From patchwork Thu Dec 7 17:39:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483795 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BA83CC10DC3 for ; Thu, 7 Dec 2023 18:51:05 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJRz-0003Ta-F5; Thu, 07 Dec 2023 13:50:19 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJRx-0003T8-OX for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:17 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJRv-0006Dv-VO for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:17 -0500 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7IDQpl023130; Thu, 7 Dec 2023 18:50:12 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=WV4sUfF5I3w3RvpHyGhWljID/vuAV76E/h7M5U6kv78=; b=GlRaKOyfLefvqxkVBaT7fPwPSab9JTMhNzlZBUZHTbzT9djiYxjlAidhHryRGr6oiKVT MclRRAdD2dyArZ/ScTHQ9oWVka4HhMaMF8pWWbrMWAF8qxO31uCCNbV1hkt3hPP43zOo GJ3zqw6gSN4OvxqKChxitPwbJ2VWwFpzdBwOGTi22GCh4TPz0JgOA2s0I/JZX08NTXJf Z/QddeV6onWCUgN6g1XMiyWUwVzbIPV3Qpxvb7zCnMyfI2ElsJlaaRMz42Ki5645kRyF ccOGXiU+eVvIe+vb1UmMQ8mAGxW6Q6a+SvHvmctIJjEaZAdcyS8GMmtzyPosTbVM4ePs cw== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utdmbmns8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:12 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7Hb9Hc039550; Thu, 7 Dec 2023 18:50:11 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vh4g-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:11 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io9xs008067; Thu, 7 Dec 2023 18:50:10 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-2; Thu, 07 Dec 2023 18:50:10 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 01/40] linux-headers: add vhost_types.h and vhost.h Date: Thu, 7 Dec 2023 09:39:14 -0800 Message-Id: <1701970793-6865-2-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-ORIG-GUID: 15U65VgEl2kNN_4GM6IkKHJ_tpagXkKs X-Proofpoint-GUID: 15U65VgEl2kNN_4GM6IkKHJ_tpagXkKs Received-SPF: pass client-ip=205.220.177.32; envelope-from=si-wei.liu@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Signed-off-by: Si-Wei Liu --- include/standard-headers/linux/vhost_types.h | 13 +++++++++++++ linux-headers/linux/vhost.h | 9 +++++++++ 2 files changed, 22 insertions(+) diff --git a/include/standard-headers/linux/vhost_types.h b/include/standard-headers/linux/vhost_types.h index 5ad07e1..c39199b 100644 --- a/include/standard-headers/linux/vhost_types.h +++ b/include/standard-headers/linux/vhost_types.h @@ -185,5 +185,18 @@ struct vhost_vdpa_iova_range { * DRIVER_OK */ #define VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK 0x6 +/* Device can be resumed */ +#define VHOST_BACKEND_F_RESUME 0x5 +/* Device supports the driver enabling virtqueues both before and after + * DRIVER_OK + */ +#define VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK 0x6 +/* Device may expose the virtqueue's descriptor area, driver area and + * device area to a different group for ASID binding than where its + * buffers may reside. Requires VHOST_BACKEND_F_IOTLB_ASID. + */ +#define VHOST_BACKEND_F_DESC_ASID 0x7 +/* IOTLB don't flush memory mapping across device reset */ +#define VHOST_BACKEND_F_IOTLB_PERSIST 0x8 #endif diff --git a/linux-headers/linux/vhost.h b/linux-headers/linux/vhost.h index f5c48b6..c61c687 100644 --- a/linux-headers/linux/vhost.h +++ b/linux-headers/linux/vhost.h @@ -219,4 +219,13 @@ */ #define VHOST_VDPA_RESUME _IO(VHOST_VIRTIO, 0x7E) +/* Get the dedicated group for the descriptor table of a virtqueue: + * read index, write group in num. + * The virtqueue index is stored in the index field of vhost_vring_state. + * The group id for the descriptor table of this specific virtqueue + * is returned via num field of vhost_vring_state. + */ +#define VHOST_VDPA_GET_VRING_DESC_GROUP _IOWR(VHOST_VIRTIO, 0x7F, \ + struct vhost_vring_state) + #endif From patchwork Thu Dec 7 17:39:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483794 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9FF7FC46CA0 for ; Thu, 7 Dec 2023 18:51:05 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJSA-0003X9-Th; Thu, 07 Dec 2023 13:50:31 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJRy-0003TS-Vb for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:19 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJRx-0006Lr-Cu for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:18 -0500 Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7FDI5Y009711; Thu, 7 Dec 2023 18:50:14 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=TrIoq3vUpKqQshngqQYAQZZona36DrqAj6vqchuF/wo=; b=KL3i9z0Dolluz6H13ajkTJhmcAgaEmtrjZFMPQzJzaxyzaOtvFTZEF4oV30RBXq2bvMo I+wbYkbNaDj4KmWLxX5iu5eTf1AdwPyJ3q3FC31JJlCb0rm3+Y0YUg/96TgixKHo/0Hi 6HMP4H4+TRBCR1j0CrnkVMcQkUxj7TEv4Jn9BZ8luT83B4VXTTMuugynaDYsuleSnaAg lCGGxmq4VfeA9oICx9hKUlyShQqtEDOiSxRNSQjlEQW0zJj6LleQN/3xOqCvgJbFfJ/X lenWTV2Sa0Anp3EQTBeRAb8fdlaQCITHo1LD/6f/EbXDB6SA2HdGABqknmVLMAg0+nJe 5Q== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utdrvmpev-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:14 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7IUvxb039487; Thu, 7 Dec 2023 18:50:13 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vh5k-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:13 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io9xu008067; Thu, 7 Dec 2023 18:50:12 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-3; Thu, 07 Dec 2023 18:50:12 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 02/40] vdpa: add vhost_vdpa_get_vring_desc_group Date: Thu, 7 Dec 2023 09:39:15 -0800 Message-Id: <1701970793-6865-3-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-GUID: bmLNHMZTQo4Ac7POoz9aONmEpHuNHn7K X-Proofpoint-ORIG-GUID: bmLNHMZTQo4Ac7POoz9aONmEpHuNHn7K Received-SPF: pass client-ip=205.220.177.32; envelope-from=si-wei.liu@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Internal API to get the descriptor group index for a specific virtqueue through the VHOST_VDPA_GET_VRING_DESC_GROUP ioctl. Signed-off-by: Si-Wei Liu Acked-by: Jason Wang --- net/vhost-vdpa.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c index 90f4128..887c329 100644 --- a/net/vhost-vdpa.c +++ b/net/vhost-vdpa.c @@ -471,6 +471,25 @@ static int64_t vhost_vdpa_get_vring_group(int device_fd, unsigned vq_index, return state.num; } +static int64_t vhost_vdpa_get_vring_desc_group(int device_fd, + unsigned vq_index, + Error **errp) +{ + struct vhost_vring_state state = { + .index = vq_index, + }; + int r = ioctl(device_fd, VHOST_VDPA_GET_VRING_DESC_GROUP, &state); + + if (unlikely(r < 0)) { + r = -errno; + error_setg_errno(errp, errno, "Cannot get VQ %u descriptor group", + vq_index); + return r; + } + + return state.num; +} + static int vhost_vdpa_set_address_space_id(struct vhost_vdpa *v, unsigned vq_group, unsigned asid_num) From patchwork Thu Dec 7 17:39:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483806 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 64EFFC10F05 for ; Thu, 7 Dec 2023 18:52:38 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJSI-0003Ys-4M; Thu, 07 Dec 2023 13:50:38 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJS6-0003U7-JB for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:28 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJS0-0006Xt-DC for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:24 -0500 Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7InWTp000621; Thu, 7 Dec 2023 18:50:17 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=BqSj62+diHUsR87x+1wTb6AuDlUE/ZCVpHyqeDC2E4E=; b=Ii27jVM2QxmN8r8GV7zmQ12Y8VH6dWmxM+ZrDYAqjRRu+pqkSCmNwW/51VVUhd8AMY2O akMycakkvLA9FdZDlF7gxZuwPZQ5f/KjsWJWcRvcNK4nDtr8V6YLvVeU97dnR/UNLvJ/ LEUpMP6Tm+utEAd9jhtKPCPZCs4PcE8q1N2meerF9IXkMkDJLoFhr92+z030uVv7GZ45 noRG091o6W5YU91bVyvVO9yY4OqTXX7pZxP5nUAllDpZxMSCW6vaRpzt1gLbWgolc2aH NipgGh76FloBYs2164BOY8jIVkMmPIUzzgHCAVvInJItLfZ6FGt59Ey1DND9c7TqB8Lr 6g== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utd0mmh4b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:16 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7IltKg039527; Thu, 7 Dec 2023 18:50:16 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vh7d-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:16 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io9xw008067; Thu, 7 Dec 2023 18:50:15 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-4; Thu, 07 Dec 2023 18:50:15 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 03/40] vdpa: probe descriptor group index for data vqs Date: Thu, 7 Dec 2023 09:39:16 -0800 Message-Id: <1701970793-6865-4-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-GUID: sr7q8q4OSq04SprcZhsiDUxgdZbDOoRP X-Proofpoint-ORIG-GUID: sr7q8q4OSq04SprcZhsiDUxgdZbDOoRP Received-SPF: pass client-ip=205.220.165.32; envelope-from=si-wei.liu@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Getting it ahead at initialization time instead of start time allows decision making independent of device status, while reducing failure possibility in starting device or during migration. Adding function vhost_vdpa_probe_desc_group() for that end. This function will be used to probe the descriptor group for data vqs. Signed-off-by: Si-Wei Liu --- net/vhost-vdpa.c | 89 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+) diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c index 887c329..0cf3147 100644 --- a/net/vhost-vdpa.c +++ b/net/vhost-vdpa.c @@ -1688,6 +1688,95 @@ out: return r; } +static int vhost_vdpa_probe_desc_group(int device_fd, uint64_t features, + int vq_index, int64_t *desc_grpidx, + Error **errp) +{ + uint64_t backend_features; + int64_t vq_group, desc_group; + uint8_t saved_status = 0; + uint8_t status = 0; + int r; + + ERRP_GUARD(); + + r = ioctl(device_fd, VHOST_GET_BACKEND_FEATURES, &backend_features); + if (unlikely(r < 0)) { + error_setg_errno(errp, errno, "Cannot get vdpa backend_features"); + return r; + } + + if (!(backend_features & BIT_ULL(VHOST_BACKEND_F_IOTLB_ASID))) { + return 0; + } + + if (!(backend_features & BIT_ULL(VHOST_BACKEND_F_DESC_ASID))) { + return 0; + } + + r = ioctl(device_fd, VHOST_VDPA_GET_STATUS, &saved_status); + if (unlikely(r)) { + error_setg_errno(errp, -r, "Cannot get device status"); + goto out; + } + + r = ioctl(device_fd, VHOST_VDPA_SET_STATUS, &status); + if (unlikely(r)) { + error_setg_errno(errp, -r, "Cannot reset device"); + goto out; + } + + r = ioctl(device_fd, VHOST_SET_FEATURES, &features); + if (unlikely(r)) { + error_setg_errno(errp, errno, "Cannot set features"); + } + + status = VIRTIO_CONFIG_S_ACKNOWLEDGE | + VIRTIO_CONFIG_S_DRIVER | + VIRTIO_CONFIG_S_FEATURES_OK; + + r = ioctl(device_fd, VHOST_VDPA_SET_STATUS, &status); + if (unlikely(r)) { + error_setg_errno(errp, -r, "Cannot set device status"); + goto out; + } + + vq_group = vhost_vdpa_get_vring_group(device_fd, vq_index, errp); + if (unlikely(vq_group < 0)) { + if (vq_group != -ENOTSUP) { + r = vq_group; + goto out; + } + + /* + * The kernel report VHOST_BACKEND_F_IOTLB_ASID if the vdpa frontend + * support ASID even if the parent driver does not. + */ + error_free(*errp); + *errp = NULL; + r = 0; + goto out; + } + + desc_group = vhost_vdpa_get_vring_desc_group(device_fd, vq_index, + errp); + if (unlikely(desc_group < 0)) { + r = desc_group; + goto out; + } else if (desc_group != vq_group) { + *desc_grpidx = desc_group; + } + r = 1; + +out: + status = 0; + ioctl(device_fd, VHOST_VDPA_SET_STATUS, &status); + if (saved_status) { + ioctl(device_fd, VHOST_VDPA_SET_STATUS, &saved_status); + } + return r; +} + static NetClientState *net_vhost_vdpa_init(NetClientState *peer, const char *device, const char *name, From patchwork Thu Dec 7 17:39:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483796 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9B722C10F05 for ; Thu, 7 Dec 2023 18:51:05 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJSC-0003XO-JO; Thu, 07 Dec 2023 13:50:32 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJS5-0003U6-8t for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:27 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJS0-0006Z3-DK for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:24 -0500 Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7HY0pW009542; Thu, 7 Dec 2023 18:50:19 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=PXaBos3iadjASSJZj10OoTka/EF5tVh7f2/9jG/IaBk=; b=QIomfy/rCMx5KAtw18PQeVo/HP37wR3SPJZ9wAymwaedglXoFqYf+cU5nH+7BTalpuD7 hjWwKOSTitIru0/IkRb/85U/1BqHg3PwiGL6XVZgqqs/5/o9Fam43UdmX32/nD6qm2OR zGQSlvScJiHTp8hulAD9vzISn03DBNPREXGIx+RFjRx7ahR56duC3B4WR2lYEHWaF3Xh osHxzhNZ+vrC9oe+hVQZire88BLpjXKMCQ2LmCFVqiW4TcTj9tKwoO9V+ESXu7eakQNd xTr84XVhawd80Nk3r21Bxb593kvbFyQYFtR+033slTYXSum1l/x0uKeWAxOComx7wjfB Ww== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utd1gcshf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:18 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7HQKhr039445; Thu, 7 Dec 2023 18:50:17 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vh90-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:17 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io900008067; Thu, 7 Dec 2023 18:50:17 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-5; Thu, 07 Dec 2023 18:50:17 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 04/40] vdpa: piggyback desc_group index when probing isolated cvq Date: Thu, 7 Dec 2023 09:39:17 -0800 Message-Id: <1701970793-6865-5-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-ORIG-GUID: h2pRchlMPATXGGR-NWKj99Zcx7z4f-Fx X-Proofpoint-GUID: h2pRchlMPATXGGR-NWKj99Zcx7z4f-Fx Received-SPF: pass client-ip=205.220.177.32; envelope-from=si-wei.liu@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Same as the previous commit, but do it for cvq instead of data vqs. Signed-off-by: Si-Wei Liu --- net/vhost-vdpa.c | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c index 0cf3147..cb5705d 100644 --- a/net/vhost-vdpa.c +++ b/net/vhost-vdpa.c @@ -1601,16 +1601,19 @@ static const VhostShadowVirtqueueOps vhost_vdpa_net_svq_ops = { }; /** - * Probe if CVQ is isolated + * Probe if CVQ is isolated, and piggyback its descriptor group + * index if supported * * @device_fd The vdpa device fd * @features Features offered by the device. * @cvq_index The control vq pair index + * @desc_grpidx The CVQ's descriptor group index to return * - * Returns <0 in case of failure, 0 if false and 1 if true. + * Returns <0 in case of failure, 0 if false and 1 if true (isolated). */ static int vhost_vdpa_probe_cvq_isolation(int device_fd, uint64_t features, - int cvq_index, Error **errp) + int cvq_index, int64_t *desc_grpidx, + Error **errp) { uint64_t backend_features; int64_t cvq_group; @@ -1667,6 +1670,13 @@ static int vhost_vdpa_probe_cvq_isolation(int device_fd, uint64_t features, goto out; } + if (backend_features & BIT_ULL(VHOST_BACKEND_F_DESC_ASID)) { + int64_t desc_group = vhost_vdpa_get_vring_desc_group(device_fd, + cvq_index, errp); + if (likely(desc_group >= 0) && desc_group != cvq_group) + *desc_grpidx = desc_group; + } + for (int i = 0; i < cvq_index; ++i) { int64_t group = vhost_vdpa_get_vring_group(device_fd, i, errp); if (unlikely(group < 0)) { @@ -1685,6 +1695,8 @@ static int vhost_vdpa_probe_cvq_isolation(int device_fd, uint64_t features, out: status = 0; ioctl(device_fd, VHOST_VDPA_SET_STATUS, &status); + status = VIRTIO_CONFIG_S_ACKNOWLEDGE | VIRTIO_CONFIG_S_DRIVER; + ioctl(device_fd, VHOST_VDPA_SET_STATUS, &status); return r; } @@ -1791,6 +1803,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer, Error **errp) { NetClientState *nc = NULL; + int64_t desc_group = -1; VhostVDPAState *s; int ret = 0; assert(name); @@ -1802,7 +1815,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer, } else { cvq_isolated = vhost_vdpa_probe_cvq_isolation(vdpa_device_fd, features, queue_pair_index * 2, - errp); + &desc_group, errp); if (unlikely(cvq_isolated < 0)) { return NULL; } From patchwork Thu Dec 7 17:39:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483805 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 921DEC4167B for ; Thu, 7 Dec 2023 18:52:34 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJSJ-0003ZM-5t; Thu, 07 Dec 2023 13:50:39 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJS6-0003U8-J2 for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:28 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJS1-0006bb-Jd for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:25 -0500 Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7EtISZ017262; Thu, 7 Dec 2023 18:50:20 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=VXUEwdiXIGYALuEd5KfW5mDWOK4SrCBkiWVuRQsjMa4=; b=QgvsHVMwQspdOf1YmV914EB0wo3+2FbdlN4orY7UmijBuCCAyc6/vkAitd1pllzL1xRE dEXeM0dXqds0nD8vif6fPLhiKhnkUDQ5tfMxRtJx7Ypz8X+GRql8eiuE27Y4mFLf9T97 NYl9RoRclTFIWdk1h9peOZUD0HOl3c5/uncZlFiwR/BDSzzcOCkea0NsFFMNte0pXWgN njc61K/SuDFuWWck+2Xl6UXR09YCtOyOAMCy/PfnxhGNCX97ACPBTnL2k/efLzEP7MJ8 rn3xfQW6GWbmwY/SFFfycvP+g/DEvZioMz1y4tHnJBYUFmeFVu5dBiUHwhC8kggMtZSz Gg== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utdrvmpf0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:20 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7Hb9ff039324; Thu, 7 Dec 2023 18:50:19 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vhap-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:19 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io902008067; Thu, 7 Dec 2023 18:50:18 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-6; Thu, 07 Dec 2023 18:50:18 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 05/40] vdpa: populate desc_group from net_vhost_vdpa_init Date: Thu, 7 Dec 2023 09:39:18 -0800 Message-Id: <1701970793-6865-6-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-GUID: xUPjONcvcF8ZQ082lNdJc9WCRsW0WXh5 X-Proofpoint-ORIG-GUID: xUPjONcvcF8ZQ082lNdJc9WCRsW0WXh5 Received-SPF: pass client-ip=205.220.177.32; envelope-from=si-wei.liu@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Add the desc_group field to struct vhost_vdpa, and get it populated when the corresponding vq is initialized at net_vhost_vdpa_init. If the vq does not have descriptor group capability, or it doesn't have a dedicated ASID group to host descriptors other than the data buffers, desc_group will be set to a negative value -1. Signed-off-by: Si-Wei Liu --- include/hw/virtio/vhost-vdpa.h | 1 + net/vhost-vdpa.c | 15 +++++++++++++-- 2 files changed, 14 insertions(+), 2 deletions(-) diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h index 6533ad2..63493ff 100644 --- a/include/hw/virtio/vhost-vdpa.h +++ b/include/hw/virtio/vhost-vdpa.h @@ -87,6 +87,7 @@ typedef struct vhost_vdpa { Error *migration_blocker; VhostVDPAHostNotifier notifier[VIRTIO_QUEUE_MAX]; IOMMUNotifier n; + int64_t desc_group; } VhostVDPA; int vhost_vdpa_get_iova_range(int fd, struct vhost_vdpa_iova_range *iova_range); diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c index cb5705d..1a738b2 100644 --- a/net/vhost-vdpa.c +++ b/net/vhost-vdpa.c @@ -1855,11 +1855,22 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer, ret = vhost_vdpa_add(nc, (void *)&s->vhost_vdpa, queue_pair_index, nvqs); if (ret) { - qemu_del_net_client(nc); - return NULL; + goto err; } + if (is_datapath) { + ret = vhost_vdpa_probe_desc_group(vdpa_device_fd, features, + 0, &desc_group, errp); + if (unlikely(ret < 0)) { + goto err; + } + } + s->vhost_vdpa.desc_group = desc_group; return nc; + +err: + qemu_del_net_client(nc); + return NULL; } static int vhost_vdpa_get_features(int fd, uint64_t *features, Error **errp) From patchwork Thu Dec 7 17:39:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483862 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6F7B9C10DCE for ; Thu, 7 Dec 2023 18:58:23 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJSK-0003ah-Ty; Thu, 07 Dec 2023 13:50:41 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJS6-0003UD-Pw for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:28 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJS4-0006c6-C7 for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:26 -0500 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7IJEpo006221; Thu, 7 Dec 2023 18:50:22 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=bYCoqd8Qwye+lfpvk+32JEchmO62alasv1pCQMgpYZI=; b=J6LYaBTF6FrqBs4x4Fb3zO6ZJLtUBPBnyzswuVdFpsNgycOLFEKnNq1jZA/S9emT9wQj M9mv1xJwlh2uh62CtaDIe0Lw2+lCQDQ+IcNPItenDqO9TxlQtoCzG80zvhdwRu5+9gnH ZHg0WL3ZD3WGsdWVihekLjyGEu1iQmR3B/hjHVFrj64budv4i4q45fh3lbOharbLTtIv BHaKWRQqlkFCPK2YRkd6yhn34FZFxCN4Q+9x+GA8iCaE7jhcaLb7Y6hBGu0cF2DdQf10 pTxv3ksvArQEIk/XCJ27xEpOWgnukrmGY+9HWL22qBu33wYBDAcv+uCbOCXP60lAdrj/ 1A== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utdabvgde-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:21 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7I1di0039548; Thu, 7 Dec 2023 18:50:20 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vhch-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:20 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io904008067; Thu, 7 Dec 2023 18:50:20 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-7; Thu, 07 Dec 2023 18:50:20 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 06/40] vhost: make svq work with gpa without iova translation Date: Thu, 7 Dec 2023 09:39:19 -0800 Message-Id: <1701970793-6865-7-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-ORIG-GUID: ZPTuHOVey7VmsYHA5_lXz1rTlTS7gSQ9 X-Proofpoint-GUID: ZPTuHOVey7VmsYHA5_lXz1rTlTS7gSQ9 Received-SPF: pass client-ip=205.220.177.32; envelope-from=si-wei.liu@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Make vhost_svq_vring_write_descs able to work with GPA directly without going through iova tree for translation. This will be needed in the next few patches where the SVQ has dedicated address space to host its virtqueues. Instead of having to translate qemu's VA to IOVA via the iova tree, with dedicated or isolated address space for SVQ descriptors, the IOVA is exactly same as the guest GPA space where translation would not be needed any more. Signed-off-by: Si-Wei Liu Reviewed-by: Eugenio Pérez --- hw/virtio/vhost-shadow-virtqueue.c | 35 +++++++++++++++++++++++------------ 1 file changed, 23 insertions(+), 12 deletions(-) diff --git a/hw/virtio/vhost-shadow-virtqueue.c b/hw/virtio/vhost-shadow-virtqueue.c index fc5f408..97ccd45 100644 --- a/hw/virtio/vhost-shadow-virtqueue.c +++ b/hw/virtio/vhost-shadow-virtqueue.c @@ -136,8 +136,8 @@ static bool vhost_svq_translate_addr(const VhostShadowVirtqueue *svq, * Return true if success, false otherwise and print error. */ static bool vhost_svq_vring_write_descs(VhostShadowVirtqueue *svq, hwaddr *sg, - const struct iovec *iovec, size_t num, - bool more_descs, bool write) + const struct iovec *iovec, hwaddr *addr, + size_t num, bool more_descs, bool write) { uint16_t i = svq->free_head, last = svq->free_head; unsigned n; @@ -149,8 +149,15 @@ static bool vhost_svq_vring_write_descs(VhostShadowVirtqueue *svq, hwaddr *sg, return true; } - ok = vhost_svq_translate_addr(svq, sg, iovec, num); - if (unlikely(!ok)) { + if (svq->iova_tree) { + ok = vhost_svq_translate_addr(svq, sg, iovec, num); + if (unlikely(!ok)) { + return false; + } + } else if (!addr) { + qemu_log_mask(LOG_GUEST_ERROR, + "No translation found for vaddr 0x%p\n", + iovec[0].iov_base); return false; } @@ -161,7 +168,7 @@ static bool vhost_svq_vring_write_descs(VhostShadowVirtqueue *svq, hwaddr *sg, } else { descs[i].flags = flags; } - descs[i].addr = cpu_to_le64(sg[n]); + descs[i].addr = cpu_to_le64(svq->iova_tree ? sg[n] : addr[n]); descs[i].len = cpu_to_le32(iovec[n].iov_len); last = i; @@ -173,9 +180,10 @@ static bool vhost_svq_vring_write_descs(VhostShadowVirtqueue *svq, hwaddr *sg, } static bool vhost_svq_add_split(VhostShadowVirtqueue *svq, - const struct iovec *out_sg, size_t out_num, - const struct iovec *in_sg, size_t in_num, - unsigned *head) + const struct iovec *out_sg, hwaddr *out_addr, + size_t out_num, + const struct iovec *in_sg, hwaddr *in_addr, + size_t in_num, unsigned *head) { unsigned avail_idx; vring_avail_t *avail = svq->vring.avail; @@ -191,13 +199,14 @@ static bool vhost_svq_add_split(VhostShadowVirtqueue *svq, return false; } - ok = vhost_svq_vring_write_descs(svq, sgs, out_sg, out_num, in_num > 0, - false); + ok = vhost_svq_vring_write_descs(svq, sgs, out_sg, out_addr, out_num, + in_num > 0, false); if (unlikely(!ok)) { return false; } - ok = vhost_svq_vring_write_descs(svq, sgs, in_sg, in_num, false, true); + ok = vhost_svq_vring_write_descs(svq, sgs, in_sg, in_addr, in_num, + false, true); if (unlikely(!ok)) { return false; } @@ -258,7 +267,9 @@ int vhost_svq_add(VhostShadowVirtqueue *svq, const struct iovec *out_sg, return -ENOSPC; } - ok = vhost_svq_add_split(svq, out_sg, out_num, in_sg, in_num, &qemu_head); + ok = vhost_svq_add_split(svq, out_sg, elem ? elem->out_addr : NULL, + out_num, in_sg, elem ? elem->in_addr : NULL, + in_num, &qemu_head); if (unlikely(!ok)) { return -EINVAL; } From patchwork Thu Dec 7 17:39:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483824 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D127DC46CA0 for ; Thu, 7 Dec 2023 18:57:00 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJSD-0003Xn-1s; Thu, 07 Dec 2023 13:50:33 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJS8-0003Vc-Ol for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:28 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJS4-0006cB-Tt for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:28 -0500 Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7IhZjg029893; Thu, 7 Dec 2023 18:50:23 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=PGF2vwqC9zOw5FzhQrsgKA+MWTLi1jrbI3MPbsVTPJs=; b=ejwpsie+fYnCTtlG6cq3uEUGAeoydRVfUGNKLBKORTJ5SQbGzSE0LEwXUBWmc64SazfA MPrpCsdyA1Y9OuZH7xpl5qAp6CjQ0AYYjYW73J7tKbBoLR8bTyM9WbXsAmI44Q+gEKc8 3VVZuAZGdYVS50er2n4OeEQ58L4RST/9DRc331YRftl6ZWy6EJy3/7wbKWivYeeQ5pTB p9ef4TzNsAlSSjtWDEZvpo5yl1dQOmItClMHVf//376Jg+Z/r6OzvbRNYk9sG8ie/+9t AYo3AkSjnmovZNk+XHYsJuA51h2pE+xsLwlGcd4CCgJMlz8RsBcuNwHmh+MehHCMdTXG qg== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utd1gcshk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:23 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7HTkJI039553; Thu, 7 Dec 2023 18:50:22 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vhe7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:22 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io906008067; Thu, 7 Dec 2023 18:50:21 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-8; Thu, 07 Dec 2023 18:50:21 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 07/40] vdpa: move around vhost_vdpa_set_address_space_id Date: Thu, 7 Dec 2023 09:39:20 -0800 Message-Id: <1701970793-6865-8-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-ORIG-GUID: YWerNjCqHu8uuacLiG32N_6fvetHPbuP X-Proofpoint-GUID: YWerNjCqHu8uuacLiG32N_6fvetHPbuP Received-SPF: pass client-ip=205.220.177.32; envelope-from=si-wei.liu@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Move it a few lines ahead to make function call easier for those before it. No funtional change involved. Signed-off-by: Si-Wei Liu Reviewed-by: Eugenio Pérez Acked-by: Jason Wang --- net/vhost-vdpa.c | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c index 1a738b2..dbfa192 100644 --- a/net/vhost-vdpa.c +++ b/net/vhost-vdpa.c @@ -335,6 +335,24 @@ static void vdpa_net_migration_state_notifier(Notifier *notifier, void *data) } } +static int vhost_vdpa_set_address_space_id(struct vhost_vdpa *v, + unsigned vq_group, + unsigned asid_num) +{ + struct vhost_vring_state asid = { + .index = vq_group, + .num = asid_num, + }; + int r; + + r = ioctl(v->shared->device_fd, VHOST_VDPA_SET_GROUP_ASID, &asid); + if (unlikely(r < 0)) { + error_report("Can't set vq group %u asid %u, errno=%d (%s)", + asid.index, asid.num, errno, g_strerror(errno)); + } + return r; +} + static void vhost_vdpa_net_data_start_first(VhostVDPAState *s) { struct vhost_vdpa *v = &s->vhost_vdpa; @@ -490,24 +508,6 @@ static int64_t vhost_vdpa_get_vring_desc_group(int device_fd, return state.num; } -static int vhost_vdpa_set_address_space_id(struct vhost_vdpa *v, - unsigned vq_group, - unsigned asid_num) -{ - struct vhost_vring_state asid = { - .index = vq_group, - .num = asid_num, - }; - int r; - - r = ioctl(v->shared->device_fd, VHOST_VDPA_SET_GROUP_ASID, &asid); - if (unlikely(r < 0)) { - error_report("Can't set vq group %u asid %u, errno=%d (%s)", - asid.index, asid.num, errno, g_strerror(errno)); - } - return r; -} - static void vhost_vdpa_cvq_unmap_buf(struct vhost_vdpa *v, void *addr) { VhostIOVATree *tree = v->shared->iova_tree; From patchwork Thu Dec 7 17:39:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483808 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0DAEFC4167B for ; Thu, 7 Dec 2023 18:52:46 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJSf-0003em-MI; Thu, 07 Dec 2023 13:51:01 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJS9-0003X0-NM for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:29 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJS6-0006id-Rm for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:29 -0500 Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7InVtB000565; Thu, 7 Dec 2023 18:50:24 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=AFnkWpXumI0VNgX4CkUuacA8JaKOefBrdVfykfxK9Es=; b=FkbouOX+lr0/aGQfIuQv0U9MnOKGISxQMtTIsCZnfgivurX+o3koo3z+yPEhmuaQdcFY kamsNbNj7xTE7QQ93xDXxmxoENK+ttvPZxZK6vxt+uFwdV3h8xAiRH5aZRDhxz0n6RHe t1KXg4IkQG/MfmJFk+9S5Wf+o9AqYPUFhbNPDabYh3/Es2o6whqsEQTyLQa7NYILpMs2 LLZCTlF306JD8LnZFkDEOMH8+QIaEnIJFkladcVnCOcOYm79i/ZJr7A21FiQfdowiM2G 7831zRg9unIiBzVxkRS3rYoxLIBF+XDiQbsqrhN4y0ej3q5KmYXFl+vnYJKvKB4NZHBf zA== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utd0mmh4k-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:24 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7IiBkc039465; Thu, 7 Dec 2023 18:50:23 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vhfr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:23 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io908008067; Thu, 7 Dec 2023 18:50:23 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-9; Thu, 07 Dec 2023 18:50:23 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 08/40] vdpa: add back vhost_vdpa_net_first_nc_vdpa Date: Thu, 7 Dec 2023 09:39:21 -0800 Message-Id: <1701970793-6865-9-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-GUID: sW8woUfvFtB6gf9_u20fckzmqpo6W94W X-Proofpoint-ORIG-GUID: sW8woUfvFtB6gf9_u20fckzmqpo6W94W Received-SPF: pass client-ip=205.220.165.32; envelope-from=si-wei.liu@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Previous commits had it removed. Now adding it back because this function will be needed by next patches. Signed-off-by: Si-Wei Liu Reviewed-by: Eugenio Pérez --- net/vhost-vdpa.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c index dbfa192..c9bfc6f 100644 --- a/net/vhost-vdpa.c +++ b/net/vhost-vdpa.c @@ -287,6 +287,16 @@ static ssize_t vhost_vdpa_receive(NetClientState *nc, const uint8_t *buf, return size; } + +/** From any vdpa net client, get the netclient of the first queue pair */ +static VhostVDPAState *vhost_vdpa_net_first_nc_vdpa(VhostVDPAState *s) +{ + NICState *nic = qemu_get_nic(s->nc.peer); + NetClientState *nc0 = qemu_get_peer(nic->ncs, 0); + + return DO_UPCAST(VhostVDPAState, nc, nc0); +} + static void vhost_vdpa_net_log_global_enable(VhostVDPAState *s, bool enable) { struct vhost_vdpa *v = &s->vhost_vdpa; @@ -566,7 +576,7 @@ dma_map_err: static int vhost_vdpa_net_cvq_start(NetClientState *nc) { - VhostVDPAState *s; + VhostVDPAState *s, *s0; struct vhost_vdpa *v; int64_t cvq_group; int r; @@ -577,7 +587,8 @@ static int vhost_vdpa_net_cvq_start(NetClientState *nc) s = DO_UPCAST(VhostVDPAState, nc, nc); v = &s->vhost_vdpa; - v->shadow_vqs_enabled = v->shared->shadow_data; + s0 = vhost_vdpa_net_first_nc_vdpa(s); + v->shadow_vqs_enabled = s0->vhost_vdpa.shadow_vqs_enabled; s->vhost_vdpa.address_space_id = VHOST_VDPA_GUEST_PA_ASID; if (v->shared->shadow_data) { From patchwork Thu Dec 7 17:39:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483797 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A2800C46CA3 for ; Thu, 7 Dec 2023 18:51:05 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJSJ-0003aE-Ql; Thu, 07 Dec 2023 13:50:40 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSA-0003XC-JC for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:30 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJS8-0006p2-RW for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:30 -0500 Received: from pps.filterd (m0333521.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7HFTxO007979; Thu, 7 Dec 2023 18:50:26 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=twV6mGc+nRkfELdaEpOqp368u6C8dpOt4AY5JIu7iLY=; b=lstrWVEIEgBeg9R9/CfuziQOw5skbokGBSRzp6tkt0MkomJH3hJent7GA78cNjrkFTC0 +9M5YNkmdMT25HemCfRzozvFueeXwM2Ss4jLofSFY83IG4g0+c0Q31Z0BrEpuRYwQ6JR 9/jdtFyIfW39tIxwJXH4LbZUjgcFl0qKjdGx0y1jqEpD9Rl2KA3MryUn+qV+gtzd24gx dGgT+ETovTckEKbaEoIFTBkr5wQMu58fzjQV+kOvsclRNSlUwiZlaWhOjJ8naahbok+z MucjJXsEK/Q94zkvb7+yHy3ahWLbG5FEkroGJS/jKmV5gaYrHKIyoH+ZZbRDndpVOFbK Yw== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utd0hmjt8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:25 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7HhN4P039752; Thu, 7 Dec 2023 18:50:25 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vhh4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:25 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io90A008067; Thu, 7 Dec 2023 18:50:24 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-10; Thu, 07 Dec 2023 18:50:24 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 09/40] vdpa: no repeat setting shadow_data Date: Thu, 7 Dec 2023 09:39:22 -0800 Message-Id: <1701970793-6865-10-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=947 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-GUID: cHp_8aKC8Y4Ixecv0cnkyEkjc74aR3XW X-Proofpoint-ORIG-GUID: cHp_8aKC8Y4Ixecv0cnkyEkjc74aR3XW Received-SPF: pass client-ip=205.220.165.32; envelope-from=si-wei.liu@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Since shadow_data is now shared in the parent data struct, it just needs to be set only once by the first vq. This change will make shadow_data independent of svq enabled state, which can be optionally turned off when SVQ descritors and device driver areas are all isolated to a separate address space. Signed-off-by: Si-Wei Liu Reviewed-by: Eugenio Pérez Acked-by: Jason Wang --- net/vhost-vdpa.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c index c9bfc6f..2555897 100644 --- a/net/vhost-vdpa.c +++ b/net/vhost-vdpa.c @@ -387,13 +387,12 @@ static int vhost_vdpa_net_data_start(NetClientState *nc) if (s->always_svq || migration_is_setup_or_active(migrate_get_current()->state)) { v->shadow_vqs_enabled = true; - v->shared->shadow_data = true; } else { v->shadow_vqs_enabled = false; - v->shared->shadow_data = false; } if (v->index == 0) { + v->shared->shadow_data = v->shadow_vqs_enabled; vhost_vdpa_net_data_start_first(s); return 0; } From patchwork Thu Dec 7 17:39:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483802 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1F667C4167B for ; Thu, 7 Dec 2023 18:52:22 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJSJ-0003ZT-6p; Thu, 07 Dec 2023 13:50:39 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSB-0003XM-Ba for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:31 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJS9-0006pB-3M for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:30 -0500 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7IJWaO006793; Thu, 7 Dec 2023 18:50:27 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=1qvAaetJbXbQEOsr+WeZTRGtI7xnUOGqRF8vnQO8kGM=; b=T77LF37L05otH+CIttKOwHPIxTBLkd3+rBS5EGqHCaRm3kCna9qtGu+QU4rkY6Y1Se1h 55uwT6HVhYu7jL/msADehEWlkUF+TaXOvz0FAw2ZRUqvE618pDng2z7RT4ffxf0zr0xT rmkPq4DZvrkdZTbXoPTALXqfLNzAscj7+cRfALKsz5SajZk3QeS+RzwwPOxyd1HL3wXT R5IEOl2ZsufuX+0r8d2lfjkwzJ6u1P2ZlkY3f2fnCPgjy0anRTINbNIbOaUKeP75bv7b TZjLEfSItyFyD+zd/WIQTqXlHXtVXLgJJIqzAtpK8GlnDUKIAmvVMJElxumpX/9vIyAG xQ== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utdabvgdh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:27 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7IZbmR039504; Thu, 7 Dec 2023 18:50:26 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vhjm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:26 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io90C008067; Thu, 7 Dec 2023 18:50:26 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-11; Thu, 07 Dec 2023 18:50:25 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 10/40] vdpa: assign svq descriptors a separate ASID when possible Date: Thu, 7 Dec 2023 09:39:23 -0800 Message-Id: <1701970793-6865-11-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-ORIG-GUID: bJLr1SmqAfSzL3C9-fvdQ3nlQTxtaNzC X-Proofpoint-GUID: bJLr1SmqAfSzL3C9-fvdQ3nlQTxtaNzC Received-SPF: pass client-ip=205.220.177.32; envelope-from=si-wei.liu@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org When backend supports the VHOST_BACKEND_F_DESC_ASID feature and all the data vqs can support one or more descriptor group to host SVQ vrings and descriptors, we assign them a different ASID than where its buffers reside in guest memory address space. With this dedicated ASID for SVQs, the IOVA for what vdpa device may care effectively becomes the GPA, thus there's no need to translate IOVA address. For this reason, shadow_data can be turned off accordingly. It doesn't mean the SVQ is not enabled, but just that the translation is not needed from iova tree perspective. We can reuse CVQ's address space ID to host SVQ descriptors because both CVQ and SVQ are emulated in the same QEMU process, which will share the same VA address space. Signed-off-by: Si-Wei Liu --- hw/virtio/vhost-vdpa.c | 5 ++++- net/vhost-vdpa.c | 57 ++++++++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 57 insertions(+), 5 deletions(-) diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index 24844b5..30dff95 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -627,6 +627,7 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) uint64_t qemu_backend_features = 0x1ULL << VHOST_BACKEND_F_IOTLB_MSG_V2 | 0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH | 0x1ULL << VHOST_BACKEND_F_IOTLB_ASID | + 0x1ULL << VHOST_BACKEND_F_DESC_ASID | 0x1ULL << VHOST_BACKEND_F_SUSPEND; int ret; @@ -1249,7 +1250,9 @@ static bool vhost_vdpa_svqs_start(struct vhost_dev *dev) goto err; } - vhost_svq_start(svq, dev->vdev, vq, v->shared->iova_tree); + vhost_svq_start(svq, dev->vdev, vq, + v->desc_group >= 0 && v->address_space_id ? + NULL : v->shared->iova_tree); ok = vhost_vdpa_svq_map_rings(dev, svq, &addr, &err); if (unlikely(!ok)) { goto err_map; diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c index 2555897..aebaa53 100644 --- a/net/vhost-vdpa.c +++ b/net/vhost-vdpa.c @@ -366,20 +366,50 @@ static int vhost_vdpa_set_address_space_id(struct vhost_vdpa *v, static void vhost_vdpa_net_data_start_first(VhostVDPAState *s) { struct vhost_vdpa *v = &s->vhost_vdpa; + int r; migration_add_notifier(&s->migration_state, vdpa_net_migration_state_notifier); + if (!v->shadow_vqs_enabled) { + if (v->desc_group >= 0 && + v->address_space_id != VHOST_VDPA_GUEST_PA_ASID) { + vhost_vdpa_set_address_space_id(v, v->desc_group, + VHOST_VDPA_GUEST_PA_ASID); + s->vhost_vdpa.address_space_id = VHOST_VDPA_GUEST_PA_ASID; + } + return; + } + /* iova_tree may be initialized by vhost_vdpa_net_load_setup */ - if (v->shadow_vqs_enabled && !v->shared->iova_tree) { + if (!v->shared->iova_tree) { v->shared->iova_tree = vhost_iova_tree_new(v->shared->iova_range.first, v->shared->iova_range.last); } + + if (s->always_svq || v->desc_group < 0) { + return; + } + + r = vhost_vdpa_set_address_space_id(v, v->desc_group, + VHOST_VDPA_NET_CVQ_ASID); + if (unlikely(r < 0)) { + /* The other data vqs should also fall back to using the same ASID */ + s->vhost_vdpa.address_space_id = VHOST_VDPA_GUEST_PA_ASID; + return; + } + + /* No translation needed on data SVQ when descriptor group is used */ + s->vhost_vdpa.address_space_id = VHOST_VDPA_NET_CVQ_ASID; + s->vhost_vdpa.shared->shadow_data = false; + return; } static int vhost_vdpa_net_data_start(NetClientState *nc) { VhostVDPAState *s = DO_UPCAST(VhostVDPAState, nc, nc); + VhostVDPAState *s0 = vhost_vdpa_net_first_nc_vdpa(s); + struct vhost_vdpa *v = &s->vhost_vdpa; assert(nc->info->type == NET_CLIENT_DRIVER_VHOST_VDPA); @@ -397,6 +427,18 @@ static int vhost_vdpa_net_data_start(NetClientState *nc) return 0; } + if (v->desc_group >= 0 && v->desc_group != s0->vhost_vdpa.desc_group) { + unsigned asid; + asid = v->shadow_vqs_enabled ? + s0->vhost_vdpa.address_space_id : VHOST_VDPA_GUEST_PA_ASID; + if (asid != s->vhost_vdpa.address_space_id) { + vhost_vdpa_set_address_space_id(v, v->desc_group, asid); + } + s->vhost_vdpa.address_space_id = asid; + } else { + s->vhost_vdpa.address_space_id = s0->vhost_vdpa.address_space_id; + } + return 0; } @@ -603,13 +645,19 @@ static int vhost_vdpa_net_cvq_start(NetClientState *nc) return 0; } - if (!s->cvq_isolated) { + if (!s->cvq_isolated && v->desc_group < 0) { + if (s0->vhost_vdpa.shadow_vqs_enabled && + s0->vhost_vdpa.desc_group >= 0 && + s0->vhost_vdpa.address_space_id) { + v->shadow_vqs_enabled = false; + } return 0; } - cvq_group = vhost_vdpa_get_vring_group(v->shared->device_fd, + cvq_group = s->cvq_isolated ? + vhost_vdpa_get_vring_group(v->shared->device_fd, v->dev->vq_index_end - 1, - &err); + &err) : v->desc_group; if (unlikely(cvq_group < 0)) { error_report_err(err); return cvq_group; @@ -1840,6 +1888,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer, s->always_svq = svq; s->migration_state.notify = NULL; s->vhost_vdpa.shadow_vqs_enabled = svq; + s->vhost_vdpa.address_space_id = VHOST_VDPA_GUEST_PA_ASID; if (queue_pair_index == 0) { vhost_vdpa_net_valid_svq_features(features, &s->vhost_vdpa.migration_blocker); From patchwork Thu Dec 7 17:39:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483823 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C72A5C4167B for ; Thu, 7 Dec 2023 18:57:00 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJSj-0003in-45; Thu, 07 Dec 2023 13:51:05 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSC-0003Xb-SR for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:32 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSB-0006rT-2t for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:32 -0500 Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7FcRpD001073; Thu, 7 Dec 2023 18:50:29 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=DJSLEXjKaBkTa2xp+KB2j4QSquhLP8LlYyhXssczufQ=; b=nBh1trneA2dARq5m1y4hIwdG4+6llvFmMeuomaGLMPhANxVh4XYdbUIuQtbfTzR1N4oo y+ont1wz188BUegm5JejhoUqUmIryS99bLFFoWgo3GWSlXm1ZYn+adCyu0yAc5J4aQJs 5ueltAG5P/hi6XmS49/IGCnJNe1ZQo3KYylF9NQOnQNEKgwztLvc04xP0PZBoR6iMmbH KslhFIwhsBaM2UVWrF5pwQj6I7cRV5JNh0cAmLFkUVHzYkhoh5IJo2yWBvBOFa4Btn7C S5TjdXtkYaZh/Mf0dH8qa2eMS7Zuwul/XGPVfsY2Ng/nhYLag48BdBSqNBCGwCdZY+0c Nw== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utdda4h9a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:28 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7HTkJO039553; Thu, 7 Dec 2023 18:50:28 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vhky-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:28 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io90E008067; Thu, 7 Dec 2023 18:50:27 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-12; Thu, 07 Dec 2023 18:50:27 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 11/40] vdpa: factor out vhost_vdpa_last_dev Date: Thu, 7 Dec 2023 09:39:24 -0800 Message-Id: <1701970793-6865-12-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-GUID: shHTDgZARB2Axot1HvYYH6fkM8Nxt79n X-Proofpoint-ORIG-GUID: shHTDgZARB2Axot1HvYYH6fkM8Nxt79n Received-SPF: pass client-ip=205.220.165.32; envelope-from=si-wei.liu@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Generalize duplicated condition check for the last vq of vdpa device to a common function. Signed-off-by: Si-Wei Liu Reviewed-by: Eugenio Pérez Acked-by: Jason Wang --- hw/virtio/vhost-vdpa.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index 30dff95..2b1cc14 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -593,6 +593,11 @@ static bool vhost_vdpa_first_dev(struct vhost_dev *dev) return v->index == 0; } +static bool vhost_vdpa_last_dev(struct vhost_dev *dev) +{ + return dev->vq_index + dev->nvqs == dev->vq_index_end; +} + static int vhost_vdpa_get_dev_features(struct vhost_dev *dev, uint64_t *features) { @@ -1432,7 +1437,7 @@ static int vhost_vdpa_dev_start(struct vhost_dev *dev, bool started) goto out_stop; } - if (dev->vq_index + dev->nvqs != dev->vq_index_end) { + if (!vhost_vdpa_last_dev(dev)) { return 0; } @@ -1467,7 +1472,7 @@ static void vhost_vdpa_reset_status(struct vhost_dev *dev) { struct vhost_vdpa *v = dev->opaque; - if (dev->vq_index + dev->nvqs != dev->vq_index_end) { + if (!vhost_vdpa_last_dev(dev)) { return; } From patchwork Thu Dec 7 17:39:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483801 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3CED2C4167B for ; Thu, 7 Dec 2023 18:51:59 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJSW-0003bv-Ip; Thu, 07 Dec 2023 13:50:53 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSE-0003YK-D8 for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:34 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSC-0006xo-Rb for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:34 -0500 Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7FbkWE000492; Thu, 7 Dec 2023 18:50:30 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=zU7xzPRuUjddVE49divM8k5v/rSS/cjQ6pq2yfzWieA=; b=JT56Z0gDbaOJHnEo2wmNaPZnyLGwvHG2/EbeFuucQOsdeVwyxS6qbKX8wS5n/3WG4q26 o0WhoXOpbG6cBw6Qsira1ier+47bbMJD6mniv33AbLpJCrOq6kQ4woMtAuLY5sXwoV3w GCQyv8l5vW+W3yM6fYR24jCFpr05stFJ35+mY0JbjidTH3pPd8WKb10+mJvN8UguEJDe vbCfeDdm03WCHlZcd/aJSaZCVeEbnySAWdCAfW1g+nWmFasnXPVieusgWwkVg//7dDQy cU6vUolFHDbMZ0AaFnGcMQmc7+Ucyjinz0V3kS/sr5mccmTby2vxvxaw/K6lxSwzjdly XQ== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utdda4h9b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:30 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7HRJY6039470; Thu, 7 Dec 2023 18:50:29 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vhne-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:29 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io90G008067; Thu, 7 Dec 2023 18:50:28 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-13; Thu, 07 Dec 2023 18:50:28 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 12/40] vdpa: check map_thread_enabled before join maps thread Date: Thu, 7 Dec 2023 09:39:25 -0800 Message-Id: <1701970793-6865-13-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-GUID: aZH4UpDxnYV3N7kxIUvzU6ryQII79Ktx X-Proofpoint-ORIG-GUID: aZH4UpDxnYV3N7kxIUvzU6ryQII79Ktx Received-SPF: pass client-ip=205.220.165.32; envelope-from=si-wei.liu@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org The next patches will also register memory listener on demand, hence the need to differentiate the map_thread case from the rest. Signed-off-by: Si-Wei Liu --- hw/virtio/vhost-vdpa.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index 2b1cc14..4f026db 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -1450,7 +1450,7 @@ static int vhost_vdpa_dev_start(struct vhost_dev *dev, bool started) if (!v->shared->listener_registered) { memory_listener_register(&v->shared->listener, dev->vdev->dma_as); v->shared->listener_registered = true; - } else { + } else if (v->shared->map_thread_enabled) { ok = vhost_vdpa_join_maps_thread(v->shared); if (unlikely(!ok)) { goto out_stop; From patchwork Thu Dec 7 17:39:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483812 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 60BD4C4167B for ; Thu, 7 Dec 2023 18:53:55 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJSL-0003bB-VY; Thu, 07 Dec 2023 13:50:42 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSF-0003YZ-1k for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:36 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSD-0006xy-7M for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:34 -0500 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7IXRY1021815; Thu, 7 Dec 2023 18:50:32 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=BfrNOM61HQk1Refk9aUivCtpc5yjhyjOFXJvrx8vuEY=; b=dmY5cGU+ZGNtQ2mD9MOouxy7sZjUV+YMUCwfcUER27O3jIeX9I4K/UWP2CGW4G+CBEwk QTbpxzXGueAQ05hzZZJDklcl4bmlOEHaV/dIXNWALpW7Nv9jp2upHC/uLj/mSQQX9obw 9PzRcJSX34TQFgwPoBz9Y2rFw+humF0hItCYeBWJYoPhQprIK9lvQIDlOW8RTN86QVyJ q0VEGMGxRjVLTvL4pxB9Mz0M9ok80g1jxp3oZ3t/tD6fBFUG/5J75IdZt/jdVr07ppQ9 XVQdVrqRQUepOT2hoM1zU15/xF28WAMaXnVjBw9YkguxPiQSt3tleNnNSownG6LXEFdv ZQ== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utdmbmnsn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:31 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7HNN8s039498; Thu, 7 Dec 2023 18:50:30 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vhpj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:30 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io90I008067; Thu, 7 Dec 2023 18:50:30 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-14; Thu, 07 Dec 2023 18:50:30 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 13/40] vdpa: ref counting VhostVDPAShared Date: Thu, 7 Dec 2023 09:39:26 -0800 Message-Id: <1701970793-6865-14-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-ORIG-GUID: pAv6TUJR0jCfrTDP25p5yCJ6gVZ-M29W X-Proofpoint-GUID: pAv6TUJR0jCfrTDP25p5yCJ6gVZ-M29W Received-SPF: pass client-ip=205.220.177.32; envelope-from=si-wei.liu@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Subsequent patches attempt to release VhostVDPAShared resources, for example iova tree to free and memory listener to unregister, in vdpa_dev_cleanup(). Instead of checking against the vq index, which is not always available in all of the callers, counting the usage by reference. Then it'll be easy to free resource upon the last deref. Signed-off-by: Si-Wei Liu --- include/hw/virtio/vhost-vdpa.h | 2 ++ net/vhost-vdpa.c | 14 ++++++++++---- 2 files changed, 12 insertions(+), 4 deletions(-) diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h index 63493ff..7b8d3bf 100644 --- a/include/hw/virtio/vhost-vdpa.h +++ b/include/hw/virtio/vhost-vdpa.h @@ -70,6 +70,8 @@ typedef struct vhost_vdpa_shared { /* Vdpa must send shadow addresses as IOTLB key for data queues, not GPA */ bool shadow_data; + + unsigned refcnt; } VhostVDPAShared; typedef struct vhost_vdpa { diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c index aebaa53..a126e5c 100644 --- a/net/vhost-vdpa.c +++ b/net/vhost-vdpa.c @@ -236,11 +236,11 @@ static void vhost_vdpa_cleanup(NetClientState *nc) g_free(s->vhost_net); s->vhost_net = NULL; } - if (s->vhost_vdpa.index != 0) { - return; + if (--s->vhost_vdpa.shared->refcnt == 0) { + qemu_close(s->vhost_vdpa.shared->device_fd); + g_free(s->vhost_vdpa.shared); } - qemu_close(s->vhost_vdpa.shared->device_fd); - g_free(s->vhost_vdpa.shared); + s->vhost_vdpa.shared = NULL; } /** Dummy SetSteeringEBPF to support RSS for vhost-vdpa backend */ @@ -1896,6 +1896,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer, s->vhost_vdpa.shared->device_fd = vdpa_device_fd; s->vhost_vdpa.shared->iova_range = iova_range; s->vhost_vdpa.shared->shadow_data = svq; + s->vhost_vdpa.shared->refcnt++; } else if (!is_datapath) { s->cvq_cmd_out_buffer = mmap(NULL, vhost_vdpa_net_cvq_cmd_page_len(), PROT_READ | PROT_WRITE, @@ -1910,6 +1911,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer, } if (queue_pair_index != 0) { s->vhost_vdpa.shared = shared; + s->vhost_vdpa.shared->refcnt++; } ret = vhost_vdpa_add(nc, (void *)&s->vhost_vdpa, queue_pair_index, nvqs); @@ -1928,6 +1930,10 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer, return nc; err: + if (--s->vhost_vdpa.shared->refcnt == 0) { + g_free(s->vhost_vdpa.shared); + } + s->vhost_vdpa.shared = NULL; qemu_del_net_client(nc); return NULL; } From patchwork Thu Dec 7 17:39:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483822 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E4AD4C4167B for ; Thu, 7 Dec 2023 18:56:53 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJSc-0003cs-K5; Thu, 07 Dec 2023 13:50:58 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSG-0003Ym-QQ for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:37 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSE-00074A-Mv for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:36 -0500 Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7GDP6M031570; Thu, 7 Dec 2023 18:50:33 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=e0mpqwBWjWSiuQVcV0ZNzMbrYl/rSZxxr7/pXeGCxhg=; b=WKTjdpyJb9DfeGSt3HQz3MFCe+mTiE3INyI3GLxJQIor33DtlGu+KNPeL11m0JYeRKD7 wbhJ3P5qa+mFXByWHUNj1+ps5D4YNbtH6ZhxUnBKxaLk6rV72LJ3KUm0EdWSaOe32rFq MfVvzbUezhxzyQ9IH/Mg5q22FxG2G7ZOxYZNL1jeYRQhXD77aDaWy9TgTGrMXoG4Q/YA 90aJyDncdmWLGgA706U/MqG6caIaBny0Rdhly9M+OOobNUsTzTKnhJsf2W3FUMD0ykZU j8qvWLv+10p3FivS2dgu9uZ7LcOzoyFVYjAtobveClXgFBxjUsS6xTHkbEW/AjGmgB6X bA== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utdrvmpf8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:33 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7HW79B039466; Thu, 7 Dec 2023 18:50:32 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vhqp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:32 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io90K008067; Thu, 7 Dec 2023 18:50:31 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-15; Thu, 07 Dec 2023 18:50:31 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 14/40] vdpa: convert iova_tree to ref count based Date: Thu, 7 Dec 2023 09:39:27 -0800 Message-Id: <1701970793-6865-15-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-GUID: G7S8G0A10AUV155r3ZU6Dkd7LnaxHw3A X-Proofpoint-ORIG-GUID: G7S8G0A10AUV155r3ZU6Dkd7LnaxHw3A Received-SPF: pass client-ip=205.220.177.32; envelope-from=si-wei.liu@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org So that it can be freed from vhost_vdpa_cleanup on the last deref. The next few patches will try to make iova tree life cycle not depend on memory listener, and there's possiblity to keep iova tree around when memory mapping is not changed across device reset. Signed-off-by: Si-Wei Liu --- net/vhost-vdpa.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c index a126e5c..7b8f047 100644 --- a/net/vhost-vdpa.c +++ b/net/vhost-vdpa.c @@ -238,6 +238,8 @@ static void vhost_vdpa_cleanup(NetClientState *nc) } if (--s->vhost_vdpa.shared->refcnt == 0) { qemu_close(s->vhost_vdpa.shared->device_fd); + g_clear_pointer(&s->vhost_vdpa.shared->iova_tree, + vhost_iova_tree_delete); g_free(s->vhost_vdpa.shared); } s->vhost_vdpa.shared = NULL; @@ -461,19 +463,12 @@ static int vhost_vdpa_net_data_load(NetClientState *nc) static void vhost_vdpa_net_client_stop(NetClientState *nc) { VhostVDPAState *s = DO_UPCAST(VhostVDPAState, nc, nc); - struct vhost_dev *dev; assert(nc->info->type == NET_CLIENT_DRIVER_VHOST_VDPA); if (s->vhost_vdpa.index == 0) { migration_remove_notifier(&s->migration_state); } - - dev = s->vhost_vdpa.dev; - if (dev->vq_index + dev->nvqs == dev->vq_index_end) { - g_clear_pointer(&s->vhost_vdpa.shared->iova_tree, - vhost_iova_tree_delete); - } } static int vhost_vdpa_net_load_setup(NetClientState *nc, NICState *nic) From patchwork Thu Dec 7 17:39:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483821 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 53690C4167B for ; Thu, 7 Dec 2023 18:56:41 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJSc-0003ct-JA; Thu, 07 Dec 2023 13:50:58 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSJ-0003Zn-BU for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:39 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSG-00074X-Hl for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:38 -0500 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7IXRY3021815; Thu, 7 Dec 2023 18:50:35 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=xDxSIA4rvNU4sQYtYIHv8uaWbJt9DwDRsPu40Jb1QUI=; b=aX/2yZCAjmgE3OKkgp4Quteporg5ySBoF79SJleuUtReVD60iY22agclRRyjVJL/oIGM lEwExZxJSFbgiHVUm7sIyj7qSb00qr/MZps89in8FuHdIfa1tzYK2ZgpPmb0c34YV7Q8 47jP0ne2dlg+/fh/4QVouGfaP7RoO+6ZDZ2mltRLmWnT2CnvWstVMt3zcprf99ry5uh/ oRvbhRgBT5jBkrlOJ9Gx2xIWNBAbs5ATJAn2JShPYzr5rnYIam4rY61oItL/XevQ2R5j pOfJBtLTix9AlvIK6KPSeffWarcYme7BpLOArPELsvvzDyNOmgu4gT9+QkNaWxgXhedv BQ== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utdmbmnst-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:34 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7Hb9fu039324; Thu, 7 Dec 2023 18:50:33 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vhrs-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:33 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io90M008067; Thu, 7 Dec 2023 18:50:33 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-16; Thu, 07 Dec 2023 18:50:33 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 15/40] vdpa: add svq_switching and flush_map to header Date: Thu, 7 Dec 2023 09:39:28 -0800 Message-Id: <1701970793-6865-16-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-ORIG-GUID: egULmAtzU0HAE4lm2Z15uVhpElIubHmc X-Proofpoint-GUID: egULmAtzU0HAE4lm2Z15uVhpElIubHmc Received-SPF: pass client-ip=205.220.177.32; envelope-from=si-wei.liu@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Will be used in next patches. Signed-off-by: Si-Wei Liu --- include/hw/virtio/vhost-vdpa.h | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h index 7b8d3bf..0fe0f60 100644 --- a/include/hw/virtio/vhost-vdpa.h +++ b/include/hw/virtio/vhost-vdpa.h @@ -72,6 +72,12 @@ typedef struct vhost_vdpa_shared { bool shadow_data; unsigned refcnt; + + /* SVQ switching is in progress? 1: turn on SVQ, -1: turn off SVQ */ + int svq_switching; + + /* Flush mappings on reset due to shared address space */ + bool flush_map; } VhostVDPAShared; typedef struct vhost_vdpa { From patchwork Thu Dec 7 17:39:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483826 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3F219C4167B for ; Thu, 7 Dec 2023 18:57:13 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJSj-0003mf-QY; Thu, 07 Dec 2023 13:51:05 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSM-0003bD-25 for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:42 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSJ-00074q-3K for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:40 -0500 Received: from pps.filterd (m0333521.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7HFTxP007979; Thu, 7 Dec 2023 18:50:36 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=q/OD/SQMRrvfSZtobhfq0UPv1L+/kMxXzcbtd0Vkdy8=; b=g52EoJ5jXOZ34zk1SZVNWxuGqQrD2O1C0iQkwXINSzKibrZbw9dd804gBoi52VSG1tCO tBrDcsKB0FuZGPC9WSrX6l33SzBlhTSMfoap+9CO5e0gdyMMGQnonN2r+rnWS5imA1SZ xpfnmp68OPvQH5Mge1UVb1lejBg6bzX2QzjnL2EUybxJWQEXIGNTjBmuXcHkrjZD3MJw 7FR3Nv2MlLNthqMr7ZtXkjHlAA//i/CplTj4oJ0CG78KTH2t4bc+/5itGwMJHxbKKuEF v05A2uc8LiqRMrvZYc6agSDPebUYvzqDbHpagMhXq3QZaA3Xy0Bw12Ddf4yKR1DKuooy OA== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utd0hmjtj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:36 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7HW79E039466; Thu, 7 Dec 2023 18:50:35 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vhtc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:35 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io90O008067; Thu, 7 Dec 2023 18:50:34 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-17; Thu, 07 Dec 2023 18:50:34 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 16/40] vdpa: indicate SVQ switching via flag Date: Thu, 7 Dec 2023 09:39:29 -0800 Message-Id: <1701970793-6865-17-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=959 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-GUID: T7qjZS8JQ86YnXnSajyUbZQKDxKYOjEt X-Proofpoint-ORIG-GUID: T7qjZS8JQ86YnXnSajyUbZQKDxKYOjEt Received-SPF: pass client-ip=205.220.165.32; envelope-from=si-wei.liu@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org svq_switching indicates the case where SVQ mode change is on going. Positive (1) means switching from the normal passthrough mode to SVQ mode, and negative (-1) meaning switch SVQ back to the passthrough; zero (0) indicates that there's no SVQ mode switch taking place. Signed-off-by: Si-Wei Liu --- net/vhost-vdpa.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c index 7b8f047..04718b2 100644 --- a/net/vhost-vdpa.c +++ b/net/vhost-vdpa.c @@ -320,6 +320,7 @@ static void vhost_vdpa_net_log_global_enable(VhostVDPAState *s, bool enable) data_queue_pairs = n->multiqueue ? n->max_queue_pairs : 1; cvq = virtio_vdev_has_feature(vdev, VIRTIO_NET_F_CTRL_VQ) ? n->max_ncs - n->max_queue_pairs : 0; + v->shared->svq_switching = enable ? 1 : -1; /* * TODO: vhost_net_stop does suspend, get_base and reset. We can be smarter * in the future and resume the device if read-only operations between @@ -332,6 +333,7 @@ static void vhost_vdpa_net_log_global_enable(VhostVDPAState *s, bool enable) if (unlikely(r < 0)) { error_report("unable to start vhost net: %s(%d)", g_strerror(-r), -r); } + v->shared->svq_switching = 0; } static void vdpa_net_migration_state_notifier(Notifier *notifier, void *data) From patchwork Thu Dec 7 17:39:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483814 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F3D24C10F05 for ; Thu, 7 Dec 2023 18:55:41 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJSg-0003eX-KG; Thu, 07 Dec 2023 13:51:03 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSL-0003bC-VD for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:42 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSJ-000750-4N for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:40 -0500 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7IJEpq006221; Thu, 7 Dec 2023 18:50:38 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=DAJgRMVF30Cp8qjJnARfP/QyVuDoThEW/b3dYAUZDFA=; b=SsiFgE/It4NP+DNrlL/5otSYr8bt53TZF8qVdpClG0OML2pAAvwGeKN+vikiaBr02Hq1 PqZ1qTSob7MzAxguAcEkxhBvEkFGv7VuCkxciiY6vXHZJgRb04Hh3yNr53AG3rHQroa8 GsWUOCOEDRaXvf33o/LnJNZ5KCe0ZDSKkIboNtNqk7aJLxyR+sRa7hYX3o3fV3XSz/ZS wLtINid2rix7QuWAD6tvV4wXhtPc74yvc0xknhz7x0Qz8VrHNac73YVLurxPYYWY63pa e844cyzLjpvGhwh9VMXXUTxy18iDl+LDNREdLqsFv7CRsICQgaeLKkeHQlcZ/mKCwTkB Dg== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utdabvgds-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:37 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7IltL1039527; Thu, 7 Dec 2023 18:50:36 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vhup-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:36 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io90Q008067; Thu, 7 Dec 2023 18:50:36 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-18; Thu, 07 Dec 2023 18:50:36 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 17/40] vdpa: judge if map can be kept across reset Date: Thu, 7 Dec 2023 09:39:30 -0800 Message-Id: <1701970793-6865-18-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-ORIG-GUID: flPk4cZaL6HQcNu2WbdikwEePuRy0azc X-Proofpoint-GUID: flPk4cZaL6HQcNu2WbdikwEePuRy0azc Received-SPF: pass client-ip=205.220.177.32; envelope-from=si-wei.liu@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org The descriptor group for SVQ ASID allows the guest memory mapping to retain across SVQ switching, same as how isolated CVQ can do with a different ASID than the guest GPA space. Introduce an evaluation function to judge whether to flush or keep iotlb maps based on virtqueue's descriptor group and cvq isolation capability. Have to hook the evaluation function to NetClient's .poll op as .vhost_reset_status runs ahead of .stop, and .vhost_dev_start don't have access to the vhost-vdpa net's information. Signed-off-by: Si-Wei Liu --- net/vhost-vdpa.c | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c index 04718b2..e9b96ed 100644 --- a/net/vhost-vdpa.c +++ b/net/vhost-vdpa.c @@ -504,12 +504,36 @@ static int vhost_vdpa_net_load_cleanup(NetClientState *nc, NICState *nic) n->parent_obj.status & VIRTIO_CONFIG_S_DRIVER_OK); } +static void vhost_vdpa_net_data_eval_flush(NetClientState *nc, bool stop) +{ + VhostVDPAState *s = DO_UPCAST(VhostVDPAState, nc, nc); + struct vhost_vdpa *v = &s->vhost_vdpa; + + if (!stop) { + return; + } + + if (s->vhost_vdpa.index == 0) { + if (s->always_svq) { + v->shared->flush_map = true; + } else if (!v->shared->svq_switching || v->desc_group >= 0) { + v->shared->flush_map = false; + } else { + v->shared->flush_map = true; + } + } else if (!s->always_svq && v->shared->svq_switching && + v->desc_group < 0) { + v->shared->flush_map = true; + } +} + static NetClientInfo net_vhost_vdpa_info = { .type = NET_CLIENT_DRIVER_VHOST_VDPA, .size = sizeof(VhostVDPAState), .receive = vhost_vdpa_receive, .start = vhost_vdpa_net_data_start, .load = vhost_vdpa_net_data_load, + .poll = vhost_vdpa_net_data_eval_flush, .stop = vhost_vdpa_net_client_stop, .cleanup = vhost_vdpa_cleanup, .has_vnet_hdr = vhost_vdpa_has_vnet_hdr, @@ -1368,12 +1392,28 @@ static int vhost_vdpa_net_cvq_load(NetClientState *nc) return 0; } +static void vhost_vdpa_net_cvq_eval_flush(NetClientState *nc, bool stop) +{ + VhostVDPAState *s = DO_UPCAST(VhostVDPAState, nc, nc); + struct vhost_vdpa *v = &s->vhost_vdpa; + + if (!stop) { + return; + } + + if (!v->shared->flush_map && !v->shared->svq_switching && + !s->cvq_isolated && v->desc_group < 0) { + v->shared->flush_map = true; + } +} + static NetClientInfo net_vhost_vdpa_cvq_info = { .type = NET_CLIENT_DRIVER_VHOST_VDPA, .size = sizeof(VhostVDPAState), .receive = vhost_vdpa_receive, .start = vhost_vdpa_net_cvq_start, .load = vhost_vdpa_net_cvq_load, + .poll = vhost_vdpa_net_cvq_eval_flush, .stop = vhost_vdpa_net_cvq_stop, .cleanup = vhost_vdpa_cleanup, .has_vnet_hdr = vhost_vdpa_has_vnet_hdr, From patchwork Thu Dec 7 17:39:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483819 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 92EC2C4167B for ; Thu, 7 Dec 2023 18:56:07 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJSe-0003eW-L9; Thu, 07 Dec 2023 13:51:00 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSN-0003bk-S5 for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:44 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSM-00075C-6F for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:43 -0500 Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7ISPbf031783; Thu, 7 Dec 2023 18:50:39 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=GzAQNxB1Ds4Wl3bN9BqEVFr1k5fVdLUM7iPUgjnRZus=; b=SMxSJt91aeBW8blW22QdImDgwZ6oW5kCW1f6ZHdpVqCKYYje2Zbgj3R2PcA1MsNuDQqd omg45NmdnMqxXdNXRwJ+9s1cswgrcqvAyUUnVF10IBpcUmnm+luxvpjILzQ6/h25l2+1 ePBvgOQ4Z5YHDCGOcYmEhhw763XikOF1ViBDYmnX5ptAWJj4pSjqDeSMawLfBGENXhNN fprMgy7dmBZgsBxlG/FouxVC/cp2VBHlTQvZbcvVOuy7BUKnNgkcwyt3GYN8SSHNiPyX +jEtnO9DTh4YKykapZeM/R3P5admIUZXK++Ljcb7LFpaMGFIdE8f70Oq4cIWsKa09ao6 WA== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utdda4h9q-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:38 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7I27i7039472; Thu, 7 Dec 2023 18:50:38 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vhvw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:38 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io90S008067; Thu, 7 Dec 2023 18:50:37 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-19; Thu, 07 Dec 2023 18:50:37 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 18/40] vdpa: unregister listener on last dev cleanup Date: Thu, 7 Dec 2023 09:39:31 -0800 Message-Id: <1701970793-6865-19-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-GUID: PVINAj-GTltoU4gOBZQ4FLMTMhfygUVu X-Proofpoint-ORIG-GUID: PVINAj-GTltoU4gOBZQ4FLMTMhfygUVu Received-SPF: pass client-ip=205.220.165.32; envelope-from=si-wei.liu@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org So that the free of iova tree struct can be safely deferred to until the last vq referencing it goes away. Signed-off-by: Si-Wei Liu --- hw/virtio/vhost-vdpa.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index 4f026db..ea2dfc8 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -815,7 +815,10 @@ static int vhost_vdpa_cleanup(struct vhost_dev *dev) } vhost_vdpa_host_notifiers_uninit(dev, dev->nvqs); - memory_listener_unregister(&v->shared->listener); + if (vhost_vdpa_last_dev(dev) && v->shared->listener_registered) { + memory_listener_unregister(&v->shared->listener); + v->shared->listener_registered = false; + } vhost_vdpa_svq_cleanup(dev); dev->opaque = NULL; From patchwork Thu Dec 7 17:39:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483813 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C01DCC4167B for ; Thu, 7 Dec 2023 18:54:38 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJSe-0003eT-Fi; Thu, 07 Dec 2023 13:51:00 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSS-0003cC-9P for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:51 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSM-00075M-Q1 for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:44 -0500 Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7FbkWF000492; Thu, 7 Dec 2023 18:50:40 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=Ts4cQAvsSa+oeK7OSgvPKqhJMDfUGW7pypfVyDb3Mq8=; b=jSJQkpOwMOL28/CRMeHbfORlgHg9RV2YoM1Ep/lUs+U7KVeP1TMl4UZ7a0PrUSd8QyxA cVJ6OEnA0cuRuML5HtE4278ZM9GuORd9y+TE+AMSuPijwlsBhNTxoatqInydwykNuAkI wYIybStxMCNdkGwUb8fUc1Id9te3Cgp7553R80B3g/IjqY8hfGrJKr2WW2mBJHpgePkc zZLKXbU2G3+dZUVd69kKRnP8QcgnsPJAI+H5BWB6tKGNp15sP+FuChf5L11SnL/IS010 Sii7NFCuOjVnXjPmjLV15MsfTnBXc8staCFKL0+tBjns65OoaX0oOlx5LyiBGoLJaG82 yA== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utdda4h9t-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:40 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7Hb9g1039324; Thu, 7 Dec 2023 18:50:39 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vhxf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:39 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io90U008067; Thu, 7 Dec 2023 18:50:39 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-20; Thu, 07 Dec 2023 18:50:39 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 19/40] vdpa: should avoid map flushing with persistent iotlb Date: Thu, 7 Dec 2023 09:39:32 -0800 Message-Id: <1701970793-6865-20-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-GUID: VmsbE7U-VPFNiBIeLW82-JtATxW-ibte X-Proofpoint-ORIG-GUID: VmsbE7U-VPFNiBIeLW82-JtATxW-ibte Received-SPF: pass client-ip=205.220.165.32; envelope-from=si-wei.liu@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Today memory listener is unregistered in vhost_vdpa_reset_status unconditionally, due to which all the maps will be flushed away from the iotlb. However, map flush is not always needed, and doing it from performance hot path may have innegligible latency impact that affects VM reboot time or brown out period during live migration. Leverage the IOTLB_PERSIST backend featuae, which ensures durable iotlb maps and not disappearing even across reset. When it is supported, we may conditionally keep the maps for cases where the guest memory mapping doesn't change. Prepare a function so that the next patch will be able to use it to keep the maps. Signed-off-by: Si-Wei Liu --- hw/virtio/trace-events | 1 + hw/virtio/vhost-vdpa.c | 20 ++++++++++++++++++++ 2 files changed, 21 insertions(+) diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index 77905d1..9725d44 100644 --- a/hw/virtio/trace-events +++ b/hw/virtio/trace-events @@ -66,6 +66,7 @@ vhost_vdpa_set_owner(void *dev) "dev: %p" vhost_vdpa_vq_get_addr(void *dev, void *vq, uint64_t desc_user_addr, uint64_t avail_user_addr, uint64_t used_user_addr) "dev: %p vq: %p desc_user_addr: 0x%"PRIx64" avail_user_addr: 0x%"PRIx64" used_user_addr: 0x%"PRIx64 vhost_vdpa_get_iova_range(void *dev, uint64_t first, uint64_t last) "dev: %p first: 0x%"PRIx64" last: 0x%"PRIx64 vhost_vdpa_set_config_call(void *dev, int fd)"dev: %p fd: %d" +vhost_vdpa_maybe_flush_map(void *dev, bool reg, bool flush, bool persist) "dev: %p registered: %d flush_map: %d iotlb_persistent: %d" # virtio.c virtqueue_alloc_element(void *elem, size_t sz, unsigned in_num, unsigned out_num) "elem %p size %zd in_num %u out_num %u" diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index ea2dfc8..31e0a55 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -1471,6 +1471,26 @@ out_stop: return ok ? 0 : -1; } +static void vhost_vdpa_maybe_flush_map(struct vhost_dev *dev) +{ + struct vhost_vdpa *v = dev->opaque; + + trace_vhost_vdpa_maybe_flush_map(dev, v->shared->listener_registered, + v->shared->flush_map, + !!(dev->backend_cap & + BIT_ULL(VHOST_BACKEND_F_IOTLB_PERSIST))); + + if (!v->shared->listener_registered) { + return; + } + + if (!(dev->backend_cap & BIT_ULL(VHOST_BACKEND_F_IOTLB_PERSIST)) || + v->shared->flush_map) { + memory_listener_unregister(&v->shared->listener); + v->shared->listener_registered = false; + } +} + static void vhost_vdpa_reset_status(struct vhost_dev *dev) { struct vhost_vdpa *v = dev->opaque; From patchwork Thu Dec 7 17:39:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483811 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E19DAC4167B for ; Thu, 7 Dec 2023 18:53:35 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJSt-0003xI-4b; Thu, 07 Dec 2023 13:51:15 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSW-0003cS-3y for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:52 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSO-00075p-Pr for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:48 -0500 Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7HVdNu010826; Thu, 7 Dec 2023 18:50:42 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=EthSTSZAGUdFGpGUUgIS40/qu5PWf2EuzefpgEB0S8Y=; b=fiNpDcI7jkLvuU7c9McRT5f3QPpsAqnKiiZtWmucBPbpUonY3I53aERnM+56LaofjjDn rWsHB5eW2auOaBt55IpF5JNnCwXCTCCnCfHeOQcS8vCtw5sBW3wb63gktQG9hEPfDxpP wKhOC/f84hDWkLyXSW5ULbqIcm/DNnI9vw1XPJKdGADXDTzRJY+xyDyW5f9eaIHhNNqT ysoVQN0Lrn5RWJxTLNQbCqm9i8/UFtiB8stZ3DFwg1CveT4sktaAJZmtKgNbiA/mfUrM 5S0/VWIjL4w8gdQLNd5d94GSxRvc/ABOf0MDMrP6EWyfDUgFeTfJrNJed4X5472sDcmx AQ== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utdc1cg00-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:41 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7HOJfY039780; Thu, 7 Dec 2023 18:50:41 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vhyn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:41 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io90W008067; Thu, 7 Dec 2023 18:50:40 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-21; Thu, 07 Dec 2023 18:50:40 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 20/40] vdpa: avoid mapping flush across reset Date: Thu, 7 Dec 2023 09:39:33 -0800 Message-Id: <1701970793-6865-21-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-GUID: W5odj9H8iuFszJ3247q5atcCldSAVR7C X-Proofpoint-ORIG-GUID: W5odj9H8iuFszJ3247q5atcCldSAVR7C Received-SPF: pass client-ip=205.220.165.32; envelope-from=si-wei.liu@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Leverage the IOTLB_PERSIST and DESC_ASID features to achieve a slightly light weight reset path, without resorting to suspend and resume. Not as best but it offers significant time saving too, which should somehow play its role in live migration down time reduction by large. It benefits two cases: - normal virtio reset in the VM, e.g. guest reboot, don't have to tear down all iotlb mapping and set up again. - SVQ switching, in which data vq's descriptor table and vrings are moved to a different ASID than where its buffers reside. Along with the use of persistent iotlb, it would save substantial time from pinning and mapping unneccessarily when moving descriptors on to or out of shadow mode. Signed-off-by: Si-Wei Liu --- hw/virtio/vhost-vdpa.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index 31e0a55..47c764b 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -633,6 +633,7 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) 0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH | 0x1ULL << VHOST_BACKEND_F_IOTLB_ASID | 0x1ULL << VHOST_BACKEND_F_DESC_ASID | + 0x1ULL << VHOST_BACKEND_F_IOTLB_PERSIST | 0x1ULL << VHOST_BACKEND_F_SUSPEND; int ret; @@ -1493,8 +1494,6 @@ static void vhost_vdpa_maybe_flush_map(struct vhost_dev *dev) static void vhost_vdpa_reset_status(struct vhost_dev *dev) { - struct vhost_vdpa *v = dev->opaque; - if (!vhost_vdpa_last_dev(dev)) { return; } @@ -1502,9 +1501,7 @@ static void vhost_vdpa_reset_status(struct vhost_dev *dev) vhost_vdpa_reset_device(dev); vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_ACKNOWLEDGE | VIRTIO_CONFIG_S_DRIVER); - memory_listener_unregister(&v->shared->listener); - v->shared->listener_registered = false; - + vhost_vdpa_maybe_flush_map(dev); } static int vhost_vdpa_set_log_base(struct vhost_dev *dev, uint64_t base, From patchwork Thu Dec 7 17:39:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483804 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 99B77C46CA0 for ; Thu, 7 Dec 2023 18:52:34 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJSf-0003ev-W2; Thu, 07 Dec 2023 13:51:03 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSa-0003d6-9h for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:58 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSS-000768-6S for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:53 -0500 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7IJFbR006249; Thu, 7 Dec 2023 18:50:43 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=rCzHnfkwKPT4JYuH9v8Cdemma7Tb5LZVKQwQbSmCGCM=; b=S83Bwx0Sm+W8pLeWpegkjjLL2R8EriAiSKpfVWAJgkZySy0m0FzY7zHtrG2maYA6hQe7 aiSCR4alCk7KgnJ11avOT/1c3qM3HSAWPeImwTSO4WQvKzLBuhvGPYjN+1EJjhQFBjEh nQRVQP9QJQAG/PuL2TWMhFGZPugDlfMCsaNxJe6zQaRyKPxsmsJdvhJr/CPr6nI/b8vH kjLLnzMTtzkOXeAMyg0WW+nZQK+aEXF1WgOKj3jIl47qrylZ0LDst/4aq/NJlOkaGWE2 u5qhfNtumh6f/Gc6buzBM10sERIxNGhkz36PsMAvKg2YGCbHdYEVbsYfzVMhdQs3jKZH vg== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utdabvge1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:43 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7Hrf0K039610; Thu, 7 Dec 2023 18:50:42 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vj0n-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:42 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io90Y008067; Thu, 7 Dec 2023 18:50:42 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-22; Thu, 07 Dec 2023 18:50:41 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 21/40] vdpa: vhost_vdpa_dma_batch_end_once rename Date: Thu, 7 Dec 2023 09:39:34 -0800 Message-Id: <1701970793-6865-22-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-ORIG-GUID: KdXaukVOZBlx2C3Dmk_NjLvQJU1DF52c X-Proofpoint-GUID: KdXaukVOZBlx2C3Dmk_NjLvQJU1DF52c Received-SPF: pass client-ip=205.220.177.32; envelope-from=si-wei.liu@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org No functional changes. Rename only. Signed-off-by: Si-Wei Liu --- hw/virtio/vhost-vdpa.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index 47c764b..013bfa2 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -191,7 +191,7 @@ static void vhost_vdpa_iotlb_batch_begin_once(VhostVDPAShared *s) s->iotlb_batch_begin_sent = true; } -static void vhost_vdpa_dma_end_batch(VhostVDPAShared *s) +static void vhost_vdpa_dma_batch_end_once(VhostVDPAShared *s) { struct vhost_msg_v2 msg = {}; int fd = s->device_fd; @@ -229,7 +229,7 @@ static void vhost_vdpa_listener_commit(MemoryListener *listener) { VhostVDPAShared *s = container_of(listener, VhostVDPAShared, listener); - vhost_vdpa_dma_end_batch(s); + vhost_vdpa_dma_batch_end_once(s); } static void vhost_vdpa_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb) @@ -1367,7 +1367,7 @@ static void *vhost_vdpa_load_map(void *opaque) vhost_vdpa_iotlb_batch_begin_once(shared); break; case VHOST_IOTLB_BATCH_END: - vhost_vdpa_dma_end_batch(shared); + vhost_vdpa_dma_batch_end_once(shared); break; default: error_report("Invalid IOTLB msg type %d", msg->iotlb.type); From patchwork Thu Dec 7 17:39:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483859 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5FBE1C10DC3 for ; Thu, 7 Dec 2023 18:58:00 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJSm-0003qE-56; Thu, 07 Dec 2023 13:51:08 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSW-0003cT-7D for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:53 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSR-00076R-3a for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:51 -0500 Received: from pps.filterd (m0333521.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7Hx09A028566; Thu, 7 Dec 2023 18:50:45 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=oSWk8NLfi5dV39YdUg3W+vNsFQfTt4s1cR4DL0KLrSY=; b=NJsCcMar4YwwEeGj0cvfLu+EtKsrs8XuD6DuVrLhUOsxgJ8+qIzWrt0if9rfgeuyE6Ci 6A1/0A3bh1vvA+s7iR30j4B7LzwqNlOeGuC0YAF6zN4JFoXKzLyHWRlPSumvBRcgPC+A xcWqAL5ECVG94vkl06kRX+NKSUnuaReceWJYBWywHdezRNgBU7yKvgQlbpm59zAx1DwC crTTdmLOxyytm1CJqhlGJTAO5GEIVI6SMJnMNkaivYztVC5qrbVUsc9wOu3isSHgjC0r tHtnh24bDHlDhDu4luBbI39yF8WWhZYdcG/rSq/s4z2Et4/QVauG4uv5A0Rnizch5RqW JQ== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utd0hmjtq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:44 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7Hcvr4039979; Thu, 7 Dec 2023 18:50:44 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vj1q-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:44 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io90a008067; Thu, 7 Dec 2023 18:50:43 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-23; Thu, 07 Dec 2023 18:50:43 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 22/40] vdpa: factor out vhost_vdpa_map_batch_begin Date: Thu, 7 Dec 2023 09:39:35 -0800 Message-Id: <1701970793-6865-23-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-GUID: 424P1H1d3w_4EdSqgyefR05OHlkVmnHS X-Proofpoint-ORIG-GUID: 424P1H1d3w_4EdSqgyefR05OHlkVmnHS Received-SPF: pass client-ip=205.220.165.32; envelope-from=si-wei.liu@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Refactoring only. No functional change. Signed-off-by: Si-Wei Liu Acked-by: Jason Wang --- hw/virtio/trace-events | 2 +- hw/virtio/vhost-vdpa.c | 25 ++++++++++++++++--------- 2 files changed, 17 insertions(+), 10 deletions(-) diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index 9725d44..b0239b8 100644 --- a/hw/virtio/trace-events +++ b/hw/virtio/trace-events @@ -32,7 +32,7 @@ vhost_user_create_notifier(int idx, void *n) "idx:%d n:%p" # vhost-vdpa.c vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, uint8_t type) "vdpa_shared:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: %"PRIu8 vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint8_t type) "vdpa_shared:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" type: %"PRIu8 -vhost_vdpa_listener_begin_batch(void *v, int fd, uint32_t msg_type, uint8_t type) "vdpa_shared:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8 +vhost_vdpa_map_batch_begin(void *v, int fd, uint32_t msg_type, uint8_t type) "vdpa_shared:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8 vhost_vdpa_listener_commit(void *v, int fd, uint32_t msg_type, uint8_t type) "vdpa_shared:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8 vhost_vdpa_listener_region_add_unaligned(void *v, const char *name, uint64_t offset_as, uint64_t offset_page) "vdpa_shared: %p region %s offset_within_address_space %"PRIu64" offset_within_region %"PRIu64 vhost_vdpa_listener_region_add(void *vdpa, uint64_t iova, uint64_t llend, void *vaddr, bool readonly) "vdpa: %p iova 0x%"PRIx64" llend 0x%"PRIx64" vaddr: %p read-only: %d" diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index 013bfa2..7a1b7f4 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -161,7 +161,7 @@ int vhost_vdpa_dma_unmap(VhostVDPAShared *s, uint32_t asid, hwaddr iova, return ret; } -static void vhost_vdpa_iotlb_batch_begin_once(VhostVDPAShared *s) +static bool vhost_vdpa_map_batch_begin(VhostVDPAShared *s) { int fd = s->device_fd; struct vhost_msg_v2 msg = { @@ -169,26 +169,33 @@ static void vhost_vdpa_iotlb_batch_begin_once(VhostVDPAShared *s) .iotlb.type = VHOST_IOTLB_BATCH_BEGIN, }; - if (!(s->backend_cap & (0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH)) || - s->iotlb_batch_begin_sent) { - return; - } - if (s->map_thread_enabled && !qemu_thread_is_self(&s->map_thread)) { struct vhost_msg_v2 *new_msg = g_new(struct vhost_msg_v2, 1); *new_msg = msg; g_async_queue_push(s->map_queue, new_msg); - return; + return false; } - trace_vhost_vdpa_listener_begin_batch(s, fd, msg.type, msg.iotlb.type); + trace_vhost_vdpa_map_batch_begin(s, fd, msg.type, msg.iotlb.type); if (write(fd, &msg, sizeof(msg)) != sizeof(msg)) { error_report("failed to write, fd=%d, errno=%d (%s)", fd, errno, strerror(errno)); } - s->iotlb_batch_begin_sent = true; + return true; +} + +static void vhost_vdpa_iotlb_batch_begin_once(VhostVDPAShared *s) +{ + if (!(s->backend_cap & (0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH)) || + s->iotlb_batch_begin_sent) { + return; + } + + if (vhost_vdpa_map_batch_begin(s)) { + s->iotlb_batch_begin_sent = true; + } } static void vhost_vdpa_dma_batch_end_once(VhostVDPAShared *s) From patchwork Thu Dec 7 17:39:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483810 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A733FC4167B for ; Thu, 7 Dec 2023 18:53:18 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJSt-0003yA-NF; Thu, 07 Dec 2023 13:51:15 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSa-0003d8-9h for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:58 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSW-00076j-6r for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:53 -0500 Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7GDP6O031570; Thu, 7 Dec 2023 18:50:46 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=DxpWETRXyV33aWVhz27vZT9JTn25BNRlFUePIY4gwgM=; b=Y078tMi9V5gxDYy1HeyHVhgeU/Xa9WL6n8dUqeNWMuDYiQTU0MqKHBJCII4tDEcjJpVg GMNGvtRcoW4Ov0LEYXXVQXzcoGtixd1a0AOfMAkGICYR1rkCz+nNbOWFLUG7836A//uo XjgJHI8N1GAvRiaUawQl0fNm7I7niCC/RyzPxsJO7HYAFz/YrWCSmkKEwNeaJ32GT63X +JqKHJN9Bwg0cc6V+hqYirPxQpw9f74mINBt6xZQdxhRH76ydONttEwjxbOXunkRJ15Y 40apCsKFMlYvXNAr2l0d870btKP3rlov5ed37IMwt8AFLDYiB/CczaJLRRCJkwR1H8/0 DA== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utdrvmpfn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:46 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7HNN94039498; Thu, 7 Dec 2023 18:50:45 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vj30-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:45 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io90c008067; Thu, 7 Dec 2023 18:50:45 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-24; Thu, 07 Dec 2023 18:50:44 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 23/40] vdpa: vhost_vdpa_dma_batch_begin_once rename Date: Thu, 7 Dec 2023 09:39:36 -0800 Message-Id: <1701970793-6865-24-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-GUID: 7v0XLWao16LyHgiiiQqFsKmf-VmYa7wd X-Proofpoint-ORIG-GUID: 7v0XLWao16LyHgiiiQqFsKmf-VmYa7wd Received-SPF: pass client-ip=205.220.177.32; envelope-from=si-wei.liu@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org No functional changes. Rename only. Signed-off-by: Si-Wei Liu --- hw/virtio/vhost-vdpa.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index 7a1b7f4..a6c6fe5 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -186,7 +186,7 @@ static bool vhost_vdpa_map_batch_begin(VhostVDPAShared *s) return true; } -static void vhost_vdpa_iotlb_batch_begin_once(VhostVDPAShared *s) +static void vhost_vdpa_dma_batch_begin_once(VhostVDPAShared *s) { if (!(s->backend_cap & (0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH)) || s->iotlb_batch_begin_sent) { @@ -411,7 +411,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener, iova = mem_region.iova; } - vhost_vdpa_iotlb_batch_begin_once(s); + vhost_vdpa_dma_batch_begin_once(s); ret = vhost_vdpa_dma_map(s, VHOST_VDPA_GUEST_PA_ASID, iova, int128_get64(llsize), vaddr, section->readonly); if (ret) { @@ -493,7 +493,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener, iova = result->iova; vhost_iova_tree_remove(s->iova_tree, *result); } - vhost_vdpa_iotlb_batch_begin_once(s); + vhost_vdpa_dma_batch_begin_once(s); /* * The unmap ioctl doesn't accept a full 64-bit. need to check it */ @@ -1371,7 +1371,7 @@ static void *vhost_vdpa_load_map(void *opaque) msg->iotlb.size); break; case VHOST_IOTLB_BATCH_BEGIN: - vhost_vdpa_iotlb_batch_begin_once(shared); + vhost_vdpa_dma_batch_begin_once(shared); break; case VHOST_IOTLB_BATCH_END: vhost_vdpa_dma_batch_end_once(shared); From patchwork Thu Dec 7 17:39:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483816 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 019C9C46CA0 for ; Thu, 7 Dec 2023 18:55:41 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJSp-0003ug-81; Thu, 07 Dec 2023 13:51:11 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSj-0003n7-Oa for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:51:05 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSg-000789-7w for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:51:05 -0500 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7IDQpq023130; Thu, 7 Dec 2023 18:50:48 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=BS7Msf1TqnmvtthANunCKzo/Ed1i9XdmfTaEX+ayVFI=; b=MEiCigPllYgrcPiGHSlkHzk4PEmr/Fva8BcydEFqm7XyDxj5dakVUl/STeitaHDtps+L j57r/8UpCU8pEemX/FyFCB/7JndRQNBNkLEbX8yDmUBvV4vVM3YOfMsE8O2sjsT7fDmw i13hoEgit6nZHC7+CFjmyyyq56vzPWSQQ0iCBsF5xb0xJ0ohOMNMYZkmarY3TJlb7iMb wcV0bul449HEJ5xtMO08PN31+NvpqACLN+pfiEJ3MGKBFx6Xk2EXQliEXk+0ZoFsW4vj 9YyqKYN5ycyJdnOUMSEpiuszn/Fk6biCyy8/VtKvj2nsHWMrDT4cPBWKtH1FxmOC6709 5w== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utdmbmnt5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:47 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7HnhAw039577; Thu, 7 Dec 2023 18:50:47 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vj49-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:46 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io90e008067; Thu, 7 Dec 2023 18:50:46 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-25; Thu, 07 Dec 2023 18:50:46 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 24/40] vdpa: factor out vhost_vdpa_dma_batch_end Date: Thu, 7 Dec 2023 09:39:37 -0800 Message-Id: <1701970793-6865-25-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-ORIG-GUID: wHLI1iagrlgzCyDbdjh7a-OQF9ypWRiT X-Proofpoint-GUID: wHLI1iagrlgzCyDbdjh7a-OQF9ypWRiT Received-SPF: pass client-ip=205.220.177.32; envelope-from=si-wei.liu@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Refactoring only. No functional change. Signed-off-by: Si-Wei Liu --- hw/virtio/trace-events | 2 +- hw/virtio/vhost-vdpa.c | 30 ++++++++++++++++++------------ 2 files changed, 19 insertions(+), 13 deletions(-) diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index b0239b8..3411a07 100644 --- a/hw/virtio/trace-events +++ b/hw/virtio/trace-events @@ -33,7 +33,7 @@ vhost_user_create_notifier(int idx, void *n) "idx:%d n:%p" vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, uint8_t type) "vdpa_shared:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: %"PRIu8 vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint8_t type) "vdpa_shared:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" type: %"PRIu8 vhost_vdpa_map_batch_begin(void *v, int fd, uint32_t msg_type, uint8_t type) "vdpa_shared:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8 -vhost_vdpa_listener_commit(void *v, int fd, uint32_t msg_type, uint8_t type) "vdpa_shared:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8 +vhost_vdpa_dma_batch_end(void *v, int fd, uint32_t msg_type, uint8_t type) "vdpa_shared:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8 vhost_vdpa_listener_region_add_unaligned(void *v, const char *name, uint64_t offset_as, uint64_t offset_page) "vdpa_shared: %p region %s offset_within_address_space %"PRIu64" offset_within_region %"PRIu64 vhost_vdpa_listener_region_add(void *vdpa, uint64_t iova, uint64_t llend, void *vaddr, bool readonly) "vdpa: %p iova 0x%"PRIx64" llend 0x%"PRIx64" vaddr: %p read-only: %d" vhost_vdpa_listener_region_del_unaligned(void *v, const char *name, uint64_t offset_as, uint64_t offset_page) "vdpa_shared: %p region %s offset_within_address_space %"PRIu64" offset_within_region %"PRIu64 diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index a6c6fe5..999a97a 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -198,19 +198,11 @@ static void vhost_vdpa_dma_batch_begin_once(VhostVDPAShared *s) } } -static void vhost_vdpa_dma_batch_end_once(VhostVDPAShared *s) +static bool vhost_vdpa_dma_batch_end(VhostVDPAShared *s) { struct vhost_msg_v2 msg = {}; int fd = s->device_fd; - if (!(s->backend_cap & (0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH))) { - return; - } - - if (!s->iotlb_batch_begin_sent) { - return; - } - msg.type = VHOST_IOTLB_MSG_V2; msg.iotlb.type = VHOST_IOTLB_BATCH_END; @@ -220,16 +212,30 @@ static void vhost_vdpa_dma_batch_end_once(VhostVDPAShared *s) *new_msg = msg; g_async_queue_push(s->map_queue, new_msg); - return; + return false; } - trace_vhost_vdpa_listener_commit(s, fd, msg.type, msg.iotlb.type); + trace_vhost_vdpa_dma_batch_end(s, fd, msg.type, msg.iotlb.type); if (write(fd, &msg, sizeof(msg)) != sizeof(msg)) { error_report("failed to write, fd=%d, errno=%d (%s)", fd, errno, strerror(errno)); } + return true; +} + +static void vhost_vdpa_dma_batch_end_once(VhostVDPAShared *s) +{ + if (!(s->backend_cap & (0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH))) { + return; + } + + if (!s->iotlb_batch_begin_sent) { + return; + } - s->iotlb_batch_begin_sent = false; + if (vhost_vdpa_dma_batch_end(s)) { + s->iotlb_batch_begin_sent = false; + } } static void vhost_vdpa_listener_commit(MemoryListener *listener) From patchwork Thu Dec 7 17:39:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483798 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 82FE8C4167B for ; Thu, 7 Dec 2023 18:51:23 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJSl-0003pK-Qp; Thu, 07 Dec 2023 13:51:07 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSa-0003d9-Ag for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:58 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSW-000796-Fi for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:56 -0500 Received: from pps.filterd (m0333521.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7HFTxV007979; Thu, 7 Dec 2023 18:50:49 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=FuWoZYg+OylOSDGU0DmVC1aA59KXXhNhadiSgZ9OM78=; b=ZtFFnG35R8XaUpae/IkQLgcxtp5InSCZ614k4r3QImIljrfxQUy8hB+gyY9+mptx8tgi M2EAxig/QdKD9SRNhbOzJmu5RF8S+7TW+oHXkqdZBKu9MHaDr6lFbPOiLVxBJDpzp+q5 B4UyFWI+E1GCIgHDARUYTyXpadOk0cldvcuna/D2KhHUCoHb37UgFfY2+vHFJ6UM8YNg FGx2Dww3L0SuW4xANb4ULJF1DAu8K9v3+ScDGnOv3yfzyi8S6DUCCUjz5ZBvyh7Rfftc 25G+/q2jYyjXr8oWHjmRJk4NZuTlSM7dkB5mBm7thkQPclgZnWmnXecPj7VvtROT27y6 4g== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utd0hmjtu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:49 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7InFew039579; Thu, 7 Dec 2023 18:50:48 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vj5c-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:48 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io90g008067; Thu, 7 Dec 2023 18:50:47 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-26; Thu, 07 Dec 2023 18:50:47 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 25/40] vdpa: add asid to dma_batch_once API Date: Thu, 7 Dec 2023 09:39:38 -0800 Message-Id: <1701970793-6865-26-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-GUID: GgEBq_STXTzs1jNyPD5vEXSIz9aX4Nug X-Proofpoint-ORIG-GUID: GgEBq_STXTzs1jNyPD5vEXSIz9aX4Nug Received-SPF: pass client-ip=205.220.165.32; envelope-from=si-wei.liu@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org So that DMA batching API can operate on other ASID than 0. Signed-off-by: Si-Wei Liu --- hw/virtio/trace-events | 4 ++-- hw/virtio/vhost-vdpa.c | 14 ++++++++------ 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index 3411a07..196f32f 100644 --- a/hw/virtio/trace-events +++ b/hw/virtio/trace-events @@ -32,8 +32,8 @@ vhost_user_create_notifier(int idx, void *n) "idx:%d n:%p" # vhost-vdpa.c vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, uint8_t type) "vdpa_shared:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: %"PRIu8 vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint8_t type) "vdpa_shared:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" type: %"PRIu8 -vhost_vdpa_map_batch_begin(void *v, int fd, uint32_t msg_type, uint8_t type) "vdpa_shared:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8 -vhost_vdpa_dma_batch_end(void *v, int fd, uint32_t msg_type, uint8_t type) "vdpa_shared:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8 +vhost_vdpa_map_batch_begin(void *v, int fd, uint32_t msg_type, uint8_t type, uint32_t asid) "vdpa_shared:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8" asid: %"PRIu32 +vhost_vdpa_dma_batch_end(void *v, int fd, uint32_t msg_type, uint8_t type, uint32_t asid) "vdpa_shared:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8" asid: %"PRIu32 vhost_vdpa_listener_region_add_unaligned(void *v, const char *name, uint64_t offset_as, uint64_t offset_page) "vdpa_shared: %p region %s offset_within_address_space %"PRIu64" offset_within_region %"PRIu64 vhost_vdpa_listener_region_add(void *vdpa, uint64_t iova, uint64_t llend, void *vaddr, bool readonly) "vdpa: %p iova 0x%"PRIx64" llend 0x%"PRIx64" vaddr: %p read-only: %d" vhost_vdpa_listener_region_del_unaligned(void *v, const char *name, uint64_t offset_as, uint64_t offset_page) "vdpa_shared: %p region %s offset_within_address_space %"PRIu64" offset_within_region %"PRIu64 diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index 999a97a..2db2832 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -161,11 +161,12 @@ int vhost_vdpa_dma_unmap(VhostVDPAShared *s, uint32_t asid, hwaddr iova, return ret; } -static bool vhost_vdpa_map_batch_begin(VhostVDPAShared *s) +static bool vhost_vdpa_map_batch_begin(VhostVDPAShared *s, uint32_t asid) { int fd = s->device_fd; struct vhost_msg_v2 msg = { .type = VHOST_IOTLB_MSG_V2, + .asid = asid, .iotlb.type = VHOST_IOTLB_BATCH_BEGIN, }; @@ -178,7 +179,7 @@ static bool vhost_vdpa_map_batch_begin(VhostVDPAShared *s) return false; } - trace_vhost_vdpa_map_batch_begin(s, fd, msg.type, msg.iotlb.type); + trace_vhost_vdpa_map_batch_begin(s, fd, msg.type, msg.iotlb.type, msg.asid); if (write(fd, &msg, sizeof(msg)) != sizeof(msg)) { error_report("failed to write, fd=%d, errno=%d (%s)", fd, errno, strerror(errno)); @@ -193,17 +194,18 @@ static void vhost_vdpa_dma_batch_begin_once(VhostVDPAShared *s) return; } - if (vhost_vdpa_map_batch_begin(s)) { + if (vhost_vdpa_map_batch_begin(s, 0)) { s->iotlb_batch_begin_sent = true; } } -static bool vhost_vdpa_dma_batch_end(VhostVDPAShared *s) +static bool vhost_vdpa_dma_batch_end(VhostVDPAShared *s, uint32_t asid) { struct vhost_msg_v2 msg = {}; int fd = s->device_fd; msg.type = VHOST_IOTLB_MSG_V2; + msg.asid = asid; msg.iotlb.type = VHOST_IOTLB_BATCH_END; if (s->map_thread_enabled && !qemu_thread_is_self(&s->map_thread)) { @@ -215,7 +217,7 @@ static bool vhost_vdpa_dma_batch_end(VhostVDPAShared *s) return false; } - trace_vhost_vdpa_dma_batch_end(s, fd, msg.type, msg.iotlb.type); + trace_vhost_vdpa_dma_batch_end(s, fd, msg.type, msg.iotlb.type, msg.asid); if (write(fd, &msg, sizeof(msg)) != sizeof(msg)) { error_report("failed to write, fd=%d, errno=%d (%s)", fd, errno, strerror(errno)); @@ -233,7 +235,7 @@ static void vhost_vdpa_dma_batch_end_once(VhostVDPAShared *s) return; } - if (vhost_vdpa_dma_batch_end(s)) { + if (vhost_vdpa_dma_batch_end(s, 0)) { s->iotlb_batch_begin_sent = false; } } From patchwork Thu Dec 7 17:39:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483807 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 681CBC10DC3 for ; Thu, 7 Dec 2023 18:52:39 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJSq-0003w4-Hb; Thu, 07 Dec 2023 13:51:12 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSb-0003dE-FJ for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:58 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSW-00079N-LL for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:56 -0500 Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7InVX4000548; Thu, 7 Dec 2023 18:50:50 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=6xgia7g1oUY0RjFvspZ/n8yXNYyUzWdL6U6DAU8F9r4=; b=hTnDnabomVnFFgH2CoIwMKRwtZ4aTSVcbwN8akolzv3wU2OpRwPR8gTeM+ImcmPWNJv+ 8Dx8aU86W3lr0PUPmH9BSkGkj5s6AgMm7eLUlL7ZgMBUUW+iqkL3O7UuJDAGaM01Qag2 adFZ7GUxcdFCU13xYsVW4hEybj75rX5iiZ3lDGotkxvu5qsBnPWXPWfBQOWtGV+74woT tdyNr5u1YyZIL+VJ/oS5wGyi6clp9E2uHQzO3eN4L2zkqTVKsTCUoaBBKRFN1LMlxHAh v6KxEJrFv5Npw4pUfJnNEq3NkL1vspIeE+Ja7SxtuqQ5p7b/Sy1wi9xv4899D8bGTGc9 EA== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utd0mmh5n-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:50 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7HVFSx039488; Thu, 7 Dec 2023 18:50:49 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vj6c-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:49 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io90i008067; Thu, 7 Dec 2023 18:50:49 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-27; Thu, 07 Dec 2023 18:50:49 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 26/40] vdpa: return int for dma_batch_once API Date: Thu, 7 Dec 2023 09:39:39 -0800 Message-Id: <1701970793-6865-27-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-GUID: QJXnP9B189LUqMPQkKgFaTfqPY7WUPVl X-Proofpoint-ORIG-GUID: QJXnP9B189LUqMPQkKgFaTfqPY7WUPVl Received-SPF: pass client-ip=205.220.165.32; envelope-from=si-wei.liu@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Return zero for success for now. Prepare for non-zero return in the next few patches. Signed-off-by: Si-Wei Liu --- hw/virtio/vhost-vdpa.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index 2db2832..e0137f0 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -187,16 +187,18 @@ static bool vhost_vdpa_map_batch_begin(VhostVDPAShared *s, uint32_t asid) return true; } -static void vhost_vdpa_dma_batch_begin_once(VhostVDPAShared *s) +static int vhost_vdpa_dma_batch_begin_once(VhostVDPAShared *s) { if (!(s->backend_cap & (0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH)) || s->iotlb_batch_begin_sent) { - return; + return 0; } if (vhost_vdpa_map_batch_begin(s, 0)) { s->iotlb_batch_begin_sent = true; } + + return 0; } static bool vhost_vdpa_dma_batch_end(VhostVDPAShared *s, uint32_t asid) @@ -225,19 +227,21 @@ static bool vhost_vdpa_dma_batch_end(VhostVDPAShared *s, uint32_t asid) return true; } -static void vhost_vdpa_dma_batch_end_once(VhostVDPAShared *s) +static int vhost_vdpa_dma_batch_end_once(VhostVDPAShared *s) { if (!(s->backend_cap & (0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH))) { - return; + return 0; } if (!s->iotlb_batch_begin_sent) { - return; + return 0; } if (vhost_vdpa_dma_batch_end(s, 0)) { s->iotlb_batch_begin_sent = false; } + + return 0; } static void vhost_vdpa_listener_commit(MemoryListener *listener) From patchwork Thu Dec 7 17:39:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483815 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D4B78C4167B for ; Thu, 7 Dec 2023 18:55:41 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJSr-0003wv-Al; Thu, 07 Dec 2023 13:51:13 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSp-0003vj-Rs for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:51:11 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSj-0007AC-VR for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:51:11 -0500 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7EtJQM015896; Thu, 7 Dec 2023 18:50:52 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=Jnkj8GEJTOW6vXOQA2mRArcmep3dLye1fLL8tT9XBa0=; b=TMY+GN35Ciijutl/r949GpyDzd63HdaIefN+03riqV6duwNhy0RPtbqJ1vlIfTY97ohr auhF9ri3wLj38ADM+d9hsPqrOS+3wEvpmW1qggEg2xmROn1sH9tTZoVmqeoQl71tVXBx MnAQnrWsLGK3LsNCkn/TH0NV7UZiA+IE+nyRKJsD94lHyAyPuOHwS/Hq1l/yXbmnJnkR v1B/bCLUTjBaNbMOJzi0U1yb+9LpGvsETl1gKwo4+qVCOZTrB4UErZYst5DYWRf5VmCl uUa9m+WbosWQCWegY9GW/YhNUrf+BKo3yvT2K/YoqJ/Kfyyve9YPVfzK3kAmoyxzv4XT cA== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utdmbmnt8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:52 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7IVo0G039480; Thu, 7 Dec 2023 18:50:51 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vj77-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:51 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io90k008067; Thu, 7 Dec 2023 18:50:50 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-28; Thu, 07 Dec 2023 18:50:50 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 27/40] vdpa: add asid to all dma_batch call sites Date: Thu, 7 Dec 2023 09:39:40 -0800 Message-Id: <1701970793-6865-28-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-ORIG-GUID: Y8IbbPfJkJuWdiEkUypV9_KhpWVQl7xj X-Proofpoint-GUID: Y8IbbPfJkJuWdiEkUypV9_KhpWVQl7xj Received-SPF: pass client-ip=205.220.177.32; envelope-from=si-wei.liu@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Will allow other callers to specifcy asid when calling the dma_batch API. Signed-off-by: Si-Wei Liu --- hw/virtio/vhost-vdpa.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index e0137f0..d3f5721 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -187,14 +187,14 @@ static bool vhost_vdpa_map_batch_begin(VhostVDPAShared *s, uint32_t asid) return true; } -static int vhost_vdpa_dma_batch_begin_once(VhostVDPAShared *s) +static int vhost_vdpa_dma_batch_begin_once(VhostVDPAShared *s, uint32_t asid) { if (!(s->backend_cap & (0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH)) || s->iotlb_batch_begin_sent) { return 0; } - if (vhost_vdpa_map_batch_begin(s, 0)) { + if (vhost_vdpa_map_batch_begin(s, asid)) { s->iotlb_batch_begin_sent = true; } @@ -227,7 +227,7 @@ static bool vhost_vdpa_dma_batch_end(VhostVDPAShared *s, uint32_t asid) return true; } -static int vhost_vdpa_dma_batch_end_once(VhostVDPAShared *s) +static int vhost_vdpa_dma_batch_end_once(VhostVDPAShared *s, uint32_t asid) { if (!(s->backend_cap & (0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH))) { return 0; @@ -237,7 +237,7 @@ static int vhost_vdpa_dma_batch_end_once(VhostVDPAShared *s) return 0; } - if (vhost_vdpa_dma_batch_end(s, 0)) { + if (vhost_vdpa_dma_batch_end(s, asid)) { s->iotlb_batch_begin_sent = false; } @@ -248,7 +248,7 @@ static void vhost_vdpa_listener_commit(MemoryListener *listener) { VhostVDPAShared *s = container_of(listener, VhostVDPAShared, listener); - vhost_vdpa_dma_batch_end_once(s); + vhost_vdpa_dma_batch_end_once(s, VHOST_VDPA_GUEST_PA_ASID); } static void vhost_vdpa_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb) @@ -423,7 +423,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener, iova = mem_region.iova; } - vhost_vdpa_dma_batch_begin_once(s); + vhost_vdpa_dma_batch_begin_once(s, VHOST_VDPA_GUEST_PA_ASID); ret = vhost_vdpa_dma_map(s, VHOST_VDPA_GUEST_PA_ASID, iova, int128_get64(llsize), vaddr, section->readonly); if (ret) { @@ -505,7 +505,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener, iova = result->iova; vhost_iova_tree_remove(s->iova_tree, *result); } - vhost_vdpa_dma_batch_begin_once(s); + vhost_vdpa_dma_batch_begin_once(s, VHOST_VDPA_GUEST_PA_ASID); /* * The unmap ioctl doesn't accept a full 64-bit. need to check it */ @@ -1383,10 +1383,10 @@ static void *vhost_vdpa_load_map(void *opaque) msg->iotlb.size); break; case VHOST_IOTLB_BATCH_BEGIN: - vhost_vdpa_dma_batch_begin_once(shared); + vhost_vdpa_dma_batch_begin_once(shared, msg->asid); break; case VHOST_IOTLB_BATCH_END: - vhost_vdpa_dma_batch_end_once(shared); + vhost_vdpa_dma_batch_end_once(shared, msg->asid); break; default: error_report("Invalid IOTLB msg type %d", msg->iotlb.type); From patchwork Thu Dec 7 17:39:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483820 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C1E0DC4167B for ; Thu, 7 Dec 2023 18:56:23 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJSu-0003yj-9t; Thu, 07 Dec 2023 13:51:16 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSd-0003dq-IL for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:59 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSa-0007Ab-7s for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:50:59 -0500 Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7HVcZ3010797; Thu, 7 Dec 2023 18:50:54 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=MS8ZsyDAdKAh6RiaLiKLs0LxEX3ZzPIG9eACCh1nyIQ=; b=KRfkzehqZY+grL0T/CA66mzcVnpwjYGhJUZk3i/JZ25wJrQmGp1CguKHLzcPuy3YUZDZ foWczM37A0fQr3Vte1CLmmsK4PjNMoxATLF5VJgivSSyhOoYpwk8MP8WFUroqS1Cf1uk 3pPkGotqqrt/IMq5vNiHaKcryshPIEtHETmwBnSMYjQOQR9yyN+Geyc9Lp8+o8yDZVnv Dr/0S4iA+F/0iqouV88wmpnn26W2Ab75xK4XVqpgb0YZMYNouAaypnd3zcn0EXlwdcfM NKzit8yOi4nF5B5AUOpt31tOSVZJuTe/M33tNuDr7fP/35nrE8K1UsOtrFXaLpRlz7Ff aA== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utdc1cg0b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:53 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7Hcvr8039979; Thu, 7 Dec 2023 18:50:52 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vj81-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:52 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io90m008067; Thu, 7 Dec 2023 18:50:52 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-29; Thu, 07 Dec 2023 18:50:52 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 28/40] vdpa: support iotlb_batch_asid Date: Thu, 7 Dec 2023 09:39:41 -0800 Message-Id: <1701970793-6865-29-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-GUID: xIk5cnj65ZpX3U_GR0QXTYhXhSti7wS- X-Proofpoint-ORIG-GUID: xIk5cnj65ZpX3U_GR0QXTYhXhSti7wS- Received-SPF: pass client-ip=205.220.165.32; envelope-from=si-wei.liu@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Then it's possible to specify ASID when calling the DMA batching API. If the ASID to work on doesn't align with the ASID for ongoing transaction, the API will fail the request and return negative, and the transaction will remain intact as if no failed request ever had occured. Signed-off-by: Si-Wei Liu --- hw/virtio/vhost-vdpa.c | 25 +++++++++++++++++++------ include/hw/virtio/vhost-vdpa.h | 1 + net/vhost-vdpa.c | 1 + 3 files changed, 21 insertions(+), 6 deletions(-) diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index d3f5721..b7896a8 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -189,15 +189,25 @@ static bool vhost_vdpa_map_batch_begin(VhostVDPAShared *s, uint32_t asid) static int vhost_vdpa_dma_batch_begin_once(VhostVDPAShared *s, uint32_t asid) { - if (!(s->backend_cap & (0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH)) || - s->iotlb_batch_begin_sent) { + if (!(s->backend_cap & (0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH))) { return 0; } - if (vhost_vdpa_map_batch_begin(s, asid)) { - s->iotlb_batch_begin_sent = true; + if (s->iotlb_batch_begin_sent && s->iotlb_batch_asid != asid) { + return -1; + } + + if (s->iotlb_batch_begin_sent) { + return 0; } + if (!vhost_vdpa_map_batch_begin(s, asid)) { + return 0; + } + + s->iotlb_batch_begin_sent = true; + s->iotlb_batch_asid = asid; + return 0; } @@ -237,10 +247,13 @@ static int vhost_vdpa_dma_batch_end_once(VhostVDPAShared *s, uint32_t asid) return 0; } - if (vhost_vdpa_dma_batch_end(s, asid)) { - s->iotlb_batch_begin_sent = false; + if (!vhost_vdpa_dma_batch_end(s, asid)) { + return 0; } + s->iotlb_batch_begin_sent = false; + s->iotlb_batch_asid = -1; + return 0; } diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h index 0fe0f60..219316f 100644 --- a/include/hw/virtio/vhost-vdpa.h +++ b/include/hw/virtio/vhost-vdpa.h @@ -61,6 +61,7 @@ typedef struct vhost_vdpa_shared { bool map_thread_enabled; bool iotlb_batch_begin_sent; + uint32_t iotlb_batch_asid; /* * The memory listener has been registered, so DMA maps have been sent to diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c index e9b96ed..bc72345 100644 --- a/net/vhost-vdpa.c +++ b/net/vhost-vdpa.c @@ -1933,6 +1933,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer, s->vhost_vdpa.shared->device_fd = vdpa_device_fd; s->vhost_vdpa.shared->iova_range = iova_range; s->vhost_vdpa.shared->shadow_data = svq; + s->vhost_vdpa.shared->iotlb_batch_asid = -1; s->vhost_vdpa.shared->refcnt++; } else if (!is_datapath) { s->cvq_cmd_out_buffer = mmap(NULL, vhost_vdpa_net_cvq_cmd_page_len(), From patchwork Thu Dec 7 17:39:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483861 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2EBF0C4167B for ; Thu, 7 Dec 2023 18:58:23 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJSu-0003yd-2l; Thu, 07 Dec 2023 13:51:16 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSq-0003vx-4b for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:51:13 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSl-0007B3-71 for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:51:11 -0500 Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7InVX5000548; Thu, 7 Dec 2023 18:50:55 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=i6V4YQb4G5hm1g+p/0CXNDXC0WWgyQ8rR9Uz9vRtnJI=; b=R1vXdULLkQlxFJb1tucQB1ZVfGngOrxUv/0BhrJ6/G3Tv69wIEpAsdXhjbSLwi5W66d2 zhpet4Lf6XFoFpkYvjysxcDzzBsekJxVsoj140CUMzlBtTauPsYsaM5ytloeGMJHKjEI DDfDdITw7l95qeaX5NI84Y89Z7Vxml8gh9XIp4HacK1oL2cH71E4wLfywPBHVjI4KT68 IpVk9QnhjOUvU/Y6loXf372spGaeNpcLBAlJodIC4Wix5eIr1FL3CYp2YVoEiK/G4jws geyctaZyTc4fXvAqoRrDz4IWSDr/MStiOgxqK3W6UebzzYZIGEL/PXPhIwR/AjATqasi YQ== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utd0mmh5t-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:54 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7HRJYR039470; Thu, 7 Dec 2023 18:50:54 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vj8t-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:54 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io90o008067; Thu, 7 Dec 2023 18:50:53 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-30; Thu, 07 Dec 2023 18:50:53 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 29/40] vdpa: expose API vhost_vdpa_dma_batch_once Date: Thu, 7 Dec 2023 09:39:42 -0800 Message-Id: <1701970793-6865-30-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-GUID: CJ0dY195PKGyL1AhpqpBieLNrl9Dqe0z X-Proofpoint-ORIG-GUID: CJ0dY195PKGyL1AhpqpBieLNrl9Dqe0z Received-SPF: pass client-ip=205.220.165.32; envelope-from=si-wei.liu@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org So that the batching API can be called from other file externally than the local. Signed-off-by: Si-Wei Liu --- hw/virtio/vhost-vdpa.c | 21 +++++++++++++++------ include/hw/virtio/vhost-vdpa.h | 3 +++ 2 files changed, 18 insertions(+), 6 deletions(-) diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index b7896a8..68dc01b 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -187,7 +187,7 @@ static bool vhost_vdpa_map_batch_begin(VhostVDPAShared *s, uint32_t asid) return true; } -static int vhost_vdpa_dma_batch_begin_once(VhostVDPAShared *s, uint32_t asid) +int vhost_vdpa_dma_batch_begin_once(VhostVDPAShared *s, uint32_t asid) { if (!(s->backend_cap & (0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH))) { return 0; @@ -237,7 +237,7 @@ static bool vhost_vdpa_dma_batch_end(VhostVDPAShared *s, uint32_t asid) return true; } -static int vhost_vdpa_dma_batch_end_once(VhostVDPAShared *s, uint32_t asid) +int vhost_vdpa_dma_batch_end_once(VhostVDPAShared *s, uint32_t asid) { if (!(s->backend_cap & (0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH))) { return 0; @@ -436,7 +436,12 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener, iova = mem_region.iova; } - vhost_vdpa_dma_batch_begin_once(s, VHOST_VDPA_GUEST_PA_ASID); + ret = vhost_vdpa_dma_batch_begin_once(s, VHOST_VDPA_GUEST_PA_ASID); + if (unlikely(ret)) { + error_report("Can't batch mapping on asid 0 (%p)", s); + goto fail_map; + } + ret = vhost_vdpa_dma_map(s, VHOST_VDPA_GUEST_PA_ASID, iova, int128_get64(llsize), vaddr, section->readonly); if (ret) { @@ -518,7 +523,11 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener, iova = result->iova; vhost_iova_tree_remove(s->iova_tree, *result); } - vhost_vdpa_dma_batch_begin_once(s, VHOST_VDPA_GUEST_PA_ASID); + ret = vhost_vdpa_dma_batch_begin_once(s, VHOST_VDPA_GUEST_PA_ASID); + if (ret) { + error_report("Can't batch mapping on asid 0 (%p)", s); + } + /* * The unmap ioctl doesn't accept a full 64-bit. need to check it */ @@ -1396,10 +1405,10 @@ static void *vhost_vdpa_load_map(void *opaque) msg->iotlb.size); break; case VHOST_IOTLB_BATCH_BEGIN: - vhost_vdpa_dma_batch_begin_once(shared, msg->asid); + r = vhost_vdpa_dma_batch_begin_once(shared, msg->asid); break; case VHOST_IOTLB_BATCH_END: - vhost_vdpa_dma_batch_end_once(shared, msg->asid); + r = vhost_vdpa_dma_batch_end_once(shared, msg->asid); break; default: error_report("Invalid IOTLB msg type %d", msg->iotlb.type); diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h index 219316f..aa13679 100644 --- a/include/hw/virtio/vhost-vdpa.h +++ b/include/hw/virtio/vhost-vdpa.h @@ -106,6 +106,9 @@ int vhost_vdpa_dma_map(VhostVDPAShared *s, uint32_t asid, hwaddr iova, hwaddr size, void *vaddr, bool readonly); int vhost_vdpa_dma_unmap(VhostVDPAShared *s, uint32_t asid, hwaddr iova, hwaddr size); +int vhost_vdpa_dma_batch_begin_once(VhostVDPAShared *s, uint32_t asid); +int vhost_vdpa_dma_batch_end_once(VhostVDPAShared *s, uint32_t asid); + int vhost_vdpa_load_setup(VhostVDPAShared *s, AddressSpace *dma_as); int vhost_vdpa_load_cleanup(VhostVDPAShared *s, bool vhost_will_start); From patchwork Thu Dec 7 17:39:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483809 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E2276C10DC3 for ; Thu, 7 Dec 2023 18:52:53 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJSu-0003ym-Ru; Thu, 07 Dec 2023 13:51:16 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSr-0003wk-7B for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:51:13 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSl-0007Bh-SH for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:51:12 -0500 Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7GDNQ5031482; Thu, 7 Dec 2023 18:50:56 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=5AgeQWyfy2Ho2yhA/SVB1fzSmEKhlEu+76jpsngIZG4=; b=NjeL04sf86y1ZL7zO8yZOuOybt/SUtemGfiYHUFqaUYRCISz4tJ1578gK3419CODdTRT yH7zjgt9pj+Hb6wAj84+fQbSUpZMDdOeCpAIm1i1uALWJFDRO4tbmxoShitQxVRFo1SU RRocGQyABqcbEtEPnOiyFcsW393gqsmfxqjDHcw1BdIq7w8nrvJDhZIgn5qelpNXrVlg Vo6xSBd//34SvStDkQkbuR2EY7OaXnlnBf6/S3e8L61UpVYpr9kPWGPV1JK/6vbfg0kf xt6vU/hM8d1k0bFUQV6TDE7nkdgKcju1bIrM6uOZ48a/kdAmrjS+28ETcf7K0OP5ZnoQ Lg== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utdrvmpfu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:56 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7ITt8L039503; Thu, 7 Dec 2023 18:50:55 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vj9s-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:55 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io90q008067; Thu, 7 Dec 2023 18:50:54 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-31; Thu, 07 Dec 2023 18:50:54 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 30/40] vdpa: batch map/unmap op per svq pair basis Date: Thu, 7 Dec 2023 09:39:43 -0800 Message-Id: <1701970793-6865-31-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-GUID: KxJwXgfLbavtyNmFEvUIyeUJEW5kqX00 X-Proofpoint-ORIG-GUID: KxJwXgfLbavtyNmFEvUIyeUJEW5kqX00 Received-SPF: pass client-ip=205.220.177.32; envelope-from=si-wei.liu@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Coalesce multiple map or unmap operations to just one so that all mapping setup or teardown can occur in a single DMA batch. Signed-off-by: Si-Wei Liu Acked-by: Jason Wang --- hw/virtio/vhost-vdpa.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index 68dc01b..d98704a 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -1288,6 +1288,7 @@ static bool vhost_vdpa_svqs_start(struct vhost_dev *dev) return true; } + vhost_vdpa_dma_batch_begin_once(v->shared, v->address_space_id); for (i = 0; i < v->shadow_vqs->len; ++i) { VirtQueue *vq = virtio_get_queue(dev->vdev, dev->vq_index + i); VhostShadowVirtqueue *svq = g_ptr_array_index(v->shadow_vqs, i); @@ -1315,6 +1316,7 @@ static bool vhost_vdpa_svqs_start(struct vhost_dev *dev) goto err_set_addr; } } + vhost_vdpa_dma_batch_end_once(v->shared, v->address_space_id); return true; @@ -1323,6 +1325,7 @@ err_set_addr: err_map: vhost_svq_stop(g_ptr_array_index(v->shadow_vqs, i)); + vhost_vdpa_dma_batch_end_once(v->shared, v->address_space_id); err: error_reportf_err(err, "Cannot setup SVQ %u: ", i); @@ -1343,6 +1346,7 @@ static void vhost_vdpa_svqs_stop(struct vhost_dev *dev) return; } + vhost_vdpa_dma_batch_begin_once(v->shared, v->address_space_id); for (unsigned i = 0; i < v->shadow_vqs->len; ++i) { VhostShadowVirtqueue *svq = g_ptr_array_index(v->shadow_vqs, i); @@ -1352,6 +1356,7 @@ static void vhost_vdpa_svqs_stop(struct vhost_dev *dev) event_notifier_cleanup(&svq->hdev_kick); event_notifier_cleanup(&svq->hdev_call); } + vhost_vdpa_dma_batch_end_once(v->shared, v->address_space_id); } static void vhost_vdpa_suspend(struct vhost_dev *dev) From patchwork Thu Dec 7 17:39:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483864 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9DF96C4167B for ; Thu, 7 Dec 2023 18:59:00 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJSx-0003zy-7R; Thu, 07 Dec 2023 13:51:19 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSv-0003yp-5e for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:51:17 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSo-0007Ck-1y for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:51:16 -0500 Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7FbtSt000883; Thu, 7 Dec 2023 18:50:57 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=C903EYO3s1FA0DFS86jy4RVM3891ps6An1EsefY13cI=; b=NI+ytsDEXxAVuxCg0ZBtfc6SRH+UAozTDrVTtB8DpxvHXG5XfXcdFe2DSkAKG8lkVviy d3GYoStY/M71SRfJytsMHVP2hMcLXHAchE38UzKY1P+l6GKGYsCxnKbAPKQI87Cs7T/Q yEWcEyAs/VW1Z14E7pYbcv83Sdv1DU1txJZJ0bgicEs3z5CxtuKOMpVa3Jxqmi6zc1AQ EaXcy3o9y4v0wFcshHijL3Z3iIHS0Q6tBx/cJPaZ2w0h4vxz+LEguFCd4zbcqaLnM5Ib ebK+8ouSepHFm0uRdeOumJteqy7jFHcCvT9zZFYvNeKKC/nxbFyIIzn+YnN7x0leZGss Tg== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utdda4hab-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:57 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7IWvDa039587; Thu, 7 Dec 2023 18:50:56 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vjb7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:56 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io90s008067; Thu, 7 Dec 2023 18:50:56 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-32; Thu, 07 Dec 2023 18:50:56 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 31/40] vdpa: batch map and unmap around cvq svq start/stop Date: Thu, 7 Dec 2023 09:39:44 -0800 Message-Id: <1701970793-6865-32-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-GUID: 1tu8oQmlk6NmeiRRP-m4mFaka_Y8mJaK X-Proofpoint-ORIG-GUID: 1tu8oQmlk6NmeiRRP-m4mFaka_Y8mJaK Received-SPF: pass client-ip=205.220.165.32; envelope-from=si-wei.liu@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Coalesce map or unmap operations to exact one DMA batch to reduce potential impact on performance. Signed-off-by: Si-Wei Liu Acked-by: Jason Wang --- net/vhost-vdpa.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c index bc72345..1c1d61f 100644 --- a/net/vhost-vdpa.c +++ b/net/vhost-vdpa.c @@ -715,10 +715,11 @@ out: v->shared->iova_range.last); } + vhost_vdpa_dma_batch_begin_once(v->shared, v->address_space_id); r = vhost_vdpa_cvq_map_buf(&s->vhost_vdpa, s->cvq_cmd_out_buffer, vhost_vdpa_net_cvq_cmd_page_len(), false); if (unlikely(r < 0)) { - return r; + goto err; } r = vhost_vdpa_cvq_map_buf(&s->vhost_vdpa, s->status, @@ -727,18 +728,23 @@ out: vhost_vdpa_cvq_unmap_buf(&s->vhost_vdpa, s->cvq_cmd_out_buffer); } +err: + vhost_vdpa_dma_batch_end_once(v->shared, v->address_space_id); return r; } static void vhost_vdpa_net_cvq_stop(NetClientState *nc) { VhostVDPAState *s = DO_UPCAST(VhostVDPAState, nc, nc); + struct vhost_vdpa *v = &s->vhost_vdpa; assert(nc->info->type == NET_CLIENT_DRIVER_VHOST_VDPA); if (s->vhost_vdpa.shadow_vqs_enabled) { + vhost_vdpa_dma_batch_begin_once(v->shared, v->address_space_id); vhost_vdpa_cvq_unmap_buf(&s->vhost_vdpa, s->cvq_cmd_out_buffer); vhost_vdpa_cvq_unmap_buf(&s->vhost_vdpa, s->status); + vhost_vdpa_dma_batch_end_once(v->shared, v->address_space_id); } vhost_vdpa_net_client_stop(nc); From patchwork Thu Dec 7 17:39:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483865 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0BD82C10DCE for ; Thu, 7 Dec 2023 18:59:03 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJSx-000408-Og; Thu, 07 Dec 2023 13:51:19 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSw-0003zX-Oc for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:51:18 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSo-0007DA-P6 for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:51:18 -0500 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7EtJQN015896; Thu, 7 Dec 2023 18:50:59 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=wOBXHNRL6OZTDk0hPYLcG1XuyRPvxOOU/1I3eVVSfQc=; b=Q8tsZkFkeKuqnhR2AVoKcXoyfTsXdN4VLji0hTjjQGhmUEwFzzQdbRJEmSrk3XQbJkUJ CnIa2KI/NsSaXAoIcErqZLl1dQlu2njcgJKuM+YlTYJiu+Hb2041nKpl9kHAWiqIW8CN bhphH7oyhZMij0Z+2buQnZBsv4VPVavjYtplY+ez1qbgzRNWXLPXxn7f1+sGqYQj3BLt Mm1CEr194IgOZhjIDoTvnSkVZmiddBZ6xujCjfnqKGW5Wlh+20pkcqhibbo4jK2On6/4 7yh/yOeUNCM9ztwAf/SPiEB6riZWBj104bP4EUD4dPYuF+05abQCIOlYOKB/6n+380XI /w== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utdmbmntd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:59 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7IVo0L039480; Thu, 7 Dec 2023 18:50:58 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vjbw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:58 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io90u008067; Thu, 7 Dec 2023 18:50:57 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-33; Thu, 07 Dec 2023 18:50:57 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 32/40] vdpa: factor out vhost_vdpa_net_get_nc_vdpa Date: Thu, 7 Dec 2023 09:39:45 -0800 Message-Id: <1701970793-6865-33-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-ORIG-GUID: M9pmOcsCX09JoRCTcbLtQ2Ory6laTJVd X-Proofpoint-GUID: M9pmOcsCX09JoRCTcbLtQ2Ory6laTJVd Received-SPF: pass client-ip=205.220.177.32; envelope-from=si-wei.liu@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Introduce new API. No functional change on existing API. Signed-off-by: Si-Wei Liu Acked-by: Jason Wang --- net/vhost-vdpa.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c index 1c1d61f..683619f 100644 --- a/net/vhost-vdpa.c +++ b/net/vhost-vdpa.c @@ -290,13 +290,18 @@ static ssize_t vhost_vdpa_receive(NetClientState *nc, const uint8_t *buf, } -/** From any vdpa net client, get the netclient of the first queue pair */ -static VhostVDPAState *vhost_vdpa_net_first_nc_vdpa(VhostVDPAState *s) +/** From any vdpa net client, get the netclient of the i-th queue pair */ +static VhostVDPAState *vhost_vdpa_net_get_nc_vdpa(VhostVDPAState *s, int i) { NICState *nic = qemu_get_nic(s->nc.peer); - NetClientState *nc0 = qemu_get_peer(nic->ncs, 0); + NetClientState *nc_i = qemu_get_peer(nic->ncs, i); + + return DO_UPCAST(VhostVDPAState, nc, nc_i); +} - return DO_UPCAST(VhostVDPAState, nc, nc0); +static VhostVDPAState *vhost_vdpa_net_first_nc_vdpa(VhostVDPAState *s) +{ + return vhost_vdpa_net_get_nc_vdpa(s, 0); } static void vhost_vdpa_net_log_global_enable(VhostVDPAState *s, bool enable) From patchwork Thu Dec 7 17:39:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483818 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AABE0C4167B for ; Thu, 7 Dec 2023 18:55:57 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJT2-00047e-Qi; Thu, 07 Dec 2023 13:51:24 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJT1-000435-5e for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:51:23 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSq-0007Dc-PQ for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:51:22 -0500 Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7InVX7000548; Thu, 7 Dec 2023 18:51:00 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=kxSpiE8Ke1O5BLHfgTteE8AERnFgFLmrr+RYhHE9PCg=; b=jY9g3wdOmkIfelJ/yBFJy6tK7LjF0+RM21dYA4b5YsbYy+EimVg9CzipYKqpE5EkrieX I4fBQ6Y0vpYb9rvuvNSsRITClvZMgFBdMi3gDkucu1xSLMEkQUiLgOGO+WjLA6gc3aMC EAd3DyFqFOuJHOw9b6Y8TA7vufWz2OYxqTg4YZU7nwBzz10MbNxeelfKGiTlWpNq42Fj Pd/Bp6zH8ZAY3e8pNQVkrSumA3mEbDXbRVJCpm+pHJkmMua5WAoO49EqvWwgZtrWAFen FO4wFtisG5xMqZ7asT+h1CGn+VUHX7O38Pax7HNu7Q9CjUr/d74FQHW+BGp+dfUKwIsQ Ew== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utd0mmh64-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:51:00 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7InFf7039579; Thu, 7 Dec 2023 18:50:59 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vjct-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:50:59 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io90w008067; Thu, 7 Dec 2023 18:50:59 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-34; Thu, 07 Dec 2023 18:50:59 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 33/40] vdpa: batch multiple dma_unmap to a single call for vm stop Date: Thu, 7 Dec 2023 09:39:46 -0800 Message-Id: <1701970793-6865-34-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-GUID: Ok6zt6Qx_oseqpQofEMTPRb2ZoIhLeDE X-Proofpoint-ORIG-GUID: Ok6zt6Qx_oseqpQofEMTPRb2ZoIhLeDE Received-SPF: pass client-ip=205.220.165.32; envelope-from=si-wei.liu@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Should help live migration downtime on source host. Below are the coalesced dma_unmap time series on 2 queue pair config (no dedicated descriptor group ASID for SVQ). 109531@1693367276.853503:vhost_vdpa_reset_device dev: 0x55c933926890 109531@1693367276.853513:vhost_vdpa_add_status dev: 0x55c933926890 status: 0x3 109531@1693367276.853520:vhost_vdpa_flush_map dev: 0x55c933926890 doit: 1 svq_flush: 0 persist: 1 109531@1693367276.853524:vhost_vdpa_set_config_call dev: 0x55c933926890 fd: -1 109531@1693367276.853579:vhost_vdpa_iotlb_begin_batch vdpa:0x7fa2aa895190 fd: 16 msg_type: 2 type: 5 109531@1693367276.853586:vhost_vdpa_dma_unmap vdpa:0x7fa2aa895190 fd: 16 msg_type: 2 asid: 0 iova: 0x1000 size: 0x2000 type: 3 109531@1693367276.853600:vhost_vdpa_dma_unmap vdpa:0x7fa2aa895190 fd: 16 msg_type: 2 asid: 0 iova: 0x3000 size: 0x1000 type: 3 109531@1693367276.853618:vhost_vdpa_dma_unmap vdpa:0x7fa2aa895190 fd: 16 msg_type: 2 asid: 0 iova: 0x4000 size: 0x2000 type: 3 109531@1693367276.853625:vhost_vdpa_dma_unmap vdpa:0x7fa2aa895190 fd: 16 msg_type: 2 asid: 0 iova: 0x6000 size: 0x1000 type: 3 109531@1693367276.853630:vhost_vdpa_dma_unmap vdpa:0x7fa2aa84c190 fd: 16 msg_type: 2 asid: 0 iova: 0x7000 size: 0x2000 type: 3 109531@1693367276.853636:vhost_vdpa_dma_unmap vdpa:0x7fa2aa84c190 fd: 16 msg_type: 2 asid: 0 iova: 0x9000 size: 0x1000 type: 3 109531@1693367276.853642:vhost_vdpa_dma_unmap vdpa:0x7fa2aa84c190 fd: 16 msg_type: 2 asid: 0 iova: 0xa000 size: 0x2000 type: 3 109531@1693367276.853648:vhost_vdpa_dma_unmap vdpa:0x7fa2aa84c190 fd: 16 msg_type: 2 asid: 0 iova: 0xc000 size: 0x1000 type: 3 109531@1693367276.853654:vhost_vdpa_dma_unmap vdpa:0x7fa2aa6b6190 fd: 16 msg_type: 2 asid: 0 iova: 0xf000 size: 0x1000 type: 3 109531@1693367276.853660:vhost_vdpa_dma_unmap vdpa:0x7fa2aa6b6190 fd: 16 msg_type: 2 asid: 0 iova: 0x10000 size: 0x1000 type: 3 109531@1693367276.853666:vhost_vdpa_dma_unmap vdpa:0x7fa2aa6b6190 fd: 16 msg_type: 2 asid: 0 iova: 0xd000 size: 0x1000 type: 3 109531@1693367276.853670:vhost_vdpa_dma_unmap vdpa:0x7fa2aa6b6190 fd: 16 msg_type: 2 asid: 0 iova: 0xe000 size: 0x1000 type: 3 109531@1693367276.853675:vhost_vdpa_iotlb_end_batch vdpa:0x7fa2aa895190 fd: 16 msg_type: 2 type: 6 109531@1693367277.014697:vhost_vdpa_get_vq_index dev: 0x55c933925de0 idx: 0 vq idx: 0 109531@1693367277.014747:vhost_vdpa_get_vq_index dev: 0x55c933925de0 idx: 1 vq idx: 1 109531@1693367277.014753:vhost_vdpa_get_vq_index dev: 0x55c9339262e0 idx: 2 vq idx: 2 109531@1693367277.014756:vhost_vdpa_get_vq_index dev: 0x55c9339262e0 idx: 3 vq idx: 3 Signed-off-by: Si-Wei Liu --- hw/virtio/vhost-vdpa.c | 7 +-- include/hw/virtio/vhost-vdpa.h | 3 ++ net/vhost-vdpa.c | 112 +++++++++++++++++++++++++++-------------- 3 files changed, 80 insertions(+), 42 deletions(-) diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index d98704a..4010fd9 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -1162,8 +1162,8 @@ static void vhost_vdpa_svq_unmap_ring(struct vhost_vdpa *v, hwaddr addr) vhost_iova_tree_remove(v->shared->iova_tree, *result); } -static void vhost_vdpa_svq_unmap_rings(struct vhost_dev *dev, - const VhostShadowVirtqueue *svq) +void vhost_vdpa_svq_unmap_rings(struct vhost_dev *dev, + const VhostShadowVirtqueue *svq) { struct vhost_vdpa *v = dev->opaque; struct vhost_vring_addr svq_addr; @@ -1346,17 +1346,14 @@ static void vhost_vdpa_svqs_stop(struct vhost_dev *dev) return; } - vhost_vdpa_dma_batch_begin_once(v->shared, v->address_space_id); for (unsigned i = 0; i < v->shadow_vqs->len; ++i) { VhostShadowVirtqueue *svq = g_ptr_array_index(v->shadow_vqs, i); vhost_svq_stop(svq); - vhost_vdpa_svq_unmap_rings(dev, svq); event_notifier_cleanup(&svq->hdev_kick); event_notifier_cleanup(&svq->hdev_call); } - vhost_vdpa_dma_batch_end_once(v->shared, v->address_space_id); } static void vhost_vdpa_suspend(struct vhost_dev *dev) diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h index aa13679..f426e2c 100644 --- a/include/hw/virtio/vhost-vdpa.h +++ b/include/hw/virtio/vhost-vdpa.h @@ -112,6 +112,9 @@ int vhost_vdpa_dma_batch_end_once(VhostVDPAShared *s, uint32_t asid); int vhost_vdpa_load_setup(VhostVDPAShared *s, AddressSpace *dma_as); int vhost_vdpa_load_cleanup(VhostVDPAShared *s, bool vhost_will_start); +void vhost_vdpa_svq_unmap_rings(struct vhost_dev *dev, + const VhostShadowVirtqueue *svq); + typedef struct vdpa_iommu { VhostVDPAShared *dev_shared; IOMMUMemoryRegion *iommu_mr; diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c index 683619f..41714d1 100644 --- a/net/vhost-vdpa.c +++ b/net/vhost-vdpa.c @@ -29,6 +29,7 @@ #include "migration/migration.h" #include "migration/misc.h" #include "hw/virtio/vhost.h" +#include "hw/virtio/vhost-vdpa.h" /* Todo:need to add the multiqueue support here */ typedef struct VhostVDPAState { @@ -467,15 +468,89 @@ static int vhost_vdpa_net_data_load(NetClientState *nc) return 0; } +static void vhost_vdpa_cvq_unmap_buf(struct vhost_vdpa *v, void *addr) +{ + VhostIOVATree *tree = v->shared->iova_tree; + DMAMap needle = { + /* + * No need to specify size or to look for more translations since + * this contiguous chunk was allocated by us. + */ + .translated_addr = (hwaddr)(uintptr_t)addr, + }; + const DMAMap *map = vhost_iova_tree_find_iova(tree, &needle); + int r; + + if (unlikely(!map)) { + error_report("Cannot locate expected map"); + return; + } + + r = vhost_vdpa_dma_unmap(v->shared, v->address_space_id, map->iova, + map->size + 1); + if (unlikely(r != 0)) { + error_report("Device cannot unmap: %s(%d)", g_strerror(r), r); + } + + vhost_iova_tree_remove(tree, *map); +} + static void vhost_vdpa_net_client_stop(NetClientState *nc) { VhostVDPAState *s = DO_UPCAST(VhostVDPAState, nc, nc); + struct vhost_vdpa *v = &s->vhost_vdpa; + struct vhost_vdpa *last_vi = NULL; + bool has_cvq = v->dev->vq_index_end % 2; + int nvqp; assert(nc->info->type == NET_CLIENT_DRIVER_VHOST_VDPA); if (s->vhost_vdpa.index == 0) { migration_remove_notifier(&s->migration_state); } + + if (v->dev->vq_index + v->dev->nvqs != v->dev->vq_index_end) { + return; + } + + nvqp = (v->dev->vq_index_end + 1) / 2; + for (int i = 0; i < nvqp; ++i) { + VhostVDPAState *s_i = vhost_vdpa_net_get_nc_vdpa(s, i); + struct vhost_vdpa *v_i = &s_i->vhost_vdpa; + + if (!v_i->shadow_vqs_enabled) { + continue; + } + if (!last_vi) { + vhost_vdpa_dma_batch_begin_once(v_i->shared, + v_i->address_space_id); + last_vi = v_i; + } else if (last_vi->address_space_id != v_i->address_space_id) { + vhost_vdpa_dma_batch_end_once(last_vi->shared, + last_vi->address_space_id); + vhost_vdpa_dma_batch_begin_once(v_i->shared, + v_i->address_space_id); + last_vi = v_i; + } + + for (unsigned j = 0; j < v_i->shadow_vqs->len; ++j) { + VhostShadowVirtqueue *svq = g_ptr_array_index(v_i->shadow_vqs, j); + + vhost_vdpa_svq_unmap_rings(v_i->dev, svq); + } + } + if (has_cvq) { + if (last_vi) { + assert(last_vi->address_space_id == v->address_space_id); + } + vhost_vdpa_cvq_unmap_buf(&s->vhost_vdpa, s->cvq_cmd_out_buffer); + vhost_vdpa_cvq_unmap_buf(&s->vhost_vdpa, s->status); + } + if (last_vi) { + vhost_vdpa_dma_batch_end_once(last_vi->shared, + last_vi->address_space_id); + last_vi = NULL; + } } static int vhost_vdpa_net_load_setup(NetClientState *nc, NICState *nic) @@ -585,33 +660,6 @@ static int64_t vhost_vdpa_get_vring_desc_group(int device_fd, return state.num; } -static void vhost_vdpa_cvq_unmap_buf(struct vhost_vdpa *v, void *addr) -{ - VhostIOVATree *tree = v->shared->iova_tree; - DMAMap needle = { - /* - * No need to specify size or to look for more translations since - * this contiguous chunk was allocated by us. - */ - .translated_addr = (hwaddr)(uintptr_t)addr, - }; - const DMAMap *map = vhost_iova_tree_find_iova(tree, &needle); - int r; - - if (unlikely(!map)) { - error_report("Cannot locate expected map"); - return; - } - - r = vhost_vdpa_dma_unmap(v->shared, v->address_space_id, map->iova, - map->size + 1); - if (unlikely(r != 0)) { - error_report("Device cannot unmap: %s(%d)", g_strerror(r), r); - } - - vhost_iova_tree_remove(tree, *map); -} - /** Map CVQ buffer. */ static int vhost_vdpa_cvq_map_buf(struct vhost_vdpa *v, void *buf, size_t size, bool write) @@ -740,18 +788,8 @@ err: static void vhost_vdpa_net_cvq_stop(NetClientState *nc) { - VhostVDPAState *s = DO_UPCAST(VhostVDPAState, nc, nc); - struct vhost_vdpa *v = &s->vhost_vdpa; - assert(nc->info->type == NET_CLIENT_DRIVER_VHOST_VDPA); - if (s->vhost_vdpa.shadow_vqs_enabled) { - vhost_vdpa_dma_batch_begin_once(v->shared, v->address_space_id); - vhost_vdpa_cvq_unmap_buf(&s->vhost_vdpa, s->cvq_cmd_out_buffer); - vhost_vdpa_cvq_unmap_buf(&s->vhost_vdpa, s->status); - vhost_vdpa_dma_batch_end_once(v->shared, v->address_space_id); - } - vhost_vdpa_net_client_stop(nc); } From patchwork Thu Dec 7 17:39:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483817 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0ED4CC4167B for ; Thu, 7 Dec 2023 18:55:48 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJT3-000486-10; Thu, 07 Dec 2023 13:51:25 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJT0-00041S-Vh for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:51:23 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSr-0007Du-HA for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:51:22 -0500 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7IJGFg006265; Thu, 7 Dec 2023 18:51:02 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type : content-transfer-encoding; s=corp-2023-11-20; bh=wtgVWgrJosAg3FEJdVTyDsO19N6fZxNG0z4YKtYzny0=; b=g+IGzD55rD/wrCswJER52o+ck8aq66lQ4Q0shwUUt6rS8cUbXs2J0Vttqru/BM7iKy2t waMHtL3CtVfnxe75zxhJkBxehjJBSP0BC8w7DeRUwgItZGQXHFsdGIdwZnvjSHn67+UI L7nPq6aoHixssDgexjg9QjZ57AnXXp1b51ECuLJg680ig+6T9IDB6nBRcScO0x5Z/6kq tk/QpX7WANvPZdBx33tClu/rx1XEBrdAKMKAbh2SRtJpcccW7BVzrvhgU/SZ6w3ogOEa VXDA7cotZd1cujH94CtowRm3VXYfhOMickMO4vfIGZKW3xj/jmIfypqW+hVqb0v+Qogr uQ== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utdabvgef-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:51:02 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7HNN9E039498; Thu, 7 Dec 2023 18:51:01 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vjdn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:51:01 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io910008067; Thu, 7 Dec 2023 18:51:00 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-35; Thu, 07 Dec 2023 18:51:00 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 34/40] vdpa: fix network breakage after cancelling migration Date: Thu, 7 Dec 2023 09:39:47 -0800 Message-Id: <1701970793-6865-35-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-ORIG-GUID: A401oTK9044HLD8zykPrYHl6X1aE1Gvl X-Proofpoint-GUID: A401oTK9044HLD8zykPrYHl6X1aE1Gvl Received-SPF: pass client-ip=205.220.177.32; envelope-from=si-wei.liu@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Fix an issue where cancellation of ongoing migration ends up with no network connectivity. When canceling migration, SVQ will be switched back to the passthrough mode, but the right call fd is not programed to the device and the svq's own call fd is still used. At the point of this transitioning period, the shadow_vqs_enabled hadn't been set back to false yet, causing the installation of call fd inadvertently bypassed. Fixes: a8ac88585da1 ("vhost: Add Shadow VirtQueue call forwarding capabilities") Cc: Eugenio Pérez Signed-off-by: Si-Wei Liu Acked-by: Jason Wang --- hw/virtio/vhost-vdpa.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index 4010fd9..8ba390d 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -1647,7 +1647,12 @@ static int vhost_vdpa_set_vring_call(struct vhost_dev *dev, /* Remember last call fd because we can switch to SVQ anytime. */ vhost_svq_set_svq_call_fd(svq, file->fd); - if (v->shadow_vqs_enabled) { + /* + * In the event of SVQ switching to off, shadow_vqs_enabled has + * not been set to false yet, but the underlying call fd will + * switch back to the guest notifier for passthrough VQs. + */ + if (v->shadow_vqs_enabled && v->shared->svq_switching >= 0) { return 0; } From patchwork Thu Dec 7 17:39:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483827 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C8D6DC4167B for ; Thu, 7 Dec 2023 18:57:16 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJT5-00049C-3D; Thu, 07 Dec 2023 13:51:27 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJT2-00047b-NL for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:51:24 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSt-0007E1-Io for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:51:24 -0500 Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7F2dui022096; Thu, 7 Dec 2023 18:51:03 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=BU8+O9Al6PiQ7o7HvWw6WPaWNTW+Fg1nkZ5IYUEQFOo=; b=CvlGWxjIDOIjN2ywMyRPU7+vQ8N1ExZVEWz1IM9IKMx3yNBPpxKVNQFSPpy4o19NMDH8 pDz32v2tOEQkJw86TmzMGym6tiKLQE+fkbgJvI/BPrqAQ53W89as5DVooglwBavdpwQH wZfUO5qMl7ZMRG4X8LcQpsU+C7q1eXxyG9vLO4C35e423o9HLzHJcTH4K8FZyPpp6JVZ xTBjQ1cPSkE5TY8mt7+Gjb1uEBT1DCQQ1MLwZV4UI+TPJvpDmNmkY/Yi57YClq9D5R7w 7PRwnH0SizCRZbyWA2QYxPlTHv0s2qQzt2A90sVjc87bfTtETbQeSHB/tfk0iTXnU49z 6g== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utd1gcsjj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:51:03 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7InXqE039476; Thu, 7 Dec 2023 18:51:02 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vjed-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:51:02 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io912008067; Thu, 7 Dec 2023 18:51:02 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-36; Thu, 07 Dec 2023 18:51:01 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 35/40] vdpa: add vhost_vdpa_set_address_space_id trace Date: Thu, 7 Dec 2023 09:39:48 -0800 Message-Id: <1701970793-6865-36-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-ORIG-GUID: 3hQOZX71Ea9Xuw3Xdpmb4Z3JMnSrwyOC X-Proofpoint-GUID: 3hQOZX71Ea9Xuw3Xdpmb4Z3JMnSrwyOC Received-SPF: pass client-ip=205.220.177.32; envelope-from=si-wei.liu@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org For better debuggability and observability. Signed-off-by: Si-Wei Liu Reviewed-by: Eugenio Pérez --- net/trace-events | 3 +++ net/vhost-vdpa.c | 3 +++ 2 files changed, 6 insertions(+) diff --git a/net/trace-events b/net/trace-events index 823a071..aab666a 100644 --- a/net/trace-events +++ b/net/trace-events @@ -23,3 +23,6 @@ colo_compare_tcp_info(const char *pkt, uint32_t seq, uint32_t ack, int hdlen, in # filter-rewriter.c colo_filter_rewriter_pkt_info(const char *func, const char *src, const char *dst, uint32_t seq, uint32_t ack, uint32_t flag) "%s: src/dst: %s/%s p: seq/ack=%u/%u flags=0x%x" colo_filter_rewriter_conn_offset(uint32_t offset) ": offset=%u" + +# vhost-vdpa.c +vhost_vdpa_set_address_space_id(void *v, unsigned vq_group, unsigned asid_num) "vhost_vdpa: %p vq_group: %u asid: %u" diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c index 41714d1..84876b0 100644 --- a/net/vhost-vdpa.c +++ b/net/vhost-vdpa.c @@ -30,6 +30,7 @@ #include "migration/misc.h" #include "hw/virtio/vhost.h" #include "hw/virtio/vhost-vdpa.h" +#include "trace.h" /* Todo:need to add the multiqueue support here */ typedef struct VhostVDPAState { @@ -365,6 +366,8 @@ static int vhost_vdpa_set_address_space_id(struct vhost_vdpa *v, }; int r; + trace_vhost_vdpa_set_address_space_id(v, vq_group, asid_num); + r = ioctl(v->shared->device_fd, VHOST_VDPA_SET_GROUP_ASID, &asid); if (unlikely(r < 0)) { error_report("Can't set vq group %u asid %u, errno=%d (%s)", From patchwork Thu Dec 7 17:39:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483825 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4F36FC4167B for ; Thu, 7 Dec 2023 18:57:05 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJT5-00049R-Nf; Thu, 07 Dec 2023 13:51:27 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJT4-00048V-0T for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:51:26 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSu-0007EB-TI for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:51:25 -0500 Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7InWTu000605; Thu, 7 Dec 2023 18:51:04 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=e90zuyHoRqXwhinSX/efqz46AUUPSKC2NBVB0QTvYoA=; b=AhW43zry5NmVUkGNWMl/MqEA69Pn43xSg4wn7+lkXXk5gyIcroi5qURn9lg5cQzjx/T6 NtXug3kvpbDbw7lxAmZn/Yw7OXvF3K8vZ/psmHY4rAYZTHYUjznYW1+xPGWaAcyAHFAW zOdsHZx7Fhq1N4yR3TCysJSt/2h3jNkCg5hvF4OS7Oj9HevxL4bHAukm8EdUYM8atKty Ag2kydiH1JaHSIWv9y7zo8KfnE002UT71QGjTF6XCizc0uvWHJ+yWo4WtZtDdaXes/IT sr5k2aIcH8v6cqu4OzoxRssSum8U3MuKMbJh1LkLuG9U+v+ODeCV0WAV9R2FIfQ1F9V8 AQ== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utd0mmh6c-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:51:04 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7IltLF039527; Thu, 7 Dec 2023 18:51:04 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vjfa-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:51:03 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io914008067; Thu, 7 Dec 2023 18:51:03 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-37; Thu, 07 Dec 2023 18:51:03 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 36/40] vdpa: add vhost_vdpa_get_vring_base trace for svq mode Date: Thu, 7 Dec 2023 09:39:49 -0800 Message-Id: <1701970793-6865-37-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-GUID: 1P7TgylDL8yqSNW9X_W77HbMchrd9so1 X-Proofpoint-ORIG-GUID: 1P7TgylDL8yqSNW9X_W77HbMchrd9so1 Received-SPF: pass client-ip=205.220.165.32; envelope-from=si-wei.liu@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org For better debuggability and observability. Signed-off-by: Si-Wei Liu Reviewed-by: Eugenio Pérez Acked-by: Jason Wang --- hw/virtio/trace-events | 2 +- hw/virtio/vhost-vdpa.c | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index 196f32f..a8d3321 100644 --- a/hw/virtio/trace-events +++ b/hw/virtio/trace-events @@ -58,7 +58,7 @@ vhost_vdpa_set_log_base(void *dev, uint64_t base, unsigned long long size, int r vhost_vdpa_set_vring_addr(void *dev, unsigned int index, unsigned int flags, uint64_t desc_user_addr, uint64_t used_user_addr, uint64_t avail_user_addr, uint64_t log_guest_addr) "dev: %p index: %u flags: 0x%x desc_user_addr: 0x%"PRIx64" used_user_addr: 0x%"PRIx64" avail_user_addr: 0x%"PRIx64" log_guest_addr: 0x%"PRIx64 vhost_vdpa_set_vring_num(void *dev, unsigned int index, unsigned int num) "dev: %p index: %u num: %u" vhost_vdpa_set_vring_base(void *dev, unsigned int index, unsigned int num) "dev: %p index: %u num: %u" -vhost_vdpa_get_vring_base(void *dev, unsigned int index, unsigned int num) "dev: %p index: %u num: %u" +vhost_vdpa_get_vring_base(void *dev, unsigned int index, unsigned int num, bool svq) "dev: %p index: %u num: %u svq: %d" vhost_vdpa_set_vring_kick(void *dev, unsigned int index, int fd) "dev: %p index: %u fd: %d" vhost_vdpa_set_vring_call(void *dev, unsigned int index, int fd) "dev: %p index: %u fd: %d" vhost_vdpa_get_features(void *dev, uint64_t features) "dev: %p features: 0x%"PRIx64 diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index 8ba390d..d66936f 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -1607,6 +1607,7 @@ static int vhost_vdpa_get_vring_base(struct vhost_dev *dev, if (v->shadow_vqs_enabled) { ring->num = virtio_queue_get_last_avail_idx(dev->vdev, ring->index); + trace_vhost_vdpa_get_vring_base(dev, ring->index, ring->num, true); return 0; } @@ -1619,7 +1620,7 @@ static int vhost_vdpa_get_vring_base(struct vhost_dev *dev, } ret = vhost_vdpa_call(dev, VHOST_GET_VRING_BASE, ring); - trace_vhost_vdpa_get_vring_base(dev, ring->index, ring->num); + trace_vhost_vdpa_get_vring_base(dev, ring->index, ring->num, false); return ret; } From patchwork Thu Dec 7 17:39:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483800 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8B317C4167B for ; Thu, 7 Dec 2023 18:51:40 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJT9-0004A1-Ol; Thu, 07 Dec 2023 13:51:32 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJT5-00049I-G8 for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:51:27 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSw-0007EN-9r for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:51:27 -0500 Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7FDI5d009711; Thu, 7 Dec 2023 18:51:07 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=TxIsJRdr2QrceXoIu2cqfn++hSDV86DDXcv+xhCGH68=; b=TAh3wzB0W8Esfs+q5MF00k9wSD20joGtk6aGGJV7gXHBdOtvYYQkE7vqBFifWpdyRDNl t9l455BptTGl/fh03RLIyLOYB51ozOAXQZf/4VgqQ7Uv6D8SbHjL5jQ8ZOdcPGR1NSBJ +u9NwTQ+M0bM0y5Bwe9E/6q+eZEXYldlbYu2K/yDe1clCRGtZsPUJuu19oIO+6lyZyGy vNnLzuScYEwqq+EqhAd8YhW9AYxNwmbCb70e4/RSDZg7Co3MELcLYCfhmUnOxoGZ/UjH fOgNOUDQoGNMGNW2cqHMvxI05wEukw7sZNWVJhmeOpL2gxwqDXx/Mfxu2kAfN5Wy6tzN dg== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utdrvmpg9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:51:06 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7HPF63039562; Thu, 7 Dec 2023 18:51:05 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vjfu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:51:05 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io916008067; Thu, 7 Dec 2023 18:51:04 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-38; Thu, 07 Dec 2023 18:51:04 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 37/40] vdpa: add vhost_vdpa_set_dev_vring_base trace for svq mode Date: Thu, 7 Dec 2023 09:39:50 -0800 Message-Id: <1701970793-6865-38-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-GUID: MBJN8BQ7O_2ZYw-8XBJxT6MgKCpRantJ X-Proofpoint-ORIG-GUID: MBJN8BQ7O_2ZYw-8XBJxT6MgKCpRantJ Received-SPF: pass client-ip=205.220.177.32; envelope-from=si-wei.liu@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org For better debuggability and observability. Signed-off-by: Si-Wei Liu Reviewed-by: Eugenio Pérez Acked-by: Jason Wang --- hw/virtio/trace-events | 2 +- hw/virtio/vhost-vdpa.c | 5 ++++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index a8d3321..5085607 100644 --- a/hw/virtio/trace-events +++ b/hw/virtio/trace-events @@ -57,7 +57,7 @@ vhost_vdpa_dev_start(void *dev, bool started) "dev: %p started: %d" vhost_vdpa_set_log_base(void *dev, uint64_t base, unsigned long long size, int refcnt, int fd, void *log) "dev: %p base: 0x%"PRIx64" size: %llu refcnt: %d fd: %d log: %p" vhost_vdpa_set_vring_addr(void *dev, unsigned int index, unsigned int flags, uint64_t desc_user_addr, uint64_t used_user_addr, uint64_t avail_user_addr, uint64_t log_guest_addr) "dev: %p index: %u flags: 0x%x desc_user_addr: 0x%"PRIx64" used_user_addr: 0x%"PRIx64" avail_user_addr: 0x%"PRIx64" log_guest_addr: 0x%"PRIx64 vhost_vdpa_set_vring_num(void *dev, unsigned int index, unsigned int num) "dev: %p index: %u num: %u" -vhost_vdpa_set_vring_base(void *dev, unsigned int index, unsigned int num) "dev: %p index: %u num: %u" +vhost_vdpa_set_dev_vring_base(void *dev, unsigned int index, unsigned int num, bool svq) "dev: %p index: %u num: %u svq: %d" vhost_vdpa_get_vring_base(void *dev, unsigned int index, unsigned int num, bool svq) "dev: %p index: %u num: %u svq: %d" vhost_vdpa_set_vring_kick(void *dev, unsigned int index, int fd) "dev: %p index: %u fd: %d" vhost_vdpa_set_vring_call(void *dev, unsigned int index, int fd) "dev: %p index: %u fd: %d" diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index d66936f..ff4f218 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -1043,7 +1043,10 @@ static int vhost_vdpa_get_config(struct vhost_dev *dev, uint8_t *config, static int vhost_vdpa_set_dev_vring_base(struct vhost_dev *dev, struct vhost_vring_state *ring) { - trace_vhost_vdpa_set_vring_base(dev, ring->index, ring->num); + struct vhost_vdpa *v = dev->opaque; + + trace_vhost_vdpa_set_dev_vring_base(dev, ring->index, ring->num, + v->shadow_vqs_enabled); return vhost_vdpa_call(dev, VHOST_SET_VRING_BASE, ring); } From patchwork Thu Dec 7 17:39:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483858 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9E7F8C46CA0 for ; Thu, 7 Dec 2023 18:58:00 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJTD-0004R9-KN; Thu, 07 Dec 2023 13:51:35 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJT5-00049K-HA for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:51:27 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSy-0007EX-D1 for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:51:27 -0500 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7EtJQQ015896; Thu, 7 Dec 2023 18:51:08 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=guD/rAL4AbaDzj4AqYJSf7e69luBS3Cz3m5iSUNPG84=; b=fSvndbRqFCmowwBNZziieAJr0s/cAI7xJRbhcr8y05J8Y1EjZQQoveG6j2QEnPCt/tXb f2XZ+dOCjrLTlhJzUrDpHpddEk6q/9p1ltLafYmQEdn5muLkZEKECp2EKHT6bFsgiOT0 iSNQ3zpt4WDJ7z8X6pBSGxAl8ULE8Uo4vq5hHeKYDuEwGRb4DwJE+r0h7vEZ1AclSqMc yMOAzCkGhrNY7qrJxeGnke+XJTVJyk9VMmMOUMsJ7xPQvw2anOEmYgtVkl1Gaw8dtuTO K4sYWo9KojT6cgFFSr8Pu7AzSAaL4GjHtB4i0tO/lr5suzk+L0jDjJ5Yh0R3ulNGTHoR Ng== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utdmbmntt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:51:08 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7InXqH039476; Thu, 7 Dec 2023 18:51:07 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vjgs-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:51:07 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io918008067; Thu, 7 Dec 2023 18:51:06 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-39; Thu, 07 Dec 2023 18:51:06 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 38/40] vdpa: add trace events for eval_flush Date: Thu, 7 Dec 2023 09:39:51 -0800 Message-Id: <1701970793-6865-39-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-ORIG-GUID: aLQAQNz4LNudA147DlFBEZSSvsO5-Ay2 X-Proofpoint-GUID: aLQAQNz4LNudA147DlFBEZSSvsO5-Ay2 Received-SPF: pass client-ip=205.220.177.32; envelope-from=si-wei.liu@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org For better debuggability and observability. Signed-off-by: Si-Wei Liu --- net/trace-events | 2 ++ net/vhost-vdpa.c | 7 +++++++ 2 files changed, 9 insertions(+) diff --git a/net/trace-events b/net/trace-events index aab666a..d650c71 100644 --- a/net/trace-events +++ b/net/trace-events @@ -26,3 +26,5 @@ colo_filter_rewriter_conn_offset(uint32_t offset) ": offset=%u" # vhost-vdpa.c vhost_vdpa_set_address_space_id(void *v, unsigned vq_group, unsigned asid_num) "vhost_vdpa: %p vq_group: %u asid: %u" +vhost_vdpa_net_data_eval_flush(void *s, int qindex, int svq_switch, bool svq_flush) "vhost_vdpa: %p qp: %d svq_switch: %d flush_map: %d" +vhost_vdpa_net_cvq_eval_flush(void *s, int qindex, int svq_switch, bool svq_flush) "vhost_vdpa: %p qp: %d svq_switch: %d flush_map: %d" diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c index 84876b0..a0bd8cd 100644 --- a/net/vhost-vdpa.c +++ b/net/vhost-vdpa.c @@ -608,6 +608,9 @@ static void vhost_vdpa_net_data_eval_flush(NetClientState *nc, bool stop) v->desc_group < 0) { v->shared->flush_map = true; } + trace_vhost_vdpa_net_data_eval_flush(v, s->vhost_vdpa.index, + v->shared->svq_switching, + v->shared->flush_map); } static NetClientInfo net_vhost_vdpa_info = { @@ -1457,6 +1460,10 @@ static void vhost_vdpa_net_cvq_eval_flush(NetClientState *nc, bool stop) !s->cvq_isolated && v->desc_group < 0) { v->shared->flush_map = true; } + + trace_vhost_vdpa_net_cvq_eval_flush(v, s->vhost_vdpa.index, + v->shared->svq_switching, + v->shared->flush_map); } static NetClientInfo net_vhost_vdpa_cvq_info = { From patchwork Thu Dec 7 17:39:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483863 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4BAE4C10DC3 for ; Thu, 7 Dec 2023 18:58:24 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJTK-0004lC-HB; Thu, 07 Dec 2023 13:51:42 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJT7-0004NR-9C for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:51:31 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJSz-0007Et-5a for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:51:28 -0500 Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7EtISd017262; Thu, 7 Dec 2023 18:51:10 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=WPm5hA3Uu3KbbEN+UqK0pesOU6z6VRZ/RI9BjPTbiq4=; b=ZvacEikrNP3nqMuDWQTOwPIW6uSC69dshZ2cDM48Fyb232n5t3y35ZuvaSd0vmuwFjZQ RSrvuBwTjBHaTPlBVt31HKBQle4smP6JDCNW30eYypW3USljaW5TrTQSX/Nx9Dj5Dn/8 X+pj//L+/onGBf+3jwToLLRsDQroiqhmAqCsNLoMq3J0y/tg9tcjKng88RfqfIUExKqx 1qRntUZwhLgxugIaA6cZIbSwi5pdobWKTE6Wu/5OjsaiIzqsxI0KI6YazrBMLFYMiD5z mSX5lUFBmjoFbeEYPYGuNDsBO5Sf/gSo8AkmvvHEC8fJKPYTCQVZqDJD70vslVj8rKxz XA== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utdrvmpgc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:51:09 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7Hb9IL039550; Thu, 7 Dec 2023 18:51:08 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vjht-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:51:08 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io91A008067; Thu, 7 Dec 2023 18:51:08 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-40; Thu, 07 Dec 2023 18:51:08 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 39/40] vdpa: add trace events for vhost_vdpa_net_load_cmd Date: Thu, 7 Dec 2023 09:39:52 -0800 Message-Id: <1701970793-6865-40-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-GUID: SfR0iKqbb0YA5_4aNqhfmt3kn_yoD1WP X-Proofpoint-ORIG-GUID: SfR0iKqbb0YA5_4aNqhfmt3kn_yoD1WP Received-SPF: pass client-ip=205.220.177.32; envelope-from=si-wei.liu@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org For better debuggability and observability. Signed-off-by: Si-Wei Liu Reviewed-by: Eugenio Pérez --- net/trace-events | 2 ++ net/vhost-vdpa.c | 2 ++ 2 files changed, 4 insertions(+) diff --git a/net/trace-events b/net/trace-events index d650c71..be087e6 100644 --- a/net/trace-events +++ b/net/trace-events @@ -28,3 +28,5 @@ colo_filter_rewriter_conn_offset(uint32_t offset) ": offset=%u" vhost_vdpa_set_address_space_id(void *v, unsigned vq_group, unsigned asid_num) "vhost_vdpa: %p vq_group: %u asid: %u" vhost_vdpa_net_data_eval_flush(void *s, int qindex, int svq_switch, bool svq_flush) "vhost_vdpa: %p qp: %d svq_switch: %d flush_map: %d" vhost_vdpa_net_cvq_eval_flush(void *s, int qindex, int svq_switch, bool svq_flush) "vhost_vdpa: %p qp: %d svq_switch: %d flush_map: %d" +vhost_vdpa_net_load_cmd(void *s, uint8_t class, uint8_t cmd, int data_num, int data_size) "vdpa state: %p class: %u cmd: %u sg_num: %d size: %d" +vhost_vdpa_net_load_cmd_retval(void *s, uint8_t class, uint8_t cmd, int r) "vdpa state: %p class: %u cmd: %u retval: %d" diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c index a0bd8cd..61da8b4 100644 --- a/net/vhost-vdpa.c +++ b/net/vhost-vdpa.c @@ -885,6 +885,7 @@ static ssize_t vhost_vdpa_net_load_cmd(VhostVDPAState *s, assert(data_size < vhost_vdpa_net_cvq_cmd_page_len() - sizeof(ctrl)); cmd_size = sizeof(ctrl) + data_size; + trace_vhost_vdpa_net_load_cmd(s, class, cmd, data_num, data_size); if (vhost_svq_available_slots(svq) < 2 || iov_size(out_cursor, 1) < cmd_size) { /* @@ -916,6 +917,7 @@ static ssize_t vhost_vdpa_net_load_cmd(VhostVDPAState *s, r = vhost_vdpa_net_cvq_add(s, &out, 1, &in, 1); if (unlikely(r < 0)) { + trace_vhost_vdpa_net_load_cmd_retval(s, class, cmd, r); return r; } From patchwork Thu Dec 7 17:39:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 13483860 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DECFBC4167B for ; Thu, 7 Dec 2023 18:58:05 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rBJTK-0004jf-ET; Thu, 07 Dec 2023 13:51:42 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJT7-0004NS-EB for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:51:31 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rBJT1-0007FC-MX for qemu-devel@nongnu.org; Thu, 07 Dec 2023 13:51:28 -0500 Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7IhZjh029893; Thu, 7 Dec 2023 18:51:11 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-11-20; bh=p1QP117Xi8eswdzOZl72BbSbFGZ9b4zYrMMFkqmCa5k=; b=mQotVYr50Gy6ZZBRQ4IQ6271jXK0ZGeU9ZEewufwU1StFlfq7c2U/lsJtgnbV7NiORrL ur8H62z2ZcDUdfXHIrWZPKwLeYD1Nlnm+To0k4Ch40UhE5jgu7bwZRJo9SK23MypDKJM TCcnp+SonTtvu2Dg5gka6fx+JMD6//tqgfM+XvyMEbo3VwvG9xtTfsf5T2dMIZM/5+Sk 9SOTHo6EbWi5Nha5YPNQS2nX4KM8kEyAqtJjD1MdUcEKwRCJjAa7sQxaAfYDLIIZrvLJ PaOeiMJJuk0XFfML23Pd4I/QVe68sxjnTjsbnpfiGlDee/HSOCYFE/s7HEpqmgRUVxbZ PQ== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3utd1gcsjq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:51:11 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7HOJfs039780; Thu, 7 Dec 2023 18:51:10 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3utan7vjjs-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Dec 2023 18:51:10 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3B7Io91C008067; Thu, 7 Dec 2023 18:51:09 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3utan7vh2k-41; Thu, 07 Dec 2023 18:51:09 +0000 From: Si-Wei Liu To: eperezma@redhat.com, jasowang@redhat.com, mst@redhat.com, dtatulea@nvidia.com, leiyang@redhat.com, yin31149@gmail.com, boris.ostrovsky@oracle.com, jonah.palmer@oracle.com Cc: qemu-devel@nongnu.org Subject: [PATCH 40/40] vdpa: add trace event for vhost_vdpa_net_load_mq Date: Thu, 7 Dec 2023 09:39:53 -0800 Message-Id: <1701970793-6865-41-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> References: <1701970793-6865-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_15,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070156 X-Proofpoint-ORIG-GUID: 1RoSeJSsy6m9DBEPScuTtfrF6h6ay9KV X-Proofpoint-GUID: 1RoSeJSsy6m9DBEPScuTtfrF6h6ay9KV Received-SPF: pass client-ip=205.220.177.32; envelope-from=si-wei.liu@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org For better debuggability and observability. Signed-off-by: Si-Wei Liu Reviewed-by: Eugenio Pérez --- net/trace-events | 1 + net/vhost-vdpa.c | 2 ++ 2 files changed, 3 insertions(+) diff --git a/net/trace-events b/net/trace-events index be087e6..c128cc4 100644 --- a/net/trace-events +++ b/net/trace-events @@ -30,3 +30,4 @@ vhost_vdpa_net_data_eval_flush(void *s, int qindex, int svq_switch, bool svq_flu vhost_vdpa_net_cvq_eval_flush(void *s, int qindex, int svq_switch, bool svq_flush) "vhost_vdpa: %p qp: %d svq_switch: %d flush_map: %d" vhost_vdpa_net_load_cmd(void *s, uint8_t class, uint8_t cmd, int data_num, int data_size) "vdpa state: %p class: %u cmd: %u sg_num: %d size: %d" vhost_vdpa_net_load_cmd_retval(void *s, uint8_t class, uint8_t cmd, int r) "vdpa state: %p class: %u cmd: %u retval: %d" +vhost_vdpa_net_load_mq(void *s, int ncurqps) "vdpa state: %p current_qpairs: %d" diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c index 61da8b4..17b8d01 100644 --- a/net/vhost-vdpa.c +++ b/net/vhost-vdpa.c @@ -1109,6 +1109,8 @@ static int vhost_vdpa_net_load_mq(VhostVDPAState *s, return 0; } + trace_vhost_vdpa_net_load_mq(s, n->curr_queue_pairs); + mq.virtqueue_pairs = cpu_to_le16(n->curr_queue_pairs); const struct iovec data = { .iov_base = &mq,