From patchwork Fri Oct 14 12:23:52 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Wang, Wei W" X-Patchwork-Id: 9376589 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 27E1B6075E for ; Fri, 14 Oct 2016 12:24:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1892E2A5DC for ; Fri, 14 Oct 2016 12:24:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0C04A2A5E8; Fri, 14 Oct 2016 12:24:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 4AC4F2A5DC for ; Fri, 14 Oct 2016 12:24:21 +0000 (UTC) Received: from localhost ([::1]:47068 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bv1X9-0001tC-S2 for patchwork-qemu-devel@patchwork.kernel.org; Fri, 14 Oct 2016 08:24:19 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:36230) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bv1Wq-0001s8-PO for qemu-devel@nongnu.org; Fri, 14 Oct 2016 08:24:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bv1Wm-0007RG-Jo for qemu-devel@nongnu.org; Fri, 14 Oct 2016 08:24:00 -0400 Received: from mga05.intel.com ([192.55.52.43]:19698) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bv1Wm-0007QU-7t for qemu-devel@nongnu.org; Fri, 14 Oct 2016 08:23:56 -0400 Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga105.fm.intel.com with ESMTP; 14 Oct 2016 05:23:53 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.31,344,1473145200"; d="scan'208";a="19636724" Received: from ww-bdw.sh.intel.com ([10.239.48.149]) by orsmga005.jf.intel.com with ESMTP; 14 Oct 2016 05:23:51 -0700 From: Wei Wang To: qemu-devel@nongnu.org, marcandre.lureau@gmail.com, mst@redhat.com, stefanha@redhat.com, pbonzini@redhat.com Date: Fri, 14 Oct 2016 20:23:52 +0800 Message-Id: <1476447832-99544-1-git-send-email-wei.w.wang@intel.com> X-Mailer: git-send-email 1.9.1 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 192.55.52.43 Subject: [Qemu-devel] [PATCH] docs/vhost-user: extend vhost-user to support the vhost-pci based inter-VM communication X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Wei Wang Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Wei Wang --- docs/specs/vhost-user.txt | 127 +++++++++++++++++++++++++++++++++++++++------- 1 file changed, 109 insertions(+), 18 deletions(-) diff --git a/docs/specs/vhost-user.txt b/docs/specs/vhost-user.txt index 7890d71..195fad6 100644 --- a/docs/specs/vhost-user.txt +++ b/docs/specs/vhost-user.txt @@ -17,31 +17,39 @@ The protocol defines 2 sides of the communication, master and slave. Master is the application that shares its virtqueues, in our case QEMU. Slave is the consumer of the virtqueues. -In the current implementation QEMU is the Master, and the Slave is intended to +In the traditional implementation QEMU is the Master, and the Slave is intended to be a software Ethernet switch running in user space, such as Snabbswitch. Master and slave can be either a client (i.e. connecting) or server (listening) in the socket communication. +The current vhost-user protocol is extended to support the vhost-pci based inter-VM +communication. In this case, Slave is a QEMU which runs a vhost-pci server, and +Master is another QEMU which runs a vhost-pci client. + Message Specification --------------------- Note that all numbers are in the machine native byte order. A vhost-user message -consists of 3 header fields and a payload: +consists of 4 header fields and a payload: ------------------------------------- -| request | flags | size | payload | ------------------------------------- +---------------------------------------------- +| request | flags | conn_id | size | payload | +---------------------------------------------- * Request: 32-bit type of the request * Flags: 32-bit bit field: - - Lower 2 bits are the version (currently 0x01) - - Bit 2 is the reply flag - needs to be sent on each reply from the slave + - Lower 2 bits are the version (currently 0x02) + - Bit 2 is the reply flag - needs to be sent on each reply - Bit 3 is the need_reply flag - see VHOST_USER_PROTOCOL_F_REPLY_ACK for details. + * Conn_id: 64-bit connection id to indentify a client socket connection. It is + introduced in version 0x02 to support the "1-server-N-client" model + and an asynchronous client read implementation. The connection id, + 0xFFFFFFFFFFFFFFFF, is used by an anonymous client (e.g. a client who + has not got its connection id from the server in the initial talk) * Size - 32-bit size of the payload - Depending on the request type, payload can be: * A single 64-bit integer @@ -72,10 +80,12 @@ Depending on the request type, payload can be: Log: a 64-bit guest address for logging * Memory regions description - --------------------------------------------------- - | num regions | padding | region0 | ... | region7 | - --------------------------------------------------- + ---------------------------------------------------------- + | uuid | num regions | padding | region0 | ... | region7 | + ---------------------------------------------------------- + UUID: a 128-bit identification of the QEMU that the memory region + belongs to Num regions: a 32-bit number of regions Padding: 32-bit @@ -97,6 +107,19 @@ Depending on the request type, payload can be: log offset: offset from start of supplied file descriptor where logging starts (i.e. where guest address 0 would be logged) +* UUID description + -------- + | uuid | + -------- + uuid: 128 bit uuid + +* Rx queue gsi description + -------------------------------------- + | num rx queues | gsi[num rx queues] | + -------------------------------------- + num rx queues: 16 bit. + gsi[]: stores the GSI of each rx queue. + In QEMU the vhost-user message is implemented with the following struct: typedef struct VhostUserMsg { @@ -109,6 +132,8 @@ typedef struct VhostUserMsg { struct vhost_vring_addr addr; VhostUserMemory memory; VhostUserLog log; + uuit_t uuid; + RxQueueGSI rx_gsi; }; } QEMU_PACKED VhostUserMsg; @@ -119,17 +144,27 @@ The protocol for vhost-user is based on the existing implementation of vhost for the Linux Kernel. Most messages that can be sent via the Unix domain socket implementing vhost-user have an equivalent ioctl to the kernel implementation. -The communication consists of master sending message requests and slave sending -message replies. Most of the requests don't require replies. Here is a list of -the ones that do: +Traditionally, the communication consists of master sending message requests +and slave sending message replies. Most of the requests don't require replies. +Here is a list of the ones that do: - * VHOST_GET_FEATURES - * VHOST_GET_PROTOCOL_FEATURES - * VHOST_GET_VRING_BASE - * VHOST_SET_LOG_BASE (if VHOST_USER_PROTOCOL_F_LOG_SHMFD) + * VHOST_USER_GET_FEATURES + * VHOST_USER_GET_PROTOCOL_FEATURES + * VHOST_USER_GET_VRING_BASE + * VHOST_USER_SET_LOG_BASE (if VHOST_USER_PROTOCOL_F_LOG_SHMFD) + * VHOST_USER_GET_CONN_ID + * VHOST_USER_SET_PVI_SENDER_UUID + * VHOST_USER_SET_PEER_CONNECTION [ Also see the section on REPLY_ACK protocol extension. ] +Currently, the communiaction also supports the Slave (server) sending messages or +asynchronously replying to the Master (client). Here is a list of them: + * VHOST_USER_SET_RX_QUEUE_GSI + * VHOST_USER_SET_PVI_SENDER_UUID + * VHOST_USER_SET_PEER_CONNECTION (the serve may actively request to disconnect + with the client) + There are several messages that the master sends with file descriptors passed in the ancillary data: @@ -259,6 +294,7 @@ Protocol features #define VHOST_USER_PROTOCOL_F_LOG_SHMFD 1 #define VHOST_USER_PROTOCOL_F_RARP 2 #define VHOST_USER_PROTOCOL_F_REPLY_ACK 3 +#define VHOST_USER_PROTOCOL_F_VHOST_PCI 4 Message types ------------- @@ -470,6 +506,61 @@ Message types The first 6 bytes of the payload contain the mac address of the guest to allow the vhost user backend to construct and broadcast the fake RARP. + * VHOST_USER_GET_CONN_ID + Id: 20 + Equivalent ioctl: N/A + Master payload: u64 + + The client sends this message to the server to ask for its connection id. + The connection id is then put into the message header (the conn_id field), + so that the server can always know who it is talking with. + + * VHOST_USER_SET_DEV_TYPE + Id: 21 + Equivalent ioctl: N/A + Master payload: u64 + + The client sends the necessary info about the producer virtio device to + the server. + This request should be sent only when VHOST_USER_PROTOCOL_F_VHOST_PCI has + been negotiated. + #define VHOST_USER_DEV_TYPE_NET 1 + + * VHOST_USER_SET_PVI_SENDER_UUID + Id: 22 + Equivalent ioctl: N/A + Master payload: uuid description + + The producer sends its QEMU uuid to the consumer, the vhost-pci device, to + enable sending PV interrupts to the consumer, and vice versa. + + * VHOST_USER_SET_RX_QUEUE_GSI + Id: 23 + Equivalent ioctl: N/A + Master payload: rx queue gsi description + + The producer device sends its rx queues' GSIs to its consumer device. + The consumer device also sends one of its rx queues' GSI to the producer + device after the queue's GSI is assigned. + This request should be sent only when VHOST_USER_PROTOCOL_F_VHOST_PCI has + been negotiated. + + * VHOST_USER_SET_PEER_CONNECTION + Id: 24 + Equivalent ioctl: N/A + Master payload: u64 + + The producer device requests to connect or disconnect with the consumer device. + The consumer device may request to disconnect with the producer device. This + request should be sent only when VHOST_USER_PROTOCOL_F_VHOST_PCI has been + negotiated. + Connection request: If the reply message indicates "success", the vhost-pci based + inter-VM communication channel has been established. + Disconnection request: If the reply message indicates "success", the vhost-pci based + inter-VM communication channel has been destroyed. + #define VHOST_USER_SET_PEER_CONNECTION_F_DISCONNECT 0 + #define VHOST_USER_SET_PEER_CONNECTION_F_CONNECT 1 + VHOST_USER_PROTOCOL_F_REPLY_ACK: ------------------------------- The original vhost-user specification only demands replies for certain