From patchwork Mon Oct 24 07:10:02 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Wang, Wei W" X-Patchwork-Id: 9391531 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 4E2F760231 for ; Mon, 24 Oct 2016 07:11:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3B06028A20 for ; Mon, 24 Oct 2016 07:11:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2FC6A28A83; Mon, 24 Oct 2016 07:11:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 6B99A28A20 for ; Mon, 24 Oct 2016 07:11:19 +0000 (UTC) Received: from localhost ([::1]:44993 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1byZPi-0004DT-L0 for patchwork-qemu-devel@patchwork.kernel.org; Mon, 24 Oct 2016 03:11:18 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37743) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1byZOY-0003br-Si for qemu-devel@nongnu.org; Mon, 24 Oct 2016 03:10:11 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1byZOV-0006pj-Mg for qemu-devel@nongnu.org; Mon, 24 Oct 2016 03:10:06 -0400 Received: from mga11.intel.com ([192.55.52.93]:23663) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1byZOV-0006p7-Ba for qemu-devel@nongnu.org; Mon, 24 Oct 2016 03:10:03 -0400 Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga102.fm.intel.com with ESMTP; 24 Oct 2016 00:10:03 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos; i="5.31,541,1473145200"; d="scan'208"; a="1074733783" Received: from ww-bdw.sh.intel.com ([10.239.48.149]) by fmsmga002.fm.intel.com with ESMTP; 24 Oct 2016 00:10:00 -0700 From: Wei Wang To: virtio-comment@lists.oasis-open.org, qemu-devel@nongnu.org, mst@redhat.com, marcandre.lureau@gmail.com, stefanha@redhat.com, pbonzini@redhat.com Date: Mon, 24 Oct 2016 15:10:02 +0800 Message-Id: <1477293002-144213-1-git-send-email-wei.w.wang@intel.com> X-Mailer: git-send-email 1.9.1 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 192.55.52.93 Subject: [Qemu-devel] [PATCH v1] docs/vhost-user: extend the vhost-user protocol to support the vhost-pci based inter-vm communication X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Wei Wang Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Wei Wang --- docs/specs/vhost-user.txt | 81 +++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 72 insertions(+), 9 deletions(-) diff --git a/docs/specs/vhost-user.txt b/docs/specs/vhost-user.txt index 7890d71..173f693 100644 --- a/docs/specs/vhost-user.txt +++ b/docs/specs/vhost-user.txt @@ -17,28 +17,37 @@ The protocol defines 2 sides of the communication, master and slave. Master is the application that shares its virtqueues, in our case QEMU. Slave is the consumer of the virtqueues. -In the current implementation QEMU is the Master, and the Slave is intended to +In the traditional implementation QEMU is the Master, and the Slave is intended to be a software Ethernet switch running in user space, such as Snabbswitch. Master and slave can be either a client (i.e. connecting) or server (listening) in the socket communication. +The current vhost-user protocol is extended to support the vhost-pci based inter-VM +communication. In this case, Slave is a QEMU which runs a vhost-pci server, and +Master is another QEMU which runs a vhost-pci client. + Message Specification --------------------- Note that all numbers are in the machine native byte order. A vhost-user message -consists of 3 header fields and a payload: +consists of 4 header fields and a payload: ------------------------------------- -| request | flags | size | payload | ------------------------------------- +---------------------------------------------- +| request | flags | conn_id | size | payload | +---------------------------------------------- * Request: 32-bit type of the request * Flags: 32-bit bit field: - Lower 2 bits are the version (currently 0x01) - - Bit 2 is the reply flag - needs to be sent on each reply from the slave + - Bit 2 is the reply flag - needs to be sent on each reply - Bit 3 is the need_reply flag - see VHOST_USER_PROTOCOL_F_REPLY_ACK for details. + * Conn_id: 64-bit connection id to indentify a client socket connection. It is + introduced in version 0x02 to support the "1-server-N-client" model + and an asynchronous client read implementation. The connection id, + 0xFFFFFFFFFFFFFFFF, is used by an anonymous client (e.g. a client who + has not got its connection id from the server in the initial talk) * Size - 32-bit size of the payload @@ -97,6 +106,13 @@ Depending on the request type, payload can be: log offset: offset from start of supplied file descriptor where logging starts (i.e. where guest address 0 would be logged) +* Device info + -------------------- + | virito id | uuid | + -------------------- + Virtio id: 16-bit virtio id of the device + UUID: 128-bit UUID to identify the QEMU instance that creates the device + In QEMU the vhost-user message is implemented with the following struct: typedef struct VhostUserMsg { @@ -109,6 +125,7 @@ typedef struct VhostUserMsg { struct vhost_vring_addr addr; VhostUserMemory memory; VhostUserLog log; + DeviceInfo dev_info; }; } QEMU_PACKED VhostUserMsg; @@ -119,17 +136,25 @@ The protocol for vhost-user is based on the existing implementation of vhost for the Linux Kernel. Most messages that can be sent via the Unix domain socket implementing vhost-user have an equivalent ioctl to the kernel implementation. -The communication consists of master sending message requests and slave sending -message replies. Most of the requests don't require replies. Here is a list of -the ones that do: +Traditionally, the communication consists of master sending message requests +and slave sending message replies. Most of the requests don't require replies. +Here is a list of the ones that do: * VHOST_GET_FEATURES * VHOST_GET_PROTOCOL_FEATURES * VHOST_GET_VRING_BASE * VHOST_SET_LOG_BASE (if VHOST_USER_PROTOCOL_F_LOG_SHMFD) + * VHOST_USER_GET_CONN_ID + * VHOST_USER_SET_PEER_CONNECTION [ Also see the section on REPLY_ACK protocol extension. ] +Currently, the communication also supports the Slave (server) sending messages +to the Master (client). Here is a list of them: + * VHOST_USER_SET_FEATURES + * VHOST_USER_SET_PEER_CONNECTION (the serve may actively request to disconnect + with the client) + There are several messages that the master sends with file descriptors passed in the ancillary data: @@ -259,6 +284,7 @@ Protocol features #define VHOST_USER_PROTOCOL_F_LOG_SHMFD 1 #define VHOST_USER_PROTOCOL_F_RARP 2 #define VHOST_USER_PROTOCOL_F_REPLY_ACK 3 +#define VHOST_USER_PROTOCOL_F_VHOST_PCI 4 Message types ------------- @@ -470,6 +496,43 @@ Message types The first 6 bytes of the payload contain the mac address of the guest to allow the vhost user backend to construct and broadcast the fake RARP. + * VHOST_USER_GET_CONN_ID + + Id: 20 + Equivalent ioctl: N/A + Master payload: u64 + + The client sends this message to the server to ask for its connection id. + The connection id is then put into the message header (the conn_id field), + so that the server can always know who it is talking with. + +* VHOST_USER_SET_DEV_INFO + + Id: 21 + Equivalent ioctl: N/A + Master payload: dev info + + The client sends the producer device info to the server. + This request should be sent only when VHOST_USER_PROTOCOL_F_VHOST_PCI has + been negotiated. + +* VHOST_USER_SET_PEER_CONNECTION + + Id: 22 + Equivalent ioctl: N/A + Master payload: u64 + + The producer device requests to connect or disconnect to the consumer device. + The consumer device may request to disconnect to the producer device. This + request should be sent only when VHOST_USER_PROTOCOL_F_VHOST_PCI has been + negotiated. + Connection request: If the reply message indicates "success", the vhost-pci based + inter-VM communication channel has been established. + Disconnection request: If the reply message indicates "success", the vhost-pci based + inter-VM communication channel has been destroyed. + #define VHOST_USER_SET_PEER_CONNECTION_F_OFF 0 + #define VHOST_USER_SET_PEER_CONNECTION_F_ON 1 + VHOST_USER_PROTOCOL_F_REPLY_ACK: ------------------------------- The original vhost-user specification only demands replies for certain