From patchwork Fri Apr 1 09:54:42 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Bligh X-Patchwork-Id: 8722081 Return-Path: X-Original-To: patchwork-qemu-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 5F898C0553 for ; Fri, 1 Apr 2016 09:54:58 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 3A2F1203B5 for ; Fri, 1 Apr 2016 09:54:57 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id F025820398 for ; Fri, 1 Apr 2016 09:54:55 +0000 (UTC) Received: from localhost ([::1]:43058 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1alvn5-0006zS-B4 for patchwork-qemu-devel@patchwork.kernel.org; Fri, 01 Apr 2016 05:54:55 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:44389) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1alvmy-0006zK-0M for qemu-devel@nongnu.org; Fri, 01 Apr 2016 05:54:49 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1alvmw-0008Cq-HP for qemu-devel@nongnu.org; Fri, 01 Apr 2016 05:54:47 -0400 Received: from mail.avalus.com ([2001:41c8:10:1dd::10]:42942) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1alvmw-0008Cc-7w for qemu-devel@nongnu.org; Fri, 01 Apr 2016 05:54:46 -0400 Received: by mail.avalus.com (Postfix) with ESMTPSA id 40297C5609E; Fri, 1 Apr 2016 10:54:45 +0100 (BST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=alex.org.uk; s=mail; t=1459504485; bh=1mrWtfKA0N+VpQe2Ryw2IDdJl1gPJb0gteDhgJAwcB0=; h=From:To:Cc:Subject:Date; b=bvYcBJLE7w08GKMFEyijXGoiEvFyeb9UHAzi+0rHITIXwhMJq7OaLNpx12b6e8s5D KmBeeV3jnaQIoiwZqIdjGWNrqMqmK+Jr+YTjBqmCNNzpVPwK4HY3uLQpRVAAZl2uTs 5aqpP7YYOQeamXmuYS1VqxYI8UTjk37/zWEp2UJw= From: Alex Bligh To: Eric Blake , Wouter Verhelst Date: Fri, 1 Apr 2016 10:54:42 +0100 Message-Id: <1459504482-64928-1-git-send-email-alex@alex.org.uk> X-Mailer: git-send-email 1.9.1 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 2001:41c8:10:1dd::10 Cc: "nbd-general@lists.sourceforge.net" , "qemu-devel@nongnu.org" , Alex Bligh Subject: [Qemu-devel] [PATCHv2] Improve documentation of FUA and FLUSH X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org X-Spam-Status: No, score=-6.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI, T_DKIM_INVALID, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Improve the documentation of NBD_CMD_FLUSH and NBD_CMD_FLAG_FUA. Specifically the latter may be set on any command, and its semantics on commands other than NBD_CMD_WRITE need explaining. Further, explain how these relate to reordering of commands. Signed-off-by: Alex Bligh --- doc/proto.md | 83 +++++++++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 68 insertions(+), 15 deletions(-) Changes since v1: * Rebased on master * Eric's modifications * Included 2 options as to how we might handle FUA on non-write commands. Let's establish whether either the kernel or Qemu have meaningful uses for read+FUA. diff --git a/doc/proto.md b/doc/proto.md index 8c83382..389b186 100644 --- a/doc/proto.md +++ b/doc/proto.md @@ -213,6 +213,47 @@ handle as was sent by the client in the corresponding request. In this way, the client can correlate which request is receiving a response. +#### Ordering of messages and writes + +The server MAY process commands out of order, and MAY reply out of +order, save that: + +* All write commands (that includes both `NBD_CMD_WRITE` and + `NBD_CMD_TRIM`) that the server completes (i.e. replies to) + prior to processing to a `NBD_CMD_FLUSH` MUST be written to non-volatile + storage prior to replying to that `NBD_CMD_FLUSH`. The server SHOULD ensure + that all write command received prior to processing the `NBD_CMD_FLUSH` + (whether they are replied to or not) are written to non-volatile + storage prior to processing an `NBD_CMD_FLUSH`; note this is a + stronger condition than the previous 'MUST' condition. This + paragraph only applies if `NBD_FLAG_SEND_FLUSH` is set within + the transmission flags, as otherwise `NBD_CMD_FLUSH` will never + be sent by the client to the server. + +* A server MUST NOT reply to a command that has `NBD_CMD_FLAG_FUA` set + +[Option #C1] + + in its command flags until the data area referred to by that command + +[Option #C1] + + in its command flags until the data (if any) written by that command + +[All options] + + is persisted to non-volatile storage. This only applies if + `NBD_FLAG_SEND_FLUSH` is set within the transmission flags, as otherwise + `NBD_CMD_FLAG_FUA` will not be set on any commands sent to the server + by the client. + +`NBD_CMD_FLUSH` is modelled on the Linux kernel empty bio with +`REQ_FLUSH` set. `NBD_CMD_FLAG_FUA` is modelled on the Linux +kernel bio with `REQ_FUA` set. In case of ambiguity in this +specification, the +[kernel documentation](https://www.kernel.org/doc/Documentation/block/writeback_cache_control.txt) +may be useful. + #### Request message The request message, sent by the client, looks as follows: @@ -480,10 +521,33 @@ affects a particular command. Clients MUST NOT set a command flag bit that is not documented for the particular command; and whether a flag is valid may depend on negotiation during the handshake phase. -- bit 0, `NBD_CMD_FLAG_FUA`; valid during `NBD_CMD_WRITE` and - `NBD_CMD_WRITE_ZEROES` commands. SHOULD be set to 1 if the client requires - "Force Unit Access" mode of operation. MUST NOT be set unless transmission - flags included `NBD_FLAG_SEND_FUA`. +- bit 0, `NBD_CMD_FLAG_FUA`. This flag is valid for all commands. This bit + MUST be set to 0 unless the `NBD_FLAG_SEND_FUA` flag ("Force Unit Access") + was set in the transmission flags field. If the `NBD_FLAG_SEND_FUA` + is set in the transmission flags field, the client MAY set + `NBD_CMD_FLAG_FUA` in any request. If this bit is set, the server + +[Option #C1] + + MUST NOT send a reply until it has ensured that any data referred to + by this request (i.e. written data on a `NBD_CMD_WRITE` or + `NBD_CMD_WRITE_ZEROES`, read data on an `NBD_CMD_READ`) has reached + permanent storage. There will be certain commands + (e.g. `NBD_CMD_DISC`) for which this flag will thus not alter behaviour + (as the command does not refer to any data), in which case the server + MUST ignore this bit. + +[Option #C2] + + MUST NOT send a reply until it has ensured that the data (if any) written + by this request (e.g. by `NBD_CMD_WRITE` or `NBD_CMD_WRITE_ZEROES`) has + reached permanent storage. There will be certain commands (e.g. `NBD_CMD_READ` + and `NBD_CMD_DISC`) for which this flag will thus not alter behaviour + (as the command does not write any data), in which case the server + MUST ignore this bit. + +[All options] + - bit 1, `NBD_CMD_MAY_TRIM`; defined by the experimental `WRITE_ZEROES` extension; see below. - bit 2, `NBD_CMD_FLAG_DF`; defined by the experimental `STRUCTURED_REPLY` @@ -532,12 +596,6 @@ The following request types exist: message. The server MAY send the reply message before the data has reached permanent storage. - If the `NBD_FLAG_SEND_FUA` flag ("Force Unit Access") was set in the - transmission flags field, the client MAY set the flag `NBD_CMD_FLAG_FUA` in - the command flags field. If this flag was set, the server MUST NOT send - the reply until it has ensured that the newly-written data has reached - permanent storage. - If an error occurs, the server SHOULD set the appropriate error code in the error field. The server MAY then close the connection. @@ -749,11 +807,6 @@ one new command and one new command flag. A client MUST NOT send a write zeroes request unless `NBD_FLAG_SEND_WRITE_ZEROES` was set in the transmission flags field. - If the `NBD_FLAG_SEND_FUA` flag was set in the transmission flags field, - the client MAY set the flag `NBD_CMD_FLAG_FUA` in the command flags field. - If this flag was set, the server MUST NOT send the reply until it has - ensured that the newly-zeroed data has reached permanent storage. - If the flag `NBD_CMD_FLAG_MAY_TRIM` was set by the client in the command flags field, the server MAY use trimming to zero out the area, but it MUST ensure that the data reads back as zero.