From patchwork Tue Mar 16 19:43:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 12143719 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA078C43332 for ; Tue, 16 Mar 2021 19:45:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7F7A264F75 for ; Tue, 16 Mar 2021 19:45:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240631AbhCPTpH (ORCPT ); Tue, 16 Mar 2021 15:45:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35666 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240580AbhCPToW (ORCPT ); Tue, 16 Mar 2021 15:44:22 -0400 Received: from mail-pf1-x429.google.com (mail-pf1-x429.google.com [IPv6:2607:f8b0:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 74B8AC06174A for ; Tue, 16 Mar 2021 12:44:22 -0700 (PDT) Received: by mail-pf1-x429.google.com with SMTP id e26so9536497pfd.9 for ; Tue, 16 Mar 2021 12:44:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ODJiWJ3Wy16MBWPH/OnBJ1HXlU+a5X9xcdG8fXTHrSk=; b=bRyG+Au/wiDppJgxVCZE8FFpwYZeBYvle5OITPi4RlDm4dUNjZs0r3Nb6YAnz48b5n a7GQezbZX4xntcsJZvf9+8vyD1lT5iqVVDwbYZEF/gKbzBVbDyWMJUsDHnHTDvd97lnk 9uQ7OGqixQg7K6Gc1k0URquk/f3o9WBn315PmGg34zTbJuYqenZ6HergJBkJ7O1cWl8w ga4ygVz2FgAl3SeHL70OeozVsQZcU5BtDle1iUKX1bTRSSqywpupGhhBDzGCAoPzUeDh /HswDRP3kp3RfN7QhJYw9i3R4ZkuVxF3QqZTx0cZdvRIIRIO9uQkA6Uh23AToc3G5t9d OxxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ODJiWJ3Wy16MBWPH/OnBJ1HXlU+a5X9xcdG8fXTHrSk=; b=JAkLriJUn8jzaWdP9UV/1mSmQk/WFHDiLnx1q3bCxM/gSBjrjkqWpDeysSSrcepytV lDxkW8By9XXjZns630tbIFGh9HBmy5l8QBLH/ZWqDb4/dKB6TkjqqLqpHV5H3XPjqUff Tgb0CHTJYVrkvR1ol7gKfdOUFNszWM3jrGm3tQN38EsLwKHjBwJPtbyR22cJ3+RjQKOB z6H9VU6FhB2/RpttF8AwddY1XZL2hQrrE26QW3QtPGzao+hiMs210GDcBrLC6SgQjnIP SLEEUnkiqr+Cqhr6kXleSo7i7kLPtaDwZzZu6RCEdhc881kQqsLkZD4whfO8xFNMgngU RCZw== X-Gm-Message-State: AOAM5325Z7ncO8fWeIEPF9IdhkxxK/cYo7HSnBgON8CdhjuZVC4vGuES 2ltnGOpX4Aw2vqmPUM+5cMeVVzf1pl23DA== X-Google-Smtp-Source: ABdhPJxzP5x/8N+oUauITZua+wsFjt5mGiPThczxV77lBUq+Z7Wc2STAd5hZZJC241bl90Xhq/UuVw== X-Received: by 2002:a63:4f59:: with SMTP id p25mr1104170pgl.335.1615923861406; Tue, 16 Mar 2021 12:44:21 -0700 (PDT) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:532]) by smtp.gmail.com with ESMTPSA id w22sm16919104pfi.133.2021.03.16.12.44.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Mar 2021 12:44:20 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v4 1/5] btrfs: add send stream v2 definitions Date: Tue, 16 Mar 2021 12:43:50 -0700 Message-Id: <1bbc7fbf0efb06a0d91cb029c1a5ab5f4525c71c.1615922753.git.osandov@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Omar Sandoval This adds the definitions of the new commands for send stream version 2 and their respective attributes: fallocate, FS_IOC_SETFLAGS (a.k.a. chattr), and encoded writes. It also documents two changes to the send stream format in v2: the receiver shouldn't assume a maximum command size, and the DATA attribute is encoded differently to allow for writes larger than 64k. These will be implemented in subsequent changes, and then the ioctl will accept the new flags. Reviewed-by: Josef Bacik Signed-off-by: Omar Sandoval --- fs/btrfs/send.c | 2 +- fs/btrfs/send.h | 30 +++++++++++++++++++++++++++++- include/uapi/linux/btrfs.h | 13 +++++++++++++ 3 files changed, 43 insertions(+), 2 deletions(-) diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c index 3bcbf2bcb869..d07570588a16 100644 --- a/fs/btrfs/send.c +++ b/fs/btrfs/send.c @@ -7267,7 +7267,7 @@ long btrfs_ioctl_send(struct file *mnt_file, struct btrfs_ioctl_send_args *arg) sctx->clone_roots_cnt = arg->clone_sources_count; - sctx->send_max_size = BTRFS_SEND_BUF_SIZE; + sctx->send_max_size = BTRFS_SEND_BUF_SIZE_V1; sctx->send_buf = kvmalloc(sctx->send_max_size, GFP_KERNEL); if (!sctx->send_buf) { ret = -ENOMEM; diff --git a/fs/btrfs/send.h b/fs/btrfs/send.h index de91488b7cd0..9f4f7b96b1eb 100644 --- a/fs/btrfs/send.h +++ b/fs/btrfs/send.h @@ -12,7 +12,11 @@ #define BTRFS_SEND_STREAM_MAGIC "btrfs-stream" #define BTRFS_SEND_STREAM_VERSION 1 -#define BTRFS_SEND_BUF_SIZE SZ_64K +/* + * In send stream v1, no command is larger than 64k. In send stream v2, no limit + * should be assumed. + */ +#define BTRFS_SEND_BUF_SIZE_V1 SZ_64K enum btrfs_tlv_type { BTRFS_TLV_U8, @@ -76,6 +80,13 @@ enum btrfs_send_cmd { BTRFS_SEND_C_END, BTRFS_SEND_C_UPDATE_EXTENT, + + /* The following commands were added in send stream v2. */ + + BTRFS_SEND_C_FALLOCATE, + BTRFS_SEND_C_SETFLAGS, + BTRFS_SEND_C_ENCODED_WRITE, + __BTRFS_SEND_C_MAX, }; #define BTRFS_SEND_C_MAX (__BTRFS_SEND_C_MAX - 1) @@ -106,6 +117,11 @@ enum { BTRFS_SEND_A_PATH_LINK, BTRFS_SEND_A_FILE_OFFSET, + /* + * In send stream v2, this attribute is special: it must be the last + * attribute in a command, its header contains only the type, and its + * length is implicitly the remaining length of the command. + */ BTRFS_SEND_A_DATA, BTRFS_SEND_A_CLONE_UUID, @@ -114,6 +130,18 @@ enum { BTRFS_SEND_A_CLONE_OFFSET, BTRFS_SEND_A_CLONE_LEN, + /* The following attributes were added in send stream v2. */ + + BTRFS_SEND_A_FALLOCATE_MODE, + + BTRFS_SEND_A_SETFLAGS_FLAGS, + + BTRFS_SEND_A_UNENCODED_FILE_LEN, + BTRFS_SEND_A_UNENCODED_LEN, + BTRFS_SEND_A_UNENCODED_OFFSET, + BTRFS_SEND_A_COMPRESSION, + BTRFS_SEND_A_ENCRYPTION, + __BTRFS_SEND_A_MAX, }; #define BTRFS_SEND_A_MAX (__BTRFS_SEND_A_MAX - 1) diff --git a/include/uapi/linux/btrfs.h b/include/uapi/linux/btrfs.h index 5df73001aad4..93aa0932234e 100644 --- a/include/uapi/linux/btrfs.h +++ b/include/uapi/linux/btrfs.h @@ -770,6 +770,19 @@ struct btrfs_ioctl_received_subvol_args { */ #define BTRFS_SEND_FLAG_OMIT_END_CMD 0x4 +/* + * Use version 2 of the send stream, which adds new commands and supports larger + * writes. + */ +#define BTRFS_SEND_FLAG_STREAM_V2 0x8 + +/* + * Send compressed data using the ENCODED_WRITE command instead of decompressing + * the data and sending it with the WRITE command. This requires + * BTRFS_SEND_FLAG_STREAM_V2. + */ +#define BTRFS_SEND_FLAG_COMPRESSED 0x10 + #define BTRFS_SEND_FLAG_MASK \ (BTRFS_SEND_FLAG_NO_FILE_DATA | \ BTRFS_SEND_FLAG_OMIT_STREAM_HEADER | \ From patchwork Tue Mar 16 19:43:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 12143725 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 998ECC433DB for ; Tue, 16 Mar 2021 19:45:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5C86464E33 for ; Tue, 16 Mar 2021 19:45:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233080AbhCPTp1 (ORCPT ); Tue, 16 Mar 2021 15:45:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35698 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240546AbhCPTo3 (ORCPT ); Tue, 16 Mar 2021 15:44:29 -0400 Received: from mail-pf1-x432.google.com (mail-pf1-x432.google.com [IPv6:2607:f8b0:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2EB40C06174A for ; Tue, 16 Mar 2021 12:44:29 -0700 (PDT) Received: by mail-pf1-x432.google.com with SMTP id r16so8220790pfh.10 for ; Tue, 16 Mar 2021 12:44:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=pgCBjItU1Bta5sWTZqTeK5Gpn8XCDGtd/okV9d9rprQ=; b=q8myNonQgk/lwMOmpbeD7ayFDoBQYXWeVfbd7jj2SegF4L0CKi2B+97D33lyNL0Due AVuB4xTxjj+Z76hQgc7OLefXB5ZgHfZriYesu+/73DWfv/+2vg02zpXn0dx9LU9L0u+Q jFEcdd96gKVEvnO+Ft1ucgv/3qVpurtstV3OVGT+Q0Fa/aaiHgTSZYW65mh4yReIGaRi tx32LibP68ZolKPFwhmTcYNZ7KxUfeBz57fvkRjTGZMIgAnRJA9BnChij70tUxdT7dzR oapJ9DioI+ADanLc//OJlyeF0xfaT7IkKzeFh9QjWW9C9TATsrWzsVx/5yUoSPdAJLrF kFiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pgCBjItU1Bta5sWTZqTeK5Gpn8XCDGtd/okV9d9rprQ=; b=n7+AXvqqHKDFm9rlE3rVvEtZ0Ai31iGmDWQIy08iZRPWZUtvNtIhPazLuG7EU6LVJk nIWV2xh1PdRk38qj09hwbLWS5jQX8f+rJd6KpfIsEWwcot2YQfkoNMoFioXZ1LTiNKmu lWgNIL3RZck9FOWiSRR6Oshj8Z+lFG+7RFChtbCL0ppq3dlI2VrV4J2e0Du5lhkFQv7u MeqM/77ZAiaT8e+E5y9a3Cm6kP695tNuDkpiQRxD1HmPWo8/sCNb7WiSp49mSLHCaq/1 yz4Knf4T9yrnyM4JWO705k1+aqYsx6JzXPP6nWtqJlXUqs3o1jaCpBXaB7KHTxeBosR7 AJxA== X-Gm-Message-State: AOAM530LHjK1aOmOajrMHRD/DN8+cYvnip4tkiDIY8IuUxJjAsJvMcT8 hgkRSg4msht1FrXgcxsnarW2+1ol2wPZFg== X-Google-Smtp-Source: ABdhPJwECwOHcP1XYFx414JGUslZxeYx+znORTdXuLzMaIayjgH8Evtlzi/IEpTYzFGICjML8dbLrQ== X-Received: by 2002:a62:2585:0:b029:1fb:bd86:2008 with SMTP id l127-20020a6225850000b02901fbbd862008mr976731pfl.77.1615923868014; Tue, 16 Mar 2021 12:44:28 -0700 (PDT) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:532]) by smtp.gmail.com with ESMTPSA id w22sm16919104pfi.133.2021.03.16.12.44.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Mar 2021 12:44:26 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v4 2/5] btrfs: send: write larger chunks when using stream v2 Date: Tue, 16 Mar 2021 12:43:53 -0700 Message-Id: X-Mailer: git-send-email 2.30.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Omar Sandoval The length field of the send stream TLV header is 16 bits. This means that the maximum amount of data that can be sent for one write is 64k minus one. However, encoded writes must be able to send the maximum compressed extent (128k) in one command. To support this, send stream version 2 encodes the DATA attribute differently: it has no length field, and the length is implicitly up to the end of containing command (which has a 32-bit length field). Although this is necessary for encoded writes, normal writes can benefit from it, too. For v2, let's bump up the send buffer to the maximum compressed extent size plus 16k for the other metadata (144k total). Since this will most likely be vmalloc'd (and always will be after the next commit), we round it up to the next page since we might as well use the rest of the page on systems with >16k pages. Signed-off-by: Omar Sandoval --- fs/btrfs/send.c | 34 ++++++++++++++++++++++++++-------- 1 file changed, 26 insertions(+), 8 deletions(-) diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c index d07570588a16..98948568017c 100644 --- a/fs/btrfs/send.c +++ b/fs/btrfs/send.c @@ -4915,14 +4915,27 @@ static inline u64 max_send_read_size(const struct send_ctx *sctx) static int put_data_header(struct send_ctx *sctx, u32 len) { - struct btrfs_tlv_header *hdr; + if (sctx->flags & BTRFS_SEND_FLAG_STREAM_V2) { + /* + * In v2, the data attribute header doesn't include a length; it + * is implicitly to the end of the command. + */ + if (sctx->send_max_size - sctx->send_size < 2 + len) + return -EOVERFLOW; + put_unaligned_le16(BTRFS_SEND_A_DATA, + sctx->send_buf + sctx->send_size); + sctx->send_size += 2; + } else { + struct btrfs_tlv_header *hdr; - if (sctx->send_max_size - sctx->send_size < sizeof(*hdr) + len) - return -EOVERFLOW; - hdr = (struct btrfs_tlv_header *)(sctx->send_buf + sctx->send_size); - put_unaligned_le16(BTRFS_SEND_A_DATA, &hdr->tlv_type); - put_unaligned_le16(len, &hdr->tlv_len); - sctx->send_size += sizeof(*hdr); + if (sctx->send_max_size - sctx->send_size < sizeof(*hdr) + len) + return -EOVERFLOW; + hdr = (struct btrfs_tlv_header *)(sctx->send_buf + + sctx->send_size); + put_unaligned_le16(BTRFS_SEND_A_DATA, &hdr->tlv_type); + put_unaligned_le16(len, &hdr->tlv_len); + sctx->send_size += sizeof(*hdr); + } return 0; } @@ -7267,7 +7280,12 @@ long btrfs_ioctl_send(struct file *mnt_file, struct btrfs_ioctl_send_args *arg) sctx->clone_roots_cnt = arg->clone_sources_count; - sctx->send_max_size = BTRFS_SEND_BUF_SIZE_V1; + if (sctx->flags & BTRFS_SEND_FLAG_STREAM_V2) { + sctx->send_max_size = ALIGN(SZ_16K + BTRFS_MAX_COMPRESSED, + PAGE_SIZE); + } else { + sctx->send_max_size = BTRFS_SEND_BUF_SIZE_V1; + } sctx->send_buf = kvmalloc(sctx->send_max_size, GFP_KERNEL); if (!sctx->send_buf) { ret = -ENOMEM; From patchwork Tue Mar 16 19:43:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 12143729 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 886C8C4332B for ; Tue, 16 Mar 2021 19:45:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6CDA464F77 for ; Tue, 16 Mar 2021 19:45:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240653AbhCPTpi (ORCPT ); Tue, 16 Mar 2021 15:45:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35718 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240594AbhCPTof (ORCPT ); Tue, 16 Mar 2021 15:44:35 -0400 Received: from mail-pg1-x52a.google.com (mail-pg1-x52a.google.com [IPv6:2607:f8b0:4864:20::52a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6E01CC061764 for ; Tue, 16 Mar 2021 12:44:31 -0700 (PDT) Received: by mail-pg1-x52a.google.com with SMTP id p21so23307024pgl.12 for ; Tue, 16 Mar 2021 12:44:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=afne12eWxC99ZM3RzCisiw1S1UD3p193Z3PzLDJZPos=; b=gjVKxFNzVzllJs/RT8W4Egbh/OE3HnDf+Y0CMunCLsQxm014Xg6DKCojAJJe5I4a2P J+4pnnGsjw4pa274nBt037XOwFcOOJk9V5VaJy98lcQ1oDXXEhyh/PtELyC4Z/gKQExY XzrLQ8UQszcVkcb/Q414EFgaAa3L3cgrFGecymz+noqUHKjU76ADky4mrcK0npU2glI6 3eMgoZ0RcdFu/vkPDQXwocLDa2/cap5PltYtN4MdcDxdWiGUiqgEcnWdwriCRUBi91xa jvGmmOscb+iWggMmd9+fU5O+EfsjQjD3bZRV0B/vTIxNXloOKaqVBM08Sf2P5PyTgjA/ aUSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=afne12eWxC99ZM3RzCisiw1S1UD3p193Z3PzLDJZPos=; b=X5AJdIkf0n+g7yEh4kxzcxeFv+rjm462eDhNAmuVakWlJMCdZvIXnGK2plrfKym8ZP NcT13EuKMpMaPKyv5illsRPBQM2924Id5VOM9v6iGVxIfK7kGbn9D8le/ZhqtINHw5jT knX76/T8YsvK6icuy8cq+T2RrvcV0Fvv4NStoRtedYiGxR4s7dJ2Mo+fPefs7GpiLOjK ctvF0Ox7m6Pgb+yg0xj+kC+mKeFzXlC6tTUoyqYVmk2SoxO0EllwaQ+hKReir+PM+nA8 NegvHasYiFIPhaZzLKDKHs0RLDph6qRw7NHWk3N4BB3IpH8mVh0luFGRedt8xrIxtcq0 WuBg== X-Gm-Message-State: AOAM530pHcFWnO7Fk5eN7wp3J2qaRFx328Ely/hBviGnGMAl4jx8inpz b7VOdwd/po3yX9cfa3ZactkFZB0/R/gRjA== X-Google-Smtp-Source: ABdhPJxYjxHpYGvE1UsK7N+mxBKfr8/iFr+Lwt+g/ozLYXOMwkRO+SVrODMOPJKOj8V6Hd+K0NIIhQ== X-Received: by 2002:a63:4708:: with SMTP id u8mr1162155pga.102.1615923870318; Tue, 16 Mar 2021 12:44:30 -0700 (PDT) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:532]) by smtp.gmail.com with ESMTPSA id w22sm16919104pfi.133.2021.03.16.12.44.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Mar 2021 12:44:29 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v4 03/11] btrfs-progs: receive: support v2 send stream DATA tlv format Date: Tue, 16 Mar 2021 12:43:54 -0700 Message-Id: <4ce95af75a4596ca20fed3d5112a600616e3e985.1615922859.git.osandov@osandov.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov The new format privileges the BTRFS_SEND_A_DATA attribute by guaranteeing it will always be the last attribute in any command that needs it, and by implicitly encoding the data length as the difference between the total command length in the command header and the sizes of the rest of the attributes (and of course the tlv_type identifying the DATA attribute). To parse the new stream, we must read the tlv_type and if it is not DATA, we proceed normally, but if it is DATA, we don't parse a tlv_len but simply compute the length. In addition, we add some bounds checking when parsing each chunk of data, as well as for the tlv_len itself. Signed-off-by: Boris Burkov --- common/send-stream.c | 36 ++++++++++++++++++++++++++---------- 1 file changed, 26 insertions(+), 10 deletions(-) diff --git a/common/send-stream.c b/common/send-stream.c index 3d3585c3..4d819185 100644 --- a/common/send-stream.c +++ b/common/send-stream.c @@ -165,28 +165,44 @@ static int read_cmd(struct btrfs_send_stream *sctx) pos = 0; while (pos < cmd_len) { - struct btrfs_tlv_header *tlv_hdr; u16 tlv_type; - u16 tlv_len; struct btrfs_send_attribute *send_attr; - tlv_hdr = (struct btrfs_tlv_header *)data; - tlv_type = le16_to_cpu(tlv_hdr->tlv_type); - tlv_len = le16_to_cpu(tlv_hdr->tlv_len); + if (cmd_len - pos < sizeof(__le16)) { + error("send stream is truncated"); + ret = -EINVAL; + goto out; + } + tlv_type = le16_to_cpu(*(__le16 *)data); if (tlv_type == 0 || tlv_type > BTRFS_SEND_A_MAX) { - error("invalid tlv in cmd tlv_type = %hu, tlv_len = %hu", - tlv_type, tlv_len); + error("invalid tlv in cmd tlv_type = %hu", tlv_type); ret = -EINVAL; goto out; } send_attr = &sctx->cmd_attrs[tlv_type]; send_attr->tlv_type = tlv_type; - send_attr->tlv_len = tlv_len; - pos += sizeof(*tlv_hdr); - data += sizeof(*tlv_hdr); + pos += sizeof(tlv_type); + data += sizeof(tlv_type); + if (sctx->version == 2 && tlv_type == BTRFS_SEND_A_DATA) { + send_attr->tlv_len = cmd_len - pos; + } else { + if (cmd_len - pos < sizeof(__le16)) { + error("send stream is truncated"); + ret = -EINVAL; + goto out; + } + send_attr->tlv_len = le16_to_cpu(*(__le16 *)data); + pos += sizeof(__le16); + data += sizeof(__le16); + } + if (cmd_len - pos < send_attr->tlv_len) { + error("send stream is truncated"); + ret = -EINVAL; + goto out; + } send_attr->data = data; pos += send_attr->tlv_len; data += send_attr->tlv_len; From patchwork Tue Mar 16 19:43:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 12143733 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15A6CC4360C for ; Tue, 16 Mar 2021 19:46:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E274264F5E for ; Tue, 16 Mar 2021 19:45:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240656AbhCPTpw (ORCPT ); Tue, 16 Mar 2021 15:45:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35826 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240614AbhCPTpB (ORCPT ); Tue, 16 Mar 2021 15:45:01 -0400 Received: from mail-pj1-x102b.google.com (mail-pj1-x102b.google.com [IPv6:2607:f8b0:4864:20::102b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 87439C0613DB for ; Tue, 16 Mar 2021 12:44:37 -0700 (PDT) Received: by mail-pj1-x102b.google.com with SMTP id x7-20020a17090a2b07b02900c0ea793940so1923491pjc.2 for ; Tue, 16 Mar 2021 12:44:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6OsUA50+8xoVcD2c3qd1aIdeBQH0nioyEP52OneAmPE=; b=KWBn17b9Obf/nBjJaUYsB3u1wpQIzCkKRQwXTYjxHnPCk0lt55ESCAoj5xqGNkZvVt +CqzYX/lucXshzWS4nmcnAUwmwrzkdgzoyZ5Jgk1a4TcEsuYMSZS/cBCKn6mbjPWYTn0 +txMTFpXbhNOTjNvilIc3iWbBbzqNQ6QR4GHn4/NUnNGsVCb4aMEnF3ZXoWbA8bzwS6P jpkE4OfFr6vHy3I+WQxSKyBcrajSvfJU3WDarg6xcbc+PkOvSa/ycDRObPS3IR7OU3R6 RjLgOwE0eD/zRvyC32FsW2vI/qoRrJZveR6jr6D0/49/5wZnqYDeAc5UlreDs8ydOjdt o+gw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6OsUA50+8xoVcD2c3qd1aIdeBQH0nioyEP52OneAmPE=; b=ri/7Y0NX/aoZDZdI5QsvST8KMMi/2Q0H9vmPOoZ2QDTtJpr3eWPaIJwB0wyn4kMA1K wNT46dH0g2TcANE6Mqt03moQgcpijRfMN6lLbqdhHXOVOSLY0hvcRrrw3N2VOQfV25Ba tSRrurIdZgrQtKPU1kYAUbOMm7onGblmlz6uWT5rYr46GkIU/V9ft2E2PnjwX/tstEBW wQ28e4lXM0VTnbXxQ40Ccc8HRSSE4sFNMl9TpM3vweC331c5IpmX8fIzu77HSAGKmsWP 3Ae4z1m4q+ydEZiwcKoBILg2m6HvtuZdSVdjD3CNfXX7MQtWPasOvQPHNvz9+cfL6ktc IFGg== X-Gm-Message-State: AOAM531IIgBKuBWPcYk1jMCDiHvvt92qWvo40aejO/P3iDknONBWAM5L UQNrqrVy8Tm5qjCzlMaFeoUtIKTFqqM97A== X-Google-Smtp-Source: ABdhPJxKk0KhlNgwdEaDwDUGXPI/3mRTybXij/0/kfz1YkKBsJKqFbxT91wmEQ3EFLCvadSY6/asLA== X-Received: by 2002:a17:902:7682:b029:e6:2bc5:f005 with SMTP id m2-20020a1709027682b02900e62bc5f005mr851951pll.32.1615923876233; Tue, 16 Mar 2021 12:44:36 -0700 (PDT) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:532]) by smtp.gmail.com with ESMTPSA id w22sm16919104pfi.133.2021.03.16.12.44.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Mar 2021 12:44:35 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v4 4/5] btrfs: send: send compressed extents with encoded writes Date: Tue, 16 Mar 2021 12:43:57 -0700 Message-Id: X-Mailer: git-send-email 2.30.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Omar Sandoval Now that all of the pieces are in place, we can use the ENCODED_WRITE command to send compressed extents when appropriate. Signed-off-by: Omar Sandoval --- fs/btrfs/ctree.h | 4 + fs/btrfs/inode.c | 6 +- fs/btrfs/send.c | 230 +++++++++++++++++++++++++++++++++++++++++++---- 3 files changed, 220 insertions(+), 20 deletions(-) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h index 33a08ab5cb0e..402ffdce81ac 100644 --- a/fs/btrfs/ctree.h +++ b/fs/btrfs/ctree.h @@ -3157,6 +3157,10 @@ int btrfs_run_delalloc_range(struct btrfs_inode *inode, struct page *locked_page int btrfs_writepage_cow_fixup(struct page *page, u64 start, u64 end); void btrfs_writepage_endio_finish_ordered(struct page *page, u64 start, u64 end, int uptodate); +int encoded_iov_compression_from_btrfs(unsigned int compress_type); +int btrfs_encoded_read_regular_fill_pages(struct inode *inode, u64 offset, + u64 disk_io_size, + struct page **pages); ssize_t btrfs_encoded_read(struct kiocb *iocb, struct iov_iter *iter); ssize_t btrfs_do_encoded_write(struct kiocb *iocb, struct iov_iter *from, struct encoded_iov *encoded); diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index c2fe76f57bf5..3c1c8879a9e1 100644 --- a/fs/btrfs/inode.c +++ b/fs/btrfs/inode.c @@ -9984,7 +9984,7 @@ void btrfs_set_range_writeback(struct extent_io_tree *tree, u64 start, u64 end) } } -static int encoded_iov_compression_from_btrfs(unsigned int compress_type) +int encoded_iov_compression_from_btrfs(unsigned int compress_type) { switch (compress_type) { case BTRFS_COMPRESS_NONE: @@ -10190,8 +10190,8 @@ static void btrfs_encoded_read_endio(struct bio *bio) bio_put(bio); } -static int btrfs_encoded_read_regular_fill_pages(struct inode *inode, u64 offset, - u64 disk_io_size, struct page **pages) +int btrfs_encoded_read_regular_fill_pages(struct inode *inode, u64 offset, + u64 disk_io_size, struct page **pages) { struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb); struct btrfs_encoded_read_private priv = { diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c index 25b1a60a568c..7516eba701af 100644 --- a/fs/btrfs/send.c +++ b/fs/btrfs/send.c @@ -595,6 +595,7 @@ static int tlv_put(struct send_ctx *sctx, u16 attr, const void *data, int len) return tlv_put(sctx, attr, &__tmp, sizeof(__tmp)); \ } +TLV_PUT_DEFINE_INT(32) TLV_PUT_DEFINE_INT(64) static int tlv_put_string(struct send_ctx *sctx, u16 attr, @@ -5213,16 +5214,211 @@ static int send_hole(struct send_ctx *sctx, u64 end) return ret; } -static int send_extent_data(struct send_ctx *sctx, - const u64 offset, - const u64 len) +static int send_encoded_inline_extent(struct send_ctx *sctx, + struct btrfs_path *path, u64 offset, + u64 len) { + struct btrfs_root *root = sctx->send_root; + struct btrfs_fs_info *fs_info = root->fs_info; + struct inode *inode; + struct fs_path *p; + struct extent_buffer *leaf = path->nodes[0]; + struct btrfs_key key; + struct btrfs_file_extent_item *ei; + u64 ram_bytes; + size_t inline_size; + int ret; + + inode = btrfs_iget(fs_info->sb, sctx->cur_ino, root); + if (IS_ERR(inode)) + return PTR_ERR(inode); + + p = fs_path_alloc(); + if (!p) { + ret = -ENOMEM; + goto out; + } + + ret = begin_cmd(sctx, BTRFS_SEND_C_ENCODED_WRITE); + if (ret < 0) + goto out; + + ret = get_cur_path(sctx, sctx->cur_ino, sctx->cur_inode_gen, p); + if (ret < 0) + goto out; + + btrfs_item_key_to_cpu(leaf, &key, path->slots[0]); + ei = btrfs_item_ptr(leaf, path->slots[0], + struct btrfs_file_extent_item); + ram_bytes = btrfs_file_extent_ram_bytes(leaf, ei); + inline_size = btrfs_file_extent_inline_item_len(leaf, + btrfs_item_nr(path->slots[0])); + + TLV_PUT_PATH(sctx, BTRFS_SEND_A_PATH, p); + TLV_PUT_U64(sctx, BTRFS_SEND_A_FILE_OFFSET, offset); + TLV_PUT_U64(sctx, BTRFS_SEND_A_UNENCODED_FILE_LEN, + min(key.offset + ram_bytes - offset, len)); + TLV_PUT_U64(sctx, BTRFS_SEND_A_UNENCODED_LEN, ram_bytes); + TLV_PUT_U64(sctx, BTRFS_SEND_A_UNENCODED_OFFSET, offset - key.offset); + ret = encoded_iov_compression_from_btrfs( + btrfs_file_extent_compression(leaf, ei)); + if (ret < 0) + goto out; + TLV_PUT_U32(sctx, BTRFS_SEND_A_COMPRESSION, ret); + TLV_PUT_U32(sctx, BTRFS_SEND_A_ENCRYPTION, 0); + + ret = put_data_header(sctx, inline_size); + if (ret < 0) + goto out; + read_extent_buffer(leaf, sctx->send_buf + sctx->send_size, + btrfs_file_extent_inline_start(ei), inline_size); + sctx->send_size += inline_size; + + ret = send_cmd(sctx); + +tlv_put_failure: +out: + fs_path_free(p); + iput(inode); + return ret; +} + +static int send_encoded_extent(struct send_ctx *sctx, struct btrfs_path *path, + u64 offset, u64 len) +{ + struct btrfs_root *root = sctx->send_root; + struct btrfs_fs_info *fs_info = root->fs_info; + struct inode *inode; + struct fs_path *p; + struct extent_buffer *leaf = path->nodes[0]; + struct btrfs_key key; + struct btrfs_file_extent_item *ei; + u64 block_start; + u64 block_len; + u32 data_offset; + struct btrfs_cmd_header *hdr; + u32 crc; + int ret; + + inode = btrfs_iget(fs_info->sb, sctx->cur_ino, root); + if (IS_ERR(inode)) + return PTR_ERR(inode); + + p = fs_path_alloc(); + if (!p) { + ret = -ENOMEM; + goto out; + } + + ret = begin_cmd(sctx, BTRFS_SEND_C_ENCODED_WRITE); + if (ret < 0) + goto out; + + ret = get_cur_path(sctx, sctx->cur_ino, sctx->cur_inode_gen, p); + if (ret < 0) + goto out; + + btrfs_item_key_to_cpu(leaf, &key, path->slots[0]); + ei = btrfs_item_ptr(leaf, path->slots[0], + struct btrfs_file_extent_item); + block_start = btrfs_file_extent_disk_bytenr(leaf, ei); + block_len = btrfs_file_extent_disk_num_bytes(leaf, ei); + + TLV_PUT_PATH(sctx, BTRFS_SEND_A_PATH, p); + TLV_PUT_U64(sctx, BTRFS_SEND_A_FILE_OFFSET, offset); + TLV_PUT_U64(sctx, BTRFS_SEND_A_UNENCODED_FILE_LEN, + min(key.offset + btrfs_file_extent_num_bytes(leaf, ei) - offset, + len)); + TLV_PUT_U64(sctx, BTRFS_SEND_A_UNENCODED_LEN, + btrfs_file_extent_ram_bytes(leaf, ei)); + TLV_PUT_U64(sctx, BTRFS_SEND_A_UNENCODED_OFFSET, + offset - key.offset + btrfs_file_extent_offset(leaf, ei)); + ret = encoded_iov_compression_from_btrfs( + btrfs_file_extent_compression(leaf, ei)); + if (ret < 0) + goto out; + TLV_PUT_U32(sctx, BTRFS_SEND_A_COMPRESSION, ret); + TLV_PUT_U32(sctx, BTRFS_SEND_A_ENCRYPTION, 0); + + ret = put_data_header(sctx, block_len); + if (ret < 0) + goto out; + + data_offset = ALIGN(sctx->send_size, PAGE_SIZE); + if (data_offset > sctx->send_max_size || + sctx->send_max_size - data_offset < block_len) { + ret = -EOVERFLOW; + goto out; + } + + ret = btrfs_encoded_read_regular_fill_pages(inode, block_start, + block_len, + sctx->send_buf_pages + + (data_offset >> PAGE_SHIFT)); + if (ret) + goto out; + + hdr = (struct btrfs_cmd_header *)sctx->send_buf; + hdr->len = cpu_to_le32(sctx->send_size + block_len - sizeof(*hdr)); + hdr->crc = 0; + crc = btrfs_crc32c(0, sctx->send_buf, sctx->send_size); + crc = btrfs_crc32c(crc, sctx->send_buf + data_offset, block_len); + hdr->crc = cpu_to_le32(crc); + + ret = write_buf(sctx->send_filp, sctx->send_buf, sctx->send_size, + &sctx->send_off); + if (!ret) { + ret = write_buf(sctx->send_filp, sctx->send_buf + data_offset, + block_len, &sctx->send_off); + } + sctx->total_send_size += sctx->send_size + block_len; + sctx->cmd_send_size[le16_to_cpu(hdr->cmd)] += + sctx->send_size + block_len; + sctx->send_size = 0; + +tlv_put_failure: +out: + fs_path_free(p); + iput(inode); + return ret; +} + +static int send_extent_data(struct send_ctx *sctx, struct btrfs_path *path, + const u64 offset, const u64 len) +{ + struct extent_buffer *leaf = path->nodes[0]; + struct btrfs_file_extent_item *ei; u64 read_size = max_send_read_size(sctx); u64 sent = 0; if (sctx->flags & BTRFS_SEND_FLAG_NO_FILE_DATA) return send_update_extent(sctx, offset, len); + ei = btrfs_item_ptr(leaf, path->slots[0], + struct btrfs_file_extent_item); + if ((sctx->flags & BTRFS_SEND_FLAG_COMPRESSED) && + btrfs_file_extent_compression(leaf, ei) != BTRFS_COMPRESS_NONE) { + bool is_inline = (btrfs_file_extent_type(leaf, ei) == + BTRFS_FILE_EXTENT_INLINE); + + /* + * Send the compressed extent unless the compressed data is + * larger than the decompressed data. This can happen if we're + * not sending the entire extent, either because it has been + * partially overwritten/truncated or because this is a part of + * the extent that we couldn't clone in clone_range(). + */ + if (is_inline && + btrfs_file_extent_inline_item_len(leaf, + btrfs_item_nr(path->slots[0])) <= len) { + return send_encoded_inline_extent(sctx, path, offset, + len); + } else if (!is_inline && + btrfs_file_extent_disk_num_bytes(leaf, ei) <= len) { + return send_encoded_extent(sctx, path, offset, len); + } + } + while (sent < len) { u64 size = min(len - sent, read_size); int ret; @@ -5293,12 +5489,9 @@ static int send_capabilities(struct send_ctx *sctx) return ret; } -static int clone_range(struct send_ctx *sctx, - struct clone_root *clone_root, - const u64 disk_byte, - u64 data_offset, - u64 offset, - u64 len) +static int clone_range(struct send_ctx *sctx, struct btrfs_path *dst_path, + struct clone_root *clone_root, const u64 disk_byte, + u64 data_offset, u64 offset, u64 len) { struct btrfs_path *path; struct btrfs_key key; @@ -5322,7 +5515,7 @@ static int clone_range(struct send_ctx *sctx, */ if (clone_root->offset == 0 && len == sctx->send_root->fs_info->sectorsize) - return send_extent_data(sctx, offset, len); + return send_extent_data(sctx, dst_path, offset, len); path = alloc_path_for_send(); if (!path) @@ -5419,7 +5612,8 @@ static int clone_range(struct send_ctx *sctx, if (hole_len > len) hole_len = len; - ret = send_extent_data(sctx, offset, hole_len); + ret = send_extent_data(sctx, dst_path, offset, + hole_len); if (ret < 0) goto out; @@ -5492,14 +5686,16 @@ static int clone_range(struct send_ctx *sctx, if (ret < 0) goto out; } - ret = send_extent_data(sctx, offset + slen, + ret = send_extent_data(sctx, dst_path, + offset + slen, clone_len - slen); } else { ret = send_clone(sctx, offset, clone_len, clone_root); } } else { - ret = send_extent_data(sctx, offset, clone_len); + ret = send_extent_data(sctx, dst_path, offset, + clone_len); } if (ret < 0) @@ -5531,7 +5727,7 @@ static int clone_range(struct send_ctx *sctx, } if (len > 0) - ret = send_extent_data(sctx, offset, len); + ret = send_extent_data(sctx, dst_path, offset, len); else ret = 0; out: @@ -5562,10 +5758,10 @@ static int send_write_or_clone(struct send_ctx *sctx, struct btrfs_file_extent_item); disk_byte = btrfs_file_extent_disk_bytenr(path->nodes[0], ei); data_offset = btrfs_file_extent_offset(path->nodes[0], ei); - ret = clone_range(sctx, clone_root, disk_byte, data_offset, - offset, end - offset); + ret = clone_range(sctx, path, clone_root, disk_byte, + data_offset, offset, end - offset); } else { - ret = send_extent_data(sctx, offset, end - offset); + ret = send_extent_data(sctx, path, offset, end - offset); } sctx->cur_inode_next_write_offset = end; return ret; From patchwork Tue Mar 16 19:43:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 12143737 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B81BBC433E9 for ; Tue, 16 Mar 2021 19:46:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9743064F52 for ; Tue, 16 Mar 2021 19:46:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240666AbhCPTqA (ORCPT ); Tue, 16 Mar 2021 15:46:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35708 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240618AbhCPTpD (ORCPT ); Tue, 16 Mar 2021 15:45:03 -0400 Received: from mail-pg1-x52c.google.com (mail-pg1-x52c.google.com [IPv6:2607:f8b0:4864:20::52c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6E15FC0613DF for ; Tue, 16 Mar 2021 12:44:41 -0700 (PDT) Received: by mail-pg1-x52c.google.com with SMTP id q5so10654514pgk.5 for ; Tue, 16 Mar 2021 12:44:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+3VnBuWPqboPwPuwDX4KEFwl4968vOutKRgPYHRDSQo=; b=Ynuu7XhXLzvhhyPyPgN7sje6eAPgaLgmFeSzcp3IPReFzV6GyodYGq+oocJJaZKtTR NqxUpwcvw+k9uIKfxE2H3id4iBaH2Mf7NL/hZkNUY1mImfiFnWtx0458NMCzC+/MrZ7P tKI8Z+KjMPj5QKDX8Vqf0PCDWk+d12xySgtfyKaqN//TgmG+Ptg8ulVK3SOxLgj8FBeM ETRiBXeqEUCgZbZQaIkFabjCAwsPsvl19GU1tAtjgfNv4ZqQNJJUGx3zaWwmoGy4Mv0g 6LHRqmtBNqrt5Nc4JOOco3/Tjzc9U5urYC1vqRlkGjYbN7kmYPz6aLcReiplxPPlZoQ7 4iCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+3VnBuWPqboPwPuwDX4KEFwl4968vOutKRgPYHRDSQo=; b=PddK0kUAxQeVTGTIQH3N0+ybktEnMNAy3M1qiKr9kUmdkpc6zqc/IIYa6NgblNnVJO jJ+thLHJXuhgWe3IV5vzV8fU7E2es956VbdwfGLUcOhYN9ujANSvepmiqE907bno7ezV fEbeY3cB8DlfjzBkk3udvhW8mJyUIvZ9DNigNWPUjEDZT6lDS55sGSq9xydEoMbllYkb m2yUyNrj51ooewErpS+DrdpCyDrBC+h+uj46NDJN6h4lIHPFyy42G9CDroYf/9UxL6b1 6fDHsYcFCF865gHeXikg+zXQQDhTUT5x0T/+4ENgFxU7JtljI0FF4xaewfJONvqiB2yd eFjA== X-Gm-Message-State: AOAM5331705fX19ORpYESolEgWLuOMsNHZxlkgaRKZDgewz5LtNuYpVB Q16bjkCEY1aURUdYdZFa86RCynPRX2G45A== X-Google-Smtp-Source: ABdhPJyTFq14PGme72u+qlyTvCUxFlVHt1qDRWBRvYM3yKTfFOldqIrj0sTqY1YfJdA+aZlT4NG55Q== X-Received: by 2002:a62:e10f:0:b029:1f5:42b6:7113 with SMTP id q15-20020a62e10f0000b02901f542b67113mr917750pfh.63.1615923880456; Tue, 16 Mar 2021 12:44:40 -0700 (PDT) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:532]) by smtp.gmail.com with ESMTPSA id w22sm16919104pfi.133.2021.03.16.12.44.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Mar 2021 12:44:39 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v4 5/5] btrfs: send: enable support for stream v2 and compressed writes Date: Tue, 16 Mar 2021 12:43:59 -0700 Message-Id: X-Mailer: git-send-email 2.30.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Omar Sandoval Now that the new support is implemented, allow the ioctl to accept the flags and update the version in sysfs. Signed-off-by: Omar Sandoval --- fs/btrfs/send.c | 10 +++++++++- fs/btrfs/send.h | 2 +- include/uapi/linux/btrfs.h | 4 +++- 3 files changed, 13 insertions(+), 3 deletions(-) diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c index 7516eba701af..cb824d1271fa 100644 --- a/fs/btrfs/send.c +++ b/fs/btrfs/send.c @@ -671,7 +671,10 @@ static int send_header(struct send_ctx *sctx) struct btrfs_stream_header hdr; strcpy(hdr.magic, BTRFS_SEND_STREAM_MAGIC); - hdr.version = cpu_to_le32(BTRFS_SEND_STREAM_VERSION); + if (sctx->flags & BTRFS_SEND_FLAG_STREAM_V2) + hdr.version = cpu_to_le32(2); + else + hdr.version = cpu_to_le32(1); return write_buf(sctx->send_filp, &hdr, sizeof(hdr), &sctx->send_off); @@ -7446,6 +7449,11 @@ long btrfs_ioctl_send(struct file *mnt_file, struct btrfs_ioctl_send_args *arg) ret = -EINVAL; goto out; } + if ((arg->flags & BTRFS_SEND_FLAG_COMPRESSED) && + !(arg->flags & BTRFS_SEND_FLAG_STREAM_V2)) { + ret = -EINVAL; + goto out; + } sctx = kzalloc(sizeof(struct send_ctx), GFP_KERNEL); if (!sctx) { diff --git a/fs/btrfs/send.h b/fs/btrfs/send.h index 9f4f7b96b1eb..9c83e14a43b2 100644 --- a/fs/btrfs/send.h +++ b/fs/btrfs/send.h @@ -10,7 +10,7 @@ #include "ctree.h" #define BTRFS_SEND_STREAM_MAGIC "btrfs-stream" -#define BTRFS_SEND_STREAM_VERSION 1 +#define BTRFS_SEND_STREAM_VERSION 2 /* * In send stream v1, no command is larger than 64k. In send stream v2, no limit diff --git a/include/uapi/linux/btrfs.h b/include/uapi/linux/btrfs.h index 93aa0932234e..b12a9a1a106c 100644 --- a/include/uapi/linux/btrfs.h +++ b/include/uapi/linux/btrfs.h @@ -786,7 +786,9 @@ struct btrfs_ioctl_received_subvol_args { #define BTRFS_SEND_FLAG_MASK \ (BTRFS_SEND_FLAG_NO_FILE_DATA | \ BTRFS_SEND_FLAG_OMIT_STREAM_HEADER | \ - BTRFS_SEND_FLAG_OMIT_END_CMD) + BTRFS_SEND_FLAG_OMIT_END_CMD | \ + BTRFS_SEND_FLAG_STREAM_V2 | \ + BTRFS_SEND_FLAG_COMPRESSED) struct btrfs_ioctl_send_args { __s64 send_fd; /* in */ From patchwork Tue Mar 16 19:44:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 12143743 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61F1FC43333 for ; Tue, 16 Mar 2021 19:46:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3DCF464E33 for ; Tue, 16 Mar 2021 19:46:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240678AbhCPTqV (ORCPT ); Tue, 16 Mar 2021 15:46:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35716 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240622AbhCPTpF (ORCPT ); Tue, 16 Mar 2021 15:45:05 -0400 Received: from mail-pg1-x52b.google.com (mail-pg1-x52b.google.com [IPv6:2607:f8b0:4864:20::52b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9EB9AC0613E1 for ; Tue, 16 Mar 2021 12:44:43 -0700 (PDT) Received: by mail-pg1-x52b.google.com with SMTP id l2so23323818pgb.1 for ; Tue, 16 Mar 2021 12:44:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=X1awuXo2o21+U9RKljM1ZM6ezbmkIQlYdJCeBAFiTbI=; b=nAgyoscW/bSw2aT9vdLmiwiUPeHJuTL/YHiU93JzvVS5E6KLG8h88TOqeCCaM9ybzE R7jp5Sw3mf3d/c8kDiDP7U7EEyJsmnfbrQOGXIUvN4EEfRKWBUenQ/+iMK4ktLx8aKHc f4xPpEtYGuXbaZ5V8u1eepSvnN0oRSLrdezjlba1FK+Nj1xRFWinw5gH4f+SyRa+9PmI A2k4jNj+fqtmnPvE4ddFw9AyDkyvcYESdafMRhfYFZipu3dI5hW5eRQWNARSTmCCKwCA sXvyFT7b0FTFQ5TxiXDbZSmRWzSiKKf9UWF6ERTepUBjGR0k67wO/Itq9V3t9JZj0raZ 866w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=X1awuXo2o21+U9RKljM1ZM6ezbmkIQlYdJCeBAFiTbI=; b=IszY9Uh+UT3TzC4ftTseFCqaHVWsa/Vh7RuzOWjE3Qjf5g0Daae9E84mIqSKfqtWJo T4/CGLgGZG9fapSw1g+zWfS2Zth1356CL9Gnai/Rd8Brqv3MEJS+Z3H0zW0qrffRZHE8 EypkHOVkrEoWxvpfekqteQJqDuLpfykF5P6ZOHIGGbiFjzGgaxvrBNe/fIcaEaozy3bL 5zhe0IAZBLzAyw4UYp8jAVpdAPHEfeynA8T/P7C/oi+Hqbj0acBIDAAm+ILfQSG3H7a6 26ocL+CUQPdivVN9x1UgfVzNhxJnYbgknd43JLhPLEo7WbpmAsK0SrAR/IewyO5kcihy WjbA== X-Gm-Message-State: AOAM531VspU8sEaqPd8BQstU0nIapOC+Lymok3WkDTAhw8fUS/Trnld/ v2L2Rl2lHDOLvH02yt+sWP5FOzC04V/FHA== X-Google-Smtp-Source: ABdhPJw42dgJ10zSKLwz1DaUShq4KEvpZ8N/vo6GYx60QkZ0yxgBeBqiR+VAtsXYs2lEXDtzsozudw== X-Received: by 2002:aa7:9f52:0:b029:1ee:db83:dac6 with SMTP id h18-20020aa79f520000b02901eedb83dac6mr871586pfr.45.1615923882245; Tue, 16 Mar 2021 12:44:42 -0700 (PDT) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:532]) by smtp.gmail.com with ESMTPSA id w22sm16919104pfi.133.2021.03.16.12.44.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Mar 2021 12:44:41 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v4 06/11] btrfs-progs: receive: process encoded_write commands Date: Tue, 16 Mar 2021 12:44:00 -0700 Message-Id: <9815618fabedbcc9d980f9e7ff3bfe0c0a258cf3.1615922859.git.osandov@osandov.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov Add a new btrfs_send_op and support for both dumping and proper receive processing which does actual encoded writes. Encoded writes are only allowed on a file descriptor opened with an extra flag that allows encoded writes, so we also add support for this flag when opening or reusing a file for writing. Signed-off-by: Boris Burkov --- cmds/receive-dump.c | 16 ++++++- cmds/receive.c | 104 ++++++++++++++++++++++++++++++++++++++++--- common/send-stream.c | 22 +++++++++ common/send-stream.h | 4 ++ stubs.h | 46 +++++++++++++++++++ 5 files changed, 186 insertions(+), 6 deletions(-) diff --git a/cmds/receive-dump.c b/cmds/receive-dump.c index 648d9314..20ec2b70 100644 --- a/cmds/receive-dump.c +++ b/cmds/receive-dump.c @@ -316,6 +316,19 @@ static int print_update_extent(const char *path, u64 offset, u64 len, offset, len); } +static int print_encoded_write(const char *path, const void *data, u64 offset, + u64 len, u64 unencoded_file_len, + u64 unencoded_len, u64 unencoded_offset, + u32 compression, u32 encryption, void *user) +{ + return PRINT_DUMP(user, path, "encoded_write", + "offset=%llu len=%llu, unencoded_file_len=%llu, " + "unencoded_len=%llu, unencoded_offset=%llu, " + "compression=%u, encryption=%u", + offset, len, unencoded_file_len, unencoded_len, + unencoded_offset, compression, encryption); +} + struct btrfs_send_ops btrfs_print_send_ops = { .subvol = print_subvol, .snapshot = print_snapshot, @@ -337,5 +350,6 @@ struct btrfs_send_ops btrfs_print_send_ops = { .chmod = print_chmod, .chown = print_chown, .utimes = print_utimes, - .update_extent = print_update_extent + .update_extent = print_update_extent, + .encoded_write = print_encoded_write, }; diff --git a/cmds/receive.c b/cmds/receive.c index b4099bc4..ad1e3021 100644 --- a/cmds/receive.c +++ b/cmds/receive.c @@ -30,12 +30,14 @@ #include #include #include +#include #include #include #include #include #include +#include #include #include @@ -52,6 +54,7 @@ #include "cmds/receive-dump.h" #include "common/help.h" #include "common/path-utils.h" +#include "stubs.h" struct btrfs_receive { @@ -60,6 +63,7 @@ struct btrfs_receive int write_fd; char write_path[PATH_MAX]; + int write_fd_allow_encoded; char *root_path; char *dest_dir_path; /* relative to root_path */ @@ -635,24 +639,65 @@ out: return ret; } -static int open_inode_for_write(struct btrfs_receive *rctx, const char *path) +static int set_write_fd_allow_encoded(struct btrfs_receive *rctx) +{ + int ret; + int flags; + + flags = fcntl(rctx->write_fd, F_GETFL); + if (flags < 0) { + ret = -errno; + error("failed to fetch old fd flags"); + goto close_fd; + } + ret = fcntl(rctx->write_fd, F_SETFL, flags | O_ALLOW_ENCODED); + if (ret < 0) { + ret = -errno; + error("failed to enable encoded writes"); + goto close_fd; + } + rctx->write_fd_allow_encoded = true; + ret = 0; + goto out; +close_fd: + close(rctx->write_fd); + rctx->write_fd = -1; + rctx->write_fd_allow_encoded = false; +out: + return ret; +} + +static int open_inode_for_write(struct btrfs_receive *rctx, const char *path, + bool allow_encoded) { int ret = 0; + int flags = O_RDWR; if (rctx->write_fd != -1) { - if (strcmp(rctx->write_path, path) == 0) + /* + * if the existing fd is for this path and the needed flags are + * satisfied, no need to open a new one + */ + if (strcmp(rctx->write_path, path) == 0) { + /* fixup the allow encoded flag, if necessary */ + if (allow_encoded && !rctx->write_fd_allow_encoded) + ret = set_write_fd_allow_encoded(rctx); goto out; + } close(rctx->write_fd); rctx->write_fd = -1; } - rctx->write_fd = open(path, O_RDWR); + if (allow_encoded) + flags |= O_ALLOW_ENCODED; + rctx->write_fd = open(path, flags); if (rctx->write_fd < 0) { ret = -errno; error("cannot open %s: %m", path); goto out; } strncpy_null(rctx->write_path, path); + rctx->write_fd_allow_encoded = allow_encoded; out: return ret; @@ -683,7 +728,7 @@ static int process_write(const char *path, const void *data, u64 offset, goto out; } - ret = open_inode_for_write(rctx, full_path); + ret = open_inode_for_write(rctx, full_path, false); if (ret < 0) goto out; @@ -726,7 +771,7 @@ static int process_clone(const char *path, u64 offset, u64 len, goto out; } - ret = open_inode_for_write(rctx, full_path); + ret = open_inode_for_write(rctx, full_path, false); if (ret < 0) goto out; @@ -987,6 +1032,54 @@ static int process_update_extent(const char *path, u64 offset, u64 len, return 0; } +static int process_encoded_write(const char *path, const void *data, u64 offset, + u64 len, u64 unencoded_file_len, u64 unencoded_len, + u64 unencoded_offset, u32 compression, u32 encryption, void *user) +{ + int ret; + ssize_t w; + struct btrfs_receive *rctx = user; + char full_path[PATH_MAX]; + struct encoded_iov encoded = { + .len = unencoded_file_len, + .unencoded_len = unencoded_len, + .unencoded_offset = unencoded_offset, + .compression = compression, + .encryption = encryption, + }; + struct iovec iov[2] = { + { &encoded, sizeof(encoded) }, + { (char *)data, len } + }; + + if (encryption) { + error("encoded_write: encryption not supported"); + return -EOPNOTSUPP; + } + + ret = path_cat_out(full_path, rctx->full_subvol_path, path); + if (ret < 0) { + error("encoded_write: path invalid: %s", path); + return ret; + } + + ret = open_inode_for_write(rctx, full_path, true); + if (ret < 0) + return ret; + + /* + * NOTE: encoded writes guarantee no partial writes, so we don't need to + * handle that possibility. + */ + w = pwritev2(rctx->write_fd, iov, 2, offset, RWF_ENCODED); + if (w < 0) { + ret = -errno; + error("encoded_write: writing to %s failed: %m", path); + return ret; + } + return 0; +} + static struct btrfs_send_ops send_ops = { .subvol = process_subvol, .snapshot = process_snapshot, @@ -1009,6 +1102,7 @@ static struct btrfs_send_ops send_ops = { .chown = process_chown, .utimes = process_utimes, .update_extent = process_update_extent, + .encoded_write = process_encoded_write, }; static int do_receive(struct btrfs_receive *rctx, const char *tomnt, diff --git a/common/send-stream.c b/common/send-stream.c index 4d819185..044e101b 100644 --- a/common/send-stream.c +++ b/common/send-stream.c @@ -354,6 +354,8 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx) struct timespec mt; u8 uuid[BTRFS_UUID_SIZE]; u8 clone_uuid[BTRFS_UUID_SIZE]; + u32 compression; + u32 encryption; u64 tmp; u64 tmp2; u64 ctransid; @@ -362,6 +364,9 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx) u64 dev; u64 clone_offset; u64 offset; + u64 unencoded_file_len; + u64 unencoded_len; + u64 unencoded_offset; int len; int xattr_len; @@ -436,6 +441,23 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx) TLV_GET(sctx, BTRFS_SEND_A_DATA, &data, &len); ret = sctx->ops->write(path, data, offset, len, sctx->user); break; + case BTRFS_SEND_C_ENCODED_WRITE: + TLV_GET_STRING(sctx, BTRFS_SEND_A_PATH, &path); + TLV_GET_U64(sctx, BTRFS_SEND_A_FILE_OFFSET, &offset); + TLV_GET_U64(sctx, BTRFS_SEND_A_UNENCODED_FILE_LEN, + &unencoded_file_len); + TLV_GET_U64(sctx, BTRFS_SEND_A_UNENCODED_LEN, &unencoded_len); + TLV_GET_U64(sctx, BTRFS_SEND_A_UNENCODED_OFFSET, + &unencoded_offset); + TLV_GET_U32(sctx, BTRFS_SEND_A_COMPRESSION, &compression); + TLV_GET_U32(sctx, BTRFS_SEND_A_ENCRYPTION, &encryption); + TLV_GET(sctx, BTRFS_SEND_A_DATA, &data, &len); + ret = sctx->ops->encoded_write(path, data, offset, len, + unencoded_file_len, + unencoded_len, unencoded_offset, + compression, encryption, + sctx->user); + break; case BTRFS_SEND_C_CLONE: TLV_GET_STRING(sctx, BTRFS_SEND_A_PATH, &path); TLV_GET_U64(sctx, BTRFS_SEND_A_FILE_OFFSET, &offset); diff --git a/common/send-stream.h b/common/send-stream.h index 39901f86..607bc007 100644 --- a/common/send-stream.h +++ b/common/send-stream.h @@ -66,6 +66,10 @@ struct btrfs_send_ops { struct timespec *mt, struct timespec *ct, void *user); int (*update_extent)(const char *path, u64 offset, u64 len, void *user); + int (*encoded_write)(const char *path, const void *data, u64 offset, + u64 len, u64 unencoded_file_len, u64 unencoded_len, + u64 unencoded_offset, u32 compression, + u32 encryption, void *user); }; int btrfs_read_and_process_send_stream(int fd, diff --git a/stubs.h b/stubs.h index b39f8a69..69e7fe23 100644 --- a/stubs.h +++ b/stubs.h @@ -1,6 +1,8 @@ #ifndef _BTRFS_STUBS_H #define _BTRFS_STUBS_H +#include +#include #include struct iovec; @@ -8,4 +10,48 @@ struct iovec; ssize_t pwritev2(int fd, const struct iovec *iov, int iovcnt, off_t offset, int flags); +#ifndef O_ALLOW_ENCODED +#if defined(__alpha__) +#define O_ALLOW_ENCODED 0200000000 +#elif defined(__hppa__) +#define O_ALLOW_ENCODED 100000000 +#elif defined(__sparc__) +#define O_ALLOW_ENCODED 0x8000000 +#else +#define O_ALLOW_ENCODED 040000000 #endif +#endif + +#ifndef ENCODED_IOV_SIZE_VER0 + +#define ENCODED_IOV_COMPRESSION_NONE 0 +#define ENCODED_IOV_COMPRESSION_BTRFS_ZLIB 1 +#define ENCODED_IOV_COMPRESSION_BTRFS_ZSTD 2 +#define ENCODED_IOV_COMPRESSION_BTRFS_LZO_4K 3 +#define ENCODED_IOV_COMPRESSION_BTRFS_LZO_8K 4 +#define ENCODED_IOV_COMPRESSION_BTRFS_LZO_16K 5 +#define ENCODED_IOV_COMPRESSION_BTRFS_LZO_32K 6 +#define ENCODED_IOV_COMPRESSION_BTRFS_LZO_64K 7 +#define ENCODED_IOV_COMPRESSION_TYPES 8 + +#define ENCODED_IOV_ENCRYPTION_NONE 0 +#define ENCODED_IOV_ENCRYPTION_TYPES 1 + +struct encoded_iov { + __aligned_u64 len; + __aligned_u64 unencoded_len; + __aligned_u64 unencoded_offset; + __u32 compression; + __u32 encryption; +}; + +#define ENCODED_IOV_SIZE_VER0 32 + +#endif /* ENCODED_IOV_SIZE_VER0 */ + +#ifndef RWF_ENCODED +/* encoded (e.g., compressed and/or encrypted) IO */ +#define RWF_ENCODED ((__kernel_rwf_t)0x00000020) +#endif + +#endif /* _BTRFS_STUBS_H */ From patchwork Tue Mar 16 19:44:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 12143739 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05C4EC43331 for ; Tue, 16 Mar 2021 19:46:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E713864F91 for ; Tue, 16 Mar 2021 19:46:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240623AbhCPTqN (ORCPT ); Tue, 16 Mar 2021 15:46:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35848 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240530AbhCPTpF (ORCPT ); Tue, 16 Mar 2021 15:45:05 -0400 Received: from mail-pf1-x42c.google.com (mail-pf1-x42c.google.com [IPv6:2607:f8b0:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 667E9C0613E3 for ; Tue, 16 Mar 2021 12:44:45 -0700 (PDT) Received: by mail-pf1-x42c.google.com with SMTP id x21so8239893pfa.3 for ; Tue, 16 Mar 2021 12:44:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=71QLsbBKRN4JEnDqyl9cH6L4uu++i8y8AUFQmN5/hC0=; b=Xb1euGmo2k8oGv+R/bRYu3VMTz7/QTIBls+XEcOGcw+QvA1qxV5abQ+FZzz5KnVTtL eRZAYmUilH8fhFZhVZfV0dp0+Us9YRlpoq3xz0l0W8j9QKKSegEa9YmYAT/AC+G7zyXc OP2KS3kWzxY6gG7yDFqjoW+uqUCDTCjxc6/sTHUSbtmqL0rVG+TCHDUM+UUQAWrgOr+U cF8/y8lNPCy9xDEivR1IU9f7y5mTZcwfJ2WmDgZAKoBse8eFHppwUSDI5zWzrJ8pu4Fr fhqjPgHMG7Jne2bC02TktNgBQ/oD4duf5hA6I4Rt9R6tjmkRpe3ojtSsaB3DXMkkKSwB B4/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=71QLsbBKRN4JEnDqyl9cH6L4uu++i8y8AUFQmN5/hC0=; b=NgCiplbLIfbER0RTcqGfA4spuTFFI0Y2WuO2PfHYSJK2JZY9udAwF9tdh+fpbpmFvq F/nEPsBO7veloVr28vhmE/ZM0fm0lj3AyPIluefcnPmjj9ANrdH4imsJowqYFFGL703k qJTqqN1rcS/jfHsUxuTlmcKNsQdT00psUBuwT/8wWM4mD6FUcciZY57Xv7xpXDi9iQSr C1IwyhT7rDhqGII2FfM2WPFQNl/WWEwreBX1OfZ6q7OubaL0x3xmjacXISA5nRH+4rSh MfpsFKylUjYlmfV8VcCkucxcXqjy/Fc/kJMHZ3e4zHLzSuLuAXHrmcKOOyigsJB0p04F RR1A== X-Gm-Message-State: AOAM530S7ZLA71O2G/4jOXIys2AICRXwgRGPgHDB7/4dWRXF9KxEuJTi xCzYjQ0zJU5ls7UcXNEtgwTGG8PTLO39Fg== X-Google-Smtp-Source: ABdhPJzLuSyiuw8SSaPCdcVGN1aojjJKp5mvQJCTBQPmv9X/1TzYIma5KKqiVtUrr0JfhBZK2J+yaw== X-Received: by 2002:aa7:9d1c:0:b029:1f1:53b2:a5b0 with SMTP id k28-20020aa79d1c0000b02901f153b2a5b0mr1008634pfp.13.1615923884196; Tue, 16 Mar 2021 12:44:44 -0700 (PDT) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:532]) by smtp.gmail.com with ESMTPSA id w22sm16919104pfi.133.2021.03.16.12.44.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Mar 2021 12:44:43 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v4 07/11] btrfs-progs: receive: encoded_write fallback to explicit decode and write Date: Tue, 16 Mar 2021 12:44:01 -0700 Message-Id: <12bad5c88e48f6e23bfb3cc76fb6a876468e8705.1615922859.git.osandov@osandov.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov An encoded_write can fail if the file system it is being applied to does not support encoded writes or if it can't find enough contiguous space to accommodate the encoded extent. In those cases, we can likely still process an encoded_write by explicitly decoding the data and doing a normal write. Add the necessary fallback path for decoding data compressed with zlib, lzo, or zstd. zlib and zstd have reusable decoding context data structures which we cache in the receive context so that we don't have to recreate them on every encoded_write. Finally, add a command line flag for force-decompress which causes receive to always use the fallback path rather than first attempting the encoded write. Signed-off-by: Boris Burkov --- Documentation/btrfs-receive.asciidoc | 4 + cmds/receive.c | 274 +++++++++++++++++++++++++-- 2 files changed, 265 insertions(+), 13 deletions(-) diff --git a/Documentation/btrfs-receive.asciidoc b/Documentation/btrfs-receive.asciidoc index e4c4d2c0..354a71dc 100644 --- a/Documentation/btrfs-receive.asciidoc +++ b/Documentation/btrfs-receive.asciidoc @@ -60,6 +60,10 @@ By default the mountpoint is searched in '/proc/self/mounts'. If '/proc' is not accessible, eg. in a chroot environment, use this option to tell us where this filesystem is mounted. +--force-decompress:: +if the stream contains compressed data (see '--compressed-data' in +`btrfs-send`(8)), always decompress it instead of writing it with encoded I/O. + --dump:: dump the stream metadata, one line per operation + diff --git a/cmds/receive.c b/cmds/receive.c index ad1e3021..f0236f50 100644 --- a/cmds/receive.c +++ b/cmds/receive.c @@ -41,6 +41,10 @@ #include #include +#include +#include +#include + #include "kernel-shared/ctree.h" #include "ioctl.h" #include "cmds/commands.h" @@ -81,6 +85,12 @@ struct btrfs_receive struct subvol_uuid_search sus; int honor_end_cmd; + + int force_decompress; + + /* Reuse stream objects for encoded_write decompression fallback */ + ZSTD_DStream *zstd_dstream; + z_stream *zlib_stream; }; static int finish_subvol(struct btrfs_receive *rctx) @@ -1032,9 +1042,222 @@ static int process_update_extent(const char *path, u64 offset, u64 len, return 0; } +static int decompress_zlib(struct btrfs_receive *rctx, const char *encoded_data, + u64 encoded_len, char *unencoded_data, + u64 unencoded_len) +{ + bool init = false; + int ret; + + if (!rctx->zlib_stream) { + init = true; + rctx->zlib_stream = malloc(sizeof(z_stream)); + if (!rctx->zlib_stream) { + error("failed to allocate zlib stream %m"); + return -ENOMEM; + } + } + rctx->zlib_stream->next_in = (void *)encoded_data; + rctx->zlib_stream->avail_in = encoded_len; + rctx->zlib_stream->next_out = (void *)unencoded_data; + rctx->zlib_stream->avail_out = unencoded_len; + + if (init) { + rctx->zlib_stream->zalloc = Z_NULL; + rctx->zlib_stream->zfree = Z_NULL; + rctx->zlib_stream->opaque = Z_NULL; + ret = inflateInit(rctx->zlib_stream); + } else { + ret = inflateReset(rctx->zlib_stream); + } + if (ret != Z_OK) { + error("zlib inflate init failed: %d", ret); + return -EIO; + } + + while (rctx->zlib_stream->avail_in > 0 && + rctx->zlib_stream->avail_out > 0) { + ret = inflate(rctx->zlib_stream, Z_FINISH); + if (ret == Z_STREAM_END) { + break; + } else if (ret != Z_OK) { + error("zlib inflate failed: %d", ret); + return -EIO; + } + } + return 0; +} + +static int decompress_zstd(struct btrfs_receive *rctx, const char *encoded_buf, + u64 encoded_len, char *unencoded_buf, + u64 unencoded_len) +{ + ZSTD_inBuffer in_buf = { + .src = encoded_buf, + .size = encoded_len + }; + ZSTD_outBuffer out_buf = { + .dst = unencoded_buf, + .size = unencoded_len + }; + size_t ret; + + if (!rctx->zstd_dstream) { + rctx->zstd_dstream = ZSTD_createDStream(); + if (!rctx->zstd_dstream) { + error("failed to create zstd dstream"); + return -ENOMEM; + } + } + ret = ZSTD_initDStream(rctx->zstd_dstream); + if (ZSTD_isError(ret)) { + error("failed to init zstd stream: %s", ZSTD_getErrorName(ret)); + return -EIO; + } + while (in_buf.pos < in_buf.size && out_buf.pos < out_buf.size) { + ret = ZSTD_decompressStream(rctx->zstd_dstream, &out_buf, &in_buf); + if (ret == 0) { + break; + } else if (ZSTD_isError(ret)) { + error("failed to decompress zstd stream: %s", + ZSTD_getErrorName(ret)); + return -EIO; + } + } + return 0; +} + +static int decompress_lzo(const char *encoded_data, u64 encoded_len, + char *unencoded_data, u64 unencoded_len, + unsigned int page_size) +{ + uint32_t total_len; + size_t in_pos, out_pos; + + if (encoded_len < 4) { + error("lzo header is truncated"); + return -EIO; + } + memcpy(&total_len, encoded_data, 4); + total_len = le32toh(total_len); + if (total_len > encoded_len) { + error("lzo header is invalid"); + return -EIO; + } + + in_pos = 4; + out_pos = 0; + while (in_pos < total_len && out_pos < unencoded_len) { + size_t page_remaining; + uint32_t src_len; + lzo_uint dst_len; + int ret; + + page_remaining = -in_pos % page_size; + if (page_remaining < 4) { + if (total_len - in_pos <= page_remaining) + break; + in_pos += page_remaining; + } + + if (total_len - in_pos < 4) { + error("lzo segment header is truncated"); + return -EIO; + } + + memcpy(&src_len, encoded_data + in_pos, 4); + src_len = le32toh(src_len); + in_pos += 4; + if (src_len > total_len - in_pos) { + error("lzo segment header is invalid"); + return -EIO; + } + + dst_len = page_size; + ret = lzo1x_decompress_safe((void *)(encoded_data + in_pos), + src_len, + (void *)(unencoded_data + out_pos), + &dst_len, NULL); + if (ret != LZO_E_OK) { + error("lzo1x_decompress_safe failed: %d", ret); + return -EIO; + } + + in_pos += src_len; + out_pos += dst_len; + } + return 0; +} + +static int decompress_and_write(struct btrfs_receive *rctx, + const char *encoded_data, u64 offset, + u64 encoded_len, u64 unencoded_file_len, + u64 unencoded_len, u64 unencoded_offset, + u32 compression) +{ + int ret = 0; + size_t pos; + ssize_t w; + char *unencoded_data; + int page_shift; + + unencoded_data = calloc(unencoded_len, 1); + if (!unencoded_data) { + error("allocating space for unencoded data failed: %m"); + return -errno; + } + + switch (compression) { + case ENCODED_IOV_COMPRESSION_BTRFS_ZLIB: + ret = decompress_zlib(rctx, encoded_data, encoded_len, + unencoded_data, unencoded_len); + if (ret) + goto out; + break; + case ENCODED_IOV_COMPRESSION_BTRFS_ZSTD: + ret = decompress_zstd(rctx, encoded_data, encoded_len, + unencoded_data, unencoded_len); + if (ret) + goto out; + break; + case ENCODED_IOV_COMPRESSION_BTRFS_LZO_4K: + case ENCODED_IOV_COMPRESSION_BTRFS_LZO_8K: + case ENCODED_IOV_COMPRESSION_BTRFS_LZO_16K: + case ENCODED_IOV_COMPRESSION_BTRFS_LZO_32K: + case ENCODED_IOV_COMPRESSION_BTRFS_LZO_64K: + page_shift = compression - ENCODED_IOV_COMPRESSION_BTRFS_LZO_4K + 12; + ret = decompress_lzo(encoded_data, encoded_len, unencoded_data, + unencoded_len, 1U << page_shift); + if (ret) + goto out; + break; + default: + error("unknown compression: %d", compression); + ret = -EOPNOTSUPP; + goto out; + } + + pos = unencoded_offset; + while (pos < unencoded_file_len) { + w = pwrite(rctx->write_fd, unencoded_data + pos, + unencoded_file_len - pos, offset); + if (w < 0) { + ret = -errno; + error("writing unencoded data failed: %m"); + goto out; + } + pos += w; + offset += w; + } +out: + free(unencoded_data); + return ret; +} + static int process_encoded_write(const char *path, const void *data, u64 offset, - u64 len, u64 unencoded_file_len, u64 unencoded_len, - u64 unencoded_offset, u32 compression, u32 encryption, void *user) + u64 len, u64 unencoded_file_len, + u64 unencoded_len, u64 unencoded_offset, + u32 compression, u32 encryption, void *user) { int ret; ssize_t w; @@ -1051,6 +1274,7 @@ static int process_encoded_write(const char *path, const void *data, u64 offset, { &encoded, sizeof(encoded) }, { (char *)data, len } }; + bool encoded_write = !rctx->force_decompress; if (encryption) { error("encoded_write: encryption not supported"); @@ -1067,17 +1291,25 @@ static int process_encoded_write(const char *path, const void *data, u64 offset, if (ret < 0) return ret; - /* - * NOTE: encoded writes guarantee no partial writes, so we don't need to - * handle that possibility. - */ - w = pwritev2(rctx->write_fd, iov, 2, offset, RWF_ENCODED); - if (w < 0) { - ret = -errno; - error("encoded_write: writing to %s failed: %m", path); - return ret; + if (encoded_write) { + /* + * NOTE: encoded writes guarantee no partial writes, so we don't + * need to handle that possibility. + */ + w = pwritev2(rctx->write_fd, iov, 2, offset, RWF_ENCODED); + if (w >= 0) + return 0; + /* Fall back for these errors, fail hard for anything else. */ + if (errno != ENOSPC && errno != EOPNOTSUPP && errno != EINVAL) { + ret = -errno; + error("encoded_write: writing to %s failed: %m", path); + return ret; + } } - return 0; + + return decompress_and_write(rctx, data, offset, len, unencoded_file_len, + unencoded_len, unencoded_offset, + compression); } static struct btrfs_send_ops send_ops = { @@ -1257,6 +1489,12 @@ out: close(rctx->dest_dir_fd); rctx->dest_dir_fd = -1; } + if (rctx->zstd_dstream) + ZSTD_freeDStream(rctx->zstd_dstream); + if (rctx->zlib_stream) { + inflateEnd(rctx->zlib_stream); + free(rctx->zlib_stream); + } return ret; } @@ -1287,6 +1525,9 @@ static const char * const cmd_receive_usage[] = { "-m ROOTMOUNT the root mount point of the destination filesystem.", " If /proc is not accessible, use this to tell us where", " this file system is mounted.", + "--force-decompress", + " if the stream contains compressed data, always", + " decompress it instead of writing it with encoded I/O", "--dump dump stream metadata, one line per operation,", " does not require the MOUNT parameter", "-v deprecated, alias for global -v option", @@ -1330,12 +1571,16 @@ static int cmd_receive(const struct cmd_struct *cmd, int argc, char **argv) optind = 0; while (1) { int c; - enum { GETOPT_VAL_DUMP = 257 }; + enum { + GETOPT_VAL_DUMP = 257, + GETOPT_VAL_FORCE_DECOMPRESS, + }; static const struct option long_opts[] = { { "max-errors", required_argument, NULL, 'E' }, { "chroot", no_argument, NULL, 'C' }, { "dump", no_argument, NULL, GETOPT_VAL_DUMP }, { "quiet", no_argument, NULL, 'q' }, + { "force-decompress", no_argument, NULL, GETOPT_VAL_FORCE_DECOMPRESS }, { NULL, 0, NULL, 0 } }; @@ -1378,6 +1623,9 @@ static int cmd_receive(const struct cmd_struct *cmd, int argc, char **argv) case GETOPT_VAL_DUMP: dump = 1; break; + case GETOPT_VAL_FORCE_DECOMPRESS: + rctx.force_decompress = 1; + break; default: usage_unknown_option(cmd, argv); } From patchwork Tue Mar 16 19:44:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 12143749 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 315D9C433DB for ; Tue, 16 Mar 2021 19:47:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 070E464F6B for ; Tue, 16 Mar 2021 19:47:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240683AbhCPTq3 (ORCPT ); Tue, 16 Mar 2021 15:46:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35856 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240629AbhCPTpG (ORCPT ); Tue, 16 Mar 2021 15:45:06 -0400 Received: from mail-pl1-x62f.google.com (mail-pl1-x62f.google.com [IPv6:2607:f8b0:4864:20::62f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0FFD5C0613E5 for ; Tue, 16 Mar 2021 12:44:47 -0700 (PDT) Received: by mail-pl1-x62f.google.com with SMTP id c16so17475315ply.0 for ; Tue, 16 Mar 2021 12:44:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Wge42oy0DQ6NR7o79fLXvBmuilouGkgkiyMUSGJksCk=; b=yh5kyUAgt/XzkRvSLUngbm+49zedTzhVfIqb9E/l2FoBTv+IGejH/DM/u6CSYb5irq oziGF0qdK/pF7gh4rh1xbBdaLc11o6ck46mUbzZ6MKn3DVl4B7h7IqEmIv8HzNYQ4pAz FZP5Ao5YxrUT9dG/+F1/9bsAB3+N4Ba+C2FpbStzEo+ap6YuwPZE9JcSPBnrRHD1vTyd 24qh5cOh9Nk0QGRcy1eF3tPo8GbmUAX7kArqSNXsCUhVpe/LBlE4cDlUG9nYSwwolCBv q9wBhTN7PzcoIndiZEffhz334cAziwNvARleiswB0evP9EoGTnKd51leD+oVCa5BBiSf jJ0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Wge42oy0DQ6NR7o79fLXvBmuilouGkgkiyMUSGJksCk=; b=kX5fso0xnlXMhwmHx+kM5oh9S5JNghRg0tUiGgOzWwYGlTNiQloVUigYdC9B8WfEM5 u8maXa8EfH4knaJiNFSgWLqYVVziKQ5cnz3nAT3NWJdHwJT9Zjj6QLw0DWwhR8aVTnuD KTz/tv2Sqx48zevgFfSLRbbT4AdtXSs1m1llGDbZNnuHVBbg+41G5k6PmFTXm/gIiARU Dsbo02X/SOyusO0Isd3nuzFQTp8lnTNkpnG12iJetBQcqf6AZN/GmUQw40cjiLNPszsm lV2LC2zSe1kuiqZw7kMhVBzHaofMiNaymGeu6coo5xg+5pcGPVWanrqihO2T1yCpmbaa /r/w== X-Gm-Message-State: AOAM5339bbhXFNa70fm8KuRvnh1gWMN+PLlq9Ot2B2YJ3bzIVwhgBt7M R9We0p6S5hAmdunKUfQs8uEiaYiRIOydOA== X-Google-Smtp-Source: ABdhPJwv2xuBFNcCxrY/DI9K/OQyCvg0WLQg1qiCLzeS7ufzXwOWb7okENo5JC/ZISaFGOXYSAy8rA== X-Received: by 2002:a17:902:e74e:b029:e5:bde4:2b80 with SMTP id p14-20020a170902e74eb02900e5bde42b80mr967896plf.44.1615923885902; Tue, 16 Mar 2021 12:44:45 -0700 (PDT) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:532]) by smtp.gmail.com with ESMTPSA id w22sm16919104pfi.133.2021.03.16.12.44.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Mar 2021 12:44:45 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v4 08/11] btrfs-progs: receive: process fallocate commands Date: Tue, 16 Mar 2021 12:44:02 -0700 Message-Id: X-Mailer: git-send-email 2.30.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov Send stream v2 can emit fallocate commands, so receive must support them as well. The implementation simply passes along the arguments to the syscall. Note that mode is encoded as a u32 in send stream but fallocate takes an int, so there is a unsigned->signed conversion there. Signed-off-by: Boris Burkov --- cmds/receive-dump.c | 9 +++++++++ cmds/receive.c | 25 +++++++++++++++++++++++++ common/send-stream.c | 9 +++++++++ common/send-stream.h | 2 ++ 4 files changed, 45 insertions(+) diff --git a/cmds/receive-dump.c b/cmds/receive-dump.c index 20ec2b70..acc0ba32 100644 --- a/cmds/receive-dump.c +++ b/cmds/receive-dump.c @@ -329,6 +329,14 @@ static int print_encoded_write(const char *path, const void *data, u64 offset, unencoded_offset, compression, encryption); } +static int print_fallocate(const char *path, int mode, u64 offset, u64 len, + void *user) +{ + return PRINT_DUMP(user, path, "fallocate", + "mode=%d offset=%llu len=%llu", + mode, offset, len); +} + struct btrfs_send_ops btrfs_print_send_ops = { .subvol = print_subvol, .snapshot = print_snapshot, @@ -352,4 +360,5 @@ struct btrfs_send_ops btrfs_print_send_ops = { .utimes = print_utimes, .update_extent = print_update_extent, .encoded_write = print_encoded_write, + .fallocate = print_fallocate, }; diff --git a/cmds/receive.c b/cmds/receive.c index f0236f50..9103dabb 100644 --- a/cmds/receive.c +++ b/cmds/receive.c @@ -1312,6 +1312,30 @@ static int process_encoded_write(const char *path, const void *data, u64 offset, compression); } +static int process_fallocate(const char *path, int mode, u64 offset, u64 len, + void *user) +{ + int ret; + struct btrfs_receive *rctx = user; + char full_path[PATH_MAX]; + + ret = path_cat_out(full_path, rctx->full_subvol_path, path); + if (ret < 0) { + error("fallocate: path invalid: %s", path); + return ret; + } + ret = open_inode_for_write(rctx, full_path, false); + if (ret < 0) + return ret; + ret = fallocate(rctx->write_fd, mode, offset, len); + if (ret < 0) { + ret = -errno; + error("fallocate: fallocate on %s failed: %m", path); + return ret; + } + return 0; +} + static struct btrfs_send_ops send_ops = { .subvol = process_subvol, .snapshot = process_snapshot, @@ -1335,6 +1359,7 @@ static struct btrfs_send_ops send_ops = { .utimes = process_utimes, .update_extent = process_update_extent, .encoded_write = process_encoded_write, + .fallocate = process_fallocate, }; static int do_receive(struct btrfs_receive *rctx, const char *tomnt, diff --git a/common/send-stream.c b/common/send-stream.c index 044e101b..bc41396e 100644 --- a/common/send-stream.c +++ b/common/send-stream.c @@ -369,6 +369,7 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx) u64 unencoded_offset; int len; int xattr_len; + int fallocate_mode; ret = read_cmd(sctx); if (ret) @@ -514,6 +515,14 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx) case BTRFS_SEND_C_END: ret = 1; break; + case BTRFS_SEND_C_FALLOCATE: + TLV_GET_STRING(sctx, BTRFS_SEND_A_PATH, &path); + TLV_GET_U32(sctx, BTRFS_SEND_A_FALLOCATE_MODE, &fallocate_mode); + TLV_GET_U64(sctx, BTRFS_SEND_A_FILE_OFFSET, &offset); + TLV_GET_U64(sctx, BTRFS_SEND_A_SIZE, &tmp); + ret = sctx->ops->fallocate(path, fallocate_mode, offset, tmp, + sctx->user); + break; } tlv_get_failed: diff --git a/common/send-stream.h b/common/send-stream.h index 607bc007..a58739bb 100644 --- a/common/send-stream.h +++ b/common/send-stream.h @@ -70,6 +70,8 @@ struct btrfs_send_ops { u64 len, u64 unencoded_file_len, u64 unencoded_len, u64 unencoded_offset, u32 compression, u32 encryption, void *user); + int (*fallocate)(const char *path, int mode, u64 offset, u64 len, + void *user); }; int btrfs_read_and_process_send_stream(int fd, From patchwork Tue Mar 16 19:44:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 12143747 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 832BFC43603 for ; Tue, 16 Mar 2021 19:46:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7062E64F5E for ; Tue, 16 Mar 2021 19:46:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231255AbhCPTq1 (ORCPT ); Tue, 16 Mar 2021 15:46:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35860 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240625AbhCPTpH (ORCPT ); Tue, 16 Mar 2021 15:45:07 -0400 Received: from mail-pf1-x436.google.com (mail-pf1-x436.google.com [IPv6:2607:f8b0:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3E042C0613E7 for ; Tue, 16 Mar 2021 12:44:49 -0700 (PDT) Received: by mail-pf1-x436.google.com with SMTP id e26so9537277pfd.9 for ; Tue, 16 Mar 2021 12:44:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=HADdv5K3OryTpYiXcmOT6Xw/vTCCME1y1dLqCdUeIrg=; b=jgk/SXUsODhQThpK+1SKnVlwNAEbHP7KKrasi5IbeIFWHWYuA2XtPXazw2aDNa4P4v PxjAjugwxcGxy5PrikedXAZ0w4OKK9vnqEBIJRTUqlozVoqXV0llfjkorDSjgQqtYcA+ kVwZ2jNdZLdg5KBNZ27yVsgTUFe166MsKMj49/FMfnuUJ2zYEKPEdHM85d6jMC76kZlC bhlGgdMEeL8dEWlLqz7YC4Yp8cw4LciQEEz6ZSMkVI36iuJhIskPecv0BzowtcabDP78 ImeAega99HfNHTJlc7pILL+BdTmrJo7mnhkz3JE8jVsGv1DKcWkF3vFwRlZ5Q05YNWL7 AoXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=HADdv5K3OryTpYiXcmOT6Xw/vTCCME1y1dLqCdUeIrg=; b=dais5Vw7H7mHOOH+pp3Ts2V02INqMGsY3l86TRmKAQ/dlNjdY+V3IyPKhOkAQGnBca ZVeQVWE+eKuewBURsS1bDkKgkJhO7qisXQJ1gqDpinWJHcBiw5+q/dQPxz+v8X+M/Dni D1pn1JqCU+hFyvwcDPL2WrH5oJjZ13i59FaKk6LW3N2woaC/n1SN6Es47uUZlra5ci85 fJhU47MtaKbGe2/VR1Rii+kIRbsHDiFNJGWvj91KSgesad5CH93w8CfXnjCi/NS0y4Pb dFSWz7YzYAl9AmXUoCLUntT71zJ4g5BUnNnp+Kqz3hCkcZe49zC6TZBvb0voO2SHfnNE 4hbg== X-Gm-Message-State: AOAM531/DyPKXhn4PSAegu4MHkdVVgMq7NLcHyangvhk3hzbC2msqAU8 I7dy0S+MC4wq6UK8uApQgD5JTPHQsTvAQw== X-Google-Smtp-Source: ABdhPJz1GhM8NXoddkaM4aO9UPIabTNLTWS3SXQr2PvCy5lkyPyLbhGdazh5ier/5zsAI3D6s/NcBw== X-Received: by 2002:a62:7b83:0:b029:1f1:5ef3:b4d9 with SMTP id w125-20020a627b830000b02901f15ef3b4d9mr1091143pfc.65.1615923888207; Tue, 16 Mar 2021 12:44:48 -0700 (PDT) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:532]) by smtp.gmail.com with ESMTPSA id w22sm16919104pfi.133.2021.03.16.12.44.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Mar 2021 12:44:47 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v4 09/11] btrfs-progs: receive: process setflags ioctl commands Date: Tue, 16 Mar 2021 12:44:03 -0700 Message-Id: <11124eac89be601d9165736cf13beaf68ab7140f.1615922859.git.osandov@osandov.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov In send stream v2, send can emit a command for setting inode flags via the setflags ioctl. Pass the flags attribute through to the ioctl call in receive. Signed-off-by: Boris Burkov --- cmds/receive-dump.c | 6 ++++++ cmds/receive.c | 24 ++++++++++++++++++++++++ common/send-stream.c | 7 +++++++ common/send-stream.h | 1 + 4 files changed, 38 insertions(+) diff --git a/cmds/receive-dump.c b/cmds/receive-dump.c index acc0ba32..40f07ad4 100644 --- a/cmds/receive-dump.c +++ b/cmds/receive-dump.c @@ -337,6 +337,11 @@ static int print_fallocate(const char *path, int mode, u64 offset, u64 len, mode, offset, len); } +static int print_setflags(const char *path, int flags, void *user) +{ + return PRINT_DUMP(user, path, "setflags", "flags=%d", flags); +} + struct btrfs_send_ops btrfs_print_send_ops = { .subvol = print_subvol, .snapshot = print_snapshot, @@ -361,4 +366,5 @@ struct btrfs_send_ops btrfs_print_send_ops = { .update_extent = print_update_extent, .encoded_write = print_encoded_write, .fallocate = print_fallocate, + .setflags = print_setflags, }; diff --git a/cmds/receive.c b/cmds/receive.c index 9103dabb..3fc83bd1 100644 --- a/cmds/receive.c +++ b/cmds/receive.c @@ -1336,6 +1336,29 @@ static int process_fallocate(const char *path, int mode, u64 offset, u64 len, return 0; } +static int process_setflags(const char *path, int flags, void *user) +{ + int ret; + struct btrfs_receive *rctx = user; + char full_path[PATH_MAX]; + + ret = path_cat_out(full_path, rctx->full_subvol_path, path); + if (ret < 0) { + error("setflags: path invalid: %s", path); + return ret; + } + ret = open_inode_for_write(rctx, full_path, false); + if (ret < 0) + return ret; + ret = ioctl(rctx->write_fd, FS_IOC_SETFLAGS, &flags); + if (ret < 0) { + ret = -errno; + error("setflags: setflags ioctl on %s failed: %m", path); + return ret; + } + return 0; +} + static struct btrfs_send_ops send_ops = { .subvol = process_subvol, .snapshot = process_snapshot, @@ -1360,6 +1383,7 @@ static struct btrfs_send_ops send_ops = { .update_extent = process_update_extent, .encoded_write = process_encoded_write, .fallocate = process_fallocate, + .setflags = process_setflags, }; static int do_receive(struct btrfs_receive *rctx, const char *tomnt, diff --git a/common/send-stream.c b/common/send-stream.c index bc41396e..b9d35a34 100644 --- a/common/send-stream.c +++ b/common/send-stream.c @@ -370,6 +370,7 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx) int len; int xattr_len; int fallocate_mode; + int setflags_flags; ret = read_cmd(sctx); if (ret) @@ -523,8 +524,14 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx) ret = sctx->ops->fallocate(path, fallocate_mode, offset, tmp, sctx->user); break; + case BTRFS_SEND_C_SETFLAGS: + TLV_GET_STRING(sctx, BTRFS_SEND_A_PATH, &path); + TLV_GET_U32(sctx, BTRFS_SEND_A_SETFLAGS_FLAGS, &setflags_flags); + ret = sctx->ops->setflags(path, setflags_flags, sctx->user); + break; } + tlv_get_failed: out: free(path); diff --git a/common/send-stream.h b/common/send-stream.h index a58739bb..5373bf69 100644 --- a/common/send-stream.h +++ b/common/send-stream.h @@ -72,6 +72,7 @@ struct btrfs_send_ops { u32 encryption, void *user); int (*fallocate)(const char *path, int mode, u64 offset, u64 len, void *user); + int (*setflags)(const char *path, int flags, void *user); }; int btrfs_read_and_process_send_stream(int fd, From patchwork Tue Mar 16 19:44:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 12143745 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 735D4C432C3 for ; Tue, 16 Mar 2021 19:46:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6118B64FC0 for ; Tue, 16 Mar 2021 19:46:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237163AbhCPTq0 (ORCPT ); Tue, 16 Mar 2021 15:46:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35892 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234342AbhCPTpQ (ORCPT ); Tue, 16 Mar 2021 15:45:16 -0400 Received: from mail-pj1-x102c.google.com (mail-pj1-x102c.google.com [IPv6:2607:f8b0:4864:20::102c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4CEF9C0613E9 for ; Tue, 16 Mar 2021 12:44:51 -0700 (PDT) Received: by mail-pj1-x102c.google.com with SMTP id cl21-20020a17090af695b02900c61ac0f0e9so4108780pjb.1 for ; Tue, 16 Mar 2021 12:44:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yIrmegELQzDBD+K/fHYvK9pDetSTfa5MQLVDCS/2lnk=; b=1Fh16vtyyGMTjU4kNzOf9HKIj+pmhYXk9POqev3tZhre3RtjVmrg2H5DUc8hREMw63 g3FMJRrnQwFoLpGlpHpyMkDHfCAEGDTp/udDHWyIZnBD+dXH+1kj2fJtx7FwOvRLsVO2 9FDD7yVzyswv9fIqbo0tgJQt09K/447/UY3+0SYDaujkFg5oVAgJcHTCSDQF3j7BV9Wb feNAIvOFl0An2jHGqfkq2wMbQdfzMwo5Y/ecTri+E3WMdkIV0DUJUru0T0DrA9lEM4KZ U+Z0gFX7AzOu4l5K6M0LWv0Tixdynq+OJ9SXD8lRhF6t5Db85eDl9JFusca9nVE8TuA/ BrHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yIrmegELQzDBD+K/fHYvK9pDetSTfa5MQLVDCS/2lnk=; b=tOlDnt3kNTKrEtLAOqoPfkgzgA+gm/UvsO79B8x+DevQ5hB07TeTab3pPW2UsAvS9N e6LpSSmo/w9iOLgSgHpmikIhmUy7wgwC4IX0HVcLc1YnDcA+KdBK1ztk/vGDz8ivOGJi 1di3QZ0uaRzdmzH8GVgm8L3weuOn/0AGenZK+hFBV3137LUPeld45HG2NSpjyeYj9/4w Fi9i/QEIrg8lKy7fAgazv/vtnmyKcd8ZdBIRqdZvmYG/asMZYALyNdTy/pC0POOJVI74 EZFEsAbLWFJ7X7HMrCDtjYE7OjcLo/DmMVkIRI1t8lQ/kzPZ3cQr2AlGON6f1tZYZc/S 8deQ== X-Gm-Message-State: AOAM5312vvkTov8kmXotObEQzfxq2krznRAuflLAELEZnwjJdpdvpaaI QnPmRRNCKPjWJkouXsDMkdtWfx6XazNHmg== X-Google-Smtp-Source: ABdhPJxkehI52wJ+H/vnUjiLinsRUHN93QRSSEgjpMWdGlF0EQu/LCeZCOGPF5ET0RGfVex/T4MmQA== X-Received: by 2002:a17:90a:458b:: with SMTP id v11mr590742pjg.189.1615923890003; Tue, 16 Mar 2021 12:44:50 -0700 (PDT) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:532]) by smtp.gmail.com with ESMTPSA id w22sm16919104pfi.133.2021.03.16.12.44.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Mar 2021 12:44:49 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v4 10/11] btrfs-progs: send: stream v2 ioctl flags Date: Tue, 16 Mar 2021 12:44:04 -0700 Message-Id: <045eb34479405c7a27e388c4272e8ec12eec2654.1615922859.git.osandov@osandov.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov To make the btrfs send ioctl use the stream v2 format requires passing BTRFS_SEND_FLAG_STREAM_V2 in flags. Further, to cause the ioctl to emit encoded_write commands for encoded extents, we must set that flag as well as BTRFS_SEND_FLAG_COMPRESSED. Finally, we bump up the version in send.h as well, since we are now fully compatible with v2. Add two command line arguments to btrfs send: --stream-version and --compressed-data. --stream-version requires an argument which it parses as an integer and sets STREAM_V2 if the argument is 2. --compressed-data does not require an argument and automatically implies STREAM_V2 as well (COMPRESSED alone causes the ioctl to error out). Some examples to illustrate edge cases: // v1, old format and no encoded_writes btrfs send subvol btrfs send --stream-version 1 subvol // v2 and compressed, we will see encoded_writes btrfs send --compressed-data subvol btrfs send --compressed-data --stream-version 2 subvol // v2 only, new format but no encoded_writes btrfs send --stream-version 2 subvol // error: compressed needs version >= 2 btrfs send --compressed-data --stream-version 1 subvol // error: invalid version (not 1 or 2) btrfs send --stream-version 3 subvol btrfs send --compressed-data --stream-version 0 subvol btrfs send --compressed-data --stream-version 10 subvol Signed-off-by: Boris Burkov --- Documentation/btrfs-send.asciidoc | 16 ++++++++- cmds/send.c | 54 ++++++++++++++++++++++++++++++- ioctl.h | 17 +++++++++- libbtrfsutil/btrfs.h | 17 +++++++++- send.h | 2 +- 5 files changed, 101 insertions(+), 5 deletions(-) diff --git a/Documentation/btrfs-send.asciidoc b/Documentation/btrfs-send.asciidoc index c4a05672..202bcd97 100644 --- a/Documentation/btrfs-send.asciidoc +++ b/Documentation/btrfs-send.asciidoc @@ -55,7 +55,21 @@ send in 'NO_FILE_DATA' mode The output stream does not contain any file data and thus cannot be used to transfer changes. This mode is faster and is useful to show the differences in metadata. --q|--quiet:::: + +--stream-version <1|2>:: +Use the given send stream version. The default is 1. Version 2 encodes file +data slightly more efficiently; it is also required for sending compressed data +directly (see '--compressed-data'). Version 2 requires at least btrfs-progs +5.12 on both the sender and receiver and at least Linux 5.12 on the sender. + +--compressed-data:: +Send data that is compressed on the filesystem directly without decompressing +it. If the receiver supports encoded I/O (see `encoded_io`(7)), it can also +write it directly without decompressing it. Otherwise, the receiver will fall +back to decompressing it and writing it normally. This implies +'--stream-version 2'. + +-q|--quiet:: (deprecated) alias for global '-q' option -v|--verbose:: (deprecated) alias for global '-v' option diff --git a/cmds/send.c b/cmds/send.c index 3bfc69f5..80eb2510 100644 --- a/cmds/send.c +++ b/cmds/send.c @@ -452,6 +452,21 @@ static const char * const cmd_send_usage[] = { " does not contain any file data and thus cannot be used", " to transfer changes. This mode is faster and useful to", " show the differences in metadata.", + "--stream-version <1|2>", + " Use the given send stream version. The default is", + " 1. Version 2 encodes file data slightly more", + " efficiently; it is also required for sending", + " compressed data directly (see --compressed-data).", + " Version 2 requires at least btrfs-progs 5.12 on both", + " the sender and receiver and at least Linux 5.12 on the", + " sender.", + "--compressed-data", + " Send data that is compressed on the filesystem", + " directly without decompressing it. If the receiver", + " supports encoded I/O, it can also write it directly", + " without decompressing it. Otherwise, the receiver will", + " fall back to decompressing it and writing it normally.", + " This implies --stream-version 2.", "-v|--verbose deprecated, alias for global -v option", "-q|--quiet deprecated, alias for global -q option", HELPINFO_INSERT_GLOBALS, @@ -463,6 +478,7 @@ static const char * const cmd_send_usage[] = { static int cmd_send(const struct cmd_struct *cmd, int argc, char **argv) { char *subvol = NULL; + char *end; int ret; char outname[PATH_MAX]; struct btrfs_send send; @@ -474,6 +490,7 @@ static int cmd_send(const struct cmd_struct *cmd, int argc, char **argv) int full_send = 1; int new_end_cmd_semantic = 0; u64 send_flags = 0; + long stream_version = 0; memset(&send, 0, sizeof(send)); send.dump_fd = fileno(stdout); @@ -492,11 +509,17 @@ static int cmd_send(const struct cmd_struct *cmd, int argc, char **argv) optind = 0; while (1) { - enum { GETOPT_VAL_SEND_NO_DATA = 256 }; + enum { + GETOPT_VAL_SEND_NO_DATA = 256, + GETOPT_VAL_SEND_STREAM_V2, + GETOPT_VAL_SEND_COMPRESSED_DATA, + }; static const struct option long_options[] = { { "verbose", no_argument, NULL, 'v' }, { "quiet", no_argument, NULL, 'q' }, { "no-data", no_argument, NULL, GETOPT_VAL_SEND_NO_DATA }, + { "stream-version", required_argument, NULL, GETOPT_VAL_SEND_STREAM_V2 }, + { "compressed-data", no_argument, NULL, GETOPT_VAL_SEND_COMPRESSED_DATA }, { NULL, 0, NULL, 0 } }; int c = getopt_long(argc, argv, "vqec:f:i:p:", long_options, NULL); @@ -585,10 +608,39 @@ static int cmd_send(const struct cmd_struct *cmd, int argc, char **argv) case GETOPT_VAL_SEND_NO_DATA: send_flags |= BTRFS_SEND_FLAG_NO_FILE_DATA; break; + case GETOPT_VAL_SEND_STREAM_V2: + stream_version = strtol(optarg, &end, 10); + if (*end != '\0' || + stream_version < 1 || stream_version > 2) { + ret = 1; + error("invalid --stream-version. valid values: {1, 2}"); + goto out; + } + if (stream_version == 2) + send_flags |= BTRFS_SEND_FLAG_STREAM_V2; + break; + case GETOPT_VAL_SEND_COMPRESSED_DATA: + send_flags |= BTRFS_SEND_FLAG_COMPRESSED; + /* + * We want to default to stream v2 if only compressed is + * set. If stream_version is explicitly set to 0, that + * will trigger its own error condition for being an + * invalid version. + */ + if (stream_version == 0) { + stream_version = 2; + send_flags |= BTRFS_SEND_FLAG_STREAM_V2; + } + break; default: usage_unknown_option(cmd, argv); } } + if (stream_version < 2 && (send_flags & BTRFS_SEND_FLAG_COMPRESSED)) { + ret = 1; + error("--compressed requires --stream-version >= 2"); + goto out; + } if (check_argc_min(argc - optind, 1)) return 1; diff --git a/ioctl.h b/ioctl.h index ade6dcb9..46de8ac8 100644 --- a/ioctl.h +++ b/ioctl.h @@ -653,10 +653,25 @@ BUILD_ASSERT(sizeof(struct btrfs_ioctl_received_subvol_args_32) == 192); */ #define BTRFS_SEND_FLAG_OMIT_END_CMD 0x4 +/* + * Use version 2 of the send stream, which adds new commands and supports larger + * writes. + */ +#define BTRFS_SEND_FLAG_STREAM_V2 0x8 + +/* + * Send compressed data using the ENCODED_WRITE command instead of decompressing + * the data and sending it with the WRITE command. This requires + * BTRFS_SEND_FLAG_STREAM_V2. + */ +#define BTRFS_SEND_FLAG_COMPRESSED 0x10 + #define BTRFS_SEND_FLAG_MASK \ (BTRFS_SEND_FLAG_NO_FILE_DATA | \ BTRFS_SEND_FLAG_OMIT_STREAM_HEADER | \ - BTRFS_SEND_FLAG_OMIT_END_CMD) + BTRFS_SEND_FLAG_OMIT_END_CMD | \ + BTRFS_SEND_FLAG_STREAM_V2 | \ + BTRFS_SEND_FLAG_COMPRESSED) struct btrfs_ioctl_send_args { __s64 send_fd; /* in */ diff --git a/libbtrfsutil/btrfs.h b/libbtrfsutil/btrfs.h index 60d51ff6..8430a40d 100644 --- a/libbtrfsutil/btrfs.h +++ b/libbtrfsutil/btrfs.h @@ -731,10 +731,25 @@ struct btrfs_ioctl_received_subvol_args { */ #define BTRFS_SEND_FLAG_OMIT_END_CMD 0x4 +/* + * Use version 2 of the send stream, which adds new commands and supports larger + * writes. + */ +#define BTRFS_SEND_FLAG_STREAM_V2 0x8 + +/* + * Send compressed data using the ENCODED_WRITE command instead of decompressing + * the data and sending it with the WRITE command. This requires + * BTRFS_SEND_FLAG_STREAM_V2. + */ +#define BTRFS_SEND_FLAG_COMPRESSED 0x10 + #define BTRFS_SEND_FLAG_MASK \ (BTRFS_SEND_FLAG_NO_FILE_DATA | \ BTRFS_SEND_FLAG_OMIT_STREAM_HEADER | \ - BTRFS_SEND_FLAG_OMIT_END_CMD) + BTRFS_SEND_FLAG_OMIT_END_CMD | \ + BTRFS_SEND_FLAG_STREAM_V2 | \ + BTRFS_SEND_FLAG_COMPRESSED) struct btrfs_ioctl_send_args { __s64 send_fd; /* in */ diff --git a/send.h b/send.h index 3c47e0c7..fac90588 100644 --- a/send.h +++ b/send.h @@ -31,7 +31,7 @@ extern "C" { #endif #define BTRFS_SEND_STREAM_MAGIC "btrfs-stream" -#define BTRFS_SEND_STREAM_VERSION 1 +#define BTRFS_SEND_STREAM_VERSION 2 #define BTRFS_SEND_BUF_SIZE_V1 SZ_64K #define BTRFS_SEND_READ_SIZE (1024 * 48) From patchwork Tue Mar 16 19:44:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 12143751 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 373D4C433E6 for ; Tue, 16 Mar 2021 19:47:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 19C9A64F77 for ; Tue, 16 Mar 2021 19:47:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240652AbhCPTqb (ORCPT ); Tue, 16 Mar 2021 15:46:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35894 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240585AbhCPTpQ (ORCPT ); Tue, 16 Mar 2021 15:45:16 -0400 Received: from mail-pj1-x1036.google.com (mail-pj1-x1036.google.com [IPv6:2607:f8b0:4864:20::1036]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 236BFC0613EF for ; Tue, 16 Mar 2021 12:44:54 -0700 (PDT) Received: by mail-pj1-x1036.google.com with SMTP id lr1-20020a17090b4b81b02900ea0a3f38c1so4120646pjb.0 for ; Tue, 16 Mar 2021 12:44:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=csjtCPkdL7tZ0Hw4uLrgE01zyi/QWaclx9nbiJQ12+k=; b=GlL1MupUUCck+C58n+nknBl7BOsxQ+q7L/a0RohyN9atfW008/oLMhFh/JYB5MXOme uUXN0HwwkFb2X3hbCWr6Fo5fcP1jBhQ71cP1N039j/kNVPBH7OIAtuv1C4is9lTPl/Uv saj/z5nRwCi6DnEVZkUkuDoHRqGn3Kn2D+pfRS3xn/jiMzELKxxjN9HqgK5XqiZCB8N0 iGFr/bpmf2NEszY28pZQXXLrR6OjUz4+GvU/qorIVndkVzTbKf17AHxIJf6THypXoKj5 h5julzh0zItmnmrIeTpdQXCrRcv7DQaFgmh7vZx3qpeSDNqbv4+gyEKdyW7jJB2Cjefc iS0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=csjtCPkdL7tZ0Hw4uLrgE01zyi/QWaclx9nbiJQ12+k=; b=TgT5HjBtrvaiengPL3/6j8wIRd0U3y1mXz6s8rd5uNorwxT4a4qcpQ6/EabLzbfQn1 D1+Q+oN32Hs9tZ9KSfOgIboj1fxDdLYs670e+PI4ryVfpvzGNYt7EPmsrpGlyrx6yr4Q a/moB1Igs1JA9URGsN/5q/OhWhYAe2zhYlPUsKpQcmDQMTuYKj5cEuNdcHRs4x1oGQ+J yqKTN0DnR9BGOV/k1rGHg3pZR0k7zxSShqo+XX63kheM1BoOXcss79ZzdfuF/9Fyqhcq Emasq/OFdk6FB52GAZdzn4L1iwMUXyVuue5Ozd1XCDE8rn1yKV9TAECOFYXEkE4ySjbZ gqfA== X-Gm-Message-State: AOAM5332vn2D3OpExa5B0Z3ME0sT4BIEa/Yd1+j72Mj3ecBC3QsdqIkH LQNaqkQCAhvYBXvt8roCnM/tkBEB0dkwZA== X-Google-Smtp-Source: ABdhPJxDtOJ9qNhyYBLvbG6xRRmNP/szyGhfc6LFfiT/+MN1/08xcVlqFYTKrPS8tuqVNVZhF8W3Zg== X-Received: by 2002:a17:902:ee95:b029:e5:e2c7:5f76 with SMTP id a21-20020a170902ee95b02900e5e2c75f76mr864196pld.25.1615923892239; Tue, 16 Mar 2021 12:44:52 -0700 (PDT) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:532]) by smtp.gmail.com with ESMTPSA id w22sm16919104pfi.133.2021.03.16.12.44.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Mar 2021 12:44:51 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v4 11/11] btrfs-progs: receive: add tests for basic encoded_write send/receive Date: Tue, 16 Mar 2021 12:44:05 -0700 Message-Id: <10e4ab8447fd360b103229d866923f265204d940.1615922859.git.osandov@osandov.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov Adapt the existing send/receive tests by passing '-o --force-compress' to the mount commands in a new test. After writing a few files in the various compression formats, send/receive them with and without --force-decompress to test both the encoded_write path and the fallback to decode+write. Signed-off-by: Boris Burkov --- .../043-receive-write-encoded/test.sh | 114 ++++++++++++++++++ 1 file changed, 114 insertions(+) create mode 100755 tests/misc-tests/043-receive-write-encoded/test.sh diff --git a/tests/misc-tests/043-receive-write-encoded/test.sh b/tests/misc-tests/043-receive-write-encoded/test.sh new file mode 100755 index 00000000..b9390e88 --- /dev/null +++ b/tests/misc-tests/043-receive-write-encoded/test.sh @@ -0,0 +1,114 @@ +#!/bin/bash +# +# test that we can send and receive encoded writes for three modes of +# transparent compression: zlib, lzo, and zstd. + +source "$TEST_TOP/common" + +check_prereq mkfs.btrfs +check_prereq btrfs + +setup_root_helper +prepare_test_dev + +here=`pwd` + +# assumes the filesystem exists, and does mount, write, snapshot, send, unmount +# for the specified encoding option +send_one() { + local str + local subv + local snap + + algorithm="$1" + shift + str="$1" + shift + + subv="subv-$algorithm" + snap="snap-$algorithm" + + run_check_mount_test_dev "-o" "compress-force=$algorithm" + cd "$TEST_MNT" || _fail "cannot chdir to TEST_MNT" + + run_check $SUDO_HELPER "$TOP/btrfs" subvolume create "$subv" + run_check $SUDO_HELPER dd if=/dev/zero of="$subv/file1" bs=1M count=1 + run_check $SUDO_HELPER dd if=/dev/zero of="$subv/file2" bs=500K count=1 + run_check $SUDO_HELPER "$TOP/btrfs" subvolume snapshot -r "$subv" "$snap" + run_check $SUDO_HELPER "$TOP/btrfs" send -f "$str" "$snap" "$@" + + cd "$here" || _fail "cannot chdir back to test directory" + run_check_umount_test_dev +} + +receive_one() { + local str + str="$1" + shift + + run_check_mkfs_test_dev + run_check_mount_test_dev + run_check $SUDO_HELPER "$TOP/btrfs" receive "$@" -v -f "$str" "$TEST_MNT" + run_check_umount_test_dev + run_check rm -f -- "$str" +} + +test_one_write_encoded() { + local str + local algorithm + algorithm="$1" + shift + str="$here/stream-$algorithm.stream" + + run_check_mkfs_test_dev + send_one "$algorithm" "$str" --compressed-data + receive_one "$str" "$@" +} + +test_one_stream_v1() { + local str + local algorithm + algorithm="$1" + shift + str="$here/stream-$algorithm.stream" + + run_check_mkfs_test_dev + send_one "$algorithm" "$str" --stream-version 1 + receive_one "$str" "$@" +} + +test_mix_write_encoded() { + local strzlib + local strlzo + local strzstd + strzlib="$here/stream-zlib.stream" + strlzo="$here/stream-lzo.stream" + strzstd="$here/stream-zstd.stream" + + run_check_mkfs_test_dev + + send_one "zlib" "$strzlib" --compressed-data + send_one "lzo" "$strlzo" --compressed-data + send_one "zstd" "$strzstd" --compressed-data + + receive_one "$strzlib" + receive_one "$strlzo" + receive_one "$strzstd" +} + +test_one_write_encoded "zlib" +test_one_write_encoded "lzo" +test_one_write_encoded "zstd" + +# with decompression forced +test_one_write_encoded "zlib" "--force-decompress" +test_one_write_encoded "lzo" "--force-decompress" +test_one_write_encoded "zstd" "--force-decompress" + +# send stream v1 +test_one_stream_v1 "zlib" +test_one_stream_v1 "lzo" +test_one_stream_v1 "zstd" + +# files use a mix of compression algorithms +test_mix_write_encoded