From patchwork Fri Aug 21 07:39:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11728467 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 651E21392 for ; Fri, 21 Aug 2020 07:40:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4CF33207DF for ; Fri, 21 Aug 2020 07:40:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="vGjTBteg" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728320AbgHUHke (ORCPT ); Fri, 21 Aug 2020 03:40:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37192 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728107AbgHUHkS (ORCPT ); Fri, 21 Aug 2020 03:40:18 -0400 Received: from mail-pj1-x1041.google.com (mail-pj1-x1041.google.com [IPv6:2607:f8b0:4864:20::1041]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C0FE8C061386 for ; Fri, 21 Aug 2020 00:40:17 -0700 (PDT) Received: by mail-pj1-x1041.google.com with SMTP id mt12so461458pjb.4 for ; Fri, 21 Aug 2020 00:40:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5SqT5HAFYRljt1AMWeBXhRdOtC1aJb3/tKpBWuypvUM=; b=vGjTBtegLouDINSX89/DUbl0rXrvGMosTXgf3yh/KvRO+XaYrs13qKTStO39lW2D1v e71FCzZqXXyTo5b5NdWFb+1SNsyZZ8ethiDK9sz1r9bj1nmuPBGtQ7q4JBohtBJNrQOQ sOwHxaI+rpxGbY/xTgxsHcDplJcvrf7KjcyapKfQRCu53rB60SgmSX03EzpYXweEtL5S KuruJfdEq/nKjJKKGIBvnwjYCcVvjlsp7rKcVwVCdbcFRDviA2Xk5/vZpD3V/ERoqMOx MbLlkRF9GoKP+ODevyrSHp0wqnZvkwbyK1QntGRoRyel+JVMu0bic9qDMBpTd61a9mYE 3JhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5SqT5HAFYRljt1AMWeBXhRdOtC1aJb3/tKpBWuypvUM=; b=HuXysuiPPHhgPwF+Nr91WWQujTmW/n56WZds9WLM0hSTgJY3V7i2XeXsRkBykK6yi7 lnkaFyiHpYQNs2EX+TBQ5gLjeHKywoMpQwvTQHybqlbJnxNVeUabJhQ2EV9Xrc6ZhCGF t2AvN+2M3wxLFfrgGKSzwbIXOpgVMiMZzP4PZxXKfVdoHy2CM/kYMSxYUSW7KP+BHtBt NBwV7zTPgfNz4aTRu/m+uqQMbzU0KCrCdoKhEDzpgBz4CJY6q0lkTDp+9uDOiM/ioUA9 VQc03mpUrYYgNFrXVROR3UKC2Jk4fZjEQuKCh2gI01JFflsFItLfAGFT0D7gY8E/W8s1 T35A== X-Gm-Message-State: AOAM531WABTomGpCmqpVNz9EmVdcCz8AHr4QvjgH6gHM5zZMrBTYTfhz 8XzifT2YzM5ho3Y74kjkiqnO5V5yV3BvpQ== X-Google-Smtp-Source: ABdhPJxN9ZmKpfnmcT+flno4G92mVF8eJPuMYv1H6pNBYN2fp+LMtlORB8iGhREqfVBYgy4HaRl81Q== X-Received: by 2002:a17:902:b60d:: with SMTP id b13mr1467416pls.48.1597995616710; Fri, 21 Aug 2020 00:40:16 -0700 (PDT) Received: from exodia.tfbnw.net ([2620:10d:c090:400::5:f2a4]) by smtp.gmail.com with ESMTPSA id jb1sm1080875pjb.9.2020.08.21.00.40.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 00:40:14 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org Subject: [PATCH 1/9] btrfs: send: get rid of i_size logic in send_write() Date: Fri, 21 Aug 2020 00:39:51 -0700 Message-Id: <16f28691ff66e8aeb280532cc146c8ee49d6cda4.1597994106.git.osandov@osandov.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: References: MIME-Version: 1.0 Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Omar Sandoval send_write()/fill_read_buf() have some logic for avoiding reading past i_size. However, everywhere that we call send_write()/send_extent_data(), we've already clamped the length down to i_size. Get rid of the i_size handling, which simplifies the next change. Signed-off-by: Omar Sandoval Reviewed-by: Filipe Manana Reviewed-by: Josef Bacik --- fs/btrfs/send.c | 37 ++++++++++--------------------------- 1 file changed, 10 insertions(+), 27 deletions(-) diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c index 7c7c09fc65e8..8af5e867e4ca 100644 --- a/fs/btrfs/send.c +++ b/fs/btrfs/send.c @@ -4794,7 +4794,7 @@ static int process_all_new_xattrs(struct send_ctx *sctx) return ret; } -static ssize_t fill_read_buf(struct send_ctx *sctx, u64 offset, u32 len) +static int fill_read_buf(struct send_ctx *sctx, u64 offset, u32 len) { struct btrfs_root *root = sctx->send_root; struct btrfs_fs_info *fs_info = root->fs_info; @@ -4804,21 +4804,13 @@ static ssize_t fill_read_buf(struct send_ctx *sctx, u64 offset, u32 len) pgoff_t index = offset >> PAGE_SHIFT; pgoff_t last_index; unsigned pg_offset = offset_in_page(offset); - ssize_t ret = 0; + int ret = 0; + size_t read = 0; inode = btrfs_iget(fs_info->sb, sctx->cur_ino, root); if (IS_ERR(inode)) return PTR_ERR(inode); - if (offset + len > i_size_read(inode)) { - if (offset > i_size_read(inode)) - len = 0; - else - len = offset - i_size_read(inode); - } - if (len == 0) - goto out; - last_index = (offset + len - 1) >> PAGE_SHIFT; /* initial readahead */ @@ -4859,16 +4851,15 @@ static ssize_t fill_read_buf(struct send_ctx *sctx, u64 offset, u32 len) } addr = kmap(page); - memcpy(sctx->read_buf + ret, addr + pg_offset, cur_len); + memcpy(sctx->read_buf + read, addr + pg_offset, cur_len); kunmap(page); unlock_page(page); put_page(page); index++; pg_offset = 0; len -= cur_len; - ret += cur_len; + read += cur_len; } -out: iput(inode); return ret; } @@ -4882,7 +4873,6 @@ static int send_write(struct send_ctx *sctx, u64 offset, u32 len) struct btrfs_fs_info *fs_info = sctx->send_root->fs_info; int ret = 0; struct fs_path *p; - ssize_t num_read = 0; p = fs_path_alloc(); if (!p) @@ -4890,12 +4880,9 @@ static int send_write(struct send_ctx *sctx, u64 offset, u32 len) btrfs_debug(fs_info, "send_write offset=%llu, len=%d", offset, len); - num_read = fill_read_buf(sctx, offset, len); - if (num_read <= 0) { - if (num_read < 0) - ret = num_read; + ret = fill_read_buf(sctx, offset, len); + if (ret < 0) goto out; - } ret = begin_cmd(sctx, BTRFS_SEND_C_WRITE); if (ret < 0) @@ -4907,16 +4894,14 @@ static int send_write(struct send_ctx *sctx, u64 offset, u32 len) TLV_PUT_PATH(sctx, BTRFS_SEND_A_PATH, p); TLV_PUT_U64(sctx, BTRFS_SEND_A_FILE_OFFSET, offset); - TLV_PUT(sctx, BTRFS_SEND_A_DATA, sctx->read_buf, num_read); + TLV_PUT(sctx, BTRFS_SEND_A_DATA, sctx->read_buf, len); ret = send_cmd(sctx); tlv_put_failure: out: fs_path_free(p); - if (ret < 0) - return ret; - return num_read; + return ret; } /* @@ -5095,9 +5080,7 @@ static int send_extent_data(struct send_ctx *sctx, ret = send_write(sctx, offset + sent, size); if (ret < 0) return ret; - if (!ret) - break; - sent += ret; + sent += size; } return 0; } From patchwork Fri Aug 21 07:39:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11728469 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6F2EC739 for ; Fri, 21 Aug 2020 07:40:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4D97B2087D for ; Fri, 21 Aug 2020 07:40:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="yF3ZUILB" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728344AbgHUHkj (ORCPT ); Fri, 21 Aug 2020 03:40:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37202 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728250AbgHUHkU (ORCPT ); Fri, 21 Aug 2020 03:40:20 -0400 Received: from mail-pf1-x443.google.com (mail-pf1-x443.google.com [IPv6:2607:f8b0:4864:20::443]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3F0DDC061343 for ; Fri, 21 Aug 2020 00:40:20 -0700 (PDT) Received: by mail-pf1-x443.google.com with SMTP id m8so671648pfh.3 for ; Fri, 21 Aug 2020 00:40:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=rV0Iw2CCSZHG6p1G4uFOZPtC+8d3NmwhtMPWnU59b4o=; b=yF3ZUILBJ6pRuJOTEVflPp/vEYfpVErHnkY9F4KHaPMN2rchhmZ4blW6ivIKX/t5kk Bc4jBnsEvq0Al4xyHfDG4KcvMwGdOqguFCG/BabKcZLu4fgFmeNlh3/vqHK4mqim4N5n wPoJpKwrMsQiiES8kp/0YZWkWxUeXm2Ar0tWDe/Zh3t62KJ9z/HdxmIdwHaUyck3BnKe UanAV49SOd2H+bW+EVEGqhpOvnCElcpIIzhcoLeKJBy4juaDdohUU1H1H4WUZ85079tf IGbrthiM0giWDVo/bWkwdWKRRtp1blu0V3440gZZ0AugisuBgt+CCGN94WASxhEDo/Nr 0Jvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rV0Iw2CCSZHG6p1G4uFOZPtC+8d3NmwhtMPWnU59b4o=; b=LTAPoBYwFJyefayCEl3jpmQ0ENWA/Or93vKGZjasuhK8ksZLteUBqJUg53OyFtQ2Ql jmx51hTWUcRpmTdiBVdMdlRBqwqwjWudQLgmQK4Cruhu7UvqUBzYPrytXzy55ahdmX9t 7Sxu+cLC2pJ5oV/wbPjfA3AM72RIcerx54IixxolavvUyQBtDro9jAsJ34P87eiKQ2Qi bJY9mQNChwpX8FmhppIiQmC0U4d/sxzaubWI9D7g0UsBrH7AfF1eJIeUHehaKIGT66dQ Qk+V6YAvSQXA7S5iNnEKVvDf66ImRbv6I+BGLqWN6DeF0LGNcJgIaaAGyFJDvXU3zzgE BGDg== X-Gm-Message-State: AOAM5305CzewY5LWrvmo4pS6D+EY7aL9+oo1CJcRwxbNR9luybEib2oF l/KoXfCZdKSMHGvuwvfhgNYLC0R3+kayEw== X-Google-Smtp-Source: ABdhPJx5BYSVv2feYz5UTuKAlmjkIsTLVPVDIM/X9wiNoLdnQYy0W7Vg3XqVJIGd7kHmLA05UdiIAQ== X-Received: by 2002:a62:9246:: with SMTP id o67mr1416722pfd.249.1597995619020; Fri, 21 Aug 2020 00:40:19 -0700 (PDT) Received: from exodia.tfbnw.net ([2620:10d:c090:400::5:f2a4]) by smtp.gmail.com with ESMTPSA id jb1sm1080875pjb.9.2020.08.21.00.40.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 00:40:17 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org Subject: [PATCH 2/9] btrfs: send: avoid copying file data Date: Fri, 21 Aug 2020 00:39:52 -0700 Message-Id: X-Mailer: git-send-email 2.28.0 In-Reply-To: References: MIME-Version: 1.0 Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Omar Sandoval send_write() currently copies from the page cache to sctx->read_buf, and then from sctx->read_buf to sctx->send_buf. Similarly, send_hole() zeroes sctx->read_buf and then copies from sctx->read_buf to sctx->send_buf. However, if we write the TLV header manually, we can copy to sctx->send_buf directly and get rid of sctx->read_buf. Signed-off-by: Omar Sandoval Reviewed-by: Filipe Manana Reviewed-by: Josef Bacik --- fs/btrfs/send.c | 65 +++++++++++++++++++++++++++++-------------------- fs/btrfs/send.h | 1 - 2 files changed, 39 insertions(+), 27 deletions(-) diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c index 8af5e867e4ca..e70f5ceb3261 100644 --- a/fs/btrfs/send.c +++ b/fs/btrfs/send.c @@ -122,8 +122,6 @@ struct send_ctx { struct file_ra_state ra; - char *read_buf; - /* * We process inodes by their increasing order, so if before an * incremental send we reverse the parent/child relationship of @@ -4794,7 +4792,25 @@ static int process_all_new_xattrs(struct send_ctx *sctx) return ret; } -static int fill_read_buf(struct send_ctx *sctx, u64 offset, u32 len) +static u64 max_send_read_size(struct send_ctx *sctx) +{ + return sctx->send_max_size - SZ_16K; +} + +static int put_data_header(struct send_ctx *sctx, u32 len) +{ + struct btrfs_tlv_header *hdr; + + if (sctx->send_max_size - sctx->send_size < sizeof(*hdr) + len) + return -EOVERFLOW; + hdr = (struct btrfs_tlv_header *)(sctx->send_buf + sctx->send_size); + hdr->tlv_type = cpu_to_le16(BTRFS_SEND_A_DATA); + hdr->tlv_len = cpu_to_le16(len); + sctx->send_size += sizeof(*hdr); + return 0; +} + +static int put_file_data(struct send_ctx *sctx, u64 offset, u32 len) { struct btrfs_root *root = sctx->send_root; struct btrfs_fs_info *fs_info = root->fs_info; @@ -4804,8 +4820,11 @@ static int fill_read_buf(struct send_ctx *sctx, u64 offset, u32 len) pgoff_t index = offset >> PAGE_SHIFT; pgoff_t last_index; unsigned pg_offset = offset_in_page(offset); - int ret = 0; - size_t read = 0; + int ret; + + ret = put_data_header(sctx, len); + if (ret) + return ret; inode = btrfs_iget(fs_info->sb, sctx->cur_ino, root); if (IS_ERR(inode)) @@ -4851,14 +4870,15 @@ static int fill_read_buf(struct send_ctx *sctx, u64 offset, u32 len) } addr = kmap(page); - memcpy(sctx->read_buf + read, addr + pg_offset, cur_len); + memcpy(sctx->send_buf + sctx->send_size, addr + pg_offset, + cur_len); kunmap(page); unlock_page(page); put_page(page); index++; pg_offset = 0; len -= cur_len; - read += cur_len; + sctx->send_size += cur_len; } iput(inode); return ret; @@ -4880,10 +4900,6 @@ static int send_write(struct send_ctx *sctx, u64 offset, u32 len) btrfs_debug(fs_info, "send_write offset=%llu, len=%d", offset, len); - ret = fill_read_buf(sctx, offset, len); - if (ret < 0) - goto out; - ret = begin_cmd(sctx, BTRFS_SEND_C_WRITE); if (ret < 0) goto out; @@ -4894,7 +4910,9 @@ static int send_write(struct send_ctx *sctx, u64 offset, u32 len) TLV_PUT_PATH(sctx, BTRFS_SEND_A_PATH, p); TLV_PUT_U64(sctx, BTRFS_SEND_A_FILE_OFFSET, offset); - TLV_PUT(sctx, BTRFS_SEND_A_DATA, sctx->read_buf, len); + ret = put_file_data(sctx, offset, len); + if (ret < 0) + goto out; ret = send_cmd(sctx); @@ -5013,8 +5031,8 @@ static int send_update_extent(struct send_ctx *sctx, static int send_hole(struct send_ctx *sctx, u64 end) { struct fs_path *p = NULL; + u64 read_size = max_send_read_size(sctx); u64 offset = sctx->cur_inode_last_extent; - u64 len; int ret = 0; /* @@ -5041,16 +5059,19 @@ static int send_hole(struct send_ctx *sctx, u64 end) ret = get_cur_path(sctx, sctx->cur_ino, sctx->cur_inode_gen, p); if (ret < 0) goto tlv_put_failure; - memset(sctx->read_buf, 0, BTRFS_SEND_READ_SIZE); while (offset < end) { - len = min_t(u64, end - offset, BTRFS_SEND_READ_SIZE); + u64 len = min(end - offset, read_size); ret = begin_cmd(sctx, BTRFS_SEND_C_WRITE); if (ret < 0) break; TLV_PUT_PATH(sctx, BTRFS_SEND_A_PATH, p); TLV_PUT_U64(sctx, BTRFS_SEND_A_FILE_OFFSET, offset); - TLV_PUT(sctx, BTRFS_SEND_A_DATA, sctx->read_buf, len); + ret = put_data_header(sctx, len); + if (ret < 0) + break; + memset(sctx->send_buf + sctx->send_size, 0, len); + sctx->send_size += len; ret = send_cmd(sctx); if (ret < 0) break; @@ -5066,17 +5087,16 @@ static int send_extent_data(struct send_ctx *sctx, const u64 offset, const u64 len) { + u64 read_size = max_send_read_size(sctx); u64 sent = 0; if (sctx->flags & BTRFS_SEND_FLAG_NO_FILE_DATA) return send_update_extent(sctx, offset, len); while (sent < len) { - u64 size = len - sent; + u64 size = min(len - sent, read_size); int ret; - if (size > BTRFS_SEND_READ_SIZE) - size = BTRFS_SEND_READ_SIZE; ret = send_write(sctx, offset + sent, size); if (ret < 0) return ret; @@ -7145,12 +7165,6 @@ long btrfs_ioctl_send(struct file *mnt_file, struct btrfs_ioctl_send_args *arg) goto out; } - sctx->read_buf = kvmalloc(BTRFS_SEND_READ_SIZE, GFP_KERNEL); - if (!sctx->read_buf) { - ret = -ENOMEM; - goto out; - } - sctx->pending_dir_moves = RB_ROOT; sctx->waiting_dir_moves = RB_ROOT; sctx->orphan_dirs = RB_ROOT; @@ -7354,7 +7368,6 @@ long btrfs_ioctl_send(struct file *mnt_file, struct btrfs_ioctl_send_args *arg) kvfree(sctx->clone_roots); kvfree(sctx->send_buf); - kvfree(sctx->read_buf); name_cache_free(sctx); diff --git a/fs/btrfs/send.h b/fs/btrfs/send.h index ead397f7034f..de91488b7cd0 100644 --- a/fs/btrfs/send.h +++ b/fs/btrfs/send.h @@ -13,7 +13,6 @@ #define BTRFS_SEND_STREAM_VERSION 1 #define BTRFS_SEND_BUF_SIZE SZ_64K -#define BTRFS_SEND_READ_SIZE (48 * SZ_1K) enum btrfs_tlv_type { BTRFS_TLV_U8, From patchwork Fri Aug 21 07:40:02 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11728503 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A467C1575 for ; Fri, 21 Aug 2020 07:41:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8D691208DB for ; Fri, 21 Aug 2020 07:41:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="BXNiuBbZ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728141AbgHUHlR (ORCPT ); Fri, 21 Aug 2020 03:41:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37220 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728353AbgHUHkm (ORCPT ); Fri, 21 Aug 2020 03:40:42 -0400 Received: from mail-pl1-x641.google.com (mail-pl1-x641.google.com [IPv6:2607:f8b0:4864:20::641]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 27C97C061388 for ; Fri, 21 Aug 2020 00:40:42 -0700 (PDT) Received: by mail-pl1-x641.google.com with SMTP id v16so525495plo.1 for ; Fri, 21 Aug 2020 00:40:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=N1N4BTnl4BaaYraexKEAlh2R7kwx90CEeerw/26WSug=; b=BXNiuBbZOZkAJvbBGbzKIlsbNo3cFOHkqLme1JEMS8uq9yYeDlgRPMn2BQoec8ylsr 8Psr06jSo9TnEsOX+E2saOj11H4du91qvpkCtPus7xGc4ygo6njrzLQ1O9ai228HjN+O 4QGbnLeSUUE53vvnbmUBFtHv7oWuQN9F0445cf8JCMtqP3KdsdMZnO+ZdR2F5fSTALo1 uZLIY4W3m2PBqsgxWpZ9uwt1PTeHKKstqFuaEO6BS6lebjSNhLK8mlPmJSfwPzUTp7kE EoVHUS6oGdVsfiH0iP02qXmwAavvtSBdHfwoFEYub6EDPUAoGvC4Yyhrki02DKyRVfSz HbGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=N1N4BTnl4BaaYraexKEAlh2R7kwx90CEeerw/26WSug=; b=BviawU7bQCLA37IZiorf3AomL0X0RNsHwUGk2afe01LcBrS8uv/lnnOe9mPccylXzv awoG90H+Ma6o61qJPWiLPvvymPyZyTMF/bytNyQwGzJKDKTXZ7JTEgk3kNzeVOFV+LFU RqNh2IdFzLuto7iM8KYIh35ZXYp47iibfZbf5Pj09xel1agMJ6tpU1fiDhq/07UO/dls 4o55GuUtXB4t8wF9L9GLRBcf0SDUxk40+Z+nDIdxRrsqS6KjDWbhtDWokyfHqUlmyjn2 X+gVMcLOwLZNbYOZ58DzVeiEL/geS6Le/kKIZ7iJ2PfNqcJ7e4O1n6/OA7S36vBjVzCe i+/Q== X-Gm-Message-State: AOAM531oO7nDxUYIJVJBAr53AvS0tibQRqbsUDFfsck49R4LkE+YPTn7 odO//OYQUNodYsiKjXWqVuXJevfRnE6nWw== X-Google-Smtp-Source: ABdhPJz+TEw8htl0uMBxwjAa8AUNwxVt2AuNzHEQXhWxlIcalWWSvFVEjLQm4Uv2WLPemXg8RBixQg== X-Received: by 2002:a17:902:407:: with SMTP id 7mr1388728ple.167.1597995641064; Fri, 21 Aug 2020 00:40:41 -0700 (PDT) Received: from exodia.tfbnw.net ([2620:10d:c090:400::5:f2a4]) by smtp.gmail.com with ESMTPSA id jb1sm1080875pjb.9.2020.08.21.00.40.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 00:40:39 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org Subject: [PATCH 03/11] btrfs-progs: receive: support v2 send stream DATA tlv format Date: Fri, 21 Aug 2020 00:40:02 -0700 Message-Id: X-Mailer: git-send-email 2.28.0 In-Reply-To: References: MIME-Version: 1.0 Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov The new format privileges the BTRFS_SEND_A_DATA attribute by guaranteeing it will always be the last attribute in any command that needs it, and by implicitly encoding the data length as the difference between the total command length in the command header and the sizes of the rest of the attributes (and of course the tlv_type identifying the DATA attribute). To parse the new stream, we must read the tlv_type and if it is not DATA, we proceed normally, but if it is DATA, we don't parse a tlv_len but simply compute the length. In addition, we add some bounds checking when parsing each chunk of data, as well as for the tlv_len itself. Signed-off-by: Boris Burkov --- common/send-stream.c | 36 ++++++++++++++++++++++++++---------- 1 file changed, 26 insertions(+), 10 deletions(-) diff --git a/common/send-stream.c b/common/send-stream.c index 51a6a94a..77d5cd04 100644 --- a/common/send-stream.c +++ b/common/send-stream.c @@ -165,28 +165,44 @@ static int read_cmd(struct btrfs_send_stream *sctx) pos = 0; while (pos < cmd_len) { - struct btrfs_tlv_header *tlv_hdr; u16 tlv_type; - u16 tlv_len; struct btrfs_send_attribute *send_attr; - tlv_hdr = (struct btrfs_tlv_header *)data; - tlv_type = le16_to_cpu(tlv_hdr->tlv_type); - tlv_len = le16_to_cpu(tlv_hdr->tlv_len); + if (cmd_len - pos < sizeof(__le16)) { + error("send stream is truncated"); + ret = -EINVAL; + goto out; + } + tlv_type = le16_to_cpu(*(__le16 *)data); if (tlv_type == 0 || tlv_type > BTRFS_SEND_A_MAX) { - error("invalid tlv in cmd tlv_type = %hu, tlv_len = %hu", - tlv_type, tlv_len); + error("invalid tlv in cmd tlv_type = %hu", tlv_type); ret = -EINVAL; goto out; } send_attr = &sctx->cmd_attrs[tlv_type]; send_attr->tlv_type = tlv_type; - send_attr->tlv_len = tlv_len; - pos += sizeof(*tlv_hdr); - data += sizeof(*tlv_hdr); + pos += sizeof(tlv_type); + data += sizeof(tlv_type); + if (sctx->version == 2 && tlv_type == BTRFS_SEND_A_DATA) { + send_attr->tlv_len = cmd_len - pos; + } else { + if (cmd_len - pos < sizeof(__le16)) { + error("send stream is truncated"); + ret = -EINVAL; + goto out; + } + send_attr->tlv_len = le16_to_cpu(*(__le16 *)data); + pos += sizeof(__le16); + data += sizeof(__le16); + } + if (cmd_len - pos < send_attr->tlv_len) { + error("send stream is truncated"); + ret = -EINVAL; + goto out; + } send_attr->data = data; pos += send_attr->tlv_len; data += send_attr->tlv_len; From patchwork Fri Aug 21 07:40:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11728499 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 604821392 for ; Fri, 21 Aug 2020 07:41:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 487E5207DF for ; Fri, 21 Aug 2020 07:41:17 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="IP6qcyz3" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728440AbgHUHlN (ORCPT ); Fri, 21 Aug 2020 03:41:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37232 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728363AbgHUHko (ORCPT ); Fri, 21 Aug 2020 03:40:44 -0400 Received: from mail-pj1-x1042.google.com (mail-pj1-x1042.google.com [IPv6:2607:f8b0:4864:20::1042]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 26DC9C06138B for ; Fri, 21 Aug 2020 00:40:44 -0700 (PDT) Received: by mail-pj1-x1042.google.com with SMTP id mt12so461899pjb.4 for ; Fri, 21 Aug 2020 00:40:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=HPllMP1dggYArlaBG8NbdTKVXA2/TOK/hKcMMPnu8LA=; b=IP6qcyz3YWCExIoUyhje7azbZk90z3J/+FZDf98t6ckQrKngJGknpqKrixQJHnL6n/ nJo6BM45hA6TVWMpLVxhEBKr7SLPZsVQvGGtl8pwv/9tZUaozMggjPKKrSOhf39LmKEe kREF5Uijlym1NEz/PLMdJvy5Y/QnT9z1YczOrRmuEwhQKtxdKseemhOOhuotGdKouSNB 7xDy2d2dLB2NLD3hUjaWuR898WiAss4ammDiyezszjDsLinXOoWPZHjw7KRR0EwD+84g zQwLNiYR2WMPjlG+cr212Sk1vTbNdDGyScUDD7S54qCiYBRttx5MatrFui/tNtqplOtl TDDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=HPllMP1dggYArlaBG8NbdTKVXA2/TOK/hKcMMPnu8LA=; b=KRJQW1smS27lxwExahT/tG7k44YfQtSxx1wopNVlfIRolSE2pel5AEOpybSJ9r1kBx DybTwyCZvPPnSsGu9BwWY8QOJv9qPbLjxAenE3foyBJgNVQ4TgQHJCsSugc10DHV1+ss PRESCudeu3V4IeZ7dCML6BWP/3zk2HwTbrW6xM77VCOiT2/TCi5KbthIDaewj+0Bfy4R l5QYn+ttlqk8MBfH+IJcddggQKtUUD2Q2bUy5GF9Q2Qbp3O2Vu/t3V4KxFbkKWf9NsVk LFgrCoX3XgS7ush6AKQE+8NK/W/DOebX8MmonBSEDvu3qRVzfQo/imG+lthFh3z+aUSq zU0Q== X-Gm-Message-State: AOAM532DBRk8ZaXl2RLQzUcQOoJbCM+W5N/X4/OA2mcYAa2qpMAxYmEU 5VFfSqbs6yHEXbmH1ZjfxU3UZ3zF8YNyww== X-Google-Smtp-Source: ABdhPJzjSz6B/wIUhrw81Khq6id6K/AFXO+kMSQ2LmrsdpT9MO9QNi0H3um10Hmj7H+rctPHlxvQAQ== X-Received: by 2002:a17:90a:c787:: with SMTP id gn7mr1497597pjb.90.1597995642917; Fri, 21 Aug 2020 00:40:42 -0700 (PDT) Received: from exodia.tfbnw.net ([2620:10d:c090:400::5:f2a4]) by smtp.gmail.com with ESMTPSA id jb1sm1080875pjb.9.2020.08.21.00.40.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 00:40:42 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org Subject: [PATCH 04/11] btrfs-progs: receive: add send stream v2 cmds and attrs to send.h Date: Fri, 21 Aug 2020 00:40:03 -0700 Message-Id: <3477da4106d103099b41705e2a84fb58c18cbd29.1597994354.git.osandov@osandov.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: References: MIME-Version: 1.0 Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov Send stream v2 adds three commands and several attributes associated to those commands. Before we implement processing them, add all the commands and attributes. This avoids leaving the enums in an intermediate state that doesn't correspond to any version of send stream. Signed-off-by: Boris Burkov --- send.h | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/send.h b/send.h index 228928a0..3c47e0c7 100644 --- a/send.h +++ b/send.h @@ -98,6 +98,11 @@ enum btrfs_send_cmd { BTRFS_SEND_C_END, BTRFS_SEND_C_UPDATE_EXTENT, + + BTRFS_SEND_C_FALLOCATE, + BTRFS_SEND_C_SETFLAGS, + BTRFS_SEND_C_ENCODED_WRITE, + __BTRFS_SEND_C_MAX, }; #define BTRFS_SEND_C_MAX (__BTRFS_SEND_C_MAX - 1) @@ -136,6 +141,16 @@ enum { BTRFS_SEND_A_CLONE_OFFSET, BTRFS_SEND_A_CLONE_LEN, + BTRFS_SEND_A_FALLOCATE_MODE, + + BTRFS_SEND_A_SETFLAGS_FLAGS, + + BTRFS_SEND_A_UNENCODED_FILE_LEN, + BTRFS_SEND_A_UNENCODED_LEN, + BTRFS_SEND_A_UNENCODED_OFFSET, + BTRFS_SEND_A_COMPRESSION, + BTRFS_SEND_A_ENCRYPTION, + __BTRFS_SEND_A_MAX, }; #define BTRFS_SEND_A_MAX (__BTRFS_SEND_A_MAX - 1) From patchwork Fri Aug 21 07:39:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11728481 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C91D11392 for ; Fri, 21 Aug 2020 07:40:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AF2982078D for ; Fri, 21 Aug 2020 07:40:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="dSg2e+H2" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728366AbgHUHko (ORCPT ); Fri, 21 Aug 2020 03:40:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37184 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727912AbgHUHk1 (ORCPT ); Fri, 21 Aug 2020 03:40:27 -0400 Received: from mail-pg1-x543.google.com (mail-pg1-x543.google.com [IPv6:2607:f8b0:4864:20::543]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E3FF9C06138B for ; Fri, 21 Aug 2020 00:40:26 -0700 (PDT) Received: by mail-pg1-x543.google.com with SMTP id v15so613816pgh.6 for ; Fri, 21 Aug 2020 00:40:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=1WFgjJ3rZW2t7bP/7ZGExcYanOQgvJvNl08xr/OUhp8=; b=dSg2e+H2NFSciBdWMN+2zt/ZVTY+R/pe0F+3RGzZK41krLC+OjGma5ZjnKJyfc3nZE Ztr9zGKRsIn8jIb4exSJiIiZkHITbW36ekN401fQengv4nO2CrTBv8NrN411ivDKZihR om7XiqZXRJU4UFpspAWhOFyjWYtvoPl+RShLOe9ucOOJ7ZVFZrgN9FKYUWCCzX3otcu3 irpkJIWAh0ZPgkR087E9sjjlLvKX5HQfYRMNmUOY/R+PWjxM4o2cvkdOIIY7tn76llsL 18Pm6SlITDGkWEF1oMdpXTL3YR5kLMJxh0/++yrWGvcJGmWNxSyFNqWQVnJPUSVsynFO yHvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1WFgjJ3rZW2t7bP/7ZGExcYanOQgvJvNl08xr/OUhp8=; b=XUN5d7mISgf6l4v8doGxfCAJ5TRAjDdOxtfPaSG+Kp3kzRVK2llUuxOwG0N9Mcml1U ZNwp9R1s02CMuQ7HrgJCl3ML9GLi9E1tPvK6q9gZyQIj7Ux2rqGE1ooQWIVDND7cjj9c iUesMIvnLcw+VqLtDaI/PpuFbCLjyUbba9Vvza2xVwa0mC6fVG1Y4pDqQ6UL5Q4zRN1p mLZ8F8aOD/E1nQQVaPSDVueNXUOotxKLP0US2dMcdBYeUK8wM+DXtpWnLjUdwSWzkkZs 6ExrKtaFJYLvLlh83jitYqFs7eRFqpOIZooApxws/ctDK0qLKqfsfHJo5dFse/I2eriY b3Vg== X-Gm-Message-State: AOAM531xgt+3ZpFCiBDclF4hueIpqKMdZU/PNQvwuhENmEkFATau6Acv 8gV3FQzOGCnmAbChNd7+xhDm8iI+ukO2Yw== X-Google-Smtp-Source: ABdhPJxKlmIXI+kh+VsqvVebfzvPnWqxe1vtPxCacXLgvBfG6sEyFX0MiDh0Ot2V48zgw7hzbioy1w== X-Received: by 2002:a63:5559:: with SMTP id f25mr1407868pgm.369.1597995625667; Fri, 21 Aug 2020 00:40:25 -0700 (PDT) Received: from exodia.tfbnw.net ([2620:10d:c090:400::5:f2a4]) by smtp.gmail.com with ESMTPSA id jb1sm1080875pjb.9.2020.08.21.00.40.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 00:40:23 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org Subject: [PATCH 5/9] btrfs: add send stream v2 definitions Date: Fri, 21 Aug 2020 00:39:55 -0700 Message-Id: X-Mailer: git-send-email 2.28.0 In-Reply-To: References: MIME-Version: 1.0 Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Omar Sandoval This adds the definitions of the new commands for send stream version 2 and their respective attributes: fallocate, FS_IOC_SETFLAGS (a.k.a. chattr), and encoded writes. It also documents two changes to the send stream format in v2: the receiver shouldn't assume a maximum command size, and the DATA attribute is encoded differently to allow for writes larger than 64k. These will be implemented in subsequent changes, and then the ioctl will accept the new flags. Signed-off-by: Omar Sandoval Reviewed-by: Josef Bacik --- fs/btrfs/send.c | 2 +- fs/btrfs/send.h | 30 +++++++++++++++++++++++++++++- include/uapi/linux/btrfs.h | 13 +++++++++++++ 3 files changed, 43 insertions(+), 2 deletions(-) diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c index 37ce21361782..e25c3391fc02 100644 --- a/fs/btrfs/send.c +++ b/fs/btrfs/send.c @@ -7136,7 +7136,7 @@ long btrfs_ioctl_send(struct file *mnt_file, struct btrfs_ioctl_send_args *arg) sctx->clone_roots_cnt = arg->clone_sources_count; - sctx->send_max_size = BTRFS_SEND_BUF_SIZE; + sctx->send_max_size = BTRFS_SEND_BUF_SIZE_V1; sctx->send_buf = kvmalloc(sctx->send_max_size, GFP_KERNEL); if (!sctx->send_buf) { ret = -ENOMEM; diff --git a/fs/btrfs/send.h b/fs/btrfs/send.h index de91488b7cd0..9f4f7b96b1eb 100644 --- a/fs/btrfs/send.h +++ b/fs/btrfs/send.h @@ -12,7 +12,11 @@ #define BTRFS_SEND_STREAM_MAGIC "btrfs-stream" #define BTRFS_SEND_STREAM_VERSION 1 -#define BTRFS_SEND_BUF_SIZE SZ_64K +/* + * In send stream v1, no command is larger than 64k. In send stream v2, no limit + * should be assumed. + */ +#define BTRFS_SEND_BUF_SIZE_V1 SZ_64K enum btrfs_tlv_type { BTRFS_TLV_U8, @@ -76,6 +80,13 @@ enum btrfs_send_cmd { BTRFS_SEND_C_END, BTRFS_SEND_C_UPDATE_EXTENT, + + /* The following commands were added in send stream v2. */ + + BTRFS_SEND_C_FALLOCATE, + BTRFS_SEND_C_SETFLAGS, + BTRFS_SEND_C_ENCODED_WRITE, + __BTRFS_SEND_C_MAX, }; #define BTRFS_SEND_C_MAX (__BTRFS_SEND_C_MAX - 1) @@ -106,6 +117,11 @@ enum { BTRFS_SEND_A_PATH_LINK, BTRFS_SEND_A_FILE_OFFSET, + /* + * In send stream v2, this attribute is special: it must be the last + * attribute in a command, its header contains only the type, and its + * length is implicitly the remaining length of the command. + */ BTRFS_SEND_A_DATA, BTRFS_SEND_A_CLONE_UUID, @@ -114,6 +130,18 @@ enum { BTRFS_SEND_A_CLONE_OFFSET, BTRFS_SEND_A_CLONE_LEN, + /* The following attributes were added in send stream v2. */ + + BTRFS_SEND_A_FALLOCATE_MODE, + + BTRFS_SEND_A_SETFLAGS_FLAGS, + + BTRFS_SEND_A_UNENCODED_FILE_LEN, + BTRFS_SEND_A_UNENCODED_LEN, + BTRFS_SEND_A_UNENCODED_OFFSET, + BTRFS_SEND_A_COMPRESSION, + BTRFS_SEND_A_ENCRYPTION, + __BTRFS_SEND_A_MAX, }; #define BTRFS_SEND_A_MAX (__BTRFS_SEND_A_MAX - 1) diff --git a/include/uapi/linux/btrfs.h b/include/uapi/linux/btrfs.h index 2c39d15a2beb..51e69f28d22d 100644 --- a/include/uapi/linux/btrfs.h +++ b/include/uapi/linux/btrfs.h @@ -769,6 +769,19 @@ struct btrfs_ioctl_received_subvol_args { */ #define BTRFS_SEND_FLAG_OMIT_END_CMD 0x4 +/* + * Use version 2 of the send stream, which adds new commands and supports larger + * writes. + */ +#define BTRFS_SEND_FLAG_STREAM_V2 0x8 + +/* + * Send compressed data using the ENCODED_WRITE command instead of decompressing + * the data and sending it with the WRITE command. This requires + * BTRFS_SEND_FLAG_STREAM_V2. + */ +#define BTRFS_SEND_FLAG_COMPRESSED 0x10 + #define BTRFS_SEND_FLAG_MASK \ (BTRFS_SEND_FLAG_NO_FILE_DATA | \ BTRFS_SEND_FLAG_OMIT_STREAM_HEADER | \ From patchwork Fri Aug 21 07:39:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11728479 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3F83F739 for ; Fri, 21 Aug 2020 07:40:49 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 25A42207DF for ; Fri, 21 Aug 2020 07:40:49 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="iau3GpV8" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728375AbgHUHkq (ORCPT ); Fri, 21 Aug 2020 03:40:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37232 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728268AbgHUHk3 (ORCPT ); Fri, 21 Aug 2020 03:40:29 -0400 Received: from mail-pf1-x442.google.com (mail-pf1-x442.google.com [IPv6:2607:f8b0:4864:20::442]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D279AC061342 for ; Fri, 21 Aug 2020 00:40:28 -0700 (PDT) Received: by mail-pf1-x442.google.com with SMTP id f193so653185pfa.12 for ; Fri, 21 Aug 2020 00:40:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=MpMHOROWH1K8dL9ToH1ce+0j8obccQVesOcDCSINSNg=; b=iau3GpV8ekB9m+DmlUkMpAcGc34fTiQC1Xkrmkdflfo80BK9SRQVIbkJcXexjwHMhf yLwurQrHHV9kx2jJUr9eCK2J26VhrOmBW2gMzW3eLN/3lVQ3en5/3/zx5QFRLq2X0qUa XSjhYR/xJkNn+x5tLTM4sRNfIugFG8JEDe7zqC7EGKLmrBJPm7AWBKjdIkZHD0ZDk/Hx vsCyLAeCDewv3odZ7axGSkfGsm+ponE2mDza8rBgO61tKP7QUSeqg3JDO31OmC8kOrWc wE6x4QH8fMoR+qUSc2Vvkrbk9rMoBM881+8o94PDHOKImxZZ9xEYL66k1TrVKYEMa5Qw Esfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=MpMHOROWH1K8dL9ToH1ce+0j8obccQVesOcDCSINSNg=; b=N4uVz2eWSbb2wd+erwEknBoLgBfBbhSoJBHVjDNu4VIp/F+Cg/VUTgK6Sb4jCS8+9a GGqzHiRGiwxCrzaV9hZDjyAFTCDJpp7Cyig0Z7V36h7eF96QjA8CmdJehCNTOTljUbDA qIXjMUsExGL9nTeqInTFveGHvct0zT2ipZVN+KECG0lnXdegNFEwZI6aabr1ItJZoZQO rSORG9BqboNuQbiQaeHdtkS3NqkY89+99t5cKATvPSdxez4DKJ9yHFb+Sgle99L54T6O Vl2ZZGyPCwP5/s3WknWxKbkMxWJWRkNzOERUk1P4FFxbS49zwZCCBJFJGCZ4i7CefF0x qzdA== X-Gm-Message-State: AOAM532V/O/ZAWbIRVMlnprb0EZzBiD/ZDrKZAbhAcZlz1HtKsP317sb 8d+vxdI9QGw0R+OyYyaDUJSbqvF3OxNJWw== X-Google-Smtp-Source: ABdhPJydIf99Q5Hfrly97cyxhvALDoesvJp9B1+WUNcmfgiAj0fKuLIqZ6rzXogAAWcrSVCLW+xVig== X-Received: by 2002:a62:8141:: with SMTP id t62mr1426534pfd.282.1597995627765; Fri, 21 Aug 2020 00:40:27 -0700 (PDT) Received: from exodia.tfbnw.net ([2620:10d:c090:400::5:f2a4]) by smtp.gmail.com with ESMTPSA id jb1sm1080875pjb.9.2020.08.21.00.40.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 00:40:26 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org Subject: [PATCH 6/9] btrfs: send: write larger chunks when using stream v2 Date: Fri, 21 Aug 2020 00:39:56 -0700 Message-Id: X-Mailer: git-send-email 2.28.0 In-Reply-To: References: MIME-Version: 1.0 Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Omar Sandoval The length field of the send stream TLV header is 16 bits. This means that the maximum amount of data that can be sent for one write is 64k minus one. However, encoded writes must be able to send the maximum compressed extent (128k) in one command. To support this, send stream version 2 encodes the DATA attribute differently: it has no length field, and the length is implicitly up to the end of containing command (which has a 32-bit length field). Although this is necessary for encoded writes, normal writes can benefit from it, too. For v2, let's bump up the send buffer to the maximum compressed extent size plus 16k for the other metadata (144k total). Since this will most likely be vmalloc'd (and always will be after the next commit), we round it up to the next page since we might as well use the rest of the page on systems with >16k pages. Signed-off-by: Omar Sandoval --- fs/btrfs/send.c | 34 ++++++++++++++++++++++++++-------- 1 file changed, 26 insertions(+), 8 deletions(-) diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c index e25c3391fc02..c0f81d302f49 100644 --- a/fs/btrfs/send.c +++ b/fs/btrfs/send.c @@ -4799,14 +4799,27 @@ static u64 max_send_read_size(struct send_ctx *sctx) static int put_data_header(struct send_ctx *sctx, u32 len) { - struct btrfs_tlv_header *hdr; + if (sctx->flags & BTRFS_SEND_FLAG_STREAM_V2) { + __le16 tlv_type; + + if (sctx->send_max_size - sctx->send_size < + sizeof(tlv_type) + len) + return -EOVERFLOW; + tlv_type = cpu_to_le16(BTRFS_SEND_A_DATA); + memcpy(sctx->send_buf + sctx->send_size, &tlv_type, + sizeof(tlv_type)); + sctx->send_size += sizeof(tlv_type); + } else { + struct btrfs_tlv_header *hdr; - if (sctx->send_max_size - sctx->send_size < sizeof(*hdr) + len) - return -EOVERFLOW; - hdr = (struct btrfs_tlv_header *)(sctx->send_buf + sctx->send_size); - hdr->tlv_type = cpu_to_le16(BTRFS_SEND_A_DATA); - hdr->tlv_len = cpu_to_le16(len); - sctx->send_size += sizeof(*hdr); + if (sctx->send_max_size - sctx->send_size < sizeof(*hdr) + len) + return -EOVERFLOW; + hdr = (struct btrfs_tlv_header *)(sctx->send_buf + + sctx->send_size); + hdr->tlv_type = cpu_to_le16(BTRFS_SEND_A_DATA); + hdr->tlv_len = cpu_to_le16(len); + sctx->send_size += sizeof(*hdr); + } return 0; } @@ -7136,7 +7149,12 @@ long btrfs_ioctl_send(struct file *mnt_file, struct btrfs_ioctl_send_args *arg) sctx->clone_roots_cnt = arg->clone_sources_count; - sctx->send_max_size = BTRFS_SEND_BUF_SIZE_V1; + if (sctx->flags & BTRFS_SEND_FLAG_STREAM_V2) { + sctx->send_max_size = ALIGN(SZ_16K + BTRFS_MAX_COMPRESSED, + PAGE_SIZE); + } else { + sctx->send_max_size = BTRFS_SEND_BUF_SIZE_V1; + } sctx->send_buf = kvmalloc(sctx->send_max_size, GFP_KERNEL); if (!sctx->send_buf) { ret = -ENOMEM; From patchwork Fri Aug 21 07:40:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11728513 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 57D70739 for ; Fri, 21 Aug 2020 07:41:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3B4B72087D for ; Fri, 21 Aug 2020 07:41:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="WGb9lD/E" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728458AbgHUHlX (ORCPT ); Fri, 21 Aug 2020 03:41:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37302 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728378AbgHUHkv (ORCPT ); Fri, 21 Aug 2020 03:40:51 -0400 Received: from mail-pj1-x1042.google.com (mail-pj1-x1042.google.com [IPv6:2607:f8b0:4864:20::1042]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BEA6EC061349 for ; Fri, 21 Aug 2020 00:40:50 -0700 (PDT) Received: by mail-pj1-x1042.google.com with SMTP id nv17so465638pjb.3 for ; Fri, 21 Aug 2020 00:40:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JJzpMUwPLLywPgZkTrDSxUbJuyFfFwLySMrCBI6an1Q=; b=WGb9lD/EEk978nTwkBnFUsrmz3dmqQKyQbLPwgQoKrWdtYB6kznfddOnyC0TdeBF0y vRqm5dQyCRM/c26lXGuBu09O8VS7qa24Q4ynJEPOGjk4trJu8hpNWtnNVfFpU4nyXrP8 5O8lbSNrFT2hzzbVAdXBHUMDyZzjVka/HpqCf+BKVu/DYHMxxizDGCprS/Bajbni5IuY xvXdpZ2lQoSHVMECaT7QhLOd5EE4Yn94SM+QNMwchDZYXVnMAotJBP2JSHUzxGyNRpYJ 8ZwGdqt8ZnbSel36mhQzN1e58pODamtIHPUft1QUHp3ZIqauxrV5EoV7DmcpUrsN83JD grgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JJzpMUwPLLywPgZkTrDSxUbJuyFfFwLySMrCBI6an1Q=; b=SB0UrFsb/5i8GrvDydBxhVcLcaWt7SxuaVNAeBb2QuELNuxkA9W1wgh44CS4t2ie05 6dO5zuHWEqseCzLnqdOcZOC/DRYH8g6XqMLiwF4PJmjngr3Q69CJrLuZOh5+IXGNzgjL Em2cWWquZBNnzVi7gBTdt5Se/NtOsHSrdOMgdh5Af13/8UGFgWqdYzWrT/13pCAG4g61 JAnvKBkuj/ndjmeriZcMKVKuiG11JKWNgO+oyykfADjqZ3ySzTdmdb81IILtgs270kBw rhJHNVFqsvOujhOBLQEgvadt3BuwWN7OB6ObDhasYQlydSLqxhkcLbBHAvOYEMzegej+ 8DGg== X-Gm-Message-State: AOAM530uD2QP8DFNuGXjG91if36gBjjxdhksziJ55GH7oNB/h54H+4NN 6g6VX7IgD1oDAfOF6B3yRcWpsHA5bjCy2w== X-Google-Smtp-Source: ABdhPJxsk3zqP0HOjMeW9POjNCWYqpmClhHKnIxSUapiYItczePylr2VHUZuKznq9sCJyt45kJAimw== X-Received: by 2002:a17:90a:bf86:: with SMTP id d6mr1394211pjs.83.1597995649334; Fri, 21 Aug 2020 00:40:49 -0700 (PDT) Received: from exodia.tfbnw.net ([2620:10d:c090:400::5:f2a4]) by smtp.gmail.com with ESMTPSA id jb1sm1080875pjb.9.2020.08.21.00.40.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 00:40:47 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org Subject: [PATCH 07/11] btrfs-progs: receive: encoded_write fallback to explicit decode and write Date: Fri, 21 Aug 2020 00:40:06 -0700 Message-Id: <8e4bf8b613ad4504376e9f00ff87ee2e51222a23.1597994354.git.osandov@osandov.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: References: MIME-Version: 1.0 Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov An encoded_write can fail if the file system it is being applied to does not support encoded writes or if it can't find enough contiguous space to accommodate the encoded extent. In those cases, we can likely still process an encoded_write by explicitly decoding the data and doing a normal write. Add the necessary fallback path for decoding data compressed with zlib, lzo, or zstd. zlib and zstd have reusable decoding context data structures which we cache in the receive context so that we don't have to recreate them on every encoded_write. Finally, add a command line flag for force-decompress which causes receive to always use the fallback path rather than first attempting the encoded write. Signed-off-by: Boris Burkov --- cmds/receive.c | 273 +++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 262 insertions(+), 11 deletions(-) diff --git a/cmds/receive.c b/cmds/receive.c index cd0f47ec..c67d4653 100644 --- a/cmds/receive.c +++ b/cmds/receive.c @@ -41,6 +41,10 @@ #include #include +#include +#include +#include + #include "kernel-shared/ctree.h" #include "ioctl.h" #include "cmds/commands.h" @@ -82,6 +86,8 @@ struct btrfs_receive int honor_end_cmd; + int force_decompress; + /* * Buffer to store capabilities from security.capabilities xattr, * usually 20 bytes, but make same room for potentially larger @@ -89,6 +95,10 @@ struct btrfs_receive */ char cached_capabilities[64]; int cached_capabilities_len; + + /* Reuse stream objects for encoded_write decompression fallback */ + ZSTD_DStream *zstd_dstream; + z_stream *zlib_stream; }; static int finish_subvol(struct btrfs_receive *rctx) @@ -1073,9 +1083,210 @@ static int process_update_extent(const char *path, u64 offset, u64 len, return 0; } +static int decompress_zlib(struct btrfs_receive *rctx, const void *encoded_data, + u64 encoded_len, char *unencoded_data, + u64 unencoded_len) +{ + int status = 0; + bool init = false; + int ret; + + if (!rctx->zlib_stream) { + init = true; + rctx->zlib_stream = malloc(sizeof(z_stream)); + if (!rctx->zlib_stream) { + error("failed to allocate zlib stream %m"); + status = -ENOMEM; + goto out; + } + } + rctx->zlib_stream->next_in = (void *)encoded_data; + rctx->zlib_stream->avail_in = encoded_len; + rctx->zlib_stream->next_out = (void *)unencoded_data; + rctx->zlib_stream->avail_out = unencoded_len; + + if (!init) + ret = inflateReset(rctx->zlib_stream); + else { + rctx->zlib_stream->zalloc = Z_NULL; + rctx->zlib_stream->zfree = Z_NULL; + rctx->zlib_stream->opaque = Z_NULL; + ret = inflateInit(rctx->zlib_stream); + } + if (ret != Z_OK) { + error("zlib inflate init failed %d", ret); + status = -EIO; + goto out; + } + + while (rctx->zlib_stream->avail_in > 0 && + rctx->zlib_stream->avail_out > 0) { + ret = inflate(rctx->zlib_stream, Z_FINISH); + if (ret == Z_STREAM_END) { + break; + } else if (ret != Z_OK) { + error("zlib inflate failed %d", ret); + status = -EIO; + break; + } + } +out: + return status; +} + +static int decompress_lzo(const void *encoded_data, u64 encoded_len, + char *unencoded_data, u64 unencoded_len) +{ + uint32_t total_len; + size_t in_pos, out_pos; + + if (encoded_len < 4) { + error("lzo header is truncated"); + return -EIO; + } + memcpy(&total_len, encoded_data, 4); + total_len = le32toh(total_len); + if (total_len > encoded_len) { + error("lzo header is invalid"); + return -EIO; + } + + in_pos = 4; + out_pos = 0; + while (in_pos < total_len && out_pos < unencoded_len) { + uint32_t src_len; + lzo_uint dst_len = unencoded_len - out_pos; + int ret; + + if (total_len - in_pos < 4) { + error("lzo segment header is truncated"); + return -EIO; + } + memcpy(&src_len, encoded_data + in_pos, 4); + src_len = le32toh(src_len); + in_pos += 4; + if (src_len > total_len - in_pos) { + error("lzo segment header is invalid\n"); + return -EIO; + } + + ret = lzo1x_decompress_safe((void *)(encoded_data + in_pos), + src_len, (void *)(unencoded_data + out_pos), &dst_len, + NULL); + if (ret != LZO_E_OK) { + error("lzo1x_decompress_safe failed: %d", ret); + return -EIO; + } + + in_pos += src_len; + out_pos += dst_len; + } + return 0; +} + +static int decompress_zstd(struct btrfs_receive *rctx, const void *encoded_buf, + u64 encoded_len, char *unencoded_buf, + u64 unencoded_len) +{ + ZSTD_inBuffer in_buf = { + .src = encoded_buf, + .size = encoded_len + }; + ZSTD_outBuffer out_buf = { + .dst = unencoded_buf, + .size = unencoded_len + }; + int status = 0; + size_t ret; + + if (!rctx->zstd_dstream) { + rctx->zstd_dstream = ZSTD_createDStream(); + if (!rctx->zstd_dstream) { + error("failed to create zstd dstream"); + status = -ENOMEM; + goto out; + } + } + ret = ZSTD_initDStream(rctx->zstd_dstream); + if (ZSTD_isError(ret)) { + error("failed to init zstd stream %s", ZSTD_getErrorName(ret)); + status = -EIO; + goto out; + } + while (in_buf.pos < in_buf.size && out_buf.pos < out_buf.size) { + ret = ZSTD_decompressStream(rctx->zstd_dstream, &out_buf, &in_buf); + if (ret == 0) { + break; + } else if (ZSTD_isError(ret)) { + error("failed to decompress zstd stream: %s", + ZSTD_getErrorName(ret)); + status = -EIO; + goto out; + } + } + +out: + return status; +} + +static int decompress_and_write(const void *encoded_data, u64 encoded_len, + u64 unencoded_file_len, u64 unencoded_len, + u64 unencoded_offset, u32 compression, + void *user) +{ + int ret = 0; + size_t pos; + ssize_t w; + struct btrfs_receive *rctx = user; + char *unencoded_data; + + unencoded_data = calloc(unencoded_len, sizeof(*unencoded_data)); + if (!unencoded_data) { + error("allocating space for unencoded data failed: %m"); + return -errno; + } + + switch (compression) { + case ENCODED_IOV_COMPRESSION_ZLIB: + ret = decompress_zlib(rctx, encoded_data, encoded_len, + unencoded_data, unencoded_len); + if (ret) + goto out; + break; + case ENCODED_IOV_COMPRESSION_LZO: + ret = decompress_lzo(encoded_data, encoded_len, + unencoded_data, unencoded_len); + if (ret) + goto out; + break; + case ENCODED_IOV_COMPRESSION_ZSTD: + ret = decompress_zstd(rctx, encoded_data, encoded_len, + unencoded_data, unencoded_len); + if (ret) + goto out; + break; + } + + pos = unencoded_offset; + while (pos < unencoded_file_len) { + w = pwrite(rctx->write_fd, unencoded_data + pos, + unencoded_file_len - pos, unencoded_offset + pos); + if (w < 0) { + ret = -errno; + error("writing unencoded data failed: %m"); + goto out; + } + pos += w; + } +out: + free(unencoded_data); + return ret; +} + static int process_encoded_write(const char *path, const void *data, u64 offset, - u64 len, u64 unencoded_file_len, u64 unencoded_len, - u64 unencoded_offset, u32 compression, u32 encryption, void *user) + u64 len, u64 unencoded_file_len, + u64 unencoded_len, u64 unencoded_offset, + u32 compression, u32 encryption, void *user) { int ret; struct btrfs_receive *rctx = user; @@ -1091,6 +1302,14 @@ static int process_encoded_write(const char *path, const void *data, u64 offset, { &encoded, sizeof(encoded) }, { (char *)data, len } }; + bool encoded_write = !rctx->force_decompress; + bool decompress = rctx->force_decompress; + + if (encryption) { + error("encoded_write: encryption not supported\n"); + ret = -EOPNOTSUPP; + goto out; + } ret = path_cat_out(full_path, rctx->full_subvol_path, path); if (ret < 0) { @@ -1102,15 +1321,37 @@ static int process_encoded_write(const char *path, const void *data, u64 offset, if (ret < 0) goto out; - /* - * NOTE: encoded writes guarantee no partial writes, - * so we don't need to handle that possibility. - */ - ret = pwritev2(rctx->write_fd, iov, 2, offset, RWF_ENCODED); - if (ret < 0) { - ret = -errno; - error("encoded_write: writing to %s failed: %m", path); + if (encoded_write) { + /* + * NOTE: encoded writes guarantee no partial writes, + * so we don't need to handle that possibility. + */ + ret = pwritev2(rctx->write_fd, iov, 2, offset, RWF_ENCODED); + if (ret < 0) { + /* + * error conditions where fallback to manual decompress + * and write make sense. + */ + if (errno == ENOSPC || + errno == EOPNOTSUPP || + errno == EINVAL) + decompress = true; + else { + ret = -errno; + error("encoded_write: writing to %s failed: %m", path); + goto out; + } + } } + + if (decompress) { + ret = decompress_and_write(data, len, unencoded_file_len, + unencoded_len, unencoded_offset, + compression, user); + if (ret < 0) + goto out; + } + ret = 0; out: return ret; } @@ -1300,6 +1541,12 @@ out: close(rctx->dest_dir_fd); rctx->dest_dir_fd = -1; } + if (rctx->zstd_dstream) + ZSTD_freeDStream(rctx->zstd_dstream); + if (rctx->zlib_stream) { + inflateEnd(rctx->zlib_stream); + free(rctx->zlib_stream); + } return ret; } @@ -1373,12 +1620,13 @@ static int cmd_receive(const struct cmd_struct *cmd, int argc, char **argv) optind = 0; while (1) { int c; - enum { GETOPT_VAL_DUMP = 257 }; + enum { GETOPT_VAL_DUMP = 257, GETOPT_VAL_FORCE_DECOMPRESS }; static const struct option long_opts[] = { { "max-errors", required_argument, NULL, 'E' }, { "chroot", no_argument, NULL, 'C' }, { "dump", no_argument, NULL, GETOPT_VAL_DUMP }, { "quiet", no_argument, NULL, 'q' }, + { "force-decompress", no_argument, NULL, GETOPT_VAL_FORCE_DECOMPRESS }, { NULL, 0, NULL, 0 } }; @@ -1421,6 +1669,9 @@ static int cmd_receive(const struct cmd_struct *cmd, int argc, char **argv) case GETOPT_VAL_DUMP: dump = 1; break; + case GETOPT_VAL_FORCE_DECOMPRESS: + rctx.force_decompress = 1; + break; default: usage_unknown_option(cmd, argv); } From patchwork Fri Aug 21 07:39:58 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11728487 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 382961575 for ; Fri, 21 Aug 2020 07:41:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1B56C207DF for ; Fri, 21 Aug 2020 07:41:04 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="KVI1paRT" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728422AbgHUHlC (ORCPT ); Fri, 21 Aug 2020 03:41:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37248 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728271AbgHUHkd (ORCPT ); Fri, 21 Aug 2020 03:40:33 -0400 Received: from mail-pl1-x641.google.com (mail-pl1-x641.google.com [IPv6:2607:f8b0:4864:20::641]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 94FB8C061386 for ; Fri, 21 Aug 2020 00:40:33 -0700 (PDT) Received: by mail-pl1-x641.google.com with SMTP id y6so520410plt.3 for ; Fri, 21 Aug 2020 00:40:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=s3KjqHk3CpEbau5lu3Gq41W5/sumx9zV5gzHgy+kkg8=; b=KVI1paRTNgwqX/ZfpzPPVT2fXDU9URrczlZgNYIPRcpp7GYtw7Mg09XAOhGhPV/zKd aIr1ePScfH51K2Itz2LDi8toWuNNdfr99Ns9PrDwlNhofllvY3tknuqKnXWt5lnLA2x9 sKRjm19PIUsopqEdfUpj2K1Ccslh6/nmQFy5S1DaIxHvt34i2zLBsr856z2arK501QIi h1pcW1JAGuE1+Ddummoty5sNCZ5X/p7vwtxtTVqbgNERNPN0vQz3THwDjsFJh//oHt7i ahYxFJ8Gmo8/9kJ++HvPDX55T0vg+W7YYhQE6VDFuj20Js0jxuJ9u+QJV+bucgaqI/XR FRGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=s3KjqHk3CpEbau5lu3Gq41W5/sumx9zV5gzHgy+kkg8=; b=Nsn2zsAM1oIYE3Azbdj/hKZvbZzFNwBY+3r5+bJb/UJk/DOlr0OlEOf+ad3l3feCXA JsJ+H11lerkAWD5IVZjvCGwzVb07VxomYsH2IL7Qs1xkL1OCtWImpxAkjSdHDW/qq9c8 NOB7IOCLGukfE+gcpKTlTiHq5NAcYG7PZUG5VulsJKT0sSHf00NBOf4HKYoo5f0xqRbF WxR/oq9jsXkAX+hMSnfgf02jeaP4FbL+45cwGx9FZg9Gb3jVkfZCuRyBliZvz6dh7aPG RtlEhxijm+piWJlN/zfwM1hkyScXUbtUIchfF8p1+cWuymWrn2a9eIYtwlqEAGGHoKPf ORPQ== X-Gm-Message-State: AOAM533dXzAQDI9Gq08Wc70wSb98A3AHkwK53n6tF7y179TmbEF59CKn PmIwxtdKOZIxM6+TaYoY0RJJmqxDphs+ng== X-Google-Smtp-Source: ABdhPJw+AtNCiDaoxgWSIXkn0Do0jx+Cnpjd8904hybFM3W8Wb0wpQUqQ6iMj7xo399AdQEkcd6kAw== X-Received: by 2002:a17:902:9f82:: with SMTP id g2mr1411405plq.254.1597995632275; Fri, 21 Aug 2020 00:40:32 -0700 (PDT) Received: from exodia.tfbnw.net ([2620:10d:c090:400::5:f2a4]) by smtp.gmail.com with ESMTPSA id jb1sm1080875pjb.9.2020.08.21.00.40.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 00:40:31 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org Subject: [PATCH 8/9] btrfs: send: send compressed extents with encoded writes Date: Fri, 21 Aug 2020 00:39:58 -0700 Message-Id: <8eeed3db43c3c31f4596051897578e481d6cda17.1597994106.git.osandov@osandov.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: References: MIME-Version: 1.0 Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Omar Sandoval Now that all of the pieces are in place, we can use the ENCODED_WRITE command to send compressed extents when appropriate. Signed-off-by: Omar Sandoval --- fs/btrfs/ctree.h | 4 + fs/btrfs/inode.c | 6 +- fs/btrfs/send.c | 230 +++++++++++++++++++++++++++++++++++++++++++---- 3 files changed, 220 insertions(+), 20 deletions(-) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h index 744f4212b5f7..918ae5471994 100644 --- a/fs/btrfs/ctree.h +++ b/fs/btrfs/ctree.h @@ -3020,6 +3020,10 @@ int btrfs_run_delalloc_range(struct btrfs_inode *inode, struct page *locked_page int btrfs_writepage_cow_fixup(struct page *page, u64 start, u64 end); void btrfs_writepage_endio_finish_ordered(struct page *page, u64 start, u64 end, int uptodate); +int encoded_iov_compression_from_btrfs(unsigned int compress_type); +int btrfs_encoded_read_regular_fill_pages(struct inode *inode, u64 offset, + u64 disk_io_size, + struct page **pages); ssize_t btrfs_encoded_read(struct kiocb *iocb, struct iov_iter *iter); ssize_t btrfs_encoded_write(struct kiocb *iocb, struct iov_iter *from, struct encoded_iov *encoded); diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index 174889774b10..e7cc966d7cf8 100644 --- a/fs/btrfs/inode.c +++ b/fs/btrfs/inode.c @@ -9818,7 +9818,7 @@ void btrfs_set_range_writeback(struct extent_io_tree *tree, u64 start, u64 end) } } -static int encoded_iov_compression_from_btrfs(unsigned int compress_type) +int encoded_iov_compression_from_btrfs(unsigned int compress_type) { switch (compress_type) { case BTRFS_COMPRESS_NONE: @@ -10019,8 +10019,8 @@ static void btrfs_encoded_read_endio(struct bio *bio) bio_put(bio); } -static int btrfs_encoded_read_regular_fill_pages(struct inode *inode, u64 offset, - u64 disk_io_size, struct page **pages) +int btrfs_encoded_read_regular_fill_pages(struct inode *inode, u64 offset, + u64 disk_io_size, struct page **pages) { struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb); struct btrfs_encoded_read_private priv = { diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c index efa6f8f27e4d..df6882b3ab2b 100644 --- a/fs/btrfs/send.c +++ b/fs/btrfs/send.c @@ -594,6 +594,7 @@ static int tlv_put(struct send_ctx *sctx, u16 attr, const void *data, int len) return tlv_put(sctx, attr, &__tmp, sizeof(__tmp)); \ } +TLV_PUT_DEFINE_INT(32) TLV_PUT_DEFINE_INT(64) static int tlv_put_string(struct send_ctx *sctx, u16 attr, @@ -5097,16 +5098,211 @@ static int send_hole(struct send_ctx *sctx, u64 end) return ret; } -static int send_extent_data(struct send_ctx *sctx, - const u64 offset, - const u64 len) +static int send_encoded_inline_extent(struct send_ctx *sctx, + struct btrfs_path *path, u64 offset, + u64 len) { + struct btrfs_root *root = sctx->send_root; + struct btrfs_fs_info *fs_info = root->fs_info; + struct inode *inode; + struct fs_path *p; + struct extent_buffer *leaf = path->nodes[0]; + struct btrfs_key key; + struct btrfs_file_extent_item *ei; + u64 ram_bytes; + size_t inline_size; + int ret; + + inode = btrfs_iget(fs_info->sb, sctx->cur_ino, root); + if (IS_ERR(inode)) + return PTR_ERR(inode); + + p = fs_path_alloc(); + if (!p) { + ret = -ENOMEM; + goto out; + } + + ret = begin_cmd(sctx, BTRFS_SEND_C_ENCODED_WRITE); + if (ret < 0) + goto out; + + ret = get_cur_path(sctx, sctx->cur_ino, sctx->cur_inode_gen, p); + if (ret < 0) + goto out; + + btrfs_item_key_to_cpu(leaf, &key, path->slots[0]); + ei = btrfs_item_ptr(leaf, path->slots[0], + struct btrfs_file_extent_item); + ram_bytes = btrfs_file_extent_ram_bytes(leaf, ei); + inline_size = btrfs_file_extent_inline_item_len(leaf, + btrfs_item_nr(path->slots[0])); + + TLV_PUT_PATH(sctx, BTRFS_SEND_A_PATH, p); + TLV_PUT_U64(sctx, BTRFS_SEND_A_FILE_OFFSET, offset); + TLV_PUT_U64(sctx, BTRFS_SEND_A_UNENCODED_FILE_LEN, + min(key.offset + ram_bytes - offset, len)); + TLV_PUT_U64(sctx, BTRFS_SEND_A_UNENCODED_LEN, ram_bytes); + TLV_PUT_U64(sctx, BTRFS_SEND_A_UNENCODED_OFFSET, offset - key.offset); + ret = encoded_iov_compression_from_btrfs( + btrfs_file_extent_compression(leaf, ei)); + if (ret < 0) + goto out; + TLV_PUT_U32(sctx, BTRFS_SEND_A_COMPRESSION, ret); + TLV_PUT_U32(sctx, BTRFS_SEND_A_ENCRYPTION, 0); + + ret = put_data_header(sctx, inline_size); + if (ret < 0) + goto out; + read_extent_buffer(leaf, sctx->send_buf + sctx->send_size, + btrfs_file_extent_inline_start(ei), inline_size); + sctx->send_size += inline_size; + + ret = send_cmd(sctx); + +tlv_put_failure: +out: + fs_path_free(p); + iput(inode); + return ret; +} + +static int send_encoded_extent(struct send_ctx *sctx, struct btrfs_path *path, + u64 offset, u64 len) +{ + struct btrfs_root *root = sctx->send_root; + struct btrfs_fs_info *fs_info = root->fs_info; + struct inode *inode; + struct fs_path *p; + struct extent_buffer *leaf = path->nodes[0]; + struct btrfs_key key; + struct btrfs_file_extent_item *ei; + u64 block_start; + u64 block_len; + u32 data_offset; + struct btrfs_cmd_header *hdr; + u32 crc; + int ret; + + inode = btrfs_iget(fs_info->sb, sctx->cur_ino, root); + if (IS_ERR(inode)) + return PTR_ERR(inode); + + p = fs_path_alloc(); + if (!p) { + ret = -ENOMEM; + goto out; + } + + ret = begin_cmd(sctx, BTRFS_SEND_C_ENCODED_WRITE); + if (ret < 0) + goto out; + + ret = get_cur_path(sctx, sctx->cur_ino, sctx->cur_inode_gen, p); + if (ret < 0) + goto out; + + btrfs_item_key_to_cpu(leaf, &key, path->slots[0]); + ei = btrfs_item_ptr(leaf, path->slots[0], + struct btrfs_file_extent_item); + block_start = btrfs_file_extent_disk_bytenr(leaf, ei); + block_len = btrfs_file_extent_disk_num_bytes(leaf, ei); + + TLV_PUT_PATH(sctx, BTRFS_SEND_A_PATH, p); + TLV_PUT_U64(sctx, BTRFS_SEND_A_FILE_OFFSET, offset); + TLV_PUT_U64(sctx, BTRFS_SEND_A_UNENCODED_FILE_LEN, + min(key.offset + btrfs_file_extent_num_bytes(leaf, ei) - offset, + len)); + TLV_PUT_U64(sctx, BTRFS_SEND_A_UNENCODED_LEN, + btrfs_file_extent_ram_bytes(leaf, ei)); + TLV_PUT_U64(sctx, BTRFS_SEND_A_UNENCODED_OFFSET, + offset - key.offset + btrfs_file_extent_offset(leaf, ei)); + ret = encoded_iov_compression_from_btrfs( + btrfs_file_extent_compression(leaf, ei)); + if (ret < 0) + goto out; + TLV_PUT_U32(sctx, BTRFS_SEND_A_COMPRESSION, ret); + TLV_PUT_U32(sctx, BTRFS_SEND_A_ENCRYPTION, 0); + + ret = put_data_header(sctx, block_len); + if (ret < 0) + goto out; + + data_offset = ALIGN(sctx->send_size, PAGE_SIZE); + if (data_offset > sctx->send_max_size || + sctx->send_max_size - data_offset < block_len) { + ret = -EOVERFLOW; + goto out; + } + + ret = btrfs_encoded_read_regular_fill_pages(inode, block_start, + block_len, + sctx->send_buf_pages + + (data_offset >> PAGE_SHIFT)); + if (ret) + goto out; + + hdr = (struct btrfs_cmd_header *)sctx->send_buf; + hdr->len = cpu_to_le32(sctx->send_size + block_len - sizeof(*hdr)); + hdr->crc = 0; + crc = btrfs_crc32c(0, sctx->send_buf, sctx->send_size); + crc = btrfs_crc32c(crc, sctx->send_buf + data_offset, block_len); + hdr->crc = cpu_to_le32(crc); + + ret = write_buf(sctx->send_filp, sctx->send_buf, sctx->send_size, + &sctx->send_off); + if (!ret) { + ret = write_buf(sctx->send_filp, sctx->send_buf + data_offset, + block_len, &sctx->send_off); + } + sctx->total_send_size += sctx->send_size + block_len; + sctx->cmd_send_size[le16_to_cpu(hdr->cmd)] += + sctx->send_size + block_len; + sctx->send_size = 0; + +tlv_put_failure: +out: + fs_path_free(p); + iput(inode); + return ret; +} + +static int send_extent_data(struct send_ctx *sctx, struct btrfs_path *path, + const u64 offset, const u64 len) +{ + struct extent_buffer *leaf = path->nodes[0]; + struct btrfs_file_extent_item *ei; u64 read_size = max_send_read_size(sctx); u64 sent = 0; if (sctx->flags & BTRFS_SEND_FLAG_NO_FILE_DATA) return send_update_extent(sctx, offset, len); + ei = btrfs_item_ptr(leaf, path->slots[0], + struct btrfs_file_extent_item); + if ((sctx->flags & BTRFS_SEND_FLAG_COMPRESSED) && + btrfs_file_extent_compression(leaf, ei) != BTRFS_COMPRESS_NONE) { + bool is_inline = (btrfs_file_extent_type(leaf, ei) == + BTRFS_FILE_EXTENT_INLINE); + + /* + * Send the compressed extent unless the compressed data is + * larger than the decompressed data. This can happen if we're + * not sending the entire extent, either because it has been + * partially overwritten/truncated or because this is a part of + * the extent that we couldn't clone in clone_range(). + */ + if (is_inline && + btrfs_file_extent_inline_item_len(leaf, + btrfs_item_nr(path->slots[0])) <= len) { + return send_encoded_inline_extent(sctx, path, offset, + len); + } else if (!is_inline && + btrfs_file_extent_disk_num_bytes(leaf, ei) <= len) { + return send_encoded_extent(sctx, path, offset, len); + } + } + while (sent < len) { u64 size = min(len - sent, read_size); int ret; @@ -5177,12 +5373,9 @@ static int send_capabilities(struct send_ctx *sctx) return ret; } -static int clone_range(struct send_ctx *sctx, - struct clone_root *clone_root, - const u64 disk_byte, - u64 data_offset, - u64 offset, - u64 len) +static int clone_range(struct send_ctx *sctx, struct btrfs_path *dst_path, + struct clone_root *clone_root, const u64 disk_byte, + u64 data_offset, u64 offset, u64 len) { struct btrfs_path *path; struct btrfs_key key; @@ -5206,7 +5399,7 @@ static int clone_range(struct send_ctx *sctx, */ if (clone_root->offset == 0 && len == sctx->send_root->fs_info->sectorsize) - return send_extent_data(sctx, offset, len); + return send_extent_data(sctx, dst_path, offset, len); path = alloc_path_for_send(); if (!path) @@ -5303,7 +5496,8 @@ static int clone_range(struct send_ctx *sctx, if (hole_len > len) hole_len = len; - ret = send_extent_data(sctx, offset, hole_len); + ret = send_extent_data(sctx, dst_path, offset, + hole_len); if (ret < 0) goto out; @@ -5376,14 +5570,16 @@ static int clone_range(struct send_ctx *sctx, if (ret < 0) goto out; } - ret = send_extent_data(sctx, offset + slen, + ret = send_extent_data(sctx, dst_path, + offset + slen, clone_len - slen); } else { ret = send_clone(sctx, offset, clone_len, clone_root); } } else { - ret = send_extent_data(sctx, offset, clone_len); + ret = send_extent_data(sctx, dst_path, offset, + clone_len); } if (ret < 0) @@ -5400,7 +5596,7 @@ static int clone_range(struct send_ctx *sctx, } if (len > 0) - ret = send_extent_data(sctx, offset, len); + ret = send_extent_data(sctx, dst_path, offset, len); else ret = 0; out: @@ -5431,10 +5627,10 @@ static int send_write_or_clone(struct send_ctx *sctx, struct btrfs_file_extent_item); disk_byte = btrfs_file_extent_disk_bytenr(path->nodes[0], ei); data_offset = btrfs_file_extent_offset(path->nodes[0], ei); - ret = clone_range(sctx, clone_root, disk_byte, data_offset, - offset, end - offset); + ret = clone_range(sctx, path, clone_root, disk_byte, + data_offset, offset, end - offset); } else { - ret = send_extent_data(sctx, offset, end - offset); + ret = send_extent_data(sctx, path, offset, end - offset); } sctx->cur_inode_next_write_offset = end; return ret; From patchwork Fri Aug 21 07:39:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11728491 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 716161392 for ; Fri, 21 Aug 2020 07:41:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 565D9207DF for ; Fri, 21 Aug 2020 07:41:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="FfIoIiJ2" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728335AbgHUHk7 (ORCPT ); Fri, 21 Aug 2020 03:40:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37202 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728324AbgHUHkg (ORCPT ); Fri, 21 Aug 2020 03:40:36 -0400 Received: from mail-pl1-x644.google.com (mail-pl1-x644.google.com [IPv6:2607:f8b0:4864:20::644]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2DCB6C061343 for ; Fri, 21 Aug 2020 00:40:36 -0700 (PDT) Received: by mail-pl1-x644.google.com with SMTP id y6so520450plt.3 for ; Fri, 21 Aug 2020 00:40:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=eHIY7+yv9iDvme5i3eTWSs94I5h2qd9+JUn/jl34owY=; b=FfIoIiJ2nK41QK2HVGnMsZpckArAX6hK2ETKBNhWm5hoyst/Q7RMTNJHh2WfQ63bcd S+6+0MxY+QR62LAO4H71HSNbmdNU8xJQeTBkhfMtO0sjB4859KDP4ydCpXytu9tEX+uR KlHutbcts1KwOzZ9Akiw/tCnxx0dN6DRNNrHXA+VHMKMGPph8T4mq54XeTGqM3Au8Hyj T44tD0X4TUpDiYJq1XlKWPFwqKNIqtxrhAc7mUCGcF6hQa1RTpRYxeZoJ5aJcRHN2dod heL2bKJq/0QcYONzF46wke951RPk9Y4NqjxvXVeHWoAkScrVXRHJjtgRTxNahr1x0pf0 0GGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=eHIY7+yv9iDvme5i3eTWSs94I5h2qd9+JUn/jl34owY=; b=dX8LRQHOXtzdKiFEmNrBF30sGGXeXlYG1S3ar/m/IsvTaHwKroF4znFYbOjsDIopes I9hOkk9Oqp2ikZiOXfgOLUc/QzzIURb9C72TZ+/q931tD7YrDKDgi04G8xlcv+7VDLZO 4w+/g+Mio7pAsn0tSrOYXl0Qpfgl2NUsYJ4gNmVCcRKTnZvqgPV+Q7xFhSz7MJr91i5q LPyFWAR25yN/3IF32hTgGH0lfhXwBzb+yinwQwHD4nWXbOfAkFjPi1SWv76NmFDBEsBs ZJ1CJmf/G3gGRi94ZtKpJELwWMVl8ayZcRA5t1XsRmiFnc8ZaiE0AC5zJTHTykKrzwEj tT6w== X-Gm-Message-State: AOAM531a3Ea1xRYU3BC5NVd8U0EJeV+pOdDyAJNwlVRKQVpoDET/pgye LFGRnrOhvjUv3jbrf0Ou9T058rFONeiD8A== X-Google-Smtp-Source: ABdhPJztre1VZSD47fIOEgc25Ha4Tb12UKVQQ6BZY0AZui9Q3RIoZ2N+y9OZyrxJj4d/4T5SBB64PQ== X-Received: by 2002:a17:902:c286:: with SMTP id i6mr1373034pld.63.1597995635098; Fri, 21 Aug 2020 00:40:35 -0700 (PDT) Received: from exodia.tfbnw.net ([2620:10d:c090:400::5:f2a4]) by smtp.gmail.com with ESMTPSA id jb1sm1080875pjb.9.2020.08.21.00.40.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 00:40:32 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org Subject: [PATCH 9/9] btrfs: send: enable support for stream v2 and compressed writes Date: Fri, 21 Aug 2020 00:39:59 -0700 Message-Id: X-Mailer: git-send-email 2.28.0 In-Reply-To: References: MIME-Version: 1.0 Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Omar Sandoval Now that the new support is implemented, allow the ioctl to accept the flags and update the version in sysfs. Signed-off-by: Omar Sandoval --- fs/btrfs/send.c | 10 +++++++++- fs/btrfs/send.h | 2 +- include/uapi/linux/btrfs.h | 4 +++- 3 files changed, 13 insertions(+), 3 deletions(-) diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c index df6882b3ab2b..e87dea7bd915 100644 --- a/fs/btrfs/send.c +++ b/fs/btrfs/send.c @@ -670,7 +670,10 @@ static int send_header(struct send_ctx *sctx) struct btrfs_stream_header hdr; strcpy(hdr.magic, BTRFS_SEND_STREAM_MAGIC); - hdr.version = cpu_to_le32(BTRFS_SEND_STREAM_VERSION); + if (sctx->flags & BTRFS_SEND_FLAG_STREAM_V2) + hdr.version = cpu_to_le32(2); + else + hdr.version = cpu_to_le32(1); return write_buf(sctx->send_filp, &hdr, sizeof(hdr), &sctx->send_off); @@ -7315,6 +7318,11 @@ long btrfs_ioctl_send(struct file *mnt_file, struct btrfs_ioctl_send_args *arg) ret = -EINVAL; goto out; } + if ((arg->flags & BTRFS_SEND_FLAG_COMPRESSED) && + !(arg->flags & BTRFS_SEND_FLAG_STREAM_V2)) { + ret = -EINVAL; + goto out; + } sctx = kzalloc(sizeof(struct send_ctx), GFP_KERNEL); if (!sctx) { diff --git a/fs/btrfs/send.h b/fs/btrfs/send.h index 9f4f7b96b1eb..9c83e14a43b2 100644 --- a/fs/btrfs/send.h +++ b/fs/btrfs/send.h @@ -10,7 +10,7 @@ #include "ctree.h" #define BTRFS_SEND_STREAM_MAGIC "btrfs-stream" -#define BTRFS_SEND_STREAM_VERSION 1 +#define BTRFS_SEND_STREAM_VERSION 2 /* * In send stream v1, no command is larger than 64k. In send stream v2, no limit diff --git a/include/uapi/linux/btrfs.h b/include/uapi/linux/btrfs.h index 51e69f28d22d..6f29c456e4d7 100644 --- a/include/uapi/linux/btrfs.h +++ b/include/uapi/linux/btrfs.h @@ -785,7 +785,9 @@ struct btrfs_ioctl_received_subvol_args { #define BTRFS_SEND_FLAG_MASK \ (BTRFS_SEND_FLAG_NO_FILE_DATA | \ BTRFS_SEND_FLAG_OMIT_STREAM_HEADER | \ - BTRFS_SEND_FLAG_OMIT_END_CMD) + BTRFS_SEND_FLAG_OMIT_END_CMD | \ + BTRFS_SEND_FLAG_STREAM_V2 | \ + BTRFS_SEND_FLAG_COMPRESSED) struct btrfs_ioctl_send_args { __s64 send_fd; /* in */ From patchwork Fri Aug 21 07:40:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11728521 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B8DE8739 for ; Fri, 21 Aug 2020 07:41:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A1F20207DF for ; Fri, 21 Aug 2020 07:41:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="eUl2gyDC" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728471AbgHUHlg (ORCPT ); Fri, 21 Aug 2020 03:41:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37330 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728400AbgHUHk4 (ORCPT ); Fri, 21 Aug 2020 03:40:56 -0400 Received: from mail-pj1-x1041.google.com (mail-pj1-x1041.google.com [IPv6:2607:f8b0:4864:20::1041]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 910BDC061343 for ; Fri, 21 Aug 2020 00:40:56 -0700 (PDT) Received: by mail-pj1-x1041.google.com with SMTP id mt12so462079pjb.4 for ; Fri, 21 Aug 2020 00:40:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yvoSycF4wGFmZEGYefbZtpG0hIG/VlSxdUHeeNSYgs0=; b=eUl2gyDCmKzsToDNfWl5aI9WA5NN6pL3sCIY+3KLlYrhjm1f9ukiy1AlZYTxLKl4NL asfxVR0yHk5HohAHb65P1vNM4IJWtebZAUzQQu8nHBFrhl3ZSROs8nL52Hv1Me48GDEP PrpN445otWPkYgg3gKGnUjPuoMuX+PZHN6Ystz2pgWubOuIdCbFsZIbLxj+YUT1krvtg 81wzD9rg3xfsFUOeioDf/LmXJfwoeuHIhSChKDDcwjw3o//gCigedIK4P1WUIs0uVeU1 xSy3w71XHvH305imyHqkS6xaenQ2N9olR1jdQp1KWUSOn55StedbfPgVquxqUR8ny7La WeGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yvoSycF4wGFmZEGYefbZtpG0hIG/VlSxdUHeeNSYgs0=; b=gctvxcgArFwfTSohQtzIkls+RxP8EPhwUTZL3t3CEWgt/8S4HOUeZXDN+6LgWaMTcW r7tqQdB4n9LxM1/gY03uwaIdJNHacKMKVHc4dcq+ohok42UeMXF3+JqxvdTNkDnKbg86 JDcvieSocHARF/MM9CuDlL9Fo5wrLzeh9O6AM/HjqO+up/Ht7h/+E/VSlBsjVSf0tuF7 rLVIJtkBGsANZ0yRH2R8BDffeNMrZmYEzc1ws1cLLVtdstbNu8Wx2IbkS89lmpLHlozQ 2IvXwgYrGemCTgN571KEdOMM7KZT9vagVz4DO5WfFNUpkmuwBImjcYRTRuI8o5O7Q5nz wgmQ== X-Gm-Message-State: AOAM532a+Juzku9usHA5QW4jfWQpE3I+zLzkTLK7DYSdMYAbrb/7KVdn GZJZLIa7eN517fqCrXtvKtVPRifrRAiCgA== X-Google-Smtp-Source: ABdhPJwHgF0DGJDsQ1TwGS7d7/rnUydxJiwSq4qInbfmmIxhUcFpFQy3UgHN33CeMdXWkQgr6cGylg== X-Received: by 2002:a17:90a:17ab:: with SMTP id q40mr1530279pja.28.1597995655218; Fri, 21 Aug 2020 00:40:55 -0700 (PDT) Received: from exodia.tfbnw.net ([2620:10d:c090:400::5:f2a4]) by smtp.gmail.com with ESMTPSA id jb1sm1080875pjb.9.2020.08.21.00.40.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 00:40:54 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org Subject: [PATCH 10/11] btrfs-progs: send: stream v2 ioctl flags Date: Fri, 21 Aug 2020 00:40:09 -0700 Message-Id: X-Mailer: git-send-email 2.28.0 In-Reply-To: References: MIME-Version: 1.0 Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov To make the btrfs send ioctl use the stream v2 format requires passing BTRFS_SEND_FLAG_STREAM_V2 in flags. Further, to cause the ioctl to emit encoded_write commands for encoded extents, we must set that flag as well as BTRFS_SEND_FLAG_COMPRESSED. Finally, we bump up the version in send.h as well, since we are now fully compatible with v2. Add two command line arguments to btrfs send: --stream-version and --compressed. --stream-version requires an argument which it parses as an integer and sets STREAM_V2 if the argument is 2. --compressed does not require an argument and automatically implies STREAM_V2 as well (COMPRESSED alone causes the ioctl to error out). Some examples to illustrate edge cases: // v1, old format and no encoded_writes btrfs send subvol btrfs send --stream-version 1 subvol // v2 and compressed, we will see encoded_writes btrfs send --compressed subvol btrfs send --compressed --stream-version 2 subvol // v2 only, new format but no encoded_writes btrfs send --stream-version 2 subvol // error: compressed needs version >= 2 btrfs send --compressed --stream-version 1 subvol // error: invalid version (not 1 or 2) btrfs send --stream-version 3 subvol btrfs send --compressed --stream-version 0 subvol btrfs send --compressed --stream-version 10 subvol Signed-off-by: Boris Burkov --- cmds/send.c | 39 +++++++++++++++++++++++++++++++++++++-- ioctl.h | 17 ++++++++++++++++- libbtrfsutil/btrfs.h | 17 ++++++++++++++++- send.h | 2 +- 4 files changed, 70 insertions(+), 5 deletions(-) diff --git a/cmds/send.c b/cmds/send.c index b8e3ba12..4c4eaa84 100644 --- a/cmds/send.c +++ b/cmds/send.c @@ -474,6 +474,7 @@ static int cmd_send(const struct cmd_struct *cmd, int argc, char **argv) int full_send = 1; int new_end_cmd_semantic = 0; u64 send_flags = 0; + long stream_version = 0; memset(&send, 0, sizeof(send)); send.dump_fd = fileno(stdout); @@ -492,11 +493,17 @@ static int cmd_send(const struct cmd_struct *cmd, int argc, char **argv) optind = 0; while (1) { - enum { GETOPT_VAL_SEND_NO_DATA = 256 }; + enum { + GETOPT_VAL_SEND_NO_DATA = 256, + GETOPT_VAL_SEND_STREAM_V2, + GETOPT_VAL_SEND_COMPRESSED + }; static const struct option long_options[] = { { "verbose", no_argument, NULL, 'v' }, { "quiet", no_argument, NULL, 'q' }, - { "no-data", no_argument, NULL, GETOPT_VAL_SEND_NO_DATA } + { "no-data", no_argument, NULL, GETOPT_VAL_SEND_NO_DATA }, + { "stream-version", required_argument, NULL, GETOPT_VAL_SEND_STREAM_V2 }, + { "compressed", no_argument, NULL, GETOPT_VAL_SEND_COMPRESSED } }; int c = getopt_long(argc, argv, "vqec:f:i:p:", long_options, NULL); @@ -584,10 +591,38 @@ static int cmd_send(const struct cmd_struct *cmd, int argc, char **argv) case GETOPT_VAL_SEND_NO_DATA: send_flags |= BTRFS_SEND_FLAG_NO_FILE_DATA; break; + case GETOPT_VAL_SEND_STREAM_V2: + stream_version = strtol(optarg, NULL, 10); + if (stream_version < 1 || stream_version > 2) { + ret = 1; + error("invalid --stream-version. valid values: {1, 2}"); + goto out; + } + if (stream_version == 2) + send_flags |= BTRFS_SEND_FLAG_STREAM_V2; + break; + case GETOPT_VAL_SEND_COMPRESSED: + send_flags |= BTRFS_SEND_FLAG_COMPRESSED; + /* + * We want to default to stream v2 if only compressed is + * set. If stream_version is explicitly set to 0, that + * will trigger its own error condition for being an + * invalid version. + */ + if (stream_version == 0) { + stream_version = 2; + send_flags |= BTRFS_SEND_FLAG_STREAM_V2; + } + break; default: usage_unknown_option(cmd, argv); } } + if (stream_version < 2 && (send_flags & BTRFS_SEND_FLAG_COMPRESSED)) { + ret = 1; + error("--compressed requires --stream-version >= 2"); + goto out; + } if (check_argc_min(argc - optind, 1)) return 1; diff --git a/ioctl.h b/ioctl.h index ade6dcb9..46de8ac8 100644 --- a/ioctl.h +++ b/ioctl.h @@ -653,10 +653,25 @@ BUILD_ASSERT(sizeof(struct btrfs_ioctl_received_subvol_args_32) == 192); */ #define BTRFS_SEND_FLAG_OMIT_END_CMD 0x4 +/* + * Use version 2 of the send stream, which adds new commands and supports larger + * writes. + */ +#define BTRFS_SEND_FLAG_STREAM_V2 0x8 + +/* + * Send compressed data using the ENCODED_WRITE command instead of decompressing + * the data and sending it with the WRITE command. This requires + * BTRFS_SEND_FLAG_STREAM_V2. + */ +#define BTRFS_SEND_FLAG_COMPRESSED 0x10 + #define BTRFS_SEND_FLAG_MASK \ (BTRFS_SEND_FLAG_NO_FILE_DATA | \ BTRFS_SEND_FLAG_OMIT_STREAM_HEADER | \ - BTRFS_SEND_FLAG_OMIT_END_CMD) + BTRFS_SEND_FLAG_OMIT_END_CMD | \ + BTRFS_SEND_FLAG_STREAM_V2 | \ + BTRFS_SEND_FLAG_COMPRESSED) struct btrfs_ioctl_send_args { __s64 send_fd; /* in */ diff --git a/libbtrfsutil/btrfs.h b/libbtrfsutil/btrfs.h index 60d51ff6..8430a40d 100644 --- a/libbtrfsutil/btrfs.h +++ b/libbtrfsutil/btrfs.h @@ -731,10 +731,25 @@ struct btrfs_ioctl_received_subvol_args { */ #define BTRFS_SEND_FLAG_OMIT_END_CMD 0x4 +/* + * Use version 2 of the send stream, which adds new commands and supports larger + * writes. + */ +#define BTRFS_SEND_FLAG_STREAM_V2 0x8 + +/* + * Send compressed data using the ENCODED_WRITE command instead of decompressing + * the data and sending it with the WRITE command. This requires + * BTRFS_SEND_FLAG_STREAM_V2. + */ +#define BTRFS_SEND_FLAG_COMPRESSED 0x10 + #define BTRFS_SEND_FLAG_MASK \ (BTRFS_SEND_FLAG_NO_FILE_DATA | \ BTRFS_SEND_FLAG_OMIT_STREAM_HEADER | \ - BTRFS_SEND_FLAG_OMIT_END_CMD) + BTRFS_SEND_FLAG_OMIT_END_CMD | \ + BTRFS_SEND_FLAG_STREAM_V2 | \ + BTRFS_SEND_FLAG_COMPRESSED) struct btrfs_ioctl_send_args { __s64 send_fd; /* in */ diff --git a/send.h b/send.h index 3c47e0c7..fac90588 100644 --- a/send.h +++ b/send.h @@ -31,7 +31,7 @@ extern "C" { #endif #define BTRFS_SEND_STREAM_MAGIC "btrfs-stream" -#define BTRFS_SEND_STREAM_VERSION 1 +#define BTRFS_SEND_STREAM_VERSION 2 #define BTRFS_SEND_BUF_SIZE_V1 SZ_64K #define BTRFS_SEND_READ_SIZE (1024 * 48) From patchwork Fri Aug 21 07:40:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11728523 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6A73C1575 for ; Fri, 21 Aug 2020 07:41:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 51B262078D for ; Fri, 21 Aug 2020 07:41:38 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="lEgZZeZz" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728407AbgHUHlg (ORCPT ); Fri, 21 Aug 2020 03:41:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37210 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728412AbgHUHk6 (ORCPT ); Fri, 21 Aug 2020 03:40:58 -0400 Received: from mail-pf1-x429.google.com (mail-pf1-x429.google.com [IPv6:2607:f8b0:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 58161C061357 for ; Fri, 21 Aug 2020 00:40:58 -0700 (PDT) Received: by mail-pf1-x429.google.com with SMTP id d22so667854pfn.5 for ; Fri, 21 Aug 2020 00:40:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=83qY1yiGV6Av8ygbOO41/2N//qJALrswYkrshLQjEaI=; b=lEgZZeZz3dHdOoJtt5d8VNuVhl8oh1GerNwL6riHZgnlCETtA26dy9skXtG8r4+LZd DoASVZ0kDL/fuv4jLYePgszynxrI+WQs5E0FTiWaU4D6mQvcGDszmyowaArlV+h11YVN 3hjp7/cObCFoR+6rs+/fCl8S6fL85bTSWEVF+P32lm2sd11oyQo7nAqx0CNNuZx0noOa YR4VEfJ1aic9wQJxsaWGUF8EZbBdgpDhGgqbImoHam6O0uu0lxCi62UG9tHVwFofrnnf H8D/gorWe6W2zjrEcgLuxKrOSXbS01dIZCJT7QXLCu4jZKnE5bl2pFVJ/6GXpEdOGXCa xSZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=83qY1yiGV6Av8ygbOO41/2N//qJALrswYkrshLQjEaI=; b=Iy3MMhia2GeKBIcxAE9+ZkINoSo/OixVmGLaoupcisdrhodKZBMMIEn/3DjSBX0qw0 WT6wyltGjaeMFcsCSTRqpgMzU+fo8aOsssBEaYdhehg0N+dbqS0hxNUEloeAs1nks7Cd 4C4UxogV9eqDIYVlvOkVwU2ruYg1zbOioFe0W7mmkLl4npIVNu7vookptvwJFTo6cl0M keBBtQ9q1d7L1CIJ70hPSbyQGKpXB47RdaPYeg0P1CmyDLIWQxPDjeHNkF3tbWfKv2X7 S2yL25IkOViGlIUsQNzIBap4APK11fNzXlIZcXX29otTWHMrC2UrRlMKuIz2+Rdjnrvw Esug== X-Gm-Message-State: AOAM532en2NNcMxW/HD1JLK0VoKryolrKeV6UGJLkdb43z5fHhd09s2s Wc0XzPkGu1oVj7RmG3xY8hjg9TNo3rpIDA== X-Google-Smtp-Source: ABdhPJwqhfIDFP068+V2W1ChVQO5SsJKLf5jPoSErAlzXLd7vD0uTnfmOQfhNU/zO9ptzsOqTDGIyA== X-Received: by 2002:a63:fe0a:: with SMTP id p10mr1391421pgh.255.1597995657124; Fri, 21 Aug 2020 00:40:57 -0700 (PDT) Received: from exodia.tfbnw.net ([2620:10d:c090:400::5:f2a4]) by smtp.gmail.com with ESMTPSA id jb1sm1080875pjb.9.2020.08.21.00.40.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 00:40:55 -0700 (PDT) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org Subject: [PATCH 11/11] btrfs-progs: receive: add tests for basic encoded_write send/receive Date: Fri, 21 Aug 2020 00:40:10 -0700 Message-Id: X-Mailer: git-send-email 2.28.0 In-Reply-To: References: MIME-Version: 1.0 Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov Adapt the existing send/receive tests by passing '-o --force-compress' to the mount commands in a new test. After writing a few files in the various compression formats, send/receive them with and without --force-decompress to test both the encoded_write path and the fallback to decode+write. Signed-off-by: Boris Burkov --- .../040-receive-write-encoded/test.sh | 114 ++++++++++++++++++ 1 file changed, 114 insertions(+) create mode 100755 tests/misc-tests/040-receive-write-encoded/test.sh diff --git a/tests/misc-tests/040-receive-write-encoded/test.sh b/tests/misc-tests/040-receive-write-encoded/test.sh new file mode 100755 index 00000000..4df6ccd6 --- /dev/null +++ b/tests/misc-tests/040-receive-write-encoded/test.sh @@ -0,0 +1,114 @@ +#!/bin/bash +# +# test that we can send and receive encoded writes for three modes of +# transparent compression: zlib, lzo, and zstd. + +source "$TEST_TOP/common" + +check_prereq mkfs.btrfs +check_prereq btrfs + +setup_root_helper +prepare_test_dev + +here=`pwd` + +# assumes the filesystem exists, and does mount, write, snapshot, send, unmount +# for the specified encoding option +send_one() { + local str + local subv + local snap + + algorithm="$1" + shift + str="$1" + shift + + subv="subv-$algorithm" + snap="snap-$algorithm" + + run_check_mount_test_dev "-o" "compress-force=$algorithm" + cd "$TEST_MNT" || _fail "cannot chdir to TEST_MNT" + + run_check $SUDO_HELPER "$TOP/btrfs" subvolume create "$subv" + run_check $SUDO_HELPER dd if=/dev/zero of="$subv/file1" bs=1M count=1 + run_check $SUDO_HELPER dd if=/dev/zero of="$subv/file2" bs=500K count=1 + run_check $SUDO_HELPER "$TOP/btrfs" subvolume snapshot -r "$subv" "$snap" + run_check $SUDO_HELPER "$TOP/btrfs" send -f "$str" "$snap" "$@" + + cd "$here" || _fail "cannot chdir back to test directory" + run_check_umount_test_dev +} + +receive_one() { + local str + str="$1" + shift + + run_check_mkfs_test_dev + run_check_mount_test_dev + run_check $SUDO_HELPER "$TOP/btrfs" receive "$@" -v -f "$str" "$TEST_MNT" + run_check_umount_test_dev + run_check rm -f -- "$str" +} + +test_one_write_encoded() { + local str + local algorithm + algorithm="$1" + shift + str="$here/stream-$algorithm.stream" + + run_check_mkfs_test_dev + send_one "$algorithm" "$str" --compressed + receive_one "$str" "$@" +} + +test_one_stream_v1() { + local str + local algorithm + algorithm="$1" + shift + str="$here/stream-$algorithm.stream" + + run_check_mkfs_test_dev + send_one "$algorithm" "$str" --stream-version 1 + receive_one "$str" "$@" +} + +test_mix_write_encoded() { + local strzlib + local strlzo + local strzstd + strzlib="$here/stream-zlib.stream" + strlzo="$here/stream-lzo.stream" + strzstd="$here/stream-zstd.stream" + + run_check_mkfs_test_dev + + send_one "zlib" "$strzlib" --compressed + send_one "lzo" "$strlzo" --compressed + send_one "zstd" "$strzstd" --compressed + + receive_one "$strzlib" + receive_one "$strlzo" + receive_one "$strzstd" +} + +test_one_write_encoded "zlib" +test_one_write_encoded "lzo" +test_one_write_encoded "zstd" + +# with decompression forced +test_one_write_encoded "zlib" "--force-decompress" +test_one_write_encoded "lzo" "--force-decompress" +test_one_write_encoded "zstd" "--force-decompress" + +# send stream v1 +test_one_stream_v1 "zlib" +test_one_stream_v1 "lzo" +test_one_stream_v1 "zstd" + +# files use a mix of compression algorithms +test_mix_write_encoded