From patchwork Fri Jan 22 20:47:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 12040519 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 80219C433E0 for ; Fri, 22 Jan 2021 20:54:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4708023B08 for ; Fri, 22 Jan 2021 20:54:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730742AbhAVUyB (ORCPT ); Fri, 22 Jan 2021 15:54:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49776 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730938AbhAVUwW (ORCPT ); Fri, 22 Jan 2021 15:52:22 -0500 Received: from mail-pg1-x52d.google.com (mail-pg1-x52d.google.com [IPv6:2607:f8b0:4864:20::52d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1A73AC06174A for ; Fri, 22 Jan 2021 12:48:21 -0800 (PST) Received: by mail-pg1-x52d.google.com with SMTP id i5so4637994pgo.1 for ; Fri, 22 Jan 2021 12:48:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=pEymZbT2e7XwdrhZLYzjzmD3uYJNqriWjFNl14kDrfE=; b=QovdwtY9DH1sp3t9+xGragFOWmOETQFOAEumTa4hdzg5zMZ/PMClCGWkvxWjinpYJu 5tmr1vdLH+Ao3IuONpG7qmbK86QOg2efPp2NHp20xtrsyN6uy/2mOaXUhcB0lF8RD8WA fSsPAYj6vQTNsPeykh55brqE2JOL6qGq+WcTTbj2yvbpKQut9CdYq/cGqLr2axEyrTBf R8ZujjSYVZnhsNIAQJsVBgw3SfUDRLCV0jKq8QCUCDeGYXo/HfuiRYPRUfr48k6bUgDr Ahpo1kquyg4BMlaptC35hrKnS3N4tGAzjzlxgoTwO5cNWh/89MbIKZZqhMBTRbnyp3SF P67g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pEymZbT2e7XwdrhZLYzjzmD3uYJNqriWjFNl14kDrfE=; b=mbsnuqW/KPnOnQB2Y01jxtaJpfP9SGLWyd36QepX6SljonTGe7eHFQNhiHgpRuxaHd XnqEv2u4TaxzcZVP3LVI098aIf+ZarnZsi7YXdaYGjHJPLX/JrQC1T3Nza7xD9pzChpH KWHCxiwYTnvOeLnqNqCyy+bhPqylKW+twSMYyBd0SuXY2WIzrtnXJjo8+wD403IDxUp5 G7+lh6fzV+3n0wG/hs8wCRoAvJkUkFSBJ+LFiEflSYvvOe6UsNjq2R1t/sx9UR/Y+10A yYS13QExQpZp0CG3fBO3s9bS/IvFUv7YMSkUM3LGbeUzkiq9K4dzK57nUH6OHXcks0jK pByg== X-Gm-Message-State: AOAM533hM2w6/zQS7p2VMVYEc9knWe5ITbIqDa7xFAW/DJ4B6c7vfYcB CkWLzEM0rpIlmTJrNOKjQGc+WOLrpyBvFw== X-Google-Smtp-Source: ABdhPJyKHxiX7QYhwimC5BveXPky/DHmXbdaeFuT0xdPTHcZ54suKNB5J5EgxU/jE7A+wm52K5c+dQ== X-Received: by 2002:a65:5241:: with SMTP id q1mr6335206pgp.143.1611348499956; Fri, 22 Jan 2021 12:48:19 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:ea88]) by smtp.gmail.com with ESMTPSA id y16sm9865617pfb.83.2021.01.22.12.48.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 Jan 2021 12:48:18 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v3 01/11] btrfs-progs: receive: support v2 send stream larger tlv_len Date: Fri, 22 Jan 2021 12:47:44 -0800 Message-Id: <2792c5c7a2d5cb9cf7c7d9c8f3ef505c460c0072.1611347859.git.osandov@osandov.com> X-Mailer: git-send-email 2.30.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov An encoded extent can be up to 128K in length, which exceeds the largest value expressible by the current send stream format's 16 bit tlv_len field. Since encoded writes cannot be split into multiple writes by btrfs send, the send stream format must change to accommodate encoded writes. Supporting this changed format requires retooling how we store the commands we have processed. Since we can no longer use btrfs_tlv_header to describe every attribute, we define a new struct btrfs_send_attribute which has a 32 bit length field, and use that to store the attribute information needed for receive processing. This is transparent to users of the various TLV_GET macros. Signed-off-by: Boris Burkov --- common/send-stream.c | 34 +++++++++++++++++++++++++--------- 1 file changed, 25 insertions(+), 9 deletions(-) diff --git a/common/send-stream.c b/common/send-stream.c index a0c52f79..cd5aa311 100644 --- a/common/send-stream.c +++ b/common/send-stream.c @@ -24,13 +24,23 @@ #include "crypto/crc32c.h" #include "common/utils.h" +struct btrfs_send_attribute { + u16 tlv_type; + /* + * Note: in btrfs_tlv_header, this is __le16, but we need 32 bits for + * attributes with file data as of version 2 of the send stream format + */ + u32 tlv_len; + char *data; +}; + struct btrfs_send_stream { char read_buf[BTRFS_SEND_BUF_SIZE]; int fd; int cmd; struct btrfs_cmd_header *cmd_hdr; - struct btrfs_tlv_header *cmd_attrs[BTRFS_SEND_A_MAX + 1]; + struct btrfs_send_attribute cmd_attrs[BTRFS_SEND_A_MAX + 1]; u32 version; /* @@ -152,6 +162,7 @@ static int read_cmd(struct btrfs_send_stream *sctx) struct btrfs_tlv_header *tlv_hdr; u16 tlv_type; u16 tlv_len; + struct btrfs_send_attribute *send_attr; tlv_hdr = (struct btrfs_tlv_header *)data; tlv_type = le16_to_cpu(tlv_hdr->tlv_type); @@ -164,10 +175,15 @@ static int read_cmd(struct btrfs_send_stream *sctx) goto out; } - sctx->cmd_attrs[tlv_type] = tlv_hdr; + send_attr = &sctx->cmd_attrs[tlv_type]; + send_attr->tlv_type = tlv_type; + send_attr->tlv_len = tlv_len; + pos += sizeof(*tlv_hdr); + data += sizeof(*tlv_hdr); - data += sizeof(*tlv_hdr) + tlv_len; - pos += sizeof(*tlv_hdr) + tlv_len; + send_attr->data = data; + pos += send_attr->tlv_len; + data += send_attr->tlv_len; } sctx->cmd = cmd; @@ -180,7 +196,7 @@ out: static int tlv_get(struct btrfs_send_stream *sctx, int attr, void **data, int *len) { int ret; - struct btrfs_tlv_header *hdr; + struct btrfs_send_attribute *send_attr; if (attr <= 0 || attr > BTRFS_SEND_A_MAX) { error("invalid attribute requested, attr = %d", attr); @@ -188,15 +204,15 @@ static int tlv_get(struct btrfs_send_stream *sctx, int attr, void **data, int *l goto out; } - hdr = sctx->cmd_attrs[attr]; - if (!hdr) { + send_attr = &sctx->cmd_attrs[attr]; + if (!send_attr->data) { error("attribute %d requested but not present", attr); ret = -ENOENT; goto out; } - *len = le16_to_cpu(hdr->tlv_len); - *data = hdr + 1; + *len = send_attr->tlv_len; + *data = send_attr->data; ret = 0; From patchwork Fri Jan 22 20:47:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 12040525 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96F84C433DB for ; Fri, 22 Jan 2021 20:54:52 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6B2AE23B06 for ; Fri, 22 Jan 2021 20:54:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730320AbhAVUyn (ORCPT ); Fri, 22 Jan 2021 15:54:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49786 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730944AbhAVUwX (ORCPT ); Fri, 22 Jan 2021 15:52:23 -0500 Received: from mail-pg1-x534.google.com (mail-pg1-x534.google.com [IPv6:2607:f8b0:4864:20::534]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C1E90C0698C7 for ; Fri, 22 Jan 2021 12:48:23 -0800 (PST) Received: by mail-pg1-x534.google.com with SMTP id 15so4606694pgx.7 for ; Fri, 22 Jan 2021 12:48:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yuS+bqDRqimJQtw+CUfBlHJuFVwwMnHAqeTtllhBo5o=; b=F05HqoGWac6z+Vge5IUSRo/6x7Z5kNzAEMl13kxlj1HFCsDymJpLiS1dJVOj/4ZVyS Ym5h+7gvpIu1uwrvzL3bqF0CXsnfpP0Bm3+5SyIP/n9HcqznrofQty8mia38pkzgyltL wv3QU/6Cm9xKRs4n1y8FR/rPHMZn2RRYpo0qhXNIA47H2A1v8ETYsjvtaalqZtgafmvm k77oFG6XdibFdhUKIAYIAW2LhK8zIl6lfsWJjToTtnDM1UMOYokRhuCExnhcSSPTN4pv tFWq0luCAHjzwD99E9+02lgnXLMpxZYWceHTKAkp/PZRhEbw/0oI8BF+BSjxzO8YQ37C 4Ctw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yuS+bqDRqimJQtw+CUfBlHJuFVwwMnHAqeTtllhBo5o=; b=BVnPFmOlkTbV/REPym8mAaXD2qYvY6YFtT3xptwhTVuDorspMmvaH2TqsNrAeYu93f tMEUDD+LFTQNxGIRw4Ts7W0RpEzI0q6YPeJlMoLGPm+SoCQyPAiyt7tBztOH/I+qvhbF fNw0rvoxoV2P0RH2nT6PCi7P5TEWik+V/AWNxGjTyT6xzYo5RugNNNahu/AVLRHgLOzm pPct7LwOmYtu4nfR/HV0kIxOCPkgBE5BGZxVgbhBnFLECGma8tV6J7jFpH/WYvAxjbSG IHJfskDEK7yxgkoTLoekT64r8BEHlmx0fYyZkdVZbcnA7CyRsNTpG2GQVERiRzes39ZK 8Qgg== X-Gm-Message-State: AOAM530+yCCJUChHiCFZQSww5A0rlN6cJPAHz18lOy0OJxOwHN5bYuwO BqC7enHTsCPAO2BSeHjITfgKnIWffHpA/Q== X-Google-Smtp-Source: ABdhPJzxNSBEXm8T/NSdQpk2RIiNIamZcCB8nMx4SGqTY+bBicDYXpfL4gXObpa2JPuOXhCGhyr2wA== X-Received: by 2002:a62:2bd4:0:b029:1ae:4d9f:60da with SMTP id r203-20020a622bd40000b02901ae4d9f60damr6793220pfr.20.1611348502594; Fri, 22 Jan 2021 12:48:22 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:ea88]) by smtp.gmail.com with ESMTPSA id y16sm9865617pfb.83.2021.01.22.12.48.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 Jan 2021 12:48:21 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v3 02/11] btrfs-progs: receive: dynamically allocate sctx->read_buf Date: Fri, 22 Jan 2021 12:47:45 -0800 Message-Id: <1bb29f037059aa09b4d61349f59d862cbfb44a1b.1611347859.git.osandov@osandov.com> X-Mailer: git-send-email 2.30.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov In send stream v2, write commands can now be an arbitrary size. For that reason, we can no longer allocate a fixed array in sctx for read_cmd. Instead, read_cmd dynamically allocates sctx->read_buf. To avoid needless reallocations, we reuse read_buf between read_cmd calls by also keeping track of the size of the allocated buffer in sctx->read_buf_sz. We do the first allocation of the old default size at the start of processing the stream, and we only reallocate if we encounter a command that needs a larger buffer. Signed-off-by: Boris Burkov --- common/send-stream.c | 55 ++++++++++++++++++++++++++++---------------- send.h | 2 +- 2 files changed, 36 insertions(+), 21 deletions(-) diff --git a/common/send-stream.c b/common/send-stream.c index cd5aa311..3d3585c3 100644 --- a/common/send-stream.c +++ b/common/send-stream.c @@ -35,11 +35,11 @@ struct btrfs_send_attribute { }; struct btrfs_send_stream { - char read_buf[BTRFS_SEND_BUF_SIZE]; + char *read_buf; + size_t read_buf_sz; int fd; int cmd; - struct btrfs_cmd_header *cmd_hdr; struct btrfs_send_attribute cmd_attrs[BTRFS_SEND_A_MAX + 1]; u32 version; @@ -111,11 +111,12 @@ static int read_cmd(struct btrfs_send_stream *sctx) u32 pos; u32 crc; u32 crc2; + struct btrfs_cmd_header *cmd_hdr; + size_t buf_len; memset(sctx->cmd_attrs, 0, sizeof(sctx->cmd_attrs)); - ASSERT(sizeof(*sctx->cmd_hdr) <= sizeof(sctx->read_buf)); - ret = read_buf(sctx, sctx->read_buf, sizeof(*sctx->cmd_hdr)); + ret = read_buf(sctx, sctx->read_buf, sizeof(*cmd_hdr)); if (ret < 0) goto out; if (ret) { @@ -124,18 +125,22 @@ static int read_cmd(struct btrfs_send_stream *sctx) goto out; } - sctx->cmd_hdr = (struct btrfs_cmd_header *)sctx->read_buf; - cmd = le16_to_cpu(sctx->cmd_hdr->cmd); - cmd_len = le32_to_cpu(sctx->cmd_hdr->len); - - if (cmd_len + sizeof(*sctx->cmd_hdr) >= sizeof(sctx->read_buf)) { - ret = -EINVAL; - error("command length %u too big for buffer %zu", - cmd_len, sizeof(sctx->read_buf)); - goto out; + cmd_hdr = (struct btrfs_cmd_header *)sctx->read_buf; + cmd_len = le32_to_cpu(cmd_hdr->len); + cmd = le16_to_cpu(cmd_hdr->cmd); + buf_len = sizeof(*cmd_hdr) + cmd_len; + if (sctx->read_buf_sz < buf_len) { + sctx->read_buf = realloc(sctx->read_buf, buf_len); + if (!sctx->read_buf) { + ret = -ENOMEM; + error("failed to reallocate read buffer for cmd"); + goto out; + } + sctx->read_buf_sz = buf_len; + /* We need to reset cmd_hdr after realloc of sctx->read_buf */ + cmd_hdr = (struct btrfs_cmd_header *)sctx->read_buf; } - - data = sctx->read_buf + sizeof(*sctx->cmd_hdr); + data = sctx->read_buf + sizeof(*cmd_hdr); ret = read_buf(sctx, data, cmd_len); if (ret < 0) goto out; @@ -145,11 +150,12 @@ static int read_cmd(struct btrfs_send_stream *sctx) goto out; } - crc = le32_to_cpu(sctx->cmd_hdr->crc); - sctx->cmd_hdr->crc = 0; + crc = le32_to_cpu(cmd_hdr->crc); + /* in send, crc is computed with header crc = 0, replicate that */ + cmd_hdr->crc = 0; crc2 = crc32c(0, (unsigned char*)sctx->read_buf, - sizeof(*sctx->cmd_hdr) + cmd_len); + sizeof(*cmd_hdr) + cmd_len); if (crc != crc2) { ret = -EINVAL; @@ -524,19 +530,28 @@ int btrfs_read_and_process_send_stream(int fd, goto out; } + sctx.read_buf = malloc(BTRFS_SEND_BUF_SIZE_V1); + if (!sctx.read_buf) { + ret = -ENOMEM; + error("unable to allocate send stream read buffer"); + goto out; + } + sctx.read_buf_sz = BTRFS_SEND_BUF_SIZE_V1; + while (1) { ret = read_and_process_cmd(&sctx); if (ret < 0) { last_err = ret; errors++; if (max_errors > 0 && errors >= max_errors) - goto out; + break; } else if (ret > 0) { if (!honor_end_cmd) ret = 0; - goto out; + break; } } + free(sctx.read_buf); out: if (last_err && !ret) diff --git a/send.h b/send.h index 8dd865ec..228928a0 100644 --- a/send.h +++ b/send.h @@ -33,7 +33,7 @@ extern "C" { #define BTRFS_SEND_STREAM_MAGIC "btrfs-stream" #define BTRFS_SEND_STREAM_VERSION 1 -#define BTRFS_SEND_BUF_SIZE SZ_64K +#define BTRFS_SEND_BUF_SIZE_V1 SZ_64K #define BTRFS_SEND_READ_SIZE (1024 * 48) enum btrfs_tlv_type { From patchwork Fri Jan 22 20:47:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 12040531 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49A47C433E0 for ; Fri, 22 Jan 2021 20:55:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1800523B06 for ; Fri, 22 Jan 2021 20:55:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730363AbhAVUzV (ORCPT ); Fri, 22 Jan 2021 15:55:21 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49350 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730242AbhAVUxJ (ORCPT ); Fri, 22 Jan 2021 15:53:09 -0500 Received: from mail-pf1-x42c.google.com (mail-pf1-x42c.google.com [IPv6:2607:f8b0:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 398CAC0698CA for ; Fri, 22 Jan 2021 12:48:28 -0800 (PST) Received: by mail-pf1-x42c.google.com with SMTP id 11so4627676pfu.4 for ; Fri, 22 Jan 2021 12:48:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Bt0vf2xnnq+L7u6gm6lH9LhabqXbnPeiGL7bkGtie2s=; b=gAT0Ly07k45VtTc+4CiL8nznhXVDFcaWwBJQFMc9cAO1mkwnFVOrZhYBQNcsu0/etw L5KjT8wznAna7OKfQgn04+MPFD9KAZuAppiCWcGJyAhmJGV6HqalreC1ZHkI9X5tpC5N Q2uiKuCrVJGkgZqwXwwFqMceCeMp+f9onIt6LIe7VK8blQ/EjiEpduaI8tUq/0ebi2il nVgKPhFxIZlCWN0A1L87ISOtD03GkR4GiOyVdfgKUgr2NAVPNt4w3hFF8G5wYHgY7zOT dPM8OPEWUg6DgZawSX72JNpATaQ2mB9WOvEtU5hyJs6qZvFnqeDoCiOPklrnQgIt3+TJ BNlw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Bt0vf2xnnq+L7u6gm6lH9LhabqXbnPeiGL7bkGtie2s=; b=VjKwZZff1lGUrTHA6pbKBAYVBBY7LUxAFU9w5i0lyD2xlXdClgFv3A3AwVRayIHC9X wzcIytSQ6ShbLm/npCOI4iWCdT44OqALVQJoOg7KfCmAYMp+XDKuo+nKpVaPckz/N7qt OqtJFa2oFQvJlx8+wVFlAacoBl+FvIkpRaIFWkEGt0xi5kxga80kuIhTxo/uFCuPT5co l4u1jcY4NlyAv0utwfmNTrjqQ8xz86eWvNLVpA3LJVxe9uJLEHMv3uFlgMcjMwIonpEf XUAxvUpC3S1NGm05rAbuTQRhrVoyQj7aE5WdMK++Reh2cnpumWtgr8r01eUp8oXsRMty yPUA== X-Gm-Message-State: AOAM532kKTV4wY964XpXXxXglkwBS7lqAoMofxthDPkmySSNqRgCZ3On 4+CiNBeUyKlD9A/lxRKjp2Mq1Q0hCqm+ow== X-Google-Smtp-Source: ABdhPJwXj/ytmwn93g/hryffgdZFG49b4q2h8dVPQMfyg+S3ztuJy8v2hsgafy2t1dMXoON4TvFlEw== X-Received: by 2002:a62:5383:0:b029:1b6:20bf:e5a1 with SMTP id h125-20020a6253830000b02901b620bfe5a1mr6860974pfb.58.1611348507188; Fri, 22 Jan 2021 12:48:27 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:ea88]) by smtp.gmail.com with ESMTPSA id y16sm9865617pfb.83.2021.01.22.12.48.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 Jan 2021 12:48:25 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v3 03/11] btrfs-progs: receive: support v2 send stream DATA tlv format Date: Fri, 22 Jan 2021 12:47:47 -0800 Message-Id: <58df81982267169d549b982091a4ccb0b3ac4c5d.1611347859.git.osandov@osandov.com> X-Mailer: git-send-email 2.30.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov The new format privileges the BTRFS_SEND_A_DATA attribute by guaranteeing it will always be the last attribute in any command that needs it, and by implicitly encoding the data length as the difference between the total command length in the command header and the sizes of the rest of the attributes (and of course the tlv_type identifying the DATA attribute). To parse the new stream, we must read the tlv_type and if it is not DATA, we proceed normally, but if it is DATA, we don't parse a tlv_len but simply compute the length. In addition, we add some bounds checking when parsing each chunk of data, as well as for the tlv_len itself. Signed-off-by: Boris Burkov --- common/send-stream.c | 36 ++++++++++++++++++++++++++---------- 1 file changed, 26 insertions(+), 10 deletions(-) diff --git a/common/send-stream.c b/common/send-stream.c index 3d3585c3..4d819185 100644 --- a/common/send-stream.c +++ b/common/send-stream.c @@ -165,28 +165,44 @@ static int read_cmd(struct btrfs_send_stream *sctx) pos = 0; while (pos < cmd_len) { - struct btrfs_tlv_header *tlv_hdr; u16 tlv_type; - u16 tlv_len; struct btrfs_send_attribute *send_attr; - tlv_hdr = (struct btrfs_tlv_header *)data; - tlv_type = le16_to_cpu(tlv_hdr->tlv_type); - tlv_len = le16_to_cpu(tlv_hdr->tlv_len); + if (cmd_len - pos < sizeof(__le16)) { + error("send stream is truncated"); + ret = -EINVAL; + goto out; + } + tlv_type = le16_to_cpu(*(__le16 *)data); if (tlv_type == 0 || tlv_type > BTRFS_SEND_A_MAX) { - error("invalid tlv in cmd tlv_type = %hu, tlv_len = %hu", - tlv_type, tlv_len); + error("invalid tlv in cmd tlv_type = %hu", tlv_type); ret = -EINVAL; goto out; } send_attr = &sctx->cmd_attrs[tlv_type]; send_attr->tlv_type = tlv_type; - send_attr->tlv_len = tlv_len; - pos += sizeof(*tlv_hdr); - data += sizeof(*tlv_hdr); + pos += sizeof(tlv_type); + data += sizeof(tlv_type); + if (sctx->version == 2 && tlv_type == BTRFS_SEND_A_DATA) { + send_attr->tlv_len = cmd_len - pos; + } else { + if (cmd_len - pos < sizeof(__le16)) { + error("send stream is truncated"); + ret = -EINVAL; + goto out; + } + send_attr->tlv_len = le16_to_cpu(*(__le16 *)data); + pos += sizeof(__le16); + data += sizeof(__le16); + } + if (cmd_len - pos < send_attr->tlv_len) { + error("send stream is truncated"); + ret = -EINVAL; + goto out; + } send_attr->data = data; pos += send_attr->tlv_len; data += send_attr->tlv_len; From patchwork Fri Jan 22 20:47:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 12040535 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 737D6C433E0 for ; Fri, 22 Jan 2021 20:56:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4167E23B08 for ; Fri, 22 Jan 2021 20:56:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728914AbhAVU4H (ORCPT ); Fri, 22 Jan 2021 15:56:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49740 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731033AbhAVUx3 (ORCPT ); Fri, 22 Jan 2021 15:53:29 -0500 Received: from mail-pl1-x631.google.com (mail-pl1-x631.google.com [IPv6:2607:f8b0:4864:20::631]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 96A11C0698CE for ; Fri, 22 Jan 2021 12:48:33 -0800 (PST) Received: by mail-pl1-x631.google.com with SMTP id t6so3996446plq.1 for ; Fri, 22 Jan 2021 12:48:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Qxtundi6ingv4jVYkWsdsz+OtIfocQW8nsRe9gYEN1w=; b=k4a2bCJwZ+LnkBLgdQid4Gy/NWng3pu59K+wfUnIo6Shccw50rSL0heuX6MYJ3T93v +iY7Ak/N6/iWEy1pBP75yZ/PvkOIdMx+Yyp1qLGQqleD2p9BSzJnfV9PHdlrXmg/e8RV +h0Z/AxZwNW7qoPLLC1DtHdJyz0BJiXl/jgrAv9mhb9JFDWPHKekFL6jMkk3CGZRAk53 DY4UtjB1yL44R9AUGqRK4c+3jMPsQaKdDXGlaQ0c/F55phZIGyZQf7UoTMoJT1tz+I+e bQr8D2ciNs4g0qYwhlSXJnTGBGoK2u1jQSjUw2TX7S0VMFIqnK9aqp502uDjlvxYW0F7 eMKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Qxtundi6ingv4jVYkWsdsz+OtIfocQW8nsRe9gYEN1w=; b=rWVGJrdLRqKkJyIcr35E8xSpvyx2BfS4CEdt0Vz7gsut9qPRh856LiQtN0DPeHcx+T Ug5xVH8CfZ+Hu+P4XsrP4xjd+A+Zrq+UnH+J/XVAK+Q9crOrBqKhy1k9umTlI0SyuuiD DYdl2xxpZmll+QE/vBaIuDKuct8545IAXZfAQ7qj4FXUlbHQodEb8EREIeUTqKBkLreY EZykA7wjNjzpl3cLGfvBWth7GicBk6hxiWV0+l4qI5IHxjM0kebztwbcKHk+YXyGAlD5 vMeoUdsqjDfvF4YxSInHmedD4SBwzDSXb0qOtBU8vrYXysVdU/zGW6Hq/lYtwwYUKS2i DOmQ== X-Gm-Message-State: AOAM531KGJWGiVd1cd96gm04mLbEmM/gvTzwhrz7Lf1IrDZIZb8hfjsN n9/tehYM5Qvrk7N1FhcvNNVHg2RT67X2/Q== X-Google-Smtp-Source: ABdhPJwOAAAT0hJlnuakhGyj8+//uqcebWDhOZD8XEywxul/ux7XHdz8ySWtu+jwpXsLCyYJaZTA3A== X-Received: by 2002:a17:90b:4acd:: with SMTP id mh13mr7526167pjb.229.1611348512464; Fri, 22 Jan 2021 12:48:32 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:ea88]) by smtp.gmail.com with ESMTPSA id y16sm9865617pfb.83.2021.01.22.12.48.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 Jan 2021 12:48:30 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v3 04/11] btrfs-progs: receive: add send stream v2 cmds and attrs to send.h Date: Fri, 22 Jan 2021 12:47:49 -0800 Message-Id: X-Mailer: git-send-email 2.30.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov Send stream v2 adds three commands and several attributes associated to those commands. Before we implement processing them, add all the commands and attributes. This avoids leaving the enums in an intermediate state that doesn't correspond to any version of send stream. Signed-off-by: Boris Burkov --- send.h | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/send.h b/send.h index 228928a0..3c47e0c7 100644 --- a/send.h +++ b/send.h @@ -98,6 +98,11 @@ enum btrfs_send_cmd { BTRFS_SEND_C_END, BTRFS_SEND_C_UPDATE_EXTENT, + + BTRFS_SEND_C_FALLOCATE, + BTRFS_SEND_C_SETFLAGS, + BTRFS_SEND_C_ENCODED_WRITE, + __BTRFS_SEND_C_MAX, }; #define BTRFS_SEND_C_MAX (__BTRFS_SEND_C_MAX - 1) @@ -136,6 +141,16 @@ enum { BTRFS_SEND_A_CLONE_OFFSET, BTRFS_SEND_A_CLONE_LEN, + BTRFS_SEND_A_FALLOCATE_MODE, + + BTRFS_SEND_A_SETFLAGS_FLAGS, + + BTRFS_SEND_A_UNENCODED_FILE_LEN, + BTRFS_SEND_A_UNENCODED_LEN, + BTRFS_SEND_A_UNENCODED_OFFSET, + BTRFS_SEND_A_COMPRESSION, + BTRFS_SEND_A_ENCRYPTION, + __BTRFS_SEND_A_MAX, }; #define BTRFS_SEND_A_MAX (__BTRFS_SEND_A_MAX - 1) From patchwork Fri Jan 22 20:47:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 12040607 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41CABC433E0 for ; Fri, 22 Jan 2021 21:11:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 01ED823A7A for ; Fri, 22 Jan 2021 21:11:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730033AbhAVVLM (ORCPT ); Fri, 22 Jan 2021 16:11:12 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49756 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731057AbhAVUxe (ORCPT ); Fri, 22 Jan 2021 15:53:34 -0500 Received: from mail-pj1-x102c.google.com (mail-pj1-x102c.google.com [IPv6:2607:f8b0:4864:20::102c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 907F0C0698D4 for ; Fri, 22 Jan 2021 12:48:40 -0800 (PST) Received: by mail-pj1-x102c.google.com with SMTP id g15so4662716pjd.2 for ; Fri, 22 Jan 2021 12:48:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=iHS144DxLbBMpm965BQLT7zCwBiEuqsfigSMglTrs0I=; b=VfGysgsvrue0+3LQNRCUg+NcyLTA13X5zLrIvchmLAMCkBiPfUgnFSEvCoYc2xTsUD hK6Wfo1YDgwp7fRN+5davLhgmNJ7CZ4771kF/bWvDL9Rn0R+icDso+Jeqh2zw+MZJP+s CvOdP1Wat8pHJ5IcZlICIZkpViatpKV2Pz4RtNSGTBtBvEVnWRTrNViXE94PcZ1YEbem y/+dCaWWsXOlq29NJho6EJr6qH9PbEDtJKP8SFgCkp6/m1Ud3kL7Rz6Pjx3+GM+SX4jf YY+s266bOqAvu4jKPhiXlC5qMiRCSAcDd9AW771izjfoS6CZWIIC7R75smQQG9gt3yjs jyBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=iHS144DxLbBMpm965BQLT7zCwBiEuqsfigSMglTrs0I=; b=JsdGhoPyS3/AIPMumF3fEHleIkUwNji65QwgwreM41UDRu3Z3XEjGO0zxU/yzmgRTz z6kIlGr3L45g4zEsRw8lOboQq/DL/j4+DH5kc2DK0Rb2A91n2X848SdrUYZlbidq8rKY XGTAYdIGwj74flhiwfQm+4LcygmdMraQJdUQDH5Y40MLGyEt6mHuJVXFW5jdbSH2ph4h NUKEylZOFvqIrhbN03ZcXGSmr8W5Ayhd3uiKcm5MMOup4Od/7/VRqqJ/OZ5swzvNV3FG XALTL5n9gLB/6qXYLInvvA5qFY12W18tT7jBdJCm13WzBZcY1VqNiTcmbfNEElE9rK3Z Qv9Q== X-Gm-Message-State: AOAM533UPqOocDOc4BGrVYwF6RBM7w1s0Z6aL3lWXK1LNPGQ3DNLn2Gg vjeGO2GHKlIEPQ9BLKVZaDgMRkeAEDV/+g== X-Google-Smtp-Source: ABdhPJzFXbgDRARyE4owNTTt/BocEaIJ9gaa9aJT7xlJClUZzGX1KC6ZGd9vG0G4mhUGanlObS+bCA== X-Received: by 2002:a17:902:7481:b029:df:e6bf:7e53 with SMTP id h1-20020a1709027481b02900dfe6bf7e53mr2559651pll.80.1611348519376; Fri, 22 Jan 2021 12:48:39 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:ea88]) by smtp.gmail.com with ESMTPSA id y16sm9865617pfb.83.2021.01.22.12.48.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 Jan 2021 12:48:37 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v3 5/5] btrfs: send: enable support for stream v2 and compressed writes Date: Fri, 22 Jan 2021 12:47:52 -0800 Message-Id: X-Mailer: git-send-email 2.30.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Omar Sandoval Now that the new support is implemented, allow the ioctl to accept the flags and update the version in sysfs. Signed-off-by: Omar Sandoval --- fs/btrfs/send.c | 10 +++++++++- fs/btrfs/send.h | 2 +- include/uapi/linux/btrfs.h | 4 +++- 3 files changed, 13 insertions(+), 3 deletions(-) diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c index 7516eba701af..cb824d1271fa 100644 --- a/fs/btrfs/send.c +++ b/fs/btrfs/send.c @@ -671,7 +671,10 @@ static int send_header(struct send_ctx *sctx) struct btrfs_stream_header hdr; strcpy(hdr.magic, BTRFS_SEND_STREAM_MAGIC); - hdr.version = cpu_to_le32(BTRFS_SEND_STREAM_VERSION); + if (sctx->flags & BTRFS_SEND_FLAG_STREAM_V2) + hdr.version = cpu_to_le32(2); + else + hdr.version = cpu_to_le32(1); return write_buf(sctx->send_filp, &hdr, sizeof(hdr), &sctx->send_off); @@ -7446,6 +7449,11 @@ long btrfs_ioctl_send(struct file *mnt_file, struct btrfs_ioctl_send_args *arg) ret = -EINVAL; goto out; } + if ((arg->flags & BTRFS_SEND_FLAG_COMPRESSED) && + !(arg->flags & BTRFS_SEND_FLAG_STREAM_V2)) { + ret = -EINVAL; + goto out; + } sctx = kzalloc(sizeof(struct send_ctx), GFP_KERNEL); if (!sctx) { diff --git a/fs/btrfs/send.h b/fs/btrfs/send.h index 9f4f7b96b1eb..9c83e14a43b2 100644 --- a/fs/btrfs/send.h +++ b/fs/btrfs/send.h @@ -10,7 +10,7 @@ #include "ctree.h" #define BTRFS_SEND_STREAM_MAGIC "btrfs-stream" -#define BTRFS_SEND_STREAM_VERSION 1 +#define BTRFS_SEND_STREAM_VERSION 2 /* * In send stream v1, no command is larger than 64k. In send stream v2, no limit diff --git a/include/uapi/linux/btrfs.h b/include/uapi/linux/btrfs.h index 93aa0932234e..b12a9a1a106c 100644 --- a/include/uapi/linux/btrfs.h +++ b/include/uapi/linux/btrfs.h @@ -786,7 +786,9 @@ struct btrfs_ioctl_received_subvol_args { #define BTRFS_SEND_FLAG_MASK \ (BTRFS_SEND_FLAG_NO_FILE_DATA | \ BTRFS_SEND_FLAG_OMIT_STREAM_HEADER | \ - BTRFS_SEND_FLAG_OMIT_END_CMD) + BTRFS_SEND_FLAG_OMIT_END_CMD | \ + BTRFS_SEND_FLAG_STREAM_V2 | \ + BTRFS_SEND_FLAG_COMPRESSED) struct btrfs_ioctl_send_args { __s64 send_fd; /* in */ From patchwork Fri Jan 22 20:47:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 12040605 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8C39C433DB for ; Fri, 22 Jan 2021 21:10:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A4B5E23AF8 for ; Fri, 22 Jan 2021 21:10:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729568AbhAVVKm (ORCPT ); Fri, 22 Jan 2021 16:10:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49758 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731060AbhAVUxf (ORCPT ); Fri, 22 Jan 2021 15:53:35 -0500 Received: from mail-pj1-x1029.google.com (mail-pj1-x1029.google.com [IPv6:2607:f8b0:4864:20::1029]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 272E7C061788 for ; Fri, 22 Jan 2021 12:48:44 -0800 (PST) Received: by mail-pj1-x1029.google.com with SMTP id b5so4672073pjl.0 for ; Fri, 22 Jan 2021 12:48:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=56tOWB99LkykgzKBFUTrZ700+h0B1BRfT52yzernMAs=; b=MlahdaOofYCxNd+q9ijockBbTYoQhlS5YNrfsO262fp+feVCNBml/R372u5pQZMUr6 /rqnBt6XrDO/r3enNCcXqRRHB7A73hNUsdaWom/F1wdKQSFO8ppwALhn4F5B7z+AZVEe SMTbewpxcO9pS48v0JdFaOlzy0HK+gweSOHWxE9daiXmY822qOCo+S/HBBgJwtodCUkT ZBiQB+B6R9syd+8SOlsjFON1nfsMjPwYXr10RGX2HFGmC5kmmiaFEJBaDNaowJ7Yjp66 MwLTMPZfNkcQjdjOl/rBdeCW78L79tVchXR3L/igkJxaJGc8CIYLYvvj8JO8A2cGv/cO 4Kbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=56tOWB99LkykgzKBFUTrZ700+h0B1BRfT52yzernMAs=; b=eP+qGtQ4ALTz1lQIWDEN8KEiEV5ASwPa8XewnEg9bf/t8/4inxCxEWcnFtnuno2GiO EbID2KhcypGeDBupxB5mfPH3k3d0qldorg6tKYZreIVpBfF4Kp79L8tCvJaUu5fHqLg3 OUr7xFtb4kfXN08OXVzLcQios4iDeUCO6CMD7lf5U/zwwUYObNrSql1mlKQIPN5NYsgR 235sPu6FIcJTdIdnR6y6KDVj84rkR9vz871naPRRBhGw8ITJW35M1nsKtqO0PqP4cYZX fFP+KsvGWkPvdRuVPBxGgXs0Efk3yRGvPqinoOs6N4iMKKD15YcRW20cboIrCXSNlkaJ ldbQ== X-Gm-Message-State: AOAM530X56GhKMP1EiQ0qNkyLtx9f7Iszx6rnyRZ1PQztV1as2pqaT2x cG1sW7R6V8oglYr1UbwsAuRXxtJFfWjbEw== X-Google-Smtp-Source: ABdhPJwWxaug5vCAgeuttcp1+HA5Mu+wulWOXfv9gziCgiq3kjF4pywboEYlzvhGMptLkh/RGzYqqQ== X-Received: by 2002:a17:902:ce89:b029:df:c98f:430d with SMTP id f9-20020a170902ce89b02900dfc98f430dmr6887333plg.18.1611348522711; Fri, 22 Jan 2021 12:48:42 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:ea88]) by smtp.gmail.com with ESMTPSA id y16sm9865617pfb.83.2021.01.22.12.48.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 Jan 2021 12:48:41 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v3 06/11] btrfs-progs: receive: process encoded_write commands Date: Fri, 22 Jan 2021 12:47:53 -0800 Message-Id: X-Mailer: git-send-email 2.30.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov Add a new btrfs_send_op and support for both dumping and proper receive processing which does actual encoded writes. Encoded writes are only allowed on a file descriptor opened with an extra flag that allows encoded writes, so we also add support for this flag when opening or reusing a file for writing. Signed-off-by: Boris Burkov --- cmds/receive-dump.c | 16 ++++++- cmds/receive.c | 104 ++++++++++++++++++++++++++++++++++++++++--- common/send-stream.c | 22 +++++++++ common/send-stream.h | 4 ++ stubs.h | 46 +++++++++++++++++++ 5 files changed, 186 insertions(+), 6 deletions(-) diff --git a/cmds/receive-dump.c b/cmds/receive-dump.c index 648d9314..20ec2b70 100644 --- a/cmds/receive-dump.c +++ b/cmds/receive-dump.c @@ -316,6 +316,19 @@ static int print_update_extent(const char *path, u64 offset, u64 len, offset, len); } +static int print_encoded_write(const char *path, const void *data, u64 offset, + u64 len, u64 unencoded_file_len, + u64 unencoded_len, u64 unencoded_offset, + u32 compression, u32 encryption, void *user) +{ + return PRINT_DUMP(user, path, "encoded_write", + "offset=%llu len=%llu, unencoded_file_len=%llu, " + "unencoded_len=%llu, unencoded_offset=%llu, " + "compression=%u, encryption=%u", + offset, len, unencoded_file_len, unencoded_len, + unencoded_offset, compression, encryption); +} + struct btrfs_send_ops btrfs_print_send_ops = { .subvol = print_subvol, .snapshot = print_snapshot, @@ -337,5 +350,6 @@ struct btrfs_send_ops btrfs_print_send_ops = { .chmod = print_chmod, .chown = print_chown, .utimes = print_utimes, - .update_extent = print_update_extent + .update_extent = print_update_extent, + .encoded_write = print_encoded_write, }; diff --git a/cmds/receive.c b/cmds/receive.c index 2aaba3ff..76a9e958 100644 --- a/cmds/receive.c +++ b/cmds/receive.c @@ -30,12 +30,14 @@ #include #include #include +#include #include #include #include #include #include +#include #include #include @@ -52,6 +54,7 @@ #include "cmds/receive-dump.h" #include "common/help.h" #include "common/path-utils.h" +#include "stubs.h" struct btrfs_receive { @@ -60,6 +63,7 @@ struct btrfs_receive int write_fd; char write_path[PATH_MAX]; + int write_fd_allow_encoded; char *root_path; char *dest_dir_path; /* relative to root_path */ @@ -643,24 +647,65 @@ out: return ret; } -static int open_inode_for_write(struct btrfs_receive *rctx, const char *path) +static int set_write_fd_allow_encoded(struct btrfs_receive *rctx) +{ + int ret; + int flags; + + flags = fcntl(rctx->write_fd, F_GETFL); + if (flags < 0) { + ret = -errno; + error("failed to fetch old fd flags"); + goto close_fd; + } + ret = fcntl(rctx->write_fd, F_SETFL, flags | O_ALLOW_ENCODED); + if (ret < 0) { + ret = -errno; + error("failed to enable encoded writes"); + goto close_fd; + } + rctx->write_fd_allow_encoded = true; + ret = 0; + goto out; +close_fd: + close(rctx->write_fd); + rctx->write_fd = -1; + rctx->write_fd_allow_encoded = false; +out: + return ret; +} + +static int open_inode_for_write(struct btrfs_receive *rctx, const char *path, + bool allow_encoded) { int ret = 0; + int flags = O_RDWR; if (rctx->write_fd != -1) { - if (strcmp(rctx->write_path, path) == 0) + /* + * if the existing fd is for this path and the needed flags are + * satisfied, no need to open a new one + */ + if (strcmp(rctx->write_path, path) == 0) { + /* fixup the allow encoded flag, if necessary */ + if (allow_encoded && !rctx->write_fd_allow_encoded) + ret = set_write_fd_allow_encoded(rctx); goto out; + } close(rctx->write_fd); rctx->write_fd = -1; } - rctx->write_fd = open(path, O_RDWR); + if (allow_encoded) + flags |= O_ALLOW_ENCODED; + rctx->write_fd = open(path, flags); if (rctx->write_fd < 0) { ret = -errno; error("cannot open %s: %m", path); goto out; } strncpy_null(rctx->write_path, path); + rctx->write_fd_allow_encoded = allow_encoded; out: return ret; @@ -691,7 +736,7 @@ static int process_write(const char *path, const void *data, u64 offset, goto out; } - ret = open_inode_for_write(rctx, full_path); + ret = open_inode_for_write(rctx, full_path, false); if (ret < 0) goto out; @@ -734,7 +779,7 @@ static int process_clone(const char *path, u64 offset, u64 len, goto out; } - ret = open_inode_for_write(rctx, full_path); + ret = open_inode_for_write(rctx, full_path, false); if (ret < 0) goto out; @@ -1028,6 +1073,54 @@ static int process_update_extent(const char *path, u64 offset, u64 len, return 0; } +static int process_encoded_write(const char *path, const void *data, u64 offset, + u64 len, u64 unencoded_file_len, u64 unencoded_len, + u64 unencoded_offset, u32 compression, u32 encryption, void *user) +{ + int ret; + ssize_t w; + struct btrfs_receive *rctx = user; + char full_path[PATH_MAX]; + struct encoded_iov encoded = { + .len = unencoded_file_len, + .unencoded_len = unencoded_len, + .unencoded_offset = unencoded_offset, + .compression = compression, + .encryption = encryption, + }; + struct iovec iov[2] = { + { &encoded, sizeof(encoded) }, + { (char *)data, len } + }; + + if (encryption) { + error("encoded_write: encryption not supported"); + return -EOPNOTSUPP; + } + + ret = path_cat_out(full_path, rctx->full_subvol_path, path); + if (ret < 0) { + error("encoded_write: path invalid: %s", path); + return ret; + } + + ret = open_inode_for_write(rctx, full_path, true); + if (ret < 0) + return ret; + + /* + * NOTE: encoded writes guarantee no partial writes, so we don't need to + * handle that possibility. + */ + w = pwritev2(rctx->write_fd, iov, 2, offset, RWF_ENCODED); + if (w < 0) { + ret = -errno; + error("encoded_write: writing to %s failed: %m", path); + return ret; + } + return 0; +} + static struct btrfs_send_ops send_ops = { .subvol = process_subvol, .snapshot = process_snapshot, @@ -1050,6 +1143,7 @@ static struct btrfs_send_ops send_ops = { .chown = process_chown, .utimes = process_utimes, .update_extent = process_update_extent, + .encoded_write = process_encoded_write, }; static int do_receive(struct btrfs_receive *rctx, const char *tomnt, diff --git a/common/send-stream.c b/common/send-stream.c index 4d819185..044e101b 100644 --- a/common/send-stream.c +++ b/common/send-stream.c @@ -354,6 +354,8 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx) struct timespec mt; u8 uuid[BTRFS_UUID_SIZE]; u8 clone_uuid[BTRFS_UUID_SIZE]; + u32 compression; + u32 encryption; u64 tmp; u64 tmp2; u64 ctransid; @@ -362,6 +364,9 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx) u64 dev; u64 clone_offset; u64 offset; + u64 unencoded_file_len; + u64 unencoded_len; + u64 unencoded_offset; int len; int xattr_len; @@ -436,6 +441,23 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx) TLV_GET(sctx, BTRFS_SEND_A_DATA, &data, &len); ret = sctx->ops->write(path, data, offset, len, sctx->user); break; + case BTRFS_SEND_C_ENCODED_WRITE: + TLV_GET_STRING(sctx, BTRFS_SEND_A_PATH, &path); + TLV_GET_U64(sctx, BTRFS_SEND_A_FILE_OFFSET, &offset); + TLV_GET_U64(sctx, BTRFS_SEND_A_UNENCODED_FILE_LEN, + &unencoded_file_len); + TLV_GET_U64(sctx, BTRFS_SEND_A_UNENCODED_LEN, &unencoded_len); + TLV_GET_U64(sctx, BTRFS_SEND_A_UNENCODED_OFFSET, + &unencoded_offset); + TLV_GET_U32(sctx, BTRFS_SEND_A_COMPRESSION, &compression); + TLV_GET_U32(sctx, BTRFS_SEND_A_ENCRYPTION, &encryption); + TLV_GET(sctx, BTRFS_SEND_A_DATA, &data, &len); + ret = sctx->ops->encoded_write(path, data, offset, len, + unencoded_file_len, + unencoded_len, unencoded_offset, + compression, encryption, + sctx->user); + break; case BTRFS_SEND_C_CLONE: TLV_GET_STRING(sctx, BTRFS_SEND_A_PATH, &path); TLV_GET_U64(sctx, BTRFS_SEND_A_FILE_OFFSET, &offset); diff --git a/common/send-stream.h b/common/send-stream.h index 39901f86..607bc007 100644 --- a/common/send-stream.h +++ b/common/send-stream.h @@ -66,6 +66,10 @@ struct btrfs_send_ops { struct timespec *mt, struct timespec *ct, void *user); int (*update_extent)(const char *path, u64 offset, u64 len, void *user); + int (*encoded_write)(const char *path, const void *data, u64 offset, + u64 len, u64 unencoded_file_len, u64 unencoded_len, + u64 unencoded_offset, u32 compression, + u32 encryption, void *user); }; int btrfs_read_and_process_send_stream(int fd, diff --git a/stubs.h b/stubs.h index b39f8a69..69e7fe23 100644 --- a/stubs.h +++ b/stubs.h @@ -1,6 +1,8 @@ #ifndef _BTRFS_STUBS_H #define _BTRFS_STUBS_H +#include +#include #include struct iovec; @@ -8,4 +10,48 @@ struct iovec; ssize_t pwritev2(int fd, const struct iovec *iov, int iovcnt, off_t offset, int flags); +#ifndef O_ALLOW_ENCODED +#if defined(__alpha__) +#define O_ALLOW_ENCODED 0200000000 +#elif defined(__hppa__) +#define O_ALLOW_ENCODED 100000000 +#elif defined(__sparc__) +#define O_ALLOW_ENCODED 0x8000000 +#else +#define O_ALLOW_ENCODED 040000000 #endif +#endif + +#ifndef ENCODED_IOV_SIZE_VER0 + +#define ENCODED_IOV_COMPRESSION_NONE 0 +#define ENCODED_IOV_COMPRESSION_BTRFS_ZLIB 1 +#define ENCODED_IOV_COMPRESSION_BTRFS_ZSTD 2 +#define ENCODED_IOV_COMPRESSION_BTRFS_LZO_4K 3 +#define ENCODED_IOV_COMPRESSION_BTRFS_LZO_8K 4 +#define ENCODED_IOV_COMPRESSION_BTRFS_LZO_16K 5 +#define ENCODED_IOV_COMPRESSION_BTRFS_LZO_32K 6 +#define ENCODED_IOV_COMPRESSION_BTRFS_LZO_64K 7 +#define ENCODED_IOV_COMPRESSION_TYPES 8 + +#define ENCODED_IOV_ENCRYPTION_NONE 0 +#define ENCODED_IOV_ENCRYPTION_TYPES 1 + +struct encoded_iov { + __aligned_u64 len; + __aligned_u64 unencoded_len; + __aligned_u64 unencoded_offset; + __u32 compression; + __u32 encryption; +}; + +#define ENCODED_IOV_SIZE_VER0 32 + +#endif /* ENCODED_IOV_SIZE_VER0 */ + +#ifndef RWF_ENCODED +/* encoded (e.g., compressed and/or encrypted) IO */ +#define RWF_ENCODED ((__kernel_rwf_t)0x00000020) +#endif + +#endif /* _BTRFS_STUBS_H */ From patchwork Fri Jan 22 20:47:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 12040603 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C07AC433E6 for ; Fri, 22 Jan 2021 21:09:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 17F2523B55 for ; Fri, 22 Jan 2021 21:09:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730560AbhAVVJJ (ORCPT ); Fri, 22 Jan 2021 16:09:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49776 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731087AbhAVUxi (ORCPT ); Fri, 22 Jan 2021 15:53:38 -0500 Received: from mail-pg1-x533.google.com (mail-pg1-x533.google.com [IPv6:2607:f8b0:4864:20::533]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 115C6C0698D5 for ; Fri, 22 Jan 2021 12:48:48 -0800 (PST) Received: by mail-pg1-x533.google.com with SMTP id n10so4601221pgl.10 for ; Fri, 22 Jan 2021 12:48:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=duTdFKWtosjJC4zNK7cu52L9NF104GoA+mBHnbVJDPo=; b=bqvlokC0DL93XPUqQQj1af0ykxBreA4TDTlNarEG/EiIkEKRwGxOkilQrZbQXtOEpX 1d7ln3a6/3+1RlT+8Uat15JeQZjFK3WpGf/s7zmgbY0H/Xz3P5TmUpvBTWUqepF6X/5d Fc1kH4VRL8DB3TN1t9zIh7By0Z0I4KxY0rooowyjd9fkJALS38wcGB1ZCCtzSLcCpbvD K+xgDMM9PDlp6KJXBOqlAgIRl5g7+wMoQs9G6UiTJrqEOVFAmPU9Ds/tAL543Lnny33Y hRoqjRdoMU8z4Fgqkzo8xu7nkL4D/yfKEdydBEc+Oe0ZkDTyacp2Uy01UUfukQvvftkZ lH8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=duTdFKWtosjJC4zNK7cu52L9NF104GoA+mBHnbVJDPo=; b=kZeiP8PnR0TX7m7MXfzw/Q5y4EuRaUqA13NT3GsAYZdUVYGElfFvYr8KoB8lTa6t+b mhWY2lIdx0ArkJ0ijcNP+Hd5cNizAzWeiuHxfQJMg9A1uhuoYZSa70xw/HqyYdOkkchF JTy9dVhbTCfb5LnwV4vU8/zVAMEW0Xabczh3cSqefRUVKJ/75gxVp/HGFH3MS7Cuvc92 qO9yEnt7avNxAOaESaWpkc4/4UGfHhyJg0banc0bAaHAZpIdNSPcOLm2+LCfzmDH8GBL ol4EsEecqnwl6FXYpcXM5CBNEVCyIT8h+fm8uvddW5uSt0am1r3aD7HAWhaab5mtI5Lt M3gA== X-Gm-Message-State: AOAM5320KXhBR/3364yUiHmUqcIi6yNCwMDhL0Uu4MZV7iFDWKPCqQ0z Og+RVY9vcvWHuBlD1lQMtXV6gWf5KPRI9Q== X-Google-Smtp-Source: ABdhPJxD+lqjIXV7RMa/uxCRnALp+qHc3OYGP+3QqlbDDsYTRsnVY6D7ZqnYRYp3gFGh/6cFXlJ3Ng== X-Received: by 2002:a63:1152:: with SMTP id 18mr6514861pgr.268.1611348526728; Fri, 22 Jan 2021 12:48:46 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:ea88]) by smtp.gmail.com with ESMTPSA id y16sm9865617pfb.83.2021.01.22.12.48.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 Jan 2021 12:48:44 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v3 07/11] btrfs-progs: receive: encoded_write fallback to explicit decode and write Date: Fri, 22 Jan 2021 12:47:54 -0800 Message-Id: <235588b5bd7dbf8950ff0fd53af1386f35878078.1611347859.git.osandov@osandov.com> X-Mailer: git-send-email 2.30.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov An encoded_write can fail if the file system it is being applied to does not support encoded writes or if it can't find enough contiguous space to accommodate the encoded extent. In those cases, we can likely still process an encoded_write by explicitly decoding the data and doing a normal write. Add the necessary fallback path for decoding data compressed with zlib, lzo, or zstd. zlib and zstd have reusable decoding context data structures which we cache in the receive context so that we don't have to recreate them on every encoded_write. Finally, add a command line flag for force-decompress which causes receive to always use the fallback path rather than first attempting the encoded write. Signed-off-by: Boris Burkov --- Documentation/btrfs-receive.asciidoc | 4 + cmds/receive.c | 274 +++++++++++++++++++++++++-- 2 files changed, 265 insertions(+), 13 deletions(-) diff --git a/Documentation/btrfs-receive.asciidoc b/Documentation/btrfs-receive.asciidoc index e4c4d2c0..354a71dc 100644 --- a/Documentation/btrfs-receive.asciidoc +++ b/Documentation/btrfs-receive.asciidoc @@ -60,6 +60,10 @@ By default the mountpoint is searched in '/proc/self/mounts'. If '/proc' is not accessible, eg. in a chroot environment, use this option to tell us where this filesystem is mounted. +--force-decompress:: +if the stream contains compressed data (see '--compressed-data' in +`btrfs-send`(8)), always decompress it instead of writing it with encoded I/O. + --dump:: dump the stream metadata, one line per operation + diff --git a/cmds/receive.c b/cmds/receive.c index 76a9e958..27e5c003 100644 --- a/cmds/receive.c +++ b/cmds/receive.c @@ -41,6 +41,10 @@ #include #include +#include +#include +#include + #include "kernel-shared/ctree.h" #include "ioctl.h" #include "cmds/commands.h" @@ -82,6 +86,8 @@ struct btrfs_receive int honor_end_cmd; + int force_decompress; + /* * Buffer to store capabilities from security.capabilities xattr, * usually 20 bytes, but make same room for potentially larger @@ -89,6 +95,10 @@ struct btrfs_receive */ char cached_capabilities[64]; int cached_capabilities_len; + + /* Reuse stream objects for encoded_write decompression fallback */ + ZSTD_DStream *zstd_dstream; + z_stream *zlib_stream; }; static int finish_subvol(struct btrfs_receive *rctx) @@ -1073,9 +1083,222 @@ static int process_update_extent(const char *path, u64 offset, u64 len, return 0; } +static int decompress_zlib(struct btrfs_receive *rctx, const char *encoded_data, + u64 encoded_len, char *unencoded_data, + u64 unencoded_len) +{ + bool init = false; + int ret; + + if (!rctx->zlib_stream) { + init = true; + rctx->zlib_stream = malloc(sizeof(z_stream)); + if (!rctx->zlib_stream) { + error("failed to allocate zlib stream %m"); + return -ENOMEM; + } + } + rctx->zlib_stream->next_in = (void *)encoded_data; + rctx->zlib_stream->avail_in = encoded_len; + rctx->zlib_stream->next_out = (void *)unencoded_data; + rctx->zlib_stream->avail_out = unencoded_len; + + if (init) { + rctx->zlib_stream->zalloc = Z_NULL; + rctx->zlib_stream->zfree = Z_NULL; + rctx->zlib_stream->opaque = Z_NULL; + ret = inflateInit(rctx->zlib_stream); + } else { + ret = inflateReset(rctx->zlib_stream); + } + if (ret != Z_OK) { + error("zlib inflate init failed: %d", ret); + return -EIO; + } + + while (rctx->zlib_stream->avail_in > 0 && + rctx->zlib_stream->avail_out > 0) { + ret = inflate(rctx->zlib_stream, Z_FINISH); + if (ret == Z_STREAM_END) { + break; + } else if (ret != Z_OK) { + error("zlib inflate failed: %d", ret); + return -EIO; + } + } + return 0; +} + +static int decompress_zstd(struct btrfs_receive *rctx, const char *encoded_buf, + u64 encoded_len, char *unencoded_buf, + u64 unencoded_len) +{ + ZSTD_inBuffer in_buf = { + .src = encoded_buf, + .size = encoded_len + }; + ZSTD_outBuffer out_buf = { + .dst = unencoded_buf, + .size = unencoded_len + }; + size_t ret; + + if (!rctx->zstd_dstream) { + rctx->zstd_dstream = ZSTD_createDStream(); + if (!rctx->zstd_dstream) { + error("failed to create zstd dstream"); + return -ENOMEM; + } + } + ret = ZSTD_initDStream(rctx->zstd_dstream); + if (ZSTD_isError(ret)) { + error("failed to init zstd stream: %s", ZSTD_getErrorName(ret)); + return -EIO; + } + while (in_buf.pos < in_buf.size && out_buf.pos < out_buf.size) { + ret = ZSTD_decompressStream(rctx->zstd_dstream, &out_buf, &in_buf); + if (ret == 0) { + break; + } else if (ZSTD_isError(ret)) { + error("failed to decompress zstd stream: %s", + ZSTD_getErrorName(ret)); + return -EIO; + } + } + return 0; +} + +static int decompress_lzo(const char *encoded_data, u64 encoded_len, + char *unencoded_data, u64 unencoded_len, + unsigned int page_size) +{ + uint32_t total_len; + size_t in_pos, out_pos; + + if (encoded_len < 4) { + error("lzo header is truncated"); + return -EIO; + } + memcpy(&total_len, encoded_data, 4); + total_len = le32toh(total_len); + if (total_len > encoded_len) { + error("lzo header is invalid"); + return -EIO; + } + + in_pos = 4; + out_pos = 0; + while (in_pos < total_len && out_pos < unencoded_len) { + size_t page_remaining; + uint32_t src_len; + lzo_uint dst_len; + int ret; + + page_remaining = -in_pos % page_size; + if (page_remaining < 4) { + if (total_len - in_pos <= page_remaining) + break; + in_pos += page_remaining; + } + + if (total_len - in_pos < 4) { + error("lzo segment header is truncated"); + return -EIO; + } + + memcpy(&src_len, encoded_data + in_pos, 4); + src_len = le32toh(src_len); + in_pos += 4; + if (src_len > total_len - in_pos) { + error("lzo segment header is invalid"); + return -EIO; + } + + dst_len = page_size; + ret = lzo1x_decompress_safe((void *)(encoded_data + in_pos), + src_len, + (void *)(unencoded_data + out_pos), + &dst_len, NULL); + if (ret != LZO_E_OK) { + error("lzo1x_decompress_safe failed: %d", ret); + return -EIO; + } + + in_pos += src_len; + out_pos += dst_len; + } + return 0; +} + +static int decompress_and_write(struct btrfs_receive *rctx, + const char *encoded_data, u64 offset, + u64 encoded_len, u64 unencoded_file_len, + u64 unencoded_len, u64 unencoded_offset, + u32 compression) +{ + int ret = 0; + size_t pos; + ssize_t w; + char *unencoded_data; + int page_shift; + + unencoded_data = calloc(unencoded_len, 1); + if (!unencoded_data) { + error("allocating space for unencoded data failed: %m"); + return -errno; + } + + switch (compression) { + case ENCODED_IOV_COMPRESSION_BTRFS_ZLIB: + ret = decompress_zlib(rctx, encoded_data, encoded_len, + unencoded_data, unencoded_len); + if (ret) + goto out; + break; + case ENCODED_IOV_COMPRESSION_BTRFS_ZSTD: + ret = decompress_zstd(rctx, encoded_data, encoded_len, + unencoded_data, unencoded_len); + if (ret) + goto out; + break; + case ENCODED_IOV_COMPRESSION_BTRFS_LZO_4K: + case ENCODED_IOV_COMPRESSION_BTRFS_LZO_8K: + case ENCODED_IOV_COMPRESSION_BTRFS_LZO_16K: + case ENCODED_IOV_COMPRESSION_BTRFS_LZO_32K: + case ENCODED_IOV_COMPRESSION_BTRFS_LZO_64K: + page_shift = compression - ENCODED_IOV_COMPRESSION_BTRFS_LZO_4K + 12; + ret = decompress_lzo(encoded_data, encoded_len, unencoded_data, + unencoded_len, 1U << page_shift); + if (ret) + goto out; + break; + default: + error("unknown compression: %d", compression); + ret = -EOPNOTSUPP; + goto out; + } + + pos = unencoded_offset; + while (pos < unencoded_file_len) { + w = pwrite(rctx->write_fd, unencoded_data + pos, + unencoded_file_len - pos, offset); + if (w < 0) { + ret = -errno; + error("writing unencoded data failed: %m"); + goto out; + } + pos += w; + offset += w; + } +out: + free(unencoded_data); + return ret; +} + static int process_encoded_write(const char *path, const void *data, u64 offset, - u64 len, u64 unencoded_file_len, u64 unencoded_len, - u64 unencoded_offset, u32 compression, u32 encryption, void *user) + u64 len, u64 unencoded_file_len, + u64 unencoded_len, u64 unencoded_offset, + u32 compression, u32 encryption, void *user) { int ret; ssize_t w; @@ -1092,6 +1315,7 @@ static int process_encoded_write(const char *path, const void *data, u64 offset, { &encoded, sizeof(encoded) }, { (char *)data, len } }; + bool encoded_write = !rctx->force_decompress; if (encryption) { error("encoded_write: encryption not supported"); @@ -1108,17 +1332,25 @@ static int process_encoded_write(const char *path, const void *data, u64 offset, if (ret < 0) return ret; - /* - * NOTE: encoded writes guarantee no partial writes, so we don't need to - * handle that possibility. - */ - w = pwritev2(rctx->write_fd, iov, 2, offset, RWF_ENCODED); - if (w < 0) { - ret = -errno; - error("encoded_write: writing to %s failed: %m", path); - return ret; + if (encoded_write) { + /* + * NOTE: encoded writes guarantee no partial writes, so we don't + * need to handle that possibility. + */ + w = pwritev2(rctx->write_fd, iov, 2, offset, RWF_ENCODED); + if (w >= 0) + return 0; + /* Fall back for these errors, fail hard for anything else. */ + if (errno != ENOSPC && errno != EOPNOTSUPP && errno != EINVAL) { + ret = -errno; + error("encoded_write: writing to %s failed: %m", path); + return ret; + } } - return 0; + + return decompress_and_write(rctx, data, offset, len, unencoded_file_len, + unencoded_len, unencoded_offset, + compression); } static struct btrfs_send_ops send_ops = { @@ -1306,6 +1538,12 @@ out: close(rctx->dest_dir_fd); rctx->dest_dir_fd = -1; } + if (rctx->zstd_dstream) + ZSTD_freeDStream(rctx->zstd_dstream); + if (rctx->zlib_stream) { + inflateEnd(rctx->zlib_stream); + free(rctx->zlib_stream); + } return ret; } @@ -1336,6 +1574,9 @@ static const char * const cmd_receive_usage[] = { "-m ROOTMOUNT the root mount point of the destination filesystem.", " If /proc is not accessible, use this to tell us where", " this file system is mounted.", + "--force-decompress", + " if the stream contains compressed data, always", + " decompress it instead of writing it with encoded I/O", "--dump dump stream metadata, one line per operation,", " does not require the MOUNT parameter", "-v deprecated, alias for global -v option", @@ -1379,12 +1620,16 @@ static int cmd_receive(const struct cmd_struct *cmd, int argc, char **argv) optind = 0; while (1) { int c; - enum { GETOPT_VAL_DUMP = 257 }; + enum { + GETOPT_VAL_DUMP = 257, + GETOPT_VAL_FORCE_DECOMPRESS, + }; static const struct option long_opts[] = { { "max-errors", required_argument, NULL, 'E' }, { "chroot", no_argument, NULL, 'C' }, { "dump", no_argument, NULL, GETOPT_VAL_DUMP }, { "quiet", no_argument, NULL, 'q' }, + { "force-decompress", no_argument, NULL, GETOPT_VAL_FORCE_DECOMPRESS }, { NULL, 0, NULL, 0 } }; @@ -1427,6 +1672,9 @@ static int cmd_receive(const struct cmd_struct *cmd, int argc, char **argv) case GETOPT_VAL_DUMP: dump = 1; break; + case GETOPT_VAL_FORCE_DECOMPRESS: + rctx.force_decompress = 1; + break; default: usage_unknown_option(cmd, argv); } From patchwork Fri Jan 22 20:47:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 12040601 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1785AC433DB for ; Fri, 22 Jan 2021 21:09:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DFA9F23AF8 for ; Fri, 22 Jan 2021 21:09:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730170AbhAVVIu (ORCPT ); Fri, 22 Jan 2021 16:08:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49788 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728381AbhAVUxk (ORCPT ); Fri, 22 Jan 2021 15:53:40 -0500 Received: from mail-pl1-x634.google.com (mail-pl1-x634.google.com [IPv6:2607:f8b0:4864:20::634]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E8EB6C0698D7 for ; Fri, 22 Jan 2021 12:48:49 -0800 (PST) Received: by mail-pl1-x634.google.com with SMTP id b8so3998242plx.0 for ; Fri, 22 Jan 2021 12:48:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=vluC4QsiHSkyry2xzC2VIhFrn1+Np2hmZb2mivQ1YTg=; b=RYRwooaVlDLl3ffkD0Wu+YBn1ROm6XyzduKrLEbtNxU5eNHa5xaJcqVMNZKgHosTIM cyqgYxDts1Y/48+aIQGujGonEZgWPouyOUYbXLRt3hRxme13PZ64P4vT0SgnbVagb0zG q7d85JpAhRsY0W9s1IoU5VnBFMydqQ72wRmZtR5Cullk+7+BduMipAzYpzArHwlU+Jrm jGoGpAeKkQGqTMbXC4dS4lcjvLLbliM2gAyRivOIhbyMc++Osz45zJ58X7uxJqMlHZra ckUUOvy6QimaHRZZhJc/16u5CqSGFq5jbXJWuKguFrgYmKcMMsQINC8UbtuDMiCTOo2X ebKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=vluC4QsiHSkyry2xzC2VIhFrn1+Np2hmZb2mivQ1YTg=; b=FlDyfJGfiDp4+YGkS0lCHrc/i7jiyPZGYQswQePy+QrtC+9tzld+hKoJr/53enxoYu hviPm7o+8eYwMh7OBAhHVaoiTDMtlF5/nQdh08qmOUdBAcV3ekg/MzTQDahVgnGm82Qb Y6t5sxOkyOLvg+4RFqzS4AurU8g9WodyhiuMt24lQ4vAYKy83/9oRpXTjzrrxRkj9T47 3MkHKn6sQNDocl0ixEVxiY3u3zrIzknSN2/IcNX6E7LZDmN4s4ZYxVBkB8yDGrrPM/ou 80jYYiKuaZMsS7Yuvd0QSP1Rh73bEIFgiyfihDWu305+jtNrJY/fcuEUfqej3J6eryGb djig== X-Gm-Message-State: AOAM533cTojqvp6njmwl3FStPLyTFOPUQBzx9T2+UJ7ZLlXtD9mp8S1h Q4K11TJyOdDlQsaDkUyZU9H2lzjQpb8Ecg== X-Google-Smtp-Source: ABdhPJxLSpX4ZzM5XqGc3wAgGKyyULQDkTsSEzp8+3tDlXR8KYC4WLEFHqByg4eRCdG2cRx6R2xpzg== X-Received: by 2002:a17:90b:1489:: with SMTP id js9mr7587958pjb.13.1611348528931; Fri, 22 Jan 2021 12:48:48 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:ea88]) by smtp.gmail.com with ESMTPSA id y16sm9865617pfb.83.2021.01.22.12.48.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 Jan 2021 12:48:47 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v3 08/11] btrfs-progs: receive: process fallocate commands Date: Fri, 22 Jan 2021 12:47:55 -0800 Message-Id: <56247ebadeed72747e5ea3db075a26e831419188.1611347859.git.osandov@osandov.com> X-Mailer: git-send-email 2.30.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov Send stream v2 can emit fallocate commands, so receive must support them as well. The implementation simply passes along the arguments to the syscall. Note that mode is encoded as a u32 in send stream but fallocate takes an int, so there is a unsigned->signed conversion there. Signed-off-by: Boris Burkov --- cmds/receive-dump.c | 9 +++++++++ cmds/receive.c | 25 +++++++++++++++++++++++++ common/send-stream.c | 9 +++++++++ common/send-stream.h | 2 ++ 4 files changed, 45 insertions(+) diff --git a/cmds/receive-dump.c b/cmds/receive-dump.c index 20ec2b70..acc0ba32 100644 --- a/cmds/receive-dump.c +++ b/cmds/receive-dump.c @@ -329,6 +329,14 @@ static int print_encoded_write(const char *path, const void *data, u64 offset, unencoded_offset, compression, encryption); } +static int print_fallocate(const char *path, int mode, u64 offset, u64 len, + void *user) +{ + return PRINT_DUMP(user, path, "fallocate", + "mode=%d offset=%llu len=%llu", + mode, offset, len); +} + struct btrfs_send_ops btrfs_print_send_ops = { .subvol = print_subvol, .snapshot = print_snapshot, @@ -352,4 +360,5 @@ struct btrfs_send_ops btrfs_print_send_ops = { .utimes = print_utimes, .update_extent = print_update_extent, .encoded_write = print_encoded_write, + .fallocate = print_fallocate, }; diff --git a/cmds/receive.c b/cmds/receive.c index 27e5c003..897da372 100644 --- a/cmds/receive.c +++ b/cmds/receive.c @@ -1353,6 +1353,30 @@ static int process_encoded_write(const char *path, const void *data, u64 offset, compression); } +static int process_fallocate(const char *path, int mode, u64 offset, u64 len, + void *user) +{ + int ret; + struct btrfs_receive *rctx = user; + char full_path[PATH_MAX]; + + ret = path_cat_out(full_path, rctx->full_subvol_path, path); + if (ret < 0) { + error("fallocate: path invalid: %s", path); + return ret; + } + ret = open_inode_for_write(rctx, full_path, false); + if (ret < 0) + return ret; + ret = fallocate(rctx->write_fd, mode, offset, len); + if (ret < 0) { + ret = -errno; + error("fallocate: fallocate on %s failed: %m", path); + return ret; + } + return 0; +} + static struct btrfs_send_ops send_ops = { .subvol = process_subvol, .snapshot = process_snapshot, @@ -1376,6 +1400,7 @@ static struct btrfs_send_ops send_ops = { .utimes = process_utimes, .update_extent = process_update_extent, .encoded_write = process_encoded_write, + .fallocate = process_fallocate, }; static int do_receive(struct btrfs_receive *rctx, const char *tomnt, diff --git a/common/send-stream.c b/common/send-stream.c index 044e101b..bc41396e 100644 --- a/common/send-stream.c +++ b/common/send-stream.c @@ -369,6 +369,7 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx) u64 unencoded_offset; int len; int xattr_len; + int fallocate_mode; ret = read_cmd(sctx); if (ret) @@ -514,6 +515,14 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx) case BTRFS_SEND_C_END: ret = 1; break; + case BTRFS_SEND_C_FALLOCATE: + TLV_GET_STRING(sctx, BTRFS_SEND_A_PATH, &path); + TLV_GET_U32(sctx, BTRFS_SEND_A_FALLOCATE_MODE, &fallocate_mode); + TLV_GET_U64(sctx, BTRFS_SEND_A_FILE_OFFSET, &offset); + TLV_GET_U64(sctx, BTRFS_SEND_A_SIZE, &tmp); + ret = sctx->ops->fallocate(path, fallocate_mode, offset, tmp, + sctx->user); + break; } tlv_get_failed: diff --git a/common/send-stream.h b/common/send-stream.h index 607bc007..a58739bb 100644 --- a/common/send-stream.h +++ b/common/send-stream.h @@ -70,6 +70,8 @@ struct btrfs_send_ops { u64 len, u64 unencoded_file_len, u64 unencoded_len, u64 unencoded_offset, u32 compression, u32 encryption, void *user); + int (*fallocate)(const char *path, int mode, u64 offset, u64 len, + void *user); }; int btrfs_read_and_process_send_stream(int fd, From patchwork Fri Jan 22 20:47:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 12040599 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0C2AC433E0 for ; Fri, 22 Jan 2021 21:08:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9387923AF8 for ; Fri, 22 Jan 2021 21:08:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729766AbhAVVIq (ORCPT ); Fri, 22 Jan 2021 16:08:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49340 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730073AbhAVUxl (ORCPT ); Fri, 22 Jan 2021 15:53:41 -0500 Received: from mail-pl1-x629.google.com (mail-pl1-x629.google.com [IPv6:2607:f8b0:4864:20::629]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F02EAC0698D9 for ; Fri, 22 Jan 2021 12:48:51 -0800 (PST) Received: by mail-pl1-x629.google.com with SMTP id q2so1208407plk.4 for ; Fri, 22 Jan 2021 12:48:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2V7g1H6LWShDD9huCMdhWXiMVqXne9qhiCCqnAMGfVE=; b=RFFVc0gv23H7tNtzOih2yo54XHbJAhM+LhAQMJiZxb4RtknuEsAiQzDkqJjfYeCeQb F45JbM/gnO2xIUl0DijvaKCI/S1SoGkYpPSMkg+YraQxJTeJMJsh2pWQzrPZUJv9Ziws SYI4oLk9amhIJ8uEGhLFwdaNGbEkV4kPqrvRPxNfYVq5CROoXFmXNu6FjoZaeChHL/wI da4GWbOTLBtYoa/0In5kRYx1NeCjLGE8uAbpdWarzZyTCdev7FSbakrFrUfzbWUqSaVP zYLKZXvjapo3vs/GR/hVEt52meNQANu8crc9WaZMUhrbDy6pQ6lu8UkQedtGBGoBznIM B77Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2V7g1H6LWShDD9huCMdhWXiMVqXne9qhiCCqnAMGfVE=; b=HzimnGpLBE7y5TRVECP/rtw7e6yJdYeo8s7/g7p5vovbfB1kKfpHpxoe7S4h1+cTKq S2UwbWQ2n0OUOUCdOoM7dW9TGgabw1oHvDU8Ul+0uw9S1r2mbHo+DQBMnnEyWwqq6Ulg 5GU4VTFx8feYgIRNtR9mYRkZ/ksQIEJkRAJCyw9l/U7HK2tJjblqxn9Zr/dw4GU984yf 7LgOFNsMs732UBWcYhMBSS7UNBKYsIbCrlIifl+oDHdOaZWhFBD8W3G6dV2ZWgaD59tV ubgR1s4O1zaHFsaLcEPBBJz2HfVlWILXoq9x0ZOjKujMyfZpDrJND2cjFQ+4G40gftLF 8+Jg== X-Gm-Message-State: AOAM530R9ppyaeLN/POKaLUGvvR92Xx1so6/axaq+Zis495OUp2hKCGW DEAPDEkAMM7Uc4yLYqMAnhJZ2C5odGl9GA== X-Google-Smtp-Source: ABdhPJyJS3WtSZ6yrKaYkqFuHLAWZpvV6Ta81xPgOooBJZmLA+GD76qo9QHlZLDBb4PJKqpXEK+6ug== X-Received: by 2002:a17:90a:e28c:: with SMTP id d12mr1193342pjz.236.1611348530976; Fri, 22 Jan 2021 12:48:50 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:ea88]) by smtp.gmail.com with ESMTPSA id y16sm9865617pfb.83.2021.01.22.12.48.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 Jan 2021 12:48:49 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v3 09/11] btrfs-progs: receive: process setflags ioctl commands Date: Fri, 22 Jan 2021 12:47:56 -0800 Message-Id: <705ef1b18d866593e610e421e1c8558e93e87656.1611347859.git.osandov@osandov.com> X-Mailer: git-send-email 2.30.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov In send stream v2, send can emit a command for setting inode flags via the setflags ioctl. Pass the flags attribute through to the ioctl call in receive. Signed-off-by: Boris Burkov --- cmds/receive-dump.c | 6 ++++++ cmds/receive.c | 24 ++++++++++++++++++++++++ common/send-stream.c | 7 +++++++ common/send-stream.h | 1 + 4 files changed, 38 insertions(+) diff --git a/cmds/receive-dump.c b/cmds/receive-dump.c index acc0ba32..40f07ad4 100644 --- a/cmds/receive-dump.c +++ b/cmds/receive-dump.c @@ -337,6 +337,11 @@ static int print_fallocate(const char *path, int mode, u64 offset, u64 len, mode, offset, len); } +static int print_setflags(const char *path, int flags, void *user) +{ + return PRINT_DUMP(user, path, "setflags", "flags=%d", flags); +} + struct btrfs_send_ops btrfs_print_send_ops = { .subvol = print_subvol, .snapshot = print_snapshot, @@ -361,4 +366,5 @@ struct btrfs_send_ops btrfs_print_send_ops = { .update_extent = print_update_extent, .encoded_write = print_encoded_write, .fallocate = print_fallocate, + .setflags = print_setflags, }; diff --git a/cmds/receive.c b/cmds/receive.c index 897da372..61d3cecb 100644 --- a/cmds/receive.c +++ b/cmds/receive.c @@ -1377,6 +1377,29 @@ static int process_fallocate(const char *path, int mode, u64 offset, u64 len, return 0; } +static int process_setflags(const char *path, int flags, void *user) +{ + int ret; + struct btrfs_receive *rctx = user; + char full_path[PATH_MAX]; + + ret = path_cat_out(full_path, rctx->full_subvol_path, path); + if (ret < 0) { + error("setflags: path invalid: %s", path); + return ret; + } + ret = open_inode_for_write(rctx, full_path, false); + if (ret < 0) + return ret; + ret = ioctl(rctx->write_fd, FS_IOC_SETFLAGS, &flags); + if (ret < 0) { + ret = -errno; + error("setflags: setflags ioctl on %s failed: %m", path); + return ret; + } + return 0; +} + static struct btrfs_send_ops send_ops = { .subvol = process_subvol, .snapshot = process_snapshot, @@ -1401,6 +1424,7 @@ static struct btrfs_send_ops send_ops = { .update_extent = process_update_extent, .encoded_write = process_encoded_write, .fallocate = process_fallocate, + .setflags = process_setflags, }; static int do_receive(struct btrfs_receive *rctx, const char *tomnt, diff --git a/common/send-stream.c b/common/send-stream.c index bc41396e..b9d35a34 100644 --- a/common/send-stream.c +++ b/common/send-stream.c @@ -370,6 +370,7 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx) int len; int xattr_len; int fallocate_mode; + int setflags_flags; ret = read_cmd(sctx); if (ret) @@ -523,8 +524,14 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx) ret = sctx->ops->fallocate(path, fallocate_mode, offset, tmp, sctx->user); break; + case BTRFS_SEND_C_SETFLAGS: + TLV_GET_STRING(sctx, BTRFS_SEND_A_PATH, &path); + TLV_GET_U32(sctx, BTRFS_SEND_A_SETFLAGS_FLAGS, &setflags_flags); + ret = sctx->ops->setflags(path, setflags_flags, sctx->user); + break; } + tlv_get_failed: out: free(path); diff --git a/common/send-stream.h b/common/send-stream.h index a58739bb..5373bf69 100644 --- a/common/send-stream.h +++ b/common/send-stream.h @@ -72,6 +72,7 @@ struct btrfs_send_ops { u32 encryption, void *user); int (*fallocate)(const char *path, int mode, u64 offset, u64 len, void *user); + int (*setflags)(const char *path, int flags, void *user); }; int btrfs_read_and_process_send_stream(int fd, From patchwork Fri Jan 22 20:47:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 12040597 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C11A3C433DB for ; Fri, 22 Jan 2021 21:07:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8F68A23B44 for ; Fri, 22 Jan 2021 21:07:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729934AbhAVVGq (ORCPT ); Fri, 22 Jan 2021 16:06:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49350 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730836AbhAVUxt (ORCPT ); Fri, 22 Jan 2021 15:53:49 -0500 Received: from mail-pl1-x629.google.com (mail-pl1-x629.google.com [IPv6:2607:f8b0:4864:20::629]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 822A6C0698DB for ; Fri, 22 Jan 2021 12:48:54 -0800 (PST) Received: by mail-pl1-x629.google.com with SMTP id s15so3965559plr.9 for ; Fri, 22 Jan 2021 12:48:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=E/JWnKjh8JJUiCDC+qgCUNoPAkGFIHrKhH6SmGrWjAU=; b=eTBhYcBLaaLyV8XtkiR5gw1x8/knvOUE2sI7MEn7do7+5Pw9+p2yxP+Ks/D0EgJHTu cuBqoDpaZFt0WpaGgm9glkzysRkxqjtSEaW49cDkDw7A9uuVb+mzTdc5TCizttXjvjl3 BqmPee09ZrGd+tAsMcEDyUTfoF/Z2CnxvpTXW50Nk/9+eDXe7IPTA2oK9csFgugkBOuJ 3jyMC6r3P39T4AfPAiARpZypmodP/vNLeYiClZNBcWBID8SPcnkDIqcK5PTLqIKnC4tO RrXZh0p9iO4iKWMonihzQYlvZBpcUw5PF8Xq5lDfJh0plcqP034ovzf3duT2tmP8wkR+ TWlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=E/JWnKjh8JJUiCDC+qgCUNoPAkGFIHrKhH6SmGrWjAU=; b=Vl84Hq/xiSvhsDN8xYIH+OMWPsDuBWR+NAt4jCD1rzKZb84sx/4umjIghDJL8esZVm TablV5EXQbFlRnvV/RcNA+E6h0s8ZTUknKxR2Mi5hXruLUDdKI8Qb5aQkEXVJUn+X1qj KJZchNjvSD3pTfcbyHf7EjL9QXSIBPHkaa+Kn0uyeHRJKIhDcIUAh8VK2SGXMITKhMwE 7atD2HKA3yYECcopSMZROwMNVLId/hbCWb57IobCCjDNDmHR5ZM3WaZk3hKmN3F/Tr7P 7Bl3OqgSJXKc0n2O8ndPCp0EnWge1olJcmffJnxFh0/PjFlHKWXb83mZsLr9bsTD+d7p xpIw== X-Gm-Message-State: AOAM531w8mwtub7cE/k4sOZryfV9dsx8ML6t1BtdZy5s0F/Guf6fpJsV 8gZCTuAyRivLuzMLw5jvrnSKiqts9ft7Eg== X-Google-Smtp-Source: ABdhPJxUfpVqwMdLWE8Z1VtdiijNuS9Q4EecFI+1Q6H6PJJbhVYibYVPk/i3KkYZ95p/sesM2XHSFw== X-Received: by 2002:a17:90b:30d4:: with SMTP id hi20mr7261836pjb.41.1611348533227; Fri, 22 Jan 2021 12:48:53 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:ea88]) by smtp.gmail.com with ESMTPSA id y16sm9865617pfb.83.2021.01.22.12.48.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 Jan 2021 12:48:51 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v3 10/11] btrfs-progs: send: stream v2 ioctl flags Date: Fri, 22 Jan 2021 12:47:57 -0800 Message-Id: <2e9257fac6e09c9f58a2b2c8c0ee1179c9e3e014.1611347859.git.osandov@osandov.com> X-Mailer: git-send-email 2.30.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov To make the btrfs send ioctl use the stream v2 format requires passing BTRFS_SEND_FLAG_STREAM_V2 in flags. Further, to cause the ioctl to emit encoded_write commands for encoded extents, we must set that flag as well as BTRFS_SEND_FLAG_COMPRESSED. Finally, we bump up the version in send.h as well, since we are now fully compatible with v2. Add two command line arguments to btrfs send: --stream-version and --compressed-data. --stream-version requires an argument which it parses as an integer and sets STREAM_V2 if the argument is 2. --compressed-data does not require an argument and automatically implies STREAM_V2 as well (COMPRESSED alone causes the ioctl to error out). Some examples to illustrate edge cases: // v1, old format and no encoded_writes btrfs send subvol btrfs send --stream-version 1 subvol // v2 and compressed, we will see encoded_writes btrfs send --compressed-data subvol btrfs send --compressed-data --stream-version 2 subvol // v2 only, new format but no encoded_writes btrfs send --stream-version 2 subvol // error: compressed needs version >= 2 btrfs send --compressed-data --stream-version 1 subvol // error: invalid version (not 1 or 2) btrfs send --stream-version 3 subvol btrfs send --compressed-data --stream-version 0 subvol btrfs send --compressed-data --stream-version 10 subvol Signed-off-by: Boris Burkov --- Documentation/btrfs-send.asciidoc | 16 ++++++++- cmds/send.c | 54 ++++++++++++++++++++++++++++++- ioctl.h | 17 +++++++++- libbtrfsutil/btrfs.h | 17 +++++++++- send.h | 2 +- 5 files changed, 101 insertions(+), 5 deletions(-) diff --git a/Documentation/btrfs-send.asciidoc b/Documentation/btrfs-send.asciidoc index c4a05672..202bcd97 100644 --- a/Documentation/btrfs-send.asciidoc +++ b/Documentation/btrfs-send.asciidoc @@ -55,7 +55,21 @@ send in 'NO_FILE_DATA' mode The output stream does not contain any file data and thus cannot be used to transfer changes. This mode is faster and is useful to show the differences in metadata. --q|--quiet:::: + +--stream-version <1|2>:: +Use the given send stream version. The default is 1. Version 2 encodes file +data slightly more efficiently; it is also required for sending compressed data +directly (see '--compressed-data'). Version 2 requires at least btrfs-progs +5.12 on both the sender and receiver and at least Linux 5.12 on the sender. + +--compressed-data:: +Send data that is compressed on the filesystem directly without decompressing +it. If the receiver supports encoded I/O (see `encoded_io`(7)), it can also +write it directly without decompressing it. Otherwise, the receiver will fall +back to decompressing it and writing it normally. This implies +'--stream-version 2'. + +-q|--quiet:: (deprecated) alias for global '-q' option -v|--verbose:: (deprecated) alias for global '-v' option diff --git a/cmds/send.c b/cmds/send.c index 3bfc69f5..80eb2510 100644 --- a/cmds/send.c +++ b/cmds/send.c @@ -452,6 +452,21 @@ static const char * const cmd_send_usage[] = { " does not contain any file data and thus cannot be used", " to transfer changes. This mode is faster and useful to", " show the differences in metadata.", + "--stream-version <1|2>", + " Use the given send stream version. The default is", + " 1. Version 2 encodes file data slightly more", + " efficiently; it is also required for sending", + " compressed data directly (see --compressed-data).", + " Version 2 requires at least btrfs-progs 5.12 on both", + " the sender and receiver and at least Linux 5.12 on the", + " sender.", + "--compressed-data", + " Send data that is compressed on the filesystem", + " directly without decompressing it. If the receiver", + " supports encoded I/O, it can also write it directly", + " without decompressing it. Otherwise, the receiver will", + " fall back to decompressing it and writing it normally.", + " This implies --stream-version 2.", "-v|--verbose deprecated, alias for global -v option", "-q|--quiet deprecated, alias for global -q option", HELPINFO_INSERT_GLOBALS, @@ -463,6 +478,7 @@ static const char * const cmd_send_usage[] = { static int cmd_send(const struct cmd_struct *cmd, int argc, char **argv) { char *subvol = NULL; + char *end; int ret; char outname[PATH_MAX]; struct btrfs_send send; @@ -474,6 +490,7 @@ static int cmd_send(const struct cmd_struct *cmd, int argc, char **argv) int full_send = 1; int new_end_cmd_semantic = 0; u64 send_flags = 0; + long stream_version = 0; memset(&send, 0, sizeof(send)); send.dump_fd = fileno(stdout); @@ -492,11 +509,17 @@ static int cmd_send(const struct cmd_struct *cmd, int argc, char **argv) optind = 0; while (1) { - enum { GETOPT_VAL_SEND_NO_DATA = 256 }; + enum { + GETOPT_VAL_SEND_NO_DATA = 256, + GETOPT_VAL_SEND_STREAM_V2, + GETOPT_VAL_SEND_COMPRESSED_DATA, + }; static const struct option long_options[] = { { "verbose", no_argument, NULL, 'v' }, { "quiet", no_argument, NULL, 'q' }, { "no-data", no_argument, NULL, GETOPT_VAL_SEND_NO_DATA }, + { "stream-version", required_argument, NULL, GETOPT_VAL_SEND_STREAM_V2 }, + { "compressed-data", no_argument, NULL, GETOPT_VAL_SEND_COMPRESSED_DATA }, { NULL, 0, NULL, 0 } }; int c = getopt_long(argc, argv, "vqec:f:i:p:", long_options, NULL); @@ -585,10 +608,39 @@ static int cmd_send(const struct cmd_struct *cmd, int argc, char **argv) case GETOPT_VAL_SEND_NO_DATA: send_flags |= BTRFS_SEND_FLAG_NO_FILE_DATA; break; + case GETOPT_VAL_SEND_STREAM_V2: + stream_version = strtol(optarg, &end, 10); + if (*end != '\0' || + stream_version < 1 || stream_version > 2) { + ret = 1; + error("invalid --stream-version. valid values: {1, 2}"); + goto out; + } + if (stream_version == 2) + send_flags |= BTRFS_SEND_FLAG_STREAM_V2; + break; + case GETOPT_VAL_SEND_COMPRESSED_DATA: + send_flags |= BTRFS_SEND_FLAG_COMPRESSED; + /* + * We want to default to stream v2 if only compressed is + * set. If stream_version is explicitly set to 0, that + * will trigger its own error condition for being an + * invalid version. + */ + if (stream_version == 0) { + stream_version = 2; + send_flags |= BTRFS_SEND_FLAG_STREAM_V2; + } + break; default: usage_unknown_option(cmd, argv); } } + if (stream_version < 2 && (send_flags & BTRFS_SEND_FLAG_COMPRESSED)) { + ret = 1; + error("--compressed requires --stream-version >= 2"); + goto out; + } if (check_argc_min(argc - optind, 1)) return 1; diff --git a/ioctl.h b/ioctl.h index ade6dcb9..46de8ac8 100644 --- a/ioctl.h +++ b/ioctl.h @@ -653,10 +653,25 @@ BUILD_ASSERT(sizeof(struct btrfs_ioctl_received_subvol_args_32) == 192); */ #define BTRFS_SEND_FLAG_OMIT_END_CMD 0x4 +/* + * Use version 2 of the send stream, which adds new commands and supports larger + * writes. + */ +#define BTRFS_SEND_FLAG_STREAM_V2 0x8 + +/* + * Send compressed data using the ENCODED_WRITE command instead of decompressing + * the data and sending it with the WRITE command. This requires + * BTRFS_SEND_FLAG_STREAM_V2. + */ +#define BTRFS_SEND_FLAG_COMPRESSED 0x10 + #define BTRFS_SEND_FLAG_MASK \ (BTRFS_SEND_FLAG_NO_FILE_DATA | \ BTRFS_SEND_FLAG_OMIT_STREAM_HEADER | \ - BTRFS_SEND_FLAG_OMIT_END_CMD) + BTRFS_SEND_FLAG_OMIT_END_CMD | \ + BTRFS_SEND_FLAG_STREAM_V2 | \ + BTRFS_SEND_FLAG_COMPRESSED) struct btrfs_ioctl_send_args { __s64 send_fd; /* in */ diff --git a/libbtrfsutil/btrfs.h b/libbtrfsutil/btrfs.h index 60d51ff6..8430a40d 100644 --- a/libbtrfsutil/btrfs.h +++ b/libbtrfsutil/btrfs.h @@ -731,10 +731,25 @@ struct btrfs_ioctl_received_subvol_args { */ #define BTRFS_SEND_FLAG_OMIT_END_CMD 0x4 +/* + * Use version 2 of the send stream, which adds new commands and supports larger + * writes. + */ +#define BTRFS_SEND_FLAG_STREAM_V2 0x8 + +/* + * Send compressed data using the ENCODED_WRITE command instead of decompressing + * the data and sending it with the WRITE command. This requires + * BTRFS_SEND_FLAG_STREAM_V2. + */ +#define BTRFS_SEND_FLAG_COMPRESSED 0x10 + #define BTRFS_SEND_FLAG_MASK \ (BTRFS_SEND_FLAG_NO_FILE_DATA | \ BTRFS_SEND_FLAG_OMIT_STREAM_HEADER | \ - BTRFS_SEND_FLAG_OMIT_END_CMD) + BTRFS_SEND_FLAG_OMIT_END_CMD | \ + BTRFS_SEND_FLAG_STREAM_V2 | \ + BTRFS_SEND_FLAG_COMPRESSED) struct btrfs_ioctl_send_args { __s64 send_fd; /* in */ diff --git a/send.h b/send.h index 3c47e0c7..fac90588 100644 --- a/send.h +++ b/send.h @@ -31,7 +31,7 @@ extern "C" { #endif #define BTRFS_SEND_STREAM_MAGIC "btrfs-stream" -#define BTRFS_SEND_STREAM_VERSION 1 +#define BTRFS_SEND_STREAM_VERSION 2 #define BTRFS_SEND_BUF_SIZE_V1 SZ_64K #define BTRFS_SEND_READ_SIZE (1024 * 48) From patchwork Fri Jan 22 20:47:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 12040551 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA3F9C433E0 for ; Fri, 22 Jan 2021 20:59:20 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 80EDE23B09 for ; Fri, 22 Jan 2021 20:59:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730497AbhAVU5s (ORCPT ); Fri, 22 Jan 2021 15:57:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50416 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730755AbhAVUyR (ORCPT ); Fri, 22 Jan 2021 15:54:17 -0500 Received: from mail-pl1-x630.google.com (mail-pl1-x630.google.com [IPv6:2607:f8b0:4864:20::630]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A650EC0698DD for ; Fri, 22 Jan 2021 12:48:56 -0800 (PST) Received: by mail-pl1-x630.google.com with SMTP id h15so1485057pli.8 for ; Fri, 22 Jan 2021 12:48:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2SYEPqYHLibi3PveqMabFu43GSxBmBmjTzenpve/yh8=; b=smdgq5CoEdiKVLQoseZrDt0Eo6B5Z3pPmV+Kj+nex+UpgLdGMRttyACCxdPrd9kNum GPaIbEz+CqSX5OJ5YzclHPpnkoTI5kZB9+aU5cfCQxVyLBG6nWpqNFE4OhdBRM6ZvefG mUmDLCUaicS3w0897/5gW/iAR47yFjFMO9CJFYMZNtnpTf6jZo+aWaA5t6JY0kTa2y9m E7j6EsR8c0j/UA7cRgd2FDAZjv7u02fdOJA4fKOFoVgnXm404sHwW0Dv5FaAIgDSWaDG gr1UVP+tNKk2HrhDN4MbRRzO9k6t2qmwDfacfL1Pt68Js+swd41jALnKbfJjmaugXanM RtMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2SYEPqYHLibi3PveqMabFu43GSxBmBmjTzenpve/yh8=; b=Nbn5/9HTlHfbTjs2vfdTucXiDyWSAQpnB0uJ3s11AoHAhquAOgWI+8LAZScw2mIs4/ MIKkImLeYjEq435gztuOeBfwbyvO2CEpX5xTDlXDDEqApW29GPoOh2MvXULBSifcPRTz u0Caorj71HHG4MPHaL8vF01kLSzZ9jlcm9k/5WR1KSrM4a6EXfkYHij3PJYOADl8wzlZ axsxf1cPxFgh+00R7WCM8+BiGCT44h0Q8m2JWMHb/whJH2VWFideVpZkykfketr+/C/G u1ZtpEJGj7lvEFDfSmh0EtPIZrAHkKPjJASCTzr8cInXov/FsS0RcqBNaaJRsgNhnU9u B1DA== X-Gm-Message-State: AOAM532an3VN48d4iiK1PvCWpcdUGNpKlIFE20OtY4gSIRqsXEVVd3Sz nSxGHklo2dRML2DzVCDK40wo4FeQ7s7iSg== X-Google-Smtp-Source: ABdhPJxc1kextmn3Pm67olcNPQ9Ly1lNQnGPodrfu09B9M3GnuwCeM5LqWLR0LwA/SKQDkzuj6Winw== X-Received: by 2002:a17:902:a504:b029:da:fbca:d49 with SMTP id s4-20020a170902a504b02900dafbca0d49mr937005plq.72.1611348535482; Fri, 22 Jan 2021 12:48:55 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:ea88]) by smtp.gmail.com with ESMTPSA id y16sm9865617pfb.83.2021.01.22.12.48.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 Jan 2021 12:48:53 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v3 11/11] btrfs-progs: receive: add tests for basic encoded_write send/receive Date: Fri, 22 Jan 2021 12:47:58 -0800 Message-Id: <6e36570df3d4a42b586a0e783ec062c7badafa1d.1611347859.git.osandov@osandov.com> X-Mailer: git-send-email 2.30.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov Adapt the existing send/receive tests by passing '-o --force-compress' to the mount commands in a new test. After writing a few files in the various compression formats, send/receive them with and without --force-decompress to test both the encoded_write path and the fallback to decode+write. Signed-off-by: Boris Burkov --- .../043-receive-write-encoded/test.sh | 114 ++++++++++++++++++ 1 file changed, 114 insertions(+) create mode 100755 tests/misc-tests/043-receive-write-encoded/test.sh diff --git a/tests/misc-tests/043-receive-write-encoded/test.sh b/tests/misc-tests/043-receive-write-encoded/test.sh new file mode 100755 index 00000000..b9390e88 --- /dev/null +++ b/tests/misc-tests/043-receive-write-encoded/test.sh @@ -0,0 +1,114 @@ +#!/bin/bash +# +# test that we can send and receive encoded writes for three modes of +# transparent compression: zlib, lzo, and zstd. + +source "$TEST_TOP/common" + +check_prereq mkfs.btrfs +check_prereq btrfs + +setup_root_helper +prepare_test_dev + +here=`pwd` + +# assumes the filesystem exists, and does mount, write, snapshot, send, unmount +# for the specified encoding option +send_one() { + local str + local subv + local snap + + algorithm="$1" + shift + str="$1" + shift + + subv="subv-$algorithm" + snap="snap-$algorithm" + + run_check_mount_test_dev "-o" "compress-force=$algorithm" + cd "$TEST_MNT" || _fail "cannot chdir to TEST_MNT" + + run_check $SUDO_HELPER "$TOP/btrfs" subvolume create "$subv" + run_check $SUDO_HELPER dd if=/dev/zero of="$subv/file1" bs=1M count=1 + run_check $SUDO_HELPER dd if=/dev/zero of="$subv/file2" bs=500K count=1 + run_check $SUDO_HELPER "$TOP/btrfs" subvolume snapshot -r "$subv" "$snap" + run_check $SUDO_HELPER "$TOP/btrfs" send -f "$str" "$snap" "$@" + + cd "$here" || _fail "cannot chdir back to test directory" + run_check_umount_test_dev +} + +receive_one() { + local str + str="$1" + shift + + run_check_mkfs_test_dev + run_check_mount_test_dev + run_check $SUDO_HELPER "$TOP/btrfs" receive "$@" -v -f "$str" "$TEST_MNT" + run_check_umount_test_dev + run_check rm -f -- "$str" +} + +test_one_write_encoded() { + local str + local algorithm + algorithm="$1" + shift + str="$here/stream-$algorithm.stream" + + run_check_mkfs_test_dev + send_one "$algorithm" "$str" --compressed-data + receive_one "$str" "$@" +} + +test_one_stream_v1() { + local str + local algorithm + algorithm="$1" + shift + str="$here/stream-$algorithm.stream" + + run_check_mkfs_test_dev + send_one "$algorithm" "$str" --stream-version 1 + receive_one "$str" "$@" +} + +test_mix_write_encoded() { + local strzlib + local strlzo + local strzstd + strzlib="$here/stream-zlib.stream" + strlzo="$here/stream-lzo.stream" + strzstd="$here/stream-zstd.stream" + + run_check_mkfs_test_dev + + send_one "zlib" "$strzlib" --compressed-data + send_one "lzo" "$strlzo" --compressed-data + send_one "zstd" "$strzstd" --compressed-data + + receive_one "$strzlib" + receive_one "$strlzo" + receive_one "$strzstd" +} + +test_one_write_encoded "zlib" +test_one_write_encoded "lzo" +test_one_write_encoded "zstd" + +# with decompression forced +test_one_write_encoded "zlib" "--force-decompress" +test_one_write_encoded "lzo" "--force-decompress" +test_one_write_encoded "zstd" "--force-decompress" + +# send stream v1 +test_one_stream_v1 "zlib" +test_one_stream_v1 "lzo" +test_one_stream_v1 "zstd" + +# files use a mix of compression algorithms +test_mix_write_encoded