From patchwork Wed Nov 18 19:18:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11915679 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76209C5519F for ; Wed, 18 Nov 2020 19:19:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0F3992220B for ; Wed, 18 Nov 2020 19:19:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="TvyNk7Qz" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727256AbgKRTTK (ORCPT ); Wed, 18 Nov 2020 14:19:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55784 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727252AbgKRTTJ (ORCPT ); Wed, 18 Nov 2020 14:19:09 -0500 Received: from mail-pg1-x543.google.com (mail-pg1-x543.google.com [IPv6:2607:f8b0:4864:20::543]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CFAC2C0613D4 for ; Wed, 18 Nov 2020 11:19:09 -0800 (PST) Received: by mail-pg1-x543.google.com with SMTP id r18so1862981pgu.6 for ; Wed, 18 Nov 2020 11:19:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=xPW+Dna2wRVVrHRfE5h6s7X411Swfso/95bBBe/E3W4=; b=TvyNk7QzgpB5yJSoGrRZyBM+BmM6MmaPBeXY299u5a8Mh7WaxRNh2OGWJ3Tv7G9UHR fNUkvb3o5FbhTBrcRzcu2vAolfe8OTgh+wxhYHppJyE8DE97mM5bUxTUca0W+ehVYkV4 yRSJP+qzJnkaYY/x7EKcuYeEu4HatNT1EyNY1PaF5gz/T0iy22vkiRUYkaNtmILkge5t AixXUekNT00Kn24cj1AhQ4wKN96Ey4l9f1Hs1UZpS7li3JRNououXp4ZWGfPHDU453oB h+pzRJ2w7lGCYpV7E8CRuU8BCpyDQc3N2A295lWiqHro8Tcfr6ccIxgznNWDJnS1Ofgz cmnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xPW+Dna2wRVVrHRfE5h6s7X411Swfso/95bBBe/E3W4=; b=DI6DA7m5udDvltn59FTNHYz+ksNFjp+BXALOuS2Ofa0Q/98nU59IuPsBlAuqjultMM FcnMVqhy4NuqP7DMHYmzmuHycjD3UvUWLynxLpmFU2JKkOgNzieT72ZflaZLfyQRcFuw w3svwZmrBzAzUUuyBHPJZoMb0x+4QwrBTO7WFnhCNesdUzq4dWlie0tVCy+tz22eWNGb zB2MuQfEU7eQ6ZPObmX53UumDKbwxy+r3E6Cy/9m2WavvaVUEjYLm/QGyXAWTwkkNC2N cT/aOHN7fbnj2ZlUITmvvGRvWeosV9B8HcQrTyjKiRQtVuU6+HmEaxYXEIaEQH51PeEA VTdA== X-Gm-Message-State: AOAM5328o8f1MuSINAkh8xwz3CFFYRqLlkDIDrZsj5SASar9uo/UsH2p hW/ehex54ApDvIynNtQ4KeLl0+Z8M6AnBg== X-Google-Smtp-Source: ABdhPJwmZL45UiAH7bJUkojbwqQ/1KAcPRf9x0k4WefT4RTlv7Z6ssxk4zGch9SrbClsJ1FTsUgW2g== X-Received: by 2002:a63:3c10:: with SMTP id j16mr9524663pga.140.1605727148644; Wed, 18 Nov 2020 11:19:08 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:8b43]) by smtp.gmail.com with ESMTPSA id l9sm3197221pjy.10.2020.11.18.11.19.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Nov 2020 11:19:07 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v2 1/5] btrfs: add send stream v2 definitions Date: Wed, 18 Nov 2020 11:18:43 -0800 Message-Id: X-Mailer: git-send-email 2.29.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Omar Sandoval This adds the definitions of the new commands for send stream version 2 and their respective attributes: fallocate, FS_IOC_SETFLAGS (a.k.a. chattr), and encoded writes. It also documents two changes to the send stream format in v2: the receiver shouldn't assume a maximum command size, and the DATA attribute is encoded differently to allow for writes larger than 64k. These will be implemented in subsequent changes, and then the ioctl will accept the new flags. Reviewed-by: Josef Bacik Signed-off-by: Omar Sandoval --- fs/btrfs/send.c | 2 +- fs/btrfs/send.h | 30 +++++++++++++++++++++++++++++- include/uapi/linux/btrfs.h | 13 +++++++++++++ 3 files changed, 43 insertions(+), 2 deletions(-) diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c index d719a2755a40..3cd090c3ffda 100644 --- a/fs/btrfs/send.c +++ b/fs/btrfs/send.c @@ -7255,7 +7255,7 @@ long btrfs_ioctl_send(struct file *mnt_file, struct btrfs_ioctl_send_args *arg) sctx->clone_roots_cnt = arg->clone_sources_count; - sctx->send_max_size = BTRFS_SEND_BUF_SIZE; + sctx->send_max_size = BTRFS_SEND_BUF_SIZE_V1; sctx->send_buf = kvmalloc(sctx->send_max_size, GFP_KERNEL); if (!sctx->send_buf) { ret = -ENOMEM; diff --git a/fs/btrfs/send.h b/fs/btrfs/send.h index de91488b7cd0..9f4f7b96b1eb 100644 --- a/fs/btrfs/send.h +++ b/fs/btrfs/send.h @@ -12,7 +12,11 @@ #define BTRFS_SEND_STREAM_MAGIC "btrfs-stream" #define BTRFS_SEND_STREAM_VERSION 1 -#define BTRFS_SEND_BUF_SIZE SZ_64K +/* + * In send stream v1, no command is larger than 64k. In send stream v2, no limit + * should be assumed. + */ +#define BTRFS_SEND_BUF_SIZE_V1 SZ_64K enum btrfs_tlv_type { BTRFS_TLV_U8, @@ -76,6 +80,13 @@ enum btrfs_send_cmd { BTRFS_SEND_C_END, BTRFS_SEND_C_UPDATE_EXTENT, + + /* The following commands were added in send stream v2. */ + + BTRFS_SEND_C_FALLOCATE, + BTRFS_SEND_C_SETFLAGS, + BTRFS_SEND_C_ENCODED_WRITE, + __BTRFS_SEND_C_MAX, }; #define BTRFS_SEND_C_MAX (__BTRFS_SEND_C_MAX - 1) @@ -106,6 +117,11 @@ enum { BTRFS_SEND_A_PATH_LINK, BTRFS_SEND_A_FILE_OFFSET, + /* + * In send stream v2, this attribute is special: it must be the last + * attribute in a command, its header contains only the type, and its + * length is implicitly the remaining length of the command. + */ BTRFS_SEND_A_DATA, BTRFS_SEND_A_CLONE_UUID, @@ -114,6 +130,18 @@ enum { BTRFS_SEND_A_CLONE_OFFSET, BTRFS_SEND_A_CLONE_LEN, + /* The following attributes were added in send stream v2. */ + + BTRFS_SEND_A_FALLOCATE_MODE, + + BTRFS_SEND_A_SETFLAGS_FLAGS, + + BTRFS_SEND_A_UNENCODED_FILE_LEN, + BTRFS_SEND_A_UNENCODED_LEN, + BTRFS_SEND_A_UNENCODED_OFFSET, + BTRFS_SEND_A_COMPRESSION, + BTRFS_SEND_A_ENCRYPTION, + __BTRFS_SEND_A_MAX, }; #define BTRFS_SEND_A_MAX (__BTRFS_SEND_A_MAX - 1) diff --git a/include/uapi/linux/btrfs.h b/include/uapi/linux/btrfs.h index 2c39d15a2beb..51e69f28d22d 100644 --- a/include/uapi/linux/btrfs.h +++ b/include/uapi/linux/btrfs.h @@ -769,6 +769,19 @@ struct btrfs_ioctl_received_subvol_args { */ #define BTRFS_SEND_FLAG_OMIT_END_CMD 0x4 +/* + * Use version 2 of the send stream, which adds new commands and supports larger + * writes. + */ +#define BTRFS_SEND_FLAG_STREAM_V2 0x8 + +/* + * Send compressed data using the ENCODED_WRITE command instead of decompressing + * the data and sending it with the WRITE command. This requires + * BTRFS_SEND_FLAG_STREAM_V2. + */ +#define BTRFS_SEND_FLAG_COMPRESSED 0x10 + #define BTRFS_SEND_FLAG_MASK \ (BTRFS_SEND_FLAG_NO_FILE_DATA | \ BTRFS_SEND_FLAG_OMIT_STREAM_HEADER | \ From patchwork Wed Nov 18 19:18:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11915683 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0510C6379F for ; Wed, 18 Nov 2020 19:19:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9E15222228 for ; Wed, 18 Nov 2020 19:19:35 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="tSKDlvfh" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727286AbgKRTTR (ORCPT ); Wed, 18 Nov 2020 14:19:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55812 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727273AbgKRTTR (ORCPT ); Wed, 18 Nov 2020 14:19:17 -0500 Received: from mail-pl1-x643.google.com (mail-pl1-x643.google.com [IPv6:2607:f8b0:4864:20::643]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DEA50C0613D4 for ; Wed, 18 Nov 2020 11:19:16 -0800 (PST) Received: by mail-pl1-x643.google.com with SMTP id 5so1527715plj.8 for ; Wed, 18 Nov 2020 11:19:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=26N9VK0p5cJDz5jy+y4kzcNpExO8G7FO7M93wxXu0UU=; b=tSKDlvfhTgTzL5lokM5AI2dcEugmR/E9HDpNCOTivrWdxD/qz4lK0H37LL+0rK7ZIC FctY+zpUH3tLvRQcOuG3k8JbbMWl0D6Pgc+Epr4rKcHfE5nJ1KHxjpjelPt9Xm+F2o4v qa7fVlKEzWV9QVQexhqatvev+UC2CuLhYpG2NZ4VY6f00TPG7aJpGcX1GlQBc64X5abY uwaYFhnR2/A4Gn53cg2jt97U45Ckl4Cg4shzqx0h0cwNu0zDEEALT7NCGlTnHMWo0eJy 45ZgBDiE229Zg0PoWrjuVS/iopvJoSYUTy7zL936nnvi8v1TTeCkIi2jnJViEB5cCEj8 OdNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=26N9VK0p5cJDz5jy+y4kzcNpExO8G7FO7M93wxXu0UU=; b=nEJ+nH3Rk+08bOxitH1BGJGifdZ/5hPXTVxyqeYCJ2RiNW2Zv8rr/S7U2Cg8ul+Hro +twGewK/vXIlhj6XZrGHWkHFyVNkDFZAQ0RrnJNn+hYTh1TsODig+v7+3NBbOKq0ve3/ A7YJj3rhuyE7OCKWHnsHt44/Q25FYEq3B3eupkXEVcyqsCe65UgUFSxLsTv5qwmRCNNu v4xg+CpBvQmtwpubKkPcUMSKUvhMPhLzbQR1FbfUWwKFowlNuG08P5H+/JTScL5UHTJD 3pBEZ+Spl+fKI+ximKPnElm2i8/uZ73gEST509FbX2PNGx8O3Lz5WQP59Jo/kDROBIkp uTIQ== X-Gm-Message-State: AOAM533/gRv0wH/eESPVC4DfHqog8FPlrEWKMCXs2DsxmIyXAEbdIG79 /ehbDAd2dUoVvn+Se480WOSuNMUd3eN/KQ== X-Google-Smtp-Source: ABdhPJwOzZpCBLIh+2XGUvA4XyTM1PJb6t6LcUd1nFPlredd6XVqU325+opbjlvlVe5hyiElPyY1Iw== X-Received: by 2002:a17:902:eb42:b029:d6:ba60:ba41 with SMTP id i2-20020a170902eb42b02900d6ba60ba41mr5653407pli.0.1605727155762; Wed, 18 Nov 2020 11:19:15 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:8b43]) by smtp.gmail.com with ESMTPSA id l9sm3197221pjy.10.2020.11.18.11.19.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Nov 2020 11:19:14 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v2 2/5] btrfs: send: write larger chunks when using stream v2 Date: Wed, 18 Nov 2020 11:18:46 -0800 Message-Id: X-Mailer: git-send-email 2.29.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Omar Sandoval The length field of the send stream TLV header is 16 bits. This means that the maximum amount of data that can be sent for one write is 64k minus one. However, encoded writes must be able to send the maximum compressed extent (128k) in one command. To support this, send stream version 2 encodes the DATA attribute differently: it has no length field, and the length is implicitly up to the end of containing command (which has a 32-bit length field). Although this is necessary for encoded writes, normal writes can benefit from it, too. For v2, let's bump up the send buffer to the maximum compressed extent size plus 16k for the other metadata (144k total). Since this will most likely be vmalloc'd (and always will be after the next commit), we round it up to the next page since we might as well use the rest of the page on systems with >16k pages. Signed-off-by: Omar Sandoval --- fs/btrfs/send.c | 34 ++++++++++++++++++++++++++-------- 1 file changed, 26 insertions(+), 8 deletions(-) diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c index 3cd090c3ffda..822458e6ffae 100644 --- a/fs/btrfs/send.c +++ b/fs/btrfs/send.c @@ -4918,14 +4918,27 @@ static inline u64 max_send_read_size(const struct send_ctx *sctx) static int put_data_header(struct send_ctx *sctx, u32 len) { - struct btrfs_tlv_header *hdr; + if (sctx->flags & BTRFS_SEND_FLAG_STREAM_V2) { + /* + * In v2, the data attribute header doesn't include a length; it + * is implicitly to the end of the command. + */ + if (sctx->send_max_size - sctx->send_size < 2 + len) + return -EOVERFLOW; + put_unaligned_le16(BTRFS_SEND_A_DATA, + sctx->send_buf + sctx->send_size); + sctx->send_size += 2; + } else { + struct btrfs_tlv_header *hdr; - if (sctx->send_max_size - sctx->send_size < sizeof(*hdr) + len) - return -EOVERFLOW; - hdr = (struct btrfs_tlv_header *)(sctx->send_buf + sctx->send_size); - put_unaligned_le16(BTRFS_SEND_A_DATA, &hdr->tlv_type); - put_unaligned_le16(len, &hdr->tlv_len); - sctx->send_size += sizeof(*hdr); + if (sctx->send_max_size - sctx->send_size < sizeof(*hdr) + len) + return -EOVERFLOW; + hdr = (struct btrfs_tlv_header *)(sctx->send_buf + + sctx->send_size); + put_unaligned_le16(BTRFS_SEND_A_DATA, &hdr->tlv_type); + put_unaligned_le16(len, &hdr->tlv_len); + sctx->send_size += sizeof(*hdr); + } return 0; } @@ -7255,7 +7268,12 @@ long btrfs_ioctl_send(struct file *mnt_file, struct btrfs_ioctl_send_args *arg) sctx->clone_roots_cnt = arg->clone_sources_count; - sctx->send_max_size = BTRFS_SEND_BUF_SIZE_V1; + if (sctx->flags & BTRFS_SEND_FLAG_STREAM_V2) { + sctx->send_max_size = ALIGN(SZ_16K + BTRFS_MAX_COMPRESSED, + PAGE_SIZE); + } else { + sctx->send_max_size = BTRFS_SEND_BUF_SIZE_V1; + } sctx->send_buf = kvmalloc(sctx->send_max_size, GFP_KERNEL); if (!sctx->send_buf) { ret = -ENOMEM; From patchwork Wed Nov 18 19:18:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11915685 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED450C64E7C for ; Wed, 18 Nov 2020 19:19:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AC3F02220B for ; Wed, 18 Nov 2020 19:19:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="r3qe8rUp" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727290AbgKRTTX (ORCPT ); Wed, 18 Nov 2020 14:19:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55830 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726416AbgKRTTW (ORCPT ); Wed, 18 Nov 2020 14:19:22 -0500 Received: from mail-pf1-x444.google.com (mail-pf1-x444.google.com [IPv6:2607:f8b0:4864:20::444]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 432F9C0613D6 for ; Wed, 18 Nov 2020 11:19:21 -0800 (PST) Received: by mail-pf1-x444.google.com with SMTP id a18so2056822pfl.3 for ; Wed, 18 Nov 2020 11:19:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6+TqCFRtTZvpeH3ZlWXpEBOxlCAPGP9uQJpmr+cQTds=; b=r3qe8rUpY/JvlCrrwvNYGPGrOVRSr9TYmIL14MpylkfVk+ElV8pBwgicMAHyGFzhRz IAElmlBBJhg7uGG/6zC1bfbEvmQkzNeigYzt9bcAfwlB46toPf3XB0mwR9Ms4JH1cVOA n+gJRZ4+LIPIOBOefRxEZTgNJlgppKDpCjZF42ZVV2xt0vCNbxGAJ8X/N8EbIFcme1TM hyoPoCfVbYOjCmsemTz/i21iEKbnGfdG1I438UWqpWERaz3dD2RidkhSlf+FEeYfq+zE AOqKELvQlcD8ztXFSHfBLEiX7mKRUxVX4cb7JxuM25SWalI8/wuWWi/RruNVT3KxKXSc hY4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6+TqCFRtTZvpeH3ZlWXpEBOxlCAPGP9uQJpmr+cQTds=; b=f3VPqm7c4Bcic+1j+EeNX26x+tLk8RvFD+tnhpF79BYOsMl2RnQ04h4hM75h39kTYe DmWEIlMdxwcU86ul8cZguxb0c6BxBV2azArflqDt2d9WglQR91pFy0jESaXy4S0LRf+E LRWgsfrKsYCGANvaxDaDAplKTVaBkIx7swqvJ3pEw5DtXfWc8ZZ2/fAkM7OIG9nrwLB9 zVi3p40TDBwo9jTDNRRGAYsWuN4NX/k4tGI/BICDgqujra4Vcwy6ribxnkjpuT2F9srq KF3K65llgIRa3nM6UdrQJZK56jWncggof2kv37Zhh+kP98Xm/kGCF7/hl58Knw/kvniW caBQ== X-Gm-Message-State: AOAM530kt9kVkayF9hr3lg2MZFpMk0SNtP+3S9ib45mGLglw11nyfYMo FSS+rmbpPknkfxsvAr/eKTaQMpPaSzPN6w== X-Google-Smtp-Source: ABdhPJwUJauYm1zSoMxCBLcPtfM2erSJk2A+9BBFFyNQciUwvihixuvSausqNYkrhRGa+3cNOMae4Q== X-Received: by 2002:a17:90a:aa8f:: with SMTP id l15mr496518pjq.135.1605727160278; Wed, 18 Nov 2020 11:19:20 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:8b43]) by smtp.gmail.com with ESMTPSA id l9sm3197221pjy.10.2020.11.18.11.19.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Nov 2020 11:19:19 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v2 3/5] btrfs: send: allocate send buffer with alloc_page() and vmap() for v2 Date: Wed, 18 Nov 2020 11:18:48 -0800 Message-Id: <81cba691d5355e218c49c44d657a6503d30c34cd.1605723600.git.osandov@fb.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Omar Sandoval For encoded writes, we need the raw pages for reading compressed data directly via a bio. So, replace kvmalloc() with vmap() so we have access to the raw pages. 144k is large enough that it usually gets allocated with vmalloc(), anyways. Signed-off-by: Omar Sandoval --- fs/btrfs/send.c | 33 +++++++++++++++++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-) diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c index 822458e6ffae..f04d815c5d16 100644 --- a/fs/btrfs/send.c +++ b/fs/btrfs/send.c @@ -81,6 +81,7 @@ struct send_ctx { char *send_buf; u32 send_size; u32 send_max_size; + struct page **send_buf_pages; u64 total_send_size; u64 cmd_send_size[BTRFS_SEND_C_MAX + 1]; u64 flags; /* 'flags' member of btrfs_ioctl_send_args is u64 */ @@ -7191,6 +7192,7 @@ long btrfs_ioctl_send(struct file *mnt_file, struct btrfs_ioctl_send_args *arg) struct btrfs_root *clone_root; struct send_ctx *sctx = NULL; u32 i; + u32 send_buf_num_pages = 0; u64 *clone_sources_tmp = NULL; int clone_sources_to_rollback = 0; size_t alloc_size; @@ -7271,10 +7273,28 @@ long btrfs_ioctl_send(struct file *mnt_file, struct btrfs_ioctl_send_args *arg) if (sctx->flags & BTRFS_SEND_FLAG_STREAM_V2) { sctx->send_max_size = ALIGN(SZ_16K + BTRFS_MAX_COMPRESSED, PAGE_SIZE); + send_buf_num_pages = sctx->send_max_size >> PAGE_SHIFT; + sctx->send_buf_pages = kcalloc(send_buf_num_pages, + sizeof(*sctx->send_buf_pages), + GFP_KERNEL); + if (!sctx->send_buf_pages) { + send_buf_num_pages = 0; + ret = -ENOMEM; + goto out; + } + for (i = 0; i < send_buf_num_pages; i++) { + sctx->send_buf_pages[i] = alloc_page(GFP_KERNEL); + if (!sctx->send_buf_pages[i]) { + ret = -ENOMEM; + goto out; + } + } + sctx->send_buf = vmap(sctx->send_buf_pages, send_buf_num_pages, + VM_MAP, PAGE_KERNEL); } else { sctx->send_max_size = BTRFS_SEND_BUF_SIZE_V1; + sctx->send_buf = kvmalloc(sctx->send_max_size, GFP_KERNEL); } - sctx->send_buf = kvmalloc(sctx->send_max_size, GFP_KERNEL); if (!sctx->send_buf) { ret = -ENOMEM; goto out; @@ -7483,7 +7503,16 @@ long btrfs_ioctl_send(struct file *mnt_file, struct btrfs_ioctl_send_args *arg) fput(sctx->send_filp); kvfree(sctx->clone_roots); - kvfree(sctx->send_buf); + if (sctx->flags & BTRFS_SEND_FLAG_STREAM_V2) { + vunmap(sctx->send_buf); + for (i = 0; i < send_buf_num_pages; i++) { + if (sctx->send_buf_pages[i]) + __free_page(sctx->send_buf_pages[i]); + } + kfree(sctx->send_buf_pages); + } else { + kvfree(sctx->send_buf); + } name_cache_free(sctx); From patchwork Wed Nov 18 19:18:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11915689 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 74C55C64E8A for ; Wed, 18 Nov 2020 19:19:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 314E52220B for ; Wed, 18 Nov 2020 19:19:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="HnwigP05" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727304AbgKRTTZ (ORCPT ); Wed, 18 Nov 2020 14:19:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55840 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726416AbgKRTTY (ORCPT ); Wed, 18 Nov 2020 14:19:24 -0500 Received: from mail-pl1-x644.google.com (mail-pl1-x644.google.com [IPv6:2607:f8b0:4864:20::644]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7177AC0613D6 for ; Wed, 18 Nov 2020 11:19:23 -0800 (PST) Received: by mail-pl1-x644.google.com with SMTP id bj5so717584plb.4 for ; Wed, 18 Nov 2020 11:19:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=P9Kdu1iOQxHZHPzLsKpiyc2lmM69laJRMX8/k+zeI9U=; b=HnwigP052OdRCU9TTvxKjsmCRiFnZCItgfljMLRQV+kUfZPqlxT76tlV5nK13Gn9ZV 8/Ab8Hh+MwMHIMPqCSjIYF+GsIRD5icuJTkNG9qX2f52JlPZo4VjO5oA7TrbT3EhX3Zr ioGBhNQsCOSJ9XDIf2jbz7arirspi7eDFG+j3aZtk+ybdQQxSANNvmlCnz2LByRnpBNV 1Z0cJPip3ujM2/hONn7eZKPW8wS8wpWkGdhDhhl8k5O0u/nMHSkVL3fKPqM+/spTJS2N Vek9l/Gd/s779eTHzJd01ifO7vS5nPgZOljuUb59vYd64tXsXBmsLENuFMzedrwkeIGp wSXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=P9Kdu1iOQxHZHPzLsKpiyc2lmM69laJRMX8/k+zeI9U=; b=UN+7uQyEOrqYRPa6RMCXFDql5amIjwl+iBQMmOU8PC8jUpC/rSk+zz+Gt+gGjOzJOG g/6/14H3kC4g5DA33Z6aoJfwXGBsHoEz2/AFUuJbGrakndqf+rotXJr9tOfIl1NH3AvI h5QMcKgFt9outK6u9eVSsmU2+yjDf9pMP9dwYTMvdJXzBuGvwHYD8cNi4Qxt8UAS/9yD gLtWsHY2isLqCDuQHfbRvh36QrkIoifJnz5Bz+Fl2fODU/cL1LMBLP/FujJP1BZ0d3aJ 8i5XatmkC6hDULJOXmCFk7AdMarVQEQpjhoLzn/1NxQLj6a/VxYLdVaWw4ofShXX3smn nXfQ== X-Gm-Message-State: AOAM530yF9x7hmamDwiuMGhvtcJ9Bry9sAV1UA8DsCFoMt8Iqgpc7OBG z0jH96B+fe+cumQndkQqMzLgM/amRwi+ew== X-Google-Smtp-Source: ABdhPJynB/3z0rxiPUbMCU413edbSddUD8pbNkI6RYQQryN1QHsqK3UrdxKuCwPlg4Lqw15YCYw9GQ== X-Received: by 2002:a17:902:bb94:b029:d6:edb2:4f41 with SMTP id m20-20020a170902bb94b02900d6edb24f41mr5762832pls.3.1605727162314; Wed, 18 Nov 2020 11:19:22 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:8b43]) by smtp.gmail.com with ESMTPSA id l9sm3197221pjy.10.2020.11.18.11.19.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Nov 2020 11:19:21 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v2 04/13] btrfs-progs: receive: support v2 send stream DATA tlv format Date: Wed, 18 Nov 2020 11:18:49 -0800 Message-Id: <514c65bbb59958b3e7504cadc3c708a48771d39b.1605723745.git.osandov@osandov.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov The new format privileges the BTRFS_SEND_A_DATA attribute by guaranteeing it will always be the last attribute in any command that needs it, and by implicitly encoding the data length as the difference between the total command length in the command header and the sizes of the rest of the attributes (and of course the tlv_type identifying the DATA attribute). To parse the new stream, we must read the tlv_type and if it is not DATA, we proceed normally, but if it is DATA, we don't parse a tlv_len but simply compute the length. In addition, we add some bounds checking when parsing each chunk of data, as well as for the tlv_len itself. Signed-off-by: Boris Burkov --- common/send-stream.c | 36 ++++++++++++++++++++++++++---------- 1 file changed, 26 insertions(+), 10 deletions(-) diff --git a/common/send-stream.c b/common/send-stream.c index 51a6a94a..77d5cd04 100644 --- a/common/send-stream.c +++ b/common/send-stream.c @@ -165,28 +165,44 @@ static int read_cmd(struct btrfs_send_stream *sctx) pos = 0; while (pos < cmd_len) { - struct btrfs_tlv_header *tlv_hdr; u16 tlv_type; - u16 tlv_len; struct btrfs_send_attribute *send_attr; - tlv_hdr = (struct btrfs_tlv_header *)data; - tlv_type = le16_to_cpu(tlv_hdr->tlv_type); - tlv_len = le16_to_cpu(tlv_hdr->tlv_len); + if (cmd_len - pos < sizeof(__le16)) { + error("send stream is truncated"); + ret = -EINVAL; + goto out; + } + tlv_type = le16_to_cpu(*(__le16 *)data); if (tlv_type == 0 || tlv_type > BTRFS_SEND_A_MAX) { - error("invalid tlv in cmd tlv_type = %hu, tlv_len = %hu", - tlv_type, tlv_len); + error("invalid tlv in cmd tlv_type = %hu", tlv_type); ret = -EINVAL; goto out; } send_attr = &sctx->cmd_attrs[tlv_type]; send_attr->tlv_type = tlv_type; - send_attr->tlv_len = tlv_len; - pos += sizeof(*tlv_hdr); - data += sizeof(*tlv_hdr); + pos += sizeof(tlv_type); + data += sizeof(tlv_type); + if (sctx->version == 2 && tlv_type == BTRFS_SEND_A_DATA) { + send_attr->tlv_len = cmd_len - pos; + } else { + if (cmd_len - pos < sizeof(__le16)) { + error("send stream is truncated"); + ret = -EINVAL; + goto out; + } + send_attr->tlv_len = le16_to_cpu(*(__le16 *)data); + pos += sizeof(__le16); + data += sizeof(__le16); + } + if (cmd_len - pos < send_attr->tlv_len) { + error("send stream is truncated"); + ret = -EINVAL; + goto out; + } send_attr->data = data; pos += send_attr->tlv_len; data += send_attr->tlv_len; From patchwork Wed Nov 18 19:18:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11915733 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DCDBFC71156 for ; Wed, 18 Nov 2020 19:19:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 437022220B for ; Wed, 18 Nov 2020 19:19:38 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="ghXL55lm" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727317AbgKRTTb (ORCPT ); Wed, 18 Nov 2020 14:19:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55856 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726416AbgKRTT1 (ORCPT ); Wed, 18 Nov 2020 14:19:27 -0500 Received: from mail-pl1-x642.google.com (mail-pl1-x642.google.com [IPv6:2607:f8b0:4864:20::642]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9657BC061A48 for ; Wed, 18 Nov 2020 11:19:27 -0800 (PST) Received: by mail-pl1-x642.google.com with SMTP id l11so1547784plt.1 for ; Wed, 18 Nov 2020 11:19:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ng3S9jzsXHM6uNgh/DWKZS8UtOoWwHqg6ejAsiMq32A=; b=ghXL55lmpcFLzMg1nP3mlUyrBZxvH+ykIs6DIk6t7TYvxlGVJWgrlxAZcwD3H0QORm tPEtf2etlYrGH0QngZgiL6IcJTagyjpXq/i+i4xTm9u5fEiUHSo35CKhcSX+zwej+5tu Iqm8UuETnJbAuyvlBmbDHkLlnKqFOEoqZWNaykZRbFvG4RmU8NZIqXw8MpI6KbL7ztn6 EHChq/+phTmop66ocDCq8gR9uEq6oW8OXF08gl25T8fy88ZqjjHToav8gUi1dy3KoyvC UzPyjS6bHmSl4mC+rayGkLJulTr/tUNB/LITuvW506Gg/2NAUl6tRx+bztPwTM2XwUDa NWXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ng3S9jzsXHM6uNgh/DWKZS8UtOoWwHqg6ejAsiMq32A=; b=desSIry54QvSbOhULdeif5nI9J+kcspaz0IAb6glTPGoyXYwTADGlotM+p9yd0iW0G +cppwgPN0FKJC5q0L4/bIHajNMVYpHuCZ8xyeLREiKrl0ZlsBx1KVLgkPkZYkqdQXiLR QeoCmQ4ceCjJ8fKvCgrFsTgAWxruDZn+PueODUc5MtLMd03p3mGKXbwbFbCE5WI6UOAZ Xiaj8eaO5FEZa6yTGY9UAL2LSg10VoTh1oqrlnoEAMSY20SxIFsdcAZ2ILfPmhv2kjJl +f0vFaf6f8GeXO47pA3BJejamskJw4dvyBKxR6xNkRi4XGz89u4AAmrRYU++Sn70pbeY woyA== X-Gm-Message-State: AOAM53089PCpir80E70XbD/0a8Y6ybwPBivf2yiz3f5yhhcJfqR25jc8 apGd/GILa30rH+nR59GZ9Uqtimn0gXOuGA== X-Google-Smtp-Source: ABdhPJyJSSBXpWt2egYFPP8WmVBzm0RRl48D0NpL29atIM2xMxu2lspygQV3KZ7xzLqMfUV1Uuh4hA== X-Received: by 2002:a17:902:c393:b029:d9:6e7:6787 with SMTP id g19-20020a170902c393b02900d906e76787mr5682353plg.78.1605727166485; Wed, 18 Nov 2020 11:19:26 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:8b43]) by smtp.gmail.com with ESMTPSA id l9sm3197221pjy.10.2020.11.18.11.19.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Nov 2020 11:19:25 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v2 05/13] btrfs-progs: receive: add send stream v2 cmds and attrs to send.h Date: Wed, 18 Nov 2020 11:18:51 -0800 Message-Id: <8495f1c759ae6fc3002cb5d9309d206a6e6f0906.1605723745.git.osandov@osandov.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov Send stream v2 adds three commands and several attributes associated to those commands. Before we implement processing them, add all the commands and attributes. This avoids leaving the enums in an intermediate state that doesn't correspond to any version of send stream. Signed-off-by: Boris Burkov --- send.h | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/send.h b/send.h index 228928a0..3c47e0c7 100644 --- a/send.h +++ b/send.h @@ -98,6 +98,11 @@ enum btrfs_send_cmd { BTRFS_SEND_C_END, BTRFS_SEND_C_UPDATE_EXTENT, + + BTRFS_SEND_C_FALLOCATE, + BTRFS_SEND_C_SETFLAGS, + BTRFS_SEND_C_ENCODED_WRITE, + __BTRFS_SEND_C_MAX, }; #define BTRFS_SEND_C_MAX (__BTRFS_SEND_C_MAX - 1) @@ -136,6 +141,16 @@ enum { BTRFS_SEND_A_CLONE_OFFSET, BTRFS_SEND_A_CLONE_LEN, + BTRFS_SEND_A_FALLOCATE_MODE, + + BTRFS_SEND_A_SETFLAGS_FLAGS, + + BTRFS_SEND_A_UNENCODED_FILE_LEN, + BTRFS_SEND_A_UNENCODED_LEN, + BTRFS_SEND_A_UNENCODED_OFFSET, + BTRFS_SEND_A_COMPRESSION, + BTRFS_SEND_A_ENCRYPTION, + __BTRFS_SEND_A_MAX, }; #define BTRFS_SEND_A_MAX (__BTRFS_SEND_A_MAX - 1) From patchwork Wed Nov 18 19:18:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11915695 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 285BDC8300A for ; Wed, 18 Nov 2020 19:19:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C7B7822227 for ; Wed, 18 Nov 2020 19:19:39 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="roqYDntK" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727343AbgKRTTf (ORCPT ); Wed, 18 Nov 2020 14:19:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55878 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727327AbgKRTTd (ORCPT ); Wed, 18 Nov 2020 14:19:33 -0500 Received: from mail-pl1-x643.google.com (mail-pl1-x643.google.com [IPv6:2607:f8b0:4864:20::643]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 667ECC0613D6 for ; Wed, 18 Nov 2020 11:19:32 -0800 (PST) Received: by mail-pl1-x643.google.com with SMTP id bj5so717809plb.4 for ; Wed, 18 Nov 2020 11:19:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BdTEku7J/1wmWyPZ7NOZDpBLdc5B/RHj8uVOrA+In/8=; b=roqYDntKPONpvGO4yrNgAnMiIvzffHjw1ItJjihDO0zAXyTqw7LaWVag9fQpotYUKd YfVB6eFRNdZ5s6MEsHcrcqaJFcYKJyMEfNku/pCinewZLtzbI4FIGXBqhiZkvNOg3A3B AAsgrrl7tDKgSZpiOIhXcoZ7ZDrPSrVaiXe02Q6CEiyYITVaT7I+IgwHFFaIsDnePx1e WhG4EGML51kiebsnX5Z3iIDjhfpdcS/8oh1i0RFjCOfvE90xXoiHIWBp+6lBvrBiS0fQ 16Y7ZDhSWkcCR2EJ/g4EwNQ7a/o4RLSksiEaCh4pS2UuBOyhD8vOYbv7lmLaJGGSgA1B RgFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BdTEku7J/1wmWyPZ7NOZDpBLdc5B/RHj8uVOrA+In/8=; b=n4L+cbfCTLtNSkp0xzT415u7AZ9/h/m2TZLAlc+lIYeTOfA+fMwsCVhWBtOQnu2k/1 ZpczTN9TW0ZFw9QyGpv2Bk6FNOP6VQ2M9j3I8/gtdM4DTCKex8T0joMJVkenRBEaSEq7 641zsO+VQuKNfIv68WcJ5gPB0nqCx14mNbur2bKxx5f393GD37eGlZoRnwOGva29fggF odR0sqME/i7Oi+niP/r1S3FWaRbW1xheehxfRM1ytBQVPCqLoqa9kui0WE3vyOBIql6d 9nQ5+b2UoQu+OENLKaltTKRWM6p6jX4W0dljTJf7PtliNITSBDFcCMdP4iaiEhwtCcBH YDTA== X-Gm-Message-State: AOAM533IuJ0xrN/L0XcxCUbLO0VUI0jPM/ofgINbm/4IsgxnZZOJ1OMc 6Z2F2M2lFBvKMRnNbsLwMC6acIpCHNvtLw== X-Google-Smtp-Source: ABdhPJzLbh0+BDId9iRc2IZ73xfiwWhzpQU1qk3S1DKX/X95Jnkcr7rywmRMtBQwOSeUicRjgXvtWg== X-Received: by 2002:a17:902:6a83:b029:d5:e98f:2437 with SMTP id n3-20020a1709026a83b02900d5e98f2437mr5625373plk.38.1605727171319; Wed, 18 Nov 2020 11:19:31 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:8b43]) by smtp.gmail.com with ESMTPSA id l9sm3197221pjy.10.2020.11.18.11.19.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Nov 2020 11:19:29 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v2 06/13] btrfs-progs: receive: add stub implementation for pwritev2 Date: Wed, 18 Nov 2020 11:18:53 -0800 Message-Id: X-Mailer: git-send-email 2.29.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov Encoded writes in receive will use pwritev2. It is possible that the system libc does not export this function, so we stub it out and detect whether to build the stub code with autoconf. This syscall has special semantics in x32 (no hi lo, just takes loff_t) so we have to detect that case and use the appropriate arguments. Signed-off-by: Boris Burkov --- Makefile | 4 ++-- configure.ac | 1 + stubs.c | 24 ++++++++++++++++++++++++ stubs.h | 11 +++++++++++ 4 files changed, 38 insertions(+), 2 deletions(-) create mode 100644 stubs.c create mode 100644 stubs.h diff --git a/Makefile b/Makefile index 381b630d..505e39d7 100644 --- a/Makefile +++ b/Makefile @@ -173,12 +173,12 @@ libbtrfs_objects = common/send-stream.o common/send-utils.o kernel-lib/rbtree.o kernel-lib/raid56.o kernel-lib/tables.o \ common/device-scan.o common/path-utils.o \ common/utils.o libbtrfsutil/subvolume.o libbtrfsutil/stubs.o \ - crypto/hash.o crypto/xxhash.o $(CRYPTO_OBJECTS) + crypto/hash.o crypto/xxhash.o $(CRYPTO_OBJECTS) stubs.o libbtrfs_headers = common/send-stream.h common/send-utils.h send.h kernel-lib/rbtree.h btrfs-list.h \ crypto/crc32c.h kernel-lib/list.h kerncompat.h \ kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \ common/extent-cache.h kernel-shared/extent_io.h ioctl.h \ - kernel-shared/ctree.h btrfsck.h version.h + kernel-shared/ctree.h btrfsck.h version.h stubs.h libbtrfsutil_major := $(shell sed -rn 's/^\#define BTRFS_UTIL_VERSION_MAJOR ([0-9])+$$/\1/p' libbtrfsutil/btrfsutil.h) libbtrfsutil_minor := $(shell sed -rn 's/^\#define BTRFS_UTIL_VERSION_MINOR ([0-9])+$$/\1/p' libbtrfsutil/btrfsutil.h) libbtrfsutil_patch := $(shell sed -rn 's/^\#define BTRFS_UTIL_VERSION_PATCH ([0-9])+$$/\1/p' libbtrfsutil/btrfsutil.h) diff --git a/configure.ac b/configure.ac index dd4adedf..eaf353cc 100644 --- a/configure.ac +++ b/configure.ac @@ -57,6 +57,7 @@ AC_CHECK_FUNCS([openat], [], [AC_MSG_ERROR([cannot find openat() function])]) AC_CHECK_FUNCS([reallocarray]) +AC_CHECK_FUNCS([pwritev2]) m4_ifndef([PKG_PROG_PKG_CONFIG], [m4_fatal([Could not locate the pkg-config autoconf diff --git a/stubs.c b/stubs.c new file mode 100644 index 00000000..ab68a411 --- /dev/null +++ b/stubs.c @@ -0,0 +1,24 @@ +#if HAVE_PWRITEV2 != 1 + +#include "stubs.h" + +#include "kerncompat.h" + +#include +#include +#include + +ssize_t pwritev2(int fd, const struct iovec *iov, int iovcnt, off_t offset, + int flags) +{ +/* these conditions indicate an x32 system, which has a different pwritev2 */ +#if defined(__x86_64__) && defined(__ILP32__) + return syscall(SYS_pwritev2, fd, iov, iovcnt, offset, flags); +#else + unsigned long hi = offset >> (BITS_PER_LONG / 2) >> (BITS_PER_LONG / 2); + unsigned long lo = offset; + + return syscall(SYS_pwritev2, fd, iov, iovcnt, lo, hi, flags); +#endif // X32 +} +#endif /* HAVE_PWRIVEV2 */ diff --git a/stubs.h b/stubs.h new file mode 100644 index 00000000..b39f8a69 --- /dev/null +++ b/stubs.h @@ -0,0 +1,11 @@ +#ifndef _BTRFS_STUBS_H +#define _BTRFS_STUBS_H + +#include + +struct iovec; + +ssize_t pwritev2(int fd, const struct iovec *iov, int iovcnt, off_t offset, + int flags); + +#endif From patchwork Wed Nov 18 19:18:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11915737 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B3F7C8300E for ; Wed, 18 Nov 2020 19:19:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 20CC72220B for ; Wed, 18 Nov 2020 19:19:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="SXRyxESn" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727339AbgKRTTe (ORCPT ); Wed, 18 Nov 2020 14:19:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55886 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727312AbgKRTTe (ORCPT ); Wed, 18 Nov 2020 14:19:34 -0500 Received: from mail-pl1-x641.google.com (mail-pl1-x641.google.com [IPv6:2607:f8b0:4864:20::641]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3D7A7C061A48 for ; Wed, 18 Nov 2020 11:19:34 -0800 (PST) Received: by mail-pl1-x641.google.com with SMTP id t18so1550561plo.0 for ; Wed, 18 Nov 2020 11:19:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=8Juha+13op+j4M860hObsmkFcXreLNLajnjj5oyj6Fc=; b=SXRyxESnxqWtvijXr4ME6DRo29Bb+r4IbukyFtWQhUiOeWuZMiGoqk8zT99vU9CfIs sFsL4VFXw2Tdeu6ztQjIN8SWzVcjYb5tSsuwX2M7LgIRx5Vsjp0xhYMqn57o+cyV2Wcc Gd5uTWmqj4v7p5sN5d7gLtojV66ED7tqqRR5lXMvzAaTJksLxc6da5hefoE8IqFedSCV ZBQmOGce5/4zoRzCiE8Jo82GHMms5/lfwKHOoA2VmDX8VgTzqdPXpFhoCMXQqoxjNZFq XWGHYeHvM8PzVrbopREJ9MeummLpqJn5ttjLSbS+YtpopW8KYFNKXrNVcy0oyTa95rGD zRSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8Juha+13op+j4M860hObsmkFcXreLNLajnjj5oyj6Fc=; b=MAcRsF7XQBZ93T/RgteUqU+cnDNuxLpmqJJE8dRTMQBVmOMUM4SpuxtCyPzXYvE6Kn Q+rzpe8bC6DvFqj2ChFK8H9W1N8b/n9wry3nPHlfevV53a+VVj7MLO/7AsZLiBPZai7F oWoO+ZKgwAB+0GvVcyuvx2h/y3blSXI8RpkAGck50q7xB8B14Oq8Mlpiepr2SwiJfi3u ZjOH+lip9WDnI2quOx/IFJF7sybO0Fa5simLk88Y7cqAgwjLb8bdf4i2BI2dFVAO5iqM 5W74CGVIRqjBYcx4WmQNokl2pWMSSzJfE0f5vffsaCkvNtFS5PW0f0+fXII/DkPCexuK xTpw== X-Gm-Message-State: AOAM531+ueETOf5Mpi98Hk4vgOBk6zHtf3AWBbRGKM220PekAqu0M/H6 IAA3pkVMaT9qP9xqd9K2QR1IIB1l/f353g== X-Google-Smtp-Source: ABdhPJwtVAIOFL011TKnuC2lFW9Qq26zYzRJ4/EAzUsgjlotJzQpQnLq3yrQQp7SThqbk1uJ/r1pkw== X-Received: by 2002:a17:902:8ec7:b029:d8:e603:304c with SMTP id x7-20020a1709028ec7b02900d8e603304cmr5393411plo.85.1605727173156; Wed, 18 Nov 2020 11:19:33 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:8b43]) by smtp.gmail.com with ESMTPSA id l9sm3197221pjy.10.2020.11.18.11.19.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Nov 2020 11:19:32 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v2 07/13] btrfs-progs: receive: open files with O_CLOEXEC Date: Wed, 18 Nov 2020 11:18:54 -0800 Message-Id: <41f3e3f92372f48037c7cee18af808d4cc5352c5.1605723745.git.osandov@osandov.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Omar Sandoval Opening with O_ALLOW_ENCODED also requires O_CLOEXEC. Let's add O_CLOEXEC now in preparation. btrfs receive doesn't exec anything, so this should be a no-op. Signed-off-by: Omar Sandoval --- cmds/receive.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/cmds/receive.c b/cmds/receive.c index 2aaba3ff..2c56cea6 100644 --- a/cmds/receive.c +++ b/cmds/receive.c @@ -654,7 +654,11 @@ static int open_inode_for_write(struct btrfs_receive *rctx, const char *path) rctx->write_fd = -1; } - rctx->write_fd = open(path, O_RDWR); + /* + * When opening with O_ALLOW_ENCODED, O_CLOEXEC must also be specified. + * We might as well always use it even though we don't exec anything. + */ + rctx->write_fd = open(path, O_RDWR | O_CLOEXEC); if (rctx->write_fd < 0) { ret = -errno; error("cannot open %s: %m", path); From patchwork Wed Nov 18 19:18:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11915743 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1AABC83010 for ; Wed, 18 Nov 2020 19:19:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9E38F2220B for ; Wed, 18 Nov 2020 19:19:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="iZIK8j5Z" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727356AbgKRTTi (ORCPT ); Wed, 18 Nov 2020 14:19:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55896 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727327AbgKRTTg (ORCPT ); Wed, 18 Nov 2020 14:19:36 -0500 Received: from mail-pl1-x643.google.com (mail-pl1-x643.google.com [IPv6:2607:f8b0:4864:20::643]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8950FC0613D6 for ; Wed, 18 Nov 2020 11:19:36 -0800 (PST) Received: by mail-pl1-x643.google.com with SMTP id s2so1526815plr.9 for ; Wed, 18 Nov 2020 11:19:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=GSKSDRR8K/+TKUuUq5SwiGfUO6KTVDHzgwHOrlirzyk=; b=iZIK8j5ZVlJW3ibHpPfRgjY/R2IWpjUmAQWNTUIUac8hV6LSOlpLK9uclduYXRW2EE 3tSywZp392AmcfprV0GK5UfY7rymX3b3iAWKAo52bXWzcK1XhSJYRFQcgsNDIHqmPlA2 kf5xvz0eSlVYL69Lw9eA4dkGfirWFyUU94WCUFYmG4NBNU4JsdjqUV5rS8vQgrj0pnmg 93P2NOSsMJHeNAyXqPP99nBqRuOMoZ1hpF05mpra2hGX08g17KHJ1+xMQTTqjXnohZJG 3yBggj3a0QCBMAFX0nuMkEZhdAdB0tDwVkEIWxG7YSdDkl0XXzgvfyYV+jjH2AjX+Rr5 NF1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=GSKSDRR8K/+TKUuUq5SwiGfUO6KTVDHzgwHOrlirzyk=; b=cnb3K6rwCrtaCaFzbD4pZ8Bjgz7Uye1EY+lESmfbSd3Q21lHTX8iWGatXR4bYHBv9I KeDzD2JYyvgChLkocRLf6KmDfeFnFIc/u6TSPMpY3BRvWttcCG56UqWBS3Ud1pmFXvg7 pQcO+w/NKSBI8U6646zZPJsBoozuDj+WVAPsFLlijKcy7cWZqqb2MbrdV4wMBtq6HkPk 5Gw2+e+YIELsSg0JxB3taH2XGiauemEVg2WD/XeY0lq9v3hgQFmIWFy5kSNzRIG2U8Rb 1JW3TMo4i6ZcbICgtA4FBpRaPnfyeGkondVX2u2sS2/PK8GQ8OQ3ELE5HFiwzTHUyeiN SYbQ== X-Gm-Message-State: AOAM533eZU++o6UtD2Wvz8jN5lgJ4R5nqzQiAoW5o+EZDuUnmc3bmSu9 QSXasq9bZAQsRTfAEuRVBIcMYdPwllvwzQ== X-Google-Smtp-Source: ABdhPJx5TBEnjr5vEOFv+PxN5aWsXsCncd4fanIQGwc7uRj2/+E+lXhHxJtEXiPIwMXWcJjYvl7R7A== X-Received: by 2002:a17:902:8c84:b029:d9:471:f0da with SMTP id t4-20020a1709028c84b02900d90471f0damr5225230plo.84.1605727175286; Wed, 18 Nov 2020 11:19:35 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:8b43]) by smtp.gmail.com with ESMTPSA id l9sm3197221pjy.10.2020.11.18.11.19.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Nov 2020 11:19:34 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v2 08/13] btrfs-progs: receive: process encoded_write commands Date: Wed, 18 Nov 2020 11:18:55 -0800 Message-Id: X-Mailer: git-send-email 2.29.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov Add a new btrfs_send_op and support for both dumping and proper receive processing which does actual encoded writes. Encoded writes are only allowed on a file descriptor opened with an extra flag that allows encoded writes, so we also add support for this flag when opening or reusing a file for writing. Signed-off-by: Boris Burkov --- cmds/receive-dump.c | 16 +++++- cmds/receive.c | 118 ++++++++++++++++++++++++++++++++++++++----- common/send-stream.c | 22 ++++++++ common/send-stream.h | 4 ++ stubs.h | 56 ++++++++++++++++++++ 5 files changed, 203 insertions(+), 13 deletions(-) diff --git a/cmds/receive-dump.c b/cmds/receive-dump.c index 648d9314..20ec2b70 100644 --- a/cmds/receive-dump.c +++ b/cmds/receive-dump.c @@ -316,6 +316,19 @@ static int print_update_extent(const char *path, u64 offset, u64 len, offset, len); } +static int print_encoded_write(const char *path, const void *data, u64 offset, + u64 len, u64 unencoded_file_len, + u64 unencoded_len, u64 unencoded_offset, + u32 compression, u32 encryption, void *user) +{ + return PRINT_DUMP(user, path, "encoded_write", + "offset=%llu len=%llu, unencoded_file_len=%llu, " + "unencoded_len=%llu, unencoded_offset=%llu, " + "compression=%u, encryption=%u", + offset, len, unencoded_file_len, unencoded_len, + unencoded_offset, compression, encryption); +} + struct btrfs_send_ops btrfs_print_send_ops = { .subvol = print_subvol, .snapshot = print_snapshot, @@ -337,5 +350,6 @@ struct btrfs_send_ops btrfs_print_send_ops = { .chmod = print_chmod, .chown = print_chown, .utimes = print_utimes, - .update_extent = print_update_extent + .update_extent = print_update_extent, + .encoded_write = print_encoded_write, }; diff --git a/cmds/receive.c b/cmds/receive.c index 2c56cea6..87f47a9a 100644 --- a/cmds/receive.c +++ b/cmds/receive.c @@ -30,12 +30,14 @@ #include #include #include +#include #include #include #include #include #include +#include #include #include @@ -52,6 +54,7 @@ #include "cmds/receive-dump.h" #include "common/help.h" #include "common/path-utils.h" +#include "stubs.h" struct btrfs_receive { @@ -60,6 +63,7 @@ struct btrfs_receive int write_fd; char write_path[PATH_MAX]; + int write_fd_allow_encoded; char *root_path; char *dest_dir_path; /* relative to root_path */ @@ -643,28 +647,69 @@ out: return ret; } -static int open_inode_for_write(struct btrfs_receive *rctx, const char *path) +static int set_write_fd_allow_encoded(struct btrfs_receive *rctx) +{ + int ret; + int flags; + + flags = fcntl(rctx->write_fd, F_GETFL); + if (flags < 0) { + ret = -errno; + error("failed to fetch old fd flags"); + goto close_fd; + } + ret = fcntl(rctx->write_fd, F_SETFL, flags | O_ALLOW_ENCODED); + if (ret < 0) { + ret = -errno; + error("failed to enable encoded writes"); + goto close_fd; + } + rctx->write_fd_allow_encoded = true; + ret = 0; + goto out; +close_fd: + close(rctx->write_fd); + rctx->write_fd = -1; + rctx->write_fd_allow_encoded = false; +out: + return ret; +} + +static int open_inode_for_write(struct btrfs_receive *rctx, const char *path, + bool allow_encoded) { int ret = 0; - - if (rctx->write_fd != -1) { - if (strcmp(rctx->write_path, path) == 0) - goto out; - close(rctx->write_fd); - rctx->write_fd = -1; - } - /* * When opening with O_ALLOW_ENCODED, O_CLOEXEC must also be specified. * We might as well always use it even though we don't exec anything. */ - rctx->write_fd = open(path, O_RDWR | O_CLOEXEC); + int flags = O_RDWR | O_CLOEXEC; + + if (rctx->write_fd != -1) { + /* + * if the existing fd is for this path and the needed flags are + * satisfied, no need to open a new one + */ + if (strcmp(rctx->write_path, path) == 0) { + /* fixup the allow encoded flag, if necessary */ + if (allow_encoded && !rctx->write_fd_allow_encoded) + ret = set_write_fd_allow_encoded(rctx); + goto out; + } + close(rctx->write_fd); + rctx->write_fd = -1; + } + + if (allow_encoded) + flags |= O_ALLOW_ENCODED; + rctx->write_fd = open(path, flags); if (rctx->write_fd < 0) { ret = -errno; error("cannot open %s: %m", path); goto out; } strncpy_null(rctx->write_path, path); + rctx->write_fd_allow_encoded = allow_encoded; out: return ret; @@ -695,7 +740,7 @@ static int process_write(const char *path, const void *data, u64 offset, goto out; } - ret = open_inode_for_write(rctx, full_path); + ret = open_inode_for_write(rctx, full_path, false); if (ret < 0) goto out; @@ -738,7 +783,7 @@ static int process_clone(const char *path, u64 offset, u64 len, goto out; } - ret = open_inode_for_write(rctx, full_path); + ret = open_inode_for_write(rctx, full_path, false); if (ret < 0) goto out; @@ -1032,6 +1077,54 @@ static int process_update_extent(const char *path, u64 offset, u64 len, return 0; } +static int process_encoded_write(const char *path, const void *data, u64 offset, + u64 len, u64 unencoded_file_len, u64 unencoded_len, + u64 unencoded_offset, u32 compression, u32 encryption, void *user) +{ + int ret; + ssize_t w; + struct btrfs_receive *rctx = user; + char full_path[PATH_MAX]; + struct encoded_iov encoded = { + .len = unencoded_file_len, + .unencoded_len = unencoded_len, + .unencoded_offset = unencoded_offset, + .compression = compression, + .encryption = encryption, + }; + struct iovec iov[2] = { + { &encoded, sizeof(encoded) }, + { (char *)data, len } + }; + + if (encryption) { + error("encoded_write: encryption not supported"); + return -EOPNOTSUPP; + } + + ret = path_cat_out(full_path, rctx->full_subvol_path, path); + if (ret < 0) { + error("encoded_write: path invalid: %s", path); + return ret; + } + + ret = open_inode_for_write(rctx, full_path, true); + if (ret < 0) + return ret; + + /* + * NOTE: encoded writes guarantee no partial writes, so we don't need to + * handle that possibility. + */ + w = pwritev2(rctx->write_fd, iov, 2, offset, RWF_ENCODED); + if (w < 0) { + ret = -errno; + error("encoded_write: writing to %s failed: %m", path); + return ret; + } + return 0; +} + static struct btrfs_send_ops send_ops = { .subvol = process_subvol, .snapshot = process_snapshot, @@ -1054,6 +1147,7 @@ static struct btrfs_send_ops send_ops = { .chown = process_chown, .utimes = process_utimes, .update_extent = process_update_extent, + .encoded_write = process_encoded_write, }; static int do_receive(struct btrfs_receive *rctx, const char *tomnt, diff --git a/common/send-stream.c b/common/send-stream.c index 77d5cd04..1376e00b 100644 --- a/common/send-stream.c +++ b/common/send-stream.c @@ -354,6 +354,8 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx) struct timespec mt; u8 uuid[BTRFS_UUID_SIZE]; u8 clone_uuid[BTRFS_UUID_SIZE]; + u32 compression; + u32 encryption; u64 tmp; u64 tmp2; u64 ctransid; @@ -362,6 +364,9 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx) u64 dev; u64 clone_offset; u64 offset; + u64 unencoded_file_len; + u64 unencoded_len; + u64 unencoded_offset; int len; int xattr_len; @@ -436,6 +441,23 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx) TLV_GET(sctx, BTRFS_SEND_A_DATA, &data, &len); ret = sctx->ops->write(path, data, offset, len, sctx->user); break; + case BTRFS_SEND_C_ENCODED_WRITE: + TLV_GET_STRING(sctx, BTRFS_SEND_A_PATH, &path); + TLV_GET_U64(sctx, BTRFS_SEND_A_FILE_OFFSET, &offset); + TLV_GET_U64(sctx, BTRFS_SEND_A_UNENCODED_FILE_LEN, + &unencoded_file_len); + TLV_GET_U64(sctx, BTRFS_SEND_A_UNENCODED_LEN, &unencoded_len); + TLV_GET_U64(sctx, BTRFS_SEND_A_UNENCODED_OFFSET, + &unencoded_offset); + TLV_GET_U32(sctx, BTRFS_SEND_A_COMPRESSION, &compression); + TLV_GET_U32(sctx, BTRFS_SEND_A_ENCRYPTION, &encryption); + TLV_GET(sctx, BTRFS_SEND_A_DATA, &data, &len); + ret = sctx->ops->encoded_write(path, data, offset, len, + unencoded_file_len, + unencoded_len, unencoded_offset, + compression, encryption, + sctx->user); + break; case BTRFS_SEND_C_CLONE: TLV_GET_STRING(sctx, BTRFS_SEND_A_PATH, &path); TLV_GET_U64(sctx, BTRFS_SEND_A_FILE_OFFSET, &offset); diff --git a/common/send-stream.h b/common/send-stream.h index 39901f86..607bc007 100644 --- a/common/send-stream.h +++ b/common/send-stream.h @@ -66,6 +66,10 @@ struct btrfs_send_ops { struct timespec *mt, struct timespec *ct, void *user); int (*update_extent)(const char *path, u64 offset, u64 len, void *user); + int (*encoded_write)(const char *path, const void *data, u64 offset, + u64 len, u64 unencoded_file_len, u64 unencoded_len, + u64 unencoded_offset, u32 compression, + u32 encryption, void *user); }; int btrfs_read_and_process_send_stream(int fd, diff --git a/stubs.h b/stubs.h index b39f8a69..7574ae8f 100644 --- a/stubs.h +++ b/stubs.h @@ -1,6 +1,8 @@ #ifndef _BTRFS_STUBS_H #define _BTRFS_STUBS_H +#include +#include #include struct iovec; @@ -8,4 +10,58 @@ struct iovec; ssize_t pwritev2(int fd, const struct iovec *iov, int iovcnt, off_t offset, int flags); +#ifndef O_ALLOW_ENCODED +#if defined(__alpha__) +#define O_ALLOW_ENCODED 0200000000 +#elif defined(__hppa__) +#define O_ALLOW_ENCODED 100000000 +#elif defined(__sparc__) +#define O_ALLOW_ENCODED 0x8000000 +#else +#define O_ALLOW_ENCODED 040000000 #endif +#endif + +#ifndef RWF_ENCODED + +enum { + ENCODED_IOV_COMPRESSION_NONE, +#define ENCODED_IOV_COMPRESSION_NONE ENCODED_IOV_COMPRESSION_NONE + ENCODED_IOV_COMPRESSION_BTRFS_ZLIB, +#define ENCODED_IOV_COMPRESSION_BTRFS_ZLIB ENCODED_IOV_COMPRESSION_BTRFS_ZLIB + ENCODED_IOV_COMPRESSION_BTRFS_ZSTD, +#define ENCODED_IOV_COMPRESSION_BTRFS_ZSTD ENCODED_IOV_COMPRESSION_BTRFS_ZSTD + ENCODED_IOV_COMPRESSION_BTRFS_LZO_4K, +#define ENCODED_IOV_COMPRESSION_BTRFS_LZO_4K ENCODED_IOV_COMPRESSION_BTRFS_LZO_4K + ENCODED_IOV_COMPRESSION_BTRFS_LZO_8K, +#define ENCODED_IOV_COMPRESSION_BTRFS_LZO_8K ENCODED_IOV_COMPRESSION_BTRFS_LZO_8K + ENCODED_IOV_COMPRESSION_BTRFS_LZO_16K, +#define ENCODED_IOV_COMPRESSION_BTRFS_LZO_16K ENCODED_IOV_COMPRESSION_BTRFS_LZO_16K + ENCODED_IOV_COMPRESSION_BTRFS_LZO_32K, +#define ENCODED_IOV_COMPRESSION_BTRFS_LZO_32K ENCODED_IOV_COMPRESSION_BTRFS_LZO_32K + ENCODED_IOV_COMPRESSION_BTRFS_LZO_64K, +#define ENCODED_IOV_COMPRESSION_BTRFS_LZO_64K ENCODED_IOV_COMPRESSION_BTRFS_LZO_64K + ENCODED_IOV_COMPRESSION_TYPES = ENCODED_IOV_COMPRESSION_BTRFS_LZO_64K, +}; + +enum { + ENCODED_IOV_ENCRYPTION_NONE, +#define ENCODED_IOV_ENCRYPTION_NONE ENCODED_IOV_ENCRYPTION_NONE + ENCODED_IOV_ENCRYPTION_TYPES = ENCODED_IOV_ENCRYPTION_NONE, +}; + +struct encoded_iov { + __aligned_u64 len; + __aligned_u64 unencoded_len; + __aligned_u64 unencoded_offset; + __u32 compression; + __u32 encryption; +}; + +#define ENCODED_IOV_SIZE_VER0 32 + +/* encoded (e.g., compressed and/or encrypted) IO */ +#define RWF_ENCODED ((__kernel_rwf_t)0x00000020) +#endif /* RWF_ENCODED */ + +#endif /* _BTRFS_STUBS_H */ From patchwork Wed Nov 18 19:18:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11915739 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8AFB5C83011 for ; Wed, 18 Nov 2020 19:19:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2A40C2220B for ; Wed, 18 Nov 2020 19:19:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="nzxpNUj6" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727360AbgKRTTk (ORCPT ); Wed, 18 Nov 2020 14:19:40 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55906 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727350AbgKRTTj (ORCPT ); Wed, 18 Nov 2020 14:19:39 -0500 Received: from mail-pf1-x444.google.com (mail-pf1-x444.google.com [IPv6:2607:f8b0:4864:20::444]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 21C96C0613D4 for ; Wed, 18 Nov 2020 11:19:39 -0800 (PST) Received: by mail-pf1-x444.google.com with SMTP id q10so2065531pfn.0 for ; Wed, 18 Nov 2020 11:19:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Uou+avacjLUgrd6sYNlplF7cpTImH06UMIPWOWB8yac=; b=nzxpNUj6PoGBVmyVq2S5Fp1qk7EKOPe6nNjaOmvNKESVqAaIkL9LtQBkbYPlh17oxN QWNpL4lDd9zQgBUi64ePtOxUDaqQczmWuQOZFHlNYGzDO17xu8DPVkyFiiL/D4wBr5jB vap+fxZY6N98wIMdyKpebAWkqFmWk46TOyHe8UD/y0BefzXbQxXE7pUN1yqSc7FjqfdS ccC3zfmoQ0hL1Yjijm3osiQC5SRVXv9u+itrHDoODV+cyRSGzd4M8iBsIrJzKKC3q+5m 5uDJK0O2S8QT66/15D6WRfaeMBaygMxT9Fi28ILgSIR2xNCPJlRvBQDZRtdUKwbUqdiO o5qQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Uou+avacjLUgrd6sYNlplF7cpTImH06UMIPWOWB8yac=; b=GZL/QEnieft0TYHRf/Xnwkemuyg+PZnDFh++2ssHzTj3ho7hOssnG1aEY/V8VjmnEK /BBd+gT/mfumOI7exg9RVlKPL3H1R8vByqLjfAWNHjvsnKlrsa0TkrazC/xGnzYKKNHb yiC/Wl80n044OxnevG9ns/XaR1JYhEvD9ePAhAdB9moGMgNPbPzAxx3jrzwVxu5AvusT N6bzTLpI4UPw6pchDH4JjsrTFUtNzLXS3gGeA9PdfUYcwoUuQjZel36VwHeaxbxS++Yo roRmS55IdQT2uR1zcov/1hs+mHLJfZkoFW2OKi7B7uZP+bGcA4OaN06U1oU9uzKM/+PB sxcg== X-Gm-Message-State: AOAM531pLKveBZlTKeeqrrXFvdH638gk2MfIStSKuLXjsWkd0BfwMPCN 7yjc1IEJxRSAfiRiDCjlsGNXecvVQ7fNQg== X-Google-Smtp-Source: ABdhPJxeeAcMunQfSTcIXD946l5RX2PtUCFBhHPmRN80hYEI6+DTzpdaJ52wUPqWfy5Qhc4NJuKu6Q== X-Received: by 2002:a05:6a00:16c4:b029:162:bf9f:6458 with SMTP id l4-20020a056a0016c4b0290162bf9f6458mr5955906pfc.55.1605727177373; Wed, 18 Nov 2020 11:19:37 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:8b43]) by smtp.gmail.com with ESMTPSA id l9sm3197221pjy.10.2020.11.18.11.19.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Nov 2020 11:19:36 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v2 09/13] btrfs-progs: receive: encoded_write fallback to explicit decode and write Date: Wed, 18 Nov 2020 11:18:56 -0800 Message-Id: <003ced29dccaac8e5bf41aa4f571f0154d011ae0.1605723745.git.osandov@osandov.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov An encoded_write can fail if the file system it is being applied to does not support encoded writes or if it can't find enough contiguous space to accommodate the encoded extent. In those cases, we can likely still process an encoded_write by explicitly decoding the data and doing a normal write. Add the necessary fallback path for decoding data compressed with zlib, lzo, or zstd. zlib and zstd have reusable decoding context data structures which we cache in the receive context so that we don't have to recreate them on every encoded_write. Finally, add a command line flag for force-decompress which causes receive to always use the fallback path rather than first attempting the encoded write. Signed-off-by: Boris Burkov --- Documentation/btrfs-receive.asciidoc | 4 + cmds/receive.c | 274 +++++++++++++++++++++++++-- 2 files changed, 265 insertions(+), 13 deletions(-) diff --git a/Documentation/btrfs-receive.asciidoc b/Documentation/btrfs-receive.asciidoc index e4c4d2c0..354a71dc 100644 --- a/Documentation/btrfs-receive.asciidoc +++ b/Documentation/btrfs-receive.asciidoc @@ -60,6 +60,10 @@ By default the mountpoint is searched in '/proc/self/mounts'. If '/proc' is not accessible, eg. in a chroot environment, use this option to tell us where this filesystem is mounted. +--force-decompress:: +if the stream contains compressed data (see '--compressed-data' in +`btrfs-send`(8)), always decompress it instead of writing it with encoded I/O. + --dump:: dump the stream metadata, one line per operation + diff --git a/cmds/receive.c b/cmds/receive.c index 87f47a9a..361c7c8a 100644 --- a/cmds/receive.c +++ b/cmds/receive.c @@ -41,6 +41,10 @@ #include #include +#include +#include +#include + #include "kernel-shared/ctree.h" #include "ioctl.h" #include "cmds/commands.h" @@ -82,6 +86,8 @@ struct btrfs_receive int honor_end_cmd; + int force_decompress; + /* * Buffer to store capabilities from security.capabilities xattr, * usually 20 bytes, but make same room for potentially larger @@ -89,6 +95,10 @@ struct btrfs_receive */ char cached_capabilities[64]; int cached_capabilities_len; + + /* Reuse stream objects for encoded_write decompression fallback */ + ZSTD_DStream *zstd_dstream; + z_stream *zlib_stream; }; static int finish_subvol(struct btrfs_receive *rctx) @@ -1077,9 +1087,222 @@ static int process_update_extent(const char *path, u64 offset, u64 len, return 0; } +static int decompress_zlib(struct btrfs_receive *rctx, const char *encoded_data, + u64 encoded_len, char *unencoded_data, + u64 unencoded_len) +{ + bool init = false; + int ret; + + if (!rctx->zlib_stream) { + init = true; + rctx->zlib_stream = malloc(sizeof(z_stream)); + if (!rctx->zlib_stream) { + error("failed to allocate zlib stream %m"); + return -ENOMEM; + } + } + rctx->zlib_stream->next_in = (void *)encoded_data; + rctx->zlib_stream->avail_in = encoded_len; + rctx->zlib_stream->next_out = (void *)unencoded_data; + rctx->zlib_stream->avail_out = unencoded_len; + + if (init) { + rctx->zlib_stream->zalloc = Z_NULL; + rctx->zlib_stream->zfree = Z_NULL; + rctx->zlib_stream->opaque = Z_NULL; + ret = inflateInit(rctx->zlib_stream); + } else { + ret = inflateReset(rctx->zlib_stream); + } + if (ret != Z_OK) { + error("zlib inflate init failed: %d", ret); + return -EIO; + } + + while (rctx->zlib_stream->avail_in > 0 && + rctx->zlib_stream->avail_out > 0) { + ret = inflate(rctx->zlib_stream, Z_FINISH); + if (ret == Z_STREAM_END) { + break; + } else if (ret != Z_OK) { + error("zlib inflate failed: %d", ret); + return -EIO; + } + } + return 0; +} + +static int decompress_zstd(struct btrfs_receive *rctx, const char *encoded_buf, + u64 encoded_len, char *unencoded_buf, + u64 unencoded_len) +{ + ZSTD_inBuffer in_buf = { + .src = encoded_buf, + .size = encoded_len + }; + ZSTD_outBuffer out_buf = { + .dst = unencoded_buf, + .size = unencoded_len + }; + size_t ret; + + if (!rctx->zstd_dstream) { + rctx->zstd_dstream = ZSTD_createDStream(); + if (!rctx->zstd_dstream) { + error("failed to create zstd dstream"); + return -ENOMEM; + } + } + ret = ZSTD_initDStream(rctx->zstd_dstream); + if (ZSTD_isError(ret)) { + error("failed to init zstd stream: %s", ZSTD_getErrorName(ret)); + return -EIO; + } + while (in_buf.pos < in_buf.size && out_buf.pos < out_buf.size) { + ret = ZSTD_decompressStream(rctx->zstd_dstream, &out_buf, &in_buf); + if (ret == 0) { + break; + } else if (ZSTD_isError(ret)) { + error("failed to decompress zstd stream: %s", + ZSTD_getErrorName(ret)); + return -EIO; + } + } + return 0; +} + +static int decompress_lzo(const char *encoded_data, u64 encoded_len, + char *unencoded_data, u64 unencoded_len, + unsigned int page_size) +{ + uint32_t total_len; + size_t in_pos, out_pos; + + if (encoded_len < 4) { + error("lzo header is truncated"); + return -EIO; + } + memcpy(&total_len, encoded_data, 4); + total_len = le32toh(total_len); + if (total_len > encoded_len) { + error("lzo header is invalid"); + return -EIO; + } + + in_pos = 4; + out_pos = 0; + while (in_pos < total_len && out_pos < unencoded_len) { + size_t page_remaining; + uint32_t src_len; + lzo_uint dst_len; + int ret; + + page_remaining = -in_pos % page_size; + if (page_remaining < 4) { + if (total_len - in_pos <= page_remaining) + break; + in_pos += page_remaining; + } + + if (total_len - in_pos < 4) { + error("lzo segment header is truncated"); + return -EIO; + } + + memcpy(&src_len, encoded_data + in_pos, 4); + src_len = le32toh(src_len); + in_pos += 4; + if (src_len > total_len - in_pos) { + error("lzo segment header is invalid"); + return -EIO; + } + + dst_len = page_size; + ret = lzo1x_decompress_safe((void *)(encoded_data + in_pos), + src_len, + (void *)(unencoded_data + out_pos), + &dst_len, NULL); + if (ret != LZO_E_OK) { + error("lzo1x_decompress_safe failed: %d", ret); + return -EIO; + } + + in_pos += src_len; + out_pos += dst_len; + } + return 0; +} + +static int decompress_and_write(struct btrfs_receive *rctx, + const char *encoded_data, u64 offset, + u64 encoded_len, u64 unencoded_file_len, + u64 unencoded_len, u64 unencoded_offset, + u32 compression) +{ + int ret = 0; + size_t pos; + ssize_t w; + char *unencoded_data; + int page_shift; + + unencoded_data = calloc(unencoded_len, 1); + if (!unencoded_data) { + error("allocating space for unencoded data failed: %m"); + return -errno; + } + + switch (compression) { + case ENCODED_IOV_COMPRESSION_BTRFS_ZLIB: + ret = decompress_zlib(rctx, encoded_data, encoded_len, + unencoded_data, unencoded_len); + if (ret) + goto out; + break; + case ENCODED_IOV_COMPRESSION_BTRFS_ZSTD: + ret = decompress_zstd(rctx, encoded_data, encoded_len, + unencoded_data, unencoded_len); + if (ret) + goto out; + break; + case ENCODED_IOV_COMPRESSION_BTRFS_LZO_4K: + case ENCODED_IOV_COMPRESSION_BTRFS_LZO_8K: + case ENCODED_IOV_COMPRESSION_BTRFS_LZO_16K: + case ENCODED_IOV_COMPRESSION_BTRFS_LZO_32K: + case ENCODED_IOV_COMPRESSION_BTRFS_LZO_64K: + page_shift = compression - ENCODED_IOV_COMPRESSION_BTRFS_LZO_4K + 12; + ret = decompress_lzo(encoded_data, encoded_len, unencoded_data, + unencoded_len, 1U << page_shift); + if (ret) + goto out; + break; + default: + error("unknown compression: %d", compression); + ret = -EOPNOTSUPP; + goto out; + } + + pos = unencoded_offset; + while (pos < unencoded_file_len) { + w = pwrite(rctx->write_fd, unencoded_data + pos, + unencoded_file_len - pos, offset); + if (w < 0) { + ret = -errno; + error("writing unencoded data failed: %m"); + goto out; + } + pos += w; + offset += w; + } +out: + free(unencoded_data); + return ret; +} + static int process_encoded_write(const char *path, const void *data, u64 offset, - u64 len, u64 unencoded_file_len, u64 unencoded_len, - u64 unencoded_offset, u32 compression, u32 encryption, void *user) + u64 len, u64 unencoded_file_len, + u64 unencoded_len, u64 unencoded_offset, + u32 compression, u32 encryption, void *user) { int ret; ssize_t w; @@ -1096,6 +1319,7 @@ static int process_encoded_write(const char *path, const void *data, u64 offset, { &encoded, sizeof(encoded) }, { (char *)data, len } }; + bool encoded_write = !rctx->force_decompress; if (encryption) { error("encoded_write: encryption not supported"); @@ -1112,17 +1336,25 @@ static int process_encoded_write(const char *path, const void *data, u64 offset, if (ret < 0) return ret; - /* - * NOTE: encoded writes guarantee no partial writes, so we don't need to - * handle that possibility. - */ - w = pwritev2(rctx->write_fd, iov, 2, offset, RWF_ENCODED); - if (w < 0) { - ret = -errno; - error("encoded_write: writing to %s failed: %m", path); - return ret; + if (encoded_write) { + /* + * NOTE: encoded writes guarantee no partial writes, so we don't + * need to handle that possibility. + */ + w = pwritev2(rctx->write_fd, iov, 2, offset, RWF_ENCODED); + if (w >= 0) + return 0; + /* Fall back for these errors, fail hard for anything else. */ + if (errno != ENOSPC && errno != EOPNOTSUPP && errno != EINVAL) { + ret = -errno; + error("encoded_write: writing to %s failed: %m", path); + return ret; + } } - return 0; + + return decompress_and_write(rctx, data, offset, len, unencoded_file_len, + unencoded_len, unencoded_offset, + compression); } static struct btrfs_send_ops send_ops = { @@ -1310,6 +1542,12 @@ out: close(rctx->dest_dir_fd); rctx->dest_dir_fd = -1; } + if (rctx->zstd_dstream) + ZSTD_freeDStream(rctx->zstd_dstream); + if (rctx->zlib_stream) { + inflateEnd(rctx->zlib_stream); + free(rctx->zlib_stream); + } return ret; } @@ -1340,6 +1578,9 @@ static const char * const cmd_receive_usage[] = { "-m ROOTMOUNT the root mount point of the destination filesystem.", " If /proc is not accessible, use this to tell us where", " this file system is mounted.", + "--force-decompress", + " if the stream contains compressed data, always", + " decompress it instead of writing it with encoded I/O", "--dump dump stream metadata, one line per operation,", " does not require the MOUNT parameter", "-v deprecated, alias for global -v option", @@ -1383,12 +1624,16 @@ static int cmd_receive(const struct cmd_struct *cmd, int argc, char **argv) optind = 0; while (1) { int c; - enum { GETOPT_VAL_DUMP = 257 }; + enum { + GETOPT_VAL_DUMP = 257, + GETOPT_VAL_FORCE_DECOMPRESS, + }; static const struct option long_opts[] = { { "max-errors", required_argument, NULL, 'E' }, { "chroot", no_argument, NULL, 'C' }, { "dump", no_argument, NULL, GETOPT_VAL_DUMP }, { "quiet", no_argument, NULL, 'q' }, + { "force-decompress", no_argument, NULL, GETOPT_VAL_FORCE_DECOMPRESS }, { NULL, 0, NULL, 0 } }; @@ -1431,6 +1676,9 @@ static int cmd_receive(const struct cmd_struct *cmd, int argc, char **argv) case GETOPT_VAL_DUMP: dump = 1; break; + case GETOPT_VAL_FORCE_DECOMPRESS: + rctx.force_decompress = 1; + break; default: usage_unknown_option(cmd, argv); } From patchwork Wed Nov 18 19:18:57 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11915741 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3372C64E7C for ; Wed, 18 Nov 2020 19:19:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6F1402220B for ; Wed, 18 Nov 2020 19:19:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="yrvZIKPZ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727370AbgKRTTn (ORCPT ); Wed, 18 Nov 2020 14:19:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55918 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727363AbgKRTTm (ORCPT ); Wed, 18 Nov 2020 14:19:42 -0500 Received: from mail-pf1-x441.google.com (mail-pf1-x441.google.com [IPv6:2607:f8b0:4864:20::441]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 37E89C0613D4 for ; Wed, 18 Nov 2020 11:19:42 -0800 (PST) Received: by mail-pf1-x441.google.com with SMTP id 10so2055668pfp.5 for ; Wed, 18 Nov 2020 11:19:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=V3j6zZr0VDMJvc7zZVcwrRVz+yGAYinHXHs1QBoNG4M=; b=yrvZIKPZURujIFP5Vio+JkFgfLuiFrd6IC2/KC0DAKghdLiaqSAXvFG9Dw96eg0IpQ U5rjt9W9lz2AkcBEFS8Ab6Euy/SWTtRYRPJb8OsfF3zOg0QD+pBao9NOBcTwI8ZZdljO uY2g0IaXfVk62dwcTpK7uZX2m+lBtjqRxI71QTPhMWrigsBzEJe0EKQILrVB6grrFOnm 7x458/pZCZ/bHxA5h6pjBNrzsTo5pOMJaDmmEw2U8qAXVhFEo8P6fhOLMFNOsuISFTTv eWHrrCH6zKfHYV6B+o//rJVn0Ch7v9Ce3rD9yovGPDuqCTueC0BP7o2MrVGAB5fMSY45 ctWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=V3j6zZr0VDMJvc7zZVcwrRVz+yGAYinHXHs1QBoNG4M=; b=rqjx7xKNnmWnQAc//m7ZpsKUL5TQ7ZyV8QRhawCp7BtzKHn2QrUA3KIcd8Z7dx9oH3 rLqjJUGJmmrYdCCcuLwzn0hJGl8pJjwX07IPw19cWuoDJfW54T8EYBIUwXUQKDUh/4OQ pPqhsHu5J8pldHQkr1cN+BRu+vMOPdDuqS/KcMAflj9wG3Skz/0VTLz66w4dIfF+Zwmz SjgDN/lu8EwB1FvjXsiv+uoh2f3nVLZHQFZyow13e5nHsNGbKIiTyyjI3h/EHaoU5QWi ieaDShljjRNU5dLEyX6gAOSPU4ZF2zG77G0bswDE15n3rpsQVW1uyAmXtyNlSqeJrb4b uIww== X-Gm-Message-State: AOAM533Z3u0dxrcR4Fwmw4IEWjMGjBPiboh/xn+xFjAatrYtkEqh0Fu4 tTHgeE1zADRxW4LUGX06cJhofcTDR/2tSA== X-Google-Smtp-Source: ABdhPJwTZa7tfZEaS8Ag0NQtDtafbzxzHuL/EumP+kMYVnhhH9fW02xHaqmCMvVWjSBDKP9VAIDR1g== X-Received: by 2002:a17:90b:705:: with SMTP id s5mr547489pjz.22.1605727181145; Wed, 18 Nov 2020 11:19:41 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:8b43]) by smtp.gmail.com with ESMTPSA id l9sm3197221pjy.10.2020.11.18.11.19.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Nov 2020 11:19:39 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v2 10/13] btrfs-progs: receive: process fallocate commands Date: Wed, 18 Nov 2020 11:18:57 -0800 Message-Id: X-Mailer: git-send-email 2.29.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov Send stream v2 can emit fallocate commands, so receive must support them as well. The implementation simply passes along the arguments to the syscall. Note that mode is encoded as a u32 in send stream but fallocate takes an int, so there is a unsigned->signed conversion there. Signed-off-by: Boris Burkov --- cmds/receive-dump.c | 9 +++++++++ cmds/receive.c | 25 +++++++++++++++++++++++++ common/send-stream.c | 9 +++++++++ common/send-stream.h | 2 ++ 4 files changed, 45 insertions(+) diff --git a/cmds/receive-dump.c b/cmds/receive-dump.c index 20ec2b70..acc0ba32 100644 --- a/cmds/receive-dump.c +++ b/cmds/receive-dump.c @@ -329,6 +329,14 @@ static int print_encoded_write(const char *path, const void *data, u64 offset, unencoded_offset, compression, encryption); } +static int print_fallocate(const char *path, int mode, u64 offset, u64 len, + void *user) +{ + return PRINT_DUMP(user, path, "fallocate", + "mode=%d offset=%llu len=%llu", + mode, offset, len); +} + struct btrfs_send_ops btrfs_print_send_ops = { .subvol = print_subvol, .snapshot = print_snapshot, @@ -352,4 +360,5 @@ struct btrfs_send_ops btrfs_print_send_ops = { .utimes = print_utimes, .update_extent = print_update_extent, .encoded_write = print_encoded_write, + .fallocate = print_fallocate, }; diff --git a/cmds/receive.c b/cmds/receive.c index 361c7c8a..ca9c3f5a 100644 --- a/cmds/receive.c +++ b/cmds/receive.c @@ -1357,6 +1357,30 @@ static int process_encoded_write(const char *path, const void *data, u64 offset, compression); } +static int process_fallocate(const char *path, int mode, u64 offset, u64 len, + void *user) +{ + int ret; + struct btrfs_receive *rctx = user; + char full_path[PATH_MAX]; + + ret = path_cat_out(full_path, rctx->full_subvol_path, path); + if (ret < 0) { + error("fallocate: path invalid: %s", path); + return ret; + } + ret = open_inode_for_write(rctx, full_path, false); + if (ret < 0) + return ret; + ret = fallocate(rctx->write_fd, mode, offset, len); + if (ret < 0) { + ret = -errno; + error("fallocate: fallocate on %s failed: %m", path); + return ret; + } + return 0; +} + static struct btrfs_send_ops send_ops = { .subvol = process_subvol, .snapshot = process_snapshot, @@ -1380,6 +1404,7 @@ static struct btrfs_send_ops send_ops = { .utimes = process_utimes, .update_extent = process_update_extent, .encoded_write = process_encoded_write, + .fallocate = process_fallocate, }; static int do_receive(struct btrfs_receive *rctx, const char *tomnt, diff --git a/common/send-stream.c b/common/send-stream.c index 1376e00b..d455cdfb 100644 --- a/common/send-stream.c +++ b/common/send-stream.c @@ -369,6 +369,7 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx) u64 unencoded_offset; int len; int xattr_len; + int fallocate_mode; ret = read_cmd(sctx); if (ret) @@ -514,6 +515,14 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx) case BTRFS_SEND_C_END: ret = 1; break; + case BTRFS_SEND_C_FALLOCATE: + TLV_GET_STRING(sctx, BTRFS_SEND_A_PATH, &path); + TLV_GET_U32(sctx, BTRFS_SEND_A_FALLOCATE_MODE, &fallocate_mode); + TLV_GET_U64(sctx, BTRFS_SEND_A_FILE_OFFSET, &offset); + TLV_GET_U64(sctx, BTRFS_SEND_A_SIZE, &tmp); + ret = sctx->ops->fallocate(path, fallocate_mode, offset, tmp, + sctx->user); + break; } tlv_get_failed: diff --git a/common/send-stream.h b/common/send-stream.h index 607bc007..a58739bb 100644 --- a/common/send-stream.h +++ b/common/send-stream.h @@ -70,6 +70,8 @@ struct btrfs_send_ops { u64 len, u64 unencoded_file_len, u64 unencoded_len, u64 unencoded_offset, u32 compression, u32 encryption, void *user); + int (*fallocate)(const char *path, int mode, u64 offset, u64 len, + void *user); }; int btrfs_read_and_process_send_stream(int fd, From patchwork Wed Nov 18 19:18:58 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11915745 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4F36C5519F for ; Wed, 18 Nov 2020 19:20:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 73EBE22227 for ; Wed, 18 Nov 2020 19:20:33 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="ctJn0jyR" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727384AbgKRTTq (ORCPT ); Wed, 18 Nov 2020 14:19:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55928 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727363AbgKRTTq (ORCPT ); Wed, 18 Nov 2020 14:19:46 -0500 Received: from mail-pl1-x644.google.com (mail-pl1-x644.google.com [IPv6:2607:f8b0:4864:20::644]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8BD77C0613D4 for ; Wed, 18 Nov 2020 11:19:44 -0800 (PST) Received: by mail-pl1-x644.google.com with SMTP id b3so1521711pls.11 for ; Wed, 18 Nov 2020 11:19:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=YMykUAF1jBHiR3hoO57Gb2atrtMf3A6zHbbf8nZiQAw=; b=ctJn0jyRQWnOpDMJwzJ+5wGIlsnQ3VJ3TXetiT77zEVZfeZInXDICwBHskcwPaagfw hVFzGIgVWpn/RmBRX3h+aVFw4RBIaLTXiMBWXUP0Pbb9VJttVxZuBwkUofvTQBbzVOgz Kct6Hqn5DOOCDvLZw1BEHq3M9yE8NW0PIpiunBFr4tWy31Zz/S5wczAJcdR2eHrNtyaB V3rR9mK5JV/NaVSjDhsX4Qki6XSsWcY3lY1WCPTrVuSxwqaEMbIxnBqRUAZCzUkWlIGs sDCcta50NGCyweNXAWog7CT4ioEj8vmkYsz98Y5K4Xn8bOj5F2rgUWIjojE5CAtLtt0o Mxog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=YMykUAF1jBHiR3hoO57Gb2atrtMf3A6zHbbf8nZiQAw=; b=oYtGPe1sumAD6+J1EJc+fK/i9BSmI/rvEK5XAQccqiqshnjJ+jR9T7q/2fGPJASH/u HnD5qxpjl8wvp3qENNPgfXRtxrFidabwyZf/187xoAxWSnjUtWhl6+KeBHqoxFdJDo7m Nmj0d9egkywW2qVABd+6ZrjX52hyzwQU3iI8GxSm74rf61usFTgDkei9QQ3vA1lsGQFA XPZcXzK4d/sL2S3BWK00Sauss6LqutbJ5WUszHjoHiUnpn0hQREUty8zdXXBLdAseAha GR8BEWm/SWfFRjVtS4sttmpTSEpPOrEd2dp5yn+jrPH/jjQ5WxfpPqipTwJXZA5iCAn8 rHew== X-Gm-Message-State: AOAM531uxA4zl1X8Cqcww094UH37U4BlQ/etwmhui9syt+sWMgRmM0wC /Nj+YGECnsc88Gpb1IAYyvOeGYK6WeEQvg== X-Google-Smtp-Source: ABdhPJyjyV39XZbRXV15OJ4+YJbkxyshR9NPZsrXzqNxm89Fj2aeHfQnMcTtwS6DMunK8J7vfAGITQ== X-Received: by 2002:a17:902:9b93:b029:d6:d0e3:10c6 with SMTP id y19-20020a1709029b93b02900d6d0e310c6mr5855545plp.41.1605727183352; Wed, 18 Nov 2020 11:19:43 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:8b43]) by smtp.gmail.com with ESMTPSA id l9sm3197221pjy.10.2020.11.18.11.19.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Nov 2020 11:19:42 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v2 11/13] btrfs-progs: receive: process setflags ioctl commands Date: Wed, 18 Nov 2020 11:18:58 -0800 Message-Id: <40b7cf2fb2602cc6029eb36c15eab361ac4e5340.1605723745.git.osandov@osandov.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov In send stream v2, send can emit a command for setting inode flags via the setflags ioctl. Pass the flags attribute through to the ioctl call in receive. Signed-off-by: Boris Burkov --- cmds/receive-dump.c | 6 ++++++ cmds/receive.c | 24 ++++++++++++++++++++++++ common/send-stream.c | 7 +++++++ common/send-stream.h | 1 + 4 files changed, 38 insertions(+) diff --git a/cmds/receive-dump.c b/cmds/receive-dump.c index acc0ba32..40f07ad4 100644 --- a/cmds/receive-dump.c +++ b/cmds/receive-dump.c @@ -337,6 +337,11 @@ static int print_fallocate(const char *path, int mode, u64 offset, u64 len, mode, offset, len); } +static int print_setflags(const char *path, int flags, void *user) +{ + return PRINT_DUMP(user, path, "setflags", "flags=%d", flags); +} + struct btrfs_send_ops btrfs_print_send_ops = { .subvol = print_subvol, .snapshot = print_snapshot, @@ -361,4 +366,5 @@ struct btrfs_send_ops btrfs_print_send_ops = { .update_extent = print_update_extent, .encoded_write = print_encoded_write, .fallocate = print_fallocate, + .setflags = print_setflags, }; diff --git a/cmds/receive.c b/cmds/receive.c index ca9c3f5a..1863f881 100644 --- a/cmds/receive.c +++ b/cmds/receive.c @@ -1381,6 +1381,29 @@ static int process_fallocate(const char *path, int mode, u64 offset, u64 len, return 0; } +static int process_setflags(const char *path, int flags, void *user) +{ + int ret; + struct btrfs_receive *rctx = user; + char full_path[PATH_MAX]; + + ret = path_cat_out(full_path, rctx->full_subvol_path, path); + if (ret < 0) { + error("setflags: path invalid: %s", path); + return ret; + } + ret = open_inode_for_write(rctx, full_path, false); + if (ret < 0) + return ret; + ret = ioctl(rctx->write_fd, FS_IOC_SETFLAGS, &flags); + if (ret < 0) { + ret = -errno; + error("setflags: setflags ioctl on %s failed: %m", path); + return ret; + } + return 0; +} + static struct btrfs_send_ops send_ops = { .subvol = process_subvol, .snapshot = process_snapshot, @@ -1405,6 +1428,7 @@ static struct btrfs_send_ops send_ops = { .update_extent = process_update_extent, .encoded_write = process_encoded_write, .fallocate = process_fallocate, + .setflags = process_setflags, }; static int do_receive(struct btrfs_receive *rctx, const char *tomnt, diff --git a/common/send-stream.c b/common/send-stream.c index d455cdfb..da0c0e5d 100644 --- a/common/send-stream.c +++ b/common/send-stream.c @@ -370,6 +370,7 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx) int len; int xattr_len; int fallocate_mode; + int setflags_flags; ret = read_cmd(sctx); if (ret) @@ -523,8 +524,14 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx) ret = sctx->ops->fallocate(path, fallocate_mode, offset, tmp, sctx->user); break; + case BTRFS_SEND_C_SETFLAGS: + TLV_GET_STRING(sctx, BTRFS_SEND_A_PATH, &path); + TLV_GET_U32(sctx, BTRFS_SEND_A_SETFLAGS_FLAGS, &setflags_flags); + ret = sctx->ops->setflags(path, setflags_flags, sctx->user); + break; } + tlv_get_failed: out: free(path); diff --git a/common/send-stream.h b/common/send-stream.h index a58739bb..5373bf69 100644 --- a/common/send-stream.h +++ b/common/send-stream.h @@ -72,6 +72,7 @@ struct btrfs_send_ops { u32 encryption, void *user); int (*fallocate)(const char *path, int mode, u64 offset, u64 len, void *user); + int (*setflags)(const char *path, int flags, void *user); }; int btrfs_read_and_process_send_stream(int fd, From patchwork Wed Nov 18 19:18:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11915747 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15DBCC56202 for ; Wed, 18 Nov 2020 19:20:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B1CFC2222C for ; Wed, 18 Nov 2020 19:20:33 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="Uz9ORVnV" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727388AbgKRTTr (ORCPT ); Wed, 18 Nov 2020 14:19:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55938 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727363AbgKRTTr (ORCPT ); Wed, 18 Nov 2020 14:19:47 -0500 Received: from mail-pl1-x641.google.com (mail-pl1-x641.google.com [IPv6:2607:f8b0:4864:20::641]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 31A4FC0613D4 for ; Wed, 18 Nov 2020 11:19:47 -0800 (PST) Received: by mail-pl1-x641.google.com with SMTP id bj5so718202plb.4 for ; Wed, 18 Nov 2020 11:19:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=andgHX9wkZy2Trx0HRVKpuTygds8q/VC0NacIobtvWI=; b=Uz9ORVnVyGjkjVWIwMCuxG++lNNP+7VuE0dYpsFjPP38DN+VEoWSvf5IUHHinpw9ag 4wvpqvlQ9/uDVPXXepKmA2bcoE4scjaqma4iEiMGFSxIGcVOMruqWmpwrgskRUVHPBPM A6xclirhrFkYNnCzfEDX1UG6i/BZAZAb6MXCkPucbfGnwdtCPtDVjKHnEKkGxFkevleP r9JE13vLdDzQkYW5zrKOkKSe3Ke1LEj8UQ3mDh82ACZIkbMXr+j67R+zkvBr6otR9mVE Tl7UaRuucQNJbOk4/RL8F0RYyigcOAG0hbKfuYpal1gRntLXfXXvfyH20TVBvWRcPEZ/ T6zg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=andgHX9wkZy2Trx0HRVKpuTygds8q/VC0NacIobtvWI=; b=rB05pJiHq8VW4l0sNMk6fBby58vXutJy9VDParQe6uzHMVW2GBSzEXvKrxfIcVvx6b TDYn+my0st9+wf3+/HO/0cdkpBhisazJDn4CHJ01cmLkEpZe1mn7vSbTNZb9mb5Zs1L9 drUp4AbzLrPXmHFdTZALXum+r5wxPAlL8g/A9vLB08RCbyiL+uf9KQBU6TpUpWKkNoUA tD3PJuUOgJn5PqwSjM2L1mG21l6Ragv3TcbKiOog6aWWbJTxnvJsfrMgsBTHD+BwT8AI 2DmlwV/75VvbgtF9O55pLrF2QcdYL6hufPlu9AJApyL9eEHH3tFwOzjb+uwtoCwBYe/S C+QQ== X-Gm-Message-State: AOAM533JjftrdL0iSEvH2t1S/jSbd/ieQzc23TLEobLcmLvAvjlsG6v8 Qhs85cKpkrI/U2CLBvH9YJPjDYM+1PZ5eQ== X-Google-Smtp-Source: ABdhPJz8l0hqY52ZeX8nSamIt+o44kFCYTONvLjz7am2GGLBPhpxeTEMBov/ijXykagRnWrPM5GUCQ== X-Received: by 2002:a17:902:7606:b029:d4:c797:a186 with SMTP id k6-20020a1709027606b02900d4c797a186mr5340154pll.38.1605727185903; Wed, 18 Nov 2020 11:19:45 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:8b43]) by smtp.gmail.com with ESMTPSA id l9sm3197221pjy.10.2020.11.18.11.19.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Nov 2020 11:19:44 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v2 12/13] btrfs-progs: send: stream v2 ioctl flags Date: Wed, 18 Nov 2020 11:18:59 -0800 Message-Id: <340dcc141473301e5811b6c1eb6963a056118d0c.1605723745.git.osandov@osandov.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov To make the btrfs send ioctl use the stream v2 format requires passing BTRFS_SEND_FLAG_STREAM_V2 in flags. Further, to cause the ioctl to emit encoded_write commands for encoded extents, we must set that flag as well as BTRFS_SEND_FLAG_COMPRESSED. Finally, we bump up the version in send.h as well, since we are now fully compatible with v2. Add two command line arguments to btrfs send: --stream-version and --compressed-data. --stream-version requires an argument which it parses as an integer and sets STREAM_V2 if the argument is 2. --compressed-data does not require an argument and automatically implies STREAM_V2 as well (COMPRESSED alone causes the ioctl to error out). Some examples to illustrate edge cases: // v1, old format and no encoded_writes btrfs send subvol btrfs send --stream-version 1 subvol // v2 and compressed, we will see encoded_writes btrfs send --compressed-data subvol btrfs send --compressed-data --stream-version 2 subvol // v2 only, new format but no encoded_writes btrfs send --stream-version 2 subvol // error: compressed needs version >= 2 btrfs send --compressed-data --stream-version 1 subvol // error: invalid version (not 1 or 2) btrfs send --stream-version 3 subvol btrfs send --compressed-data --stream-version 0 subvol btrfs send --compressed-data --stream-version 10 subvol Signed-off-by: Boris Burkov --- Documentation/btrfs-send.asciidoc | 16 ++++++++- cmds/send.c | 54 ++++++++++++++++++++++++++++++- ioctl.h | 17 +++++++++- libbtrfsutil/btrfs.h | 17 +++++++++- send.h | 2 +- 5 files changed, 101 insertions(+), 5 deletions(-) diff --git a/Documentation/btrfs-send.asciidoc b/Documentation/btrfs-send.asciidoc index c4a05672..202bcd97 100644 --- a/Documentation/btrfs-send.asciidoc +++ b/Documentation/btrfs-send.asciidoc @@ -55,7 +55,21 @@ send in 'NO_FILE_DATA' mode The output stream does not contain any file data and thus cannot be used to transfer changes. This mode is faster and is useful to show the differences in metadata. --q|--quiet:::: + +--stream-version <1|2>:: +Use the given send stream version. The default is 1. Version 2 encodes file +data slightly more efficiently; it is also required for sending compressed data +directly (see '--compressed-data'). Version 2 requires at least btrfs-progs +5.12 on both the sender and receiver and at least Linux 5.12 on the sender. + +--compressed-data:: +Send data that is compressed on the filesystem directly without decompressing +it. If the receiver supports encoded I/O (see `encoded_io`(7)), it can also +write it directly without decompressing it. Otherwise, the receiver will fall +back to decompressing it and writing it normally. This implies +'--stream-version 2'. + +-q|--quiet:: (deprecated) alias for global '-q' option -v|--verbose:: (deprecated) alias for global '-v' option diff --git a/cmds/send.c b/cmds/send.c index 3bfc69f5..80eb2510 100644 --- a/cmds/send.c +++ b/cmds/send.c @@ -452,6 +452,21 @@ static const char * const cmd_send_usage[] = { " does not contain any file data and thus cannot be used", " to transfer changes. This mode is faster and useful to", " show the differences in metadata.", + "--stream-version <1|2>", + " Use the given send stream version. The default is", + " 1. Version 2 encodes file data slightly more", + " efficiently; it is also required for sending", + " compressed data directly (see --compressed-data).", + " Version 2 requires at least btrfs-progs 5.12 on both", + " the sender and receiver and at least Linux 5.12 on the", + " sender.", + "--compressed-data", + " Send data that is compressed on the filesystem", + " directly without decompressing it. If the receiver", + " supports encoded I/O, it can also write it directly", + " without decompressing it. Otherwise, the receiver will", + " fall back to decompressing it and writing it normally.", + " This implies --stream-version 2.", "-v|--verbose deprecated, alias for global -v option", "-q|--quiet deprecated, alias for global -q option", HELPINFO_INSERT_GLOBALS, @@ -463,6 +478,7 @@ static const char * const cmd_send_usage[] = { static int cmd_send(const struct cmd_struct *cmd, int argc, char **argv) { char *subvol = NULL; + char *end; int ret; char outname[PATH_MAX]; struct btrfs_send send; @@ -474,6 +490,7 @@ static int cmd_send(const struct cmd_struct *cmd, int argc, char **argv) int full_send = 1; int new_end_cmd_semantic = 0; u64 send_flags = 0; + long stream_version = 0; memset(&send, 0, sizeof(send)); send.dump_fd = fileno(stdout); @@ -492,11 +509,17 @@ static int cmd_send(const struct cmd_struct *cmd, int argc, char **argv) optind = 0; while (1) { - enum { GETOPT_VAL_SEND_NO_DATA = 256 }; + enum { + GETOPT_VAL_SEND_NO_DATA = 256, + GETOPT_VAL_SEND_STREAM_V2, + GETOPT_VAL_SEND_COMPRESSED_DATA, + }; static const struct option long_options[] = { { "verbose", no_argument, NULL, 'v' }, { "quiet", no_argument, NULL, 'q' }, { "no-data", no_argument, NULL, GETOPT_VAL_SEND_NO_DATA }, + { "stream-version", required_argument, NULL, GETOPT_VAL_SEND_STREAM_V2 }, + { "compressed-data", no_argument, NULL, GETOPT_VAL_SEND_COMPRESSED_DATA }, { NULL, 0, NULL, 0 } }; int c = getopt_long(argc, argv, "vqec:f:i:p:", long_options, NULL); @@ -585,10 +608,39 @@ static int cmd_send(const struct cmd_struct *cmd, int argc, char **argv) case GETOPT_VAL_SEND_NO_DATA: send_flags |= BTRFS_SEND_FLAG_NO_FILE_DATA; break; + case GETOPT_VAL_SEND_STREAM_V2: + stream_version = strtol(optarg, &end, 10); + if (*end != '\0' || + stream_version < 1 || stream_version > 2) { + ret = 1; + error("invalid --stream-version. valid values: {1, 2}"); + goto out; + } + if (stream_version == 2) + send_flags |= BTRFS_SEND_FLAG_STREAM_V2; + break; + case GETOPT_VAL_SEND_COMPRESSED_DATA: + send_flags |= BTRFS_SEND_FLAG_COMPRESSED; + /* + * We want to default to stream v2 if only compressed is + * set. If stream_version is explicitly set to 0, that + * will trigger its own error condition for being an + * invalid version. + */ + if (stream_version == 0) { + stream_version = 2; + send_flags |= BTRFS_SEND_FLAG_STREAM_V2; + } + break; default: usage_unknown_option(cmd, argv); } } + if (stream_version < 2 && (send_flags & BTRFS_SEND_FLAG_COMPRESSED)) { + ret = 1; + error("--compressed requires --stream-version >= 2"); + goto out; + } if (check_argc_min(argc - optind, 1)) return 1; diff --git a/ioctl.h b/ioctl.h index ade6dcb9..46de8ac8 100644 --- a/ioctl.h +++ b/ioctl.h @@ -653,10 +653,25 @@ BUILD_ASSERT(sizeof(struct btrfs_ioctl_received_subvol_args_32) == 192); */ #define BTRFS_SEND_FLAG_OMIT_END_CMD 0x4 +/* + * Use version 2 of the send stream, which adds new commands and supports larger + * writes. + */ +#define BTRFS_SEND_FLAG_STREAM_V2 0x8 + +/* + * Send compressed data using the ENCODED_WRITE command instead of decompressing + * the data and sending it with the WRITE command. This requires + * BTRFS_SEND_FLAG_STREAM_V2. + */ +#define BTRFS_SEND_FLAG_COMPRESSED 0x10 + #define BTRFS_SEND_FLAG_MASK \ (BTRFS_SEND_FLAG_NO_FILE_DATA | \ BTRFS_SEND_FLAG_OMIT_STREAM_HEADER | \ - BTRFS_SEND_FLAG_OMIT_END_CMD) + BTRFS_SEND_FLAG_OMIT_END_CMD | \ + BTRFS_SEND_FLAG_STREAM_V2 | \ + BTRFS_SEND_FLAG_COMPRESSED) struct btrfs_ioctl_send_args { __s64 send_fd; /* in */ diff --git a/libbtrfsutil/btrfs.h b/libbtrfsutil/btrfs.h index 60d51ff6..8430a40d 100644 --- a/libbtrfsutil/btrfs.h +++ b/libbtrfsutil/btrfs.h @@ -731,10 +731,25 @@ struct btrfs_ioctl_received_subvol_args { */ #define BTRFS_SEND_FLAG_OMIT_END_CMD 0x4 +/* + * Use version 2 of the send stream, which adds new commands and supports larger + * writes. + */ +#define BTRFS_SEND_FLAG_STREAM_V2 0x8 + +/* + * Send compressed data using the ENCODED_WRITE command instead of decompressing + * the data and sending it with the WRITE command. This requires + * BTRFS_SEND_FLAG_STREAM_V2. + */ +#define BTRFS_SEND_FLAG_COMPRESSED 0x10 + #define BTRFS_SEND_FLAG_MASK \ (BTRFS_SEND_FLAG_NO_FILE_DATA | \ BTRFS_SEND_FLAG_OMIT_STREAM_HEADER | \ - BTRFS_SEND_FLAG_OMIT_END_CMD) + BTRFS_SEND_FLAG_OMIT_END_CMD | \ + BTRFS_SEND_FLAG_STREAM_V2 | \ + BTRFS_SEND_FLAG_COMPRESSED) struct btrfs_ioctl_send_args { __s64 send_fd; /* in */ diff --git a/send.h b/send.h index 3c47e0c7..fac90588 100644 --- a/send.h +++ b/send.h @@ -31,7 +31,7 @@ extern "C" { #endif #define BTRFS_SEND_STREAM_MAGIC "btrfs-stream" -#define BTRFS_SEND_STREAM_VERSION 1 +#define BTRFS_SEND_STREAM_VERSION 2 #define BTRFS_SEND_BUF_SIZE_V1 SZ_64K #define BTRFS_SEND_READ_SIZE (1024 * 48) From patchwork Wed Nov 18 19:19:00 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11915749 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 221C2C6379D for ; Wed, 18 Nov 2020 19:20:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B9DED22227 for ; Wed, 18 Nov 2020 19:20:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="g9Pgdk80" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727407AbgKRTTv (ORCPT ); Wed, 18 Nov 2020 14:19:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55946 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727377AbgKRTTu (ORCPT ); Wed, 18 Nov 2020 14:19:50 -0500 Received: from mail-pg1-x534.google.com (mail-pg1-x534.google.com [IPv6:2607:f8b0:4864:20::534]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CE253C0613D4 for ; Wed, 18 Nov 2020 11:19:48 -0800 (PST) Received: by mail-pg1-x534.google.com with SMTP id f18so1856209pgi.8 for ; Wed, 18 Nov 2020 11:19:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=qASfdMugLX6dkJNB+g4DB9CS4PoJpLhsrJsEHm5JPHo=; b=g9Pgdk80zIKJmvTMpsBJ8RyqvR6ihfHqH60Twyb9UJwXyNRCAMOC2GbyKrKUy527mR iSg5zoJNfey4DZoUsFxzXjslAhiH1JIghrUVj435BWFNsjAKxc6XW/IqlR990B1EqHoU g3o0xkNgIPCxn4ijuiWumP+wXwF/5q2HC7J/nTk9RL9Wkrrs4Wp6MHtuFKvmWHiu53CN cCFnq2kk1fKjtC38KqsTNqbC5NoU/Ek0EriZ/8aL+tg0/olB9vuvJ3HvpfFkqEwWZ1Zv ROLAoVhI9gr82YZsnYlCEjSK6YcYDDOU7HOJj11qlBYbGJMOXr1GyTWufFY42VvH+okJ efeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=qASfdMugLX6dkJNB+g4DB9CS4PoJpLhsrJsEHm5JPHo=; b=sR1o5KXLgigpqByq6F0f01+b0Du1QipKbRAfwM8LG9lWLav2UNUJRivJBelsgVM1xO jyjgoejPD3kjT7aOXaePiNsTvElzRp5spfUmnRYMtjc1IrkaGxgpaeQKeCsuDyvWkFH9 tLqGlQz7+ELWqDbxwMHR2OrerqbOPNEoAe+F0ShrvaKReJbN7O18mJ9OwfmeAvvhN18g 7F8fiNrbO/RER9z4Q0wI6TM8jurgnnh2cOOh0F/BFAVzLOeLjVDGDLangOexJDQkutBO lB+xxRxLG+RY0VSBF6EyMR7qqI+wMQWZ4gfbftNbPiQiRvSgXB6IsGyjOVsOzD3IFpGq jDEg== X-Gm-Message-State: AOAM5304h6VuREDxC0zM5q9nyXrPiZcbB6rwuy7YPgpCk2jusJS6yv+r 4L6+CQwnHOboANUwaTPFDCGdEHf52I1GKw== X-Google-Smtp-Source: ABdhPJxZ7ymPheEqg1BwDPCXVWYQJR2jOlrQETu1Ux49c10JuOuG4RR1rxx/X3E8SOPzKEs4gwW2lA== X-Received: by 2002:a63:fc1c:: with SMTP id j28mr9106892pgi.95.1605727187768; Wed, 18 Nov 2020 11:19:47 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:8b43]) by smtp.gmail.com with ESMTPSA id l9sm3197221pjy.10.2020.11.18.11.19.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Nov 2020 11:19:46 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v2 13/13] btrfs-progs: receive: add tests for basic encoded_write send/receive Date: Wed, 18 Nov 2020 11:19:00 -0800 Message-Id: X-Mailer: git-send-email 2.29.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov Adapt the existing send/receive tests by passing '-o --force-compress' to the mount commands in a new test. After writing a few files in the various compression formats, send/receive them with and without --force-decompress to test both the encoded_write path and the fallback to decode+write. Signed-off-by: Boris Burkov --- .../042-receive-write-encoded/test.sh | 114 ++++++++++++++++++ 1 file changed, 114 insertions(+) create mode 100755 tests/misc-tests/042-receive-write-encoded/test.sh diff --git a/tests/misc-tests/042-receive-write-encoded/test.sh b/tests/misc-tests/042-receive-write-encoded/test.sh new file mode 100755 index 00000000..b9390e88 --- /dev/null +++ b/tests/misc-tests/042-receive-write-encoded/test.sh @@ -0,0 +1,114 @@ +#!/bin/bash +# +# test that we can send and receive encoded writes for three modes of +# transparent compression: zlib, lzo, and zstd. + +source "$TEST_TOP/common" + +check_prereq mkfs.btrfs +check_prereq btrfs + +setup_root_helper +prepare_test_dev + +here=`pwd` + +# assumes the filesystem exists, and does mount, write, snapshot, send, unmount +# for the specified encoding option +send_one() { + local str + local subv + local snap + + algorithm="$1" + shift + str="$1" + shift + + subv="subv-$algorithm" + snap="snap-$algorithm" + + run_check_mount_test_dev "-o" "compress-force=$algorithm" + cd "$TEST_MNT" || _fail "cannot chdir to TEST_MNT" + + run_check $SUDO_HELPER "$TOP/btrfs" subvolume create "$subv" + run_check $SUDO_HELPER dd if=/dev/zero of="$subv/file1" bs=1M count=1 + run_check $SUDO_HELPER dd if=/dev/zero of="$subv/file2" bs=500K count=1 + run_check $SUDO_HELPER "$TOP/btrfs" subvolume snapshot -r "$subv" "$snap" + run_check $SUDO_HELPER "$TOP/btrfs" send -f "$str" "$snap" "$@" + + cd "$here" || _fail "cannot chdir back to test directory" + run_check_umount_test_dev +} + +receive_one() { + local str + str="$1" + shift + + run_check_mkfs_test_dev + run_check_mount_test_dev + run_check $SUDO_HELPER "$TOP/btrfs" receive "$@" -v -f "$str" "$TEST_MNT" + run_check_umount_test_dev + run_check rm -f -- "$str" +} + +test_one_write_encoded() { + local str + local algorithm + algorithm="$1" + shift + str="$here/stream-$algorithm.stream" + + run_check_mkfs_test_dev + send_one "$algorithm" "$str" --compressed-data + receive_one "$str" "$@" +} + +test_one_stream_v1() { + local str + local algorithm + algorithm="$1" + shift + str="$here/stream-$algorithm.stream" + + run_check_mkfs_test_dev + send_one "$algorithm" "$str" --stream-version 1 + receive_one "$str" "$@" +} + +test_mix_write_encoded() { + local strzlib + local strlzo + local strzstd + strzlib="$here/stream-zlib.stream" + strlzo="$here/stream-lzo.stream" + strzstd="$here/stream-zstd.stream" + + run_check_mkfs_test_dev + + send_one "zlib" "$strzlib" --compressed-data + send_one "lzo" "$strlzo" --compressed-data + send_one "zstd" "$strzstd" --compressed-data + + receive_one "$strzlib" + receive_one "$strlzo" + receive_one "$strzstd" +} + +test_one_write_encoded "zlib" +test_one_write_encoded "lzo" +test_one_write_encoded "zstd" + +# with decompression forced +test_one_write_encoded "zlib" "--force-decompress" +test_one_write_encoded "lzo" "--force-decompress" +test_one_write_encoded "zstd" "--force-decompress" + +# send stream v1 +test_one_stream_v1 "zlib" +test_one_stream_v1 "lzo" +test_one_stream_v1 "zstd" + +# files use a mix of compression algorithms +test_mix_write_encoded