From patchwork Wed Nov 18 19:18:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 11915687 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6866DC64E75 for ; Wed, 18 Nov 2020 19:19:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2597222244 for ; Wed, 18 Nov 2020 19:19:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="fMZQHOtx" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726184AbgKRTTU (ORCPT ); Wed, 18 Nov 2020 14:19:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55822 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725772AbgKRTTU (ORCPT ); Wed, 18 Nov 2020 14:19:20 -0500 Received: from mail-pf1-x442.google.com (mail-pf1-x442.google.com [IPv6:2607:f8b0:4864:20::442]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 14CE6C0613D4 for ; Wed, 18 Nov 2020 11:19:19 -0800 (PST) Received: by mail-pf1-x442.google.com with SMTP id a18so2056743pfl.3 for ; Wed, 18 Nov 2020 11:19:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/zsxWF6lVDz+7AA3ZWvsvy5Wo7+pjmYoohRaGzOoct0=; b=fMZQHOtxQhH+Wm6oQayF/Jis6x8CMcwPmGIwI8unHPGZdxEVNgU4wmCUbheZf7k+Qs tVHAWpCOqeqNRIUY+MwW4e2wg1vf+nLmG2elJL6T6gogP69u/KqwDN7HJYoUV3dKcXPk ZLeWJFBHkup16/cVKeUktQS+XbU0FguStreRV7/+pBwNMuoQW2RivtqCmakePUfS3mP9 omigH4NPnNBh9Ndb7T8MQXbTkTBTvNfsJxvAWl0+D343EQBGRsZymWMB5i8s2mvmvM12 nAJPJhNm9wTZoRkVAepPHbjTUklp+Byk4ossFZRVftpgNzqOyp2xzHNXh8SkxEHb5OvM BZng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/zsxWF6lVDz+7AA3ZWvsvy5Wo7+pjmYoohRaGzOoct0=; b=kplacl+SPU3FiCayeY6HtgirQqRyQKIodyVBXgycDgtheRquFLPcfTDgR7asU2j43v iL309qb36JIv3VI4prQ8+EKy0JZT605Y2mJZVa8Y8ArcVdrrMut3Gyl4zWUcxhMpMt5T E/rIaFmug9mRcNiJT6TFWMLVJ4CKtU0UZcpbrl2X21PO9SwiwxQKhf2lEFZnpwEe8ZC0 x6BPzqD0YmLKQ0d50tQ0AjYJLtbYShWObAFWwFRd8HIDAro6T3dgwACtnkgVsE0O4k+n 5V0EE3ZQnkApqKMLIoX/XrksCmDLJzku5lROZOOqSj7DLhOGdmlLEuXgZixWlfZ3fNGp NnBg== X-Gm-Message-State: AOAM5312D3gVSnIDdWbJLYhyY5TlrAKrp93J3KK2wtdsKolSIam77jTE VZ9HIaAqu2gso7ZrdAuHl+rQthTEFkg5Wg== X-Google-Smtp-Source: ABdhPJzQACKgpEvQnGNJzbC9MgeDK/K2QmXPVpbgyKh4g6Zq+LKBixy7Ekh0P/zAGAtJVkLPSRDxFw== X-Received: by 2002:a17:90a:154a:: with SMTP id y10mr535452pja.6.1605727158044; Wed, 18 Nov 2020 11:19:18 -0800 (PST) Received: from relinquished.tfbnw.net ([2620:10d:c090:400::5:8b43]) by smtp.gmail.com with ESMTPSA id l9sm3197221pjy.10.2020.11.18.11.19.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Nov 2020 11:19:17 -0800 (PST) From: Omar Sandoval To: linux-btrfs@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v2 03/13] btrfs-progs: receive: dynamically allocate sctx->read_buf Date: Wed, 18 Nov 2020 11:18:47 -0800 Message-Id: <0621eb3c44c2057b35ceb1ce5c1270a4c9c8edc5.1605723745.git.osandov@osandov.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Boris Burkov In send stream v2, write commands can now be an arbitrary size. For that reason, we can no longer allocate a fixed array in sctx for read_cmd. Instead, read_cmd dynamically allocates sctx->read_buf. To avoid needless reallocations, we reuse read_buf between read_cmd calls by also keeping track of the size of the allocated buffer in sctx->read_buf_sz. We do the first allocation of the old default size at the start of processing the stream, and we only reallocate if we encounter a command that needs a larger buffer. Signed-off-by: Boris Burkov --- common/send-stream.c | 55 ++++++++++++++++++++++++++++---------------- send.h | 2 +- 2 files changed, 36 insertions(+), 21 deletions(-) diff --git a/common/send-stream.c b/common/send-stream.c index 3bd21d3f..51a6a94a 100644 --- a/common/send-stream.c +++ b/common/send-stream.c @@ -36,10 +36,10 @@ struct btrfs_send_attribute { struct btrfs_send_stream { int fd; - char read_buf[BTRFS_SEND_BUF_SIZE]; + char *read_buf; + size_t read_buf_sz; int cmd; - struct btrfs_cmd_header *cmd_hdr; struct btrfs_send_attribute cmd_attrs[BTRFS_SEND_A_MAX + 1]; u32 version; @@ -111,11 +111,12 @@ static int read_cmd(struct btrfs_send_stream *sctx) u32 pos; u32 crc; u32 crc2; + struct btrfs_cmd_header *cmd_hdr; + size_t buf_len; memset(sctx->cmd_attrs, 0, sizeof(sctx->cmd_attrs)); - ASSERT(sizeof(*sctx->cmd_hdr) <= sizeof(sctx->read_buf)); - ret = read_buf(sctx, sctx->read_buf, sizeof(*sctx->cmd_hdr)); + ret = read_buf(sctx, sctx->read_buf, sizeof(*cmd_hdr)); if (ret < 0) goto out; if (ret) { @@ -124,18 +125,22 @@ static int read_cmd(struct btrfs_send_stream *sctx) goto out; } - sctx->cmd_hdr = (struct btrfs_cmd_header *)sctx->read_buf; - cmd = le16_to_cpu(sctx->cmd_hdr->cmd); - cmd_len = le32_to_cpu(sctx->cmd_hdr->len); - - if (cmd_len + sizeof(*sctx->cmd_hdr) >= sizeof(sctx->read_buf)) { - ret = -EINVAL; - error("command length %u too big for buffer %zu", - cmd_len, sizeof(sctx->read_buf)); - goto out; + cmd_hdr = (struct btrfs_cmd_header *)sctx->read_buf; + cmd_len = le32_to_cpu(cmd_hdr->len); + cmd = le16_to_cpu(cmd_hdr->cmd); + buf_len = sizeof(*cmd_hdr) + cmd_len; + if (sctx->read_buf_sz < buf_len) { + sctx->read_buf = realloc(sctx->read_buf, buf_len); + if (!sctx->read_buf) { + ret = -ENOMEM; + error("failed to reallocate read buffer for cmd"); + goto out; + } + sctx->read_buf_sz = buf_len; + /* We need to reset cmd_hdr after realloc of sctx->read_buf */ + cmd_hdr = (struct btrfs_cmd_header *)sctx->read_buf; } - - data = sctx->read_buf + sizeof(*sctx->cmd_hdr); + data = sctx->read_buf + sizeof(*cmd_hdr); ret = read_buf(sctx, data, cmd_len); if (ret < 0) goto out; @@ -145,11 +150,12 @@ static int read_cmd(struct btrfs_send_stream *sctx) goto out; } - crc = le32_to_cpu(sctx->cmd_hdr->crc); - sctx->cmd_hdr->crc = 0; + crc = le32_to_cpu(cmd_hdr->crc); + /* in send, crc is computed with header crc = 0, replicate that */ + cmd_hdr->crc = 0; crc2 = crc32c(0, (unsigned char*)sctx->read_buf, - sizeof(*sctx->cmd_hdr) + cmd_len); + sizeof(*cmd_hdr) + cmd_len); if (crc != crc2) { ret = -EINVAL; @@ -524,19 +530,28 @@ int btrfs_read_and_process_send_stream(int fd, goto out; } + sctx.read_buf = malloc(BTRFS_SEND_BUF_SIZE_V1); + if (!sctx.read_buf) { + ret = -ENOMEM; + error("unable to allocate send stream read buffer"); + goto out; + } + sctx.read_buf_sz = BTRFS_SEND_BUF_SIZE_V1; + while (1) { ret = read_and_process_cmd(&sctx); if (ret < 0) { last_err = ret; errors++; if (max_errors > 0 && errors >= max_errors) - goto out; + break; } else if (ret > 0) { if (!honor_end_cmd) ret = 0; - goto out; + break; } } + free(sctx.read_buf); out: if (last_err && !ret) diff --git a/send.h b/send.h index 8dd865ec..228928a0 100644 --- a/send.h +++ b/send.h @@ -33,7 +33,7 @@ extern "C" { #define BTRFS_SEND_STREAM_MAGIC "btrfs-stream" #define BTRFS_SEND_STREAM_VERSION 1 -#define BTRFS_SEND_BUF_SIZE SZ_64K +#define BTRFS_SEND_BUF_SIZE_V1 SZ_64K #define BTRFS_SEND_READ_SIZE (1024 * 48) enum btrfs_tlv_type {