[v12,00/17] btrfs: add ioctls and send/receive support for reading/writing compressed data

Message

Omar Sandoval Nov. 17, 2021, 8:19 p.m. UTC

From: Omar Sandoval <[e-mail]>

This series has three parts: new Btrfs ioctls for reading/writing
compressed data, support for sending compressed data via Btrfs send, and
btrfs-progs support for sending/receiving compressed data and writing it
with the new ioctl.

Patches 1 and 2 are VFS changes exporting a couple of helpers for checks
needed by reads and writes. Patches 3-8 are preparatory Btrfs changes
for compressed reads and writes. Patch 6 is a cleanup. Patch 9 adds the
compressed read ioctl and patch 10 adds the compressed write ioctl.

The main use-case for this interface is Btrfs send/receive. Currently,
when sending data from one compressed filesystem to another, the sending
side decompresses the data and the receiving side recompresses it before
writing it out. This is wasteful and can be avoided if we can just send
and write compressed extents.

Patches 11 and 12 are cleanups for Btrfs send. Patches 13-17 add the
Btrfs send support. See the previous posting for more details and
benchmarks [1]. Patches 13-15 prepare some protocol changes for send
stream v2. Patch 16 implements compressed send. Patch 17 enables send
stream v2 and compressed send in the send ioctl when requested.

These patches are based on Dave Sterba's Btrfs misc-next branch [2],
which is in turn currently based on v5.16-rc1. Test cases are here [3].

Changes since v11 [4]:

- Rebased.
- Reworked send stream protocol versioning based on Dave's approach.
- Split up cow_file_range_inline() change into refactoring change and
  functional change.
- Added enforcement that BTRFS_SEND_A_DATA is always sent last.
- Fixed uninitialized variable reported by kernel test robot.
- Cleaned up some more style nits.
- Added and clarified some comments.
- Changed "page" nomenclature to "sector" for LZO.

1: https://lore.kernel.org/linux-btrfs/[e-mail]/
2: https://github.com/kdave/btrfs-devel/tree/misc-next
3: https://github.com/osandov/xfstests/tree/btrfs-encoded-io
4: https://lore.kernel.org/linux-btrfs/[e-mail]/

Omar Sandoval (17):
  fs: export rw_verify_area()
  fs: export variant of generic_write_checks without iov_iter
  btrfs: don't advance offset for compressed bios in
    btrfs_csum_one_bio()
  btrfs: add ram_bytes and offset to btrfs_ordered_extent
  btrfs: support different disk extent size for delalloc
  btrfs: clean up cow_file_range_inline()
  btrfs: optionally extend i_size in cow_file_range_inline()
  btrfs: add definitions + documentation for encoded I/O ioctls
  btrfs: add BTRFS_IOC_ENCODED_READ
  btrfs: add BTRFS_IOC_ENCODED_WRITE
  btrfs: send: remove unused send_ctx::{total,cmd}_send_size
  btrfs: send: fix maximum command numbering
  btrfs: add send stream v2 definitions
  btrfs: send: write larger chunks when using stream v2
  btrfs: send: allocate send buffer with alloc_page() and vmap() for v2
  btrfs: send: send compressed extents with encoded writes
  btrfs: send: enable support for stream v2 and compressed writes

 fs/btrfs/compression.c     |  10 +-
 fs/btrfs/compression.h     |   6 +-
 fs/btrfs/ctree.h           |  17 +-
 fs/btrfs/delalloc-space.c  |  18 +-
 fs/btrfs/file-item.c       |  32 +-
 fs/btrfs/file.c            |  68 ++-
 fs/btrfs/inode.c           | 927 +++++++++++++++++++++++++++++++++----
 fs/btrfs/ioctl.c           | 208 +++++++++
 fs/btrfs/ordered-data.c    | 131 ++----
 fs/btrfs/ordered-data.h    |  25 +-
 fs/btrfs/relocation.c      |   2 +-
 fs/btrfs/send.c            | 324 +++++++++++--
 fs/btrfs/send.h            |  39 +-
 fs/internal.h              |   5 -
 fs/read_write.c            |  34 +-
 include/linux/fs.h         |   2 +
 include/uapi/linux/btrfs.h | 142 +++++-
 17 files changed, 1704 insertions(+), 286 deletions(-)

The btrfs-progs patches were written by Boris Burkov with some updates
from me. Patches 1-4 are preparation. Patch 5 implements encoded writes.
Patch 6 implements the fallback to decompressing. Patches 7 and 8
implement the other commands. Patch 9 adds the new `btrfs send` options.
Patch 10 adds a test case.

Boris Burkov (10):
  btrfs-progs: receive: support v2 send stream larger tlv_len
  btrfs-progs: receive: dynamically allocate sctx->read_buf
  btrfs-progs: receive: support v2 send stream DATA tlv format
  btrfs-progs: receive: add send stream v2 cmds and attrs to send.h
  btrfs-progs: receive: process encoded_write commands
  btrfs-progs: receive: encoded_write fallback to explicit decode and
    write
  btrfs-progs: receive: process fallocate commands
  btrfs-progs: receive: process setflags ioctl commands
  btrfs-progs: send: stream v2 ioctl flags
  btrfs-progs: receive: add tests for basic encoded_write send/receive

 Documentation/btrfs-receive.asciidoc          |   4 +
 Documentation/btrfs-send.asciidoc             |  18 +-
 cmds/receive-dump.c                           |  31 +-
 cmds/receive.c                                | 347 +++++++++++++++++-
 cmds/send.c                                   |  92 ++++-
 common/send-stream.c                          | 165 +++++++--
 common/send-stream.h                          |   7 +
 ioctl.h                                       | 151 +++++++-
 kernel-shared/send.h                          |  39 +-
 libbtrfs/send-stream.c                        |   2 +-
 .../052-receive-write-encoded/test.sh         | 114 ++++++
 11 files changed, 924 insertions(+), 46 deletions(-)
 create mode 100755 tests/misc-tests/052-receive-write-encoded/test.sh