[v6,00/11] fs: interface for directly reading/writing compressed data

Message

Omar Sandoval Nov. 18, 2020, 7:18 p.m. UTC

From: Omar Sandoval <[e-mail]>

This series adds an API for reading compressed data on a filesystem
without decompressing it as well as support for writing compressed data
directly to the filesystem. As with the previous submissions, I've
included a man page patch describing the API. I have test cases
(including fsstress support) and example programs which I'll send up
[1].

The main use-case is Btrfs send/receive: currently, when sending data
from one compressed filesystem to another, the sending side decompresses
the data and the receiving side recompresses it before writing it out.
This is wasteful and can be avoided if we can just send and write
compressed extents. The patches implementing the send/receive support
will be sent shortly.

Patches 1-3 add the VFS support and UAPI. Patches 4 and 5 are fixes for
patches in the Btrfs misc-next branch that conflicted with this series;
they can go into misc-next or be folded into the original patches
independently. Patches 6-9 are Btrfs prep patches. Patch 10 adds Btrfs
encoded read support and patch 11 adds Btrfs encoded write support.

These patches are based on Dave Sterba's Btrfs misc-next branch [2],
which is in turn currently based on v5.10-rc4.

Changes since v5 [3]:

- Made O_CLOEXEC mandatory in conjuction with O_ALLOW_ENCODED.
- Added _BTRFS to the ENCODED_IOV_COMPRESSION names (e.g.,
  ENCODED_IOV_COMPRESSION_ZSTD -> ENCODED_IOV_COMPRESSION_BTRFS_ZSTD).
- Split ENCODED_IOV_COMPRESSION_LZO compression mode into separate modes
  per page size. I missed that the ill-conceived Btrfs LZO format
  depends on PAGE_SIZE. Having a separate compression mode for each
  supported page size at least lets us detect mismatches.
- Fixed up other minor comments from v5.
- Added reviewed-bys.

1: https://github.com/osandov/xfstests/tree/rwf-encoded
2: https://github.com/kdave/btrfs-devel/tree/misc-next
3: https://lore.kernel.org/linux-btrfs/[e-mail]/

Omar Sandoval (11):
  iov_iter: add copy_struct_from_iter()
  fs: add O_ALLOW_ENCODED open flag
  fs: add RWF_ENCODED for reading/writing compressed data
  btrfs: fix btrfs_write_check()
  btrfs: fix check_data_csum() error message for direct I/O
  btrfs: don't advance offset for compressed bios in
    btrfs_csum_one_bio()
  btrfs: add ram_bytes and offset to btrfs_ordered_extent
  btrfs: support different disk extent size for delalloc
  btrfs: optionally extend i_size in cow_file_range_inline()
  btrfs: implement RWF_ENCODED reads
  btrfs: implement RWF_ENCODED writes

 Documentation/filesystems/encoded_io.rst |  74 ++
 Documentation/filesystems/index.rst      |   1 +
 arch/alpha/include/uapi/asm/fcntl.h      |   1 +
 arch/parisc/include/uapi/asm/fcntl.h     |   1 +
 arch/sparc/include/uapi/asm/fcntl.h      |   1 +
 fs/btrfs/compression.c                   |  12 +-
 fs/btrfs/compression.h                   |   6 +-
 fs/btrfs/ctree.h                         |   9 +-
 fs/btrfs/delalloc-space.c                |  18 +-
 fs/btrfs/file-item.c                     |  35 +-
 fs/btrfs/file.c                          |  73 +-
 fs/btrfs/inode.c                         | 933 ++++++++++++++++++++---
 fs/btrfs/ordered-data.c                  |  80 +-
 fs/btrfs/ordered-data.h                  |  18 +-
 fs/btrfs/relocation.c                    |   4 +-
 fs/fcntl.c                               |  10 +-
 fs/namei.c                               |   4 +
 fs/open.c                                |   7 +
 fs/read_write.c                          | 167 +++-
 include/linux/fcntl.h                    |   2 +-
 include/linux/fs.h                       |  11 +
 include/linux/uio.h                      |   2 +
 include/uapi/asm-generic/fcntl.h         |   4 +
 include/uapi/linux/fs.h                  |  41 +-
 lib/iov_iter.c                           |  82 ++
 25 files changed, 1384 insertions(+), 212 deletions(-)
 create mode 100644 Documentation/filesystems/encoded_io.rst