mbox series

[GIT,PULL] Btrfs updates for 6.3

Message ID cover.1676908729.git.dsterba@suse.com (mailing list archive)
State New, archived
Headers show
Series [GIT,PULL] Btrfs updates for 6.3 | expand

Pull-request

git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git for-6.3-tag

Message

David Sterba Feb. 20, 2023, 7:20 p.m. UTC
Hi,

there's a usual mix of performance improvements and new features. The
core change is reworking how checksums are processed, with followup
cleanups and simplifications. There are two minor changes in block layer
and iomap code.

Please pull, thanks.

Features:

- block group allocation class heuristics
  - pack files by size (up to 128k, up to 8M, more) to avoid
    fragmentation in block groups, assuming that file size and life time
    is correlated, in particular this may help during balance
  - with tracepoints and extensible in the future

Performance:

- send: cache directory utimes and only emit the command when necessary
  - speedup up to 10x
  - smaller final stream produced (no redundant utimes commands issued),
  - compatibility not affected

- fiemap: skip backref checks for shared leaves
  - speedup 3x on sample filesystem with all leaves shared (e.g. on
    snapshots)

- micro optimized b-tree key lookup, speedup in metadata operations
  (sample benchmark: fs_mark +10% of files/sec)

Core changes:

- change where checksumming is done in the io path
  - checksum and read repair does verification at lower layer
  - cascaded cleanups and simplifications

- raid56 refactoring and cleanups

Fixes:

- sysfs: make sure that a run-time change of a feature is correctly
  tracked by the feature files

- scrub: better reporting of tree block errors

Other:

- locally enable -Wmaybe-uninitialized after fixing all warnings

- misc cleanups, spelling fixes

Other code:

- block: export bio_split_rw

- iomap: remove IOMAP_F_ZONE_APPEND

----------------------------------------------------------------
The following changes since commit ceaa837f96adb69c0df0397937cd74991d5d821a:

  Linux 6.2-rc8 (2023-02-12 14:10:17 -0800)

are available in the Git repository at:

  git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git for-6.3-tag

for you to fetch changes up to 964a54e5e1a0d70cd80bd5a0885a1938463625b1:

  btrfs: make kobj_type structures constant (2023-02-15 19:38:55 +0100)

----------------------------------------------------------------
Boris Burkov (6):
      btrfs: pass find_free_extent_ctl to allocator tracepoints
      btrfs: add more find_free_extent tracepoints
      btrfs: introduce size class to block group allocator
      btrfs: load block group size class when caching
      btrfs: don't use size classes for zoned file systems
      btrfs: hold block group refcount during async discard

Christoph Hellwig (48):
      btrfs: remove the wait argument to btrfs_start_ordered_extent
      block: export bio_split_rw
      btrfs: better document struct btrfs_bio
      btrfs: add a btrfs_inode pointer to struct btrfs_bio
      btrfs: remove the direct I/O read checksum lookup optimization
      btrfs: simplify parameters of btrfs_lookup_bio_sums
      btrfs: refactor error handling in btrfs_submit_bio
      btrfs: save the bio iter for checksum validation in common code
      btrfs: pre-load data checksum for reads in btrfs_submit_bio
      btrfs: add a btrfs_data_csum_ok helper
      btrfs: handle checksum validation and repair at the storage layer
      btrfs: open code btrfs_bio_free_csum
      btrfs: remove btrfs_bio_for_each_sector
      btrfs: remove now unused checksumming helpers
      btrfs: remove struct btrfs_bio::device field
      btrfs: remove the io_failure_record infrastructure
      btrfs: rename btrfs_bio::iter field
      btrfs: remove struct btrfs_bio::is_metadata flag
      btrfs: open code the submit_bio_start helpers
      btrfs: simplify the btrfs_csum_one_bio calling convention
      btrfs: handle checksum generation in the storage layer
      btrfs: handle recording of zoned writes in the storage layer
      btrfs: support cloned bios in btree_csum_one_bio
      btrfs: allow btrfs_submit_bio to split bios
      btrfs: pass the iomap bio to btrfs_submit_bio
      btrfs: remove struct btrfs_io_geometry
      btrfs: open code submit_encoded_read_bio
      btrfs: remove the fs_info argument to btrfs_submit_bio
      btrfs: remove now spurious bio submission helpers
      btrfs: calculate file system wide queue limit for zoned mode
      btrfs: split zone append bios in btrfs_submit_bio
      iomap: remove IOMAP_F_ZONE_APPEND
      btrfs: raid56: simplify error handling and code flow in raid56_parity_write
      btrfs: raid56: simplify code flow in rmw_rbio
      btrfs: raid56: wait for I/O completion in submit_read_bios
      btrfs: raid56: add a bio_list_put helper
      btrfs: raid56: fold recover_assemble_read_bios into recover_rbio
      btrfs: raid56: fold rmw_read_wait_recover into rmw_read_bios
      btrfs: raid56: submit the read bios from scrub_assemble_read_bios
      btrfs: raid56: handle endio in rmw_rbio
      btrfs: raid56: handle endio in recover_rbio
      btrfs: raid56: handle endio in scrub_rbio
      btrfs: use file_offset to limit bios size in calc_bio_boundaries
      btrfs: set bbio->file_offset in alloc_new_bio
      btrfs: pass a btrfs_bio to btrfs_use_append
      btrfs: never return true for reads in btrfs_use_zone_append
      btrfs: don't rely on unchanging ->bi_bdev for zone append remaps
      btrfs: remove the bdev argument to btrfs_rmap_block

Colin Ian King (1):
      btrfs: fix spelling mistakes found using codespell

Filipe Manana (24):
      btrfs: use a negative value for BTRFS_LOG_FORCE_COMMIT
      btrfs: use a single variable to track return value for log_dir_items()
      btrfs: send: directly return from did_overwrite_ref() and simplify it
      btrfs: send: avoid unnecessary generation search at did_overwrite_ref()
      btrfs: send: directly return from will_overwrite_ref() and simplify it
      btrfs: send: avoid extra b+tree searches when checking reference overrides
      btrfs: send: remove send_progress argument from can_rmdir()
      btrfs: send: avoid duplicated orphan dir allocation and initialization
      btrfs: send: avoid unnecessary orphan dir rbtree search at can_rmdir()
      btrfs: send: reduce searches on parent root when checking if dir can be removed
      btrfs: send: iterate waiting dir move rbtree only once when processing refs
      btrfs: send: initialize all the red black trees earlier
      btrfs: send: genericize the backref cache to allow it to be reused
      btrfs: adapt lru cache to allow for 64 bits keys on 32 bits systems
      btrfs: send: cache information about created directories
      btrfs: allow a generation number to be associated with lru cache entries
      btrfs: add an api to delete a specific entry from the lru cache
      btrfs: send: use the lru cache to implement the name cache
      btrfs: send: update size of roots array for backref cache entries
      btrfs: send: cache utimes operations for directories if possible
      btrfs: assert commit root semaphore is held when accessing backref cache
      btrfs: skip backref walking during fiemap if we know the leaf is shared
      btrfs: eliminate extra call when doing binary search on extent buffer
      btrfs: do unsigned integer division in the extent buffer binary search loop

Johannes Thumshirn (4):
      btrfs: drop unused trans parameter of drop_delayed_ref
      btrfs: remove trans parameter of merge_ref
      btrfs: drop trans parameter of insert_delayed_ref
      btrfs: directly pass in fs_info to btrfs_merge_delayed_refs

Josef Bacik (15):
      btrfs: move btrfs_abort_transaction to transaction.c
      btrfs: fix uninitialized variable warning in btrfs_cleanup_ordered_extents
      btrfs: fix uninitialized variable warning in get_inode_gen
      btrfs: fix uninitialized variable warning in btrfs_update_block_group
      btrfs: fix uninitialized variable warnings in __set_extent_bit and convert_extent_bit
      btrfs: fix uninitialized variable warning in btrfs_sb_log_location
      btrfs: fix uninitialized variable warning in run_one_async_start
      btrfs: turn on -Wmaybe-uninitialized
      btrfs: always lock the block before calling btrfs_clean_tree_block
      btrfs: add trans argument to btrfs_clean_tree_block
      btrfs: replace clearing extent buffer dirty bit with btrfs_clean_block
      btrfs: do not increment dirty_metadata_bytes in set_btree_ioerr
      btrfs: rename btrfs_clean_tree_block to btrfs_clear_buffer_dirty
      btrfs: combine btrfs_clear_buffer_dirty and clear_extent_buffer_dirty
      btrfs: replace btrfs_wait_tree_block_writeback by wait_on_extent_buffer_writeback

Naohiro Aota (1):
      btrfs: zoned: fix uninitialized variable warning in btrfs_get_dev_zones

Peng Hao (1):
      btrfs: go to matching label when cleaning em in btrfs_submit_direct

Qu Wenruo (6):
      btrfs: scrub: improve tree block error reporting
      btrfs: sysfs: update fs features directory asynchronously
      btrfs: raid56: reduce overhead to calculate the bio length
      btrfs: remove stripe boundary calculation for buffered I/O
      btrfs: remove stripe boundary calculation for compressed I/O
      btrfs: remove stripe boundary calculation for encoded I/O

Thomas Weißschuh (1):
      btrfs: make kobj_type structures constant

Yushan Zhou (1):
      btrfs: use PAGE_{ALIGN, ALIGNED, ALIGN_DOWN} macro

ye xingchen (1):
      btrfs: remove duplicate include header in extent-tree.c

 block/blk-merge.c                 |   3 +-
 fs/btrfs/Makefile                 |   6 +-
 fs/btrfs/backref.c                |  33 +-
 fs/btrfs/bio.c                    | 557 +++++++++++++++++++++++++++----
 fs/btrfs/bio.h                    |  67 +---
 fs/btrfs/block-group.c            | 273 +++++++++++++--
 fs/btrfs/block-group.h            |  24 +-
 fs/btrfs/btrfs_inode.h            |  22 +-
 fs/btrfs/compression.c            | 276 +++------------
 fs/btrfs/compression.h            |   3 -
 fs/btrfs/ctree.c                  |  62 ++--
 fs/btrfs/ctree.h                  |  15 +
 fs/btrfs/defrag.c                 |   4 +-
 fs/btrfs/delayed-ref.c            |  24 +-
 fs/btrfs/delayed-ref.h            |   2 +-
 fs/btrfs/discard.c                |  41 ++-
 fs/btrfs/disk-io.c                | 225 ++-----------
 fs/btrfs/disk-io.h                |  14 +-
 fs/btrfs/extent-io-tree.c         |  10 +-
 fs/btrfs/extent-io-tree.h         |   1 -
 fs/btrfs/extent-tree.c            | 181 +++-------
 fs/btrfs/extent-tree.h            |  81 +++++
 fs/btrfs/extent_io.c              | 582 ++++----------------------------
 fs/btrfs/extent_io.h              |  36 +-
 fs/btrfs/file-item.c              |  72 ++--
 fs/btrfs/file-item.h              |   8 +-
 fs/btrfs/file.c                   |   2 +-
 fs/btrfs/free-space-tree.c        |   2 +-
 fs/btrfs/fs.c                     |   4 +
 fs/btrfs/fs.h                     |  11 +-
 fs/btrfs/inode.c                  | 641 +++++------------------------------
 fs/btrfs/ioctl.c                  |   2 +-
 fs/btrfs/lru_cache.c              | 166 +++++++++
 fs/btrfs/lru_cache.h              |  80 +++++
 fs/btrfs/lzo.c                    |   2 +-
 fs/btrfs/messages.c               |  30 --
 fs/btrfs/messages.h               |  34 --
 fs/btrfs/ordered-data.c           |  25 +-
 fs/btrfs/ordered-data.h           |   3 +-
 fs/btrfs/qgroup.c                 |   2 +-
 fs/btrfs/raid56.c                 | 334 +++++++------------
 fs/btrfs/raid56.h                 |   4 +-
 fs/btrfs/relocation.c             |   2 +-
 fs/btrfs/scrub.c                  |  51 ++-
 fs/btrfs/send.c                   | 684 ++++++++++++++++++++------------------
 fs/btrfs/super.c                  |   3 +-
 fs/btrfs/sysfs.c                  |  41 +--
 fs/btrfs/sysfs.h                  |   3 +-
 fs/btrfs/tests/extent-map-tests.c |   2 +-
 fs/btrfs/transaction.c            |  34 ++
 fs/btrfs/transaction.h            |  31 ++
 fs/btrfs/tree-log.c               |  87 ++---
 fs/btrfs/tree-log.h               |   9 +-
 fs/btrfs/volumes.c                | 116 ++-----
 fs/btrfs/volumes.h                |  18 -
 fs/btrfs/zoned.c                  | 146 ++++----
 fs/btrfs/zoned.h                  |  20 +-
 fs/iomap/direct-io.c              |  10 +-
 include/linux/bio.h               |   4 +
 include/linux/iomap.h             |   3 +-
 include/trace/events/btrfs.h      | 127 +++++--
 61 files changed, 2457 insertions(+), 2898 deletions(-)
 create mode 100644 fs/btrfs/lru_cache.c
 create mode 100644 fs/btrfs/lru_cache.h

Comments

Linus Torvalds Feb. 20, 2023, 9:02 p.m. UTC | #1
On Mon, Feb 20, 2023 at 11:26 AM David Sterba <dsterba@suse.com> wrote:
>
> Other:
>
> - locally enable -Wmaybe-uninitialized after fixing all warnings

I've pulled this, but I strongly suspect this change will get reverted.

I bet neither you nor linux-next is testing even _remotely_ a big
chunk of the different compiler versions that are out there, and the
reason flags like '-Wmaybe-uninitialized' get undone is because some
random compiler version on some random config and target archiecture
gives completely nonsensical warnings for odd reasons.

But hey, maybe the btrfs code is special.

              Linus
pr-tracker-bot@kernel.org Feb. 20, 2023, 9:11 p.m. UTC | #2
The pull request you sent on Mon, 20 Feb 2023 20:20:08 +0100:

> git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git for-6.3-tag

has been merged into torvalds/linux.git:
https://git.kernel.org/torvalds/c/885ce48739189fac6645ff42d736ee0de0b5917d

Thank you!
Ammar Faizi April 23, 2023, 2:27 a.m. UTC | #3
On 2/21/23 4:02 AM, Linus Torvalds wrote:
> On Mon, Feb 20, 2023 at 11:26 AM David Sterba <dsterba@suse.com> wrote:
>> Other:
>>
>> - locally enable -Wmaybe-uninitialized after fixing all warnings
> 
> I've pulled this, but I strongly suspect this change will get reverted.
> 
> I bet neither you nor linux-next is testing even _remotely_ a big
> chunk of the different compiler versions that are out there, and the
> reason flags like '-Wmaybe-uninitialized' get undone is because some
> random compiler version on some random config and target archiecture
> gives completely nonsensical warnings for odd reasons.
> 
> But hey, maybe the btrfs code is special.

Maybe it's too late for 6.3. So please fix this in 6.4 and backport it to
6.3 stable. If someone fixes it, kindly add:

Reported-by: Memet <zxce3@gnuweeb.org>

Compiling with GCC 9.4:

gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Linux version 6.3-rc7:

   CC [M]  fs/btrfs/volumes.o
fs/btrfs/volumes.c: In function ‘btrfs_init_new_device’:
fs/btrfs/volumes.c:2524:2: error: ‘seed_devices’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
  2524 |  list_add(&seed_devices->seed_list, &fs_devices->seed_list);
       |  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
fs/btrfs/volumes.c:2621:27: note: ‘seed_devices’ was declared here
  2621 |  struct btrfs_fs_devices *seed_devices;
       |                           ^~~~~~~~~~~~
cc1: all warnings being treated as errors
make[7]: *** [scripts/Makefile.build:252: fs/btrfs/volumes.o] Error 1
make[6]: *** [scripts/Makefile.build:494: fs/btrfs] Error 2
make[5]: *** [scripts/Makefile.build:494: fs] Error 2
make[4]: *** [Makefile:2025: .] Error 2
make[3]: *** [Makefile:357: __build_one_by_one] Error 2
make[2]: *** [debian/rules:8: build-arch] Error 2
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2
make[1]: *** [scripts/Makefile.package:120: bindeb-pkg] Error 2
make: *** [Makefile:1656: bindeb-pkg] Error 2


fs/btrfs/send.c: In function ‘get_cur_inode_state’:
fs/btrfs/send.c:1909:13: error: ‘right_gen’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
  1909 |   } else if (right_gen == gen) {
       |             ^
David Sterba April 25, 2023, 2:54 p.m. UTC | #4
On Sun, Apr 23, 2023 at 09:27:30AM +0700, Ammar Faizi wrote:
> On 2/21/23 4:02 AM, Linus Torvalds wrote:
> > On Mon, Feb 20, 2023 at 11:26 AM David Sterba <dsterba@suse.com> wrote:
> >> Other:
> >>
> >> - locally enable -Wmaybe-uninitialized after fixing all warnings
> > 
> > I've pulled this, but I strongly suspect this change will get reverted.
> > 
> > I bet neither you nor linux-next is testing even _remotely_ a big
> > chunk of the different compiler versions that are out there, and the
> > reason flags like '-Wmaybe-uninitialized' get undone is because some
> > random compiler version on some random config and target archiecture
> > gives completely nonsensical warnings for odd reasons.
> > 
> > But hey, maybe the btrfs code is special.
> 
> Maybe it's too late for 6.3. So please fix this in 6.4 and backport it to
> 6.3 stable. If someone fixes it, kindly add:

Fix for this warning is in 6.4 pull request, there's no CC:stable tag
but we can ask to add it once the code lands in master.
Ammar Faizi April 28, 2023, 3:14 a.m. UTC | #5
On Tue, Apr 25, 2023 at 04:54:12PM +0200, David Sterba wrote:
> On Sun, Apr 23, 2023 at 09:27:30AM +0700, Ammar Faizi wrote:
> > On 2/21/23 4:02 AM, Linus Torvalds wrote:
> > > On Mon, Feb 20, 2023 at 11:26 AM David Sterba <dsterba@suse.com> wrote:
> > >> Other:
> > >>
> > >> - locally enable -Wmaybe-uninitialized after fixing all warnings
> > > 
> > > I've pulled this, but I strongly suspect this change will get reverted.
> > > 
> > > I bet neither you nor linux-next is testing even _remotely_ a big
> > > chunk of the different compiler versions that are out there, and the
> > > reason flags like '-Wmaybe-uninitialized' get undone is because some
> > > random compiler version on some random config and target archiecture
> > > gives completely nonsensical warnings for odd reasons.
> > > 
> > > But hey, maybe the btrfs code is special.
> > 
> > Maybe it's too late for 6.3. So please fix this in 6.4 and backport it to
> > 6.3 stable.
> 
> Fix for this warning is in 6.4 pull request, there's no CC:stable tag
> but we can ask to add it once the code lands in master.

It landed in master.

[ Adding stable team to the Cc list ]

Hi Greg and stable team, could you please backport:

  commit 8ba7d5f5ba931be68a94b8c91bcced1622934e7a upstream
  ("btrfs: fix uninitialized variable warnings")

to v6.3 to fix gcc (10, 9, 7) build error?

The fs/btrfs/volumes.c hunk won't apply cleanly, but it can auto-merge:

  $ git cherry-pick 8ba7d5f5ba931be68a94b8c91bcced1622934e7a
  Auto-merging fs/btrfs/volumes.c
  [detached HEAD 572410288a1070c1] btrfs: fix uninitialized variable warnings
   Author: Genjian Zhang <zhanggenjian@kylinos.cn>
   Date: Fri Mar 24 10:08:38 2023 +0800
   2 files changed, 2 insertions(+), 2 deletions(-)

Thanks,
Greg Kroah-Hartman April 28, 2023, 7:34 a.m. UTC | #6
On Fri, Apr 28, 2023 at 10:14:58AM +0700, Ammar Faizi wrote:
> On Tue, Apr 25, 2023 at 04:54:12PM +0200, David Sterba wrote:
> > On Sun, Apr 23, 2023 at 09:27:30AM +0700, Ammar Faizi wrote:
> > > On 2/21/23 4:02 AM, Linus Torvalds wrote:
> > > > On Mon, Feb 20, 2023 at 11:26 AM David Sterba <dsterba@suse.com> wrote:
> > > >> Other:
> > > >>
> > > >> - locally enable -Wmaybe-uninitialized after fixing all warnings
> > > > 
> > > > I've pulled this, but I strongly suspect this change will get reverted.
> > > > 
> > > > I bet neither you nor linux-next is testing even _remotely_ a big
> > > > chunk of the different compiler versions that are out there, and the
> > > > reason flags like '-Wmaybe-uninitialized' get undone is because some
> > > > random compiler version on some random config and target archiecture
> > > > gives completely nonsensical warnings for odd reasons.
> > > > 
> > > > But hey, maybe the btrfs code is special.
> > > 
> > > Maybe it's too late for 6.3. So please fix this in 6.4 and backport it to
> > > 6.3 stable.
> > 
> > Fix for this warning is in 6.4 pull request, there's no CC:stable tag
> > but we can ask to add it once the code lands in master.
> 
> It landed in master.
> 
> [ Adding stable team to the Cc list ]
> 
> Hi Greg and stable team, could you please backport:
> 
>   commit 8ba7d5f5ba931be68a94b8c91bcced1622934e7a upstream
>   ("btrfs: fix uninitialized variable warnings")
> 
> to v6.3 to fix gcc (10, 9, 7) build error?
> 
> The fs/btrfs/volumes.c hunk won't apply cleanly, but it can auto-merge:
> 
>   $ git cherry-pick 8ba7d5f5ba931be68a94b8c91bcced1622934e7a
>   Auto-merging fs/btrfs/volumes.c
>   [detached HEAD 572410288a1070c1] btrfs: fix uninitialized variable warnings
>    Author: Genjian Zhang <zhanggenjian@kylinos.cn>
>    Date: Fri Mar 24 10:08:38 2023 +0800
>    2 files changed, 2 insertions(+), 2 deletions(-)

Now queued up, thanks.

greg k-h