[PATCHSET,v24.0,0/7] xfs: stage repair information in pageable memory

Message ID	167243835481.692498.14657125042725378987.stgit@magnolia (mailing list archive)
Headers	show Return-Path: <linux-fsdevel-owner@vger.kernel.org> Subject: [PATCHSET v24.0 0/7] xfs: stage repair information in pageable memory From: "Darrick J. Wong" <djwong@kernel.org> To: djwong@kernel.org Cc: linux-xfs@vger.kernel.org, willy@infradead.org, linux-fsdevel@vger.kernel.org Date: Fri, 30 Dec 2022 14:12:34 -0800 Message-ID: <167243835481.692498.14657125042725378987.stgit@magnolia> In-Reply-To: <Y69Unb7KRM5awJoV@magnolia> References: <Y69Unb7KRM5awJoV@magnolia> User-Agent: StGit/0.19 MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Precedence: bulk
Series	xfs: stage repair information in pageable memory \| expand [PATCHSET,v24.0,0/7] xfs: stage repair information in pageable memory [1/7] xfs: create a big array data structure [2/7] xfs: enable sorting of xfile-backed arrays [3/7] xfs: convert xfarray insertion sort to heapsort using scratchpad memory [4/7] xfs: teach xfile to pass back direct-map pages to caller [5/7] xfs: speed up xfarray sort by sorting xfile page contents directly [6/7] xfs: cache pages used for xfarray quicksort convergence [7/7] xfs: improve xfarray quicksort pivot

Message ID

167243835481.692498.14657125042725378987.stgit@magnolia (mailing list archive)

Headers

Subject: [PATCHSET v24.0 0/7] xfs: stage repair information in pageable memory
From: "Darrick J. Wong" <djwong@kernel.org>
To: djwong@kernel.org
Cc: linux-xfs@vger.kernel.org, willy@infradead.org,
        linux-fsdevel@vger.kernel.org
Date: Fri, 30 Dec 2022 14:12:34 -0800
Message-ID: <167243835481.692498.14657125042725378987.stgit@magnolia>
In-Reply-To: <Y69Unb7KRM5awJoV@magnolia>
References: <Y69Unb7KRM5awJoV@magnolia>
User-Agent: StGit/0.19
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
Precedence: bulk

Series

xfs: stage repair information in pageable memory | expand

Message

Darrick J. Wong Dec. 30, 2022, 10:12 p.m. UTC

Hi all,

In general, online repair of an indexed record set walks the filesystem
looking for records.  These records are sorted and bulk-loaded into a
new btree.  To make this happen without pinning gigabytes of metadata in
memory, first create an abstraction ('xfile') of memfd files so that
kernel code can access paged memory, and then an array abstraction
('xfarray') based on xfiles so that online repair can create an array of
new records without pinning memory.

These two data storage abstractions are critical for repair of space
metadata -- the memory used is pageable, which helps us avoid pinning
kernel memory and driving OOM problems; and they are byte-accessible
enough that we can use them like (very slow and programmatic) memory
buffers.

Later patchsets will build on this functionality to provide blob storage
and btrees.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=big-array
---
 fs/xfs/Kconfig         |    1 
 fs/xfs/Makefile        |    2 
 fs/xfs/scrub/trace.c   |    4 
 fs/xfs/scrub/trace.h   |  262 ++++++++++++
 fs/xfs/scrub/xfarray.c | 1084 ++++++++++++++++++++++++++++++++++++++++++++++++
 fs/xfs/scrub/xfarray.h |  142 ++++++
 fs/xfs/scrub/xfile.c   |  426 +++++++++++++++++++
 fs/xfs/scrub/xfile.h   |   78 +++
 8 files changed, 1998 insertions(+), 1 deletion(-)
 create mode 100644 fs/xfs/scrub/xfarray.c
 create mode 100644 fs/xfs/scrub/xfarray.h
 create mode 100644 fs/xfs/scrub/xfile.c
 create mode 100644 fs/xfs/scrub/xfile.h