| Message ID | 162914791879.197065.12619905059952917229.stgit@magnolia (mailing list archive) |
|---|---|
| Headers | show |
| Series | dax: fix broken pmem poison narrative | expand |
On Mon, Aug 16, 2021 at 02:05:18PM -0700, Darrick J. Wong wrote: > Hi all, > > Our current "advice" to people using persistent memory and FSDAX who > wish to recover upon receipt of a media error (aka 'hwpoison') event > from ACPI is to punch-hole that part of the file and then pwrite it, > which will magically cause the pmem to be reinitialized and the poison > to be cleared. > > Punching doesn't make any sense at all -- we don't allow userspace to > allocate from specific parts of the storage, and another writer could > grab the poisoned range in the meantime. In other words, the advice is > seriously overfitted to incidental xfs and ext4 behavior and can > completely fail. Worse yet, that concurrent writer now has to deal with > the poison that it didn't know about, and someone else is trying to fix. > > AFAICT, the only reason why the "punch and write" dance works at all is > that the XFS and ext4 currently call blkdev_issue_zeroout when > allocating pmem as part of a pwrite call. A pwrite without the punch > won't clear the poison, because pwrite on a DAX file calls > dax_direct_access to access the memory directly, and dax_direct_access > is only smart enough to bail out on poisoned pmem. It does not know how > to clear it. Userspace could solve the problem by calling FIEMAP and > issuing a BLKZEROOUT, but that requires rawio capabilities. > > The whole pmem poison recovery story is is wrong and needs to be > corrected ASAP before everyone else starts doing this. Therefore, > create a dax_zeroinit_range function that filesystems can call to reset > the contents of the pmem to a known value and clear any state associated > with the media error. Then, connect FALLOC_FL_ZERO_RANGE to this new > function (for DAX files) so that unprivileged userspace has a safe way > to reset the pmem and clear media errors. This is a sample copy of a SIGBUS handler that will dump out the siginfo data, call ZERO_RANGE to clear the poison, and then simulates being fortunate enough to be able to reconstruct the file contents from scratch. Note that I haven't tested this even with simulated pmem because I cannot figure out how to inject a poison error into the pmem in such a way that the nvdimm driver records it in the badblocks table. madvise(HWPOISON) calls the SIGBUS handler, but that code path never goes outside of the memory manager. int fd = open(...); char *data = mmap(fd, ... MAP_SYNC); static void handle_sigbus(int signal, siginfo_t *info, void *dontcare) { char *buf; loff_t err_offset = (char *)info->si_addr - data; loff_t err_len = (1ULL << info->si_addr_lsb); ssize_t ret; printf("RECEIVED SIGBUS (POISON HANDLER)!\n"); printf(" signal %d\n", info->si_signo); printf(" errno %d\n", info->si_errno); printf(" addr %p\n", info->si_addr); printf(" addr_lsb %d\n", info->si_addr_lsb); if (info->si_signo != SIGBUS) { printf(" code 0x%x\n", info->si_code); return; } switch (info->si_code) { case BUS_ADRALN: printf(" code: BUS_ADRALN\n"); break; case BUS_ADRERR: printf(" code: BUS_ADRERR\n"); break; case BUS_OBJERR: printf(" code: BUS_OBJERR\n"); break; case BUS_MCEERR_AR: printf(" code: BUS_MCEERR_AR\n"); break; case BUS_MCEERR_AO: printf(" code: BUS_MCEERR_AO\n"); break; default: printf(" code 0x%x\n", info->si_code); break; } printf(" err_offset %lld\n", (unsigned long long)err_offset); printf(" err_len %lld\n", (unsigned long long)err_len); if (info->si_code != BUS_MCEERR_AR) return; /* clear poison and reset pmem to initial value */ ret = fallocate(fd, FALLOC_FL_ZERO_RANGE, err_offset, err_len); if (ret) { perror("fallocate"); exit(9); } /* simulate being lucky enough to be able to reconstruct the data */ buf = malloc(err_len); if (!buf) { perror("malloc pwrite buf"); exit(10); } memset(buf, 0x59, err_len); ret = pwrite(fd, buf, err_len, err_offset); if (ret < 0) { perror("pwrite"); exit(11); } if (ret != err_len) { fprintf(stderr, "short write %zd bytes, wanted %lld\n", ret, (long long)err_len); exit(12); } free(buf); } --D > > If you're going to start using this mess, you probably ought to just > pull from my git trees, which are linked below. > > This is an extraordinary way to destroy everything. Enjoy! > Comments and questions are, as always, welcome. > > --D > > kernel git tree: > https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=dax-zeroinit-clear-poison-5.15 > --- > fs/dax.c | 72 +++++++++++++++++++++++++++++++++++++++++++++++++++ > fs/ext4/extents.c | 19 +++++++++++++ > fs/xfs/xfs_file.c | 20 ++++++++++++++ > include/linux/dax.h | 7 +++++ > 4 files changed, 118 insertions(+) >
On Mon, Aug 16, 2021 at 02:05:18PM -0700, Darrick J. Wong wrote: > AFAICT, the only reason why the "punch and write" dance works at all is > that the XFS and ext4 currently call blkdev_issue_zeroout when > allocating pmem as part of a pwrite call. A pwrite without the punch > won't clear the poison, because pwrite on a DAX file calls > dax_direct_access to access the memory directly, and dax_direct_access > is only smart enough to bail out on poisoned pmem. It does not know how > to clear it. Userspace could solve the problem by calling FIEMAP and > issuing a BLKZEROOUT, but that requires rawio capabilities. > > The whole pmem poison recovery story is is wrong and needs to be > corrected ASAP before everyone else starts doing this. Therefore, > create a dax_zeroinit_range function that filesystems can call to reset > the contents of the pmem to a known value and clear any state associated > with the media error. Then, connect FALLOC_FL_ZERO_RANGE to this new > function (for DAX files) so that unprivileged userspace has a safe way > to reset the pmem and clear media errors. I agree with the problem statement, but I don't think the fix is significantly better than what we have, as it still magically overloads other behavior. I'd rather have an explicit operation to clear the poison both at the syscall level (maybe another falloc mode), and the internal kernel API level (new method in dax_operations). Also for the next iteration please split the iomap changes from the usage in xfs.
On Tue, Aug 17, 2021 at 08:39:00AM +0100, Christoph Hellwig wrote: > On Mon, Aug 16, 2021 at 02:05:18PM -0700, Darrick J. Wong wrote: > > AFAICT, the only reason why the "punch and write" dance works at all is > > that the XFS and ext4 currently call blkdev_issue_zeroout when > > allocating pmem as part of a pwrite call. A pwrite without the punch > > won't clear the poison, because pwrite on a DAX file calls > > dax_direct_access to access the memory directly, and dax_direct_access > > is only smart enough to bail out on poisoned pmem. It does not know how > > to clear it. Userspace could solve the problem by calling FIEMAP and > > issuing a BLKZEROOUT, but that requires rawio capabilities. > > > > The whole pmem poison recovery story is is wrong and needs to be > > corrected ASAP before everyone else starts doing this. Therefore, > > create a dax_zeroinit_range function that filesystems can call to reset > > the contents of the pmem to a known value and clear any state associated > > with the media error. Then, connect FALLOC_FL_ZERO_RANGE to this new > > function (for DAX files) so that unprivileged userspace has a safe way > > to reset the pmem and clear media errors. > > I agree with the problem statement, but I don't think the fix is > significantly better than what we have, as it still magically overloads > other behavior. I'd rather have an explicit operation to clear the > poison both at the syscall level (maybe another falloc mode), and the > internal kernel API level (new method in dax_operations). I've long wondered why we can't just pass a write flag to the direct_access functions so that pmem_dax_direct_access can clear the poison. Then we ought to be able to tell userspace that they can recover from write errors by pwrite() or triggering a write fault on the page, I think. That's how userspace recovers from IO errors on traditional disks; I've never understood why it has to be any different now. > Also for the next iteration please split the iomap changes from the > usage in xfs. ok. --D
On 8/17/2021 12:39 AM, Christoph Hellwig wrote: > On Mon, Aug 16, 2021 at 02:05:18PM -0700, Darrick J. Wong wrote: >> AFAICT, the only reason why the "punch and write" dance works at all is >> that the XFS and ext4 currently call blkdev_issue_zeroout when >> allocating pmem as part of a pwrite call. A pwrite without the punch >> won't clear the poison, because pwrite on a DAX file calls >> dax_direct_access to access the memory directly, and dax_direct_access >> is only smart enough to bail out on poisoned pmem. It does not know how >> to clear it. Userspace could solve the problem by calling FIEMAP and >> issuing a BLKZEROOUT, but that requires rawio capabilities. >> >> The whole pmem poison recovery story is is wrong and needs to be >> corrected ASAP before everyone else starts doing this. Therefore, >> create a dax_zeroinit_range function that filesystems can call to reset >> the contents of the pmem to a known value and clear any state associated >> with the media error. Then, connect FALLOC_FL_ZERO_RANGE to this new >> function (for DAX files) so that unprivileged userspace has a safe way >> to reset the pmem and clear media errors. > > I agree with the problem statement, but I don't think the fix is > significantly better than what we have, as it still magically overloads This fix guarantees contiguous pmem backend, versus the existing holepunch+pwrite ends up allocating an unused(by the filesystem) pmem range from the pmem device, and the poison block stay poisoned, not cleared by the existing method. This brings 3 problem: 1. the backend become fragmented which negatively impacts the RDMA behavior, 2. what if the pmem device run out of extra clean blocks? 3. the user interface that clears poison when device is unmounted is risky in that it could damage filesystem metadata. That said, another angle to view this patch is that, FALLOC_FL_ZERO_RANGE is supposed to zero-range, for non-direct access block device, it's okay to not actually write zeros to the media, but pmem device is what-you-see-is-what-you-get, it seems making sense to actually writes zero. thanks, -jane > other behavior. I'd rather have an explicit operation to clear the > poison both at the syscall level (maybe another falloc mode), and the > internal kernel API level (new method in dax_operations). > > Also for the next iteration please split the iomap changes from the > usage in xfs. >