| Message ID | 1461015341-20153-9-git-send-email-jack@suse.cz (mailing list archive) |
|---|---|
| State | Not Applicable |
| Headers | show |
On Mon, Apr 18, 2016 at 11:35:31PM +0200, Jan Kara wrote: > Currently ext4 treats DAX IO the same way as direct IO. I.e., it > allocates unwritten extents before IO is done and converts unwritten > extents afterwards. However this way DAX IO can race with page fault to > the same area: > > ext4_ext_direct_IO() dax_fault() > dax_io() > get_block() - allocates unwritten extent > copy_from_iter_pmem() > get_block() - converts > unwritten block to > written and zeroes it > out > ext4_convert_unwritten_extents() > > So data written with DAX IO gets lost. Similarly dax_new_buf() called > from dax_io() can overwrite data that has been already written to the > block via mmap. > > Fix the problem by using pre-zeroed blocks for DAX IO the same way as we > use them for DAX mmap. The downside of this solution is that every > allocating write writes each block twice (once zeros, once data). Fixing > the race with locking is possible as well however we would need to > lock-out faults for the whole range written to by DAX IO. And that is > not easy to do without locking-out faults for the whole file which seems > too aggressive. > > Signed-off-by: Jan Kara <jack@suse.cz> Just a couple of simplifications - feel free to ignore them if you don't think they are worth the effort. > --- > fs/ext4/ext4.h | 11 +++++++++-- > fs/ext4/file.c | 4 ++-- > fs/ext4/inode.c | 42 +++++++++++++++++++++++++++++++++--------- > 3 files changed, 44 insertions(+), 13 deletions(-) > > diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h > index 35792b430fb6..173da8faff81 100644 > --- a/fs/ext4/ext4.h > +++ b/fs/ext4/ext4.h > @@ -2521,8 +2521,8 @@ struct buffer_head *ext4_getblk(handle_t *, struct inode *, ext4_lblk_t, int); > struct buffer_head *ext4_bread(handle_t *, struct inode *, ext4_lblk_t, int); > int ext4_get_block_unwritten(struct inode *inode, sector_t iblock, > struct buffer_head *bh_result, int create); > -int ext4_dax_mmap_get_block(struct inode *inode, sector_t iblock, > - struct buffer_head *bh_result, int create); > +int ext4_dax_get_block(struct inode *inode, sector_t iblock, > + struct buffer_head *bh_result, int create); > int ext4_get_block(struct inode *inode, sector_t iblock, > struct buffer_head *bh_result, int create); > int ext4_dio_get_block(struct inode *inode, sector_t iblock, > @@ -3328,6 +3328,13 @@ static inline void ext4_clear_io_unwritten_flag(ext4_io_end_t *io_end) > } > } > > +static inline bool ext4_aligned_io(struct inode *inode, loff_t off, loff_t len) > +{ > + int blksize = 1 << inode->i_blkbits; > + > + return IS_ALIGNED(off, blksize) && IS_ALIGNED(off + len, blksize); This could be just a tiny bit simpler by doing return IS_ALIGNED(off, blksize) && IS_ALIGNED(len, blksize); ^^^ You've already made sure 'off' is aligned, so if 'len' is aligned 'off+len' will be aligned. > diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c > index 23fd0e0a9223..6d5d5c1db293 100644 > --- a/fs/ext4/inode.c > +++ b/fs/ext4/inode.c > @@ -3215,12 +3215,17 @@ static int ext4_releasepage(struct page *page, gfp_t wait) > } > > #ifdef CONFIG_FS_DAX > -int ext4_dax_mmap_get_block(struct inode *inode, sector_t iblock, > - struct buffer_head *bh_result, int create) > +/* > + * Get block function for DAX IO and mmap faults. It takes care of converting > + * unwritten extents to written ones and initializes new / converted blocks > + * to zeros. > + */ > +int ext4_dax_get_block(struct inode *inode, sector_t iblock, > + struct buffer_head *bh_result, int create) > { > int ret; > > - ext4_debug("ext4_dax_mmap_get_block: inode %lu, create flag %d\n", > + ext4_debug("ext4_dax_get_block: inode %lu, create flag %d\n", > inode->i_ino, create); This pattern could be improved by using "%s" and __func__ for the function name. That way you don't have to hunt through all your debug code and update strings when you rename a function. More importantly it prevents the strings from getting out of sync with the function name, resulting in confusing debug messages. > if (!create) > return _ext4_get_block(inode, iblock, bh_result, 0); > @@ -3233,9 +3238,9 @@ int ext4_dax_mmap_get_block(struct inode *inode, sector_t iblock, > > if (buffer_unwritten(bh_result)) { > /* > - * We are protected by i_mmap_sem so we know block cannot go > - * away from under us even though we dropped i_data_sem. > - * Convert extent to written and write zeros there. > + * We are protected by i_mmap_sem or i_mutex so we know block > + * cannot go away from under us even though we dropped > + * i_data_sem. Convert extent to written and write zeros there. > */ > ret = ext4_get_block_trans(inode, iblock, bh_result, > EXT4_GET_BLOCKS_CONVERT | > @@ -3250,6 +3255,14 @@ int ext4_dax_mmap_get_block(struct inode *inode, sector_t iblock, > clear_buffer_new(bh_result); > return 0; > } > +#else > +/* Just define empty function, it will never get called. */ > +int ext4_dax_get_block(struct inode *inode, sector_t iblock, > + struct buffer_head *bh_result, int create) > +{ > + BUG(); > + return 0; > +} You don't need this stub. All the uses of ext4_dax_get_block() are either within their own '#ifdef CONFIG_FS_DAX' sections, or they are in an "if (IS_DAX)" conditional. The latter will also be compiled out if CONFIG_FS_DAX isn't defined. This is because of the way that S_DAX is defined: #define S_DAX 8192 /* Direct Access, avoiding the page cache */ #else #define S_DAX 0 /* Make all the DAX code disappear */ #endif
On Fri 29-04-16 12:01:58, Ross Zwisler wrote: > On Mon, Apr 18, 2016 at 11:35:31PM +0200, Jan Kara wrote: > > --- > > fs/ext4/ext4.h | 11 +++++++++-- > > fs/ext4/file.c | 4 ++-- > > fs/ext4/inode.c | 42 +++++++++++++++++++++++++++++++++--------- > > 3 files changed, 44 insertions(+), 13 deletions(-) > > > > diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h > > index 35792b430fb6..173da8faff81 100644 > > --- a/fs/ext4/ext4.h > > +++ b/fs/ext4/ext4.h > > @@ -2521,8 +2521,8 @@ struct buffer_head *ext4_getblk(handle_t *, struct inode *, ext4_lblk_t, int); > > struct buffer_head *ext4_bread(handle_t *, struct inode *, ext4_lblk_t, int); > > int ext4_get_block_unwritten(struct inode *inode, sector_t iblock, > > struct buffer_head *bh_result, int create); > > -int ext4_dax_mmap_get_block(struct inode *inode, sector_t iblock, > > - struct buffer_head *bh_result, int create); > > +int ext4_dax_get_block(struct inode *inode, sector_t iblock, > > + struct buffer_head *bh_result, int create); > > int ext4_get_block(struct inode *inode, sector_t iblock, > > struct buffer_head *bh_result, int create); > > int ext4_dio_get_block(struct inode *inode, sector_t iblock, > > @@ -3328,6 +3328,13 @@ static inline void ext4_clear_io_unwritten_flag(ext4_io_end_t *io_end) > > } > > } > > > > +static inline bool ext4_aligned_io(struct inode *inode, loff_t off, loff_t len) > > +{ > > + int blksize = 1 << inode->i_blkbits; > > + > > + return IS_ALIGNED(off, blksize) && IS_ALIGNED(off + len, blksize); > > This could be just a tiny bit simpler by doing > > return IS_ALIGNED(off, blksize) && IS_ALIGNED(len, blksize); > ^^^ > > You've already made sure 'off' is aligned, so if 'len' is aligned 'off+len' > will be aligned. Good point, done. > > diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c > > index 23fd0e0a9223..6d5d5c1db293 100644 > > --- a/fs/ext4/inode.c > > +++ b/fs/ext4/inode.c > > @@ -3215,12 +3215,17 @@ static int ext4_releasepage(struct page *page, gfp_t wait) > > } > > > > #ifdef CONFIG_FS_DAX > > -int ext4_dax_mmap_get_block(struct inode *inode, sector_t iblock, > > - struct buffer_head *bh_result, int create) > > +/* > > + * Get block function for DAX IO and mmap faults. It takes care of converting > > + * unwritten extents to written ones and initializes new / converted blocks > > + * to zeros. > > + */ > > +int ext4_dax_get_block(struct inode *inode, sector_t iblock, > > + struct buffer_head *bh_result, int create) > > { > > int ret; > > > > - ext4_debug("ext4_dax_mmap_get_block: inode %lu, create flag %d\n", > > + ext4_debug("ext4_dax_get_block: inode %lu, create flag %d\n", > > inode->i_ino, create); > > This pattern could be improved by using "%s" and __func__ for the function > name. That way you don't have to hunt through all your debug code and update > strings when you rename a function. More importantly it prevents the strings > from getting out of sync with the function name, resulting in confusing debug > messages. Actually, ext4_debug() already automatically prepends the function name. So I've just discarded it from the format string. > > if (!create) > > return _ext4_get_block(inode, iblock, bh_result, 0); > > @@ -3233,9 +3238,9 @@ int ext4_dax_mmap_get_block(struct inode *inode, sector_t iblock, > > > > if (buffer_unwritten(bh_result)) { > > /* > > - * We are protected by i_mmap_sem so we know block cannot go > > - * away from under us even though we dropped i_data_sem. > > - * Convert extent to written and write zeros there. > > + * We are protected by i_mmap_sem or i_mutex so we know block > > + * cannot go away from under us even though we dropped > > + * i_data_sem. Convert extent to written and write zeros there. > > */ > > ret = ext4_get_block_trans(inode, iblock, bh_result, > > EXT4_GET_BLOCKS_CONVERT | > > @@ -3250,6 +3255,14 @@ int ext4_dax_mmap_get_block(struct inode *inode, sector_t iblock, > > clear_buffer_new(bh_result); > > return 0; > > } > > +#else > > +/* Just define empty function, it will never get called. */ > > +int ext4_dax_get_block(struct inode *inode, sector_t iblock, > > + struct buffer_head *bh_result, int create) > > +{ > > + BUG(); > > + return 0; > > +} > > You don't need this stub. All the uses of ext4_dax_get_block() are either > within their own '#ifdef CONFIG_FS_DAX' sections, or they are in an > "if (IS_DAX)" conditional. The latter will also be compiled out if > CONFIG_FS_DAX isn't defined. This is because of the way that S_DAX is > defined: > > #define S_DAX 8192 /* Direct Access, avoiding the page cache */ > #else > #define S_DAX 0 /* Make all the DAX code disappear */ > #endif OK, I agree it's likely not needed but I'm somewhat wary of relying on this compiler optimization. In some more complex cases for some compilers they needn't be able to infer that the code is actually dead and you'll get compilation error. IMO not worth those 7 lines of trivial code... So I've kept this. Honza
diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h index 35792b430fb6..173da8faff81 100644 --- a/fs/ext4/ext4.h +++ b/fs/ext4/ext4.h @@ -2521,8 +2521,8 @@ struct buffer_head *ext4_getblk(handle_t *, struct inode *, ext4_lblk_t, int); struct buffer_head *ext4_bread(handle_t *, struct inode *, ext4_lblk_t, int); int ext4_get_block_unwritten(struct inode *inode, sector_t iblock, struct buffer_head *bh_result, int create); -int ext4_dax_mmap_get_block(struct inode *inode, sector_t iblock, - struct buffer_head *bh_result, int create); +int ext4_dax_get_block(struct inode *inode, sector_t iblock, + struct buffer_head *bh_result, int create); int ext4_get_block(struct inode *inode, sector_t iblock, struct buffer_head *bh_result, int create); int ext4_dio_get_block(struct inode *inode, sector_t iblock, @@ -3328,6 +3328,13 @@ static inline void ext4_clear_io_unwritten_flag(ext4_io_end_t *io_end) } } +static inline bool ext4_aligned_io(struct inode *inode, loff_t off, loff_t len) +{ + int blksize = 1 << inode->i_blkbits; + + return IS_ALIGNED(off, blksize) && IS_ALIGNED(off + len, blksize); +} + #endif /* __KERNEL__ */ #define EFSBADCRC EBADMSG /* Bad CRC detected */ diff --git a/fs/ext4/file.c b/fs/ext4/file.c index b3a9c6eeadbc..2e9aa49a95fa 100644 --- a/fs/ext4/file.c +++ b/fs/ext4/file.c @@ -207,7 +207,7 @@ static int ext4_dax_fault(struct vm_area_struct *vma, struct vm_fault *vmf) if (IS_ERR(handle)) result = VM_FAULT_SIGBUS; else - result = __dax_fault(vma, vmf, ext4_dax_mmap_get_block); + result = __dax_fault(vma, vmf, ext4_dax_get_block); if (write) { if (!IS_ERR(handle)) @@ -243,7 +243,7 @@ static int ext4_dax_pmd_fault(struct vm_area_struct *vma, unsigned long addr, result = VM_FAULT_SIGBUS; else result = __dax_pmd_fault(vma, addr, pmd, flags, - ext4_dax_mmap_get_block); + ext4_dax_get_block); if (write) { if (!IS_ERR(handle)) diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c index 23fd0e0a9223..6d5d5c1db293 100644 --- a/fs/ext4/inode.c +++ b/fs/ext4/inode.c @@ -3215,12 +3215,17 @@ static int ext4_releasepage(struct page *page, gfp_t wait) } #ifdef CONFIG_FS_DAX -int ext4_dax_mmap_get_block(struct inode *inode, sector_t iblock, - struct buffer_head *bh_result, int create) +/* + * Get block function for DAX IO and mmap faults. It takes care of converting + * unwritten extents to written ones and initializes new / converted blocks + * to zeros. + */ +int ext4_dax_get_block(struct inode *inode, sector_t iblock, + struct buffer_head *bh_result, int create) { int ret; - ext4_debug("ext4_dax_mmap_get_block: inode %lu, create flag %d\n", + ext4_debug("ext4_dax_get_block: inode %lu, create flag %d\n", inode->i_ino, create); if (!create) return _ext4_get_block(inode, iblock, bh_result, 0); @@ -3233,9 +3238,9 @@ int ext4_dax_mmap_get_block(struct inode *inode, sector_t iblock, if (buffer_unwritten(bh_result)) { /* - * We are protected by i_mmap_sem so we know block cannot go - * away from under us even though we dropped i_data_sem. - * Convert extent to written and write zeros there. + * We are protected by i_mmap_sem or i_mutex so we know block + * cannot go away from under us even though we dropped + * i_data_sem. Convert extent to written and write zeros there. */ ret = ext4_get_block_trans(inode, iblock, bh_result, EXT4_GET_BLOCKS_CONVERT | @@ -3250,6 +3255,14 @@ int ext4_dax_mmap_get_block(struct inode *inode, sector_t iblock, clear_buffer_new(bh_result); return 0; } +#else +/* Just define empty function, it will never get called. */ +int ext4_dax_get_block(struct inode *inode, sector_t iblock, + struct buffer_head *bh_result, int create) +{ + BUG(); + return 0; +} #endif static int ext4_end_io_dio(struct kiocb *iocb, loff_t offset, @@ -3371,8 +3384,20 @@ static ssize_t ext4_direct_IO_write(struct kiocb *iocb, struct iov_iter *iter, iocb->private = NULL; if (overwrite) get_block_func = ext4_dio_get_block_overwrite; - else if (!ext4_test_inode_flag(inode, EXT4_INODE_EXTENTS) || - round_down(offset, 1 << inode->i_blkbits) >= inode->i_size) { + else if (IS_DAX(inode)) { + /* + * We can avoid zeroing for aligned DAX writes beyond EOF. Other + * writes need zeroing either because they can race with page + * faults or because they use partial blocks. + */ + if (round_down(offset, 1<<inode->i_blkbits) >= inode->i_size && + ext4_aligned_io(inode, offset, count)) + get_block_func = ext4_dio_get_block; + else + get_block_func = ext4_dax_get_block; + dio_flags = DIO_LOCKING; + } else if (!ext4_test_inode_flag(inode, EXT4_INODE_EXTENTS) || + round_down(offset, 1 << inode->i_blkbits) >= inode->i_size) { get_block_func = ext4_dio_get_block; dio_flags = DIO_LOCKING | DIO_SKIP_HOLES; } else if (is_sync_kiocb(iocb)) { @@ -3386,7 +3411,6 @@ static ssize_t ext4_direct_IO_write(struct kiocb *iocb, struct iov_iter *iter, BUG_ON(ext4_encrypted_inode(inode) && S_ISREG(inode->i_mode)); #endif if (IS_DAX(inode)) { - dio_flags &= ~DIO_SKIP_HOLES; ret = dax_do_io(iocb, inode, iter, offset, get_block_func, ext4_end_io_dio, dio_flags); } else
Currently ext4 treats DAX IO the same way as direct IO. I.e., it allocates unwritten extents before IO is done and converts unwritten extents afterwards. However this way DAX IO can race with page fault to the same area: ext4_ext_direct_IO() dax_fault() dax_io() get_block() - allocates unwritten extent copy_from_iter_pmem() get_block() - converts unwritten block to written and zeroes it out ext4_convert_unwritten_extents() So data written with DAX IO gets lost. Similarly dax_new_buf() called from dax_io() can overwrite data that has been already written to the block via mmap. Fix the problem by using pre-zeroed blocks for DAX IO the same way as we use them for DAX mmap. The downside of this solution is that every allocating write writes each block twice (once zeros, once data). Fixing the race with locking is possible as well however we would need to lock-out faults for the whole range written to by DAX IO. And that is not easy to do without locking-out faults for the whole file which seems too aggressive. Signed-off-by: Jan Kara <jack@suse.cz> --- fs/ext4/ext4.h | 11 +++++++++-- fs/ext4/file.c | 4 ++-- fs/ext4/inode.c | 42 +++++++++++++++++++++++++++++++++--------- 3 files changed, 44 insertions(+), 13 deletions(-)