| Message ID | x49ehlzrt01.fsf@segfault.boston.devel.redhat.com (mailing list archive) |
|---|---|
| State | Deferred, archived |
| Headers | show |
Jeff Moyer <jmoyer@redhat.com> writes: > Mikulas Patocka <mpatocka@redhat.com> writes: > >> Hi Jeff >> >> Thanks for testing. >> >> It would be interesting ... what happens if you take the patch 3, leave >> "struct percpu_rw_semaphore bd_block_size_semaphore" in "struct >> block_device", but remove any use of the semaphore from fs/block_dev.c? - >> will the performance be like unpatched kernel or like patch 3? It could be >> that the change in the alignment affects performance on your CPU too, just >> differently than on my CPU. > > It turns out to be exactly the same performance as with the 3rd patch > applied, so I guess it does have something to do with cache alignment. > Here is the patch (against vanilla) I ended up testing. Let me know if > I've botched it somehow. > > So, I next up I'll play similar tricks to what you did (padding struct > block_device in all kernels) to eliminate the differences due to > structure alignment and provide a clear picture of what the locking > effects are. After trying again with the same padding you used in the struct bdev_inode, I see no performance differences between any of the patches. I tried bumping up the number of threads to saturate the number of cpus on a single NUMA node on my hardware, but that resulted in lower IOPS to the device, and hence consumption of less CPU time. So, I believe my results to be inconclusive. After talking with Vivek about the problem, he had mentioned that it might be worth investigating whether bd_block_size could be protected using SRCU. I looked into it, and the one thing I couldn't reconcile is updating both the bd_block_size and the inode->i_blkbits at the same time. It would involve (afaiui) adding fields to both the inode and the block_device data structures and using rcu_assign_pointer and rcu_dereference to modify and access the fields, and both fields would need to protected by the same struct srcu_struct. I'm not sure whether that's a desirable approach. When I started to implement it, it got ugly pretty quickly. What do others think? For now, my preference is to get the full patch set in. I will continue to investigate the performance impact of the data structure size changes that I've been seeing. So, for the four patches: Acked-by: Jeff Moyer <jmoyer@redhat.com> Jens, can you have a look at the patch set? We are seeing problem reports of this in the wild[1][2]. Cheers, Jeff [1] https://bugzilla.redhat.com/show_bug.cgi?id=824107 [2] https://bugzilla.redhat.com/show_bug.cgi?id=812129 -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
On 2012-09-25 19:49, Jeff Moyer wrote: > Jeff Moyer <jmoyer@redhat.com> writes: > >> Mikulas Patocka <mpatocka@redhat.com> writes: >> >>> Hi Jeff >>> >>> Thanks for testing. >>> >>> It would be interesting ... what happens if you take the patch 3, leave >>> "struct percpu_rw_semaphore bd_block_size_semaphore" in "struct >>> block_device", but remove any use of the semaphore from fs/block_dev.c? - >>> will the performance be like unpatched kernel or like patch 3? It could be >>> that the change in the alignment affects performance on your CPU too, just >>> differently than on my CPU. >> >> It turns out to be exactly the same performance as with the 3rd patch >> applied, so I guess it does have something to do with cache alignment. >> Here is the patch (against vanilla) I ended up testing. Let me know if >> I've botched it somehow. >> >> So, I next up I'll play similar tricks to what you did (padding struct >> block_device in all kernels) to eliminate the differences due to >> structure alignment and provide a clear picture of what the locking >> effects are. > > After trying again with the same padding you used in the struct > bdev_inode, I see no performance differences between any of the > patches. I tried bumping up the number of threads to saturate the > number of cpus on a single NUMA node on my hardware, but that resulted > in lower IOPS to the device, and hence consumption of less CPU time. > So, I believe my results to be inconclusive. > > After talking with Vivek about the problem, he had mentioned that it > might be worth investigating whether bd_block_size could be protected > using SRCU. I looked into it, and the one thing I couldn't reconcile is > updating both the bd_block_size and the inode->i_blkbits at the same > time. It would involve (afaiui) adding fields to both the inode and the > block_device data structures and using rcu_assign_pointer and > rcu_dereference to modify and access the fields, and both fields would > need to protected by the same struct srcu_struct. I'm not sure whether > that's a desirable approach. When I started to implement it, it got > ugly pretty quickly. What do others think? > > For now, my preference is to get the full patch set in. I will continue > to investigate the performance impact of the data structure size changes > that I've been seeing. > > So, for the four patches: > > Acked-by: Jeff Moyer <jmoyer@redhat.com> > > Jens, can you have a look at the patch set? We are seeing problem > reports of this in the wild[1][2]. I'll queue it up for 3.7. I can run my regular testing on the 8-way, it has a nack for showing scaling problems very nicely in aio/dio. As long as we're not adding per-inode cache line dirtying per IO (and the per-cpu rw sem looks OK), then I don't think there's too much to worry about.
On 2012-09-25 19:59, Jens Axboe wrote: > On 2012-09-25 19:49, Jeff Moyer wrote: >> Jeff Moyer <jmoyer@redhat.com> writes: >> >>> Mikulas Patocka <mpatocka@redhat.com> writes: >>> >>>> Hi Jeff >>>> >>>> Thanks for testing. >>>> >>>> It would be interesting ... what happens if you take the patch 3, leave >>>> "struct percpu_rw_semaphore bd_block_size_semaphore" in "struct >>>> block_device", but remove any use of the semaphore from fs/block_dev.c? - >>>> will the performance be like unpatched kernel or like patch 3? It could be >>>> that the change in the alignment affects performance on your CPU too, just >>>> differently than on my CPU. >>> >>> It turns out to be exactly the same performance as with the 3rd patch >>> applied, so I guess it does have something to do with cache alignment. >>> Here is the patch (against vanilla) I ended up testing. Let me know if >>> I've botched it somehow. >>> >>> So, I next up I'll play similar tricks to what you did (padding struct >>> block_device in all kernels) to eliminate the differences due to >>> structure alignment and provide a clear picture of what the locking >>> effects are. >> >> After trying again with the same padding you used in the struct >> bdev_inode, I see no performance differences between any of the >> patches. I tried bumping up the number of threads to saturate the >> number of cpus on a single NUMA node on my hardware, but that resulted >> in lower IOPS to the device, and hence consumption of less CPU time. >> So, I believe my results to be inconclusive. >> >> After talking with Vivek about the problem, he had mentioned that it >> might be worth investigating whether bd_block_size could be protected >> using SRCU. I looked into it, and the one thing I couldn't reconcile is >> updating both the bd_block_size and the inode->i_blkbits at the same >> time. It would involve (afaiui) adding fields to both the inode and the >> block_device data structures and using rcu_assign_pointer and >> rcu_dereference to modify and access the fields, and both fields would >> need to protected by the same struct srcu_struct. I'm not sure whether >> that's a desirable approach. When I started to implement it, it got >> ugly pretty quickly. What do others think? >> >> For now, my preference is to get the full patch set in. I will continue >> to investigate the performance impact of the data structure size changes >> that I've been seeing. >> >> So, for the four patches: >> >> Acked-by: Jeff Moyer <jmoyer@redhat.com> >> >> Jens, can you have a look at the patch set? We are seeing problem >> reports of this in the wild[1][2]. > > I'll queue it up for 3.7. I can run my regular testing on the 8-way, it > has a nack for showing scaling problems very nicely in aio/dio. As long > as we're not adding per-inode cache line dirtying per IO (and the > per-cpu rw sem looks OK), then I don't think there's too much to worry > about. I take that back. The series doesn't apply to my current tree. Not too unexpected, since it's some weeks old. But more importantly, please send this is a "real" patch series. I don't want to see two implementations of rw semaphores. I think it's perfectly fine to first do a regular rw sem, then a last patch adding the cache friendly variant from Eric and converting to that. In other words, get rid of 3/4.
On Tue, 25 Sep 2012, Jeff Moyer wrote: > Jeff Moyer <jmoyer@redhat.com> writes: > > > Mikulas Patocka <mpatocka@redhat.com> writes: > > > >> Hi Jeff > >> > >> Thanks for testing. > >> > >> It would be interesting ... what happens if you take the patch 3, leave > >> "struct percpu_rw_semaphore bd_block_size_semaphore" in "struct > >> block_device", but remove any use of the semaphore from fs/block_dev.c? - > >> will the performance be like unpatched kernel or like patch 3? It could be > >> that the change in the alignment affects performance on your CPU too, just > >> differently than on my CPU. > > > > It turns out to be exactly the same performance as with the 3rd patch > > applied, so I guess it does have something to do with cache alignment. > > Here is the patch (against vanilla) I ended up testing. Let me know if > > I've botched it somehow. > > > > So, I next up I'll play similar tricks to what you did (padding struct > > block_device in all kernels) to eliminate the differences due to > > structure alignment and provide a clear picture of what the locking > > effects are. > > After trying again with the same padding you used in the struct > bdev_inode, I see no performance differences between any of the > patches. I tried bumping up the number of threads to saturate the > number of cpus on a single NUMA node on my hardware, but that resulted > in lower IOPS to the device, and hence consumption of less CPU time. > So, I believe my results to be inconclusive. For me, the fourth patch with RCU-based locks performed better, so I am submitting that. > After talking with Vivek about the problem, he had mentioned that it > might be worth investigating whether bd_block_size could be protected > using SRCU. I looked into it, and the one thing I couldn't reconcile is > updating both the bd_block_size and the inode->i_blkbits at the same > time. It would involve (afaiui) adding fields to both the inode and the > block_device data structures and using rcu_assign_pointer and > rcu_dereference to modify and access the fields, and both fields would > need to protected by the same struct srcu_struct. I'm not sure whether > that's a desirable approach. When I started to implement it, it got > ugly pretty quickly. What do others think? Using RCU doesn't seem sensible to me (except for lock implementation, as it is in patch 4). The major problem is that the block layer reads blocksize multiple times and when different values are read, a crash may happen - RCU doesn't protect you against that - if you read a variable multiple times in a RCU-protected section, you can still get different results. If we wanted to use RCU, we would have to read blocksize just once and pass the value between all functions involved - that would result in a massive code change. > For now, my preference is to get the full patch set in. I will continue > to investigate the performance impact of the data structure size changes > that I've been seeing. Yes, we should get the patches to the kernel. Mikulas > So, for the four patches: > > Acked-by: Jeff Moyer <jmoyer@redhat.com> > > Jens, can you have a look at the patch set? We are seeing problem > reports of this in the wild[1][2]. > > Cheers, > Jeff > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=824107 > [2] https://bugzilla.redhat.com/show_bug.cgi?id=812129 > -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
Mikulas Patocka <mpatocka@redhat.com> writes: > On Tue, 25 Sep 2012, Jeff Moyer wrote: > >> Jeff Moyer <jmoyer@redhat.com> writes: >> >> > Mikulas Patocka <mpatocka@redhat.com> writes: >> > >> >> Hi Jeff >> >> >> >> Thanks for testing. >> >> >> >> It would be interesting ... what happens if you take the patch 3, leave >> >> "struct percpu_rw_semaphore bd_block_size_semaphore" in "struct >> >> block_device", but remove any use of the semaphore from fs/block_dev.c? - >> >> will the performance be like unpatched kernel or like patch 3? It could be >> >> that the change in the alignment affects performance on your CPU too, just >> >> differently than on my CPU. >> > >> > It turns out to be exactly the same performance as with the 3rd patch >> > applied, so I guess it does have something to do with cache alignment. >> > Here is the patch (against vanilla) I ended up testing. Let me know if >> > I've botched it somehow. >> > >> > So, I next up I'll play similar tricks to what you did (padding struct >> > block_device in all kernels) to eliminate the differences due to >> > structure alignment and provide a clear picture of what the locking >> > effects are. >> >> After trying again with the same padding you used in the struct >> bdev_inode, I see no performance differences between any of the >> patches. I tried bumping up the number of threads to saturate the >> number of cpus on a single NUMA node on my hardware, but that resulted >> in lower IOPS to the device, and hence consumption of less CPU time. >> So, I believe my results to be inconclusive. > > For me, the fourth patch with RCU-based locks performed better, so I am > submitting that. > >> After talking with Vivek about the problem, he had mentioned that it >> might be worth investigating whether bd_block_size could be protected >> using SRCU. I looked into it, and the one thing I couldn't reconcile is >> updating both the bd_block_size and the inode->i_blkbits at the same >> time. It would involve (afaiui) adding fields to both the inode and the >> block_device data structures and using rcu_assign_pointer and >> rcu_dereference to modify and access the fields, and both fields would >> need to protected by the same struct srcu_struct. I'm not sure whether >> that's a desirable approach. When I started to implement it, it got >> ugly pretty quickly. What do others think? > > Using RCU doesn't seem sensible to me (except for lock implementation, as > it is in patch 4). The major problem is that the block layer reads > blocksize multiple times and when different values are read, a crash may > happen - RCU doesn't protect you against that - if you read a variable > multiple times in a RCU-protected section, you can still get different > results. SRCU is sleepable, so could be (I think) used in the same manner as your rw semaphore. The only difference is that it would require changing the bd_blocksize and the i_blkbits to pointers and protecting them both with the same srcu struct. Then, the inode i_blkbits would also need to be special cased, so that we only require such handling when it is associated with a block device. It got messy. > If we wanted to use RCU, we would have to read blocksize just once and > pass the value between all functions involved - that would result in a > massive code change. If we did that, we wouldn't need rcu at all, would we? Cheers, Jeff -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
On Wed, 26 Sep 2012, Jeff Moyer wrote: > Mikulas Patocka <mpatocka@redhat.com> writes: > > > On Tue, 25 Sep 2012, Jeff Moyer wrote: > > > >> Jeff Moyer <jmoyer@redhat.com> writes: > >> > >> > Mikulas Patocka <mpatocka@redhat.com> writes: > >> > > >> >> Hi Jeff > >> >> > >> >> Thanks for testing. > >> >> > >> >> It would be interesting ... what happens if you take the patch 3, leave > >> >> "struct percpu_rw_semaphore bd_block_size_semaphore" in "struct > >> >> block_device", but remove any use of the semaphore from fs/block_dev.c? - > >> >> will the performance be like unpatched kernel or like patch 3? It could be > >> >> that the change in the alignment affects performance on your CPU too, just > >> >> differently than on my CPU. > >> > > >> > It turns out to be exactly the same performance as with the 3rd patch > >> > applied, so I guess it does have something to do with cache alignment. > >> > Here is the patch (against vanilla) I ended up testing. Let me know if > >> > I've botched it somehow. > >> > > >> > So, I next up I'll play similar tricks to what you did (padding struct > >> > block_device in all kernels) to eliminate the differences due to > >> > structure alignment and provide a clear picture of what the locking > >> > effects are. > >> > >> After trying again with the same padding you used in the struct > >> bdev_inode, I see no performance differences between any of the > >> patches. I tried bumping up the number of threads to saturate the > >> number of cpus on a single NUMA node on my hardware, but that resulted > >> in lower IOPS to the device, and hence consumption of less CPU time. > >> So, I believe my results to be inconclusive. > > > > For me, the fourth patch with RCU-based locks performed better, so I am > > submitting that. > > > >> After talking with Vivek about the problem, he had mentioned that it > >> might be worth investigating whether bd_block_size could be protected > >> using SRCU. I looked into it, and the one thing I couldn't reconcile is > >> updating both the bd_block_size and the inode->i_blkbits at the same > >> time. It would involve (afaiui) adding fields to both the inode and the > >> block_device data structures and using rcu_assign_pointer and > >> rcu_dereference to modify and access the fields, and both fields would > >> need to protected by the same struct srcu_struct. I'm not sure whether > >> that's a desirable approach. When I started to implement it, it got > >> ugly pretty quickly. What do others think? > > > > Using RCU doesn't seem sensible to me (except for lock implementation, as > > it is in patch 4). The major problem is that the block layer reads > > blocksize multiple times and when different values are read, a crash may > > happen - RCU doesn't protect you against that - if you read a variable > > multiple times in a RCU-protected section, you can still get different > > results. > > SRCU is sleepable, so could be (I think) used in the same manner as your > rw semaphore. The only difference is that it would require changing the > bd_blocksize and the i_blkbits to pointers and protecting them both with > the same srcu struct. Then, the inode i_blkbits would also need to be > special cased, so that we only require such handling when it is > associated with a block device. It got messy. No, it couldn't be used this way. If you do srcu_read_lock(&srcu) ptr1 = srcu_dereference(pointer, &srcu); ptr2 = srcu_dereference(pointer, &srcu); srcu_read_unlock(&srcu) it doesn't guarantee that ptr1 == ptr2. All that it guarantees is that when synchronize_srcu exits, there are no references to the old structure. But after rcu_assign_pointer and before synchronize_srcu exits, readers can read both old and new value of the pointer and it is not specified which value do they read. > > If we wanted to use RCU, we would have to read blocksize just once and > > pass the value between all functions involved - that would result in a > > massive code change. > > If we did that, we wouldn't need rcu at all, would we? Yes, we wouldn't need RCU then. Mikulas > Cheers, > Jeff > -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
diff --git a/drivers/char/raw.c b/drivers/char/raw.c index 54a3a6d..0bb207e 100644 --- a/drivers/char/raw.c +++ b/drivers/char/raw.c @@ -285,7 +285,7 @@ static long raw_ctl_compat_ioctl(struct file *file, unsigned int cmd, static const struct file_operations raw_fops = { .read = do_sync_read, - .aio_read = generic_file_aio_read, + .aio_read = blkdev_aio_read, .write = do_sync_write, .aio_write = blkdev_aio_write, .fsync = blkdev_fsync, diff --git a/fs/block_dev.c b/fs/block_dev.c index 38e721b..c7514b5 100644 --- a/fs/block_dev.c +++ b/fs/block_dev.c @@ -116,6 +116,8 @@ EXPORT_SYMBOL(invalidate_bdev); int set_blocksize(struct block_device *bdev, int size) { + struct address_space *mapping; + /* Size must be a power of two, and between 512 and PAGE_SIZE */ if (size > PAGE_SIZE || size < 512 || !is_power_of_2(size)) return -EINVAL; @@ -124,6 +126,16 @@ int set_blocksize(struct block_device *bdev, int size) if (size < bdev_logical_block_size(bdev)) return -EINVAL; + /* Check that the block device is not memory mapped */ + mapping = bdev->bd_inode->i_mapping; + mutex_lock(&mapping->i_mmap_mutex); + if (!prio_tree_empty(&mapping->i_mmap) || + !list_empty(&mapping->i_mmap_nonlinear)) { + mutex_unlock(&mapping->i_mmap_mutex); + return -EBUSY; + } + mutex_unlock(&mapping->i_mmap_mutex); + /* Don't change the size if it is same as current */ if (bdev->bd_block_size != size) { sync_blockdev(bdev); @@ -131,6 +143,7 @@ int set_blocksize(struct block_device *bdev, int size) bdev->bd_inode->i_blkbits = blksize_bits(size); kill_bdev(bdev); } + return 0; } @@ -441,6 +454,12 @@ static struct inode *bdev_alloc_inode(struct super_block *sb) struct bdev_inode *ei = kmem_cache_alloc(bdev_cachep, GFP_KERNEL); if (!ei) return NULL; + + if (unlikely(percpu_init_rwsem(&ei->bdev.bd_block_size_semaphore))) { + kmem_cache_free(bdev_cachep, ei); + return NULL; + } + return &ei->vfs_inode; } @@ -449,6 +468,8 @@ static void bdev_i_callback(struct rcu_head *head) struct inode *inode = container_of(head, struct inode, i_rcu); struct bdev_inode *bdi = BDEV_I(inode); + percpu_free_rwsem(&bdi->bdev.bd_block_size_semaphore); + kmem_cache_free(bdev_cachep, bdi); } @@ -1567,6 +1588,19 @@ static long block_ioctl(struct file *file, unsigned cmd, unsigned long arg) return blkdev_ioctl(bdev, mode, cmd, arg); } +ssize_t blkdev_aio_read(struct kiocb *iocb, const struct iovec *iov, + unsigned long nr_segs, loff_t pos) +{ + ssize_t ret; + struct block_device *bdev = I_BDEV(iocb->ki_filp->f_mapping->host); + percpu_rwsem_ptr p; + + ret = generic_file_aio_read(iocb, iov, nr_segs, pos); + + return ret; +} +EXPORT_SYMBOL_GPL(blkdev_aio_read); + /* * Write data to the block device. Only intended for the block device itself * and the raw driver which basically is a fake block device. @@ -1578,6 +1612,7 @@ ssize_t blkdev_aio_write(struct kiocb *iocb, const struct iovec *iov, unsigned long nr_segs, loff_t pos) { struct file *file = iocb->ki_filp; + struct block_device *bdev = I_BDEV(file->f_mapping->host); struct blk_plug plug; ssize_t ret; @@ -1597,6 +1632,16 @@ ssize_t blkdev_aio_write(struct kiocb *iocb, const struct iovec *iov, } EXPORT_SYMBOL_GPL(blkdev_aio_write); +int blkdev_mmap(struct file *file, struct vm_area_struct *vma) +{ + int ret; + struct block_device *bdev = I_BDEV(file->f_mapping->host); + + ret = generic_file_mmap(file, vma); + + return ret; +} + /* * Try to release a page associated with block device when the system * is under memory pressure. @@ -1627,9 +1672,9 @@ const struct file_operations def_blk_fops = { .llseek = block_llseek, .read = do_sync_read, .write = do_sync_write, - .aio_read = generic_file_aio_read, + .aio_read = blkdev_aio_read, .aio_write = blkdev_aio_write, - .mmap = generic_file_mmap, + .mmap = blkdev_mmap, .fsync = blkdev_fsync, .unlocked_ioctl = block_ioctl, #ifdef CONFIG_COMPAT diff --git a/include/linux/fs.h b/include/linux/fs.h index aa11047..15c481d 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -10,6 +10,7 @@ #include <linux/ioctl.h> #include <linux/blk_types.h> #include <linux/types.h> +#include <linux/percpu-rwsem.h> /* * It's silly to have NR_OPEN bigger than NR_FILE, but you can change @@ -724,6 +725,8 @@ struct block_device { int bd_fsfreeze_count; /* Mutex for freeze */ struct mutex bd_fsfreeze_mutex; + /* A semaphore that prevents I/O while block size is being changed */ + struct percpu_rw_semaphore bd_block_size_semaphore; }; /* @@ -2564,6 +2567,8 @@ extern int generic_segment_checks(const struct iovec *iov, unsigned long *nr_segs, size_t *count, int access_flags); /* fs/block_dev.c */ +extern ssize_t blkdev_aio_read(struct kiocb *iocb, const struct iovec *iov, + unsigned long nr_segs, loff_t pos); extern ssize_t blkdev_aio_write(struct kiocb *iocb, const struct iovec *iov, unsigned long nr_segs, loff_t pos); extern int blkdev_fsync(struct file *filp, loff_t start, loff_t end,