diff mbox

[dm-devel] split scsi passthrough fields out of struct request V2

Message ID e1d5f4e6-c63f-545d-710f-d4d10fa8d2e0@fb.com (mailing list archive)
State Not Applicable, archived
Headers show

Commit Message

Jens Axboe Jan. 26, 2017, 9:51 p.m. UTC 
On 01/26/2017 02:47 PM, Bart Van Assche wrote:
> (gdb) list *(blk_mq_sched_get_request+0x310)
> 0xffffffff8132dcf0 is in blk_mq_sched_get_request (block/blk-mq-sched.c:136).
> 131                                     rq->rq_flags |= RQF_QUEUED;
> 132                     } else
> 133                             rq = __blk_mq_alloc_request(data, op);
> 134             } else {
> 135                     rq = __blk_mq_alloc_request(data, op);
> 136                     data->hctx->tags->rqs[rq->tag] = rq;
> 137             }
> 138
> 139             if (rq) {
> 140                     if (!op_is_flush(op)) {
> 
> (gdb) disas blk_mq_sched_get_request
> [ ... ]
>    0xffffffff8132dce3 <+771>:   callq  0xffffffff81324ab0 <__blk_mq_alloc_request>
>    0xffffffff8132dce8 <+776>:   mov    %rax,%rcx
>    0xffffffff8132dceb <+779>:   mov    0x18(%r12),%rax
>    0xffffffff8132dcf0 <+784>:   movslq 0x5c(%rcx),%rdx
> [ ... ]
> (gdb) print &((struct request *)0)->tag
> $1 = (int *) 0x5c <irq_stack_union+92>
> 
> I think this means that rq == NULL and that a test for rq is missing after the
> __blk_mq_alloc_request() call?

That is exactly what it means, looks like that one path doesn't handle
that.  You'd have to exhaust the pool with atomic allocs for this to
trigger, we don't do that at all in the normal IO path. So good catch,
must be the dm part that enables this since it does NOWAIT allocations.

Comments

Bart Van Assche Jan. 26, 2017, 11:14 p.m. UTC | #1 
On Thu, 2017-01-26 at 14:51 -0700, Jens Axboe wrote:
> That is exactly what it means, looks like that one path doesn't handle
> that.  You'd have to exhaust the pool with atomic allocs for this to
> trigger, we don't do that at all in the normal IO path. So good catch,
> must be the dm part that enables this since it does NOWAIT allocations.
> 
> 
> diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c
> index 3136696f4991..c27613de80c5 100644
> --- a/block/blk-mq-sched.c
> +++ b/block/blk-mq-sched.c
> @@ -134,7 +134,8 @@ struct request *blk_mq_sched_get_request(struct request_queue *q,
>  			rq = __blk_mq_alloc_request(data, op);
>  	} else {
>  		rq = __blk_mq_alloc_request(data, op);
> -		data->hctx->tags->rqs[rq->tag] = rq;
> +		if (rq)
> +			data->hctx->tags->rqs[rq->tag] = rq;
>  	}
>  
>  	if (rq) {

Hello Jens,

With these two patches applied the scheduling-while-atomic complaint and
the oops are gone. However, some tasks get stuck. Is the console output
below enough to figure out what is going on or do you want me to bisect
this? I don't think that any requests got stuck since no pending requests
are shown in /sys/block/*/mq/*/{pending,*/rq_list}.

Thanks,

Bart.

[  663.217074] sysrq: SysRq : Show Blocked State
[  663.217111]   task                        PC stack   pid father
[  663.217237] kworker/10:0    D    0    71      2 0x00000000
[  663.217267] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.217289] Call Trace:
[  663.217313]  __schedule+0x2da/0xb00
[  663.217337]  ? bit_wait+0x50/0x50
[  663.217360]  schedule+0x38/0x90
[  663.217383]  schedule_timeout+0x2fe/0x640
[  663.217406]  ? mark_held_locks+0x6f/0xa0
[  663.217430]  ? ktime_get+0x74/0x130
[  663.217452]  ? bit_wait+0x50/0x50
[  663.217473]  ? trace_hardirqs_on_caller+0xf9/0x1b0
[  663.217497]  ? trace_hardirqs_on+0xd/0x10
[  663.217520]  ? ktime_get+0x98/0x130
[  663.217542]  ? __delayacct_blkio_start+0x1a/0x30
[  663.217564]  ? bit_wait+0x50/0x50
[  663.217586]  io_schedule_timeout+0x9f/0x110
[  663.217609]  bit_wait_io+0x16/0x60
[  663.217637]  __wait_on_bit+0x53/0x80
[  663.217659]  ? bit_wait+0x50/0x50
[  663.217680]  out_of_line_wait_on_bit+0x6e/0x80
[  663.217703]  ? prepare_to_wait_event+0x170/0x170
[  663.217727]  sync_mapping_buffers+0x22f/0x390
[  663.217750]  __generic_file_fsync+0x4d/0x90
[  663.217772]  ext4_sync_file+0x2b4/0x540
[  663.217793]  vfs_fsync_range+0x46/0xa0
[  663.217814]  dio_complete+0x181/0x1b0
[  663.217835]  dio_aio_complete_work+0x17/0x20
[  663.217856]  process_one_work+0x208/0x6a0
[  663.217878]  ? process_one_work+0x18d/0x6a0
[  663.217899]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.217919]  worker_thread+0x49/0x4a0
[  663.217941]  kthread+0x107/0x140
[  663.217962]  ? process_one_work+0x6a0/0x6a0
[  663.217982]  ? kthread_create_on_node+0x40/0x40
[  663.218003]  ? acpi_ps_create_op+0x1c0/0x35a
[  663.218025]  ? acpi_ps_alloc_op+0x3f/0x80
[  663.218045]  ? acpi_os_acquire_object+0x28/0x2a
[  663.218068]  ret_from_fork+0x2e/0x40
[  663.218112] kworker/u24:8   D    0   141      2 0x00000000
[  663.218139] Workqueue: writeback wb_workfn (flush-254:0)
[  663.218160] Call Trace:
[  663.218182]  __schedule+0x2da/0xb00
[  663.218209]  schedule+0x38/0x90
[  663.218233]  schedule_timeout+0x2fe/0x640
[  663.218263]  ? ktime_get+0x74/0x130
[  663.218284]  ? trace_hardirqs_on_caller+0xf9/0x1b0
[  663.218307]  ? trace_hardirqs_on+0xd/0x10
[  663.218329]  ? ktime_get+0x98/0x130
[  663.218352]  ? __delayacct_blkio_start+0x1a/0x30
[  663.218448]  io_schedule_timeout+0x9f/0x110
[  663.218475]  blk_mq_get_tag+0x158/0x260
[  663.218499]  ? remove_wait_queue+0x70/0x70
[  663.218525]  __blk_mq_alloc_request+0x16/0xe0
[  663.218548]  blk_mq_sched_get_request+0x279/0x370
[  663.218571]  ? blk_mq_sched_bypass_insert+0x70/0x70
[  663.218596]  ? generic_make_request+0xca/0x290
[  663.218619]  blk_sq_make_request+0x111/0xc90
[  663.218642]  ? blk_queue_enter+0x2d/0x280
[  663.218665]  ? generic_make_request+0xca/0x290
[  663.218688]  generic_make_request+0xd7/0x290
[  663.218712]  ? _raw_spin_unlock_irqrestore+0x31/0x50
[  663.218737]  submit_bio+0x5f/0x120
[  663.218761]  submit_bh_wbc+0x14d/0x180
[  663.218783]  __block_write_full_page+0x193/0x3f0
[  663.218805]  ? I_BDEV+0x10/0x10
[  663.218826]  ? I_BDEV+0x10/0x10
[  663.218847]  block_write_full_page+0xd0/0x120
[  663.218870]  blkdev_writepage+0x13/0x20
[  663.218892]  __writepage+0x11/0x40
[  663.218914]  write_cache_pages+0x216/0x640
[  663.218935]  ? wb_position_ratio+0x1f0/0x1f0
[  663.218957]  generic_writepages+0x41/0x60
[  663.218979]  blkdev_writepages+0x2a/0x30
[  663.219000]  do_writepages+0x1c/0x30
[  663.219022]  __writeback_single_inode+0x57/0x720
[  663.219042]  ? _raw_spin_unlock+0x22/0x30
[  663.219064]  writeback_sb_inodes+0x2f4/0x6a0
[  663.219086]  __writeback_inodes_wb+0x8d/0xc0
[  663.219107]  wb_writeback+0x33c/0x530
[  663.219127]  ? mark_held_locks+0x6f/0xa0
[  663.219149]  wb_workfn+0x356/0x630
[  663.219170]  process_one_work+0x208/0x6a0
[  663.219191]  ? process_one_work+0x18d/0x6a0
[  663.219212]  worker_thread+0x49/0x4a0
[  663.219233]  kthread+0x107/0x140
[  663.219253]  ? process_one_work+0x6a0/0x6a0
[  663.219275]  ? kthread_create_on_node+0x40/0x40
[  663.219299]  ret_from_fork+0x2e/0x40
[  663.219323] kworker/10:1    D    0   155      2 0x00000000
[  663.219349] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.219371] Call Trace:
[  663.219393]  __schedule+0x2da/0xb00
[  663.219421]  ? bit_wait+0x50/0x50
[  663.219443]  schedule+0x38/0x90
[  663.219465]  schedule_timeout+0x2fe/0x640
[  663.219487]  ? mark_held_locks+0x6f/0xa0
[  663.219509]  ? ktime_get+0x74/0x130
[  663.219531]  ? bit_wait+0x50/0x50
[  663.219553]  ? trace_hardirqs_on_caller+0xf9/0x1b0
[  663.219576]  ? trace_hardirqs_on+0xd/0x10
[  663.219598]  ? ktime_get+0x98/0x130
[  663.219619]  ? __delayacct_blkio_start+0x1a/0x30
[  663.219643]  ? bit_wait+0x50/0x50
[  663.219665]  io_schedule_timeout+0x9f/0x110
[  663.219688]  bit_wait_io+0x16/0x60
[  663.219711]  __wait_on_bit+0x53/0x80
[  663.219734]  ? bit_wait+0x50/0x50
[  663.219756]  out_of_line_wait_on_bit+0x6e/0x80
[  663.219779]  ? prepare_to_wait_event+0x170/0x170
[  663.219803]  sync_mapping_buffers+0x22f/0x390
[  663.219826]  __generic_file_fsync+0x4d/0x90
[  663.219848]  ext4_sync_file+0x2b4/0x540
[  663.219870]  vfs_fsync_range+0x46/0xa0
[  663.219892]  dio_complete+0x181/0x1b0
[  663.219915]  dio_aio_complete_work+0x17/0x20
[  663.219939]  process_one_work+0x208/0x6a0
[  663.219962]  ? process_one_work+0x18d/0x6a0
[  663.219986]  worker_thread+0x49/0x4a0
[  663.220011]  kthread+0x107/0x140
[  663.220038]  ? process_one_work+0x6a0/0x6a0
[  663.220063]  ? kthread_create_on_node+0x40/0x40
[  663.220090]  ret_from_fork+0x2e/0x40
[  663.220121] kworker/4:2     D    0   284      2 0x00000000
[  663.220148] Workqueue: srp_remove srp_remove_work [ib_srp]
[  663.220171] Call Trace:
[  663.220193]  __schedule+0x2da/0xb00
[  663.220214]  schedule+0x38/0x90
[  663.220236]  blk_mq_freeze_queue_wait+0x51/0xa0
[  663.220258]  ? remove_wait_queue+0x70/0x70
[  663.220281]  blk_mq_freeze_queue+0x15/0x20
[  663.220302]  blk_freeze_queue+0x9/0x10
[  663.220324]  blk_cleanup_queue+0xdd/0x290
[  663.220346]  __scsi_remove_device+0x49/0xd0
[  663.220368]  scsi_forget_host+0x5b/0x60
[  663.220390]  scsi_remove_host+0x6c/0x110
[  663.220412]  srp_remove_work+0x8b/0x220 [ib_srp]
[  663.220434]  process_one_work+0x208/0x6a0
[  663.220454]  ? process_one_work+0x18d/0x6a0
[  663.220475]  worker_thread+0x49/0x4a0
[  663.220496]  kthread+0x107/0x140
[  663.220516]  ? process_one_work+0x6a0/0x6a0
[  663.220537]  ? kthread_create_on_node+0x40/0x40
[  663.220560]  ret_from_fork+0x2e/0x40
[  663.220581] kworker/10:2    D    0   285      2 0x00000000
[  663.220605] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.220626] Call Trace:
[  663.220648]  __schedule+0x2da/0xb00
[  663.220669]  ? bit_wait+0x50/0x50
[  663.220690]  schedule+0x38/0x90
[  663.220711]  schedule_timeout+0x2fe/0x640
[  663.220732]  ? mark_held_locks+0x6f/0xa0
[  663.220753]  ? ktime_get+0x74/0x130
[  663.220774]  ? bit_wait+0x50/0x50
[  663.220794]  ? trace_hardirqs_on_caller+0xf9/0x1b0
[  663.220816]  ? trace_hardirqs_on+0xd/0x10
[  663.220837]  ? ktime_get+0x98/0x130
[  663.220858]  ? __delayacct_blkio_start+0x1a/0x30
[  663.220879]  ? bit_wait+0x50/0x50
[  663.220899]  io_schedule_timeout+0x9f/0x110
[  663.220920]  bit_wait_io+0x16/0x60
[  663.220941]  __wait_on_bit+0x53/0x80
[  663.220963]  ? bit_wait+0x50/0x50
[  663.220993]  out_of_line_wait_on_bit+0x6e/0x80
[  663.221015]  ? prepare_to_wait_event+0x170/0x170
[  663.221037]  sync_mapping_buffers+0x22f/0x390
[  663.221059]  __generic_file_fsync+0x4d/0x90
[  663.221081]  ext4_sync_file+0x2b4/0x540
[  663.221103]  vfs_fsync_range+0x46/0xa0
[  663.221124]  dio_complete+0x181/0x1b0
[  663.221146]  dio_aio_complete_work+0x17/0x20
[  663.221170]  process_one_work+0x208/0x6a0
[  663.221191]  ? process_one_work+0x18d/0x6a0
[  663.221214]  worker_thread+0x49/0x4a0
[  663.221237]  kthread+0x107/0x140
[  663.221259]  ? process_one_work+0x6a0/0x6a0
[  663.221281]  ? kthread_create_on_node+0x40/0x40
[  663.221304]  ret_from_fork+0x2e/0x40
[  663.221330] kworker/10:3    D    0   405      2 0x00000000
[  663.221354] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.221379] Call Trace:
[  663.221401]  __schedule+0x2da/0xb00
[  663.221424]  ? bit_wait+0x50/0x50
[  663.221446]  schedule+0x38/0x90
[  663.221470]  schedule_timeout+0x2fe/0x640
[  663.221495]  ? ktime_get+0x74/0x130
[  663.221520]  ? bit_wait+0x50/0x50
[  663.221546]  ? trace_hardirqs_on_caller+0xf9/0x1b0
[  663.221574]  ? trace_hardirqs_on+0xd/0x10
[  663.221600]  ? ktime_get+0x98/0x130
[  663.221622]  ? __delayacct_blkio_start+0x1a/0x30
[  663.221646]  ? bit_wait+0x50/0x50
[  663.221668]  io_schedule_timeout+0x9f/0x110
[  663.221690]  bit_wait_io+0x16/0x60
[  663.221712]  __wait_on_bit+0x53/0x80
[  663.221734]  ? bit_wait+0x50/0x50
[  663.221756]  out_of_line_wait_on_bit+0x6e/0x80
[  663.221778]  ? prepare_to_wait_event+0x170/0x170
[  663.221801]  __sync_dirty_buffer+0xdc/0x130
[  663.221822]  sync_dirty_buffer+0xe/0x10
[  663.221844]  ext4_write_inode+0x121/0x140
[  663.221866]  __writeback_single_inode+0x3ae/0x720
[  663.221887]  ? _raw_spin_unlock+0x22/0x30
[  663.221909]  writeback_single_inode+0xd0/0x190
[  663.221932]  sync_inode_metadata+0x2f/0x40
[  663.221953]  __generic_file_fsync+0x74/0x90
[  663.221975]  ext4_sync_file+0x2b4/0x540
[  663.221995]  vfs_fsync_range+0x46/0xa0
[  663.222016]  dio_complete+0x181/0x1b0
[  663.222036]  dio_aio_complete_work+0x17/0x20
[  663.222057]  process_one_work+0x208/0x6a0
[  663.222078]  ? process_one_work+0x18d/0x6a0
[  663.222099]  worker_thread+0x49/0x4a0
[  663.222121]  kthread+0x107/0x140
[  663.222141]  ? process_one_work+0x6a0/0x6a0
[  663.222161]  ? kthread_create_on_node+0x40/0x40
[  663.222183]  ret_from_fork+0x2e/0x40
[  663.222277] fio             D    0  9265   8846 0x00000000
[  663.222302] Call Trace:
[  663.222325]  __schedule+0x2da/0xb00
[  663.222347]  ? _raw_spin_unlock_irq+0x27/0x40
[  663.222372]  schedule+0x38/0x90
[  663.222396]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.222423]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.222450]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.222479]  ? ___slab_alloc+0x178/0x520
[  663.222503]  call_rwsem_down_write_failed+0x17/0x30
[  663.222528]  down_write+0x5a/0x70
[  663.222553]  ? ext4_file_write_iter+0x45/0x360
[  663.222576]  ext4_file_write_iter+0x45/0x360
[  663.222600]  ? __sb_start_write+0xde/0x200
[  663.222621]  ? aio_write+0x14e/0x160
[  663.222643]  aio_write+0xd1/0x160
[  663.222795]  ? __might_fault+0x3e/0x90
[  663.222823]  do_io_submit+0x37d/0x900
[  663.222848]  ? do_io_submit+0x1ac/0x900
[  663.222871]  SyS_io_submit+0xb/0x10
[  663.222893]  entry_SYSCALL_64_fastpath+0x18/0xad
[  663.222915] RIP: 0033:0x7fc269d02787
[  663.222937] RSP: 002b:00007fc249e6f948 EFLAGS: 00000202 ORIG_RAX: 00000000000000d1
[  663.222961] RAX: ffffffffffffffda RBX: 0000000000000046 RCX: 00007fc269d02787
[  663.222984] RDX: 00007fc24004f190 RSI: 0000000000000001 RDI: 00007fc27bbfc000
[  663.223006] RBP: 0000000000000038 R08: 0000000000000001 R09: 00007fc24004cfe0
[  663.223029] R10: 00007fc240039000 R11: 0000000000000202 R12: 00007fc24f052000
[  663.223051] R13: 00007fc24004f360 R14: 0000000000000000 R15: 0000000000000001
[  663.223074] fio             D    0  9266   8846 0x00000000
[  663.223286] Call Trace:
[  663.223311]  __schedule+0x2da/0xb00
[  663.223333]  ? _raw_spin_unlock_irq+0x27/0x40
[  663.223356]  schedule+0x38/0x90
[  663.223379]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.223402]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.223424]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.223447]  ? ___slab_alloc+0x178/0x520
[  663.223470]  call_rwsem_down_write_failed+0x17/0x30
[  663.223493]  down_write+0x5a/0x70
[  663.223514]  ? ext4_file_write_iter+0x45/0x360
[  663.223536]  ext4_file_write_iter+0x45/0x360
[  663.223558]  ? __sb_start_write+0xde/0x200
[  663.223579]  ? aio_write+0x14e/0x160
[  663.223599]  aio_write+0xd1/0x160
[  663.223621]  ? __might_fault+0x3e/0x90
[  663.223642]  do_io_submit+0x37d/0x900
[  663.223663]  ? do_io_submit+0x1ac/0x900
[  663.223684]  SyS_io_submit+0xb/0x10
[  663.223705]  entry_SYSCALL_64_fastpath+0x18/0xad
[  663.223725] RIP: 0033:0x7fc269d02787
[  663.223747] RSP: 002b:00007fc24a670948 EFLAGS: 00000202 ORIG_RAX: 00000000000000d1
[  663.223769] RAX: ffffffffffffffda RBX: 0000000000000046 RCX: 00007fc269d02787
[  663.223790] RDX: 00007fc23804f188 RSI: 0000000000000001 RDI: 00007fc27bc08000
[  663.223811] RBP: 0000000000000065 R08: 0000000000000001 R09: 00007fc23804d2a0
[  663.223833] R10: 00007fc238050000 R11: 0000000000000202 R12: 0000000000000020
[  663.223855] R13: 00007fc23804ffe0 R14: 0000000000001000 R15: 0000000000000080
[  663.223877] fio             D    0  9267   8846 0x00000000
[  663.224191] Call Trace:
[  663.224215]  __schedule+0x2da/0xb00
[  663.224236]  schedule+0x38/0x90
[  663.224258]  schedule_timeout+0x2fe/0x640
[  663.224280]  ? mark_held_locks+0x6f/0xa0
[  663.224303]  ? ktime_get+0x74/0x130
[  663.224325]  ? trace_hardirqs_on_caller+0xf9/0x1b0
[  663.224346]  ? trace_hardirqs_on+0xd/0x10
[  663.224368]  ? ktime_get+0x98/0x130
[  663.224389]  ? __delayacct_blkio_start+0x1a/0x30
[  663.224411]  io_schedule_timeout+0x9f/0x110
[  663.224433]  blk_mq_get_tag+0x158/0x260
[  663.224454]  ? remove_wait_queue+0x70/0x70
[  663.224476]  __blk_mq_alloc_request+0x16/0xe0
[  663.224521]  blk_mq_sched_get_request+0x279/0x370
[  663.224544]  ? blk_mq_sched_bypass_insert+0x70/0x70
[  663.224567]  ? generic_make_request+0xca/0x290
[  663.224592]  blk_sq_make_request+0x111/0xc90
[  663.224620]  ? blk_queue_enter+0x2d/0x280
[  663.224643]  ? generic_make_request+0xca/0x290
[  663.224666]  generic_make_request+0xd7/0x290
[  663.224690]  submit_bio+0x5f/0x120
[  663.224711]  ? trace_hardirqs_on+0xd/0x10
[  663.224733]  do_blockdev_direct_IO+0x280f/0x31f0
[  663.224758]  ? debug_lockdep_rcu_enabled+0x1d/0x20
[  663.224781]  ? _raw_spin_unlock+0x22/0x30
[  663.224806]  ? ext4_get_block_trans+0xe0/0xe0
[  663.224829]  __blockdev_direct_IO+0x35/0x40
[  663.224852]  ext4_direct_IO+0x19c/0x7b0
[  663.224874]  generic_file_direct_write+0xa6/0x150
[  663.224897]  __generic_file_write_iter+0xbb/0x1c0
[  663.224920]  ext4_file_write_iter+0x77/0x360
[  663.224946]  ? __sb_start_write+0xde/0x200
[  663.224967]  ? aio_write+0x14e/0x160
[  663.224988]  aio_write+0xd1/0x160
[  663.225009]  ? __might_fault+0x3e/0x90
[  663.225029]  do_io_submit+0x37d/0x900
[  663.225051]  ? do_io_submit+0x1ac/0x900
[  663.225076]  SyS_io_submit+0xb/0x10
[  663.225102]  entry_SYSCALL_64_fastpath+0x18/0xad
[  663.225128] RIP: 0033:0x7fc269d02787
[  663.225151] RSP: 002b:00007fc24ae71948 EFLAGS: 00000202 ORIG_RAX: 00000000000000d1
[  663.225175] RAX: ffffffffffffffda RBX: 0000000000000046 RCX: 00007fc269d02787
[  663.225205] RDX: 00007fc22804f2c8 RSI: 0000000000000001 RDI: 00007fc27bc0e000
[  663.225232] RBP: 00000000000000b0 R08: 0000000000000001 R09: 00007fc228045fa0
[  663.225259] R10: 00007fc228012000 R11: 0000000000000202 R12: 00007fc24f06e110
[  663.225283] R13: 00007fc22804f360 R14: 0000000000000000 R15: 0000000000000001
[  663.225309] fio             D    0  9268   8846 0x00000000
[  663.225338] Call Trace:
[  663.225366]  __schedule+0x2da/0xb00
[  663.225392]  schedule+0x38/0x90
[  663.225420]  schedule_timeout+0x2fe/0x640
[  663.225446]  ? mark_held_locks+0x6f/0xa0
[  663.225483]  ? ktime_get+0x74/0x130
[  663.225484]  ? trace_hardirqs_on_caller+0xf9/0x1b0
[  663.225486]  ? trace_hardirqs_on+0xd/0x10
[  663.225487]  ? ktime_get+0x98/0x130
[  663.225489]  ? __delayacct_blkio_start+0x1a/0x30
[  663.225491]  io_schedule_timeout+0x9f/0x110
[  663.225493]  blk_mq_get_tag+0x158/0x260
[  663.225494]  ? remove_wait_queue+0x70/0x70
[  663.225496]  __blk_mq_alloc_request+0x16/0xe0
[  663.225499]  blk_mq_sched_get_request+0x279/0x370
[  663.225500]  ? blk_mq_sched_bypass_insert+0x70/0x70
[  663.225503]  ? generic_make_request+0xca/0x290
[  663.225505]  blk_sq_make_request+0x111/0xc90
[  663.225507]  ? blk_queue_enter+0x2d/0x280
[  663.225509]  ? generic_make_request+0xca/0x290
[  663.225511]  generic_make_request+0xd7/0x290
[  663.225513]  submit_bio+0x5f/0x120
[  663.225514]  ? trace_hardirqs_on+0xd/0x10
[  663.225516]  do_blockdev_direct_IO+0x280f/0x31f0
[  663.225518]  ? debug_lockdep_rcu_enabled+0x1d/0x20
[  663.225520]  ? _raw_spin_unlock+0x22/0x30
[  663.225522]  ? ext4_get_block_trans+0xe0/0xe0
[  663.225524]  __blockdev_direct_IO+0x35/0x40
[  663.225526]  ext4_direct_IO+0x19c/0x7b0
[  663.225528]  generic_file_direct_write+0xa6/0x150
[  663.225530]  __generic_file_write_iter+0xbb/0x1c0
[  663.225531]  ext4_file_write_iter+0x77/0x360
[  663.225533]  ? __sb_start_write+0xde/0x200
[  663.225534]  ? aio_write+0x14e/0x160
[  663.225536]  aio_write+0xd1/0x160
[  663.225537]  ? __might_fault+0x3e/0x90
[  663.225539]  do_io_submit+0x37d/0x900
[  663.225540]  ? do_io_submit+0x1ac/0x900
[  663.225542]  SyS_io_submit+0xb/0x10
[  663.225543]  entry_SYSCALL_64_fastpath+0x18/0xad
[  663.225544] RIP: 0033:0x7fc269d02787
[  663.225545] RSP: 002b:00007fc24f04e948 EFLAGS: 00000212 ORIG_RAX: 00000000000000d1
[  663.225549] RAX: ffffffffffffffda RBX: 0000000000000046 RCX: 00007fc269d02787
[  663.225550] RDX: 00007fc22004f1d0 RSI: 0000000000000001 RDI: 00007fc27bc00000
[  663.225551] RBP: 0000000000000065 R08: 0000000000000001 R09: 00007fc22004b8e0
[  663.225552] R10: 00007fc220031000 R11: 0000000000000212 R12: 0000000000000020
[  663.225552] R13: 00007fc22004ffe0 R14: 0000000000001000 R15: 0000000000000080
[  663.225554] fio             D    0  9269   8846 0x00000000
[  663.225557] Call Trace:
[  663.225559]  __schedule+0x2da/0xb00
[  663.225560]  ? _raw_spin_unlock_irq+0x27/0x40
[  663.225562]  schedule+0x38/0x90
[  663.225564]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.225566]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.225568]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.225570]  ? ___slab_alloc+0x178/0x520
[  663.225573]  call_rwsem_down_write_failed+0x17/0x30
[  663.225575]  down_write+0x5a/0x70
[  663.225576]  ? ext4_file_write_iter+0x45/0x360
[  663.225578]  ext4_file_write_iter+0x45/0x360
[  663.225579]  ? __sb_start_write+0xde/0x200
[  663.225581]  ? aio_write+0x14e/0x160
[  663.225582]  aio_write+0xd1/0x160
[  663.225584]  ? __might_fault+0x3e/0x90
[  663.225585]  do_io_submit+0x37d/0x900
[  663.225586]  ? do_io_submit+0x1ac/0x900
[  663.225588]  SyS_io_submit+0xb/0x10
[  663.225589]  entry_SYSCALL_64_fastpath+0x18/0xad
[  663.225590] RIP: 0033:0x7fc269d02787
[  663.225591] RSP: 002b:00007fc24e84d948 EFLAGS: 00000202 ORIG_RAX: 00000000000000d1
[  663.225593] RAX: ffffffffffffffda RBX: 0000000000000046 RCX: 00007fc269d02787
[  663.225594] RDX: 00007fc23004f1e8 RSI: 0000000000000001 RDI: 00007fc27bc06000
[  663.225595] RBP: 0000000000000065 R08: 0000000000000001 R09: 00007fc23004b020
[  663.225595] R10: 00007fc23002e000 R11: 0000000000000202 R12: 0000000000000020
[  663.225596] R13: 00007fc23004ffe0 R14: 0000000000001000 R15: 0000000000000080
[  663.225598] fio             D    0  9270   8846 0x00000000
[  663.225600] Call Trace:
[  663.225602]  __schedule+0x2da/0xb00
[  663.225603]  ? _raw_spin_unlock_irq+0x27/0x40
[  663.225605]  schedule+0x38/0x90
[  663.225607]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.225609]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.225611]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.225614]  ? ___slab_alloc+0x178/0x520
[  663.225616]  call_rwsem_down_write_failed+0x17/0x30
[  663.225618]  down_write+0x5a/0x70
[  663.225620]  ? ext4_file_write_iter+0x45/0x360
[  663.225622]  ext4_file_write_iter+0x45/0x360
[  663.225623]  ? __sb_start_write+0xde/0x200
[  663.225624]  ? aio_write+0x14e/0x160
[  663.225626]  aio_write+0xd1/0x160
[  663.225627]  ? __might_fault+0x3e/0x90
[  663.225629]  do_io_submit+0x37d/0x900
[  663.225630]  ? do_io_submit+0x1ac/0x900
[  663.225632]  SyS_io_submit+0xb/0x10
[  663.225633]  entry_SYSCALL_64_fastpath+0x18/0xad
[  663.225634] RIP: 0033:0x7fc269d02787
[  663.225635] RSP: 002b:00007fc24e04c948 EFLAGS: 00000202 ORIG_RAX: 00000000000000d1
[  663.225636] RAX: ffffffffffffffda RBX: 0000000000000046 RCX: 00007fc269d02787
[  663.225637] RDX: 00007fc21804f298 RSI: 0000000000000001 RDI: 00007fc27bc10000
[  663.225638] RBP: 0000000011bb7409 R08: 0000000000000001 R09: 00007fc2180470e0
[  663.225639] R10: 00007fc218018000 R11: 0000000000000202 R12: 0000000068baf4ba
[  663.225640] R13: 00000000165c8e46 R14: 000000002adf21b1 R15: 00000000169850e2
[  663.225642] fio             D    0  9271   8846 0x00000000
[  663.225644] Call Trace:
[  663.225646]  __schedule+0x2da/0xb00
[  663.225647]  ? _raw_spin_unlock_irq+0x27/0x40
[  663.225649]  schedule+0x38/0x90
[  663.225651]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.225653]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.225656]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.225658]  ? ___slab_alloc+0x178/0x520
[  663.225660]  call_rwsem_down_write_failed+0x17/0x30
[  663.225662]  down_write+0x5a/0x70
[  663.225664]  ? ext4_file_write_iter+0x45/0x360
[  663.225665]  ext4_file_write_iter+0x45/0x360
[  663.225667]  ? __sb_start_write+0xde/0x200
[  663.225668]  ? aio_write+0x14e/0x160
[  663.225669]  aio_write+0xd1/0x160
[  663.225671]  ? __might_fault+0x3e/0x90
[  663.225673]  do_io_submit+0x37d/0x900
[  663.225674]  ? do_io_submit+0x1ac/0x900
[  663.225676]  SyS_io_submit+0xb/0x10
[  663.225677]  entry_SYSCALL_64_fastpath+0x18/0xad
[  663.225678] RIP: 0033:0x7fc269d02787
[  663.225679] RSP: 002b:00007fc24d84b948 EFLAGS: 00000206 ORIG_RAX: 00000000000000d1
[  663.225681] RAX: ffffffffffffffda RBX: 0000000000000046 RCX: 00007fc269d02787
[  663.225682] RDX: 00007fc21004f210 RSI: 0000000000000001 RDI: 00007fc27bc0c000
[  663.225682] RBP: 0000000000000060 R08: 0000000000000001 R09: 00007fc21004a1e0
[  663.225683] R10: 00007fc210029000 R11: 0000000000000206 R12: 00007fc24f0a6330
[  663.225684] R13: 00007fc21004f360 R14: 0000000000000000 R15: 0000000000000001
[  663.225686] fio             D    0  9272   8846 0x00000000
[  663.225689] Call Trace:
[  663.225691]  __schedule+0x2da/0xb00
[  663.225693]  schedule+0x38/0x90
[  663.225695]  schedule_timeout+0x2fe/0x640
[  663.225696]  ? mark_held_locks+0x6f/0xa0
[  663.225699]  ? ktime_get+0x74/0x130
[  663.225700]  ? trace_hardirqs_on_caller+0xf9/0x1b0
[  663.225702]  ? trace_hardirqs_on+0xd/0x10
[  663.225703]  ? ktime_get+0x98/0x130
[  663.225705]  ? __delayacct_blkio_start+0x1a/0x30
[  663.225707]  io_schedule_timeout+0x9f/0x110
[  663.225708]  blk_mq_get_tag+0x158/0x260
[  663.225710]  ? remove_wait_queue+0x70/0x70
[  663.225712]  __blk_mq_alloc_request+0x16/0xe0
[  663.225714]  blk_mq_sched_get_request+0x279/0x370
[  663.225715]  ? blk_mq_sched_bypass_insert+0x70/0x70
[  663.225718]  ? generic_make_request+0xca/0x290
[  663.225720]  blk_sq_make_request+0x111/0xc90
[  663.225722]  ? blk_queue_enter+0x2d/0x280
[  663.225724]  ? generic_make_request+0xca/0x290
[  663.225727]  generic_make_request+0xd7/0x290
[  663.225730]  submit_bio+0x5f/0x120
[  663.225731]  ? trace_hardirqs_on+0xd/0x10
[  663.225733]  do_blockdev_direct_IO+0x280f/0x31f0
[  663.225735]  ? debug_lockdep_rcu_enabled+0x1d/0x20
[  663.225737]  ? _raw_spin_unlock+0x22/0x30
[  663.225739]  ? ext4_get_block_trans+0xe0/0xe0
[  663.225741]  __blockdev_direct_IO+0x35/0x40
[  663.225743]  ext4_direct_IO+0x19c/0x7b0
[  663.225745]  generic_file_direct_write+0xa6/0x150
[  663.225747]  __generic_file_write_iter+0xbb/0x1c0
[  663.225749]  ext4_file_write_iter+0x77/0x360
[  663.225750]  ? __sb_start_write+0xde/0x200
[  663.225752]  ? aio_write+0x14e/0x160
[  663.225753]  aio_write+0xd1/0x160
[  663.225755]  ? __might_fault+0x3e/0x90
[  663.225756]  do_io_submit+0x37d/0x900
[  663.225758]  ? do_io_submit+0x1ac/0x900
[  663.225759]  SyS_io_submit+0xb/0x10
[  663.225761]  entry_SYSCALL_64_fastpath+0x18/0xad
[  663.225762] RIP: 0033:0x7fc269d02787
[  663.225763] RSP: 002b:00007fc24d04a948 EFLAGS: 00000206 ORIG_RAX: 00000000000000d1
[  663.225764] RAX: ffffffffffffffda RBX: 0000000000000046 RCX: 00007fc269d02787
[  663.225765] RDX: 00007fc20804f260 RSI: 0000000000000001 RDI: 00007fc27bbfa000
[  663.225766] RBP: 00000000000000b8 R08: 0000000000000001 R09: 00007fc208048520
[  663.225767] R10: 00007fc20801f000 R11: 0000000000000206 R12: 00007fc24f0b43b8
[  663.225768] R13: 00007fc20804f360 R14: 0000000000000000 R15: 0000000000000001
[  663.225770] fio             D    0  9273   8846 0x00000000
[  663.225772] Call Trace:
[  663.225774]  __schedule+0x2da/0xb00
[  663.225776]  schedule+0x38/0x90
[  663.225778]  schedule_timeout+0x2fe/0x640
[  663.225780]  ? mark_held_locks+0x6f/0xa0
[  663.225782]  ? ktime_get+0x74/0x130
[  663.225783]  ? trace_hardirqs_on_caller+0xf9/0x1b0
[  663.225784]  ? trace_hardirqs_on+0xd/0x10
[  663.225786]  ? ktime_get+0x98/0x130
[  663.225788]  ? __delayacct_blkio_start+0x1a/0x30
[  663.225790]  io_schedule_timeout+0x9f/0x110
[  663.225791]  blk_mq_get_tag+0x158/0x260
[  663.225793]  ? remove_wait_queue+0x70/0x70
[  663.225795]  __blk_mq_alloc_request+0x16/0xe0
[  663.225797]  blk_mq_sched_get_request+0x279/0x370
[  663.225798]  ? blk_mq_sched_bypass_insert+0x70/0x70
[  663.225800]  ? generic_make_request+0xca/0x290
[  663.225803]  blk_sq_make_request+0x111/0xc90
[  663.225805]  ? blk_queue_enter+0x2d/0x280
[  663.225807]  ? generic_make_request+0xca/0x290
[  663.225809]  generic_make_request+0xd7/0x290
[  663.225811]  submit_bio+0x5f/0x120
[  663.225813]  ? trace_hardirqs_on+0xd/0x10
[  663.225814]  do_blockdev_direct_IO+0x280f/0x31f0
[  663.225816]  ? debug_lockdep_rcu_enabled+0x1d/0x20
[  663.225818]  ? _raw_spin_unlock+0x22/0x30
[  663.225820]  ? ext4_get_block_trans+0xe0/0xe0
[  663.225822]  __blockdev_direct_IO+0x35/0x40
[  663.225824]  ext4_direct_IO+0x19c/0x7b0
[  663.225826]  generic_file_direct_write+0xa6/0x150
[  663.225828]  __generic_file_write_iter+0xbb/0x1c0
[  663.225829]  ext4_file_write_iter+0x77/0x360
[  663.225831]  ? __sb_start_write+0xde/0x200
[  663.225832]  ? aio_write+0x14e/0x160
[  663.225833]  aio_write+0xd1/0x160
[  663.225835]  ? __might_fault+0x3e/0x90
[  663.225837]  do_io_submit+0x37d/0x900
[  663.225838]  ? do_io_submit+0x1ac/0x900
[  663.225839]  SyS_io_submit+0xb/0x10
[  663.225841]  entry_SYSCALL_64_fastpath+0x18/0xad
[  663.225842] RIP: 0033:0x7fc269d02787
[  663.225842] RSP: 002b:00007fc24c849948 EFLAGS: 00000202 ORIG_RAX: 00000000000000d1
[  663.225844] RAX: ffffffffffffffda RBX: 0000000000000046 RCX: 00007fc269d02787
[  663.225845] RDX: 00007fc1d004f200 RSI: 0000000000000001 RDI: 00007fc27bbf2000
[  663.225846] RBP: 0000000000000050 R08: 0000000000000001 R09: 00007fc1d004a7a0
[  663.225847] R10: 00007fc1d002b000 R11: 0000000000000202 R12: 00007fc24f0c2440
[  663.225848] R13: 00007fc1d004f360 R14: 0000000000000000 R15: 0000000000000001
[  663.225849] fio             D    0  9274   8846 0x00000000
[  663.225851] Call Trace:
[  663.225853]  __schedule+0x2da/0xb00
[  663.225855]  schedule+0x38/0x90
[  663.225857]  schedule_timeout+0x2fe/0x640
[  663.225858]  ? mark_held_locks+0x6f/0xa0
[  663.225860]  ? ktime_get+0x74/0x130
[  663.225861]  ? trace_hardirqs_on_caller+0xf9/0x1b0
[  663.225863]  ? trace_hardirqs_on+0xd/0x10
[  663.225864]  ? ktime_get+0x98/0x130
[  663.225866]  ? __delayacct_blkio_start+0x1a/0x30
[  663.225868]  io_schedule_timeout+0x9f/0x110
[  663.225869]  blk_mq_get_tag+0x158/0x260
[  663.225871]  ? remove_wait_queue+0x70/0x70
[  663.225873]  __blk_mq_alloc_request+0x16/0xe0
[  663.225875]  blk_mq_sched_get_request+0x279/0x370
[  663.225876]  ? blk_mq_sched_bypass_insert+0x70/0x70
[  663.225878]  ? generic_make_request+0xca/0x290
[  663.225880]  blk_sq_make_request+0x111/0xc90
[  663.225882]  ? blk_queue_enter+0x2d/0x280
[  663.225885]  ? generic_make_request+0xca/0x290
[  663.225887]  generic_make_request+0xd7/0x290
[  663.225889]  submit_bio+0x5f/0x120
[  663.225890]  ? trace_hardirqs_on+0xd/0x10
[  663.225892]  do_blockdev_direct_IO+0x280f/0x31f0
[  663.225894]  ? debug_lockdep_rcu_enabled+0x1d/0x20
[  663.225895]  ? _raw_spin_unlock+0x22/0x30
[  663.225898]  ? ext4_get_block_trans+0xe0/0xe0
[  663.225901]  __blockdev_direct_IO+0x35/0x40
[  663.225903]  ext4_direct_IO+0x19c/0x7b0
[  663.225905]  generic_file_direct_write+0xa6/0x150
[  663.225907]  __generic_file_write_iter+0xbb/0x1c0
[  663.225908]  ext4_file_write_iter+0x77/0x360
[  663.225910]  ? __sb_start_write+0xde/0x200
[  663.225911]  ? aio_write+0x14e/0x160
[  663.225913]  aio_write+0xd1/0x160
[  663.225915]  ? __might_fault+0x3e/0x90
[  663.225916]  do_io_submit+0x37d/0x900
[  663.225918]  ? do_io_submit+0x1ac/0x900
[  663.225920]  SyS_io_submit+0xb/0x10
[  663.225921]  entry_SYSCALL_64_fastpath+0x18/0xad
[  663.225922] RIP: 0033:0x7fc269d02787
[  663.225923] RSP: 002b:00007fc24c048948 EFLAGS: 00000206 ORIG_RAX: 00000000000000d1
[  663.225925] RAX: ffffffffffffffda RBX: 0000000000000046 RCX: 00007fc269d02787
[  663.225926] RDX: 00007fc20004f1f0 RSI: 0000000000000001 RDI: 00007fc27bc02000
[  663.225927] RBP: 000000001d36c979 R08: 0000000000000001 R09: 00007fc20004ad60
[  663.225927] R10: 00007fc20002d000 R11: 0000000000000206 R12: 00000000d322e5e6
[  663.225928] R13: 000000002b3b7b00 R14: 00000000573a44d8 R15: 0000000001e412a3
[  663.225930] fio             D    0  9275   8846 0x00000000
[  663.225932] Call Trace:
[  663.225935]  __schedule+0x2da/0xb00
[  663.225936]  schedule+0x38/0x90
[  663.225939]  schedule_timeout+0x2fe/0x640
[  663.225940]  ? mark_held_locks+0x6f/0xa0
[  663.225942]  ? ktime_get+0x74/0x130
[  663.225943]  ? trace_hardirqs_on_caller+0xf9/0x1b0
[  663.225945]  ? trace_hardirqs_on+0xd/0x10
[  663.225946]  ? ktime_get+0x98/0x130
[  663.225948]  ? __delayacct_blkio_start+0x1a/0x30
[  663.225949]  io_schedule_timeout+0x9f/0x110
[  663.225951]  blk_mq_get_tag+0x158/0x260
[  663.225952]  ? remove_wait_queue+0x70/0x70
[  663.225954]  __blk_mq_alloc_request+0x16/0xe0
[  663.225956]  blk_mq_sched_get_request+0x279/0x370
[  663.225958]  ? blk_mq_sched_bypass_insert+0x70/0x70
[  663.225960]  ? generic_make_request+0xca/0x290
[  663.225963]  blk_sq_make_request+0x111/0xc90
[  663.225965]  ? blk_queue_enter+0x2d/0x280
[  663.225967]  ? generic_make_request+0xca/0x290
[  663.225969]  generic_make_request+0xd7/0x290
[  663.225971]  submit_bio+0x5f/0x120
[  663.225973]  ? trace_hardirqs_on+0xd/0x10
[  663.225974]  do_blockdev_direct_IO+0x280f/0x31f0
[  663.225976]  ? debug_lockdep_rcu_enabled+0x1d/0x20
[  663.225977]  ? _raw_spin_unlock+0x22/0x30
[  663.225980]  ? ext4_get_block_trans+0xe0/0xe0
[  663.225981]  __blockdev_direct_IO+0x35/0x40
[  663.225983]  ext4_direct_IO+0x19c/0x7b0
[  663.225985]  generic_file_direct_write+0xa6/0x150
[  663.225987]  __generic_file_write_iter+0xbb/0x1c0
[  663.225988]  ext4_file_write_iter+0x77/0x360
[  663.225990]  ? __sb_start_write+0xde/0x200
[  663.225991]  ? aio_write+0x14e/0x160
[  663.225992]  aio_write+0xd1/0x160
[  663.225994]  ? __might_fault+0x3e/0x90
[  663.225995]  do_io_submit+0x37d/0x900
[  663.225997]  ? do_io_submit+0x1ac/0x900
[  663.225998]  SyS_io_submit+0xb/0x10
[  663.226000]  entry_SYSCALL_64_fastpath+0x18/0xad
[  663.226000] RIP: 0033:0x7fc269d02787
[  663.226001] RSP: 002b:00007fc24b847948 EFLAGS: 00000202 ORIG_RAX: 00000000000000d1
[  663.226003] RAX: ffffffffffffffda RBX: 0000000000000046 RCX: 00007fc269d02787
[  663.226004] RDX: 00007fc1f804f1c0 RSI: 0000000000000001 RDI: 00007fc27bbf8000
[  663.226005] RBP: 0000000000000065 R08: 0000000000000001 R09: 00007fc1f804bea0
[  663.226006] R10: 00007fc1f8033000 R11: 0000000000000202 R12: 0000000000000020
[  663.226007] R13: 00007fc1f804ffe0 R14: 0000000000001000 R15: 0000000000000080
[  663.226009] fio             D    0  9276   8846 0x00000000
[  663.226011] Call Trace:
[  663.226013]  __schedule+0x2da/0xb00
[  663.226014]  schedule+0x38/0x90
[  663.226016]  schedule_timeout+0x2fe/0x640
[  663.226018]  ? mark_held_locks+0x6f/0xa0
[  663.226019]  ? ktime_get+0x74/0x130
[  663.226021]  ? trace_hardirqs_on_caller+0xf9/0x1b0
[  663.226022]  ? trace_hardirqs_on+0xd/0x10
[  663.226024]  ? ktime_get+0x98/0x130
[  663.226025]  ? __delayacct_blkio_start+0x1a/0x30
[  663.226027]  io_schedule_timeout+0x9f/0x110
[  663.226028]  blk_mq_get_tag+0x158/0x260
[  663.226030]  ? remove_wait_queue+0x70/0x70
[  663.226032]  __blk_mq_alloc_request+0x16/0xe0
[  663.226033]  blk_mq_sched_get_request+0x279/0x370
[  663.226035]  ? blk_mq_sched_bypass_insert+0x70/0x70
[  663.226038]  ? generic_make_request+0xca/0x290
[  663.226040]  blk_sq_make_request+0x111/0xc90
[  663.226042]  ? blk_queue_enter+0x2d/0x280
[  663.226044]  ? generic_make_request+0xca/0x290
[  663.226046]  generic_make_request+0xd7/0x290
[  663.226048]  submit_bio+0x5f/0x120
[  663.226049]  ? trace_hardirqs_on+0xd/0x10
[  663.226051]  do_blockdev_direct_IO+0x280f/0x31f0
[  663.226053]  ? debug_lockdep_rcu_enabled+0x1d/0x20
[  663.226054]  ? _raw_spin_unlock+0x22/0x30
[  663.226057]  ? ext4_get_block_trans+0xe0/0xe0
[  663.226058]  __blockdev_direct_IO+0x35/0x40
[  663.226060]  ext4_direct_IO+0x19c/0x7b0
[  663.226062]  generic_file_direct_write+0xa6/0x150
[  663.226064]  __generic_file_write_iter+0xbb/0x1c0
[  663.226066]  ext4_file_write_iter+0x77/0x360
[  663.226068]  ? __sb_start_write+0xde/0x200
[  663.226070]  ? aio_write+0x14e/0x160
[  663.226072]  aio_write+0xd1/0x160
[  663.226074]  ? __might_fault+0x3e/0x90
[  663.226076]  do_io_submit+0x37d/0x900
[  663.226077]  ? do_io_submit+0x1ac/0x900
[  663.226079]  SyS_io_submit+0xb/0x10
[  663.226080]  entry_SYSCALL_64_fastpath+0x18/0xad
[  663.226081] RIP: 0033:0x7fc269d02787
[  663.226082] RSP: 002b:00007fc248e6d948 EFLAGS: 00000202 ORIG_RAX: 00000000000000d1
[  663.226084] RAX: ffffffffffffffda RBX: 0000000000000046 RCX: 00007fc269d02787
[  663.226085] RDX: 00007fc1f004f158 RSI: 0000000000000001 RDI: 00007fc27bbfe000
[  663.226086] RBP: 0000000000000815 R08: 0000000000000001 R09: 00007fc1f004e3e0
[  663.226087] R10: 00007fc1f0040000 R11: 0000000000000202 R12: 00000000000006d0
[  663.226088] R13: 00007fc1f004e930 R14: 0000000000001000 R15: 0000000000000830
[  663.226089] fio             D    0  9277   8846 0x00000000
[  663.226091] Call Trace:
[  663.226094]  __schedule+0x2da/0xb00
[  663.226095]  schedule+0x38/0x90
[  663.226097]  schedule_timeout+0x2fe/0x640
[  663.226099]  ? mark_held_locks+0x6f/0xa0
[  663.226101]  ? ktime_get+0x74/0x130
[  663.226102]  ? trace_hardirqs_on_caller+0xf9/0x1b0
[  663.226103]  ? trace_hardirqs_on+0xd/0x10
[  663.226105]  ? ktime_get+0x98/0x130
[  663.226107]  ? __delayacct_blkio_start+0x1a/0x30
[  663.226109]  io_schedule_timeout+0x9f/0x110
[  663.226110]  blk_mq_get_tag+0x158/0x260
[  663.226112]  ? remove_wait_queue+0x70/0x70
[  663.226114]  __blk_mq_alloc_request+0x16/0xe0
[  663.226115]  blk_mq_sched_get_request+0x279/0x370
[  663.226117]  ? blk_mq_sched_bypass_insert+0x70/0x70
[  663.226119]  ? generic_make_request+0xca/0x290
[  663.226121]  blk_sq_make_request+0x111/0xc90
[  663.226123]  ? blk_queue_enter+0x2d/0x280
[  663.226125]  ? generic_make_request+0xca/0x290
[  663.226127]  generic_make_request+0xd7/0x290
[  663.226130]  submit_bio+0x5f/0x120
[  663.226131]  ? trace_hardirqs_on+0xd/0x10
[  663.226133]  do_blockdev_direct_IO+0x280f/0x31f0
[  663.226135]  ? debug_lockdep_rcu_enabled+0x1d/0x20
[  663.226136]  ? _raw_spin_unlock+0x22/0x30
[  663.226138]  ? ext4_get_block_trans+0xe0/0xe0
[  663.226140]  __blockdev_direct_IO+0x35/0x40
[  663.226142]  ext4_direct_IO+0x19c/0x7b0
[  663.226144]  generic_file_direct_write+0xa6/0x150
[  663.226146]  __generic_file_write_iter+0xbb/0x1c0
[  663.226148]  ext4_file_write_iter+0x77/0x360
[  663.226149]  ? __sb_start_write+0xde/0x200
[  663.226150]  ? aio_write+0x14e/0x160
[  663.226152]  aio_write+0xd1/0x160
[  663.226153]  ? __might_fault+0x3e/0x90
[  663.226155]  do_io_submit+0x37d/0x900
[  663.226156]  ? do_io_submit+0x1ac/0x900
[  663.226157]  SyS_io_submit+0xb/0x10
[  663.226159]  entry_SYSCALL_64_fastpath+0x18/0xad
[  663.226160] RIP: 0033:0x7fc269d02787
[  663.226160] RSP: 002b:00007fc24866c948 EFLAGS: 00000202 ORIG_RAX: 00000000000000d1
[  663.226162] RAX: ffffffffffffffda RBX: 0000000000000046 RCX: 00007fc269d02787
[  663.226163] RDX: 00007fc1d804f1e8 RSI: 0000000000000001 RDI: 00007fc27bbf6000
[  663.226164] RBP: 0000000000000065 R08: 0000000000000001 R09: 00007fc1d804b020
[  663.226165] R10: 00007fc1d802e000 R11: 0000000000000202 R12: 0000000000000020
[  663.226166] R13: 00007fc1d804ffe0 R14: 0000000000001000 R15: 0000000000000080
[  663.226167] fio             D    0  9278   8846 0x00000000
[  663.226169] Call Trace:
[  663.226171]  __schedule+0x2da/0xb00
[  663.226172]  ? _raw_spin_unlock_irq+0x27/0x40
[  663.226174]  schedule+0x38/0x90
[  663.226176]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.226178]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.226181]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.226183]  ? ___slab_alloc+0x178/0x520
[  663.226186]  call_rwsem_down_write_failed+0x17/0x30
[  663.226188]  down_write+0x5a/0x70
[  663.226190]  ? ext4_file_write_iter+0x45/0x360
[  663.226191]  ext4_file_write_iter+0x45/0x360
[  663.226193]  ? __sb_start_write+0xde/0x200
[  663.226194]  ? aio_write+0x14e/0x160
[  663.226195]  aio_write+0xd1/0x160
[  663.226197]  ? __might_fault+0x3e/0x90
[  663.226198]  do_io_submit+0x37d/0x900
[  663.226201]  ? do_io_submit+0x1ac/0x900
[  663.226202]  SyS_io_submit+0xb/0x10
[  663.226204]  entry_SYSCALL_64_fastpath+0x18/0xad
[  663.226205] RIP: 0033:0x7fc269d02787
[  663.226206] RSP: 002b:00007fc247e6b948 EFLAGS: 00000206 ORIG_RAX: 00000000000000d1
[  663.226208] RAX: ffffffffffffffda RBX: 0000000000000046 RCX: 00007fc269d02787
[  663.226209] RDX: 00007fc23c04f288 RSI: 0000000000000001 RDI: 00007fc27bc0a000
[  663.226210] RBP: 0000000000000065 R08: 0000000000000001 R09: 00007fc23c0476a0
[  663.226211] R10: 00007fc23c01a000 R11: 0000000000000206 R12: 0000000000000020
[  663.226212] R13: 00007fc23c04ffe0 R14: 0000000000001000 R15: 0000000000000080
[  663.226214] fio             D    0  9279   8846 0x00000000
[  663.226216] Call Trace:
[  663.226218]  __schedule+0x2da/0xb00
[  663.226220]  schedule+0x38/0x90
[  663.226222]  schedule_timeout+0x2fe/0x640
[  663.226223]  ? mark_held_locks+0x6f/0xa0
[  663.226225]  ? ktime_get+0x74/0x130
[  663.226227]  ? trace_hardirqs_on_caller+0xf9/0x1b0
[  663.226228]  ? trace_hardirqs_on+0xd/0x10
[  663.226230]  ? ktime_get+0x98/0x130
[  663.226231]  ? __delayacct_blkio_start+0x1a/0x30
[  663.226233]  io_schedule_timeout+0x9f/0x110
[  663.226235]  blk_mq_get_tag+0x158/0x260
[  663.226237]  ? remove_wait_queue+0x70/0x70
[  663.226239]  __blk_mq_alloc_request+0x16/0xe0
[  663.226241]  blk_mq_sched_get_request+0x279/0x370
[  663.226243]  ? blk_mq_sched_bypass_insert+0x70/0x70
[  663.226245]  ? generic_make_request+0xca/0x290
[  663.226247]  blk_sq_make_request+0x111/0xc90
[  663.226249]  ? blk_queue_enter+0x2d/0x280
[  663.226251]  ? generic_make_request+0xca/0x290
[  663.226253]  generic_make_request+0xd7/0x290
[  663.226255]  submit_bio+0x5f/0x120
[  663.226257]  ? trace_hardirqs_on+0xd/0x10
[  663.226258]  do_blockdev_direct_IO+0x280f/0x31f0
[  663.226261]  ? ext4_get_block_trans+0xe0/0xe0
[  663.226263]  __blockdev_direct_IO+0x35/0x40
[  663.226265]  ext4_direct_IO+0x19c/0x7b0
[  663.226267]  generic_file_direct_write+0xa6/0x150
[  663.226269]  __generic_file_write_iter+0xbb/0x1c0
[  663.226270]  ext4_file_write_iter+0x77/0x360
[  663.226272]  ? __sb_start_write+0xde/0x200
[  663.226274]  ? aio_write+0x14e/0x160
[  663.226275]  aio_write+0xd1/0x160
[  663.226276]  ? __might_fault+0x3e/0x90
[  663.226278]  do_io_submit+0x37d/0x900
[  663.226279]  ? do_io_submit+0x1ac/0x900
[  663.226281]  SyS_io_submit+0xb/0x10
[  663.226282]  entry_SYSCALL_64_fastpath+0x18/0xad
[  663.226283] RIP: 0033:0x7fc269d02787
[  663.226284] RSP: 002b:00007fc24766a948 EFLAGS: 00000202 ORIG_RAX: 00000000000000d1
[  663.226285] RAX: ffffffffffffffda RBX: 0000000000000046 RCX: 00007fc269d02787
[  663.226286] RDX: 00007fc1e804f248 RSI: 0000000000000001 RDI: 00007fc27bbf4000
[  663.226287] RBP: 0000000000000088 R08: 0000000000000001 R09: 00007fc1e8048da0
[  663.226288] R10: 00007fc1e8022000 R11: 0000000000000202 R12: 00007fc24f116770
[  663.226289] R13: 00007fc1e804f360 R14: 0000000000000000 R15: 0000000000000001
[  663.226291] fio             D    0  9280   8846 0x00000000
[  663.226293] Call Trace:
[  663.226295]  __schedule+0x2da/0xb00
[  663.226297]  schedule+0x38/0x90
[  663.226298]  schedule_timeout+0x2fe/0x640
[  663.226300]  ? mark_held_locks+0x6f/0xa0
[  663.226302]  ? ktime_get+0x74/0x130
[  663.226303]  ? trace_hardirqs_on_caller+0xf9/0x1b0
[  663.226305]  ? trace_hardirqs_on+0xd/0x10
[  663.226307]  ? ktime_get+0x98/0x130
[  663.226309]  ? __delayacct_blkio_start+0x1a/0x30
[  663.226310]  io_schedule_timeout+0x9f/0x110
[  663.226312]  blk_mq_get_tag+0x158/0x260
[  663.226313]  ? remove_wait_queue+0x70/0x70
[  663.226315]  __blk_mq_alloc_request+0x16/0xe0
[  663.226317]  blk_mq_sched_get_request+0x279/0x370
[  663.226319]  ? blk_mq_sched_bypass_insert+0x70/0x70
[  663.226321]  ? generic_make_request+0xca/0x290
[  663.226323]  blk_sq_make_request+0x111/0xc90
[  663.226325]  ? blk_queue_enter+0x2d/0x280
[  663.226327]  ? generic_make_request+0xca/0x290
[  663.226329]  generic_make_request+0xd7/0x290
[  663.226331]  submit_bio+0x5f/0x120
[  663.226333]  ? trace_hardirqs_on+0xd/0x10
[  663.226334]  do_blockdev_direct_IO+0x280f/0x31f0
[  663.226337]  ? debug_lockdep_rcu_enabled+0x1d/0x20
[  663.226338]  ? _raw_spin_unlock+0x22/0x30
[  663.226341]  ? ext4_get_block_trans+0xe0/0xe0
[  663.226343]  __blockdev_direct_IO+0x35/0x40
[  663.226345]  ext4_direct_IO+0x19c/0x7b0
[  663.226347]  generic_file_direct_write+0xa6/0x150
[  663.226349]  __generic_file_write_iter+0xbb/0x1c0
[  663.226350]  ext4_file_write_iter+0x77/0x360
[  663.226352]  ? __sb_start_write+0xde/0x200
[  663.226354]  ? aio_write+0x14e/0x160
[  663.226355]  aio_write+0xd1/0x160
[  663.226357]  ? __might_fault+0x3e/0x90
[  663.226358]  do_io_submit+0x37d/0x900
[  663.226360]  ? do_io_submit+0x1ac/0x900
[  663.226362]  SyS_io_submit+0xb/0x10
[  663.226363]  entry_SYSCALL_64_fastpath+0x18/0xad
[  663.226364] RIP: 0033:0x7fc269d02787
[  663.226365] RSP: 002b:00007fc246e69948 EFLAGS: 00000202 ORIG_RAX: 00000000000000d1
[  663.226367] RAX: ffffffffffffffda RBX: 0000000000000046 RCX: 00007fc269d02787
[  663.226368] RDX: 00007fc1e004f188 RSI: 0000000000000001 RDI: 00007fc27bc04000
[  663.226369] RBP: 0000000000000065 R08: 0000000000000001 R09: 00007fc1e004d2a0
[  663.226370] R10: 00007fc1e0050000 R11: 0000000000000202 R12: 0000000000000020
[  663.226371] R13: 00007fc1e004ffe0 R14: 0000000000001000 R15: 0000000000000080
[  663.226374] kworker/10:4    D    0  9296      2 0x00000000
[  663.226377] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.226378] Call Trace:
[  663.226381]  __schedule+0x2da/0xb00
[  663.226383]  ? bit_wait+0x50/0x50
[  663.226384]  schedule+0x38/0x90
[  663.226386]  schedule_timeout+0x2fe/0x640
[  663.226388]  ? mark_held_locks+0x6f/0xa0
[  663.226390]  ? ktime_get+0x74/0x130
[  663.226392]  ? bit_wait+0x50/0x50
[  663.226393]  ? trace_hardirqs_on_caller+0xf9/0x1b0
[  663.226395]  ? trace_hardirqs_on+0xd/0x10
[  663.226396]  ? ktime_get+0x98/0x130
[  663.226398]  ? __delayacct_blkio_start+0x1a/0x30
[  663.226400]  ? bit_wait+0x50/0x50
[  663.226401]  io_schedule_timeout+0x9f/0x110
[  663.226404]  bit_wait_io+0x16/0x60
[  663.226406]  __wait_on_bit+0x53/0x80
[  663.226407]  ? bit_wait+0x50/0x50
[  663.226409]  out_of_line_wait_on_bit+0x6e/0x80
[  663.226411]  ? prepare_to_wait_event+0x170/0x170
[  663.226413]  sync_mapping_buffers+0x22f/0x390
[  663.226415]  __generic_file_fsync+0x4d/0x90
[  663.226418]  ext4_sync_file+0x2b4/0x540
[  663.226420]  vfs_fsync_range+0x46/0xa0
[  663.226421]  dio_complete+0x181/0x1b0
[  663.226423]  dio_aio_complete_work+0x17/0x20
[  663.226424]  process_one_work+0x208/0x6a0
[  663.226425]  ? process_one_work+0x18d/0x6a0
[  663.226427]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.226429]  worker_thread+0x49/0x4a0
[  663.226431]  kthread+0x107/0x140
[  663.226432]  ? process_one_work+0x6a0/0x6a0
[  663.226434]  ? kthread_create_on_node+0x40/0x40
[  663.226436]  ret_from_fork+0x2e/0x40
[  663.226438] kworker/10:5    D    0  9297      2 0x00000000
[  663.226441] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.226442] Call Trace:
[  663.226444]  __schedule+0x2da/0xb00
[  663.226446]  schedule+0x38/0x90
[  663.226448]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.226450]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.226452]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.226454]  call_rwsem_down_write_failed+0x17/0x30
[  663.226457]  down_write+0x5a/0x70
[  663.226458]  ? __generic_file_fsync+0x43/0x90
[  663.226460]  __generic_file_fsync+0x43/0x90
[  663.226462]  ext4_sync_file+0x2b4/0x540
[  663.226463]  vfs_fsync_range+0x46/0xa0
[  663.226465]  dio_complete+0x181/0x1b0
[  663.226466]  dio_aio_complete_work+0x17/0x20
[  663.226468]  process_one_work+0x208/0x6a0
[  663.226470]  ? process_one_work+0x18d/0x6a0
[  663.226472]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.226473]  worker_thread+0x49/0x4a0
[  663.226475]  kthread+0x107/0x140
[  663.226477]  ? process_one_work+0x6a0/0x6a0
[  663.226479]  ? kthread_create_on_node+0x40/0x40
[  663.226480]  ret_from_fork+0x2e/0x40
[  663.226482] kworker/10:6    D    0  9298      2 0x00000000
[  663.226485] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.226486] Call Trace:
[  663.226488]  __schedule+0x2da/0xb00
[  663.226490]  schedule+0x38/0x90
[  663.226492]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.226494]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.226496]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.226498]  call_rwsem_down_write_failed+0x17/0x30
[  663.226500]  down_write+0x5a/0x70
[  663.226503]  ? __generic_file_fsync+0x43/0x90
[  663.226504]  __generic_file_fsync+0x43/0x90
[  663.226506]  ext4_sync_file+0x2b4/0x540
[  663.226508]  vfs_fsync_range+0x46/0xa0
[  663.226509]  dio_complete+0x181/0x1b0
[  663.226510]  dio_aio_complete_work+0x17/0x20
[  663.226512]  process_one_work+0x208/0x6a0
[  663.226513]  ? process_one_work+0x18d/0x6a0
[  663.226515]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.226517]  worker_thread+0x49/0x4a0
[  663.226519]  kthread+0x107/0x140
[  663.226521]  ? process_one_work+0x6a0/0x6a0
[  663.226522]  ? kthread_create_on_node+0x40/0x40
[  663.226524]  ret_from_fork+0x2e/0x40
[  663.226525] kworker/10:7    D    0  9299      2 0x00000000
[  663.226528] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.226529] Call Trace:
[  663.226531]  __schedule+0x2da/0xb00
[  663.226533]  schedule+0x38/0x90
[  663.226535]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.226537]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.226539]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.226541]  call_rwsem_down_write_failed+0x17/0x30
[  663.226543]  down_write+0x5a/0x70
[  663.226545]  ? __generic_file_fsync+0x43/0x90
[  663.226546]  __generic_file_fsync+0x43/0x90
[  663.226548]  ext4_sync_file+0x2b4/0x540
[  663.226549]  vfs_fsync_range+0x46/0xa0
[  663.226550]  dio_complete+0x181/0x1b0
[  663.226552]  dio_aio_complete_work+0x17/0x20
[  663.226553]  process_one_work+0x208/0x6a0
[  663.226554]  ? process_one_work+0x18d/0x6a0
[  663.226556]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.226557]  worker_thread+0x49/0x4a0
[  663.226559]  kthread+0x107/0x140
[  663.226560]  ? process_one_work+0x6a0/0x6a0
[  663.226562]  ? kthread_create_on_node+0x40/0x40
[  663.226563]  ret_from_fork+0x2e/0x40
[  663.226565] kworker/10:8    D    0  9300      2 0x00000000
[  663.226568] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.226569] Call Trace:
[  663.226571]  __schedule+0x2da/0xb00
[  663.226573]  schedule+0x38/0x90
[  663.226575]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.226577]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.226578]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.226581]  call_rwsem_down_write_failed+0x17/0x30
[  663.226583]  down_write+0x5a/0x70
[  663.226584]  ? __generic_file_fsync+0x43/0x90
[  663.226586]  __generic_file_fsync+0x43/0x90
[  663.226588]  ext4_sync_file+0x2b4/0x540
[  663.226589]  vfs_fsync_range+0x46/0xa0
[  663.226591]  dio_complete+0x181/0x1b0
[  663.226592]  dio_aio_complete_work+0x17/0x20
[  663.226594]  process_one_work+0x208/0x6a0
[  663.226595]  ? process_one_work+0x18d/0x6a0
[  663.226597]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.226599]  worker_thread+0x49/0x4a0
[  663.226601]  kthread+0x107/0x140
[  663.226602]  ? process_one_work+0x6a0/0x6a0
[  663.226604]  ? kthread_create_on_node+0x40/0x40
[  663.226606]  ret_from_fork+0x2e/0x40
[  663.226607] kworker/10:9    D    0  9301      2 0x00000000
[  663.226610] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.226611] Call Trace:
[  663.226613]  __schedule+0x2da/0xb00
[  663.226615]  schedule+0x38/0x90
[  663.226617]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.226619]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.226621]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.226624]  call_rwsem_down_write_failed+0x17/0x30
[  663.226626]  down_write+0x5a/0x70
[  663.226628]  ? __generic_file_fsync+0x43/0x90
[  663.226629]  __generic_file_fsync+0x43/0x90
[  663.226632]  ext4_sync_file+0x2b4/0x540
[  663.226633]  vfs_fsync_range+0x46/0xa0
[  663.226635]  dio_complete+0x181/0x1b0
[  663.226636]  dio_aio_complete_work+0x17/0x20
[  663.226638]  process_one_work+0x208/0x6a0
[  663.226639]  ? process_one_work+0x18d/0x6a0
[  663.226641]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.226642]  worker_thread+0x49/0x4a0
[  663.226644]  kthread+0x107/0x140
[  663.226646]  ? process_one_work+0x6a0/0x6a0
[  663.226647]  ? kthread_create_on_node+0x40/0x40
[  663.226649]  ret_from_fork+0x2e/0x40
[  663.226650] kworker/10:10   D    0  9302      2 0x00000000
[  663.226653] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.226654] Call Trace:
[  663.226656]  __schedule+0x2da/0xb00
[  663.226658]  schedule+0x38/0x90
[  663.226660]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.226662]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.226664]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.226667]  call_rwsem_down_write_failed+0x17/0x30
[  663.226669]  down_write+0x5a/0x70
[  663.226671]  ? __generic_file_fsync+0x43/0x90
[  663.226673]  __generic_file_fsync+0x43/0x90
[  663.226674]  ext4_sync_file+0x2b4/0x540
[  663.226676]  vfs_fsync_range+0x46/0xa0
[  663.226677]  dio_complete+0x181/0x1b0
[  663.226678]  dio_aio_complete_work+0x17/0x20
[  663.226680]  process_one_work+0x208/0x6a0
[  663.226681]  ? process_one_work+0x18d/0x6a0
[  663.226683]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.226684]  worker_thread+0x49/0x4a0
[  663.226686]  kthread+0x107/0x140
[  663.226687]  ? process_one_work+0x6a0/0x6a0
[  663.226688]  ? kthread_create_on_node+0x40/0x40
[  663.226690]  ret_from_fork+0x2e/0x40
[  663.226692] kworker/10:11   D    0  9303      2 0x00000000
[  663.226695] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.226696] Call Trace:
[  663.226698]  __schedule+0x2da/0xb00
[  663.226700]  schedule+0x38/0x90
[  663.226702]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.226704]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.226706]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.226708]  call_rwsem_down_write_failed+0x17/0x30
[  663.226710]  down_write+0x5a/0x70
[  663.226712]  ? __generic_file_fsync+0x43/0x90
[  663.226713]  __generic_file_fsync+0x43/0x90
[  663.226715]  ext4_sync_file+0x2b4/0x540
[  663.226717]  vfs_fsync_range+0x46/0xa0
[  663.226718]  dio_complete+0x181/0x1b0
[  663.226719]  dio_aio_complete_work+0x17/0x20
[  663.226721]  process_one_work+0x208/0x6a0
[  663.226722]  ? process_one_work+0x18d/0x6a0
[  663.226724]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.226726]  worker_thread+0x49/0x4a0
[  663.226728]  kthread+0x107/0x140
[  663.226729]  ? process_one_work+0x6a0/0x6a0
[  663.226731]  ? kthread_create_on_node+0x40/0x40
[  663.226732]  ret_from_fork+0x2e/0x40
[  663.226734] kworker/10:12   D    0  9304      2 0x00000000
[  663.226737] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.226738] Call Trace:
[  663.226740]  __schedule+0x2da/0xb00
[  663.226742]  schedule+0x38/0x90
[  663.226744]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.226746]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.226748]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.226751]  call_rwsem_down_write_failed+0x17/0x30
[  663.226753]  down_write+0x5a/0x70
[  663.226755]  ? __generic_file_fsync+0x43/0x90
[  663.226757]  __generic_file_fsync+0x43/0x90
[  663.226758]  ext4_sync_file+0x2b4/0x540
[  663.226760]  vfs_fsync_range+0x46/0xa0
[  663.226761]  dio_complete+0x181/0x1b0
[  663.226763]  dio_aio_complete_work+0x17/0x20
[  663.226764]  process_one_work+0x208/0x6a0
[  663.226765]  ? process_one_work+0x18d/0x6a0
[  663.226767]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.226769]  worker_thread+0x49/0x4a0
[  663.226771]  kthread+0x107/0x140
[  663.226772]  ? process_one_work+0x6a0/0x6a0
[  663.226773]  ? kthread_create_on_node+0x40/0x40
[  663.226775]  ret_from_fork+0x2e/0x40
[  663.226777] kworker/10:13   D    0  9305      2 0x00000000
[  663.226779] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.226781] Call Trace:
[  663.226783]  __schedule+0x2da/0xb00
[  663.226785]  ? bit_wait+0x50/0x50
[  663.226787]  schedule+0x38/0x90
[  663.226789]  schedule_timeout+0x2fe/0x640
[  663.226790]  ? mark_held_locks+0x6f/0xa0
[  663.226792]  ? ktime_get+0x74/0x130
[  663.226794]  ? bit_wait+0x50/0x50
[  663.226795]  ? trace_hardirqs_on_caller+0xf9/0x1b0
[  663.226796]  ? trace_hardirqs_on+0xd/0x10
[  663.226798]  ? ktime_get+0x98/0x130
[  663.226800]  ? __delayacct_blkio_start+0x1a/0x30
[  663.226801]  ? bit_wait+0x50/0x50
[  663.226803]  io_schedule_timeout+0x9f/0x110
[  663.226805]  bit_wait_io+0x16/0x60
[  663.226808]  __wait_on_bit+0x53/0x80
[  663.226810]  ? bit_wait+0x50/0x50
[  663.226811]  out_of_line_wait_on_bit+0x6e/0x80
[  663.226813]  ? prepare_to_wait_event+0x170/0x170
[  663.226816]  sync_mapping_buffers+0x22f/0x390
[  663.226818]  __generic_file_fsync+0x4d/0x90
[  663.226820]  ext4_sync_file+0x2b4/0x540
[  663.226822]  vfs_fsync_range+0x46/0xa0
[  663.226823]  dio_complete+0x181/0x1b0
[  663.226824]  dio_aio_complete_work+0x17/0x20
[  663.226826]  process_one_work+0x208/0x6a0
[  663.226827]  ? process_one_work+0x18d/0x6a0
[  663.226829]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.226830]  worker_thread+0x49/0x4a0
[  663.226832]  kthread+0x107/0x140
[  663.226834]  ? process_one_work+0x6a0/0x6a0
[  663.226835]  ? kthread_create_on_node+0x40/0x40
[  663.226837]  ret_from_fork+0x2e/0x40
[  663.226838] kworker/10:14   D    0  9306      2 0x00000000
[  663.226841] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.226842] Call Trace:
[  663.226844]  __schedule+0x2da/0xb00
[  663.226846]  schedule+0x38/0x90
[  663.226848]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.226851]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.226853]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.226855]  ? trace_hardirqs_on+0xd/0x10
[  663.226857]  call_rwsem_down_write_failed+0x17/0x30
[  663.226859]  down_write+0x5a/0x70
[  663.226861]  ? __generic_file_fsync+0x43/0x90
[  663.226863]  __generic_file_fsync+0x43/0x90
[  663.226864]  ext4_sync_file+0x2b4/0x540
[  663.226866]  vfs_fsync_range+0x46/0xa0
[  663.226867]  dio_complete+0x181/0x1b0
[  663.226869]  dio_aio_complete_work+0x17/0x20
[  663.226870]  process_one_work+0x208/0x6a0
[  663.226872]  ? process_one_work+0x18d/0x6a0
[  663.226874]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.226875]  worker_thread+0x49/0x4a0
[  663.226877]  kthread+0x107/0x140
[  663.226878]  ? process_one_work+0x6a0/0x6a0
[  663.226880]  ? kthread_create_on_node+0x40/0x40
[  663.226881]  ret_from_fork+0x2e/0x40
[  663.226883] kworker/10:15   D    0  9307      2 0x00000000
[  663.226885] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.226886] Call Trace:
[  663.226888]  __schedule+0x2da/0xb00
[  663.226890]  schedule+0x38/0x90
[  663.226892]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.226894]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.226896]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.226898]  call_rwsem_down_write_failed+0x17/0x30
[  663.226900]  down_write+0x5a/0x70
[  663.226901]  ? __generic_file_fsync+0x43/0x90
[  663.226903]  __generic_file_fsync+0x43/0x90
[  663.226905]  ext4_sync_file+0x2b4/0x540
[  663.226906]  vfs_fsync_range+0x46/0xa0
[  663.226907]  dio_complete+0x181/0x1b0
[  663.226909]  dio_aio_complete_work+0x17/0x20
[  663.226910]  process_one_work+0x208/0x6a0
[  663.226912]  ? process_one_work+0x18d/0x6a0
[  663.226914]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.226916]  worker_thread+0x49/0x4a0
[  663.226917]  kthread+0x107/0x140
[  663.226918]  ? process_one_work+0x6a0/0x6a0
[  663.226920]  ? kthread_create_on_node+0x40/0x40
[  663.226921]  ret_from_fork+0x2e/0x40
[  663.226923] kworker/10:16   D    0  9308      2 0x00000000
[  663.226926] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.226927] Call Trace:
[  663.226929]  __schedule+0x2da/0xb00
[  663.226931]  schedule+0x38/0x90
[  663.226933]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.226935]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.226937]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.226939]  call_rwsem_down_write_failed+0x17/0x30
[  663.226941]  down_write+0x5a/0x70
[  663.226943]  ? __generic_file_fsync+0x43/0x90
[  663.226945]  __generic_file_fsync+0x43/0x90
[  663.226946]  ext4_sync_file+0x2b4/0x540
[  663.226948]  vfs_fsync_range+0x46/0xa0
[  663.226949]  dio_complete+0x181/0x1b0
[  663.226951]  dio_aio_complete_work+0x17/0x20
[  663.226952]  process_one_work+0x208/0x6a0
[  663.226954]  ? process_one_work+0x18d/0x6a0
[  663.226956]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.226957]  worker_thread+0x49/0x4a0
[  663.226959]  kthread+0x107/0x140
[  663.226960]  ? process_one_work+0x6a0/0x6a0
[  663.226962]  ? kthread_create_on_node+0x40/0x40
[  663.226964]  ret_from_fork+0x2e/0x40
[  663.226965] kworker/10:17   D    0  9309      2 0x00000000
[  663.226968] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.226969] Call Trace:
[  663.226971]  __schedule+0x2da/0xb00
[  663.226973]  schedule+0x38/0x90
[  663.226975]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.226978]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.226980]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.226982]  call_rwsem_down_write_failed+0x17/0x30
[  663.226984]  down_write+0x5a/0x70
[  663.226986]  ? __generic_file_fsync+0x43/0x90
[  663.226988]  __generic_file_fsync+0x43/0x90
[  663.226989]  ext4_sync_file+0x2b4/0x540
[  663.226992]  vfs_fsync_range+0x46/0xa0
[  663.226993]  dio_complete+0x181/0x1b0
[  663.226994]  dio_aio_complete_work+0x17/0x20
[  663.226995]  process_one_work+0x208/0x6a0
[  663.226996]  ? process_one_work+0x18d/0x6a0
[  663.226998]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.227000]  worker_thread+0x49/0x4a0
[  663.227001]  kthread+0x107/0x140
[  663.227002]  ? process_one_work+0x6a0/0x6a0
[  663.227004]  ? kthread_create_on_node+0x40/0x40
[  663.227005]  ret_from_fork+0x2e/0x40
[  663.227007] kworker/10:18   D    0  9310      2 0x00000000
[  663.227010] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.227011] Call Trace:
[  663.227013]  __schedule+0x2da/0xb00
[  663.227015]  schedule+0x38/0x90
[  663.227017]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.227019]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.227020]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.227023]  call_rwsem_down_write_failed+0x17/0x30
[  663.227025]  down_write+0x5a/0x70
[  663.227027]  ? __generic_file_fsync+0x43/0x90
[  663.227028]  __generic_file_fsync+0x43/0x90
[  663.227031]  ext4_sync_file+0x2b4/0x540
[  663.227032]  vfs_fsync_range+0x46/0xa0
[  663.227034]  dio_complete+0x181/0x1b0
[  663.227035]  dio_aio_complete_work+0x17/0x20
[  663.227036]  process_one_work+0x208/0x6a0
[  663.227037]  ? process_one_work+0x18d/0x6a0
[  663.227039]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.227041]  worker_thread+0x49/0x4a0
[  663.227043]  kthread+0x107/0x140
[  663.227044]  ? process_one_work+0x6a0/0x6a0
[  663.227046]  ? kthread_create_on_node+0x40/0x40
[  663.227047]  ret_from_fork+0x2e/0x40
[  663.227049] kworker/10:19   D    0  9311      2 0x00000000
[  663.227052] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.227053] Call Trace:
[  663.227055]  __schedule+0x2da/0xb00
[  663.227057]  schedule+0x38/0x90
[  663.227059]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.227061]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.227063]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.227066]  call_rwsem_down_write_failed+0x17/0x30
[  663.227068]  down_write+0x5a/0x70
[  663.227070]  ? __generic_file_fsync+0x43/0x90
[  663.227072]  __generic_file_fsync+0x43/0x90
[  663.227074]  ext4_sync_file+0x2b4/0x540
[  663.227075]  vfs_fsync_range+0x46/0xa0
[  663.227077]  dio_complete+0x181/0x1b0
[  663.227078]  dio_aio_complete_work+0x17/0x20
[  663.227079]  process_one_work+0x208/0x6a0
[  663.227080]  ? process_one_work+0x18d/0x6a0
[  663.227082]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.227083]  worker_thread+0x49/0x4a0
[  663.227085]  kthread+0x107/0x140
[  663.227087]  ? process_one_work+0x6a0/0x6a0
[  663.227088]  ? kthread_create_on_node+0x40/0x40
[  663.227090]  ret_from_fork+0x2e/0x40
[  663.227092] kworker/10:20   D    0  9312      2 0x00000000
[  663.227094] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.227095] Call Trace:
[  663.227098]  __schedule+0x2da/0xb00
[  663.227099]  ? _raw_spin_unlock_irq+0x27/0x40
[  663.227101]  schedule+0x38/0x90
[  663.227103]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.227105]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.227106]  ? trace_hardirqs_on+0xd/0x10
[  663.227108]  call_rwsem_down_write_failed+0x17/0x30
[  663.227111]  down_write+0x5a/0x70
[  663.227112]  ? __generic_file_fsync+0x43/0x90
[  663.227114]  __generic_file_fsync+0x43/0x90
[  663.227115]  ext4_sync_file+0x2b4/0x540
[  663.227117]  vfs_fsync_range+0x46/0xa0
[  663.227118]  dio_complete+0x181/0x1b0
[  663.227119]  dio_aio_complete_work+0x17/0x20
[  663.227121]  process_one_work+0x208/0x6a0
[  663.227122]  ? process_one_work+0x18d/0x6a0
[  663.227123]  worker_thread+0x49/0x4a0
[  663.227125]  kthread+0x107/0x140
[  663.227126]  ? process_one_work+0x6a0/0x6a0
[  663.227128]  ? kthread_create_on_node+0x40/0x40
[  663.227129]  ret_from_fork+0x2e/0x40
[  663.227131] kworker/10:21   D    0  9313      2 0x00000000
[  663.227134] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.227135] Call Trace:
[  663.227137]  __schedule+0x2da/0xb00
[  663.227139]  schedule+0x38/0x90
[  663.227141]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.227143]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.227145]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.227147]  call_rwsem_down_write_failed+0x17/0x30
[  663.227149]  down_write+0x5a/0x70
[  663.227151]  ? __generic_file_fsync+0x43/0x90
[  663.227153]  __generic_file_fsync+0x43/0x90
[  663.227155]  ext4_sync_file+0x2b4/0x540
[  663.227156]  vfs_fsync_range+0x46/0xa0
[  663.227157]  dio_complete+0x181/0x1b0
[  663.227159]  dio_aio_complete_work+0x17/0x20
[  663.227160]  process_one_work+0x208/0x6a0
[  663.227161]  ? process_one_work+0x18d/0x6a0
[  663.227163]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.227165]  worker_thread+0x49/0x4a0
[  663.227167]  kthread+0x107/0x140
[  663.227168]  ? process_one_work+0x6a0/0x6a0
[  663.227170]  ? kthread_create_on_node+0x40/0x40
[  663.227171]  ret_from_fork+0x2e/0x40
[  663.227173] kworker/10:22   D    0  9314      2 0x00000000
[  663.227176] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.227177] Call Trace:
[  663.227179]  __schedule+0x2da/0xb00
[  663.227181]  schedule+0x38/0x90
[  663.227183]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.227185]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.227187]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.227190]  call_rwsem_down_write_failed+0x17/0x30
[  663.227192]  down_write+0x5a/0x70
[  663.227194]  ? __generic_file_fsync+0x43/0x90
[  663.227195]  __generic_file_fsync+0x43/0x90
[  663.227197]  ext4_sync_file+0x2b4/0x540
[  663.227199]  vfs_fsync_range+0x46/0xa0
[  663.227200]  dio_complete+0x181/0x1b0
[  663.227202]  dio_aio_complete_work+0x17/0x20
[  663.227203]  process_one_work+0x208/0x6a0
[  663.227204]  ? process_one_work+0x18d/0x6a0
[  663.227206]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.227207]  worker_thread+0x49/0x4a0
[  663.227209]  kthread+0x107/0x140
[  663.227210]  ? process_one_work+0x6a0/0x6a0
[  663.227212]  ? kthread_create_on_node+0x40/0x40
[  663.227213]  ret_from_fork+0x2e/0x40
[  663.227215] kworker/10:23   D    0  9315      2 0x00000000
[  663.227218] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.227219] Call Trace:
[  663.227221]  __schedule+0x2da/0xb00
[  663.227223]  schedule+0x38/0x90
[  663.227225]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.227226]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.227228]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.227231]  call_rwsem_down_write_failed+0x17/0x30
[  663.227233]  down_write+0x5a/0x70
[  663.227234]  ? __generic_file_fsync+0x43/0x90
[  663.227236]  __generic_file_fsync+0x43/0x90
[  663.227237]  ext4_sync_file+0x2b4/0x540
[  663.227239]  vfs_fsync_range+0x46/0xa0
[  663.227240]  dio_complete+0x181/0x1b0
[  663.227241]  dio_aio_complete_work+0x17/0x20
[  663.227242]  process_one_work+0x208/0x6a0
[  663.227243]  ? process_one_work+0x18d/0x6a0
[  663.227246]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.227247]  worker_thread+0x49/0x4a0
[  663.227249]  kthread+0x107/0x140
[  663.227250]  ? process_one_work+0x6a0/0x6a0
[  663.227252]  ? kthread_create_on_node+0x40/0x40
[  663.227253]  ret_from_fork+0x2e/0x40
[  663.227255] kworker/10:24   D    0  9316      2 0x00000000
[  663.227257] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.227258] Call Trace:
[  663.227260]  __schedule+0x2da/0xb00
[  663.227262]  schedule+0x38/0x90
[  663.227264]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.227265]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.227267]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.227269]  ? trace_hardirqs_on+0xd/0x10
[  663.227271]  call_rwsem_down_write_failed+0x17/0x30
[  663.227273]  down_write+0x5a/0x70
[  663.227274]  ? __generic_file_fsync+0x43/0x90
[  663.227276]  __generic_file_fsync+0x43/0x90
[  663.227278]  ext4_sync_file+0x2b4/0x540
[  663.227280]  vfs_fsync_range+0x46/0xa0
[  663.227281]  dio_complete+0x181/0x1b0
[  663.227282]  dio_aio_complete_work+0x17/0x20
[  663.227284]  process_one_work+0x208/0x6a0
[  663.227285]  ? process_one_work+0x18d/0x6a0
[  663.227287]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.227288]  worker_thread+0x49/0x4a0
[  663.227290]  kthread+0x107/0x140
[  663.227291]  ? process_one_work+0x6a0/0x6a0
[  663.227293]  ? kthread_create_on_node+0x40/0x40
[  663.227295]  ret_from_fork+0x2e/0x40
[  663.227296] kworker/10:25   D    0  9317      2 0x00000000
[  663.227299] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.227300] Call Trace:
[  663.227302]  __schedule+0x2da/0xb00
[  663.227304]  schedule+0x38/0x90
[  663.227306]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.227309]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.227311]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.227314]  call_rwsem_down_write_failed+0x17/0x30
[  663.227316]  down_write+0x5a/0x70
[  663.227318]  ? __generic_file_fsync+0x43/0x90
[  663.227320]  __generic_file_fsync+0x43/0x90
[  663.227321]  ext4_sync_file+0x2b4/0x540
[  663.227323]  vfs_fsync_range+0x46/0xa0
[  663.227324]  dio_complete+0x181/0x1b0
[  663.227326]  dio_aio_complete_work+0x17/0x20
[  663.227327]  process_one_work+0x208/0x6a0
[  663.227328]  ? process_one_work+0x18d/0x6a0
[  663.227330]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.227332]  worker_thread+0x49/0x4a0
[  663.227334]  kthread+0x107/0x140
[  663.227335]  ? process_one_work+0x6a0/0x6a0
[  663.227337]  ? kthread_create_on_node+0x40/0x40
[  663.227338]  ret_from_fork+0x2e/0x40
[  663.227340] kworker/10:26   D    0  9318      2 0x00000000
[  663.227342] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.227343] Call Trace:
[  663.227345]  __schedule+0x2da/0xb00
[  663.227347]  schedule+0x38/0x90
[  663.227349]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.227351]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.227353]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.227356]  call_rwsem_down_write_failed+0x17/0x30
[  663.227358]  down_write+0x5a/0x70
[  663.227359]  ? __generic_file_fsync+0x43/0x90
[  663.227361]  __generic_file_fsync+0x43/0x90
[  663.227363]  ext4_sync_file+0x2b4/0x540
[  663.227365]  vfs_fsync_range+0x46/0xa0
[  663.227366]  dio_complete+0x181/0x1b0
[  663.227367]  dio_aio_complete_work+0x17/0x20
[  663.227368]  process_one_work+0x208/0x6a0
[  663.227369]  ? process_one_work+0x18d/0x6a0
[  663.227371]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.227373]  worker_thread+0x49/0x4a0
[  663.227374]  kthread+0x107/0x140
[  663.227376]  ? process_one_work+0x6a0/0x6a0
[  663.227377]  ? kthread_create_on_node+0x40/0x40
[  663.227378]  ret_from_fork+0x2e/0x40
[  663.227380] kworker/10:27   D    0  9319      2 0x00000000
[  663.227383] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.227383] Call Trace:
[  663.227385]  __schedule+0x2da/0xb00
[  663.227387]  schedule+0x38/0x90
[  663.227389]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.227391]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.227393]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.227395]  call_rwsem_down_write_failed+0x17/0x30
[  663.227397]  down_write+0x5a/0x70
[  663.227400]  ? __generic_file_fsync+0x43/0x90
[  663.227401]  __generic_file_fsync+0x43/0x90
[  663.227403]  ext4_sync_file+0x2b4/0x540
[  663.227404]  vfs_fsync_range+0x46/0xa0
[  663.227406]  dio_complete+0x181/0x1b0
[  663.227407]  dio_aio_complete_work+0x17/0x20
[  663.227409]  process_one_work+0x208/0x6a0
[  663.227410]  ? process_one_work+0x18d/0x6a0
[  663.227412]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.227414]  worker_thread+0x49/0x4a0
[  663.227416]  kthread+0x107/0x140
[  663.227417]  ? process_one_work+0x6a0/0x6a0
[  663.227419]  ? kthread_create_on_node+0x40/0x40
[  663.227420]  ret_from_fork+0x2e/0x40
[  663.227422] kworker/10:28   D    0  9320      2 0x00000000
[  663.227425] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.227426] Call Trace:
[  663.227428]  __schedule+0x2da/0xb00
[  663.227430]  schedule+0x38/0x90
[  663.227432]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.227435]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.227437]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.227440]  call_rwsem_down_write_failed+0x17/0x30
[  663.227442]  down_write+0x5a/0x70
[  663.227443]  ? __generic_file_fsync+0x43/0x90
[  663.227445]  __generic_file_fsync+0x43/0x90
[  663.227447]  ext4_sync_file+0x2b4/0x540
[  663.227448]  vfs_fsync_range+0x46/0xa0
[  663.227449]  dio_complete+0x181/0x1b0
[  663.227451]  dio_aio_complete_work+0x17/0x20
[  663.227452]  process_one_work+0x208/0x6a0
[  663.227454]  ? process_one_work+0x18d/0x6a0
[  663.227455]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.227457]  worker_thread+0x49/0x4a0
[  663.227459]  kthread+0x107/0x140
[  663.227460]  ? process_one_work+0x6a0/0x6a0
[  663.227462]  ? kthread_create_on_node+0x40/0x40
[  663.227463]  ret_from_fork+0x2e/0x40
[  663.227466] kworker/10:29   D    0  9321      2 0x00000000
[  663.227468] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.227469] Call Trace:
[  663.227471]  __schedule+0x2da/0xb00
[  663.227473]  schedule+0x38/0x90
[  663.227475]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.227477]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.227479]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.227481]  call_rwsem_down_write_failed+0x17/0x30
[  663.227483]  down_write+0x5a/0x70
[  663.227485]  ? __generic_file_fsync+0x43/0x90
[  663.227486]  __generic_file_fsync+0x43/0x90
[  663.227488]  ext4_sync_file+0x2b4/0x540
[  663.227489]  vfs_fsync_range+0x46/0xa0
[  663.227490]  dio_complete+0x181/0x1b0
[  663.227492]  dio_aio_complete_work+0x17/0x20
[  663.227493]  process_one_work+0x208/0x6a0
[  663.227494]  ? process_one_work+0x18d/0x6a0
[  663.227496]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.227497]  worker_thread+0x49/0x4a0
[  663.227499]  kthread+0x107/0x140
[  663.227501]  ? process_one_work+0x6a0/0x6a0
[  663.227502]  ? kthread_create_on_node+0x40/0x40
[  663.227504]  ret_from_fork+0x2e/0x40
[  663.227505] kworker/10:30   D    0  9322      2 0x00000000
[  663.227508] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.227508] Call Trace:
[  663.227510]  __schedule+0x2da/0xb00
[  663.227512]  schedule+0x38/0x90
[  663.227514]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.227516]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.227518]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.227520]  ? trace_hardirqs_on+0xd/0x10
[  663.227523]  call_rwsem_down_write_failed+0x17/0x30
[  663.227525]  down_write+0x5a/0x70
[  663.227527]  ? __generic_file_fsync+0x43/0x90
[  663.227528]  __generic_file_fsync+0x43/0x90
[  663.227530]  ext4_sync_file+0x2b4/0x540
[  663.227532]  vfs_fsync_range+0x46/0xa0
[  663.227533]  dio_complete+0x181/0x1b0
[  663.227535]  dio_aio_complete_work+0x17/0x20
[  663.227536]  process_one_work+0x208/0x6a0
[  663.227537]  ? process_one_work+0x18d/0x6a0
[  663.227539]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.227540]  worker_thread+0x49/0x4a0
[  663.227542]  kthread+0x107/0x140
[  663.227544]  ? process_one_work+0x6a0/0x6a0
[  663.227545]  ? kthread_create_on_node+0x40/0x40
[  663.227547]  ret_from_fork+0x2e/0x40
[  663.227549] kworker/10:31   D    0  9323      2 0x00000000
[  663.227551] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.227552] Call Trace:
[  663.227555]  __schedule+0x2da/0xb00
[  663.227556]  schedule+0x38/0x90
[  663.227559]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.227561]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.227563]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.227565]  call_rwsem_down_write_failed+0x17/0x30
[  663.227568]  down_write+0x5a/0x70
[  663.227569]  ? __generic_file_fsync+0x43/0x90
[  663.227571]  __generic_file_fsync+0x43/0x90
[  663.227572]  ext4_sync_file+0x2b4/0x540
[  663.227574]  vfs_fsync_range+0x46/0xa0
[  663.227575]  dio_complete+0x181/0x1b0
[  663.227577]  dio_aio_complete_work+0x17/0x20
[  663.227578]  process_one_work+0x208/0x6a0
[  663.227579]  ? process_one_work+0x18d/0x6a0
[  663.227581]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.227582]  worker_thread+0x49/0x4a0
[  663.227584]  kthread+0x107/0x140
[  663.227586]  ? process_one_work+0x6a0/0x6a0
[  663.227587]  ? kthread_create_on_node+0x40/0x40
[  663.227589]  ret_from_fork+0x2e/0x40
[  663.227591] kworker/10:32   D    0  9324      2 0x00000000
[  663.227593] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.227594] Call Trace:
[  663.227596]  __schedule+0x2da/0xb00
[  663.227598]  schedule+0x38/0x90
[  663.227601]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.227603]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.227604]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.227607]  call_rwsem_down_write_failed+0x17/0x30
[  663.227609]  down_write+0x5a/0x70
[  663.227610]  ? __generic_file_fsync+0x43/0x90
[  663.227612]  __generic_file_fsync+0x43/0x90
[  663.227614]  ext4_sync_file+0x2b4/0x540
[  663.227616]  vfs_fsync_range+0x46/0xa0
[  663.227617]  dio_complete+0x181/0x1b0
[  663.227618]  dio_aio_complete_work+0x17/0x20
[  663.227619]  process_one_work+0x208/0x6a0
[  663.227620]  ? process_one_work+0x18d/0x6a0
[  663.227622]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.227624]  worker_thread+0x49/0x4a0
[  663.227625]  kthread+0x107/0x140
[  663.227626]  ? process_one_work+0x6a0/0x6a0
[  663.227628]  ? kthread_create_on_node+0x40/0x40
[  663.227629]  ret_from_fork+0x2e/0x40
[  663.227631] kworker/10:33   D    0  9325      2 0x00000000
[  663.227634] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.227635] Call Trace:
[  663.227637]  __schedule+0x2da/0xb00
[  663.227639]  schedule+0x38/0x90
[  663.227641]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.227643]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.227645]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.227647]  call_rwsem_down_write_failed+0x17/0x30
[  663.227649]  down_write+0x5a/0x70
[  663.227651]  ? __generic_file_fsync+0x43/0x90
[  663.227653]  __generic_file_fsync+0x43/0x90
[  663.227654]  ext4_sync_file+0x2b4/0x540
[  663.227656]  vfs_fsync_range+0x46/0xa0
[  663.227657]  dio_complete+0x181/0x1b0
[  663.227659]  dio_aio_complete_work+0x17/0x20
[  663.227660]  process_one_work+0x208/0x6a0
[  663.227661]  ? process_one_work+0x18d/0x6a0
[  663.227663]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.227665]  worker_thread+0x49/0x4a0
[  663.227667]  kthread+0x107/0x140
[  663.227668]  ? process_one_work+0x6a0/0x6a0
[  663.227670]  ? kthread_create_on_node+0x40/0x40
[  663.227672]  ret_from_fork+0x2e/0x40
[  663.227673] kworker/10:34   D    0  9326      2 0x00000000
[  663.227676] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.227677] Call Trace:
[  663.227679]  __schedule+0x2da/0xb00
[  663.227681]  schedule+0x38/0x90
[  663.227683]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.227685]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.227687]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.227690]  call_rwsem_down_write_failed+0x17/0x30
[  663.227692]  down_write+0x5a/0x70
[  663.227693]  ? __generic_file_fsync+0x43/0x90
[  663.227695]  __generic_file_fsync+0x43/0x90
[  663.227697]  ext4_sync_file+0x2b4/0x540
[  663.227698]  vfs_fsync_range+0x46/0xa0
[  663.227700]  dio_complete+0x181/0x1b0
[  663.227701]  dio_aio_complete_work+0x17/0x20
[  663.227702]  process_one_work+0x208/0x6a0
[  663.227703]  ? process_one_work+0x18d/0x6a0
[  663.227705]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.227707]  worker_thread+0x49/0x4a0
[  663.227709]  kthread+0x107/0x140
[  663.227710]  ? process_one_work+0x6a0/0x6a0
[  663.227711]  ? kthread_create_on_node+0x40/0x40
[  663.227713]  ret_from_fork+0x2e/0x40
[  663.227715] kworker/10:35   D    0  9327      2 0x00000000
[  663.227717] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.227718] Call Trace:
[  663.227720]  __schedule+0x2da/0xb00
[  663.227722]  schedule+0x38/0x90
[  663.227724]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.227726]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.227728]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.227730]  call_rwsem_down_write_failed+0x17/0x30
[  663.227733]  down_write+0x5a/0x70
[  663.227734]  ? __generic_file_fsync+0x43/0x90
[  663.227736]  __generic_file_fsync+0x43/0x90
[  663.227738]  ext4_sync_file+0x2b4/0x540
[  663.227739]  vfs_fsync_range+0x46/0xa0
[  663.227741]  dio_complete+0x181/0x1b0
[  663.227742]  dio_aio_complete_work+0x17/0x20
[  663.227743]  process_one_work+0x208/0x6a0
[  663.227745]  ? process_one_work+0x18d/0x6a0
[  663.227747]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.227748]  worker_thread+0x49/0x4a0
[  663.227750]  kthread+0x107/0x140
[  663.227751]  ? process_one_work+0x6a0/0x6a0
[  663.227753]  ? kthread_create_on_node+0x40/0x40
[  663.227754]  ret_from_fork+0x2e/0x40
[  663.227756] kworker/10:36   D    0  9328      2 0x00000000
[  663.227759] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.227760] Call Trace:
[  663.227762]  __schedule+0x2da/0xb00
[  663.227764]  schedule+0x38/0x90
[  663.227767]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.227769]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.227771]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.227773]  call_rwsem_down_write_failed+0x17/0x30
[  663.227775]  down_write+0x5a/0x70
[  663.227777]  ? __generic_file_fsync+0x43/0x90
[  663.227779]  __generic_file_fsync+0x43/0x90
[  663.227780]  ext4_sync_file+0x2b4/0x540
[  663.227782]  vfs_fsync_range+0x46/0xa0
[  663.227783]  dio_complete+0x181/0x1b0
[  663.227785]  dio_aio_complete_work+0x17/0x20
[  663.227786]  process_one_work+0x208/0x6a0
[  663.227787]  ? process_one_work+0x18d/0x6a0
[  663.227789]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.227790]  worker_thread+0x49/0x4a0
[  663.227792]  kthread+0x107/0x140
[  663.227794]  ? process_one_work+0x6a0/0x6a0
[  663.227795]  ? kthread_create_on_node+0x40/0x40
[  663.227797]  ret_from_fork+0x2e/0x40
[  663.227799] kworker/10:37   D    0  9329      2 0x00000000
[  663.227802] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.227803] Call Trace:
[  663.227805]  __schedule+0x2da/0xb00
[  663.227807]  schedule+0x38/0x90
[  663.227809]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.227810]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.227812]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.227815]  call_rwsem_down_write_failed+0x17/0x30
[  663.227817]  down_write+0x5a/0x70
[  663.227818]  ? __generic_file_fsync+0x43/0x90
[  663.227820]  __generic_file_fsync+0x43/0x90
[  663.227821]  ext4_sync_file+0x2b4/0x540
[  663.227823]  vfs_fsync_range+0x46/0xa0
[  663.227824]  dio_complete+0x181/0x1b0
[  663.227825]  dio_aio_complete_work+0x17/0x20
[  663.227826]  process_one_work+0x208/0x6a0
[  663.227828]  ? process_one_work+0x18d/0x6a0
[  663.227829]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.227831]  worker_thread+0x49/0x4a0
[  663.227833]  kthread+0x107/0x140
[  663.227834]  ? process_one_work+0x6a0/0x6a0
[  663.227836]  ? kthread_create_on_node+0x40/0x40
[  663.227837]  ret_from_fork+0x2e/0x40
[  663.227838] kworker/10:38   D    0  9330      2 0x00000000
[  663.227841] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.227842] Call Trace:
[  663.227844]  __schedule+0x2da/0xb00
[  663.227845]  schedule+0x38/0x90
[  663.227847]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.227849]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.227851]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.227853]  call_rwsem_down_write_failed+0x17/0x30
[  663.227855]  down_write+0x5a/0x70
[  663.227857]  ? __generic_file_fsync+0x43/0x90
[  663.227858]  __generic_file_fsync+0x43/0x90
[  663.227860]  ext4_sync_file+0x2b4/0x540
[  663.227861]  vfs_fsync_range+0x46/0xa0
[  663.227863]  dio_complete+0x181/0x1b0
[  663.227864]  dio_aio_complete_work+0x17/0x20
[  663.227865]  process_one_work+0x208/0x6a0
[  663.227866]  ? process_one_work+0x18d/0x6a0
[  663.227868]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.227869]  worker_thread+0x49/0x4a0
[  663.227871]  kthread+0x107/0x140
[  663.227872]  ? process_one_work+0x6a0/0x6a0
[  663.227874]  ? kthread_create_on_node+0x40/0x40
[  663.227876]  ret_from_fork+0x2e/0x40
[  663.227877] kworker/10:39   D    0  9331      2 0x00000000
[  663.227880] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.227881] Call Trace:
[  663.227883]  __schedule+0x2da/0xb00
[  663.227885]  schedule+0x38/0x90
[  663.227887]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.227889]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.227891]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.227894]  call_rwsem_down_write_failed+0x17/0x30
[  663.227896]  down_write+0x5a/0x70
[  663.227897]  ? __generic_file_fsync+0x43/0x90
[  663.227899]  __generic_file_fsync+0x43/0x90
[  663.227901]  ext4_sync_file+0x2b4/0x540
[  663.227902]  vfs_fsync_range+0x46/0xa0
[  663.227904]  dio_complete+0x181/0x1b0
[  663.227905]  dio_aio_complete_work+0x17/0x20
[  663.227907]  process_one_work+0x208/0x6a0
[  663.227908]  ? process_one_work+0x18d/0x6a0
[  663.227910]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.227911]  worker_thread+0x49/0x4a0
[  663.227913]  kthread+0x107/0x140
[  663.227915]  ? process_one_work+0x6a0/0x6a0
[  663.227916]  ? kthread_create_on_node+0x40/0x40
[  663.227918]  ret_from_fork+0x2e/0x40
[  663.227920] kworker/10:40   D    0  9332      2 0x00000000
[  663.227922] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.227923] Call Trace:
[  663.227925]  __schedule+0x2da/0xb00
[  663.227927]  schedule+0x38/0x90
[  663.227929]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.227931]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.227933]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.227936]  call_rwsem_down_write_failed+0x17/0x30
[  663.227938]  down_write+0x5a/0x70
[  663.227939]  ? __generic_file_fsync+0x43/0x90
[  663.227941]  __generic_file_fsync+0x43/0x90
[  663.227942]  ext4_sync_file+0x2b4/0x540
[  663.227944]  vfs_fsync_range+0x46/0xa0
[  663.227945]  dio_complete+0x181/0x1b0
[  663.227947]  dio_aio_complete_work+0x17/0x20
[  663.227948]  process_one_work+0x208/0x6a0
[  663.227949]  ? process_one_work+0x18d/0x6a0
[  663.227951]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.227953]  worker_thread+0x49/0x4a0
[  663.227954]  kthread+0x107/0x140
[  663.227956]  ? process_one_work+0x6a0/0x6a0
[  663.227957]  ? kthread_create_on_node+0x40/0x40
[  663.227959]  ret_from_fork+0x2e/0x40
[  663.227960] kworker/10:41   D    0  9333      2 0x00000000
[  663.227963] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.227964] Call Trace:
[  663.227966]  __schedule+0x2da/0xb00
[  663.227968]  schedule+0x38/0x90
[  663.227969]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.227971]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.227973]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.227976]  call_rwsem_down_write_failed+0x17/0x30
[  663.227978]  down_write+0x5a/0x70
[  663.227979]  ? __generic_file_fsync+0x43/0x90
[  663.227981]  __generic_file_fsync+0x43/0x90
[  663.227982]  ext4_sync_file+0x2b4/0x540
[  663.227983]  vfs_fsync_range+0x46/0xa0
[  663.227985]  dio_complete+0x181/0x1b0
[  663.227986]  dio_aio_complete_work+0x17/0x20
[  663.227987]  process_one_work+0x208/0x6a0
[  663.227988]  ? process_one_work+0x18d/0x6a0
[  663.227990]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.227991]  worker_thread+0x49/0x4a0
[  663.227993]  kthread+0x107/0x140
[  663.227994]  ? process_one_work+0x6a0/0x6a0
[  663.227996]  ? kthread_create_on_node+0x40/0x40
[  663.227997]  ret_from_fork+0x2e/0x40
[  663.227999] kworker/10:42   D    0  9334      2 0x00000000
[  663.228001] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.228002] Call Trace:
[  663.228004]  __schedule+0x2da/0xb00
[  663.228006]  schedule+0x38/0x90
[  663.228008]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.228011]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.228013]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.228015]  call_rwsem_down_write_failed+0x17/0x30
[  663.228017]  down_write+0x5a/0x70
[  663.228019]  ? __generic_file_fsync+0x43/0x90
[  663.228020]  __generic_file_fsync+0x43/0x90
[  663.228022]  ext4_sync_file+0x2b4/0x540
[  663.228024]  vfs_fsync_range+0x46/0xa0
[  663.228025]  dio_complete+0x181/0x1b0
[  663.228027]  dio_aio_complete_work+0x17/0x20
[  663.228028]  process_one_work+0x208/0x6a0
[  663.228029]  ? process_one_work+0x18d/0x6a0
[  663.228031]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.228033]  worker_thread+0x49/0x4a0
[  663.228035]  kthread+0x107/0x140
[  663.228036]  ? process_one_work+0x6a0/0x6a0
[  663.228038]  ? kthread_create_on_node+0x40/0x40
[  663.228039]  ret_from_fork+0x2e/0x40
[  663.228042] kworker/10:43   D    0  9335      2 0x00000000
[  663.228044] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.228045] Call Trace:
[  663.228048]  __schedule+0x2da/0xb00
[  663.228050]  schedule+0x38/0x90
[  663.228052]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.228054]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.228056]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.228058]  call_rwsem_down_write_failed+0x17/0x30
[  663.228060]  down_write+0x5a/0x70
[  663.228062]  ? __generic_file_fsync+0x43/0x90
[  663.228064]  __generic_file_fsync+0x43/0x90
[  663.228065]  ext4_sync_file+0x2b4/0x540
[  663.228067]  vfs_fsync_range+0x46/0xa0
[  663.228068]  dio_complete+0x181/0x1b0
[  663.228069]  dio_aio_complete_work+0x17/0x20
[  663.228071]  process_one_work+0x208/0x6a0
[  663.228072]  ? process_one_work+0x18d/0x6a0
[  663.228074]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.228075]  worker_thread+0x49/0x4a0
[  663.228077]  kthread+0x107/0x140
[  663.228079]  ? process_one_work+0x6a0/0x6a0
[  663.228081]  ? kthread_create_on_node+0x40/0x40
[  663.228082]  ret_from_fork+0x2e/0x40
[  663.228084] kworker/10:44   D    0  9336      2 0x00000000
[  663.228087] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.228088] Call Trace:
[  663.228090]  __schedule+0x2da/0xb00
[  663.228092]  schedule+0x38/0x90
[  663.228093]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.228095]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.228097]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.228100]  call_rwsem_down_write_failed+0x17/0x30
[  663.228101]  down_write+0x5a/0x70
[  663.228103]  ? __generic_file_fsync+0x43/0x90
[  663.228104]  __generic_file_fsync+0x43/0x90
[  663.228106]  ext4_sync_file+0x2b4/0x540
[  663.228108]  vfs_fsync_range+0x46/0xa0
[  663.228109]  dio_complete+0x181/0x1b0
[  663.228110]  dio_aio_complete_work+0x17/0x20
[  663.228112]  process_one_work+0x208/0x6a0
[  663.228114]  ? process_one_work+0x18d/0x6a0
[  663.228115]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.228117]  worker_thread+0x49/0x4a0
[  663.228118]  kthread+0x107/0x140
[  663.228120]  ? process_one_work+0x6a0/0x6a0
[  663.228122]  ? kthread_create_on_node+0x40/0x40
[  663.228123]  ret_from_fork+0x2e/0x40
[  663.228125] kworker/10:45   D    0  9337      2 0x00000000
[  663.228127] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.228128] Call Trace:
[  663.228130]  __schedule+0x2da/0xb00
[  663.228132]  schedule+0x38/0x90
[  663.228134]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.228136]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.228138]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.228141]  call_rwsem_down_write_failed+0x17/0x30
[  663.228143]  down_write+0x5a/0x70
[  663.228145]  ? __generic_file_fsync+0x43/0x90
[  663.228146]  __generic_file_fsync+0x43/0x90
[  663.228148]  ext4_sync_file+0x2b4/0x540
[  663.228149]  vfs_fsync_range+0x46/0xa0
[  663.228151]  dio_complete+0x181/0x1b0
[  663.228152]  dio_aio_complete_work+0x17/0x20
[  663.228153]  process_one_work+0x208/0x6a0
[  663.228155]  ? process_one_work+0x18d/0x6a0
[  663.228157]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.228158]  worker_thread+0x49/0x4a0
[  663.228160]  kthread+0x107/0x140
[  663.228162]  ? process_one_work+0x6a0/0x6a0
[  663.228164]  ? kthread_create_on_node+0x40/0x40
[  663.228166]  ? ___slab_alloc+0x4db/0x520
[  663.228168]  ? mempool_alloc_slab+0x10/0x20
[  663.228169]  ret_from_fork+0x2e/0x40
[  663.228171] kworker/10:46   D    0  9338      2 0x00000000
[  663.228174] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.228175] Call Trace:
[  663.228177]  __schedule+0x2da/0xb00
[  663.228179]  schedule+0x38/0x90
[  663.228181]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.228183]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.228185]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.228188]  call_rwsem_down_write_failed+0x17/0x30
[  663.228190]  down_write+0x5a/0x70
[  663.228192]  ? __generic_file_fsync+0x43/0x90
[  663.228193]  __generic_file_fsync+0x43/0x90
[  663.228195]  ext4_sync_file+0x2b4/0x540
[  663.228196]  vfs_fsync_range+0x46/0xa0
[  663.228198]  dio_complete+0x181/0x1b0
[  663.228199]  dio_aio_complete_work+0x17/0x20
[  663.228200]  process_one_work+0x208/0x6a0
[  663.228202]  ? process_one_work+0x18d/0x6a0
[  663.228204]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.228206]  worker_thread+0x49/0x4a0
[  663.228207]  kthread+0x107/0x140
[  663.228208]  ? process_one_work+0x6a0/0x6a0
[  663.228210]  ? kthread_create_on_node+0x40/0x40
[  663.228212]  ? ___slab_alloc+0x4db/0x520
[  663.228214]  ? mempool_alloc_slab+0x10/0x20
[  663.228215]  ret_from_fork+0x2e/0x40
[  663.228217] kworker/10:47   D    0  9339      2 0x00000000
[  663.228219] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.228220] Call Trace:
[  663.228222]  __schedule+0x2da/0xb00
[  663.228224]  schedule+0x38/0x90
[  663.228226]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.228228]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.228230]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.228232]  call_rwsem_down_write_failed+0x17/0x30
[  663.228234]  down_write+0x5a/0x70
[  663.228236]  ? __generic_file_fsync+0x43/0x90
[  663.228238]  __generic_file_fsync+0x43/0x90
[  663.228239]  ext4_sync_file+0x2b4/0x540
[  663.228241]  vfs_fsync_range+0x46/0xa0
[  663.228243]  dio_complete+0x181/0x1b0
[  663.228244]  dio_aio_complete_work+0x17/0x20
[  663.228245]  process_one_work+0x208/0x6a0
[  663.228246]  ? process_one_work+0x18d/0x6a0
[  663.228248]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.228249]  worker_thread+0x49/0x4a0
[  663.228251]  kthread+0x107/0x140
[  663.228253]  ? process_one_work+0x6a0/0x6a0
[  663.228254]  ? kthread_create_on_node+0x40/0x40
[  663.228256]  ret_from_fork+0x2e/0x40
[  663.228258] kworker/10:48   D    0  9340      2 0x00000000
[  663.228261] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.228262] Call Trace:
[  663.228264]  __schedule+0x2da/0xb00
[  663.228266]  schedule+0x38/0x90
[  663.228268]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.228270]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.228272]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.228274]  call_rwsem_down_write_failed+0x17/0x30
[  663.228276]  down_write+0x5a/0x70
[  663.228278]  ? __generic_file_fsync+0x43/0x90
[  663.228280]  __generic_file_fsync+0x43/0x90
[  663.228282]  ext4_sync_file+0x2b4/0x540
[  663.228283]  vfs_fsync_range+0x46/0xa0
[  663.228285]  dio_complete+0x181/0x1b0
[  663.228286]  dio_aio_complete_work+0x17/0x20
[  663.228287]  process_one_work+0x208/0x6a0
[  663.228289]  ? process_one_work+0x18d/0x6a0
[  663.228291]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.228292]  worker_thread+0x49/0x4a0
[  663.228294]  kthread+0x107/0x140
[  663.228295]  ? process_one_work+0x6a0/0x6a0
[  663.228297]  ? kthread_create_on_node+0x40/0x40
[  663.228298]  ret_from_fork+0x2e/0x40
[  663.228300] kworker/10:49   D    0  9341      2 0x00000000
[  663.228303] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.228304] Call Trace:
[  663.228306]  __schedule+0x2da/0xb00
[  663.228308]  schedule+0x38/0x90
[  663.228310]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.228312]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.228313]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.228316]  call_rwsem_down_write_failed+0x17/0x30
[  663.228318]  down_write+0x5a/0x70
[  663.228319]  ? __generic_file_fsync+0x43/0x90
[  663.228321]  __generic_file_fsync+0x43/0x90
[  663.228323]  ext4_sync_file+0x2b4/0x540
[  663.228324]  vfs_fsync_range+0x46/0xa0
[  663.228326]  dio_complete+0x181/0x1b0
[  663.228327]  dio_aio_complete_work+0x17/0x20
[  663.228328]  process_one_work+0x208/0x6a0
[  663.228329]  ? process_one_work+0x18d/0x6a0
[  663.228331]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.228332]  worker_thread+0x49/0x4a0
[  663.228334]  kthread+0x107/0x140
[  663.228335]  ? process_one_work+0x6a0/0x6a0
[  663.228337]  ? kthread_create_on_node+0x40/0x40
[  663.228338]  ret_from_fork+0x2e/0x40
[  663.228340] kworker/10:50   D    0  9342      2 0x00000000
[  663.228342] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.228343] Call Trace:
[  663.228345]  __schedule+0x2da/0xb00
[  663.228347]  schedule+0x38/0x90
[  663.228348]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.228350]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.228352]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.228355]  call_rwsem_down_write_failed+0x17/0x30
[  663.228357]  down_write+0x5a/0x70
[  663.228359]  ? __generic_file_fsync+0x43/0x90
[  663.228360]  __generic_file_fsync+0x43/0x90
[  663.228362]  ext4_sync_file+0x2b4/0x540
[  663.228363]  vfs_fsync_range+0x46/0xa0
[  663.228364]  dio_complete+0x181/0x1b0
[  663.228366]  dio_aio_complete_work+0x17/0x20
[  663.228367]  process_one_work+0x208/0x6a0
[  663.228368]  ? process_one_work+0x18d/0x6a0
[  663.228370]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.228372]  worker_thread+0x49/0x4a0
[  663.228373]  kthread+0x107/0x140
[  663.228375]  ? process_one_work+0x6a0/0x6a0
[  663.228376]  ? kthread_create_on_node+0x40/0x40
[  663.228378]  ret_from_fork+0x2e/0x40
[  663.228379] kworker/10:51   D    0  9343      2 0x00000000
[  663.228382] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.228383] Call Trace:
[  663.228385]  __schedule+0x2da/0xb00
[  663.228387]  schedule+0x38/0x90
[  663.228390]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.228392]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.228394]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.228396]  call_rwsem_down_write_failed+0x17/0x30
[  663.228399]  down_write+0x5a/0x70
[  663.228400]  ? __generic_file_fsync+0x43/0x90
[  663.228402]  __generic_file_fsync+0x43/0x90
[  663.228404]  ext4_sync_file+0x2b4/0x540
[  663.228405]  vfs_fsync_range+0x46/0xa0
[  663.228407]  dio_complete+0x181/0x1b0
[  663.228408]  dio_aio_complete_work+0x17/0x20
[  663.228410]  process_one_work+0x208/0x6a0
[  663.228411]  ? process_one_work+0x18d/0x6a0
[  663.228413]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.228414]  worker_thread+0x49/0x4a0
[  663.228416]  kthread+0x107/0x140
[  663.228417]  ? process_one_work+0x6a0/0x6a0
[  663.228419]  ? kthread_create_on_node+0x40/0x40
[  663.228422]  ? kmem_cache_alloc+0x230/0x2c0
[  663.228424]  ? __slab_alloc+0x3e/0x70
[  663.228426]  ? ___slab_alloc+0x4db/0x520
[  663.228427]  ret_from_fork+0x2e/0x40
[  663.228429] kworker/10:52   D    0  9344      2 0x00000000
[  663.228432] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.228433] Call Trace:
[  663.228435]  __schedule+0x2da/0xb00
[  663.228436]  schedule+0x38/0x90
[  663.228438]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.228440]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.228442]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.228445]  call_rwsem_down_write_failed+0x17/0x30
[  663.228447]  down_write+0x5a/0x70
[  663.228449]  ? __generic_file_fsync+0x43/0x90
[  663.228450]  __generic_file_fsync+0x43/0x90
[  663.228452]  ext4_sync_file+0x2b4/0x540
[  663.228454]  vfs_fsync_range+0x46/0xa0
[  663.228455]  dio_complete+0x181/0x1b0
[  663.228456]  dio_aio_complete_work+0x17/0x20
[  663.228458]  process_one_work+0x208/0x6a0
[  663.228459]  ? process_one_work+0x18d/0x6a0
[  663.228460]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.228462]  worker_thread+0x49/0x4a0
[  663.228464]  kthread+0x107/0x140
[  663.228465]  ? process_one_work+0x6a0/0x6a0
[  663.228466]  ? kthread_create_on_node+0x40/0x40
[  663.228468]  ? ___slab_alloc+0x4db/0x520
[  663.228470]  ? mempool_alloc_slab+0x10/0x20
[  663.228471]  ret_from_fork+0x2e/0x40
[  663.228473] kworker/10:53   D    0  9345      2 0x00000000
[  663.228475] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.228476] Call Trace:
[  663.228478]  __schedule+0x2da/0xb00
[  663.228480]  schedule+0x38/0x90
[  663.228482]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.228484]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.228486]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.228488]  ? trace_hardirqs_on+0xd/0x10
[  663.228491]  call_rwsem_down_write_failed+0x17/0x30
[  663.228493]  down_write+0x5a/0x70
[  663.228495]  ? __generic_file_fsync+0x43/0x90
[  663.228497]  __generic_file_fsync+0x43/0x90
[  663.228498]  ext4_sync_file+0x2b4/0x540
[  663.228500]  vfs_fsync_range+0x46/0xa0
[  663.228501]  dio_complete+0x181/0x1b0
[  663.228503]  dio_aio_complete_work+0x17/0x20
[  663.228504]  process_one_work+0x208/0x6a0
[  663.228505]  ? process_one_work+0x18d/0x6a0
[  663.228507]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.228508]  worker_thread+0x49/0x4a0
[  663.228510]  kthread+0x107/0x140
[  663.228511]  ? process_one_work+0x6a0/0x6a0
[  663.228513]  ? kthread_create_on_node+0x40/0x40
[  663.228515]  ret_from_fork+0x2e/0x40
[  663.228517] kworker/10:54   D    0  9346      2 0x00000000
[  663.228520] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.228521] Call Trace:
[  663.228523]  __schedule+0x2da/0xb00
[  663.228525]  schedule+0x38/0x90
[  663.228527]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.228529]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.228531]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.228533]  ? trace_hardirqs_on+0xd/0x10
[  663.228535]  call_rwsem_down_write_failed+0x17/0x30
[  663.228537]  down_write+0x5a/0x70
[  663.228539]  ? __generic_file_fsync+0x43/0x90
[  663.228540]  __generic_file_fsync+0x43/0x90
[  663.228542]  ext4_sync_file+0x2b4/0x540
[  663.228544]  vfs_fsync_range+0x46/0xa0
[  663.228545]  dio_complete+0x181/0x1b0
[  663.228546]  dio_aio_complete_work+0x17/0x20
[  663.228548]  process_one_work+0x208/0x6a0
[  663.228549]  ? process_one_work+0x18d/0x6a0
[  663.228551]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.228552]  worker_thread+0x49/0x4a0
[  663.228554]  kthread+0x107/0x140
[  663.228555]  ? process_one_work+0x6a0/0x6a0
[  663.228557]  ? kthread_create_on_node+0x40/0x40
[  663.228559]  ret_from_fork+0x2e/0x40
[  663.228561] kworker/10:55   D    0  9347      2 0x00000000
[  663.228564] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.228565] Call Trace:
[  663.228567]  __schedule+0x2da/0xb00
[  663.228568]  schedule+0x38/0x90
[  663.228570]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.228572]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.228574]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.228576]  call_rwsem_down_write_failed+0x17/0x30
[  663.228578]  down_write+0x5a/0x70
[  663.228580]  ? __generic_file_fsync+0x43/0x90
[  663.228581]  __generic_file_fsync+0x43/0x90
[  663.228583]  ext4_sync_file+0x2b4/0x540
[  663.228584]  vfs_fsync_range+0x46/0xa0
[  663.228586]  dio_complete+0x181/0x1b0
[  663.228587]  dio_aio_complete_work+0x17/0x20
[  663.228589]  process_one_work+0x208/0x6a0
[  663.228590]  ? process_one_work+0x18d/0x6a0
[  663.228592]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.228593]  worker_thread+0x49/0x4a0
[  663.228595]  kthread+0x107/0x140
[  663.228597]  ? process_one_work+0x6a0/0x6a0
[  663.228598]  ? kthread_create_on_node+0x40/0x40
[  663.228600]  ret_from_fork+0x2e/0x40
[  663.228602] kworker/10:56   D    0  9348      2 0x00000000
[  663.228604] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.228605] Call Trace:
[  663.228607]  __schedule+0x2da/0xb00
[  663.228609]  schedule+0x38/0x90
[  663.228611]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.228613]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.228615]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.228617]  call_rwsem_down_write_failed+0x17/0x30
[  663.228619]  down_write+0x5a/0x70
[  663.228621]  ? __generic_file_fsync+0x43/0x90
[  663.228623]  __generic_file_fsync+0x43/0x90
[  663.228625]  ext4_sync_file+0x2b4/0x540
[  663.228626]  vfs_fsync_range+0x46/0xa0
[  663.228627]  dio_complete+0x181/0x1b0
[  663.228629]  dio_aio_complete_work+0x17/0x20
[  663.228630]  process_one_work+0x208/0x6a0
[  663.228632]  ? process_one_work+0x18d/0x6a0
[  663.228633]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.228635]  worker_thread+0x49/0x4a0
[  663.228637]  kthread+0x107/0x140
[  663.228638]  ? process_one_work+0x6a0/0x6a0
[  663.228640]  ? kthread_create_on_node+0x40/0x40
[  663.228641]  ret_from_fork+0x2e/0x40
[  663.228643] kworker/10:57   D    0  9349      2 0x00000000
[  663.228645] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.228647] Call Trace:
[  663.228649]  __schedule+0x2da/0xb00
[  663.228651]  schedule+0x38/0x90
[  663.228653]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.228655]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.228657]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.228660]  call_rwsem_down_write_failed+0x17/0x30
[  663.228662]  down_write+0x5a/0x70
[  663.228663]  ? __generic_file_fsync+0x43/0x90
[  663.228665]  __generic_file_fsync+0x43/0x90
[  663.228666]  ext4_sync_file+0x2b4/0x540
[  663.228668]  vfs_fsync_range+0x46/0xa0
[  663.228669]  dio_complete+0x181/0x1b0
[  663.228670]  dio_aio_complete_work+0x17/0x20
[  663.228671]  process_one_work+0x208/0x6a0
[  663.228672]  ? process_one_work+0x18d/0x6a0
[  663.228674]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.228675]  worker_thread+0x49/0x4a0
[  663.228677]  kthread+0x107/0x140
[  663.228678]  ? process_one_work+0x6a0/0x6a0
[  663.228680]  ? kthread_create_on_node+0x40/0x40
[  663.228681]  ret_from_fork+0x2e/0x40
[  663.228683] kworker/10:58   D    0  9350      2 0x00000000
[  663.228686] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.228687] Call Trace:
[  663.228689]  __schedule+0x2da/0xb00
[  663.228690]  schedule+0x38/0x90
[  663.228692]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.228694]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.228696]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.228698]  call_rwsem_down_write_failed+0x17/0x30
[  663.228701]  down_write+0x5a/0x70
[  663.228702]  ? __generic_file_fsync+0x43/0x90
[  663.228704]  __generic_file_fsync+0x43/0x90
[  663.228706]  ext4_sync_file+0x2b4/0x540
[  663.228707]  vfs_fsync_range+0x46/0xa0
[  663.228709]  dio_complete+0x181/0x1b0
[  663.228710]  dio_aio_complete_work+0x17/0x20
[  663.228711]  process_one_work+0x208/0x6a0
[  663.228712]  ? process_one_work+0x18d/0x6a0
[  663.228714]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.228716]  worker_thread+0x49/0x4a0
[  663.228718]  kthread+0x107/0x140
[  663.228719]  ? process_one_work+0x6a0/0x6a0
[  663.228721]  ? kthread_create_on_node+0x40/0x40
[  663.228722]  ret_from_fork+0x2e/0x40
[  663.228724] kworker/10:59   D    0  9351      2 0x00000000
[  663.228727] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.228728] Call Trace:
[  663.228730]  __schedule+0x2da/0xb00
[  663.228732]  schedule+0x38/0x90
[  663.228734]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.228736]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.228738]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.228740]  call_rwsem_down_write_failed+0x17/0x30
[  663.228742]  down_write+0x5a/0x70
[  663.228744]  ? __generic_file_fsync+0x43/0x90
[  663.228746]  __generic_file_fsync+0x43/0x90
[  663.228748]  ext4_sync_file+0x2b4/0x540
[  663.228750]  vfs_fsync_range+0x46/0xa0
[  663.228751]  dio_complete+0x181/0x1b0
[  663.228752]  dio_aio_complete_work+0x17/0x20
[  663.228754]  process_one_work+0x208/0x6a0
[  663.228755]  ? process_one_work+0x18d/0x6a0
[  663.228757]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.228758]  worker_thread+0x49/0x4a0
[  663.228760]  kthread+0x107/0x140
[  663.228761]  ? process_one_work+0x6a0/0x6a0
[  663.228763]  ? kthread_create_on_node+0x40/0x40
[  663.228764]  ret_from_fork+0x2e/0x40
[  663.228766] kworker/10:60   D    0  9352      2 0x00000000
[  663.228769] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.228770] Call Trace:
[  663.228772]  __schedule+0x2da/0xb00
[  663.228774]  schedule+0x38/0x90
[  663.228776]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.228778]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.228780]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.228782]  call_rwsem_down_write_failed+0x17/0x30
[  663.228784]  down_write+0x5a/0x70
[  663.228786]  ? __generic_file_fsync+0x43/0x90
[  663.228788]  __generic_file_fsync+0x43/0x90
[  663.228790]  ext4_sync_file+0x2b4/0x540
[  663.228791]  vfs_fsync_range+0x46/0xa0
[  663.228792]  dio_complete+0x181/0x1b0
[  663.228794]  dio_aio_complete_work+0x17/0x20
[  663.228795]  process_one_work+0x208/0x6a0
[  663.228796]  ? process_one_work+0x18d/0x6a0
[  663.228798]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.228799]  worker_thread+0x49/0x4a0
[  663.228801]  kthread+0x107/0x140
[  663.228802]  ? process_one_work+0x6a0/0x6a0
[  663.228804]  ? kthread_create_on_node+0x40/0x40
[  663.228805]  ret_from_fork+0x2e/0x40
[  663.228806] kworker/10:61   D    0  9353      2 0x00000000
[  663.228809] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.228811] Call Trace:
[  663.228813]  __schedule+0x2da/0xb00
[  663.228814]  schedule+0x38/0x90
[  663.228816]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.228818]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.228820]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.228823]  call_rwsem_down_write_failed+0x17/0x30
[  663.228825]  down_write+0x5a/0x70
[  663.228826]  ? __generic_file_fsync+0x43/0x90
[  663.228828]  __generic_file_fsync+0x43/0x90
[  663.228830]  ext4_sync_file+0x2b4/0x540
[  663.228831]  vfs_fsync_range+0x46/0xa0
[  663.228832]  dio_complete+0x181/0x1b0
[  663.228834]  dio_aio_complete_work+0x17/0x20
[  663.228835]  process_one_work+0x208/0x6a0
[  663.228836]  ? process_one_work+0x18d/0x6a0
[  663.228838]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.228840]  worker_thread+0x49/0x4a0
[  663.228842]  kthread+0x107/0x140
[  663.228843]  ? process_one_work+0x6a0/0x6a0
[  663.228845]  ? kthread_create_on_node+0x40/0x40
[  663.228847]  ret_from_fork+0x2e/0x40
[  663.228848] kworker/10:62   D    0  9354      2 0x00000000
[  663.228851] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.228852] Call Trace:
[  663.228854]  __schedule+0x2da/0xb00
[  663.228856]  schedule+0x38/0x90
[  663.228858]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.228860]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.228862]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.228865]  call_rwsem_down_write_failed+0x17/0x30
[  663.228867]  down_write+0x5a/0x70
[  663.228868]  ? __generic_file_fsync+0x43/0x90
[  663.228870]  __generic_file_fsync+0x43/0x90
[  663.228872]  ext4_sync_file+0x2b4/0x540
[  663.228873]  vfs_fsync_range+0x46/0xa0
[  663.228876]  dio_complete+0x181/0x1b0
[  663.228877]  dio_aio_complete_work+0x17/0x20
[  663.228879]  process_one_work+0x208/0x6a0
[  663.228880]  ? process_one_work+0x18d/0x6a0
[  663.228882]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.228883]  worker_thread+0x49/0x4a0
[  663.228885]  kthread+0x107/0x140
[  663.228886]  ? process_one_work+0x6a0/0x6a0
[  663.228888]  ? kthread_create_on_node+0x40/0x40
[  663.228889]  ret_from_fork+0x2e/0x40
[  663.228891] kworker/10:63   D    0  9355      2 0x00000000
[  663.228893] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.228894] Call Trace:
[  663.228896]  __schedule+0x2da/0xb00
[  663.228898]  schedule+0x38/0x90
[  663.228900]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.228902]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.228904]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.228905]  ? trace_hardirqs_on+0xd/0x10
[  663.228907]  call_rwsem_down_write_failed+0x17/0x30
[  663.228909]  down_write+0x5a/0x70
[  663.228911]  ? __generic_file_fsync+0x43/0x90
[  663.228913]  __generic_file_fsync+0x43/0x90
[  663.228914]  ext4_sync_file+0x2b4/0x540
[  663.228916]  vfs_fsync_range+0x46/0xa0
[  663.228917]  dio_complete+0x181/0x1b0
[  663.228918]  dio_aio_complete_work+0x17/0x20
[  663.228920]  process_one_work+0x208/0x6a0
[  663.228921]  ? process_one_work+0x18d/0x6a0
[  663.228923]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.228924]  worker_thread+0x49/0x4a0
[  663.228926]  kthread+0x107/0x140
[  663.228927]  ? process_one_work+0x6a0/0x6a0
[  663.228929]  ? kthread_create_on_node+0x40/0x40
[  663.228931]  ret_from_fork+0x2e/0x40
[  663.228933] kworker/10:64   D    0  9356      2 0x00000000
[  663.228935] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.228936] Call Trace:
[  663.228938]  __schedule+0x2da/0xb00
[  663.228940]  schedule+0x38/0x90
[  663.228943]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.228945]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.228947]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.228949]  call_rwsem_down_write_failed+0x17/0x30
[  663.228951]  down_write+0x5a/0x70
[  663.228953]  ? __generic_file_fsync+0x43/0x90
[  663.228955]  __generic_file_fsync+0x43/0x90
[  663.228956]  ext4_sync_file+0x2b4/0x540
[  663.228958]  vfs_fsync_range+0x46/0xa0
[  663.228959]  dio_complete+0x181/0x1b0
[  663.228961]  dio_aio_complete_work+0x17/0x20
[  663.228962]  process_one_work+0x208/0x6a0
[  663.228963]  ? process_one_work+0x18d/0x6a0
[  663.228965]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.228967]  worker_thread+0x49/0x4a0
[  663.228969]  kthread+0x107/0x140
[  663.228970]  ? process_one_work+0x6a0/0x6a0
[  663.228972]  ? kthread_create_on_node+0x40/0x40
[  663.228974]  ret_from_fork+0x2e/0x40
[  663.228976] kworker/10:65   D    0  9357      2 0x00000000
[  663.228978] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.228979] Call Trace:
[  663.228981]  __schedule+0x2da/0xb00
[  663.228983]  schedule+0x38/0x90
[  663.228985]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.228987]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.228989]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.228992]  call_rwsem_down_write_failed+0x17/0x30
[  663.228994]  down_write+0x5a/0x70
[  663.228995]  ? __generic_file_fsync+0x43/0x90
[  663.228997]  __generic_file_fsync+0x43/0x90
[  663.228998]  ext4_sync_file+0x2b4/0x540
[  663.229000]  vfs_fsync_range+0x46/0xa0
[  663.229001]  dio_complete+0x181/0x1b0
[  663.229002]  dio_aio_complete_work+0x17/0x20
[  663.229003]  process_one_work+0x208/0x6a0
[  663.229004]  ? process_one_work+0x18d/0x6a0
[  663.229007]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.229008]  worker_thread+0x49/0x4a0
[  663.229010]  kthread+0x107/0x140
[  663.229011]  ? process_one_work+0x6a0/0x6a0
[  663.229013]  ? kthread_create_on_node+0x40/0x40
[  663.229014]  ret_from_fork+0x2e/0x40
[  663.229016] kworker/10:66   D    0  9358      2 0x00000000
[  663.229018] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.229019] Call Trace:
[  663.229021]  __schedule+0x2da/0xb00
[  663.229023]  schedule+0x38/0x90
[  663.229025]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.229027]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.229029]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.229032]  call_rwsem_down_write_failed+0x17/0x30
[  663.229034]  down_write+0x5a/0x70
[  663.229036]  ? __generic_file_fsync+0x43/0x90
[  663.229038]  __generic_file_fsync+0x43/0x90
[  663.229040]  ext4_sync_file+0x2b4/0x540
[  663.229041]  vfs_fsync_range+0x46/0xa0
[  663.229043]  dio_complete+0x181/0x1b0
[  663.229044]  dio_aio_complete_work+0x17/0x20
[  663.229045]  process_one_work+0x208/0x6a0
[  663.229047]  ? process_one_work+0x18d/0x6a0
[  663.229049]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.229050]  worker_thread+0x49/0x4a0
[  663.229052]  kthread+0x107/0x140
[  663.229053]  ? process_one_work+0x6a0/0x6a0
[  663.229055]  ? kthread_create_on_node+0x40/0x40
[  663.229057]  ret_from_fork+0x2e/0x40
[  663.229058] kworker/10:67   D    0  9359      2 0x00000000
[  663.229061] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.229062] Call Trace:
[  663.229064]  __schedule+0x2da/0xb00
[  663.229066]  schedule+0x38/0x90
[  663.229068]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.229070]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.229072]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.229074]  ? trace_hardirqs_on+0xd/0x10
[  663.229076]  call_rwsem_down_write_failed+0x17/0x30
[  663.229079]  down_write+0x5a/0x70
[  663.229081]  ? __generic_file_fsync+0x43/0x90
[  663.229083]  __generic_file_fsync+0x43/0x90
[  663.229085]  ext4_sync_file+0x2b4/0x540
[  663.229086]  vfs_fsync_range+0x46/0xa0
[  663.229087]  dio_complete+0x181/0x1b0
[  663.229089]  dio_aio_complete_work+0x17/0x20
[  663.229090]  process_one_work+0x208/0x6a0
[  663.229092]  ? process_one_work+0x18d/0x6a0
[  663.229094]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.229095]  worker_thread+0x49/0x4a0
[  663.229097]  kthread+0x107/0x140
[  663.229098]  ? process_one_work+0x6a0/0x6a0
[  663.229099]  ? kthread_create_on_node+0x40/0x40
[  663.229101]  ret_from_fork+0x2e/0x40
[  663.229103] kworker/10:68   D    0  9360      2 0x00000000
[  663.229105] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.229106] Call Trace:
[  663.229108]  __schedule+0x2da/0xb00
[  663.229110]  schedule+0x38/0x90
[  663.229112]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.229113]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.229115]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.229118]  call_rwsem_down_write_failed+0x17/0x30
[  663.229120]  down_write+0x5a/0x70
[  663.229121]  ? __generic_file_fsync+0x43/0x90
[  663.229123]  __generic_file_fsync+0x43/0x90
[  663.229124]  ext4_sync_file+0x2b4/0x540
[  663.229126]  vfs_fsync_range+0x46/0xa0
[  663.229127]  dio_complete+0x181/0x1b0
[  663.229128]  dio_aio_complete_work+0x17/0x20
[  663.229130]  process_one_work+0x208/0x6a0
[  663.229131]  ? process_one_work+0x18d/0x6a0
[  663.229132]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.229134]  worker_thread+0x49/0x4a0
[  663.229136]  kthread+0x107/0x140
[  663.229137]  ? process_one_work+0x6a0/0x6a0
[  663.229138]  ? kthread_create_on_node+0x40/0x40
[  663.229140]  ret_from_fork+0x2e/0x40
[  663.229141] kworker/10:69   D    0  9361      2 0x00000000
[  663.229144] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.229145] Call Trace:
[  663.229147]  __schedule+0x2da/0xb00
[  663.229148]  schedule+0x38/0x90
[  663.229150]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.229152]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.229154]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.229156]  ? trace_hardirqs_on+0xd/0x10
[  663.229158]  call_rwsem_down_write_failed+0x17/0x30
[  663.229161]  down_write+0x5a/0x70
[  663.229163]  ? __generic_file_fsync+0x43/0x90
[  663.229164]  __generic_file_fsync+0x43/0x90
[  663.229166]  ext4_sync_file+0x2b4/0x540
[  663.229167]  vfs_fsync_range+0x46/0xa0
[  663.229169]  dio_complete+0x181/0x1b0
[  663.229170]  dio_aio_complete_work+0x17/0x20
[  663.229172]  process_one_work+0x208/0x6a0
[  663.229173]  ? process_one_work+0x18d/0x6a0
[  663.229175]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.229176]  worker_thread+0x49/0x4a0
[  663.229178]  kthread+0x107/0x140
[  663.229179]  ? process_one_work+0x6a0/0x6a0
[  663.229181]  ? kthread_create_on_node+0x40/0x40
[  663.229182]  ret_from_fork+0x2e/0x40
[  663.229184] kworker/10:70   D    0  9362      2 0x00000000
[  663.229187] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.229188] Call Trace:
[  663.229190]  __schedule+0x2da/0xb00
[  663.229192]  schedule+0x38/0x90
[  663.229194]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.229197]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.229199]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.229200]  ? trace_hardirqs_on+0xd/0x10
[  663.229202]  call_rwsem_down_write_failed+0x17/0x30
[  663.229205]  down_write+0x5a/0x70
[  663.229206]  ? __generic_file_fsync+0x43/0x90
[  663.229208]  __generic_file_fsync+0x43/0x90
[  663.229210]  ext4_sync_file+0x2b4/0x540
[  663.229211]  vfs_fsync_range+0x46/0xa0
[  663.229213]  dio_complete+0x181/0x1b0
[  663.229214]  dio_aio_complete_work+0x17/0x20
[  663.229215]  process_one_work+0x208/0x6a0
[  663.229216]  ? process_one_work+0x18d/0x6a0
[  663.229218]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.229219]  worker_thread+0x49/0x4a0
[  663.229221]  kthread+0x107/0x140
[  663.229223]  ? process_one_work+0x6a0/0x6a0
[  663.229224]  ? kthread_create_on_node+0x40/0x40
[  663.229225]  ret_from_fork+0x2e/0x40
[  663.229227] kworker/10:71   D    0  9363      2 0x00000000
[  663.229230] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.229231] Call Trace:
[  663.229233]  __schedule+0x2da/0xb00
[  663.229235]  schedule+0x38/0x90
[  663.229237]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.229239]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.229241]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.229243]  ? trace_hardirqs_on+0xd/0x10
[  663.229245]  call_rwsem_down_write_failed+0x17/0x30
[  663.229247]  down_write+0x5a/0x70
[  663.229248]  ? __generic_file_fsync+0x43/0x90
[  663.229250]  __generic_file_fsync+0x43/0x90
[  663.229251]  ext4_sync_file+0x2b4/0x540
[  663.229253]  vfs_fsync_range+0x46/0xa0
[  663.229254]  dio_complete+0x181/0x1b0
[  663.229256]  dio_aio_complete_work+0x17/0x20
[  663.229257]  process_one_work+0x208/0x6a0
[  663.229259]  ? process_one_work+0x18d/0x6a0
[  663.229260]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.229262]  worker_thread+0x49/0x4a0
[  663.229264]  kthread+0x107/0x140
[  663.229265]  ? process_one_work+0x6a0/0x6a0
[  663.229266]  ? kthread_create_on_node+0x40/0x40
[  663.229268]  ret_from_fork+0x2e/0x40
[  663.229269] kworker/10:72   D    0  9364      2 0x00000000
[  663.229272] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.229273] Call Trace:
[  663.229275]  __schedule+0x2da/0xb00
[  663.229277]  schedule+0x38/0x90
[  663.229279]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.229281]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.229283]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.229285]  ? trace_hardirqs_on+0xd/0x10
[  663.229287]  call_rwsem_down_write_failed+0x17/0x30
[  663.229289]  down_write+0x5a/0x70
[  663.229291]  ? __generic_file_fsync+0x43/0x90
[  663.229292]  __generic_file_fsync+0x43/0x90
[  663.229294]  ext4_sync_file+0x2b4/0x540
[  663.229296]  vfs_fsync_range+0x46/0xa0
[  663.229297]  dio_complete+0x181/0x1b0
[  663.229298]  dio_aio_complete_work+0x17/0x20
[  663.229299]  process_one_work+0x208/0x6a0
[  663.229301]  ? process_one_work+0x18d/0x6a0
[  663.229303]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.229304]  worker_thread+0x49/0x4a0
[  663.229306]  kthread+0x107/0x140
[  663.229308]  ? process_one_work+0x6a0/0x6a0
[  663.229310]  ? kthread_create_on_node+0x40/0x40
[  663.229311]  ret_from_fork+0x2e/0x40
[  663.229313] kworker/10:73   D    0  9365      2 0x00000000
[  663.229316] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.229317] Call Trace:
[  663.229319]  __schedule+0x2da/0xb00
[  663.229321]  schedule+0x38/0x90
[  663.229323]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.229325]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.229327]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.229328]  ? trace_hardirqs_on+0xd/0x10
[  663.229330]  call_rwsem_down_write_failed+0x17/0x30
[  663.229333]  down_write+0x5a/0x70
[  663.229334]  ? __generic_file_fsync+0x43/0x90
[  663.229336]  __generic_file_fsync+0x43/0x90
[  663.229338]  ext4_sync_file+0x2b4/0x540
[  663.229340]  vfs_fsync_range+0x46/0xa0
[  663.229341]  dio_complete+0x181/0x1b0
[  663.229342]  dio_aio_complete_work+0x17/0x20
[  663.229343]  process_one_work+0x208/0x6a0
[  663.229345]  ? process_one_work+0x18d/0x6a0
[  663.229347]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.229348]  worker_thread+0x49/0x4a0
[  663.229350]  kthread+0x107/0x140
[  663.229353]  ? process_one_work+0x6a0/0x6a0
[  663.229354]  ? kthread_create_on_node+0x40/0x40
[  663.229356]  ret_from_fork+0x2e/0x40
[  663.229357] kworker/10:74   D    0  9366      2 0x00000000
[  663.229360] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.229361] Call Trace:
[  663.229363]  __schedule+0x2da/0xb00
[  663.229365]  schedule+0x38/0x90
[  663.229366]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.229369]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.229371]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.229373]  call_rwsem_down_write_failed+0x17/0x30
[  663.229375]  down_write+0x5a/0x70
[  663.229377]  ? __generic_file_fsync+0x43/0x90
[  663.229378]  __generic_file_fsync+0x43/0x90
[  663.229380]  ext4_sync_file+0x2b4/0x540
[  663.229381]  vfs_fsync_range+0x46/0xa0
[  663.229383]  dio_complete+0x181/0x1b0
[  663.229384]  dio_aio_complete_work+0x17/0x20
[  663.229385]  process_one_work+0x208/0x6a0
[  663.229387]  ? process_one_work+0x18d/0x6a0
[  663.229388]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.229390]  worker_thread+0x49/0x4a0
[  663.229392]  kthread+0x107/0x140
[  663.229393]  ? process_one_work+0x6a0/0x6a0
[  663.229395]  ? kthread_create_on_node+0x40/0x40
[  663.229396]  ret_from_fork+0x2e/0x40
[  663.229398] kworker/10:75   D    0  9367      2 0x00000000
[  663.229401] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.229403] Call Trace:
[  663.229405]  __schedule+0x2da/0xb00
[  663.229407]  schedule+0x38/0x90
[  663.229409]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.229411]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.229413]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.229415]  call_rwsem_down_write_failed+0x17/0x30
[  663.229417]  down_write+0x5a/0x70
[  663.229419]  ? __generic_file_fsync+0x43/0x90
[  663.229421]  __generic_file_fsync+0x43/0x90
[  663.229423]  ext4_sync_file+0x2b4/0x540
[  663.229424]  vfs_fsync_range+0x46/0xa0
[  663.229425]  dio_complete+0x181/0x1b0
[  663.229427]  dio_aio_complete_work+0x17/0x20
[  663.229428]  process_one_work+0x208/0x6a0
[  663.229429]  ? process_one_work+0x18d/0x6a0
[  663.229431]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.229433]  worker_thread+0x49/0x4a0
[  663.229435]  kthread+0x107/0x140
[  663.229436]  ? process_one_work+0x6a0/0x6a0
[  663.229438]  ? kthread_create_on_node+0x40/0x40
[  663.229439]  ret_from_fork+0x2e/0x40
[  663.229441] kworker/10:76   D    0  9368      2 0x00000000
[  663.229443] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.229444] Call Trace:
[  663.229446]  __schedule+0x2da/0xb00
[  663.229448]  schedule+0x38/0x90
[  663.229450]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.229453]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.229455]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.229457]  call_rwsem_down_write_failed+0x17/0x30
[  663.229459]  down_write+0x5a/0x70
[  663.229461]  ? __generic_file_fsync+0x43/0x90
[  663.229462]  __generic_file_fsync+0x43/0x90
[  663.229464]  ext4_sync_file+0x2b4/0x540
[  663.229465]  vfs_fsync_range+0x46/0xa0
[  663.229467]  dio_complete+0x181/0x1b0
[  663.229468]  dio_aio_complete_work+0x17/0x20
[  663.229470]  process_one_work+0x208/0x6a0
[  663.229471]  ? process_one_work+0x18d/0x6a0
[  663.229473]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.229474]  worker_thread+0x49/0x4a0
[  663.229476]  kthread+0x107/0x140
[  663.229477]  ? process_one_work+0x6a0/0x6a0
[  663.229479]  ? kthread_create_on_node+0x40/0x40
[  663.229481]  ret_from_fork+0x2e/0x40
[  663.229482] kworker/10:77   D    0  9369      2 0x00000000
[  663.229485] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.229486] Call Trace:
[  663.229488]  __schedule+0x2da/0xb00
[  663.229490]  schedule+0x38/0x90
[  663.229491]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.229493]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.229495]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.229498]  call_rwsem_down_write_failed+0x17/0x30
[  663.229499]  down_write+0x5a/0x70
[  663.229502]  ? __generic_file_fsync+0x43/0x90
[  663.229503]  __generic_file_fsync+0x43/0x90
[  663.229505]  ext4_sync_file+0x2b4/0x540
[  663.229506]  vfs_fsync_range+0x46/0xa0
[  663.229507]  dio_complete+0x181/0x1b0
[  663.229509]  dio_aio_complete_work+0x17/0x20
[  663.229510]  process_one_work+0x208/0x6a0
[  663.229511]  ? process_one_work+0x18d/0x6a0
[  663.229513]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.229514]  worker_thread+0x49/0x4a0
[  663.229516]  kthread+0x107/0x140
[  663.229517]  ? process_one_work+0x6a0/0x6a0
[  663.229519]  ? kthread_create_on_node+0x40/0x40
[  663.229520]  ret_from_fork+0x2e/0x40
[  663.229522] kworker/10:78   D    0  9370      2 0x00000000
[  663.229524] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.229525] Call Trace:
[  663.229527]  __schedule+0x2da/0xb00
[  663.229529]  schedule+0x38/0x90
[  663.229530]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.229532]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.229535]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.229536]  ? trace_hardirqs_on_caller+0xf9/0x1b0
[  663.229539]  call_rwsem_down_write_failed+0x17/0x30
[  663.229541]  down_write+0x5a/0x70
[  663.229543]  ? __generic_file_fsync+0x43/0x90
[  663.229545]  __generic_file_fsync+0x43/0x90
[  663.229546]  ext4_sync_file+0x2b4/0x540
[  663.229548]  vfs_fsync_range+0x46/0xa0
[  663.229549]  dio_complete+0x181/0x1b0
[  663.229551]  dio_aio_complete_work+0x17/0x20
[  663.229552]  process_one_work+0x208/0x6a0
[  663.229553]  ? process_one_work+0x18d/0x6a0
[  663.229555]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.229556]  worker_thread+0x49/0x4a0
[  663.229558]  kthread+0x107/0x140
[  663.229560]  ? process_one_work+0x6a0/0x6a0
[  663.229561]  ? kthread_create_on_node+0x40/0x40
[  663.229563]  ret_from_fork+0x2e/0x40
[  663.229565] kworker/10:79   D    0  9371      2 0x00000000
[  663.229568] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.229569] Call Trace:
[  663.229571]  __schedule+0x2da/0xb00
[  663.229573]  schedule+0x38/0x90
[  663.229575]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.229577]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.229579]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.229582]  call_rwsem_down_write_failed+0x17/0x30
[  663.229584]  down_write+0x5a/0x70
[  663.229586]  ? __generic_file_fsync+0x43/0x90
[  663.229587]  __generic_file_fsync+0x43/0x90
[  663.229589]  ext4_sync_file+0x2b4/0x540
[  663.229591]  vfs_fsync_range+0x46/0xa0
[  663.229592]  dio_complete+0x181/0x1b0
[  663.229593]  dio_aio_complete_work+0x17/0x20
[  663.229595]  process_one_work+0x208/0x6a0
[  663.229596]  ? process_one_work+0x18d/0x6a0
[  663.229597]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.229599]  worker_thread+0x49/0x4a0
[  663.229601]  kthread+0x107/0x140
[  663.229602]  ? process_one_work+0x6a0/0x6a0
[  663.229604]  ? kthread_create_on_node+0x40/0x40
[  663.229605]  ret_from_fork+0x2e/0x40
[  663.229607] kworker/10:80   D    0  9372      2 0x00000000
[  663.229609] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.229610] Call Trace:
[  663.229612]  __schedule+0x2da/0xb00
[  663.229614]  schedule+0x38/0x90
[  663.229616]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.229618]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.229620]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.229622]  call_rwsem_down_write_failed+0x17/0x30
[  663.229624]  down_write+0x5a/0x70
[  663.229626]  ? __generic_file_fsync+0x43/0x90
[  663.229628]  __generic_file_fsync+0x43/0x90
[  663.229629]  ext4_sync_file+0x2b4/0x540
[  663.229631]  vfs_fsync_range+0x46/0xa0
[  663.229633]  dio_complete+0x181/0x1b0
[  663.229634]  dio_aio_complete_work+0x17/0x20
[  663.229635]  process_one_work+0x208/0x6a0
[  663.229636]  ? process_one_work+0x18d/0x6a0
[  663.229638]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.229639]  worker_thread+0x49/0x4a0
[  663.229641]  kthread+0x107/0x140
[  663.229643]  ? process_one_work+0x6a0/0x6a0
[  663.229644]  ? kthread_create_on_node+0x40/0x40
[  663.229646]  ret_from_fork+0x2e/0x40
[  663.229648] kworker/10:81   D    0  9373      2 0x00000000
[  663.229650] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.229651] Call Trace:
[  663.229653]  __schedule+0x2da/0xb00
[  663.229655]  schedule+0x38/0x90
[  663.229657]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.229659]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.229661]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.229664]  call_rwsem_down_write_failed+0x17/0x30
[  663.229666]  down_write+0x5a/0x70
[  663.229669]  ? __generic_file_fsync+0x43/0x90
[  663.229671]  __generic_file_fsync+0x43/0x90
[  663.229672]  ext4_sync_file+0x2b4/0x540
[  663.229674]  vfs_fsync_range+0x46/0xa0
[  663.229675]  dio_complete+0x181/0x1b0
[  663.229677]  dio_aio_complete_work+0x17/0x20
[  663.229678]  process_one_work+0x208/0x6a0
[  663.229679]  ? process_one_work+0x18d/0x6a0
[  663.229681]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.229683]  worker_thread+0x49/0x4a0
[  663.229685]  kthread+0x107/0x140
[  663.229686]  ? process_one_work+0x6a0/0x6a0
[  663.229688]  ? kthread_create_on_node+0x40/0x40
[  663.229689]  ret_from_fork+0x2e/0x40
[  663.229691] kworker/10:82   D    0  9374      2 0x00000000
[  663.229693] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.229694] Call Trace:
[  663.229697]  __schedule+0x2da/0xb00
[  663.229699]  schedule+0x38/0x90
[  663.229701]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.229703]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.229705]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.229708]  call_rwsem_down_write_failed+0x17/0x30
[  663.229710]  down_write+0x5a/0x70
[  663.229711]  ? __generic_file_fsync+0x43/0x90
[  663.229713]  __generic_file_fsync+0x43/0x90
[  663.229715]  ext4_sync_file+0x2b4/0x540
[  663.229716]  vfs_fsync_range+0x46/0xa0
[  663.229717]  dio_complete+0x181/0x1b0
[  663.229719]  dio_aio_complete_work+0x17/0x20
[  663.229720]  process_one_work+0x208/0x6a0
[  663.229721]  ? process_one_work+0x18d/0x6a0
[  663.229723]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.229724]  worker_thread+0x49/0x4a0
[  663.229726]  kthread+0x107/0x140
[  663.229727]  ? process_one_work+0x6a0/0x6a0
[  663.229729]  ? kthread_create_on_node+0x40/0x40
[  663.229731]  ret_from_fork+0x2e/0x40
[  663.229732] kworker/10:83   D    0  9375      2 0x00000000
[  663.229735] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.229736] Call Trace:
[  663.229738]  __schedule+0x2da/0xb00
[  663.229739]  schedule+0x38/0x90
[  663.229741]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.229743]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.229745]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.229747]  call_rwsem_down_write_failed+0x17/0x30
[  663.229749]  down_write+0x5a/0x70
[  663.229751]  ? __generic_file_fsync+0x43/0x90
[  663.229752]  __generic_file_fsync+0x43/0x90
[  663.229754]  ext4_sync_file+0x2b4/0x540
[  663.229755]  vfs_fsync_range+0x46/0xa0
[  663.229757]  dio_complete+0x181/0x1b0
[  663.229759]  dio_aio_complete_work+0x17/0x20
[  663.229760]  process_one_work+0x208/0x6a0
[  663.229762]  ? process_one_work+0x18d/0x6a0
[  663.229763]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.229765]  worker_thread+0x49/0x4a0
[  663.229767]  kthread+0x107/0x140
[  663.229768]  ? process_one_work+0x6a0/0x6a0
[  663.229770]  ? kthread_create_on_node+0x40/0x40
[  663.229771]  ret_from_fork+0x2e/0x40
[  663.229773] kworker/10:84   D    0  9376      2 0x00000000
[  663.229775] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.229777] Call Trace:
[  663.229779]  __schedule+0x2da/0xb00
[  663.229780]  schedule+0x38/0x90
[  663.229783]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.229785]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.229787]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.229789]  call_rwsem_down_write_failed+0x17/0x30
[  663.229791]  down_write+0x5a/0x70
[  663.229793]  ? __generic_file_fsync+0x43/0x90
[  663.229795]  __generic_file_fsync+0x43/0x90
[  663.229797]  ext4_sync_file+0x2b4/0x540
[  663.229798]  vfs_fsync_range+0x46/0xa0
[  663.229800]  dio_complete+0x181/0x1b0
[  663.229801]  dio_aio_complete_work+0x17/0x20
[  663.229803]  process_one_work+0x208/0x6a0
[  663.229804]  ? process_one_work+0x18d/0x6a0
[  663.229806]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.229807]  worker_thread+0x49/0x4a0
[  663.229809]  kthread+0x107/0x140
[  663.229810]  ? process_one_work+0x6a0/0x6a0
[  663.229812]  ? kthread_create_on_node+0x40/0x40
[  663.229814]  ret_from_fork+0x2e/0x40
[  663.229815] kworker/10:85   D    0  9377      2 0x00000000
[  663.229818] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.229819] Call Trace:
[  663.229821]  __schedule+0x2da/0xb00
[  663.229823]  schedule+0x38/0x90
[  663.229825]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.229828]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.229830]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.229832]  call_rwsem_down_write_failed+0x17/0x30
[  663.229834]  down_write+0x5a/0x70
[  663.229835]  ? __generic_file_fsync+0x43/0x90
[  663.229837]  __generic_file_fsync+0x43/0x90
[  663.229838]  ext4_sync_file+0x2b4/0x540
[  663.229840]  vfs_fsync_range+0x46/0xa0
[  663.229841]  dio_complete+0x181/0x1b0
[  663.229842]  dio_aio_complete_work+0x17/0x20
[  663.229844]  process_one_work+0x208/0x6a0
[  663.229845]  ? process_one_work+0x18d/0x6a0
[  663.229847]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.229849]  worker_thread+0x49/0x4a0
[  663.229850]  kthread+0x107/0x140
[  663.229852]  ? process_one_work+0x6a0/0x6a0
[  663.229853]  ? kthread_create_on_node+0x40/0x40
[  663.229854]  ret_from_fork+0x2e/0x40
[  663.229856] kworker/10:86   D    0  9378      2 0x00000000
[  663.229859] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.229860] Call Trace:
[  663.229862]  __schedule+0x2da/0xb00
[  663.229864]  schedule+0x38/0x90
[  663.229866]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.229868]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.229870]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.229873]  call_rwsem_down_write_failed+0x17/0x30
[  663.229875]  down_write+0x5a/0x70
[  663.229876]  ? __generic_file_fsync+0x43/0x90
[  663.229878]  __generic_file_fsync+0x43/0x90
[  663.229880]  ext4_sync_file+0x2b4/0x540
[  663.229881]  vfs_fsync_range+0x46/0xa0
[  663.229883]  dio_complete+0x181/0x1b0
[  663.229884]  dio_aio_complete_work+0x17/0x20
[  663.229885]  process_one_work+0x208/0x6a0
[  663.229887]  ? process_one_work+0x18d/0x6a0
[  663.229888]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.229890]  worker_thread+0x49/0x4a0
[  663.229893]  kthread+0x107/0x140
[  663.229894]  ? process_one_work+0x6a0/0x6a0
[  663.229896]  ? kthread_create_on_node+0x40/0x40
[  663.229897]  ret_from_fork+0x2e/0x40
[  663.229899] kworker/10:87   D    0  9379      2 0x00000000
[  663.229902] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.229903] Call Trace:
[  663.229905]  __schedule+0x2da/0xb00
[  663.229906]  schedule+0x38/0x90
[  663.229908]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.229910]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.229912]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.229915]  call_rwsem_down_write_failed+0x17/0x30
[  663.229917]  down_write+0x5a/0x70
[  663.229919]  ? __generic_file_fsync+0x43/0x90
[  663.229920]  __generic_file_fsync+0x43/0x90
[  663.229922]  ext4_sync_file+0x2b4/0x540
[  663.229924]  vfs_fsync_range+0x46/0xa0
[  663.229926]  dio_complete+0x181/0x1b0
[  663.229927]  dio_aio_complete_work+0x17/0x20
[  663.229928]  process_one_work+0x208/0x6a0
[  663.229929]  ? process_one_work+0x18d/0x6a0
[  663.229931]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.229932]  worker_thread+0x49/0x4a0
[  663.229934]  kthread+0x107/0x140
[  663.229936]  ? process_one_work+0x6a0/0x6a0
[  663.229938]  ? kthread_create_on_node+0x40/0x40
[  663.229939]  ret_from_fork+0x2e/0x40
[  663.229941] kworker/10:88   D    0  9380      2 0x00000000
[  663.229943] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.229944] Call Trace:
[  663.229946]  __schedule+0x2da/0xb00
[  663.229948]  schedule+0x38/0x90
[  663.229950]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.229952]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.229954]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.229956]  call_rwsem_down_write_failed+0x17/0x30
[  663.229958]  down_write+0x5a/0x70
[  663.229960]  ? __generic_file_fsync+0x43/0x90
[  663.229962]  __generic_file_fsync+0x43/0x90
[  663.229963]  ext4_sync_file+0x2b4/0x540
[  663.229965]  vfs_fsync_range+0x46/0xa0
[  663.229966]  dio_complete+0x181/0x1b0
[  663.229967]  dio_aio_complete_work+0x17/0x20
[  663.229968]  process_one_work+0x208/0x6a0
[  663.229970]  ? process_one_work+0x18d/0x6a0
[  663.229971]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.229973]  worker_thread+0x49/0x4a0
[  663.229975]  kthread+0x107/0x140
[  663.229976]  ? process_one_work+0x6a0/0x6a0
[  663.229978]  ? kthread_create_on_node+0x40/0x40
[  663.229979]  ret_from_fork+0x2e/0x40
[  663.229981] kworker/10:89   D    0  9381      2 0x00000000
[  663.229983] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.229984] Call Trace:
[  663.229986]  __schedule+0x2da/0xb00
[  663.229988]  schedule+0x38/0x90
[  663.229990]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.229992]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.229994]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.229997]  call_rwsem_down_write_failed+0x17/0x30
[  663.229999]  down_write+0x5a/0x70
[  663.230001]  ? __generic_file_fsync+0x43/0x90
[  663.230002]  __generic_file_fsync+0x43/0x90
[  663.230004]  ext4_sync_file+0x2b4/0x540
[  663.230006]  vfs_fsync_range+0x46/0xa0
[  663.230007]  dio_complete+0x181/0x1b0
[  663.230008]  dio_aio_complete_work+0x17/0x20
[  663.230010]  process_one_work+0x208/0x6a0
[  663.230011]  ? process_one_work+0x18d/0x6a0
[  663.230013]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.230015]  worker_thread+0x49/0x4a0
[  663.230016]  kthread+0x107/0x140
[  663.230018]  ? process_one_work+0x6a0/0x6a0
[  663.230019]  ? kthread_create_on_node+0x40/0x40
[  663.230021]  ret_from_fork+0x2e/0x40
[  663.230023] kworker/10:90   D    0  9382      2 0x00000000
[  663.230026] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.230027] Call Trace:
[  663.230029]  __schedule+0x2da/0xb00
[  663.230031]  schedule+0x38/0x90
[  663.230033]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.230035]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.230037]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.230039]  call_rwsem_down_write_failed+0x17/0x30
[  663.230041]  down_write+0x5a/0x70
[  663.230042]  ? __generic_file_fsync+0x43/0x90
[  663.230044]  __generic_file_fsync+0x43/0x90
[  663.230046]  ext4_sync_file+0x2b4/0x540
[  663.230047]  vfs_fsync_range+0x46/0xa0
[  663.230049]  dio_complete+0x181/0x1b0
[  663.230050]  dio_aio_complete_work+0x17/0x20
[  663.230051]  process_one_work+0x208/0x6a0
[  663.230053]  ? process_one_work+0x18d/0x6a0
[  663.230054]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.230056]  worker_thread+0x49/0x4a0
[  663.230058]  kthread+0x107/0x140
[  663.230059]  ? process_one_work+0x6a0/0x6a0
[  663.230060]  ? kthread_create_on_node+0x40/0x40
[  663.230062]  ret_from_fork+0x2e/0x40
[  663.230063] kworker/10:91   D    0  9383      2 0x00000000
[  663.230066] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.230067] Call Trace:
[  663.230069]  __schedule+0x2da/0xb00
[  663.230070]  schedule+0x38/0x90
[  663.230072]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.230074]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.230076]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.230079]  call_rwsem_down_write_failed+0x17/0x30
[  663.230081]  down_write+0x5a/0x70
[  663.230083]  ? __generic_file_fsync+0x43/0x90
[  663.230084]  __generic_file_fsync+0x43/0x90
[  663.230086]  ext4_sync_file+0x2b4/0x540
[  663.230087]  vfs_fsync_range+0x46/0xa0
[  663.230089]  dio_complete+0x181/0x1b0
[  663.230090]  dio_aio_complete_work+0x17/0x20
[  663.230092]  process_one_work+0x208/0x6a0
[  663.230093]  ? process_one_work+0x18d/0x6a0
[  663.230095]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.230096]  worker_thread+0x49/0x4a0
[  663.230098]  kthread+0x107/0x140
[  663.230099]  ? process_one_work+0x6a0/0x6a0
[  663.230101]  ? kthread_create_on_node+0x40/0x40
[  663.230102]  ret_from_fork+0x2e/0x40
[  663.230104] kworker/10:92   D    0  9384      2 0x00000000
[  663.230107] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.230108] Call Trace:
[  663.230110]  __schedule+0x2da/0xb00
[  663.230112]  schedule+0x38/0x90
[  663.230114]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.230116]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.230118]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.230121]  call_rwsem_down_write_failed+0x17/0x30
[  663.230123]  down_write+0x5a/0x70
[  663.230125]  ? __generic_file_fsync+0x43/0x90
[  663.230126]  __generic_file_fsync+0x43/0x90
[  663.230128]  ext4_sync_file+0x2b4/0x540
[  663.230130]  vfs_fsync_range+0x46/0xa0
[  663.230131]  dio_complete+0x181/0x1b0
[  663.230132]  dio_aio_complete_work+0x17/0x20
[  663.230134]  process_one_work+0x208/0x6a0
[  663.230135]  ? process_one_work+0x18d/0x6a0
[  663.230137]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.230138]  worker_thread+0x49/0x4a0
[  663.230140]  kthread+0x107/0x140
[  663.230142]  ? process_one_work+0x6a0/0x6a0
[  663.230143]  ? kthread_create_on_node+0x40/0x40
[  663.230145]  ret_from_fork+0x2e/0x40
[  663.230146] kworker/10:93   D    0  9385      2 0x00000000
[  663.230149] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.230150] Call Trace:
[  663.230152]  __schedule+0x2da/0xb00
[  663.230153]  schedule+0x38/0x90
[  663.230155]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.230157]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.230159]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.230161]  call_rwsem_down_write_failed+0x17/0x30
[  663.230163]  down_write+0x5a/0x70
[  663.230165]  ? __generic_file_fsync+0x43/0x90
[  663.230166]  __generic_file_fsync+0x43/0x90
[  663.230168]  ext4_sync_file+0x2b4/0x540
[  663.230169]  vfs_fsync_range+0x46/0xa0
[  663.230171]  dio_complete+0x181/0x1b0
[  663.230172]  dio_aio_complete_work+0x17/0x20
[  663.230173]  process_one_work+0x208/0x6a0
[  663.230174]  ? process_one_work+0x18d/0x6a0
[  663.230176]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.230177]  worker_thread+0x49/0x4a0
[  663.230179]  kthread+0x107/0x140
[  663.230180]  ? process_one_work+0x6a0/0x6a0
[  663.230182]  ? kthread_create_on_node+0x40/0x40
[  663.230183]  ret_from_fork+0x2e/0x40
[  663.230185] kworker/10:94   D    0  9386      2 0x00000000
[  663.230187] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.230188] Call Trace:
[  663.230190]  __schedule+0x2da/0xb00
[  663.230192]  schedule+0x38/0x90
[  663.230194]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.230195]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.230197]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.230199]  ? trace_hardirqs_on+0xd/0x10
[  663.230201]  call_rwsem_down_write_failed+0x17/0x30
[  663.230203]  down_write+0x5a/0x70
[  663.230205]  ? __generic_file_fsync+0x43/0x90
[  663.230206]  __generic_file_fsync+0x43/0x90
[  663.230208]  ext4_sync_file+0x2b4/0x540
[  663.230209]  vfs_fsync_range+0x46/0xa0
[  663.230211]  dio_complete+0x181/0x1b0
[  663.230212]  dio_aio_complete_work+0x17/0x20
[  663.230214]  process_one_work+0x208/0x6a0
[  663.230215]  ? process_one_work+0x18d/0x6a0
[  663.230217]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.230218]  worker_thread+0x49/0x4a0
[  663.230220]  kthread+0x107/0x140
[  663.230221]  ? process_one_work+0x6a0/0x6a0
[  663.230223]  ? kthread_create_on_node+0x40/0x40
[  663.230224]  ret_from_fork+0x2e/0x40
[  663.230226] kworker/10:95   D    0  9387      2 0x00000000
[  663.230230] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.230231] Call Trace:
[  663.230233]  __schedule+0x2da/0xb00
[  663.230235]  schedule+0x38/0x90
[  663.230237]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.230239]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.230241]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.230243]  call_rwsem_down_write_failed+0x17/0x30
[  663.230246]  down_write+0x5a/0x70
[  663.230247]  ? __generic_file_fsync+0x43/0x90
[  663.230249]  __generic_file_fsync+0x43/0x90
[  663.230251]  ext4_sync_file+0x2b4/0x540
[  663.230252]  vfs_fsync_range+0x46/0xa0
[  663.230254]  dio_complete+0x181/0x1b0
[  663.230255]  dio_aio_complete_work+0x17/0x20
[  663.230256]  process_one_work+0x208/0x6a0
[  663.230258]  ? process_one_work+0x18d/0x6a0
[  663.230259]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.230261]  worker_thread+0x49/0x4a0
[  663.230262]  kthread+0x107/0x140
[  663.230264]  ? process_one_work+0x6a0/0x6a0
[  663.230265]  ? kthread_create_on_node+0x40/0x40
[  663.230267]  ret_from_fork+0x2e/0x40
[  663.230268] kworker/10:96   D    0  9388      2 0x00000000
[  663.230271] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.230272] Call Trace:
[  663.230274]  __schedule+0x2da/0xb00
[  663.230276]  schedule+0x38/0x90
[  663.230278]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.230280]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.230282]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.230285]  call_rwsem_down_write_failed+0x17/0x30
[  663.230287]  down_write+0x5a/0x70
[  663.230288]  ? __generic_file_fsync+0x43/0x90
[  663.230290]  __generic_file_fsync+0x43/0x90
[  663.230291]  ext4_sync_file+0x2b4/0x540
[  663.230293]  vfs_fsync_range+0x46/0xa0
[  663.230294]  dio_complete+0x181/0x1b0
[  663.230295]  dio_aio_complete_work+0x17/0x20
[  663.230296]  process_one_work+0x208/0x6a0
[  663.230298]  ? process_one_work+0x18d/0x6a0
[  663.230299]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.230301]  worker_thread+0x49/0x4a0
[  663.230303]  kthread+0x107/0x140
[  663.230304]  ? process_one_work+0x6a0/0x6a0
[  663.230305]  ? kthread_create_on_node+0x40/0x40
[  663.230307]  ret_from_fork+0x2e/0x40
[  663.230308] kworker/10:97   D    0  9389      2 0x00000000
[  663.230311] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.230312] Call Trace:
[  663.230314]  __schedule+0x2da/0xb00
[  663.230316]  schedule+0x38/0x90
[  663.230317]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.230319]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.230321]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.230324]  call_rwsem_down_write_failed+0x17/0x30
[  663.230326]  down_write+0x5a/0x70
[  663.230327]  ? __generic_file_fsync+0x43/0x90
[  663.230329]  __generic_file_fsync+0x43/0x90
[  663.230331]  ext4_sync_file+0x2b4/0x540
[  663.230333]  vfs_fsync_range+0x46/0xa0
[  663.230335]  dio_complete+0x181/0x1b0
[  663.230336]  dio_aio_complete_work+0x17/0x20
[  663.230337]  process_one_work+0x208/0x6a0
[  663.230338]  ? process_one_work+0x18d/0x6a0
[  663.230340]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.230342]  worker_thread+0x49/0x4a0
[  663.230344]  kthread+0x107/0x140
[  663.230345]  ? process_one_work+0x6a0/0x6a0
[  663.230347]  ? kthread_create_on_node+0x40/0x40
[  663.230348]  ret_from_fork+0x2e/0x40
[  663.230350] kworker/10:98   D    0  9390      2 0x00000000
[  663.230353] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.230354] Call Trace:
[  663.230356]  __schedule+0x2da/0xb00
[  663.230358]  schedule+0x38/0x90
[  663.230360]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.230362]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.230364]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.230367]  call_rwsem_down_write_failed+0x17/0x30
[  663.230369]  down_write+0x5a/0x70
[  663.230371]  ? __generic_file_fsync+0x43/0x90
[  663.230372]  __generic_file_fsync+0x43/0x90
[  663.230374]  ext4_sync_file+0x2b4/0x540
[  663.230376]  vfs_fsync_range+0x46/0xa0
[  663.230377]  dio_complete+0x181/0x1b0
[  663.230378]  dio_aio_complete_work+0x17/0x20
[  663.230380]  process_one_work+0x208/0x6a0
[  663.230381]  ? process_one_work+0x18d/0x6a0
[  663.230383]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.230384]  worker_thread+0x49/0x4a0
[  663.230386]  kthread+0x107/0x140
[  663.230387]  ? process_one_work+0x6a0/0x6a0
[  663.230390]  ? kthread_create_on_node+0x40/0x40
[  663.230391]  ret_from_fork+0x2e/0x40
[  663.230393] kworker/10:99   D    0  9391      2 0x00000000
[  663.230396] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.230397] Call Trace:
[  663.230399]  __schedule+0x2da/0xb00
[  663.230401]  schedule+0x38/0x90
[  663.230403]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.230405]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.230407]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.230409]  call_rwsem_down_write_failed+0x17/0x30
[  663.230411]  down_write+0x5a/0x70
[  663.230413]  ? __generic_file_fsync+0x43/0x90
[  663.230414]  __generic_file_fsync+0x43/0x90
[  663.230416]  ext4_sync_file+0x2b4/0x540
[  663.230417]  vfs_fsync_range+0x46/0xa0
[  663.230419]  dio_complete+0x181/0x1b0
[  663.230420]  dio_aio_complete_work+0x17/0x20
[  663.230421]  process_one_work+0x208/0x6a0
[  663.230422]  ? process_one_work+0x18d/0x6a0
[  663.230424]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.230425]  worker_thread+0x49/0x4a0
[  663.230427]  kthread+0x107/0x140
[  663.230428]  ? process_one_work+0x6a0/0x6a0
[  663.230431]  ? kthread_create_on_node+0x40/0x40
[  663.230432]  ret_from_fork+0x2e/0x40
[  663.230434] kworker/10:100  D    0  9392      2 0x00000000
[  663.230436] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.230437] Call Trace:
[  663.230440]  __schedule+0x2da/0xb00
[  663.230442]  schedule+0x38/0x90
[  663.230445]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.230447]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.230448]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.230451]  call_rwsem_down_write_failed+0x17/0x30
[  663.230453]  down_write+0x5a/0x70
[  663.230455]  ? __generic_file_fsync+0x43/0x90
[  663.230456]  __generic_file_fsync+0x43/0x90
[  663.230458]  ext4_sync_file+0x2b4/0x540
[  663.230460]  vfs_fsync_range+0x46/0xa0
[  663.230461]  dio_complete+0x181/0x1b0
[  663.230463]  dio_aio_complete_work+0x17/0x20
[  663.230464]  process_one_work+0x208/0x6a0
[  663.230465]  ? process_one_work+0x18d/0x6a0
[  663.230467]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.230469]  worker_thread+0x49/0x4a0
[  663.230471]  kthread+0x107/0x140
[  663.230473]  ? process_one_work+0x6a0/0x6a0
[  663.230474]  ? kthread_create_on_node+0x40/0x40
[  663.230476]  ret_from_fork+0x2e/0x40
[  663.230478] kworker/10:101  D    0  9393      2 0x00000000
[  663.230480] Workqueue: dio/dm-0 dio_aio_complete_work
[  663.230481] Call Trace:
[  663.230483]  __schedule+0x2da/0xb00
[  663.230485]  ? _raw_spin_unlock_irq+0x27/0x40
[  663.230487]  schedule+0x38/0x90
[  663.230489]  ? rwsem_down_write_failed+0x2ac/0x4b0
[  663.230491]  rwsem_down_write_failed+0x2b1/0x4b0
[  663.230493]  ? rwsem_down_write_failed+0x50/0x4b0
[  663.230494]  ? trace_hardirqs_on+0xd/0x10
[  663.230496]  call_rwsem_down_write_failed+0x17/0x30
[  663.230499]  down_write+0x5a/0x70
[  663.230500]  ? __generic_file_fsync+0x43/0x90
[  663.230502]  __generic_file_fsync+0x43/0x90
[  663.230504]  ext4_sync_file+0x2b4/0x540
[  663.230506]  vfs_fsync_range+0x46/0xa0
[  663.230507]  dio_complete+0x181/0x1b0
[  663.230508]  dio_aio_complete_work+0x17/0x20
[  663.230509]  process_one_work+0x208/0x6a0
[  663.230510]  ? process_one_work+0x18d/0x6a0
[  663.230512]  ? __mutex_unlock_slowpath+0x3e/0x290
[  663.230514]  worker_thread+0x49/0x4a0
[  663.230515]  kthread+0x107/0x140
[  663.230517]  ? process_one_work+0x6a0/0x6a0
[  663.230518]  ? kthread_create_on_node+0x40/0x40
[  663.230519]  ret_from_fork+0x2e/0x40--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Jens Axboe Jan. 26, 2017, 11:26 p.m. UTC | #2 
On 01/26/2017 04:14 PM, Bart Van Assche wrote:
> On Thu, 2017-01-26 at 14:51 -0700, Jens Axboe wrote:
>> That is exactly what it means, looks like that one path doesn't handle
>> that.  You'd have to exhaust the pool with atomic allocs for this to
>> trigger, we don't do that at all in the normal IO path. So good catch,
>> must be the dm part that enables this since it does NOWAIT allocations.
>>
>>
>> diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c
>> index 3136696f4991..c27613de80c5 100644
>> --- a/block/blk-mq-sched.c
>> +++ b/block/blk-mq-sched.c
>> @@ -134,7 +134,8 @@ struct request *blk_mq_sched_get_request(struct request_queue *q,
>>  			rq = __blk_mq_alloc_request(data, op);
>>  	} else {
>>  		rq = __blk_mq_alloc_request(data, op);
>> -		data->hctx->tags->rqs[rq->tag] = rq;
>> +		if (rq)
>> +			data->hctx->tags->rqs[rq->tag] = rq;
>>  	}
>>  
>>  	if (rq) {
> 
> Hello Jens,
> 
> With these two patches applied the scheduling-while-atomic complaint and
> the oops are gone. However, some tasks get stuck. Is the console output
> below enough to figure out what is going on or do you want me to bisect
> this? I don't think that any requests got stuck since no pending requests
> are shown in /sys/block/*/mq/*/{pending,*/rq_list}.

What device is stuck? Is it running with an mq scheduler attached, or
with "none"?

Would also be great to see the output of /sys/block/*/mq/*/tags and
sched_tags so we can see if they have anything pending.

From a quick look at the below, it looks like a request leak. Bisection
would most likely be very helpful.
Bart Van Assche Jan. 26, 2017, 11:47 p.m. UTC | #3 
On Thu, 2017-01-26 at 16:26 -0700, Jens Axboe wrote:
> What device is stuck? Is it running with an mq scheduler attached, or
> with "none"?
> 
> Would also be great to see the output of /sys/block/*/mq/*/tags and
> sched_tags so we can see if they have anything pending.
> 
> From a quick look at the below, it looks like a request leak. Bisection
> would most likely be very helpful.

Hello Jens,

This happens with and without scheduler attached. The most recent test I ran
was with the deadline scheduler configured as default scheduler for all blk-mq
devices (CONFIG_DEFAULT_SQ_IOSCHED="mq-deadline" and
CONFIG_DEFAULT_MQ_IOSCHED="mq-deadline"). The block devices that hang are
/dev/dm-0 and /dev/dm-1. The tags and sched_tags data is as follows:

# (cd /sys/class/block && grep -aH '' dm*/mq/*/tags)
dm-0/mq/0/tags:nr_tags=2048, reserved_tags=0, bits_per_word=64
dm-0/mq/0/tags:nr_free=1795, nr_reserved=0
dm-0/mq/0/tags:active_queues=0
dm-1/mq/0/tags:nr_tags=2048, reserved_tags=0, bits_per_word=64
dm-1/mq/0/tags:nr_free=2047, nr_reserved=0
dm-1/mq/0/tags:active_queues=0
# (cd /sys/class/block && grep -aH '' dm*/mq/*/sched_tags)
dm-0/mq/0/sched_tags:nr_tags=256, reserved_tags=0, bits_per_word=64
dm-0/mq/0/sched_tags:nr_free=0, nr_reserved=0
dm-0/mq/0/sched_tags:active_queues=0
dm-1/mq/0/sched_tags:nr_tags=256, reserved_tags=0, bits_per_word=64
dm-1/mq/0/sched_tags:nr_free=254, nr_reserved=0
dm-1/mq/0/sched_tags:active_queues=0

Bart.--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Jens Axboe Jan. 26, 2017, 11:50 p.m. UTC | #4 
On 01/26/2017 04:47 PM, Bart Van Assche wrote:
> On Thu, 2017-01-26 at 16:26 -0700, Jens Axboe wrote:
>> What device is stuck? Is it running with an mq scheduler attached, or
>> with "none"?
>>
>> Would also be great to see the output of /sys/block/*/mq/*/tags and
>> sched_tags so we can see if they have anything pending.
>>
>> From a quick look at the below, it looks like a request leak. Bisection
>> would most likely be very helpful.
> 
> Hello Jens,
> 
> This happens with and without scheduler attached. The most recent test I ran
> was with the deadline scheduler configured as default scheduler for all blk-mq
> devices (CONFIG_DEFAULT_SQ_IOSCHED="mq-deadline" and
> CONFIG_DEFAULT_MQ_IOSCHED="mq-deadline"). The block devices that hang are
> /dev/dm-0 and /dev/dm-1. The tags and sched_tags data is as follows:
> 
> # (cd /sys/class/block && grep -aH '' dm*/mq/*/tags)
> dm-0/mq/0/tags:nr_tags=2048, reserved_tags=0, bits_per_word=64
> dm-0/mq/0/tags:nr_free=1795, nr_reserved=0
> dm-0/mq/0/tags:active_queues=0
> dm-1/mq/0/tags:nr_tags=2048, reserved_tags=0, bits_per_word=64
> dm-1/mq/0/tags:nr_free=2047, nr_reserved=0
> dm-1/mq/0/tags:active_queues=0
> # (cd /sys/class/block && grep -aH '' dm*/mq/*/sched_tags)
> dm-0/mq/0/sched_tags:nr_tags=256, reserved_tags=0, bits_per_word=64
> dm-0/mq/0/sched_tags:nr_free=0, nr_reserved=0
> dm-0/mq/0/sched_tags:active_queues=0
> dm-1/mq/0/sched_tags:nr_tags=256, reserved_tags=0, bits_per_word=64
> dm-1/mq/0/sched_tags:nr_free=254, nr_reserved=0
> dm-1/mq/0/sched_tags:active_queues=0

Clearly we are missing some requests. How do I setup dm similarly to
you?

Does it reproduce without Christoph's patchset?
Jens Axboe Jan. 27, 2017, 12:33 a.m. UTC | #5 
On 01/26/2017 04:50 PM, Jens Axboe wrote:
> On 01/26/2017 04:47 PM, Bart Van Assche wrote:
>> On Thu, 2017-01-26 at 16:26 -0700, Jens Axboe wrote:
>>> What device is stuck? Is it running with an mq scheduler attached, or
>>> with "none"?
>>>
>>> Would also be great to see the output of /sys/block/*/mq/*/tags and
>>> sched_tags so we can see if they have anything pending.
>>>
>>> From a quick look at the below, it looks like a request leak. Bisection
>>> would most likely be very helpful.
>>
>> Hello Jens,
>>
>> This happens with and without scheduler attached. The most recent test I ran
>> was with the deadline scheduler configured as default scheduler for all blk-mq
>> devices (CONFIG_DEFAULT_SQ_IOSCHED="mq-deadline" and
>> CONFIG_DEFAULT_MQ_IOSCHED="mq-deadline"). The block devices that hang are
>> /dev/dm-0 and /dev/dm-1. The tags and sched_tags data is as follows:
>>
>> # (cd /sys/class/block && grep -aH '' dm*/mq/*/tags)
>> dm-0/mq/0/tags:nr_tags=2048, reserved_tags=0, bits_per_word=64
>> dm-0/mq/0/tags:nr_free=1795, nr_reserved=0
>> dm-0/mq/0/tags:active_queues=0
>> dm-1/mq/0/tags:nr_tags=2048, reserved_tags=0, bits_per_word=64
>> dm-1/mq/0/tags:nr_free=2047, nr_reserved=0
>> dm-1/mq/0/tags:active_queues=0
>> # (cd /sys/class/block && grep -aH '' dm*/mq/*/sched_tags)
>> dm-0/mq/0/sched_tags:nr_tags=256, reserved_tags=0, bits_per_word=64
>> dm-0/mq/0/sched_tags:nr_free=0, nr_reserved=0
>> dm-0/mq/0/sched_tags:active_queues=0
>> dm-1/mq/0/sched_tags:nr_tags=256, reserved_tags=0, bits_per_word=64
>> dm-1/mq/0/sched_tags:nr_free=254, nr_reserved=0
>> dm-1/mq/0/sched_tags:active_queues=0
> 
> Clearly we are missing some requests. How do I setup dm similarly to
> you?
> 
> Does it reproduce without Christoph's patchset?

I have dm-mpath running using blk_mq and with mq-deadline on both dm and
the lower level device, and it seems to be running just fine here.
Note, this is without Christoph's patchset, I'll try that next once
xfstest finishes.
Bart Van Assche Jan. 27, 2017, 12:38 a.m. UTC | #6 
On Thu, 2017-01-26 at 16:50 -0700, Jens Axboe wrote:
> Clearly we are missing some requests. How do I setup dm similarly to
> you?
> 
> Does it reproduce without Christoph's patchset?

Hello Jens,

I see similar behavior with the blk-mq-sched branch of
git://git.kernel.dk/linux-block.git (git commit ID 0efe27068ecf):
booting happens much slower than usual and I/O hangs if I run the
srp-test software.

Regarding creating a similar dm setup: I hope that in the future it
will become possible to run the srp-test software without any special
hardware and with in-tree drivers. Today running the srp-test software
with in-tree drivers namely requires IB hardware. This is how to run the
srp-test software today with in-tree drivers:
* Find a system with at least two InfiniBand ports.
* Make sure that the appropriate IB driver in the kernel is enabled and
  also that LIO (CONFIG_TARGET_CORE=m and CONFIG_TCM_FILEIO=m), ib_srp,
  ib_srpt and dm-mpath are built as kernel modules.
* If none of the IB ports are connected to an IB switch, connect the
  two ports to each other and configure and start the opensm software
  such that the port states change from "Initializing" to "Active".
* Check with "ibstat | grep State: Active" that at least one port is
  in the active state.
* Configure multipathd as explained in
  https://github.com/bvanassche/srp-test/blob/master/README.md.
* Restart multipathd to make sure it picks up /etc/multipath.conf.
* Clone https://github.com/bvanassche/srp-test and start it as follows:
  srp-test/run_tests -t 02-mq

Bart.--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Jens Axboe Jan. 27, 2017, 12:41 a.m. UTC | #7 
On 01/26/2017 05:38 PM, Bart Van Assche wrote:
> On Thu, 2017-01-26 at 16:50 -0700, Jens Axboe wrote:
>> Clearly we are missing some requests. How do I setup dm similarly to
>> you?
>>
>> Does it reproduce without Christoph's patchset?
> 
> Hello Jens,
> 
> I see similar behavior with the blk-mq-sched branch of
> git://git.kernel.dk/linux-block.git (git commit ID 0efe27068ecf):
> booting happens much slower than usual and I/O hangs if I run the
> srp-test software.

Please don't run that, run for-4.11/block and merge it to master.
Same behavior?

> Regarding creating a similar dm setup: I hope that in the future it
> will become possible to run the srp-test software without any special
> hardware and with in-tree drivers. Today running the srp-test software
> with in-tree drivers namely requires IB hardware. This is how to run the
> srp-test software today with in-tree drivers:
> * Find a system with at least two InfiniBand ports.
> * Make sure that the appropriate IB driver in the kernel is enabled and
>   also that LIO (CONFIG_TARGET_CORE=m and CONFIG_TCM_FILEIO=m), ib_srp,
>   ib_srpt and dm-mpath are built as kernel modules.
> * If none of the IB ports are connected to an IB switch, connect the
>   two ports to each other and configure and start the opensm software
>   such that the port states change from "Initializing" to "Active".
> * Check with "ibstat | grep State: Active" that at least one port is
>   in the active state.
> * Configure multipathd as explained in
>   https://github.com/bvanassche/srp-test/blob/master/README.md.
> * Restart multipathd to make sure it picks up /etc/multipath.conf.
> * Clone https://github.com/bvanassche/srp-test and start it as follows:
>   srp-test/run_tests -t 02-mq

I can't run that. Any chance of a test case that doesn't require IB?
Bart Van Assche Jan. 27, 2017, 1:15 a.m. UTC | #8 
On Thu, 2017-01-26 at 17:41 -0700, Jens Axboe wrote:
> On 01/26/2017 05:38 PM, Bart Van Assche wrote:
> > I see similar behavior with the blk-mq-sched branch of
> > git://git.kernel.dk/linux-block.git (git commit ID 0efe27068ecf):
> > booting happens much slower than usual and I/O hangs if I run the
> > srp-test software.
> 
> Please don't run that, run for-4.11/block and merge it to master.
> Same behavior?

I have not yet had the chance to run the srp-test software against that
kernel. But I already see that booting takes more than ten times longer
than usual. Note: as far as I know the dm-mpath driver is not involved
in the boot process of my test system.

> > Regarding creating a similar dm setup: I hope that in the future it
> > will become possible to run the srp-test software without any special
> > hardware and with in-tree drivers. Today running the srp-test software
> > with in-tree drivers namely requires IB hardware. This is how to run the
> > srp-test software today with in-tree drivers:
> > * Find a system with at least two InfiniBand ports.
> > * Make sure that the appropriate IB driver in the kernel is enabled and
> >   also that LIO (CONFIG_TARGET_CORE=m and CONFIG_TCM_FILEIO=m), ib_srp,
> >   ib_srpt and dm-mpath are built as kernel modules.
> > * If none of the IB ports are connected to an IB switch, connect the
> >   two ports to each other and configure and start the opensm software
> >   such that the port states change from "Initializing" to "Active".
> > * Check with "ibstat | grep State: Active" that at least one port is
> >   in the active state.
> > * Configure multipathd as explained in
> >   https://github.com/bvanassche/srp-test/blob/master/README.md.
> > * Restart multipathd to make sure it picks up /etc/multipath.conf.
> > * Clone https://github.com/bvanassche/srp-test and start it as follows:
> >   srp-test/run_tests -t 02-mq
> 
> I can't run that. Any chance of a test case that doesn't require IB?

It is possible to run that test on top of the SoftRoCE driver. I will first
check myself whether the latest version of the SoftRoCE driver is stable
enough to run srp-test on top of it (see also
https://github.com/dledford/linux/commits/k.o/for-4.11).

Bart.--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Jens Axboe Jan. 27, 2017, 1:22 a.m. UTC | #9 
On 01/26/2017 06:15 PM, Bart Van Assche wrote:
> On Thu, 2017-01-26 at 17:41 -0700, Jens Axboe wrote:
>> On 01/26/2017 05:38 PM, Bart Van Assche wrote:
>>> I see similar behavior with the blk-mq-sched branch of
>>> git://git.kernel.dk/linux-block.git (git commit ID 0efe27068ecf):
>>> booting happens much slower than usual and I/O hangs if I run the
>>> srp-test software.
>>
>> Please don't run that, run for-4.11/block and merge it to master.
>> Same behavior?
> 
> I have not yet had the chance to run the srp-test software against that
> kernel. But I already see that booting takes more than ten times longer
> than usual. Note: as far as I know the dm-mpath driver is not involved
> in the boot process of my test system.

What's your boot device? I've been booting this on a variety of setups,
no problems observed. It's booting my laptop, and on SCSI and SATA as
well. What is your root drive? What is the queue depth of it?
Controller?
Bart Van Assche Jan. 27, 2017, 5:02 p.m. UTC | #10 
On Thu, 2017-01-26 at 18:22 -0700, Jens Axboe wrote:
> What's your boot device? I've been booting this on a variety of setups,
> no problems observed. It's booting my laptop, and on SCSI and SATA as
> well. What is your root drive? What is the queue depth of it?
> Controller?

The boot device in my test setup is a SATA hard disk:

# cat /proc/cmdline  
BOOT_IMAGE=/boot/vmlinuz-4.10.0-rc5-dbg+ root=UUID=60a4b064-b3ef-4d28-96d3-3c13ecbec43e resume=/dev/sda2 showopts
# ls -l /dev/disk/by-uuid/60a4b064-b3ef-4d28-96d3-3c13ecbec43e
lrwxrwxrwx 1 root root 10 Jan 27 08:43 /dev/disk/by-uuid/60a4b064-b3ef-4d28-96d3-3c13ecbec43e -> ../../sda1
# cat /sys/block/sda/queue/nr_requests  
31
# lsscsi | grep sda
[0:0:0:0]    disk    ATA      ST1000NM0033-9ZM GA67  /dev/sda
# hdparm -i /dev/sda

/dev/sda:
 Model=ST1000NM0033-9ZM173, FwRev=GA67, SerialNo=Z1W2HM75
 Config={ HardSect NotMFM HdSw>15uSec Fixed DTR>10Mbs RotSpdTol>.5% }
 RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=0
 BuffType=unknown, BuffSize=unknown, MaxMultSect=16, MultSect=off
 CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=1953525168
 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
 PIO modes:  pio0 pio1 pio2 pio3 pio4  
 DMA modes:  mdma0 mdma1 mdma2  
 UDMA modes: udma0 udma1 udma2 udma3 udma4 udma5 *udma6  
 AdvancedPM=no WriteCache=disabled
 Drive conforms to: Unspecified:  ATA/ATAPI-4,5,6,7

 * signifies the current active mode

Bart.--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
diff mbox

Patch

diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c
index 3136696f4991..c27613de80c5 100644
--- a/block/blk-mq-sched.c
+++ b/block/blk-mq-sched.c
@@ -134,7 +134,8 @@  struct request *blk_mq_sched_get_request(struct request_queue *q,
 			rq = __blk_mq_alloc_request(data, op);
 	} else {
 		rq = __blk_mq_alloc_request(data, op);
-		data->hctx->tags->rqs[rq->tag] = rq;
+		if (rq)
+			data->hctx->tags->rqs[rq->tag] = rq;
 	}
 
 	if (rq) {