diff mbox

[V2,0/2] block: fix queue freeze and cleanup

Message ID 1513201741.2413.7.camel@wdc.com (mailing list archive)
State New, archived
Headers show

Commit Message

Bart Van Assche Dec. 13, 2017, 9:49 p.m. UTC 
On Fri, 2017-12-01 at 16:49 -0200, Mauricio Faria de Oliveira wrote:
> LR [c00000000057c7fc] __blk_run_queue+0x6c/0xb0
> Call Trace:
> [c0000001fb083970] [c0000001fb0839e0] 0xc0000001fb0839e0 (unreliable)
> [c0000001fb0839a0] [c00000000057ce0c] blk_run_queue+0x4c/0x80
> [c0000001fb0839d0] [c000000000591f54] blk_freeze_queue_start+0xa4/0xb0
> [c0000001fb083a00] [c00000000057d5cc] blk_set_queue_dying+0x6c/0x190
> [c0000001fb083a30] [c0000000008a3fbc] __dm_destroy+0xac/0x300
> [c0000001fb083ad0] [c0000000008af6a4] dev_remove+0x154/0x1d0
> [c0000001fb083b20] [c0000000008affd0] ctl_ioctl+0x360/0x4f0
> [c0000001fb083d10] [c0000000008b0198] dm_ctl_ioctl+0x38/0x50
> [c0000001fb083d50] [c0000000003863b8] do_vfs_ioctl+0xd8/0x8c0
> [c0000001fb083df0] [c000000000386c08] SyS_ioctl+0x68/0x100
> [c0000001fb083e30] [c00000000000b760] system_call+0x58/0x6c
> Instruction dump:
> XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
> XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
> ---[ end trace e1710ec836e5526f ]---
> 
> Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b

Hello Mauricio,

Would it be possible to repeat your test with the patch below applied on your
kernel tree? This patch has just been posted on the dm-devel mailing list.

Thanks,

Bart.


From: Bart Van Assche <bart.vanassche@wdc.com>
Date: Wed, 13 Dec 2017 13:07:14 -0800
Subject: [PATCH] dm: Fix a recently introduced reference counting bug

This patch avoids that the following message occurs sporadically
in the system log (revealing that pgpath->path.dev->name became
a dangling pointer):

device-mapper: table: 254:2: device kkkkkkkkkkkkkkkkkkk?????????x0?a?????E??????????????E??????F?????2?????pF??????PF?????9[F??????]F???????#???????#??????'f????? not in table devices list

This patch also fixes the following kernel crash:

general protection fault: 0000 [#1] PREEMPT SMP
RIP: 0010:multipath_busy+0x77/0xd0 [dm_multipath]
Call Trace:
 dm_mq_queue_rq+0x44/0x110 [dm_mod]
 blk_mq_dispatch_rq_list+0x73/0x440
 blk_mq_do_dispatch_sched+0x60/0xe0
 blk_mq_sched_dispatch_requests+0x11a/0x1a0
 __blk_mq_run_hw_queue+0x11f/0x1c0
 __blk_mq_delay_run_hw_queue+0x95/0xe0
 blk_mq_run_hw_queue+0x25/0x80
 blk_mq_flush_plug_list+0x197/0x420
 blk_flush_plug_list+0xe4/0x270
 blk_finish_plug+0x27/0x40
 __do_page_cache_readahead+0x2b4/0x370
 force_page_cache_readahead+0xb4/0x110
 generic_file_read_iter+0x755/0x970
 __vfs_read+0xd2/0x140
 vfs_read+0x9b/0x140
 SyS_read+0x45/0xa0
 do_syscall_64+0x56/0x1a0
 entry_SYSCALL64_slow_path+0x25/0x25

From the disassembly of multipath_busy (0x77 = 119):

./include/linux/blkdev.h:
992             return bdev->bd_disk->queue;    /* this is never NULL */
   0x00000000000006b4 <+116>:   mov    (%rax),%rax
   0x00000000000006b7 <+119>:   mov    0xe0(%rax),%rax

Fixes: commit 2a0b4682e09d ("dm: convert dm_dev_internal.count from atomic_t to refcount_t")
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Cc: Elena Reshetova <elena.reshetova@intel.com>
Cc: Kees Cook <keescook@chromium.org>
Cc: David Windsor <dwindsor@gmail.com>
Cc: Hans Liljestrand <ishkamiel@gmail.com>
Cc: Hannes Reinecke <hare@suse.com>
Cc: stable@vger.kernel.org # v4.15
---
 drivers/md/dm-table.c | 2 ++
 1 file changed, 2 insertions(+)

-- 
2.15.1

Comments

Mauricio Faria de Oliveira Dec. 20, 2017, 2:34 p.m. UTC | #1 
Hi Bart,

On 12/13/2017 07:49 PM, Bart Van Assche wrote:
> Would it be possible to repeat your test with the patch below applied on your
> kernel tree? This patch has just been posted on the dm-devel mailing list.

Sorry for the delay. I missed this.

Unfortunately the oops problem still happens on PATCH v2 and that patch
(actually, the version that ended up in v4.15-rc4 [1] by Mike Snitzer).

The problem does not happen with PATCH v3 (a.k.a. v1) -- i.e., v3 is OK.

Thanks!


Test-case w/ PATCH v3:
---

[root@guest ~]# uname -r
4.15.0-rc4.mingleiv3

[root@guest ~]# reboot
...
systemd-shutdown[1]: Detaching DM devices.
systemd-shutdown[1]: Detaching DM 253:2.
shutdown: 7 output lines suppressed due to ratelimiting
dracut Warning: Killing all remaining processes
dracut Warning: Killing all remaining processes
XFS (dm-0): Unmounting Filesystem
dracut Warning: Unmounted /oldroot.
dracut: Disassembling device-mapper devices
Rebooting.
sd 0:0:0:0: [sda] Synchronizing SCSI cache
reboot: Restarting system


Test-case w/ PATCH v2:
---

[root@guest ~]# uname -r
4.15.0-rc4.mingleiv2

[root@guest ~]# reboot
...
systemd-shutdown[1]: Detaching DM devices.
systemd-shutdown[1]: Detaching DM 253:2.
Unable to handle kernel paging request for instruction fetch
Faulting instruction address: 0x00000000
Oops: Kernel access of bad area, sig: 11 [#1]
LE SMP NR_CPUS=2048 NUMA pSeries
Modules linked in: vmx_crypto virtio_balloon ip_tables x_tables xfs 
libcrc32c virtio_net virtio_scsi crc32c_vpmsum virtio_pci virtio_ring 
virtio autofs4
CPU: 3 PID: 1 Comm: systemd-shutdow Not tainted 4.15.0-rc4.mingleiv2 #4
<...>
NIP [0000000000000000]           (null)
LR [c00000000057d0dc] __blk_run_queue+0x6c/0xb0
Call Trace:
[c0000001fb083970] [c0000001fb0839e0] 0xc0000001fb0839e0 (unreliable)
[c0000001fb0839a0] [c00000000057d6ec] blk_run_queue+0x4c/0x80
[c0000001fb0839d0] [c000000000592834] blk_freeze_queue_start+0xa4/0xb0
[c0000001fb083a00] [c00000000057deac] blk_set_queue_dying+0x6c/0x190
[c0000001fb083a30] [c0000000008a4b7c] __dm_destroy+0xac/0x300
[c0000001fb083ad0] [c0000000008b0244] dev_remove+0x154/0x1d0
[c0000001fb083b20] [c0000000008b0b70] ctl_ioctl+0x360/0x4f0
[c0000001fb083d10] [c0000000008b0d38] dm_ctl_ioctl+0x38/0x50
[c0000001fb083d50] [c0000000003867d8] do_vfs_ioctl+0xd8/0x8c0
[c0000001fb083df0] [c000000000387028] SyS_ioctl+0x68/0x100
[c0000001fb083e30] [c00000000000b760] system_call+0x58/0x6c
Instruction dump:
XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
---[ end trace 0fceefbe4fc1cd29 ]---

Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b



[1] 
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-4.15&id=afc567a4977b2d798e05153dd131a3c8d4758c0c
diff mbox

Patch

diff --git a/drivers/md/dm-table.c b/drivers/md/dm-table.c
index 88130b5d95f9..ee5c389e7256 100644
--- a/drivers/md/dm-table.c
+++ b/drivers/md/dm-table.c
@@ -459,6 +459,8 @@  int dm_get_device(struct dm_target *ti, const char *path, fmode_t mode,
 		if (r)
 			return r;
 		refcount_inc(&dd->count);
+	} else {
+		refcount_inc(&dd->count);
 	}
 
 	*result = dd->dm_dev;