| Message ID | D2F74AA5.88BF%Shiva.Krishna@nimblestorage.com (mailing list archive) |
|---|---|
| State | Not Applicable, archived |
| Delegated to: | christophe varoqui |
| Headers | show |
1. When all paths fail, reinstate_path and fail_path frequently switch, so a previously failed path is automatically reinstated,
Could someone tell me why is that? moreover, dm_reinstate_path not being called in multipathd.
Mar 2 16:06:50 cvknode129 kernel: [68389.817587] sd 3:0:0:0: rejecting I/O to offline device
Mar 2 16:06:50 cvknode129 kernel: [68389.817598] device-mapper: multipath: Failing path 8:16. 3483
Mar 2 16:06:50 cvknode129 kernel: [68389.817601] CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G OE 4.1.0-generic #2
Mar 2 16:06:50 cvknode129 kernel: [68389.817602] Hardware name: HP ProLiant BL460c Gen8, BIOS I31 02/25/2012
Mar 2 16:06:50 cvknode129 kernel: [68389.817603] ffff880816673500 ffff88081be8bd08 ffffffff817e8924 0000000000000007
Mar 2 16:06:50 cvknode129 kernel: [68389.817605] ffff880816673528 ffff88081be8bd38 ffffffffc019c75e 00000000fffffffb
Mar 2 16:06:50 cvknode129 kernel: [68389.817607] ffff88003698b458 ffff880816673500 0000000000000010 ffff88081be8bd88
Mar 2 16:06:50 cvknode129 kernel: [68389.817609] Call Trace:
Mar 2 16:06:50 cvknode129 kernel: [68389.817612] [<ffffffff817e8924>] dump_stack+0x45/0x57
Mar 2 16:06:50 cvknode129 kernel: [68389.817622] [<ffffffffc019c75e>] fail_path+0x7e/0xf0 [dm_multipath]
Mar 2 16:06:50 cvknode129 kernel: [68389.817624] [<ffffffffc019d60e>] multipath_end_io+0x5e/0x190 [dm_multipath]
Mar 2 16:06:50 cvknode129 kernel: [68389.817627] [<ffffffff81020e19>] ? sched_clock+0x9/0x10
Mar 2 16:06:50 cvknode129 kernel: [68389.817629] [<ffffffff81666cf3>] dm_softirq_done+0xd3/0x250
Mar 2 16:06:50 cvknode129 kernel: [68389.817642] [<ffffffff81015686>] ? __switch_to+0x1e6/0x580
Mar 2 16:06:50 cvknode129 kernel: [68389.817645] [<ffffffff81397d9b>] blk_done_softirq+0x7b/0x90
Mar 2 16:06:50 cvknode129 kernel: [68389.817647] [<ffffffff810820fe>] __do_softirq+0xde/0x2d0
Mar 2 16:06:50 cvknode129 kernel: [68389.817649] [<ffffffff81082310>] run_ksoftirqd+0x20/0x60
Mar 2 16:06:50 cvknode129 kernel: [68389.817650] [<ffffffff810a0fd6>] smpboot_thread_fn+0x116/0x170
Mar 2 16:06:50 cvknode129 kernel: [68389.817652] [<ffffffff810a0ec0>] ? sort_range+0x30/0x30
Mar 2 16:06:50 cvknode129 kernel: [68389.817653] [<ffffffff8109db59>] kthread+0xc9/0xe0
Mar 2 16:06:50 cvknode129 kernel: [68389.817655] [<ffffffff8109da90>] ? flush_kthread_worker+0x90/0x90
Mar 2 16:06:50 cvknode129 kernel: [68389.817657] [<ffffffff817f0ca2>] ret_from_fork+0x42/0x70
Mar 2 16:06:50 cvknode129 kernel: [68389.817658] [<ffffffff8109da90>] ? flush_kthread_worker+0x90/0x90
Mar 2 16:06:50 cvknode129 kernel: [68389.843090] CPU: 3 PID: 43112 Comm: multipath Tainted: G OE 4.1.0-generic #2
Mar 2 16:06:50 cvknode129 kernel: [68389.843092] Hardware name: HP ProLiant BL460c Gen8, BIOS I31 02/25/2012
Mar 2 16:06:50 cvknode129 kernel: [68389.843093] ffff880816673528 ffff8807efcdfb78 ffffffff817e8924 ffff88081952e580
Mar 2 16:06:50 cvknode129 kernel: [68389.843095] 0000000000000000 ffff8807efcdfbc8 ffffffffc019e20e 0000000000000000
Mar 2 16:06:50 cvknode129 kernel: [68389.843097] ffff880036a81440 0000000000000008 ffff88081a288e78 ffffc90006d49040
Mar 2 16:06:50 cvknode129 kernel: [68389.843098] Call Trace:
Mar 2 16:06:50 cvknode129 kernel: [68389.843101] [<ffffffff817e8924>] dump_stack+0x45/0x57
Mar 2 16:06:50 cvknode129 kernel: [68389.843106] [<ffffffffc019e20e>] reinstate_path+0xae/0x1a0 [dm_multipath]
Mar 2 16:06:50 cvknode129 kernel: [68389.843108] [<ffffffffc019e160>] ? multipath_map+0x20/0x20 [dm_multipath]
Mar 2 16:06:50 cvknode129 kernel: [68389.843113] [<ffffffffc019d9ce>] multipath_message+0x18e/0x360 [dm_multipath]
Mar 2 16:06:50 cvknode129 kernel: [68389.843115] [<ffffffff8166dbc5>] target_message+0x255/0x340
Mar 2 16:06:50 cvknode129 kernel: [68389.843116] [<ffffffff8166d970>] ? __dev_status+0x150/0x150
Mar 2 16:06:50 cvknode129 kernel: [68389.843118] [<ffffffff8166ef7a>] ctl_ioctl+0x24a/0x520
Mar 2 16:06:50 cvknode129 kernel: [68389.843120] [<ffffffff8166f263>] dm_ctl_ioctl+0x13/0x20
Mar 2 16:06:50 cvknode129 kernel: [68389.843122] [<ffffffff812142d6>] do_vfs_ioctl+0x86/0x530
Mar 2 16:06:50 cvknode129 kernel: [68389.843124] [<ffffffff8106a10f>] ? __do_page_fault+0x1af/0x470
Mar 2 16:06:50 cvknode129 kernel: [68389.843126] [<ffffffff81214811>] SyS_ioctl+0x91/0xb0
Mar 2 16:06:50 cvknode129 kernel: [68389.843128] [<ffffffff817f0872>] system_call_fastpath+0x16/0x75
Mar 2 16:06:50 cvknode129 kernel: [68389.843255] sd 5:0:0:0: rejecting I/O to offline device
Mar 2 16:06:50 cvknode129 kernel: [68389.843266] device-mapper: multipath: Failing path 8:48. 3514
device {
vendor "MacroSAN"
product "LU"
path_grouping_policy "group_by_prio"
path_checker "tur"
features "1 queue_if_no_path"
hardware_handler "0"
prio "alua"
failback 15
rr_weight "priorities"
no_path_retry 30
rr_min_io 1000
}
3600b34223c09334d0467dfcd5d0000da dm-0 MacroSAN,LU
size=50G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| `- 5:0:0:0 sdd 8:48 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
`- 3:0:0:0 sdb 8:16 active ready running
Thanks!
Best regards!
________________________________
zhangguanghui
-------------------------------------------------------------------------------------------------------------------------------------
????????????????????????????????????????
????????????????????????????????????????
????????????????????????????????????????
???
This e-mail and its attachments contain confidential information from H3C, which is
intended only for the person or entity whose address is listed above. Any use of the
information contained herein in any way (including, but not limited to, total or partial
disclosure, reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender
by phone or email immediately and delete it!
--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
1. When all paths fail, reinstate_path and fail_path frequently switch, so a previously failed path is automatically reinstated,
Could someone tell me why is that? moreover, dm_reinstate_path not being called in multipathd.
dm_reinstate_path is indeed initiated by multipathd when path current status from the cheker module is found to be PATH_UP / PATH_GHOST(online but path unusable) which is not same as previous status.
The chekerloop thread of multipathd daemon will be monitoring all the paths periodically via check_path, where it initiates fail_path, re-instate etc.,
messages to device mapper based on the current paths status (PATH_DOWN/ PATH_UP/PATH_GHOST/ PATH_SHAKY etc.,).
- As a first step, check_path initiates path_offline to check path status to see if it is offline/online from sysfs. If the path is offline(PATH_DOWN) or PATH_SHAKY and different from previous status it initiates fail_path msg to device mapper. If path is online (PATHUP) it initiates respective checker (in this case tur) to ping and check the new status of the path.
o If check_path founds from the tur checker module that the new path status is PATH_UP (tur cmd succeeds) and PATH_GHOST (LOGICAL UNIT NOT ACCESSIBLE , TARGET PORT IN STANDBY STATE) and not same as previous status then it initiate reinstate_path msg to device mapper.
o If check_path founds from the tur checker module that the new path status is PATH_DOWN (tur cmd fails) or PATH_SHAKY(not used in tur checker anyway) and not same as previous status then it initiate fail_path msg to device mapper.
In the below situation it looks to be the case where the paths are not-stable/flakey resulting in reinstate_path/ fail_path by the multipathd at each tur cmd failure or success.
Mar 2 16:06:50 cvknode129 kernel: [68389.817587] sd 3:0:0:0: rejecting I/O to offline device
Mar 2 16:06:50 cvknode129 kernel: [68389.817598] device-mapper: multipath: Failing path 8:16. 3483
Mar 2 16:06:50 cvknode129 kernel: [68389.817601] CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G OE 4.1.0-generic #2
Mar 2 16:06:50 cvknode129 kernel: [68389.817602] Hardware name: HP ProLiant BL460c Gen8, BIOS I31 02/25/2012
Mar 2 16:06:50 cvknode129 kernel: [68389.817603] ffff880816673500 ffff88081be8bd08 ffffffff817e8924 0000000000000007
Mar 2 16:06:50 cvknode129 kernel: [68389.817605] ffff880816673528 ffff88081be8bd38 ffffffffc019c75e 00000000fffffffb
Mar 2 16:06:50 cvknode129 kernel: [68389.817607] ffff88003698b458 ffff880816673500 0000000000000010 ffff88081be8bd88
Mar 2 16:06:50 cvknode129 kernel: [68389.817609] Call Trace:
Mar 2 16:06:50 cvknode129 kernel: [68389.817612] [<ffffffff817e8924>] dump_stack+0x45/0x57
Mar 2 16:06:50 cvknode129 kernel: [68389.817622] [<ffffffffc019c75e>] fail_path+0x7e/0xf0 [dm_multipath]
Mar 2 16:06:50 cvknode129 kernel: [68389.817624] [<ffffffffc019d60e>] multipath_end_io+0x5e/0x190 [dm_multipath]
Mar 2 16:06:50 cvknode129 kernel: [68389.817627] [<ffffffff81020e19>] ? sched_clock+0x9/0x10
Mar 2 16:06:50 cvknode129 kernel: [68389.817629] [<ffffffff81666cf3>] dm_softirq_done+0xd3/0x250
Mar 2 16:06:50 cvknode129 kernel: [68389.817642] [<ffffffff81015686>] ? __switch_to+0x1e6/0x580
Mar 2 16:06:50 cvknode129 kernel: [68389.817645] [<ffffffff81397d9b>] blk_done_softirq+0x7b/0x90
Mar 2 16:06:50 cvknode129 kernel: [68389.817647] [<ffffffff810820fe>] __do_softirq+0xde/0x2d0
Mar 2 16:06:50 cvknode129 kernel: [68389.817649] [<ffffffff81082310>] run_ksoftirqd+0x20/0x60
Mar 2 16:06:50 cvknode129 kernel: [68389.817650] [<ffffffff810a0fd6>] smpboot_thread_fn+0x116/0x170
Mar 2 16:06:50 cvknode129 kernel: [68389.817652] [<ffffffff810a0ec0>] ? sort_range+0x30/0x30
Mar 2 16:06:50 cvknode129 kernel: [68389.817653] [<ffffffff8109db59>] kthread+0xc9/0xe0
Mar 2 16:06:50 cvknode129 kernel: [68389.817655] [<ffffffff8109da90>] ? flush_kthread_worker+0x90/0x90
Mar 2 16:06:50 cvknode129 kernel: [68389.817657] [<ffffffff817f0ca2>] ret_from_fork+0x42/0x70
Mar 2 16:06:50 cvknode129 kernel: [68389.817658] [<ffffffff8109da90>] ? flush_kthread_worker+0x90/0x90
Mar 2 16:06:50 cvknode129 kernel: [68389.843090] CPU: 3 PID: 43112 Comm: multipath Tainted: G OE 4.1.0-generic #2
Mar 2 16:06:50 cvknode129 kernel: [68389.843092] Hardware name: HP ProLiant BL460c Gen8, BIOS I31 02/25/2012
Mar 2 16:06:50 cvknode129 kernel: [68389.843093] ffff880816673528 ffff8807efcdfb78 ffffffff817e8924 ffff88081952e580
Mar 2 16:06:50 cvknode129 kernel: [68389.843095] 0000000000000000 ffff8807efcdfbc8 ffffffffc019e20e 0000000000000000
Mar 2 16:06:50 cvknode129 kernel: [68389.843097] ffff880036a81440 0000000000000008 ffff88081a288e78 ffffc90006d49040
Mar 2 16:06:50 cvknode129 kernel: [68389.843098] Call Trace:
Mar 2 16:06:50 cvknode129 kernel: [68389.843101] [<ffffffff817e8924>] dump_stack+0x45/0x57
Mar 2 16:06:50 cvknode129 kernel: [68389.843106] [<ffffffffc019e20e>] reinstate_path+0xae/0x1a0 [dm_multipath]
Mar 2 16:06:50 cvknode129 kernel: [68389.843108] [<ffffffffc019e160>] ? multipath_map+0x20/0x20 [dm_multipath]
Mar 2 16:06:50 cvknode129 kernel: [68389.843113] [<ffffffffc019d9ce>] multipath_message+0x18e/0x360 [dm_multipath]
Mar 2 16:06:50 cvknode129 kernel: [68389.843115] [<ffffffff8166dbc5>] target_message+0x255/0x340
Mar 2 16:06:50 cvknode129 kernel: [68389.843116] [<ffffffff8166d970>] ? __dev_status+0x150/0x150
Mar 2 16:06:50 cvknode129 kernel: [68389.843118] [<ffffffff8166ef7a>] ctl_ioctl+0x24a/0x520
Mar 2 16:06:50 cvknode129 kernel: [68389.843120] [<ffffffff8166f263>] dm_ctl_ioctl+0x13/0x20
Mar 2 16:06:50 cvknode129 kernel: [68389.843122] [<ffffffff812142d6>] do_vfs_ioctl+0x86/0x530
Mar 2 16:06:50 cvknode129 kernel: [68389.843124] [<ffffffff8106a10f>] ? __do_page_fault+0x1af/0x470
Mar 2 16:06:50 cvknode129 kernel: [68389.843126] [<ffffffff81214811>] SyS_ioctl+0x91/0xb0
Mar 2 16:06:50 cvknode129 kernel: [68389.843128] [<ffffffff817f0872>] system_call_fastpath+0x16/0x75
Mar 2 16:06:50 cvknode129 kernel: [68389.843255] sd 5:0:0:0: rejecting I/O to offline device
Mar 2 16:06:50 cvknode129 kernel: [68389.843266] device-mapper: multipath: Failing path 8:48. 3514
device {
vendor "MacroSAN"
product "LU"
path_grouping_policy "group_by_prio"
path_checker "tur"
features "1 queue_if_no_path"
hardware_handler "0"
prio "alua"
failback 15
rr_weight "priorities"
no_path_retry 30
rr_min_io 1000
}
3600b34223c09334d0467dfcd5d0000da dm-0 MacroSAN,LU
size=50G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| `- 5:0:0:0 sdd 8:48 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
`- 3:0:0:0 sdb 8:16 active ready running
Thanks!
Best regards!
________________________________
zhangguanghui
-------------------------------------------------------------------------------------------------------------------------------------
????????????????????????????????????????
????????????????????????????????????????
????????????????????????????????????????
???
This e-mail and its attachments contain confidential information from H3C, which is
intended only for the person or entity whose address is listed above. Any use of the
information contained herein in any way (including, but not limited to, total or partial
disclosure, reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender
by phone or email immediately and delete it!
--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
Thanks a lots , The chekerloop thread of multipathd daemon will be monitoring all the paths periodically via check_path.
As far as I know, each tur cmd takes a 5s interval in default, a previously failed path may be automatically reinstated.
According to the log, it looks to be a endless loop(reinstate_path and fail_path frequently switch in 1 second),
only in the case of prio alua and two path groups. when the case of prio const and one path group is normal.
so I don't know what is that.
________________________________
zhangguanghui
???? Nalla, Ravikanth<mailto:ravikanth.nalla@hpe.com>
????? 2016-03-02 21:05
???? zhangguanghui 10102 (CCPL)<mailto:zhang.guanghui@h3c.com>; dm-devel-bounces@redhat.com<mailto:dm-devel-bounces@redhat.com>; dm-devel@redhat.com<mailto:dm-devel@redhat.com>
??? RE: multipath-tools: prevent unnecessary reinstate?
1. When all paths fail, reinstate_path and fail_path frequently switch, so a previously failed path is automatically reinstated,
Could someone tell me why is that? moreover, dm_reinstate_path not being called in multipathd.
dm_reinstate_path is indeed initiated by multipathd when path current status from the cheker module is found to be PATH_UP / PATH_GHOST(online but path unusable) which is not same as previous status.
The chekerloop thread of multipathd daemon will be monitoring all the paths periodically via check_path, where it initiates fail_path, re-instate etc.,
messages to device mapper based on the current paths status (PATH_DOWN/ PATH_UP/PATH_GHOST/ PATH_SHAKY etc.,).
- As a first step, check_path initiates path_offline to check path status to see if it is offline/online from sysfs. If the path is offline(PATH_DOWN) or PATH_SHAKY and different from previous status it initiates fail_path msg to device mapper. If path is online (PATHUP) it initiates respective checker (in this case tur) to ping and check the new status of the path.
o If check_path founds from the tur checker module that the new path status is PATH_UP (tur cmd succeeds) and PATH_GHOST (LOGICAL UNIT NOT ACCESSIBLE , TARGET PORT IN STANDBY STATE) and not same as previous status then it initiate reinstate_path msg to device mapper.
o If check_path founds from the tur checker module that the new path status is PATH_DOWN (tur cmd fails) or PATH_SHAKY(not used in tur checker anyway) and not same as previous status then it initiate fail_path msg to device mapper.
In the below situation it looks to be the case where the paths are not-stable/flakey resulting in reinstate_path/ fail_path by the multipathd at each tur cmd failure or success.
Mar 2 16:06:50 cvknode129 kernel: [68389.817587] sd 3:0:0:0: rejecting I/O to offline device
Mar 2 16:06:50 cvknode129 kernel: [68389.817598] device-mapper: multipath: Failing path 8:16. 3483
Mar 2 16:06:50 cvknode129 kernel: [68389.817601] CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G OE 4.1.0-generic #2
Mar 2 16:06:50 cvknode129 kernel: [68389.817602] Hardware name: HP ProLiant BL460c Gen8, BIOS I31 02/25/2012
Mar 2 16:06:50 cvknode129 kernel: [68389.817603] ffff880816673500 ffff88081be8bd08 ffffffff817e8924 0000000000000007
Mar 2 16:06:50 cvknode129 kernel: [68389.817605] ffff880816673528 ffff88081be8bd38 ffffffffc019c75e 00000000fffffffb
Mar 2 16:06:50 cvknode129 kernel: [68389.817607] ffff88003698b458 ffff880816673500 0000000000000010 ffff88081be8bd88
Mar 2 16:06:50 cvknode129 kernel: [68389.817609] Call Trace:
Mar 2 16:06:50 cvknode129 kernel: [68389.817612] [<ffffffff817e8924>] dump_stack+0x45/0x57
Mar 2 16:06:50 cvknode129 kernel: [68389.817622] [<ffffffffc019c75e>] fail_path+0x7e/0xf0 [dm_multipath]
Mar 2 16:06:50 cvknode129 kernel: [68389.817624] [<ffffffffc019d60e>] multipath_end_io+0x5e/0x190 [dm_multipath]
Mar 2 16:06:50 cvknode129 kernel: [68389.817627] [<ffffffff81020e19>] ? sched_clock+0x9/0x10
Mar 2 16:06:50 cvknode129 kernel: [68389.817629] [<ffffffff81666cf3>] dm_softirq_done+0xd3/0x250
Mar 2 16:06:50 cvknode129 kernel: [68389.817642] [<ffffffff81015686>] ? __switch_to+0x1e6/0x580
Mar 2 16:06:50 cvknode129 kernel: [68389.817645] [<ffffffff81397d9b>] blk_done_softirq+0x7b/0x90
Mar 2 16:06:50 cvknode129 kernel: [68389.817647] [<ffffffff810820fe>] __do_softirq+0xde/0x2d0
Mar 2 16:06:50 cvknode129 kernel: [68389.817649] [<ffffffff81082310>] run_ksoftirqd+0x20/0x60
Mar 2 16:06:50 cvknode129 kernel: [68389.817650] [<ffffffff810a0fd6>] smpboot_thread_fn+0x116/0x170
Mar 2 16:06:50 cvknode129 kernel: [68389.817652] [<ffffffff810a0ec0>] ? sort_range+0x30/0x30
Mar 2 16:06:50 cvknode129 kernel: [68389.817653] [<ffffffff8109db59>] kthread+0xc9/0xe0
Mar 2 16:06:50 cvknode129 kernel: [68389.817655] [<ffffffff8109da90>] ? flush_kthread_worker+0x90/0x90
Mar 2 16:06:50 cvknode129 kernel: [68389.817657] [<ffffffff817f0ca2>] ret_from_fork+0x42/0x70
Mar 2 16:06:50 cvknode129 kernel: [68389.817658] [<ffffffff8109da90>] ? flush_kthread_worker+0x90/0x90
Mar 2 16:06:50 cvknode129 kernel: [68389.843090] CPU: 3 PID: 43112 Comm: multipath Tainted: G OE 4.1.0-generic #2
Mar 2 16:06:50 cvknode129 kernel: [68389.843092] Hardware name: HP ProLiant BL460c Gen8, BIOS I31 02/25/2012
Mar 2 16:06:50 cvknode129 kernel: [68389.843093] ffff880816673528 ffff8807efcdfb78 ffffffff817e8924 ffff88081952e580
Mar 2 16:06:50 cvknode129 kernel: [68389.843095] 0000000000000000 ffff8807efcdfbc8 ffffffffc019e20e 0000000000000000
Mar 2 16:06:50 cvknode129 kernel: [68389.843097] ffff880036a81440 0000000000000008 ffff88081a288e78 ffffc90006d49040
Mar 2 16:06:50 cvknode129 kernel: [68389.843098] Call Trace:
Mar 2 16:06:50 cvknode129 kernel: [68389.843101] [<ffffffff817e8924>] dump_stack+0x45/0x57
Mar 2 16:06:50 cvknode129 kernel: [68389.843106] [<ffffffffc019e20e>] reinstate_path+0xae/0x1a0 [dm_multipath]
Mar 2 16:06:50 cvknode129 kernel: [68389.843108] [<ffffffffc019e160>] ? multipath_map+0x20/0x20 [dm_multipath]
Mar 2 16:06:50 cvknode129 kernel: [68389.843113] [<ffffffffc019d9ce>] multipath_message+0x18e/0x360 [dm_multipath]
Mar 2 16:06:50 cvknode129 kernel: [68389.843115] [<ffffffff8166dbc5>] target_message+0x255/0x340
Mar 2 16:06:50 cvknode129 kernel: [68389.843116] [<ffffffff8166d970>] ? __dev_status+0x150/0x150
Mar 2 16:06:50 cvknode129 kernel: [68389.843118] [<ffffffff8166ef7a>] ctl_ioctl+0x24a/0x520
Mar 2 16:06:50 cvknode129 kernel: [68389.843120] [<ffffffff8166f263>] dm_ctl_ioctl+0x13/0x20
Mar 2 16:06:50 cvknode129 kernel: [68389.843122] [<ffffffff812142d6>] do_vfs_ioctl+0x86/0x530
Mar 2 16:06:50 cvknode129 kernel: [68389.843124] [<ffffffff8106a10f>] ? __do_page_fault+0x1af/0x470
Mar 2 16:06:50 cvknode129 kernel: [68389.843126] [<ffffffff81214811>] SyS_ioctl+0x91/0xb0
Mar 2 16:06:50 cvknode129 kernel: [68389.843128] [<ffffffff817f0872>] system_call_fastpath+0x16/0x75
Mar 2 16:06:50 cvknode129 kernel: [68389.843255] sd 5:0:0:0: rejecting I/O to offline device
Mar 2 16:06:50 cvknode129 kernel: [68389.843266] device-mapper: multipath: Failing path 8:48. 3514
device {
vendor "MacroSAN"
product "LU"
path_grouping_policy "group_by_prio"
path_checker "tur"
features "1 queue_if_no_path"
hardware_handler "0"
prio "alua"
failback 15
rr_weight "priorities"
no_path_retry 30
rr_min_io 1000
}
3600b34223c09334d0467dfcd5d0000da dm-0 MacroSAN,LU
size=50G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| `- 5:0:0:0 sdd 8:48 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
`- 3:0:0:0 sdb 8:16 active ready running
Thanks!
Best regards!
________________________________
zhangguanghui
-------------------------------------------------------------------------------------------------------------------------------------
????????????????????????????????????????
????????????????????????????????????????
????????????????????????????????????????
???
This e-mail and its attachments contain confidential information from H3C, which is
intended only for the person or entity whose address is listed above. Any use of the
information contained herein in any way (including, but not limited to, total or partial
disclosure, reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender
by phone or email immediately and delete it!
--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
Thanks a lots , the problem has been solved , the multipath-tool deb building leads to the dm-mpath.rules lost.
________________________________
zhangguanghui
???? zhangguanghui 10102 (CCPL)<mailto:zhang.guanghui@h3c.com>
????? 2016-03-03 09:51
???? Nalla, Ravikanth<mailto:ravikanth.nalla@hpe.com>; dm-devel-bounces@redhat.com<mailto:dm-devel-bounces@redhat.com>; dm-devel@redhat.com<mailto:dm-devel@redhat.com>
??? Re: RE: multipath-tools: prevent unnecessary reinstate?
Thanks a lots , The chekerloop thread of multipathd daemon will be monitoring all the paths periodically via check_path.
As far as I know, each tur cmd takes a 5s interval in default, a previously failed path may be automatically reinstated.
According to the log, it looks to be a endless loop(reinstate_path and fail_path frequently switch in 1 second),
only in the case of prio alua and two path groups. when the case of prio const and one path group is normal.
so I don't know what is that.
________________________________
zhangguanghui
???? Nalla, Ravikanth<mailto:ravikanth.nalla@hpe.com>
????? 2016-03-02 21:05
???? zhangguanghui 10102 (CCPL)<mailto:zhang.guanghui@h3c.com>; dm-devel-bounces@redhat.com<mailto:dm-devel-bounces@redhat.com>; dm-devel@redhat.com<mailto:dm-devel@redhat.com>
??? RE: multipath-tools: prevent unnecessary reinstate?
1. When all paths fail, reinstate_path and fail_path frequently switch, so a previously failed path is automatically reinstated,
Could someone tell me why is that? moreover, dm_reinstate_path not being called in multipathd.
dm_reinstate_path is indeed initiated by multipathd when path current status from the cheker module is found to be PATH_UP / PATH_GHOST(online but path unusable) which is not same as previous status.
The chekerloop thread of multipathd daemon will be monitoring all the paths periodically via check_path, where it initiates fail_path, re-instate etc.,
messages to device mapper based on the current paths status (PATH_DOWN/ PATH_UP/PATH_GHOST/ PATH_SHAKY etc.,).
- As a first step, check_path initiates path_offline to check path status to see if it is offline/online from sysfs. If the path is offline(PATH_DOWN) or PATH_SHAKY and different from previous status it initiates fail_path msg to device mapper. If path is online (PATHUP) it initiates respective checker (in this case tur) to ping and check the new status of the path.
o If check_path founds from the tur checker module that the new path status is PATH_UP (tur cmd succeeds) and PATH_GHOST (LOGICAL UNIT NOT ACCESSIBLE , TARGET PORT IN STANDBY STATE) and not same as previous status then it initiate reinstate_path msg to device mapper.
o If check_path founds from the tur checker module that the new path status is PATH_DOWN (tur cmd fails) or PATH_SHAKY(not used in tur checker anyway) and not same as previous status then it initiate fail_path msg to device mapper.
In the below situation it looks to be the case where the paths are not-stable/flakey resulting in reinstate_path/ fail_path by the multipathd at each tur cmd failure or success.
Mar 2 16:06:50 cvknode129 kernel: [68389.817587] sd 3:0:0:0: rejecting I/O to offline device
Mar 2 16:06:50 cvknode129 kernel: [68389.817598] device-mapper: multipath: Failing path 8:16. 3483
Mar 2 16:06:50 cvknode129 kernel: [68389.817601] CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G OE 4.1.0-generic #2
Mar 2 16:06:50 cvknode129 kernel: [68389.817602] Hardware name: HP ProLiant BL460c Gen8, BIOS I31 02/25/2012
Mar 2 16:06:50 cvknode129 kernel: [68389.817603] ffff880816673500 ffff88081be8bd08 ffffffff817e8924 0000000000000007
Mar 2 16:06:50 cvknode129 kernel: [68389.817605] ffff880816673528 ffff88081be8bd38 ffffffffc019c75e 00000000fffffffb
Mar 2 16:06:50 cvknode129 kernel: [68389.817607] ffff88003698b458 ffff880816673500 0000000000000010 ffff88081be8bd88
Mar 2 16:06:50 cvknode129 kernel: [68389.817609] Call Trace:
Mar 2 16:06:50 cvknode129 kernel: [68389.817612] [<ffffffff817e8924>] dump_stack+0x45/0x57
Mar 2 16:06:50 cvknode129 kernel: [68389.817622] [<ffffffffc019c75e>] fail_path+0x7e/0xf0 [dm_multipath]
Mar 2 16:06:50 cvknode129 kernel: [68389.817624] [<ffffffffc019d60e>] multipath_end_io+0x5e/0x190 [dm_multipath]
Mar 2 16:06:50 cvknode129 kernel: [68389.817627] [<ffffffff81020e19>] ? sched_clock+0x9/0x10
Mar 2 16:06:50 cvknode129 kernel: [68389.817629] [<ffffffff81666cf3>] dm_softirq_done+0xd3/0x250
Mar 2 16:06:50 cvknode129 kernel: [68389.817642] [<ffffffff81015686>] ? __switch_to+0x1e6/0x580
Mar 2 16:06:50 cvknode129 kernel: [68389.817645] [<ffffffff81397d9b>] blk_done_softirq+0x7b/0x90
Mar 2 16:06:50 cvknode129 kernel: [68389.817647] [<ffffffff810820fe>] __do_softirq+0xde/0x2d0
Mar 2 16:06:50 cvknode129 kernel: [68389.817649] [<ffffffff81082310>] run_ksoftirqd+0x20/0x60
Mar 2 16:06:50 cvknode129 kernel: [68389.817650] [<ffffffff810a0fd6>] smpboot_thread_fn+0x116/0x170
Mar 2 16:06:50 cvknode129 kernel: [68389.817652] [<ffffffff810a0ec0>] ? sort_range+0x30/0x30
Mar 2 16:06:50 cvknode129 kernel: [68389.817653] [<ffffffff8109db59>] kthread+0xc9/0xe0
Mar 2 16:06:50 cvknode129 kernel: [68389.817655] [<ffffffff8109da90>] ? flush_kthread_worker+0x90/0x90
Mar 2 16:06:50 cvknode129 kernel: [68389.817657] [<ffffffff817f0ca2>] ret_from_fork+0x42/0x70
Mar 2 16:06:50 cvknode129 kernel: [68389.817658] [<ffffffff8109da90>] ? flush_kthread_worker+0x90/0x90
Mar 2 16:06:50 cvknode129 kernel: [68389.843090] CPU: 3 PID: 43112 Comm: multipath Tainted: G OE 4.1.0-generic #2
Mar 2 16:06:50 cvknode129 kernel: [68389.843092] Hardware name: HP ProLiant BL460c Gen8, BIOS I31 02/25/2012
Mar 2 16:06:50 cvknode129 kernel: [68389.843093] ffff880816673528 ffff8807efcdfb78 ffffffff817e8924 ffff88081952e580
Mar 2 16:06:50 cvknode129 kernel: [68389.843095] 0000000000000000 ffff8807efcdfbc8 ffffffffc019e20e 0000000000000000
Mar 2 16:06:50 cvknode129 kernel: [68389.843097] ffff880036a81440 0000000000000008 ffff88081a288e78 ffffc90006d49040
Mar 2 16:06:50 cvknode129 kernel: [68389.843098] Call Trace:
Mar 2 16:06:50 cvknode129 kernel: [68389.843101] [<ffffffff817e8924>] dump_stack+0x45/0x57
Mar 2 16:06:50 cvknode129 kernel: [68389.843106] [<ffffffffc019e20e>] reinstate_path+0xae/0x1a0 [dm_multipath]
Mar 2 16:06:50 cvknode129 kernel: [68389.843108] [<ffffffffc019e160>] ? multipath_map+0x20/0x20 [dm_multipath]
Mar 2 16:06:50 cvknode129 kernel: [68389.843113] [<ffffffffc019d9ce>] multipath_message+0x18e/0x360 [dm_multipath]
Mar 2 16:06:50 cvknode129 kernel: [68389.843115] [<ffffffff8166dbc5>] target_message+0x255/0x340
Mar 2 16:06:50 cvknode129 kernel: [68389.843116] [<ffffffff8166d970>] ? __dev_status+0x150/0x150
Mar 2 16:06:50 cvknode129 kernel: [68389.843118] [<ffffffff8166ef7a>] ctl_ioctl+0x24a/0x520
Mar 2 16:06:50 cvknode129 kernel: [68389.843120] [<ffffffff8166f263>] dm_ctl_ioctl+0x13/0x20
Mar 2 16:06:50 cvknode129 kernel: [68389.843122] [<ffffffff812142d6>] do_vfs_ioctl+0x86/0x530
Mar 2 16:06:50 cvknode129 kernel: [68389.843124] [<ffffffff8106a10f>] ? __do_page_fault+0x1af/0x470
Mar 2 16:06:50 cvknode129 kernel: [68389.843126] [<ffffffff81214811>] SyS_ioctl+0x91/0xb0
Mar 2 16:06:50 cvknode129 kernel: [68389.843128] [<ffffffff817f0872>] system_call_fastpath+0x16/0x75
Mar 2 16:06:50 cvknode129 kernel: [68389.843255] sd 5:0:0:0: rejecting I/O to offline device
Mar 2 16:06:50 cvknode129 kernel: [68389.843266] device-mapper: multipath: Failing path 8:48. 3514
device {
vendor "MacroSAN"
product "LU"
path_grouping_policy "group_by_prio"
path_checker "tur"
features "1 queue_if_no_path"
hardware_handler "0"
prio "alua"
failback 15
rr_weight "priorities"
no_path_retry 30
rr_min_io 1000
}
3600b34223c09334d0467dfcd5d0000da dm-0 MacroSAN,LU
size=50G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| `- 5:0:0:0 sdd 8:48 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
`- 3:0:0:0 sdb 8:16 active ready running
Thanks!
Best regards!
________________________________
zhangguanghui
-------------------------------------------------------------------------------------------------------------------------------------
????????????????????????????????????????
????????????????????????????????????????
????????????????????????????????????????
???
This e-mail and its attachments contain confidential information from H3C, which is
intended only for the person or entity whose address is listed above. Any use of the
information contained herein in any way (including, but not limited to, total or partial
disclosure, reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender
by phone or email immediately and delete it!
--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
diff --git a/libmultipath/propsel.c b/libmultipath/propsel.c index f64d5e4..890d4b1 100644 --- a/libmultipath/propsel.c +++ b/libmultipath/propsel.c @@ -373,9 +373,11 @@ detect_prio(struct path * pp) { int ret; struct prio *p = &pp->prio; + int tpgs = 0; - if (get_target_port_group_support(pp->fd) <= 0) + if ((tpgs = get_target_port_group_support(pp->fd)) <= 0) return; + pp->tpgs = tpgs; ret = get_target_port_group(pp->fd); if (ret < 0) return; @@ -415,6 +417,15 @@ select_prio (struct path * pp) prio_get(p, DEFAULT_PRIO, DEFAULT_PRIO_ARGS); origin = "(internal default)"; out: + /* + * fetch tpgs mode for alua, if its not already obtained + */ + if (!strncmp(prio_name(p), PRIO_ALUA, PRIO_NAME_LEN)) { + int tpgs = 0; + if(!pp->tpgs && + (tpgs = get_target_port_group_support(pp->fd)) >= 0) + pp->tpgs = tpgs; + } condlog(3, "%s: prio = %s %s", pp->dev, prio_name(p), origin); condlog(3, "%s: prio args = \"%s\" %s", pp->dev, prio_args(p), origin); return 0; diff --git a/libmultipath/structs.h b/libmultipath/structs.h index 5f68a8e..ef5fb7e 100644 --- a/libmultipath/structs.h +++ b/libmultipath/structs.h @@ -193,6 +193,7 @@ struct path { int detect_prio; int watch_checks; int wait_checks; + int tpgs; char * uid_attribute; char * getuid; struct prio prio; diff --git a/multipathd/main.c b/multipathd/main.c index 04f6d02..b6b0053 100644 --- a/multipathd/main.c +++ b/multipathd/main.c @@ -62,6 +62,7 @@ static int use_watchdog; #include <pgpolicies.h> #include <uevent.h> #include <log.h> +#include "prioritizers/alua_rtpg.h" #include "main.h" #include "pidfile.h" @@ -1161,6 +1162,7 @@ check_path (struct vectors * vecs, struct path * pp)
multipathd treats "stand-by" path as active(ghost) and reinstate path.This causes I/O hang issues and lots of "change" udev events in cases where only stand-by paths are present in multipath map and target supports only implicit tpgs mode(active/passive arrays) This can happen during system boot where only stand-by paths are discovered first and continuous retry of I/O's by dm-multipath and change(failed) events are hogging multipathd and slowing down the entire boot process with large number of volumes mapped (~100s). Selecting path checkers other than tur is not a solution as well, as they will continuously throw messages saying paths are failed for stand-by state. This patch will prevent re-instate of stand-by paths in this situation and thus prevent unnecessary i/o with failed events during boot or when active array goes down temporarily. With this, for targets supporting only alua and implicit tpgs mode, stand-by paths are not reinstated by path checker unless there is an active path in the map. Detailed steps to hit the I/O hang issue even with no_path_retry: devices { device { vendor "Nimble" product "Server" path_grouping_policy group_by_prio prio alua hardware_handler "1 alua" path_checker tur failback immediate fast_io_fail_tmo 10 no_path_retry 30 path_selector "round-robin 0" } } 1. Delete all active paths. mpathal (2e0176ad6309077166c9ce90033bfa248_1) dm-2 Nimble,Server size=20G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 8:0:5:1 sdg 8:96 active ghost running `- 8:0:6:1 sdh 8:112 active ghost running 2. Issue I/O on mpath with zero active paths. dd if=/dev/mapper/mpathal of=/dev/null bs=512 count=1 iflag=direct & Nov 23 15:10:36 hitdev-rhel67 multipathd: mpathal: sdg - tur checker reports path is in standby state Nov 23 15:10:36 hitdev-rhel67 multipathd: 8:96: reinstated Nov 23 15:10:36 hitdev-rhel67 multipathd: mpathal: queue_if_no_path enabled Nov 23 15:10:36 hitdev-rhel67 multipathd: mpathal: Recovered to normal mode Nov 23 15:10:36 hitdev-rhel67 multipathd: mpathal: remaining active paths:1 Nov 23 15:10:36 hitdev-rhel67 kernel: sd 8:0:5:1: alua: port group 02 state S non-preferred supports tolusna Nov 23 15:10:36 hitdev-rhel67 kernel: device-mapper: multipath: Failing path 8:96. Nov 23 15:10:36 hitdev-rhel67 multipathd: 8:96: mark as failed Nov 23 15:10:36 hitdev-rhel67 multipathd: mpathal: Entering recovery mode: max_retries=20 Nov 23 15:10:36 hitdev-rhel67 multipathd: mpathal: remaining active paths: 0 3. Monitor udev events every 5 seconds. # udevadm monitor monitor will print the received events for: monitor will print the received events for: UDEV - the event which udev sends out after rule processing KERNEL - the kernel uevent KERNEL[1448320131.061837] change /devices/virtual/block/dm-2 (block) KERNEL[1448320131.064802] change /devices/virtual/block/dm-2 (block) UDEV [1448320131.082838] change /devices/virtual/block/dm-2 (block) UDEV [1448320131.102134] change /devices/virtual/block/dm-2 (block) KERNEL[1448320135.551737] change /devices/virtual/block/dm-2 (block) KERNEL[1448320135.552701] change /devices/virtual/block/dm-2 (block) UDEV [1448320135.571634] change /devices/virtual/block/dm-2 (block) UDEV [1448320135.591017] change /devices/virtual/block/dm-2 (block) KERNEL[1448320136.553368] change /devices/virtual/block/dm-2 (block) KERNEL[1448320136.554298] change /devices/virtual/block/dm-2 (block) UDEV [1448320136.572733] change /devices/virtual/block/dm-2 (block) UDEV [1448320136.592089] change /devices/virtual/block/dm-2 (block) KERNEL[1448320140.555389] change /devices/virtual/block/dm-2 (block) KERNEL[1448320140.556369] change /devices/virtual/block/dm-2 (block) UDEV [1448320140.574944] change /devices/virtual/block/dm-2 (block) UDEV [1448320140.594076] change /devices/virtual/block/dm-2 (block) Signed-off-by: ShivaKrishna Merla <shiva.krishna@nimblestorage.com> --- libmultipath/propsel.c | 13 ++++++++++++- libmultipath/structs.h | 1 + multipathd/main.c | 19 ++++++++++++++++--- 3 files changed, 29 insertions(+), 4 deletions(-) int new_path_up = 0; int chkr_new_path_up = 0; int add_active; + int ignore_reinstate = 0; int oldchkrstate = pp->chkrstate; if (pp->initialized && !pp->mpp) @@ -1235,6 +1237,16 @@ check_path (struct vectors * vecs, struct path * pp) pp->wait_checks = 0; } + /* + * don't reinstate failed path, if its in stand-by + * and if target supports only implicit tpgs mode. + * this will prevent unnecessary i/o by dm on stand-by + * paths if there are no other active paths in map. + */ + ignore_reinstate = (newstate == PATH_GHOST && + pp->mpp->nr_active == 0 && + pp->tpgs == TPGS_IMPLICIT) ? 1 : 0; + pp->chkrstate = newstate; if (newstate != pp->state) { int oldstate = pp->state; @@ -1297,7 +1309,7 @@ check_path (struct vectors * vecs, struct path * pp) pp->watch_checks--; add_active = 0; } - if (reinstate_path(pp, add_active)) { + if (!ignore_reinstate && reinstate_path(pp, add_active)) { condlog(3, "%s: reload map", pp->dev); ev_add_path(pp, vecs); pp->tick = 1; @@ -1316,8 +1328,9 @@ check_path (struct vectors * vecs, struct path * pp) enable_group(pp); } else if (newstate == PATH_UP || newstate == PATH_GHOST) { - if (pp->dmstate == PSTATE_FAILED || - pp->dmstate == PSTATE_UNDEF) { + if ((pp->dmstate == PSTATE_FAILED || + pp->dmstate == PSTATE_UNDEF) && + !ignore_reinstate) { /* Clear IO errors */ if (reinstate_path(pp, 0)) { condlog(3, "%s: reload map", pp->dev); -- -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel