| Message ID | 20140702220223.GA23894@redhat.com (mailing list archive) |
|---|---|
| State | Superseded, archived |
| Delegated to: | Mike Snitzer |
| Headers | show |
On 07/03/2014 12:02 AM, Mike Snitzer wrote: > On Fri, Jun 27 2014 at 9:33am -0400, > Mike Snitzer <snitzer@redhat.com> wrote: > >> On Fri, Jun 27 2014 at 9:02am -0400, >> Bart Van Assche <bvanassche@acm.org> wrote: >> >>> Hello, >>> >>> While running a cable pull simulation test with dm_multipath on top of >>> the SRP initiator driver I noticed that after a few iterations I/O locks >>> up instead of dm_multipath processing the path failure properly (see also >>> below for a call trace). At least kernel versions 3.15 and 3.16-rc2 are >>> vulnerable. This issue does not occur with kernel 3.14. I have tried to >>> bisect this but gave up when I noticed that I/O locked up completely with >>> a kernel built from git commit ID e809917735ebf1b9a56c24e877ce0d320baee2ec >>> (dm mpath: push back requests instead of queueing). But with the bisect I >>> have been able to narrow down this issue to one of the patches in "Merge >>> tag 'dm-3.15-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/ >>> device-mapper/linux-dm". Does anyone have a suggestion how to analyze this >>> further or how to fix this ? > > I still don't have a _known_ fix for your issue but I reviewed commit > e809917735ebf1b9a56c24e877ce0d320baee2ec closer and identified what > looks to be a regression in logic for multipath_busy, it now calls > !pg_ready() instead of directly checking pg_init_in_progress. I think > this is needed (Hannes, what do you think?): > > diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c > index 3f6fd9d..561ead6 100644 > --- a/drivers/md/dm-mpath.c > +++ b/drivers/md/dm-mpath.c > @@ -373,7 +373,7 @@ static int __must_push_back(struct multipath *m) > dm_noflush_suspending(m->ti))); > } > > -#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required) > +#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required && !(m)->pg_init_in_progress) > > /* > * Map cloned requests > D'oh. You're correct. Can you try with this patch? Cheers, Hannes
On 07/03/14 00:02, Mike Snitzer wrote: > On Fri, Jun 27 2014 at 9:33am -0400, > Mike Snitzer <snitzer@redhat.com> wrote: > >> On Fri, Jun 27 2014 at 9:02am -0400, >> Bart Van Assche <bvanassche@acm.org> wrote: >> >>> Hello, >>> >>> While running a cable pull simulation test with dm_multipath on top of >>> the SRP initiator driver I noticed that after a few iterations I/O locks >>> up instead of dm_multipath processing the path failure properly (see also >>> below for a call trace). At least kernel versions 3.15 and 3.16-rc2 are >>> vulnerable. This issue does not occur with kernel 3.14. I have tried to >>> bisect this but gave up when I noticed that I/O locked up completely with >>> a kernel built from git commit ID e809917735ebf1b9a56c24e877ce0d320baee2ec >>> (dm mpath: push back requests instead of queueing). But with the bisect I >>> have been able to narrow down this issue to one of the patches in "Merge >>> tag 'dm-3.15-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/ >>> device-mapper/linux-dm". Does anyone have a suggestion how to analyze this >>> further or how to fix this ? > > I still don't have a _known_ fix for your issue but I reviewed commit > e809917735ebf1b9a56c24e877ce0d320baee2ec closer and identified what > looks to be a regression in logic for multipath_busy, it now calls > !pg_ready() instead of directly checking pg_init_in_progress. I think > this is needed (Hannes, what do you think?): > > diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c > index 3f6fd9d..561ead6 100644 > --- a/drivers/md/dm-mpath.c > +++ b/drivers/md/dm-mpath.c > @@ -373,7 +373,7 @@ static int __must_push_back(struct multipath *m) > dm_noflush_suspending(m->ti))); > } > > -#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required) > +#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required && !(m)->pg_init_in_progress) > > /* > * Map cloned requests Hello Mike, Sorry but even with this patch applied and additionally with commit IDs 86d56134f1b6 ("kobject: Make support for uevent_helper optional") and bcccff93af35 ("kobject: don't block for each kobject_uevent") reverted my multipath test still hangs after a few iterations. I also reran the same test with kernel 3.14.3 and it is still running after 30 iterations. Bart. -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
On 07/03/2014 03:56 PM, Bart Van Assche wrote: > On 07/03/14 00:02, Mike Snitzer wrote: >> On Fri, Jun 27 2014 at 9:33am -0400, >> Mike Snitzer <snitzer@redhat.com> wrote: >> >>> On Fri, Jun 27 2014 at 9:02am -0400, >>> Bart Van Assche <bvanassche@acm.org> wrote: >>> >>>> Hello, >>>> >>>> While running a cable pull simulation test with dm_multipath on top of >>>> the SRP initiator driver I noticed that after a few iterations I/O locks >>>> up instead of dm_multipath processing the path failure properly (see also >>>> below for a call trace). At least kernel versions 3.15 and 3.16-rc2 are >>>> vulnerable. This issue does not occur with kernel 3.14. I have tried to >>>> bisect this but gave up when I noticed that I/O locked up completely with >>>> a kernel built from git commit ID e809917735ebf1b9a56c24e877ce0d320baee2ec >>>> (dm mpath: push back requests instead of queueing). But with the bisect I >>>> have been able to narrow down this issue to one of the patches in "Merge >>>> tag 'dm-3.15-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/ >>>> device-mapper/linux-dm". Does anyone have a suggestion how to analyze this >>>> further or how to fix this ? >> >> I still don't have a _known_ fix for your issue but I reviewed commit >> e809917735ebf1b9a56c24e877ce0d320baee2ec closer and identified what >> looks to be a regression in logic for multipath_busy, it now calls >> !pg_ready() instead of directly checking pg_init_in_progress. I think >> this is needed (Hannes, what do you think?): >> >> diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c >> index 3f6fd9d..561ead6 100644 >> --- a/drivers/md/dm-mpath.c >> +++ b/drivers/md/dm-mpath.c >> @@ -373,7 +373,7 @@ static int __must_push_back(struct multipath *m) >> dm_noflush_suspending(m->ti))); >> } >> >> -#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required) >> +#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required && !(m)->pg_init_in_progress) >> >> /* >> * Map cloned requests > > Hello Mike, > > Sorry but even with this patch applied and additionally with commit IDs > 86d56134f1b6 ("kobject: Make support for uevent_helper optional") and > bcccff93af35 ("kobject: don't block for each kobject_uevent") reverted > my multipath test still hangs after a few iterations. I also reran the > same test with kernel 3.14.3 and it is still running after 30 iterations. > Hmm. Would've been too easy. Sigh. Cheers, Hannes
On Thu, Jul 03 2014 at 9:56am -0400, Bart Van Assche <bvanassche@acm.org> wrote: > On 07/03/14 00:02, Mike Snitzer wrote: > > On Fri, Jun 27 2014 at 9:33am -0400, > > Mike Snitzer <snitzer@redhat.com> wrote: > > > >> On Fri, Jun 27 2014 at 9:02am -0400, > >> Bart Van Assche <bvanassche@acm.org> wrote: > >> > >>> Hello, > >>> > >>> While running a cable pull simulation test with dm_multipath on top of > >>> the SRP initiator driver I noticed that after a few iterations I/O locks > >>> up instead of dm_multipath processing the path failure properly (see also > >>> below for a call trace). At least kernel versions 3.15 and 3.16-rc2 are > >>> vulnerable. This issue does not occur with kernel 3.14. I have tried to > >>> bisect this but gave up when I noticed that I/O locked up completely with > >>> a kernel built from git commit ID e809917735ebf1b9a56c24e877ce0d320baee2ec > >>> (dm mpath: push back requests instead of queueing). But with the bisect I > >>> have been able to narrow down this issue to one of the patches in "Merge > >>> tag 'dm-3.15-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/ > >>> device-mapper/linux-dm". Does anyone have a suggestion how to analyze this > >>> further or how to fix this ? > > > > I still don't have a _known_ fix for your issue but I reviewed commit > > e809917735ebf1b9a56c24e877ce0d320baee2ec closer and identified what > > looks to be a regression in logic for multipath_busy, it now calls > > !pg_ready() instead of directly checking pg_init_in_progress. I think > > this is needed (Hannes, what do you think?): > > > > diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c > > index 3f6fd9d..561ead6 100644 > > --- a/drivers/md/dm-mpath.c > > +++ b/drivers/md/dm-mpath.c > > @@ -373,7 +373,7 @@ static int __must_push_back(struct multipath *m) > > dm_noflush_suspending(m->ti))); > > } > > > > -#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required) > > +#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required && !(m)->pg_init_in_progress) > > > > /* > > * Map cloned requests > > Hello Mike, > > Sorry but even with this patch applied and additionally with commit IDs > 86d56134f1b6 ("kobject: Make support for uevent_helper optional") and > bcccff93af35 ("kobject: don't block for each kobject_uevent") reverted > my multipath test still hangs after a few iterations. I also reran the > same test with kernel 3.14.3 and it is still running after 30 iterations. OK, thanks for testing though! I still think the patch is needed. You are using queue_if_no_path, do you see hangs due to paths not being restored after the "cable" is restored? Any errors in the multipathd userspace logging? Or abnormal errors in kernel? Basically I'm looking for some other clue besides the hung task timeout spew. How easy would it be to replicate your testbed? Is it uniquely FIO hw dependent? How are you simulating the cable pull tests? I'd love to setup a testbed that would enable me to chase this more interactively rather than punting to you for testing. Hannes, do you have a testbed for heavy cable pull testing? Are you able to replicate these hangs? -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
On 07/03/2014 04:05 PM, Mike Snitzer wrote: > On Thu, Jul 03 2014 at 9:56am -0400, > Bart Van Assche <bvanassche@acm.org> wrote: > >> On 07/03/14 00:02, Mike Snitzer wrote: >>> On Fri, Jun 27 2014 at 9:33am -0400, >>> Mike Snitzer <snitzer@redhat.com> wrote: >>> >>>> On Fri, Jun 27 2014 at 9:02am -0400, >>>> Bart Van Assche <bvanassche@acm.org> wrote: >>>> >>>>> Hello, >>>>> >>>>> While running a cable pull simulation test with dm_multipath on top of >>>>> the SRP initiator driver I noticed that after a few iterations I/O locks >>>>> up instead of dm_multipath processing the path failure properly (see also >>>>> below for a call trace). At least kernel versions 3.15 and 3.16-rc2 are >>>>> vulnerable. This issue does not occur with kernel 3.14. I have tried to >>>>> bisect this but gave up when I noticed that I/O locked up completely with >>>>> a kernel built from git commit ID e809917735ebf1b9a56c24e877ce0d320baee2ec >>>>> (dm mpath: push back requests instead of queueing). But with the bisect I >>>>> have been able to narrow down this issue to one of the patches in "Merge >>>>> tag 'dm-3.15-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/ >>>>> device-mapper/linux-dm". Does anyone have a suggestion how to analyze this >>>>> further or how to fix this ? >>> >>> I still don't have a _known_ fix for your issue but I reviewed commit >>> e809917735ebf1b9a56c24e877ce0d320baee2ec closer and identified what >>> looks to be a regression in logic for multipath_busy, it now calls >>> !pg_ready() instead of directly checking pg_init_in_progress. I think >>> this is needed (Hannes, what do you think?): >>> >>> diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c >>> index 3f6fd9d..561ead6 100644 >>> --- a/drivers/md/dm-mpath.c >>> +++ b/drivers/md/dm-mpath.c >>> @@ -373,7 +373,7 @@ static int __must_push_back(struct multipath *m) >>> dm_noflush_suspending(m->ti))); >>> } >>> >>> -#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required) >>> +#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required && !(m)->pg_init_in_progress) >>> >>> /* >>> * Map cloned requests >> >> Hello Mike, >> >> Sorry but even with this patch applied and additionally with commit IDs >> 86d56134f1b6 ("kobject: Make support for uevent_helper optional") and >> bcccff93af35 ("kobject: don't block for each kobject_uevent") reverted >> my multipath test still hangs after a few iterations. I also reran the >> same test with kernel 3.14.3 and it is still running after 30 iterations. > > OK, thanks for testing though! I still think the patch is needed. > > You are using queue_if_no_path, do you see hangs due to paths not being > restored after the "cable" is restored? Any errors in the multipathd > userspace logging? Or abnormal errors in kernel? Basically I'm looking > for some other clue besides the hung task timeout spew. > > How easy would it be to replicate your testbed? Is it uniquely FIO hw > dependent? How are you simulating the cable pull tests? > > I'd love to setup a testbed that would enable me to chase this more > interactively rather than punting to you for testing. > > Hannes, do you have a testbed for heavy cable pull testing? Are you > able to replicate these hangs? > Yes, I do. But sadly I've been tied up with polishing up SLES12 (release deadline is looming nearer) and for some inexplicable reason management seems to find releasing a product more important than working on mainline issue ... But I hope to find some time soonish (ie start of next week) to work on this; it's the very next thing on my to-do list. Cheers, Hannes
On Thu, Jul 03 2014 at 10:15am -0400, Hannes Reinecke <hare@suse.de> wrote: > On 07/03/2014 04:05 PM, Mike Snitzer wrote: > > > >Hannes, do you have a testbed for heavy cable pull testing? Are you > >able to replicate these hangs? > > > Yes, I do. But sadly I've been tied up with polishing up SLES12 > (release deadline is looming nearer) and for some inexplicable > reason management seems to find releasing a product more important > than working on mainline issue ... OK, so long as you didn't include these dm-mpath changes.. but if you did then you could kill 2 birds with one stone ;) > But I hope to find some time soonish (ie start of next week) to work > on this; it's the very next thing on my to-do list. OK, I really appreciate it, thanks. -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
On 07/03/14 16:05, Mike Snitzer wrote: > How easy would it be to replicate your testbed? Is it uniquely FIO hw > dependent? How are you simulating the cable pull tests? > > I'd love to setup a testbed that would enable me to chase this more > interactively rather than punting to you for testing. Hello Mike, The only nonstandard hardware that is required to run my test is a pair of InfiniBand HCA's and an IB cable to connect these back-to-back. The test I ran is as follows: * Let an SRP initiator log in to an SRP target system. * Start multipathd and srpd. * Start a fio data integrity test on the initiator system on top of /dev/dm-0. * From the target system simulate a cable pull by disabling IB traffic via the ibportstate command. * After a random delay, unload and reload SCST and the IB stack. This makes the IB ports operational again. * After a random delay, repeat the previous two steps. If you want I can send you the scripts I use to run this test and also the instructions that are necessary to build and install the SCST SRP target driver. Bart. -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
On Thu, Jul 03 2014 at 10:34am -0400, Bart Van Assche <bvanassche@acm.org> wrote: > On 07/03/14 16:05, Mike Snitzer wrote: > > How easy would it be to replicate your testbed? Is it uniquely FIO hw > > dependent? How are you simulating the cable pull tests? > > > > I'd love to setup a testbed that would enable me to chase this more > > interactively rather than punting to you for testing. > > Hello Mike, > > The only nonstandard hardware that is required to run my test is a pair > of InfiniBand HCA's and an IB cable to connect these back-to-back. The > test I ran is as follows: > * Let an SRP initiator log in to an SRP target system. > * Start multipathd and srpd. > * Start a fio data integrity test on the initiator system on top of > /dev/dm-0. > * From the target system simulate a cable pull by disabling IB traffic > via the ibportstate command. > * After a random delay, unload and reload SCST and the IB stack. This > makes the IB ports operational again. > * After a random delay, repeat the previous two steps. I'll work on getting some IB cards. But I _should_ be able to achieve the same using iSCSI right? > If you want I can send you the scripts I use to run this test and also > the instructions that are necessary to build and install the SCST SRP > target driver. Please do, thanks! Also, Red Hat has a rather extensive battery of dm-mpath FC cable pull tests for RHEL but these dm-mpath changes haven't been included in any RHEL yet.. I _could_ port these upstream changes to a test RHEL7 kernel just to leverage the RHEL-based testing. -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
On 07/03/14 23:34, Bart Van Assche wrote: > On 07/03/14 16:05, Mike Snitzer wrote: > The only nonstandard hardware that is required to run my test is a pair > of InfiniBand HCA's and an IB cable to connect these back-to-back. The > test I ran is as follows: > * Let an SRP initiator log in to an SRP target system. > * Start multipathd and srpd. > * Start a fio data integrity test on the initiator system on top of > /dev/dm-0. > * From the target system simulate a cable pull by disabling IB traffic > via the ibportstate command. > * After a random delay, unload and reload SCST and the IB stack. This > makes the IB ports operational again. > * After a random delay, repeat the previous two steps. > > If you want I can send you the scripts I use to run this test and also > the instructions that are necessary to build and install the SCST SRP > target driver. Hi Bart, could you collect output of 'dmsetup table' on the test machine? My particular interest is whether pg_init is involved or not since Hannes's patch made major surgery on pg_init processing. Also, output of the followings while the problem is happening might help: dmsetup status dmsetup info -c cat /proc/diskstat multipathd -k"show paths" multipathd -k"show maps status" multipathd -k"show maps stats"
On 07/03/14 17:00, Mike Snitzer wrote: > On Thu, Jul 03 2014 at 10:34am -0400, > Bart Van Assche <bvanassche@acm.org> wrote: > >> On 07/03/14 16:05, Mike Snitzer wrote: >>> How easy would it be to replicate your testbed? Is it uniquely FIO hw >>> dependent? How are you simulating the cable pull tests? >>> >>> I'd love to setup a testbed that would enable me to chase this more >>> interactively rather than punting to you for testing. >> >> Hello Mike, >> >> The only nonstandard hardware that is required to run my test is a pair >> of InfiniBand HCA's and an IB cable to connect these back-to-back. The >> test I ran is as follows: >> * Let an SRP initiator log in to an SRP target system. >> * Start multipathd and srpd. >> * Start a fio data integrity test on the initiator system on top of >> /dev/dm-0. >> * From the target system simulate a cable pull by disabling IB traffic >> via the ibportstate command. >> * After a random delay, unload and reload SCST and the IB stack. This >> makes the IB ports operational again. >> * After a random delay, repeat the previous two steps. > > I'll work on getting some IB cards. But I _should_ be able to achieve > the same using iSCSI right? I'm not sure. There are differences between the SRP and iSCSI initiator that could matter here, e.g. that the SRP initiator triggers scsi_remove_host() some time after a path failure occurred but the iSCSI initiator not. So far I have not yet been able to trigger this issue with the iSCSI initiator with replacement_timeout = 1 and by using the following loop to simulate path failures: while true; do iptables -A INPUT -p tcp --destination-port 3260 -j DROP; sleep 10; iptables -D INPUT -p tcp --destination-port 3260 -j DROP; sleep 10; done >> If you want I can send you the scripts I use to run this test and also >> the instructions that are necessary to build and install the SCST SRP >> target driver. > > Please do, thanks! The test I run at the initiator side is as follows: # modprobe ib_srp # systemctl restart srpd # systemctl start multipathd # mkfs.ext4 -FO ^has_journal /dev/dm-0 # umount /mnt; fsck /dev/dm-0 && mount /dev/dm-0 /mnt && rm -f /mnt/test* && fio --verify=md5 --rw=randwrite --size=10M --bs=4K --iodepth=64 --sync=1 --direct=1 --ioengine=libaio --directory=/mnt --name=test --thread --numjobs=1 --loops=$((10**9)) The script I run at the target side is as follows (should also be possible with the upstream SRP target driver instead of SCST): * Download, build and install SCST. * Create a configuration file (/etc/scst.conf) in which /dev/ram0 is exported via the vdisk_blockio driver. * Start SCST. * Run the attached toggle-ib-port-loop script e.g. as follows: initiator=${initiator_host_name} toggle-ib-port-loop Bart. #!/bin/bash # How to start this test. # On the initiator system, run: # ~bart/bin/reload-srp-initiator # /etc/init.d/srpd start # mkfs.ext4 -O ^has_journal /dev/sdb # /etc/init.d/multipathd start # umount /mnt; mount /dev/dm-0 /mnt && rm -f /mnt/test* && ~bart/bin/fio-stress-test-6 /mnt 16 # On the target system, run: # initiator=antec ~bart/software/tools/toggle-ib-port-loop function port_guid() { local gid guid gid="$(</sys/class/infiniband/mlx4_0/ports/$1/gids/0)" || return $? guid="${gid#fe80:0000:0000:0000}" echo "0x${guid//:/}" } if [ -z "${initiator}" ]; then echo "Error: variable \${initiator} has not been set" exit 1 fi guid1="$(port_guid 1)" guid2="$(port_guid 2)" set -x /etc/init.d/srpd stop while true; do ssh ${initiator} ibportstate -G "$guid1" 1 disable ssh ${initiator} ibportstate -G "$guid2" 2 disable sleep $((RANDOM*150/32767)) /etc/init.d/scst stop /etc/init.d/opensmd stop /etc/init.d/openibd stop for m in mlx4_en mlx4_ib mlx4_core; do modprobe -r $m done /etc/init.d/openibd start /etc/init.d/opensmd start umount /dev/sr1 ibstat | sed -n 's/^[[:blank:]]*Port GUID: 0x\(..\)\(..\)\(..\)....\(..\)\(..\)\(..\)/00:\2:\3:\4:\5:\6/p' | while read a; do p="$(cd /sys/class/net && grep -lw $a */address)" if [ -n "$p" ]; then ifup "$(dirname $p)" fi done /etc/init.d/scst restart sleep $((30 + RANDOM*30/32767)) done -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
On 07/04/14 05:10, Junichi Nomura wrote: > could you collect output of 'dmsetup table' on the test machine? > My particular interest is whether pg_init is involved or not > since Hannes's patch made major surgery on pg_init processing. > > Also, output of the followings while the problem is happening might help: > dmsetup status > dmsetup info -c > cat /proc/diskstat > multipathd -k"show paths" > multipathd -k"show maps status" > multipathd -k"show maps stats" Hello Junichi, Thanks for looking into this issue. I have attached the requested information to this e-mail for a test run with kernel 3.16-rc4 at the initiator side. Bart. # dmsetup table 26133373139386639: 0 5368709120 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 65:16 1000 1 65:192 1000 1 26230643330303865: 0 5368709120 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 65:48 1000 1 65:208 1000 1 23834333461383137: 0 5368709120 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:208 1000 1 65:128 1000 1 26361626563396362: 0 5368709120 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 65:80 1000 1 65:224 1000 1 26363633765623538: 0 256000 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:32 1000 1 8:128 1000 1 23832346438613834: 0 256000 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:96 1000 1 65:0 1000 1 26638323034336331: 0 256000 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:80 1000 1 8:224 1000 1 23937393633303630: 0 5368709120 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:240 1000 1 65:160 1000 1 23931656631326633: 0 256000 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:112 1000 1 65:32 1000 1 26665353936313532: 0 5368709120 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 65:176 1000 1 66:16 1000 1 26135303862613661: 0 256000 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:144 1000 1 65:64 1000 1 26439316335316263: 0 5368709120 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 65:112 1000 1 65:240 1000 1 26564666266393235: 0 5368709120 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 65:144 1000 1 66:0 1000 1 26562383264626236: 0 256000 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:64 1000 1 8:192 1000 1 26236616132323164: 0 256000 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:176 1000 1 65:96 1000 1 26466363537333266: 0 256000 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:48 1000 1 8:160 1000 1 # dmsetup status 26133373139386639: 0 5368709120 multipath 2 0 0 0 1 1 A 0 2 2 65:16 A 0 0 1 65:192 A 0 0 1 26230643330303865: 0 5368709120 multipath 2 0 0 0 1 1 A 0 2 2 65:48 A 0 0 1 65:208 A 0 0 1 23834333461383137: 0 5368709120 multipath 2 1 0 0 1 1 E 0 2 2 8:208 A 0 0 1 65:128 A 0 0 1 26361626563396362: 0 5368709120 multipath 2 1 0 0 1 1 E 0 2 2 65:80 A 0 0 1 65:224 A 0 0 1 26363633765623538: 0 256000 multipath 2 0 0 0 1 1 A 0 2 2 8:32 A 0 0 1 8:128 A 0 0 1 23832346438613834: 0 256000 multipath 2 1 0 0 1 1 E 0 2 2 8:96 A 0 0 1 65:0 A 0 0 1 26638323034336331: 0 256000 multipath 2 1 0 0 1 1 E 0 2 2 8:80 A 0 0 1 8:224 A 0 0 1 23937393633303630: 0 5368709120 multipath 2 0 0 0 1 1 A 0 2 2 8:240 A 0 0 1 65:160 A 0 0 1 23931656631326633: 0 256000 multipath 2 1 0 0 1 1 E 0 2 2 8:112 A 0 0 1 65:32 A 0 0 1 26665353936313532: 0 5368709120 multipath 2 0 0 0 1 1 A 0 2 2 65:176 A 0 0 1 66:16 A 0 0 1 26135303862613661: 0 256000 multipath 2 0 0 0 1 1 A 0 2 2 8:144 A 0 0 1 65:64 A 0 0 1 26439316335316263: 0 5368709120 multipath 2 0 0 0 1 1 A 0 2 2 65:112 A 0 0 1 65:240 A 0 0 1 26564666266393235: 0 5368709120 multipath 2 0 0 0 1 1 A 0 2 2 65:144 A 0 0 1 66:0 A 0 0 1 26562383264626236: 0 256000 multipath 2 1 0 0 1 1 E 0 2 2 8:64 A 0 0 1 8:192 A 0 0 1 26236616132323164: 0 256000 multipath 2 0 0 0 1 1 A 0 2 2 8:176 A 0 0 1 65:96 A 0 0 1 26466363537333266: 0 256000 multipath 2 1 0 0 1 1 E 0 2 2 8:48 A 0 0 1 8:160 A 0 0 1 # dmsetup info -c Name Maj Min Stat Open Targ Event UUID 26133373139386639 253 10 L--w 0 1 0 mpath-26133373139386639 26230643330303865 253 11 L--w 0 1 0 mpath-26230643330303865 23834333461383137 253 9 L--w 1 1 0 mpath-23834333461383137 26361626563396362 253 13 L--w 1 1 0 mpath-26361626563396362 26363633765623538 253 0 L--w 1 1 5 mpath-26363633765623538 23832346438613834 253 4 L--w 1 1 0 mpath-23832346438613834 26638323034336331 253 3 L--w 1 1 0 mpath-26638323034336331 23937393633303630 253 8 L--w 0 1 0 mpath-23937393633303630 23931656631326633 253 5 L--w 1 1 0 mpath-23931656631326633 26665353936313532 253 15 L--w 0 1 0 mpath-26665353936313532 26135303862613661 253 6 L--w 0 1 0 mpath-26135303862613661 26439316335316263 253 12 L--w 0 1 0 mpath-26439316335316263 26564666266393235 253 14 L--w 0 1 0 mpath-26564666266393235 26562383264626236 253 2 L--w 1 1 0 mpath-26562383264626236 26236616132323164 253 7 L--w 0 1 0 mpath-26236616132323164 26466363537333266 253 1 L--w 1 1 0 mpath-26466363537333266 # cat /proc/diskstats 8 0 sda 63494 687 2082482 45080 11530 6225 362768 690500 0 23530 734810 8 1 sda1 170 0 1360 190 0 0 0 0 0 190 190 8 2 sda2 63143 687 2079674 44620 11509 6225 362768 690490 0 23500 734330 8 16 sdb 374 6 3040 460 0 0 0 0 0 460 460 8 17 sdb1 180 0 1440 190 0 0 0 0 0 190 190 11 0 sr0 0 0 0 0 0 0 0 0 0 0 0 253 0 dm-0 283565 1406 2268002 22758620 295490 134 2560844 532030 0 671930 23290480 8 32 sdc 1489 0 11912 30130 1168 0 9320 1020 0 1640 31150 8 48 sdd 362 0 2896 1310 0 0 0 0 0 1310 1310 8 64 sde 347 0 2776 1180 0 0 0 0 0 1170 1180 8 80 sdf 346 0 2768 910 0 0 0 0 0 900 910 8 96 sdg 389 0 3112 1020 0 0 0 0 0 1020 1020 8 112 sdh 417 0 3336 1080 0 0 0 0 0 1070 1080 8 128 sdi 234 0 1872 450 1132 0 9050 1590 0 2030 2040 8 144 sdj 422 0 3400 1050 0 0 0 0 0 1040 1050 8 160 sdk 234 0 1872 520 0 0 0 0 0 520 520 8 176 sdl 421 0 3392 1030 0 0 0 0 0 1020 1030 8 192 sdm 234 0 1872 510 0 0 0 0 0 510 510 8 208 sdn 369 0 2952 880 0 0 0 0 0 880 880 8 224 sdo 234 0 1872 500 0 0 0 0 0 500 500 253 1 dm-1 127 0 1016 1070 0 0 0 0 1 278360 278820 8 240 sdp 425 0 3424 890 0 0 0 0 0 880 890 65 0 sdq 234 0 1872 500 0 0 0 0 0 500 500 65 16 sdr 424 0 3416 910 0 0 0 0 0 910 910 253 2 dm-2 112 0 896 1050 0 0 0 0 1 278240 278710 65 32 sds 233 0 1864 590 0 0 0 0 0 590 590 65 48 sdt 426 0 3432 860 0 0 0 0 0 850 860 65 64 sdu 421 0 3392 830 0 0 0 0 0 830 830 253 3 dm-3 111 0 888 1120 0 0 0 0 1 278170 278700 65 96 sdw 422 0 3400 910 0 0 0 0 0 910 910 65 80 sdv 404 0 3232 900 0 0 0 0 0 900 900 253 4 dm-4 154 0 1232 1310 0 0 0 0 1 278110 278720 65 112 sdx 425 0 3424 960 0 0 0 0 0 960 960 65 128 sdy 235 0 1880 570 0 0 0 0 0 570 570 253 5 dm-5 183 0 1464 1410 0 0 0 0 1 278050 278700 65 144 sdz 425 0 3424 820 0 0 0 0 0 820 820 65 160 sdaa 425 0 3424 830 0 0 0 0 0 830 830 253 6 dm-6 374 6 3040 1750 0 0 0 0 0 940 1750 65 176 sdab 426 0 3432 820 0 0 0 0 0 820 820 65 192 sdac 428 0 3448 830 0 0 0 0 0 830 830 65 208 sdad 425 0 3424 740 0 0 0 0 0 740 740 253 7 dm-7 374 6 3040 1620 0 0 0 0 0 880 1620 65 224 sdae 236 0 1888 500 0 0 0 0 0 500 500 65 240 sdaf 427 0 3440 640 0 0 0 0 0 640 640 66 0 sdag 427 0 3440 530 0 0 0 0 0 530 530 66 16 sdah 426 0 3432 540 0 0 0 0 0 540 540 253 8 dm-8 378 6 3072 1190 0 0 0 0 0 640 1190 253 9 dm-9 134 0 1072 520 0 0 0 0 1 277420 277640 253 10 dm-10 378 6 3072 1180 0 0 0 0 0 660 1180 253 11 dm-11 378 6 3072 1000 0 0 0 0 0 530 1000 253 12 dm-12 378 6 3072 990 0 0 0 0 0 560 990 253 13 dm-13 168 0 1344 500 0 0 0 0 1 276920 277150 253 14 dm-14 378 6 3072 910 0 0 0 0 0 480 910 253 15 dm-15 378 6 3072 950 0 0 0 0 0 500 950 # multipathd -k"show paths" hcil dev dev_t pri dm_st chk_st dev_st next_check 15:0:0:0 sdc 8:32 1 active ready running XX........ 1/4 15:0:0:1 sdd 8:48 1 active ready running XX........ 1/4 15:0:0:2 sde 8:64 1 active ready running XX........ 1/4 15:0:0:3 sdf 8:80 1 active ready running XX........ 1/4 15:0:0:4 sdg 8:96 1 active ready running XX........ 1/4 15:0:0:5 sdh 8:112 1 active ready running XX........ 1/4 15:0:0:6 sdj 8:144 1 active ready running XX........ 1/4 16:0:0:0 sdi 8:128 1 active ready running XX........ 1/4 15:0:0:7 sdl 8:176 1 active ready running XX........ 1/4 16:0:0:1 sdk 8:160 1 active ready running XX........ 1/4 16:0:0:2 sdm 8:192 1 active ready running XX........ 1/4 16:0:0:3 sdo 8:224 1 active ready running XX........ 1/4 15:0:0:11 sdp 8:240 1 active ready running XX........ 1/4 15:0:0:10 sdn 8:208 1 active ready running XX........ 1/4 16:0:0:4 sdq 65:0 1 active ready running XX........ 1/4 15:0:0:12 sdr 65:16 1 active ready running XX........ 1/4 16:0:0:5 sds 65:32 1 active ready running XX........ 1/4 16:0:0:6 sdu 65:64 1 active ready running XX........ 1/4 15:0:0:13 sdt 65:48 1 active ready running XX........ 1/4 16:0:0:10 sdy 65:128 1 active ready running XX........ 1/4 16:0:0:7 sdw 65:96 1 active ready running XX........ 1/4 15:0:0:15 sdx 65:112 1 active ready running XX........ 1/4 15:0:0:14 sdv 65:80 1 active ready running XX........ 1/4 15:0:0:16 sdz 65:144 1 active ready running XX........ 1/4 15:0:0:17 sdab 65:176 1 active ready running XX........ 1/4 16:0:0:12 sdac 65:192 1 active ready running XX........ 1/4 16:0:0:11 sdaa 65:160 1 active ready running XX........ 1/4 16:0:0:13 sdad 65:208 1 active ready running XX........ 1/4 16:0:0:14 sdae 65:224 1 active ready running XX........ 1/4 16:0:0:15 sdaf 65:240 1 active ready running XX........ 1/4 16:0:0:17 sdah 66:16 1 active ready running XX........ 1/4 16:0:0:16 sdag 66:0 1 active ready running XX........ 1/4 # multipathd -k"show maps status" name failback queueing paths dm-st write_prot 26363633765623538 - on 2 active rw 26466363537333266 - on 2 active rw 26562383264626236 - on 2 active rw 26638323034336331 - on 2 active rw 23832346438613834 - on 2 active rw 23931656631326633 - on 2 active rw 26135303862613661 - on 2 active rw 26236616132323164 - on 2 active rw 23937393633303630 - on 2 active rw 23834333461383137 - on 2 active rw 26133373139386639 - on 2 active rw 26230643330303865 - on 2 active rw 26439316335316263 - on 2 active rw 26361626563396362 - on 2 active rw 26564666266393235 - on 2 active rw 26665353936313532 - on 2 active rw # multipathd -k"show maps stats" name path_faults switch_grp map_loads total_q_time q_timeouts 26363633765623538 5 0 9 0 0 26466363537333266 0 0 2 0 0 26562383264626236 0 0 2 0 0 26638323034336331 0 0 2 0 0 23832346438613834 0 0 2 0 0 23931656631326633 0 0 2 0 0 26135303862613661 0 0 2 0 0 26236616132323164 0 0 2 0 0 23937393633303630 0 0 2 0 0 23834333461383137 0 0 2 0 0 26133373139386639 0 0 2 0 0 26230643330303865 0 0 2 0 0 26439316335316263 0 0 2 0 0 26361626563396362 0 0 2 0 0 26564666266393235 0 0 2 0 0 26665353936313532 0 0 2 0 0 SysRq : Show Blocked State task PC stack pid father systemd-udevd D ffff88083b6f8000 0 8283 358 0x00000000 ffff8807c69f3b30 0000000000000002 ffff88083b6f8000 ffff8807c69f3fd8 00000000000131c0 00000000000131c0 ffffffff81a154c0 ffff8807d7578698 ffff8807d75786a0 0000000000000246 ffff88083b6f8000 0000000000000000 Call Trace: [<ffffffff814bc3a3>] schedule_preempt_disabled+0x33/0x80 [<ffffffff814bd3dc>] mutex_lock_nested+0x15c/0x3d0 [<ffffffff811d4b56>] ? __blkdev_get+0x66/0x4c0 [<ffffffff811d4b56>] __blkdev_get+0x66/0x4c0 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff811d5195>] blkdev_get+0x1e5/0x380 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff811d53db>] blkdev_open+0x5b/0x80 [<ffffffff81195fc2>] do_dentry_open.isra.15+0x1e2/0x320 [<ffffffff81196210>] finish_open+0x30/0x40 [<ffffffff811a82cd>] do_last.isra.61+0xa3d/0x1150 [<ffffffff811a4725>] ? path_init+0x5d5/0x750 [<ffffffff810a6900>] ? trace_hardirqs_on_caller+0xc0/0x1c0 [<ffffffff811a8a58>] ? path_openat+0x78/0x620 [<ffffffff811a8a97>] path_openat+0xb7/0x620 [<ffffffff811a967a>] do_filp_open+0x3a/0x90 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50 [<ffffffff811b8a1c>] ? __alloc_fd+0xac/0x1b0 [<ffffffff81197a5e>] do_sys_open+0x12e/0x210 [<ffffffff8101209d>] ? syscall_trace_enter+0xfd/0x200 [<ffffffff81197b5e>] SyS_open+0x1e/0x20 [<ffffffff814c1ea2>] tracesys+0xd0/0xd5 systemd-udevd D ffff88083b322230 0 8317 358 0x00000000 ffff8807cfa27b30 0000000000000002 ffff88083b322230 ffff8807cfa27fd8 00000000000131c0 00000000000131c0 ffff88083b490000 ffff88083ec1a098 ffff88083ec1a0a0 0000000000000246 ffff88083b322230 0000000000000000 Call Trace: [<ffffffff814bc3a3>] schedule_preempt_disabled+0x33/0x80 [<ffffffff814bd3dc>] mutex_lock_nested+0x15c/0x3d0 [<ffffffff811d4b56>] ? __blkdev_get+0x66/0x4c0 [<ffffffff811d4b56>] __blkdev_get+0x66/0x4c0 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff811d5195>] blkdev_get+0x1e5/0x380 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff811d53db>] blkdev_open+0x5b/0x80 [<ffffffff81195fc2>] do_dentry_open.isra.15+0x1e2/0x320 [<ffffffff81196210>] finish_open+0x30/0x40 [<ffffffff811a82cd>] do_last.isra.61+0xa3d/0x1150 [<ffffffff811a4725>] ? path_init+0x5d5/0x750 [<ffffffff810a6900>] ? trace_hardirqs_on_caller+0xc0/0x1c0 [<ffffffff811a8a58>] ? path_openat+0x78/0x620 [<ffffffff811a8a97>] path_openat+0xb7/0x620 [<ffffffff811a967a>] do_filp_open+0x3a/0x90 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50 [<ffffffff811b8a1c>] ? __alloc_fd+0xac/0x1b0 [<ffffffff81197a5e>] do_sys_open+0x12e/0x210 [<ffffffff8101209d>] ? syscall_trace_enter+0xfd/0x200 [<ffffffff81197b5e>] SyS_open+0x1e/0x20 [<ffffffff814c1ea2>] tracesys+0xd0/0xd5 systemd-udevd D ffff88083b00a230 0 8329 358 0x00000000 ffff8807ee1b3b30 0000000000000002 ffff88083b00a230 ffff8807ee1b3fd8 00000000000131c0 00000000000131c0 ffff88083baea230 ffff88083ec1b418 ffff88083ec1b420 0000000000000246 ffff88083b00a230 0000000000000000 Call Trace: [<ffffffff814bc3a3>] schedule_preempt_disabled+0x33/0x80 [<ffffffff814bd3dc>] mutex_lock_nested+0x15c/0x3d0 [<ffffffff811d4b56>] ? __blkdev_get+0x66/0x4c0 [<ffffffff811d4b56>] __blkdev_get+0x66/0x4c0 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff811d5195>] blkdev_get+0x1e5/0x380 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff811d53db>] blkdev_open+0x5b/0x80 [<ffffffff81195fc2>] do_dentry_open.isra.15+0x1e2/0x320 [<ffffffff81196210>] finish_open+0x30/0x40 [<ffffffff811a82cd>] do_last.isra.61+0xa3d/0x1150 [<ffffffff811a4725>] ? path_init+0x5d5/0x750 [<ffffffff810a6900>] ? trace_hardirqs_on_caller+0xc0/0x1c0 [<ffffffff811a8a58>] ? path_openat+0x78/0x620 [<ffffffff811a8a97>] path_openat+0xb7/0x620 [<ffffffff811a967a>] do_filp_open+0x3a/0x90 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50 [<ffffffff811b8a1c>] ? __alloc_fd+0xac/0x1b0 [<ffffffff81197a5e>] do_sys_open+0x12e/0x210 [<ffffffff8101209d>] ? syscall_trace_enter+0xfd/0x200 [<ffffffff81197b5e>] SyS_open+0x1e/0x20 [<ffffffff814c1ea2>] tracesys+0xd0/0xd5 systemd-udevd D ffff8807d1290000 0 8330 358 0x00000000 ffff8807cee23b30 0000000000000002 ffff8807d1290000 ffff8807cee23fd8 00000000000131c0 00000000000131c0 ffff88083b492230 ffff8807d77a6e98 ffff8807d77a6ea0 0000000000000246 ffff8807d1290000 0000000000000000 Call Trace: [<ffffffff814bc3a3>] schedule_preempt_disabled+0x33/0x80 [<ffffffff814bd3dc>] mutex_lock_nested+0x15c/0x3d0 [<ffffffff811d4b56>] ? __blkdev_get+0x66/0x4c0 [<ffffffff811d4b56>] __blkdev_get+0x66/0x4c0 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff811d5195>] blkdev_get+0x1e5/0x380 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff811d53db>] blkdev_open+0x5b/0x80 [<ffffffff81195fc2>] do_dentry_open.isra.15+0x1e2/0x320 [<ffffffff81196210>] finish_open+0x30/0x40 [<ffffffff811a82cd>] do_last.isra.61+0xa3d/0x1150 [<ffffffff811a4725>] ? path_init+0x5d5/0x750 [<ffffffff810a6900>] ? trace_hardirqs_on_caller+0xc0/0x1c0 [<ffffffff811a8a58>] ? path_openat+0x78/0x620 [<ffffffff811a8a97>] path_openat+0xb7/0x620 [<ffffffff811a967a>] do_filp_open+0x3a/0x90 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50 [<ffffffff811b8a1c>] ? __alloc_fd+0xac/0x1b0 [<ffffffff81197a5e>] do_sys_open+0x12e/0x210 [<ffffffff8101209d>] ? syscall_trace_enter+0xfd/0x200 [<ffffffff81197b5e>] SyS_open+0x1e/0x20 [<ffffffff814c1ea2>] tracesys+0xd0/0xd5 systemd-udevd D ffff8807d168c460 0 8334 358 0x00000000 ffff8807d13bfb30 0000000000000002 ffff8807d168c460 ffff8807d13bffd8 00000000000131c0 00000000000131c0 ffff88083b422230 ffff8807d77a5498 ffff8807d77a54a0 0000000000000246 ffff8807d168c460 0000000000000000 Call Trace: [<ffffffff814bc3a3>] schedule_preempt_disabled+0x33/0x80 [<ffffffff814bd3dc>] mutex_lock_nested+0x15c/0x3d0 [<ffffffff811d4b56>] ? __blkdev_get+0x66/0x4c0 [<ffffffff811d4b56>] __blkdev_get+0x66/0x4c0 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff811d5195>] blkdev_get+0x1e5/0x380 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff811d53db>] blkdev_open+0x5b/0x80 [<ffffffff81195fc2>] do_dentry_open.isra.15+0x1e2/0x320 [<ffffffff81196210>] finish_open+0x30/0x40 [<ffffffff811a82cd>] do_last.isra.61+0xa3d/0x1150 [<ffffffff811a4725>] ? path_init+0x5d5/0x750 [<ffffffff810a6900>] ? trace_hardirqs_on_caller+0xc0/0x1c0 [<ffffffff811a8a58>] ? path_openat+0x78/0x620 [<ffffffff811a8a97>] path_openat+0xb7/0x620 [<ffffffff811a967a>] do_filp_open+0x3a/0x90 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50 [<ffffffff811b8a1c>] ? __alloc_fd+0xac/0x1b0 [<ffffffff81197a5e>] do_sys_open+0x12e/0x210 [<ffffffff8101209d>] ? syscall_trace_enter+0xfd/0x200 [<ffffffff81197b5e>] SyS_open+0x1e/0x20 [<ffffffff814c1ea2>] tracesys+0xd0/0xd5 systemd-udevd D ffff8807d1688000 0 8335 358 0x00000000 ffff8807cf5f3b30 0000000000000002 ffff8807d1688000 ffff8807cf5f3fd8 00000000000131c0 00000000000131c0 ffff88083b494460 ffff88083ec1a718 ffff88083ec1a720 0000000000000246 ffff8807d1688000 0000000000000000 Call Trace: [<ffffffff814bc3a3>] schedule_preempt_disabled+0x33/0x80 [<ffffffff814bd3dc>] mutex_lock_nested+0x15c/0x3d0 [<ffffffff811d4b56>] ? __blkdev_get+0x66/0x4c0 [<ffffffff811d4b56>] __blkdev_get+0x66/0x4c0 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff811d5195>] blkdev_get+0x1e5/0x380 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff811d53db>] blkdev_open+0x5b/0x80 [<ffffffff81195fc2>] do_dentry_open.isra.15+0x1e2/0x320 [<ffffffff81196210>] finish_open+0x30/0x40 [<ffffffff811a82cd>] do_last.isra.61+0xa3d/0x1150 [<ffffffff811a4725>] ? path_init+0x5d5/0x750 [<ffffffff810a6900>] ? trace_hardirqs_on_caller+0xc0/0x1c0 [<ffffffff811a8a58>] ? path_openat+0x78/0x620 [<ffffffff811a8a97>] path_openat+0xb7/0x620 [<ffffffff811a967a>] do_filp_open+0x3a/0x90 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50 [<ffffffff811b8a1c>] ? __alloc_fd+0xac/0x1b0 [<ffffffff81197a5e>] do_sys_open+0x12e/0x210 [<ffffffff8101209d>] ? syscall_trace_enter+0xfd/0x200 [<ffffffff81197b5e>] SyS_open+0x1e/0x20 [<ffffffff814c1ea2>] tracesys+0xd0/0xd5 systemd-udevd D ffff88082df38000 0 8341 358 0x00000000 ffff8807d1763b30 0000000000000002 ffff88082df38000 ffff8807d1763fd8 00000000000131c0 00000000000131c0 ffff88083b422230 ffff8807d757e818 ffff8807d757e820 0000000000000246 ffff88082df38000 0000000000000000 Call Trace: [<ffffffff814bc3a3>] schedule_preempt_disabled+0x33/0x80 [<ffffffff814bd3dc>] mutex_lock_nested+0x15c/0x3d0 [<ffffffff811d4b56>] ? __blkdev_get+0x66/0x4c0 [<ffffffff811d4b56>] __blkdev_get+0x66/0x4c0 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff811d5195>] blkdev_get+0x1e5/0x380 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50 [<ffffffff811d53db>] blkdev_open+0x5b/0x80 [<ffffffff81195fc2>] do_dentry_open.isra.15+0x1e2/0x320 [<ffffffff81196210>] finish_open+0x30/0x40 [<ffffffff811a82cd>] do_last.isra.61+0xa3d/0x1150 [<ffffffff811a4725>] ? path_init+0x5d5/0x750 [<ffffffff810a6900>] ? trace_hardirqs_on_caller+0xc0/0x1c0 [<ffffffff811a8a58>] ? path_openat+0x78/0x620 [<ffffffff811a8a97>] path_openat+0xb7/0x620 [<ffffffff811a967a>] do_filp_open+0x3a/0x90 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50 [<ffffffff811b8a1c>] ? __alloc_fd+0xac/0x1b0 [<ffffffff81197a5e>] do_sys_open+0x12e/0x210 [<ffffffff8101209d>] ? syscall_trace_enter+0xfd/0x200 [<ffffffff81197b5e>] SyS_open+0x1e/0x20 [<ffffffff814c1ea2>] tracesys+0xd0/0xd5 blkid D ffff8807d3110000 0 8422 1 0x00000006 ffff8807d59879b8 0000000000000002 ffff8807d3110000 ffff8807d5987fd8 00000000000131c0 00000000000131c0 ffff88083b4c4460 ffff88085fd33ad0 ffff88085ff83038 ffff8807d5987a40 0000000000000002 ffffffff81135db0 Call Trace: [<ffffffff81135db0>] ? wait_on_page_read+0x60/0x60 [<ffffffff814bc0ed>] io_schedule+0x9d/0x130 [<ffffffff81135dbe>] sleep_on_page+0xe/0x20 [<ffffffff814bc738>] __wait_on_bit_lock+0x48/0xb0 [<ffffffff81135eca>] __lock_page+0x6a/0x70 [<ffffffff8109c8e0>] ? autoremove_wake_function+0x40/0x40 [<ffffffff811474cf>] truncate_inode_pages_range+0x3ff/0x690 [<ffffffff81147775>] truncate_inode_pages+0x15/0x20 [<ffffffff811d3425>] kill_bdev+0x35/0x40 [<ffffffff811d49a9>] __blkdev_put+0x69/0x1b0 [<ffffffff811d5450>] blkdev_put+0x50/0x160 [<ffffffff811d5615>] blkdev_close+0x25/0x30 [<ffffffff8119a37a>] __fput+0xea/0x1f0 [<ffffffff8119a4ce>] ____fput+0xe/0x10 [<ffffffff81074ddc>] task_work_run+0xac/0xe0 [<ffffffff8104ff67>] do_exit+0x2c7/0xc60 [<ffffffff81064091>] ? get_signal_to_deliver+0xd1/0x940 [<ffffffff814c118c>] ? _raw_spin_unlock_irq+0x2c/0x60 [<ffffffff81051cac>] do_group_exit+0x4c/0xc0 [<ffffffff810642a1>] get_signal_to_deliver+0x2e1/0x940 [<ffffffff81002528>] do_signal+0x48/0x630 [<ffffffff811d3f77>] ? blkdev_read_iter+0x37/0x40 [<ffffffff8119809e>] ? new_sync_read+0x7e/0xb0 [<ffffffff8127f0d7>] ? debug_smp_processor_id+0x17/0x20 [<ffffffff810ba5d6>] ? rcu_eqs_exit_common.isra.43+0x26/0xe0 [<ffffffff8113539a>] ? context_tracking_user_exit+0x5a/0x1a0 [<ffffffff810a693d>] ? trace_hardirqs_on_caller+0xfd/0x1c0 [<ffffffff81002b81>] do_notify_resume+0x71/0xc0 [<ffffffff814c1f98>] int_signal+0x12/0x17 blkid D ffff8807d3632230 0 8458 1 0x00000006 ffff8807d360f9b8 0000000000000002 ffff8807d3632230 ffff8807d360ffd8 00000000000131c0 00000000000131c0 ffff88083b492230 ffff88085fc93ad0 ffff88085ff7fa38 ffff8807d360fa40 0000000000000002 ffffffff81135db0 Call Trace: [<ffffffff81135db0>] ? wait_on_page_read+0x60/0x60 [<ffffffff814bc0ed>] io_schedule+0x9d/0x130 [<ffffffff81135dbe>] sleep_on_page+0xe/0x20 [<ffffffff814bc738>] __wait_on_bit_lock+0x48/0xb0 [<ffffffff81135eca>] __lock_page+0x6a/0x70 [<ffffffff8109c8e0>] ? autoremove_wake_function+0x40/0x40 [<ffffffff811474cf>] truncate_inode_pages_range+0x3ff/0x690 [<ffffffff81147775>] truncate_inode_pages+0x15/0x20 [<ffffffff811d3425>] kill_bdev+0x35/0x40 [<ffffffff811d49a9>] __blkdev_put+0x69/0x1b0 [<ffffffff811d5450>] blkdev_put+0x50/0x160 [<ffffffff811d5615>] blkdev_close+0x25/0x30 [<ffffffff8119a37a>] __fput+0xea/0x1f0 [<ffffffff8119a4ce>] ____fput+0xe/0x10 [<ffffffff81074ddc>] task_work_run+0xac/0xe0 [<ffffffff8104ff67>] do_exit+0x2c7/0xc60 [<ffffffff81064091>] ? get_signal_to_deliver+0xd1/0x940 [<ffffffff814c118c>] ? _raw_spin_unlock_irq+0x2c/0x60 [<ffffffff81051cac>] do_group_exit+0x4c/0xc0 [<ffffffff810642a1>] get_signal_to_deliver+0x2e1/0x940 [<ffffffff81002528>] do_signal+0x48/0x630 [<ffffffff811d3f77>] ? blkdev_read_iter+0x37/0x40 [<ffffffff8119809e>] ? new_sync_read+0x7e/0xb0 [<ffffffff8127f0d7>] ? debug_smp_processor_id+0x17/0x20 [<ffffffff810ba5d6>] ? rcu_eqs_exit_common.isra.43+0x26/0xe0 [<ffffffff8113539a>] ? context_tracking_user_exit+0x5a/0x1a0 [<ffffffff810a693d>] ? trace_hardirqs_on_caller+0xfd/0x1c0 [<ffffffff81002b81>] do_notify_resume+0x71/0xc0 [<ffffffff814c1f98>] int_signal+0x12/0x17 blkid D ffff8807d4432230 0 8467 1 0x00000006 ffff8807d4daf9b8 0000000000000002 ffff8807d4432230 ffff8807d4daffd8 00000000000131c0 00000000000131c0 ffff88083b4c2230 ffff88085fd53ad0 ffff88085ffa4338 ffff8807d4dafa40 0000000000000002 ffffffff81135db0 Call Trace: [<ffffffff81135db0>] ? wait_on_page_read+0x60/0x60 [<ffffffff814bc0ed>] io_schedule+0x9d/0x130 [<ffffffff81135dbe>] sleep_on_page+0xe/0x20 [<ffffffff814bc738>] __wait_on_bit_lock+0x48/0xb0 [<ffffffff81135eca>] __lock_page+0x6a/0x70 [<ffffffff8109c8e0>] ? autoremove_wake_function+0x40/0x40 [<ffffffff811474cf>] truncate_inode_pages_range+0x3ff/0x690 [<ffffffff81147775>] truncate_inode_pages+0x15/0x20 [<ffffffff811d3425>] kill_bdev+0x35/0x40 [<ffffffff811d49a9>] __blkdev_put+0x69/0x1b0 [<ffffffff811d5450>] blkdev_put+0x50/0x160 [<ffffffff811d5615>] blkdev_close+0x25/0x30 [<ffffffff8119a37a>] __fput+0xea/0x1f0 [<ffffffff8119a4ce>] ____fput+0xe/0x10 [<ffffffff81074ddc>] task_work_run+0xac/0xe0 [<ffffffff8104ff67>] do_exit+0x2c7/0xc60 [<ffffffff81064091>] ? get_signal_to_deliver+0xd1/0x940 [<ffffffff814c118c>] ? _raw_spin_unlock_irq+0x2c/0x60 [<ffffffff81051cac>] do_group_exit+0x4c/0xc0 [<ffffffff810642a1>] get_signal_to_deliver+0x2e1/0x940 [<ffffffff81002528>] do_signal+0x48/0x630 [<ffffffff811d3f77>] ? blkdev_read_iter+0x37/0x40 [<ffffffff8119809e>] ? new_sync_read+0x7e/0xb0 [<ffffffff8127f0d7>] ? debug_smp_processor_id+0x17/0x20 [<ffffffff810ba5d6>] ? rcu_eqs_exit_common.isra.43+0x26/0xe0 [<ffffffff8113539a>] ? context_tracking_user_exit+0x5a/0x1a0 [<ffffffff810a693d>] ? trace_hardirqs_on_caller+0xfd/0x1c0 [<ffffffff81002b81>] do_notify_resume+0x71/0xc0 [<ffffffff814c1f98>] int_signal+0x12/0x17 blkid D ffff8807d275a230 0 8472 1 0x00000006 ffff8807d27bb9b8 0000000000000002 ffff8807d275a230 ffff8807d27bbfd8 00000000000131c0 00000000000131c0 ffff8807d1290000 ffff88085fc93ad0 ffff88085ff87838 ffff8807d27bba40 0000000000000002 ffffffff81135db0 Call Trace: [<ffffffff81135db0>] ? wait_on_page_read+0x60/0x60 [<ffffffff814bc0ed>] io_schedule+0x9d/0x130 [<ffffffff81135dbe>] sleep_on_page+0xe/0x20 [<ffffffff814bc738>] __wait_on_bit_lock+0x48/0xb0 [<ffffffff81135eca>] __lock_page+0x6a/0x70 [<ffffffff8109c8e0>] ? autoremove_wake_function+0x40/0x40 [<ffffffff811474cf>] truncate_inode_pages_range+0x3ff/0x690 [<ffffffff81147775>] truncate_inode_pages+0x15/0x20 [<ffffffff811d3425>] kill_bdev+0x35/0x40 [<ffffffff811d49a9>] __blkdev_put+0x69/0x1b0 [<ffffffff811d5450>] blkdev_put+0x50/0x160 [<ffffffff811d5615>] blkdev_close+0x25/0x30 [<ffffffff8119a37a>] __fput+0xea/0x1f0 [<ffffffff8119a4ce>] ____fput+0xe/0x10 [<ffffffff81074ddc>] task_work_run+0xac/0xe0 [<ffffffff8104ff67>] do_exit+0x2c7/0xc60 [<ffffffff81064091>] ? get_signal_to_deliver+0xd1/0x940 [<ffffffff814c118c>] ? _raw_spin_unlock_irq+0x2c/0x60 [<ffffffff81051cac>] do_group_exit+0x4c/0xc0 [<ffffffff810642a1>] get_signal_to_deliver+0x2e1/0x940 [<ffffffff81002528>] do_signal+0x48/0x630 [<ffffffff811d3f77>] ? blkdev_read_iter+0x37/0x40 [<ffffffff8119809e>] ? new_sync_read+0x7e/0xb0 [<ffffffff8127f0d7>] ? debug_smp_processor_id+0x17/0x20 [<ffffffff810ba5d6>] ? rcu_eqs_exit_common.isra.43+0x26/0xe0 [<ffffffff8113539a>] ? context_tracking_user_exit+0x5a/0x1a0 [<ffffffff810a693d>] ? trace_hardirqs_on_caller+0xfd/0x1c0 [<ffffffff81002b81>] do_notify_resume+0x71/0xc0 [<ffffffff814c1f98>] int_signal+0x12/0x17 blkid D ffff8807c6b10000 0 8482 1 0x00000006 ffff8807d4dbf9b8 0000000000000002 ffff8807c6b10000 ffff8807d4dbffd8 00000000000131c0 00000000000131c0 ffff88082df38000 ffff88085fd73ad0 ffff88085ff74638 ffff8807d4dbfa40 0000000000000002 ffffffff81135db0 Call Trace: [<ffffffff81135db0>] ? wait_on_page_read+0x60/0x60 [<ffffffff814bc0ed>] io_schedule+0x9d/0x130 [<ffffffff81135dbe>] sleep_on_page+0xe/0x20 [<ffffffff814bc738>] __wait_on_bit_lock+0x48/0xb0 [<ffffffff81135eca>] __lock_page+0x6a/0x70 [<ffffffff8109c8e0>] ? autoremove_wake_function+0x40/0x40 [<ffffffff811474cf>] truncate_inode_pages_range+0x3ff/0x690 [<ffffffff81147775>] truncate_inode_pages+0x15/0x20 [<ffffffff811d3425>] kill_bdev+0x35/0x40 [<ffffffff811d49a9>] __blkdev_put+0x69/0x1b0 [<ffffffff811d5450>] blkdev_put+0x50/0x160 [<ffffffff811d5615>] blkdev_close+0x25/0x30 [<ffffffff8119a37a>] __fput+0xea/0x1f0 [<ffffffff8119a4ce>] ____fput+0xe/0x10 [<ffffffff81074ddc>] task_work_run+0xac/0xe0 [<ffffffff8104ff67>] do_exit+0x2c7/0xc60 [<ffffffff81064091>] ? get_signal_to_deliver+0xd1/0x940 [<ffffffff814c118c>] ? _raw_spin_unlock_irq+0x2c/0x60 [<ffffffff81051cac>] do_group_exit+0x4c/0xc0 [<ffffffff810642a1>] get_signal_to_deliver+0x2e1/0x940 [<ffffffff81002528>] do_signal+0x48/0x630 [<ffffffff811d3f77>] ? blkdev_read_iter+0x37/0x40 [<ffffffff8119809e>] ? new_sync_read+0x7e/0xb0 [<ffffffff8127f0d7>] ? debug_smp_processor_id+0x17/0x20 [<ffffffff810ba5d6>] ? rcu_eqs_exit_common.isra.43+0x26/0xe0 [<ffffffff8113539a>] ? context_tracking_user_exit+0x5a/0x1a0 [<ffffffff810a693d>] ? trace_hardirqs_on_caller+0xfd/0x1c0 [<ffffffff81002b81>] do_notify_resume+0x71/0xc0 [<ffffffff814c1f98>] int_signal+0x12/0x17 blkid D ffff8807c6b88000 0 8549 1 0x00000006 ffff8807c6bb39b8 0000000000000002 ffff8807c6b88000 ffff8807c6bb3fd8 00000000000131c0 00000000000131c0 ffff8807d168c460 ffff88085fc33ad0 ffff88085ff88a38 ffff8807c6bb3a40 0000000000000002 ffffffff81135db0 Call Trace: [<ffffffff81135db0>] ? wait_on_page_read+0x60/0x60 [<ffffffff814bc0ed>] io_schedule+0x9d/0x130 [<ffffffff81135dbe>] sleep_on_page+0xe/0x20 [<ffffffff814bc738>] __wait_on_bit_lock+0x48/0xb0 [<ffffffff81135eca>] __lock_page+0x6a/0x70 [<ffffffff8109c8e0>] ? autoremove_wake_function+0x40/0x40 [<ffffffff811474cf>] truncate_inode_pages_range+0x3ff/0x690 [<ffffffff81147775>] truncate_inode_pages+0x15/0x20 [<ffffffff811d3425>] kill_bdev+0x35/0x40 [<ffffffff811d49a9>] __blkdev_put+0x69/0x1b0 [<ffffffff811d5450>] blkdev_put+0x50/0x160 [<ffffffff811d5615>] blkdev_close+0x25/0x30 [<ffffffff8119a37a>] __fput+0xea/0x1f0 [<ffffffff8119a4ce>] ____fput+0xe/0x10 [<ffffffff81074ddc>] task_work_run+0xac/0xe0 [<ffffffff8104ff67>] do_exit+0x2c7/0xc60 [<ffffffff81064091>] ? get_signal_to_deliver+0xd1/0x940 [<ffffffff814c118c>] ? _raw_spin_unlock_irq+0x2c/0x60 [<ffffffff81051cac>] do_group_exit+0x4c/0xc0 [<ffffffff810642a1>] get_signal_to_deliver+0x2e1/0x940 [<ffffffff81002528>] do_signal+0x48/0x630 [<ffffffff811d3f77>] ? blkdev_read_iter+0x37/0x40 [<ffffffff8119809e>] ? new_sync_read+0x7e/0xb0 [<ffffffff8127f0d7>] ? debug_smp_processor_id+0x17/0x20 [<ffffffff810ba5d6>] ? rcu_eqs_exit_common.isra.43+0x26/0xe0 [<ffffffff8113539a>] ? context_tracking_user_exit+0x5a/0x1a0 [<ffffffff810a693d>] ? trace_hardirqs_on_caller+0xfd/0x1c0 [<ffffffff81002b81>] do_notify_resume+0x71/0xc0 [<ffffffff814c1f98>] int_signal+0x12/0x17 blkid D ffff8807c7268000 0 8615 1 0x00000006 ffff8807d43b39b8 0000000000000002 ffff8807c7268000 ffff8807d43b3fd8 00000000000131c0 00000000000131c0 ffff88083b4a8000 ffff88085fcb3ad0 ffff88085ffb0038 ffff8807d43b3a40 0000000000000002 ffffffff81135db0 Call Trace: [<ffffffff81135db0>] ? wait_on_page_read+0x60/0x60 [<ffffffff814bc0ed>] io_schedule+0x9d/0x130 [<ffffffff81135dbe>] sleep_on_page+0xe/0x20 [<ffffffff814bc738>] __wait_on_bit_lock+0x48/0xb0 [<ffffffff81135eca>] __lock_page+0x6a/0x70 [<ffffffff8109c8e0>] ? autoremove_wake_function+0x40/0x40 [<ffffffff811474cf>] truncate_inode_pages_range+0x3ff/0x690 [<ffffffff81147775>] truncate_inode_pages+0x15/0x20 [<ffffffff811d3425>] kill_bdev+0x35/0x40 [<ffffffff811d49a9>] __blkdev_put+0x69/0x1b0 [<ffffffff811d5450>] blkdev_put+0x50/0x160 [<ffffffff811d5615>] blkdev_close+0x25/0x30 [<ffffffff8119a37a>] __fput+0xea/0x1f0 [<ffffffff8119a4ce>] ____fput+0xe/0x10 [<ffffffff81074ddc>] task_work_run+0xac/0xe0 [<ffffffff8104ff67>] do_exit+0x2c7/0xc60 [<ffffffff81064091>] ? get_signal_to_deliver+0xd1/0x940 [<ffffffff814c118c>] ? _raw_spin_unlock_irq+0x2c/0x60 [<ffffffff81051cac>] do_group_exit+0x4c/0xc0 [<ffffffff810642a1>] get_signal_to_deliver+0x2e1/0x940 [<ffffffff81002528>] do_signal+0x48/0x630 [<ffffffff811d3f77>] ? blkdev_read_iter+0x37/0x40 [<ffffffff8119809e>] ? new_sync_read+0x7e/0xb0 [<ffffffff8127f0d7>] ? debug_smp_processor_id+0x17/0x20 [<ffffffff810ba5d6>] ? rcu_eqs_exit_common.isra.43+0x26/0xe0 [<ffffffff8113539a>] ? context_tracking_user_exit+0x5a/0x1a0 [<ffffffff810a693d>] ? trace_hardirqs_on_caller+0xfd/0x1c0 [<ffffffff81002b81>] do_notify_resume+0x71/0xc0 [<ffffffff814c1f98>] int_signal+0x12/0x17 -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel
diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c index 3f6fd9d..561ead6 100644 --- a/drivers/md/dm-mpath.c +++ b/drivers/md/dm-mpath.c @@ -373,7 +373,7 @@ static int __must_push_back(struct multipath *m) dm_noflush_suspending(m->ti))); } -#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required) +#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required && !(m)->pg_init_in_progress) /* * Map cloned requests