| Message ID | 20210222132405.91369-1-hare@suse.de (mailing list archive) |
|---|---|
| Headers | show |
| Series | scsi: enable reserved commands for LLDDs | expand |
On 22/02/2021 13:23, Hannes Reinecke wrote: > Hi all, + > > quite some drivers use internal commands for various purposes, most > commonly sending TMFs or querying the HBA status. > While these commands use the same submission mechanism than normal > I/O commands, they will not be counted as outstanding commands, > requiring those drivers to implement their own mechanism to figure > out outstanding commands. > The block layer already has the concept of 'reserved' tags for > precisely this purpose, namely non-I/O tags which live off a separate > tag pool. That guarantees that these commands can always be sent, > and won't be influenced by tag starvation from the I/O tag pool. > This patchset enables the use of reserved tags for the SCSI midlayer > by allocating a virtual LUN for the HBA itself which just serves > as a resource to allocate valid tags from. > This removes quite some hacks which were required for some > drivers (eg. fnic or snic), and allows the use of tagset > iterators within the drivers. > > The entire patchset can be found at > > git://git.kernel.org/pub/scm/linux/kernel/git/hare/scsi-devel.git > reserved-tags.v7 Thanks for doing this, I'll have a look.
On 23/02/2021 10:16, John Garry wrote: > On 22/02/2021 13:23, Hannes Reinecke wrote: >> Hi all, > > + > >> >> quite some drivers use internal commands for various purposes, most >> commonly sending TMFs or querying the HBA status. >> While these commands use the same submission mechanism than normal >> I/O commands, they will not be counted as outstanding commands, >> requiring those drivers to implement their own mechanism to figure >> out outstanding commands. >> The block layer already has the concept of 'reserved' tags for >> precisely this purpose, namely non-I/O tags which live off a separate >> tag pool. That guarantees that these commands can always be sent, >> and won't be influenced by tag starvation from the I/O tag pool. >> This patchset enables the use of reserved tags for the SCSI midlayer >> by allocating a virtual LUN for the HBA itself which just serves >> as a resource to allocate valid tags from. >> This removes quite some hacks which were required for some >> drivers (eg. fnic or snic), and allows the use of tagset >> iterators within the drivers. >> >> The entire patchset can be found at >> >> git://git.kernel.org/pub/scm/linux/kernel/git/hare/scsi-devel.git >> reserved-tags.v7 > > > Thanks for doing this, I'll have a look. So I got this working eventually for hisi_sas - a few fixes needed: https://github.com/hisilicon/kernel-dev/commits/private-topic-sas-5.11-resv7 I'll have a look at the core patches tomorrow. However, at this point, how about convert just a couple of drivers (the ones which you can test) to get it merged as a start? 31 patches is too many. thanks, John
On 2/23/21 6:50 PM, John Garry wrote: > On 23/02/2021 10:16, John Garry wrote: >> On 22/02/2021 13:23, Hannes Reinecke wrote: >>> Hi all, >> >> + >> >>> >>> quite some drivers use internal commands for various purposes, most >>> commonly sending TMFs or querying the HBA status. >>> While these commands use the same submission mechanism than normal >>> I/O commands, they will not be counted as outstanding commands, >>> requiring those drivers to implement their own mechanism to figure >>> out outstanding commands. >>> The block layer already has the concept of 'reserved' tags for >>> precisely this purpose, namely non-I/O tags which live off a separate >>> tag pool. That guarantees that these commands can always be sent, >>> and won't be influenced by tag starvation from the I/O tag pool. >>> This patchset enables the use of reserved tags for the SCSI midlayer >>> by allocating a virtual LUN for the HBA itself which just serves >>> as a resource to allocate valid tags from. >>> This removes quite some hacks which were required for some >>> drivers (eg. fnic or snic), and allows the use of tagset >>> iterators within the drivers. >>> >>> The entire patchset can be found at >>> >>> git://git.kernel.org/pub/scm/linux/kernel/git/hare/scsi-devel.git >>> reserved-tags.v7 >> >> >> Thanks for doing this, I'll have a look. > > So I got this working eventually for hisi_sas - a few fixes needed: > https://github.com/hisilicon/kernel-dev/commits/private-topic-sas-5.11-resv7 > > > I'll have a look at the core patches tomorrow. However, at this point, > how about convert just a couple of drivers (the ones which you can test) > to get it merged as a start? 31 patches is too many. > Yeah, I know. Actually, I just sent it out so that you can have a look at my libsas slow task rework. But I can easily split it off for just fnic/snic/aacraid/hpsa, and leave the libsas slow task stuff for the next round. Cheers, Hannes
On 24/02/2021 06:54, Hannes Reinecke wrote: >> >> So I got this working eventually for hisi_sas - a few fixes needed: >> https://github.com/hisilicon/kernel-dev/commits/private-topic-sas-5.11-resv7 >> >> >> I'll have a look at the core patches tomorrow. However, at this point, >> how about convert just a couple of drivers (the ones which you can >> test) to get it merged as a start? 31 patches is too many. >> > Yeah, I know. Actually, I just sent it out so that you can have a look > at my libsas slow task rework. > Fine, but obviously that libsas and users stuff needs more work. > But I can easily split it off for just fnic/snic/aacraid/hpsa, and leave > the libsas slow task stuff for the next round. I think that's better. But even then, it's still a large patchset. Maybe 1 or 2 drivers is enough to start with... and whatever you can test, as I don't have HW to test those. Thanks, John
-----Original Message-----
From: Hannes Reinecke [mailto:hare@suse.de]
Subject: [PATCHv7 00/31] scsi: enable reserved commands for LLDDs
Hi all,
quite some drivers use internal commands for various purposes, most commonly sending TMFs or querying the HBA status.
While these commands use the same submission mechanism than normal I/O commands, they will not be counted as outstanding commands, requiring those drivers to implement their own mechanism to figure out outstanding commands.
The block layer already has the concept of 'reserved' tags for precisely this purpose, namely non-I/O tags which live off a separate tag pool. That guarantees that these commands can always be sent, and won't be influenced by tag starvation from the I/O tag pool.
This patchset enables the use of reserved tags for the SCSI midlayer by allocating a virtual LUN for the HBA itself which just serves as a resource to allocate valid tags from.
This removes quite some hacks which were required for some drivers (eg. fnic or snic), and allows the use of tagset iterators within the drivers.
The entire patchset can be found at
git://git.kernel.org/pub/scm/linux/kernel/git/hare/scsi-devel.git
reserved-tags.v7
Don: Cloned and kernel built. I'll have some test results by end of next week or so...
Thanks,
Don
--
2.29.2
On 2/22/21 7:23 AM, Hannes Reinecke wrote: > Hi all, > > quite some drivers use internal commands for various purposes, most > commonly sending TMFs or querying the HBA status. > While these commands use the same submission mechanism than normal > I/O commands, they will not be counted as outstanding commands, > requiring those drivers to implement their own mechanism to figure > out outstanding commands. > The block layer already has the concept of 'reserved' tags for > precisely this purpose, namely non-I/O tags which live off a separate > tag pool. That guarantees that these commands can always be sent, > and won't be influenced by tag starvation from the I/O tag pool. > This patchset enables the use of reserved tags for the SCSI midlayer > by allocating a virtual LUN for the HBA itself which just serves > as a resource to allocate valid tags from. > This removes quite some hacks which were required for some > drivers (eg. fnic or snic), and allows the use of tagset > iterators within the drivers. > Hey Hannes, I was trying to port some iscsi patches to this set. One question I had is how to handle if my driver implements init_cmd_priv, and wants to use the reserved cmds for a non scsi IO. My case I want to use them for cmds like a iscsi nop/ping, device/target reset or login request. There is no bit to way to tell if at init_cmd_priv time the cmd will be for a reserved or non reserved cmd right? If not, I was wondering should I do: 1. in libiscsi, allocate an array of size $reserved_cmds with non_scsi_cmds structs. When I need to do a non scsi cmd do blk_mq_get_tag on the host's tags to get a reserved tag then use that to lookup a struct in my array? 2. in libiscsi when I need to do a non scsi cmd do a scsi_get_internal_cmd. At this time allocate the non_scsi_cmd struct parts.
On 3/12/21 12:53 AM, michael.christie@oracle.com wrote: > On 2/22/21 7:23 AM, Hannes Reinecke wrote: >> Hi all, >> >> quite some drivers use internal commands for various purposes, most >> commonly sending TMFs or querying the HBA status. >> While these commands use the same submission mechanism than normal >> I/O commands, they will not be counted as outstanding commands, >> requiring those drivers to implement their own mechanism to figure >> out outstanding commands. >> The block layer already has the concept of 'reserved' tags for >> precisely this purpose, namely non-I/O tags which live off a separate >> tag pool. That guarantees that these commands can always be sent, >> and won't be influenced by tag starvation from the I/O tag pool. >> This patchset enables the use of reserved tags for the SCSI midlayer >> by allocating a virtual LUN for the HBA itself which just serves >> as a resource to allocate valid tags from. >> This removes quite some hacks which were required for some >> drivers (eg. fnic or snic), and allows the use of tagset >> iterators within the drivers. >> > > Hey Hannes, > > I was trying to port some iscsi patches to this set. One question I had > is how to handle if my driver implements init_cmd_priv, and wants to use > the reserved cmds for a non scsi IO. My case I want to use them for cmds > like a iscsi nop/ping, device/target reset or login request. > > There is no bit to way to tell if at init_cmd_priv time the cmd will be > for a reserved or non reserved cmd right? If not, I was wondering should > I do: > > 1. in libiscsi, allocate an array of size $reserved_cmds with non_scsi_cmds > structs. When I need to do a non scsi cmd do blk_mq_get_tag on the host's > tags to get a reserved tag then use that to lookup a struct in my array? > > 2. in libiscsi when I need to do a non scsi cmd do a scsi_get_internal_cmd. > At this time allocate the non_scsi_cmd struct parts. > You sure you will need to allocate additional stuff? The request already comes with the request, scsi, and driver private bits (ie the additional space from .cmd_size) allocated. And yes, you can tell in init_cmd_priv() if the command is coming from the private pool; I had a helper 'req_is_reserved' once, I thought it's still there ... Cheers, Hannes
-----Original Message-----
Subject: RE: [PATCHv7 00/31] scsi: enable reserved commands for LLDDs
-----Original Message-----
From: Hannes Reinecke [mailto:hare@suse.de]
Subject: [PATCHv7 00/31] scsi: enable reserved commands for LLDDs
Hi all,
quite some drivers use internal commands for various purposes, most commonly sending TMFs or querying the HBA status.
While these commands use the same submission mechanism than normal I/O commands, they will not be counted as outstanding commands, requiring those drivers to implement their own mechanism to figure out outstanding commands.
The block layer already has the concept of 'reserved' tags for precisely this purpose, namely non-I/O tags which live off a separate tag pool. That guarantees that these commands can always be sent, and won't be influenced by tag starvation from the I/O tag pool.
This patchset enables the use of reserved tags for the SCSI midlayer by allocating a virtual LUN for the HBA itself which just serves as a resource to allocate valid tags from.
This removes quite some hacks which were required for some drivers (eg. fnic or snic), and allows the use of tagset iterators within the drivers.
The entire patchset can be found at
git://git.kernel.org/pub/scm/linux/kernel/git/hare/scsi-devel.git
reserved-tags.v7
Don: Cloned and kernel built. I'll have some test results by end of next week or so...
Thanks,
Don
--
2.29.2
Don:
03/16/2021
Have run a lot of heavy I/O tests. The driver/OS holds together until there are reset operations. The resets do not complete because of 1 patch pending on Martin Peterson's 5.13/scsi-queue tree
f749d8b7a989 scsi: hpsa: Correct dev cmds outstanding for retried cmds
and pending patch
https://patchwork.kernel.org/project/linux-scsi/patch/161540317205.18786.5821926127237311408.stgit@brunhilda/ hpsa: fix regression issue for old controllers
My testing consisted of running 7 operations in parallel using a lot of HBA, LVs, and AIO LVs.
1. mkfs
2. mount
3. rsync to mounted volumes
4. fio to mounted file systems.
5. umount
6. fsck
7. fio to raw disks.
So far, so good, but I need to add in tests that exercise the reserved slots. I'll start that soon after I complete my resulting logfile checks.
Thanks,
Don
On 12/03/2021 16:08, Hannes Reinecke wrote: > On 3/12/21 12:53 AM, michael.christie@oracle.com wrote: >> On 2/22/21 7:23 AM, Hannes Reinecke wrote: >>> Hi all, >>> >>> quite some drivers use internal commands for various purposes, most >>> commonly sending TMFs or querying the HBA status. >>> While these commands use the same submission mechanism than normal >>> I/O commands, they will not be counted as outstanding commands, >>> requiring those drivers to implement their own mechanism to figure >>> out outstanding commands. >>> The block layer already has the concept of 'reserved' tags for >>> precisely this purpose, namely non-I/O tags which live off a separate >>> tag pool. That guarantees that these commands can always be sent, >>> and won't be influenced by tag starvation from the I/O tag pool. >>> This patchset enables the use of reserved tags for the SCSI midlayer >>> by allocating a virtual LUN for the HBA itself which just serves >>> as a resource to allocate valid tags from. >>> This removes quite some hacks which were required for some >>> drivers (eg. fnic or snic), and allows the use of tagset >>> iterators within the drivers. >>> >> >> Hey Hannes, >> >> I was trying to port some iscsi patches to this set. One question I had >> is how to handle if my driver implements init_cmd_priv, and wants to use >> the reserved cmds for a non scsi IO. My case I want to use them for cmds >> like a iscsi nop/ping, device/target reset or login request. >> >> There is no bit to way to tell if at init_cmd_priv time the cmd will be >> for a reserved or non reserved cmd right? If not, I was wondering should >> I do: If I'm not mistaken, init_cmd_priv has no in-tree user today. Any plans to add one ... to see what it's about? Thanks, John >> >> 1. in libiscsi, allocate an array of size $reserved_cmds with non_scsi_cmds >> structs. When I need to do a non scsi cmd do blk_mq_get_tag on the host's >> tags to get a reserved tag then use that to lookup a struct in my array? >> >> 2. in libiscsi when I need to do a non scsi cmd do a scsi_get_internal_cmd. >> At this time allocate the non_scsi_cmd struct parts. >> > You sure you will need to allocate additional stuff? > The request already comes with the request, scsi, and driver private > bits (ie the additional space from .cmd_size) allocated. > > And yes, you can tell in init_cmd_priv() if the command is coming from > the private pool; I had a helper 'req_is_reserved' once, I thought it's > still there ... > > Cheers, > > Hannes >
-----Original Message----- From: Don Brace - C33706 Sent: Tuesday, March 16, 2021 12:58 PM To: Don Brace - C33706 <Don.Brace@microchip.com>; hare@suse.de; martin.petersen@oracle.com Cc: james.bottomley@hansenpartnership.com; hch@lst.de; john.garry@huawei.com; linux-scsi@vger.kernel.org Subject: RE: [PATCHv7 00/31] scsi: enable reserved commands for LLDDs Hi all, quite some drivers use internal commands for various purposes, most commonly sending TMFs or querying the HBA status. While these commands use the same submission mechanism than normal I/O commands, they will not be counted as outstanding commands, requiring those drivers to implement their own mechanism to figure out outstanding commands. The block layer already has the concept of 'reserved' tags for precisely this purpose, namely non-I/O tags which live off a separate tag pool. That guarantees that these commands can always be sent, and won't be influenced by tag starvation from the I/O tag pool. This patchset enables the use of reserved tags for the SCSI midlayer by allocating a virtual LUN for the HBA itself which just serves as a resource to allocate valid tags from. This removes quite some hacks which were required for some drivers (eg. fnic or snic), and allows the use of tagset iterators within the drivers. The entire patchset can be found at git://git.kernel.org/pub/scm/linux/kernel/git/hare/scsi-devel.git reserved-tags.v7 Don: Cloned and kernel built. I'll have some test results by end of next week or so... Thanks, Don -- 2.29.2 Don: 03/16/2021 Have run a lot of heavy I/O tests. The driver/OS holds together until there are reset operations. The resets do not complete because of 1 patch pending on Martin Peterson's 5.13/scsi-queue tree f749d8b7a989 scsi: hpsa: Correct dev cmds outstanding for retried cmds and pending patch https://patchwork.kernel.org/project/linux-scsi/patch/161540317205.18786.5821926127237311408.stgit@brunhilda/ hpsa: fix regression issue for old controllers My testing consisted of running 7 operations in parallel using a lot of HBA, LVs, and AIO LVs. 1. mkfs 2. mount 3. rsync to mounted volumes 4. fio to mounted file systems. 5. umount 6. fsck 7. fio to raw disks. So far, so good, but I need to add in tests that exercise the reserved slots. I'll start that soon after I complete my resulting logfile checks. Thanks, Don 03/29/2021 Don: After applying my patch commit 6651ea81e097b369ececdc53d1e04bc40955d92a (HEAD -> reserved-tags.v7) Author: Don Brace <don.brace@microchip.com> Date: Mon Feb 15 16:26:57 2021 -0600 scsi: hpsa: Correct dev cmds outstanding for retried cmds I re-ran my Heavy I/O scripts and testing held up. For the hpsa driver: Acked-by: Don Brace <don.brace@microchip.com> Tested-by: Don Brace <don.brace@microchip.com>
Hi all, quite some drivers use internal commands for various purposes, most commonly sending TMFs or querying the HBA status. While these commands use the same submission mechanism than normal I/O commands, they will not be counted as outstanding commands, requiring those drivers to implement their own mechanism to figure out outstanding commands. The block layer already has the concept of 'reserved' tags for precisely this purpose, namely non-I/O tags which live off a separate tag pool. That guarantees that these commands can always be sent, and won't be influenced by tag starvation from the I/O tag pool. This patchset enables the use of reserved tags for the SCSI midlayer by allocating a virtual LUN for the HBA itself which just serves as a resource to allocate valid tags from. This removes quite some hacks which were required for some drivers (eg. fnic or snic), and allows the use of tagset iterators within the drivers. The entire patchset can be found at git://git.kernel.org/pub/scm/linux/kernel/git/hare/scsi-devel.git reserved-tags.v7 As usual, comments and reviews are welcome. Changes to v6: - Remove patch to drop gdth - Rework libsas to use a tag per slow task - Update hisi_sas, pm8001, and mv_sas Changes to v5: - Remove patch for csiostor - Warn on normal commands in scsi_put_reserved_cmd() - Fixup aacraid to not only scsi_put_internal_cmd() for reserved commands - Add 'nr_reserved_cmds' field to host template - Reshuffle patches Changes to v4: - Fixup kbuild warning - Include reviews from Bart Changes to v3: - Kill gdth - Only convert fnic, snic, hpsa, and aacraid - Drop command emulation for pseudo host device - make 'can_queue' exclude the number or reserved tags - Drop persistent commands proposal - Sanitize host device handling Changes to v2: - Update patches from John Garry - Use virtual LUN as suggested by Christoph - Improve SCSI Host device to present a real SCSI device - Implement 'persistent' commands for AENs - Convert Megaraid SAS Changes to v1: - Make scsi_{get, put}_reserved_cmd() for Scsi host - Previously we separate scsi_{get, put}_reserved_cmd() for sdev and scsi_host_get_reserved_cmd() for the host - Fix how Scsi_Host.can_queue is set in the virtio-scsi change - Drop Scsi_Host.use_reserved_cmd_q - Drop scsi_is_reserved_cmd() - Add support in libsas and associated HBA drivers - Allocate reserved command in slow task - Switch hisi_sas to use reserved Scsi command - Reorder the series a little - Some tidying Hannes Reinecke (29): block: add flag for internal commands scsi: add scsi_{get,put}_internal_cmd() helper fnic: use internal commands fnic: use scsi_host_busy_iter() to traverse commands fnic: check for started requests in fnic_wq_copy_cleanup_handler() scsi: use real inquiry data when initialising devices scsi: Use dummy inquiry data for the host device scsi: revamp host device handling snic: use reserved commands snic: use tagset iter for traversing commands snic: check for started requests in snic_hba_reset_cmpl_handler() scsi: implement reserved command handling hpsa: move hpsa_hba_inquiry after scsi_add_host() hpsa: use reserved commands hpsa: use scsi_host_busy_iter() to traverse outstanding commands hpsa: drop refcount field from CommandList aacraid: move scsi_add_host() aacraid: store target id in host_scribble aacraid: use scsi_get_internal_cmd() aacraid: use scsi_host_busy_iter() to traverse outstanding commands mv_sas: kill mvsas_debug_issue_ssp_tmf() pm8001: kill pm8001_issue_ssp_tmf() pm8001: kill 'dev' argument from pm8001_exec_internal_task_abort() pm8001: use libsas-provided domain devices for SATA libsas: add SCSI target pointer to struct domain_device libsas: add tag to struct sas_task hisi_sas: use task tag to reference the slot mv_sas: use reserved tags and drop private tag allocation pm8001: use block-layer tags for ccb allocation John Garry (2): scsi: libsas,hisi_sas,mvsas,pm8001: Allocate Scsi_cmd for slow task scsi: hisi_sas: Use libsas slow task SCSI command block/blk-exec.c | 5 + drivers/scsi/aacraid/aachba.c | 137 ++-- drivers/scsi/aacraid/aacraid.h | 10 +- drivers/scsi/aacraid/commctrl.c | 25 +- drivers/scsi/aacraid/comminit.c | 2 +- drivers/scsi/aacraid/commsup.c | 109 ++- drivers/scsi/aacraid/dpcsup.c | 2 +- drivers/scsi/aacraid/linit.c | 175 +++-- drivers/scsi/fnic/fnic_scsi.c | 944 +++++++++++-------------- drivers/scsi/hisi_sas/hisi_sas.h | 1 - drivers/scsi/hisi_sas/hisi_sas_main.c | 137 ++-- drivers/scsi/hisi_sas/hisi_sas_v3_hw.c | 5 +- drivers/scsi/hosts.c | 3 + drivers/scsi/hpsa.c | 368 +++++----- drivers/scsi/hpsa.h | 3 +- drivers/scsi/hpsa_cmd.h | 1 - drivers/scsi/libsas/sas_ata.c | 4 + drivers/scsi/libsas/sas_expander.c | 7 +- drivers/scsi/libsas/sas_init.c | 63 +- drivers/scsi/libsas/sas_scsi_host.c | 4 +- drivers/scsi/mvsas/mv_init.c | 10 +- drivers/scsi/mvsas/mv_sas.c | 136 +--- drivers/scsi/mvsas/mv_sas.h | 13 +- drivers/scsi/pm8001/pm8001_hwi.c | 154 ++-- drivers/scsi/pm8001/pm8001_init.c | 24 +- drivers/scsi/pm8001/pm8001_sas.c | 193 +++-- drivers/scsi/pm8001/pm8001_sas.h | 14 +- drivers/scsi/pm8001/pm80xx_hwi.c | 143 ++-- drivers/scsi/scsi_devinfo.c | 1 + drivers/scsi/scsi_lib.c | 55 +- drivers/scsi/scsi_scan.c | 96 ++- drivers/scsi/scsi_sysfs.c | 3 +- drivers/scsi/snic/snic.h | 4 +- drivers/scsi/snic/snic_main.c | 7 + drivers/scsi/snic/snic_scsi.c | 525 +++++++------- include/linux/blk_types.h | 2 + include/linux/blkdev.h | 5 + include/scsi/libsas.h | 11 +- include/scsi/scsi_device.h | 4 + include/scsi/scsi_host.h | 36 +- 40 files changed, 1647 insertions(+), 1794 deletions(-)