| Message ID | 20200220200838.15809-1-dgilbert@interlog.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | host managed ZBC + doublestore | expand |
Missed putting the driver name in the subject ... On 2020-02-20 3:08 p.m., Douglas Gilbert wrote: > This patchset is a follow-on from an earlier patchset titled: > [RFC v2 0/6] scsi_debug: random doublestore verify > posted on the linux-scsi list on 20200109. > > The major addition is support for host-managed ZBC devices. > The bulk of the work in this area was done by Damien Le Moal. > It allows ZBC devices with a mix of conventional and > "sequential write required" zones to be specified. These > follow the same model as direct access devices in this > driver. Namely, each device has its own metadata (including > its write pointer(s)) but all (scsi_debug) devices share the > same user data store (see doublestore below). > > Significantly, this simulation passes the test/zbc_tests.sh > script in the https://github.com/hgst/libzbc repository. > > The lower numbered patches in this set contain various > measures to improve the speed and usefulness of this driver. > It is being used to test the rewrite of the SCSI generic (sg) > driver which is still underway. Disk to disk copies are the > test of choice by the author [DG]. Comparing the results > between (simulated) disks is useless since all scsi_debug > devices share the same user data store. This limitation was > circumvented by adding the doublestore parameter. When set, > doublestore doubles the data store and allocates them in an > alternating pattern to each scsi_debug device. To enhance > the comparisons, simulations of VERIFY(10 and 16) commands > have been added. A further enhancement is to simulate the > PRE-FETCH command (which does nothing as the data is > already cached). > > doublestore can also help with ZBC testing. Single threaded > copy commands like dd (and ddpt) can be used to copy one > non-empty zone into another empty zone. Multiple-threaded > copies (e.g. sgp_dd) can do out-of-order WRITEs which > become a WRITE VIOLATION error in a "sequential write > required" zone. > > The author [DG] found that precise command duration timing > gave a false impression of how "bulletproof" the sg driver > state machines and locking were. The first patch involves > randomizing the command durations and it did expose various > issues in the driver under test (sg). > > Since all scsi_debug memory store accesses are done in the > context of queuecommand() call, the *_irqsave() and > *_irqrestore() variants of the associated locks have been > removed. That could be a problem if queuecommand() can ever > be called form an interrupt or related context. > > Finally to address the discrepancy between command duration > times seen by the sg driver compared to what was set with > this driver's ndelay option, this driver's timekeeping for > short durations was made more accurate. > > If and when this patchset is accepted, this page will be > updated: http://sg.danny.cz/sg/sdebug26.html > This patchset is against Martin Petersen's git repository > and its 5.7/scsi-queue branch. > > Changes since v2 (RFC): > - add support for host-managed zbc devices with > conventional and "sequential write required" > zones [DLM] > > Changes since v1 (RFC): > - testing with version 1 caused several strange crashes that > turned out to be caused by a code trick to read in the > data-out buffer but _not_ place it in the big fake_storep > array. This approach failed badly when multiple threads > were doing verifies at the same time. > - replace the code trick with a new do_dout_fetch() function > - since the code trick was borrowed from the COMPARE AND > WRITE implementation [resp_comp_write()] using > do_dout_fetch() fixes the same bug in the existing driver > which hasn't been reported (yet). > > Damien Le Moal (3): > scsi_debug: zone_max_open module parameter > scsi_debug: zone_nr_conv module parameter > scsi_debug: zone_size_mb module parameter > > Douglas Gilbert (12): > scsi_debug: randomize command completion time > scsi_debug: add doublestore option > scsi_debug: implement verify(10), add verify(16) > scsi_debug: weaken rwlock around ramdisk access > scsi_debug: improve command duration calculation > scsi_debug: implement pre-fetch commands > scsi_debug: expand zbc support > scsi_debug: add zone commands > scsi_debug: zbc module parameter > scsi_debug: re-arrange parameters alphabetically > scsi_debug: zbc parameter can be string > scsi_debug: bump to version 1.89 > > drivers/scsi/scsi_debug.c | 1593 ++++++++++++++++++++++++++++++++----- > 1 file changed, 1389 insertions(+), 204 deletions(-) >
This patchset is a follow-on from an earlier patchset titled: [RFC v2 0/6] scsi_debug: random doublestore verify posted on the linux-scsi list on 20200109. The major addition is support for host-managed ZBC devices. The bulk of the work in this area was done by Damien Le Moal. It allows ZBC devices with a mix of conventional and "sequential write required" zones to be specified. These follow the same model as direct access devices in this driver. Namely, each device has its own metadata (including its write pointer(s)) but all (scsi_debug) devices share the same user data store (see doublestore below). Significantly, this simulation passes the test/zbc_tests.sh script in the https://github.com/hgst/libzbc repository. The lower numbered patches in this set contain various measures to improve the speed and usefulness of this driver. It is being used to test the rewrite of the SCSI generic (sg) driver which is still underway. Disk to disk copies are the test of choice by the author [DG]. Comparing the results between (simulated) disks is useless since all scsi_debug devices share the same user data store. This limitation was circumvented by adding the doublestore parameter. When set, doublestore doubles the data store and allocates them in an alternating pattern to each scsi_debug device. To enhance the comparisons, simulations of VERIFY(10 and 16) commands have been added. A further enhancement is to simulate the PRE-FETCH command (which does nothing as the data is already cached). doublestore can also help with ZBC testing. Single threaded copy commands like dd (and ddpt) can be used to copy one non-empty zone into another empty zone. Multiple-threaded copies (e.g. sgp_dd) can do out-of-order WRITEs which become a WRITE VIOLATION error in a "sequential write required" zone. The author [DG] found that precise command duration timing gave a false impression of how "bulletproof" the sg driver state machines and locking were. The first patch involves randomizing the command durations and it did expose various issues in the driver under test (sg). Since all scsi_debug memory store accesses are done in the context of queuecommand() call, the *_irqsave() and *_irqrestore() variants of the associated locks have been removed. That could be a problem if queuecommand() can ever be called form an interrupt or related context. Finally to address the discrepancy between command duration times seen by the sg driver compared to what was set with this driver's ndelay option, this driver's timekeeping for short durations was made more accurate. If and when this patchset is accepted, this page will be updated: http://sg.danny.cz/sg/sdebug26.html This patchset is against Martin Petersen's git repository and its 5.7/scsi-queue branch. Changes since v2 (RFC): - add support for host-managed zbc devices with conventional and "sequential write required" zones [DLM] Changes since v1 (RFC): - testing with version 1 caused several strange crashes that turned out to be caused by a code trick to read in the data-out buffer but _not_ place it in the big fake_storep array. This approach failed badly when multiple threads were doing verifies at the same time. - replace the code trick with a new do_dout_fetch() function - since the code trick was borrowed from the COMPARE AND WRITE implementation [resp_comp_write()] using do_dout_fetch() fixes the same bug in the existing driver which hasn't been reported (yet). Damien Le Moal (3): scsi_debug: zone_max_open module parameter scsi_debug: zone_nr_conv module parameter scsi_debug: zone_size_mb module parameter Douglas Gilbert (12): scsi_debug: randomize command completion time scsi_debug: add doublestore option scsi_debug: implement verify(10), add verify(16) scsi_debug: weaken rwlock around ramdisk access scsi_debug: improve command duration calculation scsi_debug: implement pre-fetch commands scsi_debug: expand zbc support scsi_debug: add zone commands scsi_debug: zbc module parameter scsi_debug: re-arrange parameters alphabetically scsi_debug: zbc parameter can be string scsi_debug: bump to version 1.89 drivers/scsi/scsi_debug.c | 1593 ++++++++++++++++++++++++++++++++----- 1 file changed, 1389 insertions(+), 204 deletions(-)