| Message ID | 20230407203143.2189681-1-fenghua.yu@intel.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Enable DSA 2.0 Event Log and completion record faulting features | expand |
On 07-04-23, 13:31, Fenghua Yu wrote: > Applications can send 64B descriptors to the DSA device via CPU > instructions MOVDIR64B or ENQCMD. The application can choose to have > the device write back a completion record (CR) in system memory to > indicate the status of the descriptor submitted on completion. > > With the DSA hardware, the device is able to do on demand paging through > the hardware by faulting in the user pages that do not have physical memory > page backing with assistance from IOMMU. In the spec this was designated as > the block on fault feature. While this hardware feature made operation > simpler, it also stalls the device engines while the memory pages are being > faulted in through Page Request Service (PRS). For applications sharing the > same workqueue (wq) or wqs in the same group, operations are stalled if > there are no free engines. To avoid slowing the performance of all other > running applications sharing the same device engine(s), PRS can to be > disabled and software can deal with partial completion. > > The block on fault feature on DSA 1.0 can be disabled for the wq. However, > PRS is not completely disabled for the whole path. It is not disabled for > CRs or batch list for a batch operation. > > The other issue is the DSA 1.0 error reporting mechanism, SWERROR register. > The SWERROR register can only report a single error at a time until the > driver reads and acknowledges the error. The follow on errors cannot be > reported until the current error is "cleared" by the software by writing > a bit to the SWERR register. If a large number of faults arrive and the > software cannot clear them fast enough, overflowed errors will be dropped > by the device. > > A CR is the optional 32 bytes (DSA) or 64 bytes (IAA) status that is > written back for a submitted descriptor. If the address for the CR faults, > the error is reported to the SWERROR register instead. > > With DSA 2.0 hardware [1], the event log feature is added. All errors are > reported as an entry in a circular buffer reside in the system memory. > The system admin is responsible to configure the size of the circular > buffer large enough per device to handle the potential errors that may be > reported. If the buffer is full and another error needs to be reported, > the device engine will block until there's a free slot in the buffer. > An event log entry for a faulted CR will contain the error information, > the CR address that faulted, and the expected CR content the device had > originally intended to write. > > DSA 2.0 also introduces per wq PRS disable knob. This will disable all PRS > operations for the specific wq. The device will still have Address > Translation Service (ATS) on. When ATS fails on a memory address for a CR, > an eventlog entry will be written by the hardware into the event log > ring buffer. The driver software is expected to parse the event log entry, > fault in the address of the CR, and the write the content of the CR to > the memory address. > > This patch series will implement the DSA 2 event log support. The support > for the handling of the faulted user CR is added. The driver is also > adding the same support for batch operation descriptors. With a batch > operation the handling of the event log entry is a bit more complex. > The faulting CR could be for the batch descriptor or any of the operation > descriptors within the batch. The hardware generates a batch identifier > that is used by the driver software to correlate the event log entries for > the relevant descriptors of that batch. > > The faulting of source and destination addresses for the operation is not > handled by the driver. That is left to be handled by the user application > by faulting in the memory and re-submit the remaining operation. Applied, thanks
On 4/12/23 10:18, Vinod Koul wrote: > On 07-04-23, 13:31, Fenghua Yu wrote: >> Applications can send 64B descriptors to the DSA device via CPU >> instructions MOVDIR64B or ENQCMD. The application can choose to have >> the device write back a completion record (CR) in system memory to >> indicate the status of the descriptor submitted on completion. > Applied, thanks > Thank you very much, Vinod! This is really good news! -Fenghua
Applications can send 64B descriptors to the DSA device via CPU instructions MOVDIR64B or ENQCMD. The application can choose to have the device write back a completion record (CR) in system memory to indicate the status of the descriptor submitted on completion. With the DSA hardware, the device is able to do on demand paging through the hardware by faulting in the user pages that do not have physical memory page backing with assistance from IOMMU. In the spec this was designated as the block on fault feature. While this hardware feature made operation simpler, it also stalls the device engines while the memory pages are being faulted in through Page Request Service (PRS). For applications sharing the same workqueue (wq) or wqs in the same group, operations are stalled if there are no free engines. To avoid slowing the performance of all other running applications sharing the same device engine(s), PRS can to be disabled and software can deal with partial completion. The block on fault feature on DSA 1.0 can be disabled for the wq. However, PRS is not completely disabled for the whole path. It is not disabled for CRs or batch list for a batch operation. The other issue is the DSA 1.0 error reporting mechanism, SWERROR register. The SWERROR register can only report a single error at a time until the driver reads and acknowledges the error. The follow on errors cannot be reported until the current error is "cleared" by the software by writing a bit to the SWERR register. If a large number of faults arrive and the software cannot clear them fast enough, overflowed errors will be dropped by the device. A CR is the optional 32 bytes (DSA) or 64 bytes (IAA) status that is written back for a submitted descriptor. If the address for the CR faults, the error is reported to the SWERROR register instead. With DSA 2.0 hardware [1], the event log feature is added. All errors are reported as an entry in a circular buffer reside in the system memory. The system admin is responsible to configure the size of the circular buffer large enough per device to handle the potential errors that may be reported. If the buffer is full and another error needs to be reported, the device engine will block until there's a free slot in the buffer. An event log entry for a faulted CR will contain the error information, the CR address that faulted, and the expected CR content the device had originally intended to write. DSA 2.0 also introduces per wq PRS disable knob. This will disable all PRS operations for the specific wq. The device will still have Address Translation Service (ATS) on. When ATS fails on a memory address for a CR, an eventlog entry will be written by the hardware into the event log ring buffer. The driver software is expected to parse the event log entry, fault in the address of the CR, and the write the content of the CR to the memory address. This patch series will implement the DSA 2 event log support. The support for the handling of the faulted user CR is added. The driver is also adding the same support for batch operation descriptors. With a batch operation the handling of the event log entry is a bit more complex. The faulting CR could be for the batch descriptor or any of the operation descriptors within the batch. The hardware generates a batch identifier that is used by the driver software to correlate the event log entries for the relevant descriptors of that batch. The faulting of source and destination addresses for the operation is not handled by the driver. That is left to be handled by the user application by faulting in the memory and re-submit the remaining operation. This series consists of three parts: 1. Patch 1: Make misc interrupt one shot. Event Log interrupt depends on this patch. This patch was released before but is not in upstream yet: https://lore.kernel.org/dmaengine/165125374675.311834.10460196228320964350.stgit@djiang5-desk3.ch.intel.com/ 2. Patches 2-15: Enable Event Log and Completion Record faulting. 3. Patch 16: Configure PRS disable per WQ. This series is applied cleanly on top of "Expose IAA 2.0 device capabilities" series (which is in dmaengine next branch now): https://lore.kernel.org/lkml/20230303213732.3357494-1-fenghua.yu@intel.com/ Change log: v4: - Use kthread_use_mm(), copy_to_user(), and kthread_unuse_mm() to switch to the mm, copy completion record to the mm, and switch back to the current mm. It's simpler than previous access_remote_vm() or emulation ways (Jason Gunthorpe, Christoph Hellwig, Tony Luck). - Change dev_err() to dev_dbg_ratelimited() in CR page fault handler (Tony Luck). v3: - Since iommu_sva_find() will be removed in IOMMU and access_remote_vm() cannot be exported, the completion record copy function idxd_copy_cr() is rewritten by maintaining and finding mm in xarray and copy completion record to the mm. Please check discussion on iommu_sva_find() will be removed and access_remote_vm() cannot be exported: 1. https://lore.kernel.org/lkml/ZAjSsm4%2FPDRqViwa@nvidia.com/ 2. https://lore.kernel.org/lkml/20230306163138.587484-1-fenghua.yu@intel.com/T/#m1fc97725a0e56ea269c8bdabacee447070d51846 v2: - Define and export iommu_access_remote_vm() for IDXD driver to write completion record to user address space. This change removes patch 8 and 9 in v1 (Alistair Popple) https://lore.kernel.org/lkml/20230306163138.587484-1-fenghua.yu@intel.com/ Dave Jiang (15): dmaengine: idxd: make misc interrupt one shot dmaengine: idxd: add event log size sysfs attribute dmaengine: idxd: setup event log configuration dmaengine: idxd: add interrupt handling for event log dmanegine: idxd: add debugfs for event log dump dmaengine: idxd: add per DSA wq workqueue for processing cr faults dmaengine: idxd: create kmem cache for event log fault items dmaengine: idxd: process user page faults for completion record dmaengine: idxd: add descs_completed field for completion record dmaengine: idxd: process batch descriptor completion record faults dmaengine: idxd: add per file user counters for completion record faults dmaengine: idxd: add a device to represent the file opened dmaengine: idxd: expose fault counters to sysfs dmaengine: idxd: add pid to exported sysfs attribute for opened file dmaengine: idxd: add per wq PRS disable Fenghua Yu (1): dmaengine: idxd: add idxd_copy_cr() to copy user completion record during page fault handling .../ABI/stable/sysfs-driver-dma-idxd | 43 +++ drivers/dma/idxd/Makefile | 2 +- drivers/dma/idxd/cdev.c | 334 ++++++++++++++++-- drivers/dma/idxd/debugfs.c | 138 ++++++++ drivers/dma/idxd/device.c | 113 +++++- drivers/dma/idxd/idxd.h | 65 ++++ drivers/dma/idxd/init.c | 53 +++ drivers/dma/idxd/irq.c | 210 +++++++++-- drivers/dma/idxd/registers.h | 105 +++++- drivers/dma/idxd/sysfs.c | 112 +++++- include/uapi/linux/idxd.h | 15 +- 11 files changed, 1126 insertions(+), 64 deletions(-) create mode 100644 drivers/dma/idxd/debugfs.c