| Message ID | 20240326124358.2466259-1-liyihang9@huawei.com (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | [v2] scsi: libsas: Allocation SMP request is aligned to ARCH_DMA_MINALIGN | expand |
On 3/26/24 21:43, Yihang Li wrote: > This series [1] reducing the kmalloc() minimum alignment on arm64 to 8 > (from 128). In libsas, this will cause SMP requests to be 8-byte-aligned > through kmalloc() allocation. However, for the hisi_sas hardware, all > commands address must be 16-byte-aligned. Otherwise, the commands fail to > be executed. > > ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA > operations, so use ARCH_DMA_MINALIGN as the alignment for SMP request. > > Link: https://lkml.kernel.org/r/20230612153201.554742-1-catalin.marinas@arm.com [1] > Signed-off-by: Yihang Li <liyihang9@huawei.com> > --- > Changes since v1: > - Directly modify alloc_smp_req() instead of using handler callback. > --- > drivers/scsi/libsas/sas_expander.c | 5 ++++- > 1 file changed, 4 insertions(+), 1 deletion(-) > > diff --git a/drivers/scsi/libsas/sas_expander.c b/drivers/scsi/libsas/sas_expander.c > index a2204674b680..941abc7298df 100644 > --- a/drivers/scsi/libsas/sas_expander.c > +++ b/drivers/scsi/libsas/sas_expander.c > @@ -135,7 +135,10 @@ static int smp_execute_task(struct domain_device *dev, void *req, int req_size, > > static inline void *alloc_smp_req(int size) > { > - u8 *p = kzalloc(size, GFP_KERNEL); > + u8 *p; > + > + size = ALIGN(size, ARCH_DMA_MINALIGN); > + p = kzalloc(size, GFP_KERNEL); Nit: why not: p = kzalloc(ALIGN(size, ARCH_DMA_MINALIGN), GFP_KERNEL); > if (p) > p[0] = SMP_REQUEST; > return p; Otherwise looks OK to me. John, Unrelated to this patch, but I wonder if the GFP_KERNEL used here shouldn't be GFP_NOIO... Is this ever called in the IO path or error recovery ?
On 26/03/2024 13:14, Damien Le Moal wrote: > On 3/26/24 21:43, Yihang Li wrote: >> This series [1] reducing the kmalloc() minimum alignment on arm64 to 8 >> (from 128). In libsas, this will cause SMP requests to be 8-byte-aligned >> through kmalloc() allocation. However, for the hisi_sas hardware, all >> commands address must be 16-byte-aligned. > Otherwise, the commands fail to >> be executed. >> >> ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA >> operations, so use ARCH_DMA_MINALIGN as the alignment for SMP request. >> >> Link: https://lkml.kernel.org/r/20230612153201.554742-1-catalin.marinas@arm.com [1] >> Signed-off-by: Yihang Li <liyihang9@huawei.com> >> --- >> Changes since v1: >> - Directly modify alloc_smp_req() instead of using handler callback. >> --- >> drivers/scsi/libsas/sas_expander.c | 5 ++++- >> 1 file changed, 4 insertions(+), 1 deletion(-) >> >> diff --git a/drivers/scsi/libsas/sas_expander.c b/drivers/scsi/libsas/sas_expander.c >> index a2204674b680..941abc7298df 100644 >> --- a/drivers/scsi/libsas/sas_expander.c >> +++ b/drivers/scsi/libsas/sas_expander.c >> @@ -135,7 +135,10 @@ static int smp_execute_task(struct domain_device *dev, void *req, int req_size, >> >> static inline void *alloc_smp_req(int size) >> { >> - u8 *p = kzalloc(size, GFP_KERNEL); >> + u8 *p; >> + >> + size = ALIGN(size, ARCH_DMA_MINALIGN); If this is a hisi_sas requirement, then why use ARCH_DMA_MINALIGN and not 16B as minimum alignment? Or are we really talking about an arch requirement? >> + p = kzalloc(size, GFP_KERNEL); > > Nit: why not: > > p = kzalloc(ALIGN(size, ARCH_DMA_MINALIGN), GFP_KERNEL); > >> if (p) >> p[0] = SMP_REQUEST; >> return p; > > Otherwise looks OK to me. > > John, > > Unrelated to this patch, but I wonder if the GFP_KERNEL used here shouldn't be > GFP_NOIO... Is this ever called in the IO path or error recovery ? > These should not be called in the IO path - as they are management functions. But I am quite confident that they can be called in SCSI error handling (for libsas). Thanks, John
On 3/26/24 22:32, John Garry wrote: >> John, >> >> Unrelated to this patch, but I wonder if the GFP_KERNEL used here shouldn't be >> GFP_NOIO... Is this ever called in the IO path or error recovery ? >> > > These should not be called in the IO path - as they are management > functions. But I am quite confident that they can be called in SCSI > error handling (for libsas). So it sounds like GFP_NOIO would be a lot safer... > > Thanks, > John
On Tue, Mar 26, 2024 at 01:32:09PM +0000, John Garry wrote: > > > + u8 *p; > > > + > > > + size = ALIGN(size, ARCH_DMA_MINALIGN); > > > If this is a hisi_sas requirement, then why use ARCH_DMA_MINALIGN and not > 16B as minimum alignment? > > Or are we really talking about an arch requirement? One thing is that we should never allocate unaligned memory for anything DMA mapped, or data will be corrupted by non-coherent DMA. So ARCH_DMA_MINALIGN needs to be here. If specific hardware has further requirements we'll need to communicated it through a field or op vector.
On 2024/3/28 14:35, Christoph Hellwig wrote: > On Tue, Mar 26, 2024 at 01:32:09PM +0000, John Garry wrote: >>>> + u8 *p; >>>> + >>>> + size = ALIGN(size, ARCH_DMA_MINALIGN); >> >> >> If this is a hisi_sas requirement, then why use ARCH_DMA_MINALIGN and not >> 16B as minimum alignment? >> >> Or are we really talking about an arch requirement? > > One thing is that we should never allocate unaligned memory for > anything DMA mapped, or data will be corrupted by non-coherent DMA. > So ARCH_DMA_MINALIGN needs to be here. If specific hardware has > further requirements we'll need to communicated it through a field > or op vector. Got it. Looks like it's still going to be aligned to ARCH_DMA_MINALIGN. Thanks, Yihang > > > . >
On 3/28/24 15:59, Yihang Li wrote: > > > On 2024/3/28 14:35, Christoph Hellwig wrote: >> On Tue, Mar 26, 2024 at 01:32:09PM +0000, John Garry wrote: >>>>> + u8 *p; >>>>> + >>>>> + size = ALIGN(size, ARCH_DMA_MINALIGN); >>> >>> >>> If this is a hisi_sas requirement, then why use ARCH_DMA_MINALIGN and not >>> 16B as minimum alignment? >>> >>> Or are we really talking about an arch requirement? >> >> One thing is that we should never allocate unaligned memory for >> anything DMA mapped, or data will be corrupted by non-coherent DMA. >> So ARCH_DMA_MINALIGN needs to be here. If specific hardware has >> further requirements we'll need to communicated it through a field >> or op vector. > > Got it. Looks like it's still going to be aligned to ARCH_DMA_MINALIGN. But I thought that the original issue was that some arch have ARCH_DMA_MINALIGN down to 8B but hisi driver needs at least 16 ? So in the end, you need something like: size = ALIGN(size, max(16, ARCH_DMA_MINALIGN)); no ? And define a macro for the "16" value to document it. Something like: #define SAS_SMP_REQ_ALIGN 16 Not sure about the name... Naming is hard :) > > Thanks, > Yihang > >> >> >> . >>
On Thu, Mar 28, 2024 at 04:23:22PM +0900, Damien Le Moal wrote: > But I thought that the original issue was that some arch have ARCH_DMA_MINALIGN > down to 8B but hisi driver needs at least 16 ? > > So in the end, you need something like: > > size = ALIGN(size, max(16, ARCH_DMA_MINALIGN)); > > no ? I don't think we ever have an 8 byte dma minalign. With 8-byte aligned addresses dma_mapping_error could run into problems.
On 3/28/24 16:29, Christoph Hellwig wrote: > On Thu, Mar 28, 2024 at 04:23:22PM +0900, Damien Le Moal wrote: >> But I thought that the original issue was that some arch have ARCH_DMA_MINALIGN >> down to 8B but hisi driver needs at least 16 ? >> >> So in the end, you need something like: >> >> size = ALIGN(size, max(16, ARCH_DMA_MINALIGN)); >> >> no ? > > I don't think we ever have an 8 byte dma minalign. With 8-byte > aligned addresses dma_mapping_error could run into problems. My bad: it is kmalloc() that can return something aligned to 8B... So "size = ALIGN(size, ARCH_DMA_MINALIGN);" is the right thing to do.
On Thu, Mar 28, 2024 at 04:36:12PM +0900, Damien Le Moal wrote:
> My bad: it is kmalloc() that can return something aligned to 8B...
Yes, that's new on arm64, and possibly soon riscv.
On 2024/3/28 15:45, Christoph Hellwig wrote: > On Thu, Mar 28, 2024 at 04:36:12PM +0900, Damien Le Moal wrote: >> My bad: it is kmalloc() that can return something aligned to 8B... > > Yes, that's new on arm64, and possibly soon riscv. Thanks for the discussion, I will still aligned to ARCH_DMA_MINALIGN in v4.
diff --git a/drivers/scsi/libsas/sas_expander.c b/drivers/scsi/libsas/sas_expander.c index a2204674b680..941abc7298df 100644 --- a/drivers/scsi/libsas/sas_expander.c +++ b/drivers/scsi/libsas/sas_expander.c @@ -135,7 +135,10 @@ static int smp_execute_task(struct domain_device *dev, void *req, int req_size, static inline void *alloc_smp_req(int size) { - u8 *p = kzalloc(size, GFP_KERNEL); + u8 *p; + + size = ALIGN(size, ARCH_DMA_MINALIGN); + p = kzalloc(size, GFP_KERNEL); if (p) p[0] = SMP_REQUEST; return p;
This series [1] reducing the kmalloc() minimum alignment on arm64 to 8 (from 128). In libsas, this will cause SMP requests to be 8-byte-aligned through kmalloc() allocation. However, for the hisi_sas hardware, all commands address must be 16-byte-aligned. Otherwise, the commands fail to be executed. ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA operations, so use ARCH_DMA_MINALIGN as the alignment for SMP request. Link: https://lkml.kernel.org/r/20230612153201.554742-1-catalin.marinas@arm.com [1] Signed-off-by: Yihang Li <liyihang9@huawei.com> --- Changes since v1: - Directly modify alloc_smp_req() instead of using handler callback. --- drivers/scsi/libsas/sas_expander.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)