mbox series

[v2,0/4] qla2xxx: A couple crash fixes

Message ID 20200831161854.70879-1-dwagner@suse.de (mailing list archive)
Headers show
Series qla2xxx: A couple crash fixes | expand

Message

Daniel Wagner Aug. 31, 2020, 4:18 p.m. UTC
changes since v1:

 - added dummy warn function to patch#1
 - added log entry to patch#4

as suggested by Martin



Initial cover letter:

The first crash we observed is due memory corruption in the srb memory
pool. Unforuntatly, I couldn't find the source of the problem but the
workaround by resetting the cleanup callbacks 'fixes' this problem
(patch #1). I think as intermeditate step this should be merged until
the real cause can be identified.

The second crash is due a race condition(?) in the firmware. The sts
entries are not updated in time which leads to this crash pattern
which several customers have reported:

 #0 [c00000ffffd1bb80] scsi_dma_unmap at d00000001e4904d4 [scsi_mod]
 #1 [c00000ffffd1bbe0] qla2x00_sp_compl at d0000000204803cc [qla2xxx]
 #2 [c00000ffffd1bc20] qla24xx_process_response_queue at d0000000204c5810 [qla2xxx]
 #3 [c00000ffffd1bd50] qla24xx_msix_rsp_q at d0000000204c8fd8 [qla2xxx]
 #4 [c00000ffffd1bde0] __handle_irq_event_percpu at c000000000189510
 #5 [c00000ffffd1bea0] handle_irq_event_percpu at c00000000018978c
 #6 [c00000ffffd1bee0] handle_irq_event at c00000000018984c
 #7 [c00000ffffd1bf10] handle_fasteoi_irq at c00000000018efc0
 #8 [c00000ffffd1bf40] generic_handle_irq at c000000000187f10
 #9 [c00000ffffd1bf60] __do_irq at c000000000018784
 #10 [c00000ffffd1bf90] call_do_irq at c00000000002caa4
 #11 [c00000ecca417a00] do_IRQ at c000000000018970
 #12 [c00000ecca417a50] restore_check_irq_replay at c00000000000de98

From analyzing the crash dump it was clear that
qla24xx_mbx_iocb_entry() calls sp->done (qla2x00_sp_compl) which
crashes because the response is not a mailbox entry, it is a status
entry. Patch #4 changes the process logic for mailbox commands so that
the sp is parsed before calling the correct proccess function.

Thanks,
Daniel

Daniel Wagner (4):
  qla2xxx: Warn if done() or free() are called on an already freed srb
  qla2xxx: Simplify return value logic in qla2x00_get_sp_from_handle()
  qla2xxx: Drop unused function argument from
    qla2x00_get_sp_from_handle()
  qla2xxx: Handle incorrect entry_type entries

 drivers/scsi/qla2xxx/qla_gbl.h    |  3 +-
 drivers/scsi/qla2xxx/qla_init.c   | 10 ++++++
 drivers/scsi/qla2xxx/qla_inline.h |  5 +++
 drivers/scsi/qla2xxx/qla_isr.c    | 74 +++++++++++++++++++++++----------------
 drivers/scsi/qla2xxx/qla_mr.c     |  9 ++---
 5 files changed, 62 insertions(+), 39 deletions(-)

Comments

Martin Wilck Aug. 31, 2020, 7:11 p.m. UTC | #1
On Mon, 2020-08-31 at 18:18 +0200, Daniel Wagner wrote:
> changes since v1:
> 
>  - added dummy warn function to patch#1
>  - added log entry to patch#4
> 
> as suggested by Martin
> 
> 
> 
> Initial cover letter:
> 
> The first crash we observed is due memory corruption in the srb
> memory
> pool. Unforuntatly, I couldn't find the source of the problem but the
> workaround by resetting the cleanup callbacks 'fixes' this problem
> (patch #1). I think as intermeditate step this should be merged until
> the real cause can be identified.
> 
> The second crash is due a race condition(?) in the firmware. The sts
> entries are not updated in time which leads to this crash pattern
> which several customers have reported:
> 
>  #0 [c00000ffffd1bb80] scsi_dma_unmap at d00000001e4904d4 [scsi_mod]
>  #1 [c00000ffffd1bbe0] qla2x00_sp_compl at d0000000204803cc [qla2xxx]
>  #2 [c00000ffffd1bc20] qla24xx_process_response_queue at
> d0000000204c5810 [qla2xxx]
>  #3 [c00000ffffd1bd50] qla24xx_msix_rsp_q at d0000000204c8fd8
> [qla2xxx]
>  #4 [c00000ffffd1bde0] __handle_irq_event_percpu at c000000000189510
>  #5 [c00000ffffd1bea0] handle_irq_event_percpu at c00000000018978c
>  #6 [c00000ffffd1bee0] handle_irq_event at c00000000018984c
>  #7 [c00000ffffd1bf10] handle_fasteoi_irq at c00000000018efc0
>  #8 [c00000ffffd1bf40] generic_handle_irq at c000000000187f10
>  #9 [c00000ffffd1bf60] __do_irq at c000000000018784
>  #10 [c00000ffffd1bf90] call_do_irq at c00000000002caa4
>  #11 [c00000ecca417a00] do_IRQ at c000000000018970
>  #12 [c00000ecca417a50] restore_check_irq_replay at c00000000000de98
> 
> From analyzing the crash dump it was clear that
> qla24xx_mbx_iocb_entry() calls sp->done (qla2x00_sp_compl) which
> crashes because the response is not a mailbox entry, it is a status
> entry. Patch #4 changes the process logic for mailbox commands so
> that
> the sp is parsed before calling the correct proccess function.
> 
> Thanks,
> Daniel
> 
> Daniel Wagner (4):
>   qla2xxx: Warn if done() or free() are called on an already freed
> srb
>   qla2xxx: Simplify return value logic in
> qla2x00_get_sp_from_handle()
>   qla2xxx: Drop unused function argument from
>     qla2x00_get_sp_from_handle()
>   qla2xxx: Handle incorrect entry_type entries
> 
>  drivers/scsi/qla2xxx/qla_gbl.h    |  3 +-
>  drivers/scsi/qla2xxx/qla_init.c   | 10 ++++++
>  drivers/scsi/qla2xxx/qla_inline.h |  5 +++
>  drivers/scsi/qla2xxx/qla_isr.c    | 74 +++++++++++++++++++++++----
> ------------
>  drivers/scsi/qla2xxx/qla_mr.c     |  9 ++---
>  5 files changed, 62 insertions(+), 39 deletions(-)
> 

For the set:
Reviewed-by: Martin Wilck <mwilck@suse.com>
Arun Easi Sept. 8, 2020, 6:41 a.m. UTC | #2
Hi Daniel,

On Mon, 31 Aug 2020, 9:18am, Daniel Wagner wrote:

> changes since v1:
> 
>  - added dummy warn function to patch#1
>  - added log entry to patch#4
> 
> as suggested by Martin
> 
> 
> 
> Initial cover letter:
> 
> The first crash we observed is due memory corruption in the srb memory
> pool. Unforuntatly, I couldn't find the source of the problem but the
> workaround by resetting the cleanup callbacks 'fixes' this problem
> (patch #1). I think as intermeditate step this should be merged until
> the real cause can be identified.
> 
> The second crash is due a race condition(?) in the firmware. The sts
> entries are not updated in time which leads to this crash pattern
> which several customers have reported:
> 
>  #0 [c00000ffffd1bb80] scsi_dma_unmap at d00000001e4904d4 [scsi_mod]
>  #1 [c00000ffffd1bbe0] qla2x00_sp_compl at d0000000204803cc [qla2xxx]
>  #2 [c00000ffffd1bc20] qla24xx_process_response_queue at d0000000204c5810 [qla2xxx]
>  #3 [c00000ffffd1bd50] qla24xx_msix_rsp_q at d0000000204c8fd8 [qla2xxx]
>  #4 [c00000ffffd1bde0] __handle_irq_event_percpu at c000000000189510
>  #5 [c00000ffffd1bea0] handle_irq_event_percpu at c00000000018978c
>  #6 [c00000ffffd1bee0] handle_irq_event at c00000000018984c
>  #7 [c00000ffffd1bf10] handle_fasteoi_irq at c00000000018efc0
>  #8 [c00000ffffd1bf40] generic_handle_irq at c000000000187f10
>  #9 [c00000ffffd1bf60] __do_irq at c000000000018784
>  #10 [c00000ffffd1bf90] call_do_irq at c00000000002caa4
>  #11 [c00000ecca417a00] do_IRQ at c000000000018970
>  #12 [c00000ecca417a50] restore_check_irq_replay at c00000000000de98
> 
> From analyzing the crash dump it was clear that
> qla24xx_mbx_iocb_entry() calls sp->done (qla2x00_sp_compl) which
> crashes because the response is not a mailbox entry, it is a status
> entry. Patch #4 changes the process logic for mailbox commands so that
> the sp is parsed before calling the correct proccess function.
> 
> Thanks,
> Daniel
> 
> Daniel Wagner (4):
>   qla2xxx: Warn if done() or free() are called on an already freed srb
>   qla2xxx: Simplify return value logic in qla2x00_get_sp_from_handle()
>   qla2xxx: Drop unused function argument from
>     qla2x00_get_sp_from_handle()
>   qla2xxx: Handle incorrect entry_type entries
> 
>  drivers/scsi/qla2xxx/qla_gbl.h    |  3 +-
>  drivers/scsi/qla2xxx/qla_init.c   | 10 ++++++
>  drivers/scsi/qla2xxx/qla_inline.h |  5 +++
>  drivers/scsi/qla2xxx/qla_isr.c    | 74 +++++++++++++++++++++++----------------
>  drivers/scsi/qla2xxx/qla_mr.c     |  9 ++---
>  5 files changed, 62 insertions(+), 39 deletions(-)
> 
> 

Thanks for the patches, my comments in the respective patches.

Regards,
-Arun