| Message ID | 20200831161854.70879-1-dwagner@suse.de (mailing list archive) |
|---|---|
| Headers | show |
| Series | qla2xxx: A couple crash fixes | expand |
On Mon, 2020-08-31 at 18:18 +0200, Daniel Wagner wrote: > changes since v1: > > - added dummy warn function to patch#1 > - added log entry to patch#4 > > as suggested by Martin > > > > Initial cover letter: > > The first crash we observed is due memory corruption in the srb > memory > pool. Unforuntatly, I couldn't find the source of the problem but the > workaround by resetting the cleanup callbacks 'fixes' this problem > (patch #1). I think as intermeditate step this should be merged until > the real cause can be identified. > > The second crash is due a race condition(?) in the firmware. The sts > entries are not updated in time which leads to this crash pattern > which several customers have reported: > > #0 [c00000ffffd1bb80] scsi_dma_unmap at d00000001e4904d4 [scsi_mod] > #1 [c00000ffffd1bbe0] qla2x00_sp_compl at d0000000204803cc [qla2xxx] > #2 [c00000ffffd1bc20] qla24xx_process_response_queue at > d0000000204c5810 [qla2xxx] > #3 [c00000ffffd1bd50] qla24xx_msix_rsp_q at d0000000204c8fd8 > [qla2xxx] > #4 [c00000ffffd1bde0] __handle_irq_event_percpu at c000000000189510 > #5 [c00000ffffd1bea0] handle_irq_event_percpu at c00000000018978c > #6 [c00000ffffd1bee0] handle_irq_event at c00000000018984c > #7 [c00000ffffd1bf10] handle_fasteoi_irq at c00000000018efc0 > #8 [c00000ffffd1bf40] generic_handle_irq at c000000000187f10 > #9 [c00000ffffd1bf60] __do_irq at c000000000018784 > #10 [c00000ffffd1bf90] call_do_irq at c00000000002caa4 > #11 [c00000ecca417a00] do_IRQ at c000000000018970 > #12 [c00000ecca417a50] restore_check_irq_replay at c00000000000de98 > > From analyzing the crash dump it was clear that > qla24xx_mbx_iocb_entry() calls sp->done (qla2x00_sp_compl) which > crashes because the response is not a mailbox entry, it is a status > entry. Patch #4 changes the process logic for mailbox commands so > that > the sp is parsed before calling the correct proccess function. > > Thanks, > Daniel > > Daniel Wagner (4): > qla2xxx: Warn if done() or free() are called on an already freed > srb > qla2xxx: Simplify return value logic in > qla2x00_get_sp_from_handle() > qla2xxx: Drop unused function argument from > qla2x00_get_sp_from_handle() > qla2xxx: Handle incorrect entry_type entries > > drivers/scsi/qla2xxx/qla_gbl.h | 3 +- > drivers/scsi/qla2xxx/qla_init.c | 10 ++++++ > drivers/scsi/qla2xxx/qla_inline.h | 5 +++ > drivers/scsi/qla2xxx/qla_isr.c | 74 +++++++++++++++++++++++---- > ------------ > drivers/scsi/qla2xxx/qla_mr.c | 9 ++--- > 5 files changed, 62 insertions(+), 39 deletions(-) > For the set: Reviewed-by: Martin Wilck <mwilck@suse.com>
Hi Daniel, On Mon, 31 Aug 2020, 9:18am, Daniel Wagner wrote: > changes since v1: > > - added dummy warn function to patch#1 > - added log entry to patch#4 > > as suggested by Martin > > > > Initial cover letter: > > The first crash we observed is due memory corruption in the srb memory > pool. Unforuntatly, I couldn't find the source of the problem but the > workaround by resetting the cleanup callbacks 'fixes' this problem > (patch #1). I think as intermeditate step this should be merged until > the real cause can be identified. > > The second crash is due a race condition(?) in the firmware. The sts > entries are not updated in time which leads to this crash pattern > which several customers have reported: > > #0 [c00000ffffd1bb80] scsi_dma_unmap at d00000001e4904d4 [scsi_mod] > #1 [c00000ffffd1bbe0] qla2x00_sp_compl at d0000000204803cc [qla2xxx] > #2 [c00000ffffd1bc20] qla24xx_process_response_queue at d0000000204c5810 [qla2xxx] > #3 [c00000ffffd1bd50] qla24xx_msix_rsp_q at d0000000204c8fd8 [qla2xxx] > #4 [c00000ffffd1bde0] __handle_irq_event_percpu at c000000000189510 > #5 [c00000ffffd1bea0] handle_irq_event_percpu at c00000000018978c > #6 [c00000ffffd1bee0] handle_irq_event at c00000000018984c > #7 [c00000ffffd1bf10] handle_fasteoi_irq at c00000000018efc0 > #8 [c00000ffffd1bf40] generic_handle_irq at c000000000187f10 > #9 [c00000ffffd1bf60] __do_irq at c000000000018784 > #10 [c00000ffffd1bf90] call_do_irq at c00000000002caa4 > #11 [c00000ecca417a00] do_IRQ at c000000000018970 > #12 [c00000ecca417a50] restore_check_irq_replay at c00000000000de98 > > From analyzing the crash dump it was clear that > qla24xx_mbx_iocb_entry() calls sp->done (qla2x00_sp_compl) which > crashes because the response is not a mailbox entry, it is a status > entry. Patch #4 changes the process logic for mailbox commands so that > the sp is parsed before calling the correct proccess function. > > Thanks, > Daniel > > Daniel Wagner (4): > qla2xxx: Warn if done() or free() are called on an already freed srb > qla2xxx: Simplify return value logic in qla2x00_get_sp_from_handle() > qla2xxx: Drop unused function argument from > qla2x00_get_sp_from_handle() > qla2xxx: Handle incorrect entry_type entries > > drivers/scsi/qla2xxx/qla_gbl.h | 3 +- > drivers/scsi/qla2xxx/qla_init.c | 10 ++++++ > drivers/scsi/qla2xxx/qla_inline.h | 5 +++ > drivers/scsi/qla2xxx/qla_isr.c | 74 +++++++++++++++++++++++---------------- > drivers/scsi/qla2xxx/qla_mr.c | 9 ++--- > 5 files changed, 62 insertions(+), 39 deletions(-) > > Thanks for the patches, my comments in the respective patches. Regards, -Arun
changes since v1: - added dummy warn function to patch#1 - added log entry to patch#4 as suggested by Martin Initial cover letter: The first crash we observed is due memory corruption in the srb memory pool. Unforuntatly, I couldn't find the source of the problem but the workaround by resetting the cleanup callbacks 'fixes' this problem (patch #1). I think as intermeditate step this should be merged until the real cause can be identified. The second crash is due a race condition(?) in the firmware. The sts entries are not updated in time which leads to this crash pattern which several customers have reported: #0 [c00000ffffd1bb80] scsi_dma_unmap at d00000001e4904d4 [scsi_mod] #1 [c00000ffffd1bbe0] qla2x00_sp_compl at d0000000204803cc [qla2xxx] #2 [c00000ffffd1bc20] qla24xx_process_response_queue at d0000000204c5810 [qla2xxx] #3 [c00000ffffd1bd50] qla24xx_msix_rsp_q at d0000000204c8fd8 [qla2xxx] #4 [c00000ffffd1bde0] __handle_irq_event_percpu at c000000000189510 #5 [c00000ffffd1bea0] handle_irq_event_percpu at c00000000018978c #6 [c00000ffffd1bee0] handle_irq_event at c00000000018984c #7 [c00000ffffd1bf10] handle_fasteoi_irq at c00000000018efc0 #8 [c00000ffffd1bf40] generic_handle_irq at c000000000187f10 #9 [c00000ffffd1bf60] __do_irq at c000000000018784 #10 [c00000ffffd1bf90] call_do_irq at c00000000002caa4 #11 [c00000ecca417a00] do_IRQ at c000000000018970 #12 [c00000ecca417a50] restore_check_irq_replay at c00000000000de98 From analyzing the crash dump it was clear that qla24xx_mbx_iocb_entry() calls sp->done (qla2x00_sp_compl) which crashes because the response is not a mailbox entry, it is a status entry. Patch #4 changes the process logic for mailbox commands so that the sp is parsed before calling the correct proccess function. Thanks, Daniel Daniel Wagner (4): qla2xxx: Warn if done() or free() are called on an already freed srb qla2xxx: Simplify return value logic in qla2x00_get_sp_from_handle() qla2xxx: Drop unused function argument from qla2x00_get_sp_from_handle() qla2xxx: Handle incorrect entry_type entries drivers/scsi/qla2xxx/qla_gbl.h | 3 +- drivers/scsi/qla2xxx/qla_init.c | 10 ++++++ drivers/scsi/qla2xxx/qla_inline.h | 5 +++ drivers/scsi/qla2xxx/qla_isr.c | 74 +++++++++++++++++++++++---------------- drivers/scsi/qla2xxx/qla_mr.c | 9 ++--- 5 files changed, 62 insertions(+), 39 deletions(-)