From patchwork Mon Feb 15 07:40:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kashyap Desai X-Patchwork-Id: 12088815 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DATE_IN_PAST_06_12, DKIMWL_WL_HIGH,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MIME_HEADER_CTYPE_ONLY,SPF_HELO_NONE,SPF_PASS, T_TVD_MIME_NO_HEADERS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D084BC433DB for ; Mon, 15 Feb 2021 16:08:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9477064DF0 for ; Mon, 15 Feb 2021 16:08:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232067AbhBOQHh (ORCPT ); Mon, 15 Feb 2021 11:07:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43432 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232296AbhBOPxL (ORCPT ); Mon, 15 Feb 2021 10:53:11 -0500 Received: from mail-pf1-x436.google.com (mail-pf1-x436.google.com [IPv6:2607:f8b0:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6BDFAC06178A for ; Mon, 15 Feb 2021 07:40:27 -0800 (PST) Received: by mail-pf1-x436.google.com with SMTP id x136so4403847pfc.2 for ; Mon, 15 Feb 2021 07:40:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=XGWB1BBx5MRHe5YKpjzmk3p1CXZow0D617mh7N0Z1+4=; b=PdVnDMQWQWk9GaCgzPCA1D9RejbWpCZpB+1xjDujak/KbJ0zhtvBmQLXh8vyNb92vv PbcIN3DazMH1nTeOOOo1oryieoWjwSRiRTEWfVPww7+0Shk5S0bHD1AxO+uKAhnbMi5A dCL6q+NJjEs+NmMTyWEEU1RnYkxyu2McWKTwg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=XGWB1BBx5MRHe5YKpjzmk3p1CXZow0D617mh7N0Z1+4=; b=oaLPnlffk+B8xTZgGVddx8dlVeczbmjxVK8dR5AVIpKX3j3pkN+o4fJpwczuhd8ipj T92SdR9TqFx1OKDSPyPcYKMXHPiZj/kySvKkhRkt8pPH+KiFZjK/EWq63a/aJDyNPvGS /0F9YMz65JGLxrqIE3TWdvF/j3OD06w0y5ZnMBaPhTjY2eI6UPpt/hXpTeHlNfRdY4is hrI0cRWHwya7KP9j5BXnwceCjrs8oO9oZpEAJMgnQ11j9OW8kh7mmmXAA+q/km/0Bjb+ gMRG5y5xciJ1VzSrrGcNvOnROnKKgKp/5bsKLeivOoLrzDSOZa1lEVPv1qNZoJY4qAHP vuTA== X-Gm-Message-State: AOAM532Nl4dj3f+m9UYO0imrQmSNszxdvzgmwQhcxi9zUsY47jKxDv9T /wX/S6Ooqc1BaB5B+GPCp6+xx77qtanUSKY2b06qIE1Vdbjqk60p1+TK86C6/RDORKbGOEC+8po y1UstqTyVaemaxOLlChReiiEnhACBD7Z35DGJK8y8CR/oEcdMlDwcyWq1tKPW5rDLVsq3HOXD2f QQ5v3wg6Dy X-Google-Smtp-Source: ABdhPJwOyR1OmAYtMg3aBKut9tJkP7RSLeB0pUovOFWp6vaOsPBbIadW0rTVGJCqPUGyM2YLljXFvw== X-Received: by 2002:a63:df11:: with SMTP id u17mr15142811pgg.345.1613403626618; Mon, 15 Feb 2021 07:40:26 -0800 (PST) Received: from drv-bst-rhel8.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id m4sm18031341pgu.4.2021.02.15.07.40.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Feb 2021 07:40:26 -0800 (PST) From: Kashyap Desai To: linux-scsi@vger.kernel.org Cc: Kashyap Desai , sumit.saxena@broadcom.com, chandrakanth.patil@broadcom.com, linux-block@vger.kernel.org Subject: [PATCH v4 1/5] add io_uring with IOPOLL support in scsi layer Date: Mon, 15 Feb 2021 13:10:44 +0530 Message-Id: <20210215074048.19424-2-kashyap.desai@broadcom.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20210215074048.19424-1-kashyap.desai@broadcom.com> References: <20210215074048.19424-1-kashyap.desai@broadcom.com> Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org io_uring with IOPOLL is not currently supported in scsi mid layer. Outside of that everything else should work and no extra support in the driver is needed. Currently io_uring with IOPOLL support is only available in block layer. This patch is to extend support of mq_poll in scsi layer. Signed-off-by: Kashyap Desai Reviewed-by: Hannes Reinecke Reviewed-by: John Garry Cc: sumit.saxena@broadcom.com Cc: chandrakanth.patil@broadcom.com Cc: linux-block@vger.kernel.org --- drivers/scsi/scsi_lib.c | 16 ++++++++++++++++ include/scsi/scsi_cmnd.h | 1 + include/scsi/scsi_host.h | 11 +++++++++++ 3 files changed, 28 insertions(+) diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c index d0ae586565f8..8c29bf0e4cfd 100644 --- a/drivers/scsi/scsi_lib.c +++ b/drivers/scsi/scsi_lib.c @@ -1789,6 +1789,19 @@ static void scsi_mq_exit_request(struct blk_mq_tag_set *set, struct request *rq, cmd->sense_buffer); } + +static int scsi_mq_poll(struct blk_mq_hw_ctx *hctx) +{ + struct request_queue *q = hctx->queue; + struct scsi_device *sdev = q->queuedata; + struct Scsi_Host *shost = sdev->host; + + if (shost->hostt->mq_poll) + return shost->hostt->mq_poll(shost, hctx->queue_num); + + return 0; +} + static int scsi_map_queues(struct blk_mq_tag_set *set) { struct Scsi_Host *shost = container_of(set, struct Scsi_Host, tag_set); @@ -1856,6 +1869,7 @@ static const struct blk_mq_ops scsi_mq_ops_no_commit = { .cleanup_rq = scsi_cleanup_rq, .busy = scsi_mq_lld_busy, .map_queues = scsi_map_queues, + .poll = scsi_mq_poll, }; @@ -1884,6 +1898,7 @@ static const struct blk_mq_ops scsi_mq_ops = { .cleanup_rq = scsi_cleanup_rq, .busy = scsi_mq_lld_busy, .map_queues = scsi_map_queues, + .poll = scsi_mq_poll, }; struct request_queue *scsi_mq_alloc_queue(struct scsi_device *sdev) @@ -1916,6 +1931,7 @@ int scsi_mq_setup_tags(struct Scsi_Host *shost) else tag_set->ops = &scsi_mq_ops_no_commit; tag_set->nr_hw_queues = shost->nr_hw_queues ? : 1; + tag_set->nr_maps = shost->nr_maps ? : 1; tag_set->queue_depth = shost->can_queue; tag_set->cmd_size = cmd_size; tag_set->numa_node = NUMA_NO_NODE; diff --git a/include/scsi/scsi_cmnd.h b/include/scsi/scsi_cmnd.h index ace15b5dc956..1d8a0f6ea8c5 100644 --- a/include/scsi/scsi_cmnd.h +++ b/include/scsi/scsi_cmnd.h @@ -10,6 +10,7 @@ #include #include #include +#include #include struct Scsi_Host; diff --git a/include/scsi/scsi_host.h b/include/scsi/scsi_host.h index e30fd963b97d..3d627bf7b951 100644 --- a/include/scsi/scsi_host.h +++ b/include/scsi/scsi_host.h @@ -270,6 +270,16 @@ struct scsi_host_template { */ int (* map_queues)(struct Scsi_Host *shost); + /* + * SCSI interface of blk_poll - poll for IO completions. + * Possible interface only if scsi LLD expose multiple h/w queues. + * + * Return value: Number of completed entries found. + * + * Status: OPTIONAL + */ + int (* mq_poll)(struct Scsi_Host *shost, unsigned int queue_num); + /* * Check if scatterlists need to be padded for DMA draining. * @@ -616,6 +626,7 @@ struct Scsi_Host { * the total queue depth is can_queue. */ unsigned nr_hw_queues; + unsigned nr_maps; unsigned active_mode:2; unsigned unchecked_isa_dma:1; From patchwork Mon Feb 15 07:40:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kashyap Desai X-Patchwork-Id: 12088469 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DATE_IN_PAST_06_12, DKIMWL_WL_HIGH,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MIME_HEADER_CTYPE_ONLY,SPF_HELO_NONE,SPF_PASS, T_TVD_MIME_NO_HEADERS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66419C433E6 for ; Mon, 15 Feb 2021 15:44:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2B78064E8E for ; Mon, 15 Feb 2021 15:44:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231436AbhBOPoP (ORCPT ); Mon, 15 Feb 2021 10:44:15 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41040 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230232AbhBOPnZ (ORCPT ); Mon, 15 Feb 2021 10:43:25 -0500 Received: from mail-pj1-x102b.google.com (mail-pj1-x102b.google.com [IPv6:2607:f8b0:4864:20::102b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C9831C06178C for ; Mon, 15 Feb 2021 07:40:29 -0800 (PST) Received: by mail-pj1-x102b.google.com with SMTP id l18so4132891pji.3 for ; Mon, 15 Feb 2021 07:40:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=s5C6MDrHPuMCawmzOm6qQQTsGxX+e2GqXd2tUvS1FuY=; b=SBLTD/ndwdTnxwN+nZs1cPgG42DSzh6lJZvflPtyOF2gYb0CrUm0VLBC4jQbFEw/la N1RowxNW+Uu589xsSTmUVNPdczd0C/fbYwKCL9kykHz8O9zywGnaCUkV7bX87nKXt7tG Dxk9abEv8nxG+qfyo+KoO9bCkvynmn9NIALwI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=s5C6MDrHPuMCawmzOm6qQQTsGxX+e2GqXd2tUvS1FuY=; b=COMXpZn4anJZ0s2DnCBYXSGLhTLroDfISVFzYW0AHF8AXhkPwWVDJq+yOTVApUwq5S lPpxKqYslpd26Bra6F2sKnwM5Q+FVJiZpS5aokN6Ge2L3Tu7YBWdxvTX5Xo3sL8wLQ/5 gNxsBl1+aWKTsQlU3kUmpgq8Du0BZxIcjY2aa2R00mIyL1XtXRS3DBmPqJmBHvRQiXU8 Qmw7I5lcV8PbIPQraJ9OxMYN3k8AVEbd3SPMLNlNFMjtbOsJERlyXkkjUhzR36POikbF XKcMJFZ/fGARFa4GXRzLgibf6m0Prddq31OFWixiUFxlscsWD1B9BM30tbx0wPxfQvWY WC/Q== X-Gm-Message-State: AOAM532scYj0YmtzpY1ExLsAVjlchcnPvuwDzYMXbxYJ3tU2d4cFuELJ xd/2Q+Mc47PKspKoAnlIjSDZTAnM4jXlE7xSIvEjTvNtLE8UYmazzgs/58d00N93AuXYa36Cy0s /yxHnxTkaKg4FBYizIKL0UtpEseUo5yISDU1msgQ0JrqAd6tq6OI/fhihkIZiFuAkQzb/YKc23y QH0r7YfkJo X-Google-Smtp-Source: ABdhPJx3WzSPqP5P0FOIs/xLY+w+n7g7NkgZOt4f31vv+8/M2ozuGlMJn1Kv6+P2uuGGPDkpErSV5A== X-Received: by 2002:a17:90a:206:: with SMTP id c6mr17062411pjc.50.1613403628970; Mon, 15 Feb 2021 07:40:28 -0800 (PST) Received: from drv-bst-rhel8.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id m4sm18031341pgu.4.2021.02.15.07.40.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Feb 2021 07:40:28 -0800 (PST) From: Kashyap Desai To: linux-scsi@vger.kernel.org Cc: Kashyap Desai , sumit.saxena@broadcom.com, chandrakanth.patil@broadcom.com, linux-block@vger.kernel.org Subject: [PATCH v4 2/5] megaraid_sas: iouring iopoll support Date: Mon, 15 Feb 2021 13:10:45 +0530 Message-Id: <20210215074048.19424-3-kashyap.desai@broadcom.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20210215074048.19424-1-kashyap.desai@broadcom.com> References: <20210215074048.19424-1-kashyap.desai@broadcom.com> Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org Add support of iouring iopoll interface. This feature requires shared hosttag support in kernel and driver. Driver will work in non-IRQ mode = There will not be any msix vector associated for poll_queues and h/w can still work in this mode. MegaRaid h/w is single submission queue and multiple reply queue, but using shared host tagset support it will enable simulated multiple hw queue. Driver allocates some extra reply queues and it will be marked as poll_queue. These poll_queues will not have associated msix vectors. All the IO completion on this queue will be done from IOPOLL interface. megaraid_sas driver having 8 poll_queues and using io_uring hiprio=1 settings, It can reach 3.2M IOPs and there is zero interrupt generated by h/w. This feature can be enabled using module parameter poll_queues. Signed-off-by: Kashyap Desai Reviewed-by: Hannes Reinecke Cc: sumit.saxena@broadcom.com Cc: chandrakanth.patil@broadcom.com Cc: linux-block@vger.kernel.org --- drivers/scsi/megaraid/megaraid_sas.h | 3 + drivers/scsi/megaraid/megaraid_sas_base.c | 87 ++++++++++++++++++--- drivers/scsi/megaraid/megaraid_sas_fusion.c | 42 +++++++++- drivers/scsi/megaraid/megaraid_sas_fusion.h | 2 + 4 files changed, 123 insertions(+), 11 deletions(-) diff --git a/drivers/scsi/megaraid/megaraid_sas.h b/drivers/scsi/megaraid/megaraid_sas.h index 0f808d63580e..d8b1797e2768 100644 --- a/drivers/scsi/megaraid/megaraid_sas.h +++ b/drivers/scsi/megaraid/megaraid_sas.h @@ -2212,6 +2212,7 @@ struct megasas_irq_context { struct irq_poll irqpoll; bool irq_poll_scheduled; bool irq_line_enable; + atomic_t in_used; }; struct MR_DRV_SYSTEM_INFO { @@ -2446,6 +2447,7 @@ struct megasas_instance { bool support_pci_lane_margining; u8 low_latency_index_start; int perf_mode; + int iopoll_q_count; }; struct MR_LD_VF_MAP { @@ -2726,5 +2728,6 @@ void megasas_init_debugfs(void); void megasas_exit_debugfs(void); void megasas_setup_debugfs(struct megasas_instance *instance); void megasas_destroy_debugfs(struct megasas_instance *instance); +int megasas_blk_mq_poll(struct Scsi_Host *shost, unsigned int queue_num); #endif /*LSI_MEGARAID_SAS_H */ diff --git a/drivers/scsi/megaraid/megaraid_sas_base.c b/drivers/scsi/megaraid/megaraid_sas_base.c index 63a4f48bdc75..25673d0ee524 100644 --- a/drivers/scsi/megaraid/megaraid_sas_base.c +++ b/drivers/scsi/megaraid/megaraid_sas_base.c @@ -114,6 +114,15 @@ unsigned int enable_sdev_max_qd; module_param(enable_sdev_max_qd, int, 0444); MODULE_PARM_DESC(enable_sdev_max_qd, "Enable sdev max qd as can_queue. Default: 0"); +int poll_queues; +module_param(poll_queues, int, 0444); +MODULE_PARM_DESC(poll_queues, "Number of queues to be use for io_uring poll mode.\n\t\t" + "This parameter is effective only if host_tagset_enable=1 &\n\t\t" + "It is not applicable for MFI_SERIES. &\n\t\t" + "Driver will work in latency mode. &\n\t\t" + "High iops queues are not allocated &\n\t\t" + ); + int host_tagset_enable = 1; module_param(host_tagset_enable, int, 0444); MODULE_PARM_DESC(host_tagset_enable, "Shared host tagset enable/disable Default: enable(1)"); @@ -207,6 +216,7 @@ static bool support_pci_lane_margining; static spinlock_t poll_aen_lock; extern struct dentry *megasas_debugfs_root; +extern int megasas_blk_mq_poll(struct Scsi_Host *shost, unsigned int queue_num); void megasas_complete_cmd(struct megasas_instance *instance, struct megasas_cmd *cmd, @@ -3127,14 +3137,37 @@ megasas_bios_param(struct scsi_device *sdev, struct block_device *bdev, static int megasas_map_queues(struct Scsi_Host *shost) { struct megasas_instance *instance; + int qoff = 0, offset; + struct blk_mq_queue_map *map; instance = (struct megasas_instance *)shost->hostdata; if (shost->nr_hw_queues == 1) return 0; - return blk_mq_pci_map_queues(&shost->tag_set.map[HCTX_TYPE_DEFAULT], - instance->pdev, instance->low_latency_index_start); + offset = instance->low_latency_index_start; + + /* Setup Default hctx */ + map = &shost->tag_set.map[HCTX_TYPE_DEFAULT]; + map->nr_queues = instance->msix_vectors - offset; + map->queue_offset = 0; + blk_mq_pci_map_queues(map, instance->pdev, offset); + qoff += map->nr_queues; + offset += map->nr_queues; + + /* Setup Poll hctx */ + map = &shost->tag_set.map[HCTX_TYPE_POLL]; + map->nr_queues = instance->iopoll_q_count; + if (map->nr_queues) { + /* + * The poll queue(s) doesn't have an IRQ (and hence IRQ + * affinity), so use the regular blk-mq cpu mapping + */ + map->queue_offset = qoff; + blk_mq_map_queues(map); + } + + return 0; } static void megasas_aen_polling(struct work_struct *work); @@ -3446,6 +3479,7 @@ static struct scsi_host_template megasas_template = { .shost_attrs = megaraid_host_attrs, .bios_param = megasas_bios_param, .map_queues = megasas_map_queues, + .mq_poll = megasas_blk_mq_poll, .change_queue_depth = scsi_change_queue_depth, .max_segment_size = 0xffffffff, }; @@ -5834,13 +5868,16 @@ __megasas_alloc_irq_vectors(struct megasas_instance *instance) irq_flags = PCI_IRQ_MSIX; if (instance->smp_affinity_enable) - irq_flags |= PCI_IRQ_AFFINITY; + irq_flags |= PCI_IRQ_AFFINITY | PCI_IRQ_ALL_TYPES; else descp = NULL; + /* Do not allocate msix vectors for poll_queues. + * msix_vectors is always within a range of FW supported reply queue. + */ i = pci_alloc_irq_vectors_affinity(instance->pdev, instance->low_latency_index_start, - instance->msix_vectors, irq_flags, descp); + instance->msix_vectors - instance->iopoll_q_count, irq_flags, descp); return i; } @@ -5856,10 +5893,30 @@ megasas_alloc_irq_vectors(struct megasas_instance *instance) int i; unsigned int num_msix_req; + instance->iopoll_q_count = 0; + if ((instance->adapter_type != MFI_SERIES) && + poll_queues) { + + instance->perf_mode = MR_LATENCY_PERF_MODE; + instance->low_latency_index_start = 1; + + /* reserve for default and non-mananged pre-vector. */ + if (instance->msix_vectors > (poll_queues + 2)) + instance->iopoll_q_count = poll_queues; + else + instance->iopoll_q_count = 0; + + num_msix_req = num_online_cpus() + instance->low_latency_index_start; + instance->msix_vectors = min(num_msix_req, + instance->msix_vectors); + + } + i = __megasas_alloc_irq_vectors(instance); - if ((instance->perf_mode == MR_BALANCED_PERF_MODE) && - (i != instance->msix_vectors)) { + if (((instance->perf_mode == MR_BALANCED_PERF_MODE) + || instance->iopoll_q_count) && + (i != (instance->msix_vectors - instance->iopoll_q_count))) { if (instance->msix_vectors) pci_free_irq_vectors(instance->pdev); /* Disable Balanced IOPS mode and try realloc vectors */ @@ -5870,12 +5927,15 @@ megasas_alloc_irq_vectors(struct megasas_instance *instance) instance->msix_vectors = min(num_msix_req, instance->msix_vectors); + instance->iopoll_q_count = 0; i = __megasas_alloc_irq_vectors(instance); } dev_info(&instance->pdev->dev, - "requested/available msix %d/%d\n", instance->msix_vectors, i); + "requested/available msix %d/%d poll_queue %d\n", + instance->msix_vectors - instance->iopoll_q_count, + i, instance->iopoll_q_count); if (i > 0) instance->msix_vectors = i; @@ -6841,12 +6901,18 @@ static int megasas_io_attach(struct megasas_instance *instance) instance->smp_affinity_enable) { host->host_tagset = 1; host->nr_hw_queues = instance->msix_vectors - - instance->low_latency_index_start; + instance->low_latency_index_start + instance->iopoll_q_count; + if (instance->iopoll_q_count) + host->nr_maps = 3; + } else { + instance->iopoll_q_count = 0; } dev_info(&instance->pdev->dev, - "Max firmware commands: %d shared with nr_hw_queues = %d\n", - instance->max_fw_cmds, host->nr_hw_queues); + "Max firmware commands: %d shared with default " + "hw_queues = %d poll_queues %d\n", instance->max_fw_cmds, + host->nr_hw_queues - instance->iopoll_q_count, + instance->iopoll_q_count); /* * Notify the mid-layer about the new controller */ @@ -8859,6 +8925,7 @@ static int __init megasas_init(void) msix_vectors = 1; rdpq_enable = 0; dual_qdepth_disable = 1; + poll_queues = 0; } /* diff --git a/drivers/scsi/megaraid/megaraid_sas_fusion.c b/drivers/scsi/megaraid/megaraid_sas_fusion.c index 38fc9467c625..10b8157044bb 100644 --- a/drivers/scsi/megaraid/megaraid_sas_fusion.c +++ b/drivers/scsi/megaraid/megaraid_sas_fusion.c @@ -685,6 +685,8 @@ megasas_alloc_reply_fusion(struct megasas_instance *instance) fusion = instance->ctrl_context; count = instance->msix_vectors > 0 ? instance->msix_vectors : 1; + count += instance->iopoll_q_count; + fusion->reply_frames_desc_pool = dma_pool_create("mr_reply", &instance->pdev->dev, fusion->reply_alloc_sz * count, 16, 0); @@ -779,6 +781,7 @@ megasas_alloc_rdpq_fusion(struct megasas_instance *instance) } msix_count = instance->msix_vectors > 0 ? instance->msix_vectors : 1; + msix_count += instance->iopoll_q_count; fusion->reply_frames_desc_pool = dma_pool_create("mr_rdpq", &instance->pdev->dev, @@ -1129,7 +1132,7 @@ megasas_ioc_init_fusion(struct megasas_instance *instance) MPI2_IOCINIT_MSGFLAG_RDPQ_ARRAY_MODE : 0; IOCInitMessage->SystemRequestFrameBaseAddress = cpu_to_le64(fusion->io_request_frames_phys); IOCInitMessage->SenseBufferAddressHigh = cpu_to_le32(upper_32_bits(fusion->sense_phys_addr)); - IOCInitMessage->HostMSIxVectors = instance->msix_vectors; + IOCInitMessage->HostMSIxVectors = instance->msix_vectors + instance->iopoll_q_count; IOCInitMessage->HostPageSize = MR_DEFAULT_NVME_PAGE_SHIFT; time = ktime_get_real(); @@ -1823,6 +1826,8 @@ megasas_init_adapter_fusion(struct megasas_instance *instance) sizeof(union MPI2_SGE_IO_UNION))/16; count = instance->msix_vectors > 0 ? instance->msix_vectors : 1; + count += instance->iopoll_q_count; + for (i = 0 ; i < count; i++) fusion->last_reply_idx[i] = 0; @@ -1835,6 +1840,9 @@ megasas_init_adapter_fusion(struct megasas_instance *instance) MEGASAS_FUSION_IOCTL_CMDS); sema_init(&instance->ioctl_sem, MEGASAS_FUSION_IOCTL_CMDS); + for (i = 0; i < MAX_MSIX_QUEUES_FUSION; i++) + atomic_set(&fusion->busy_mq_poll[i], 0); + if (megasas_alloc_ioc_init_frame(instance)) return 1; @@ -3500,6 +3508,9 @@ complete_cmd_fusion(struct megasas_instance *instance, u32 MSIxIndex, if (reply_descript_type == MPI2_RPY_DESCRIPT_FLAGS_UNUSED) return IRQ_NONE; + if (irq_context && !atomic_add_unless(&irq_context->in_used, 1, 1)) + return 0; + num_completed = 0; while (d_val.u.low != cpu_to_le32(UINT_MAX) && @@ -3613,6 +3624,7 @@ complete_cmd_fusion(struct megasas_instance *instance, u32 MSIxIndex, irq_context->irq_line_enable = true; irq_poll_sched(&irq_context->irqpoll); } + atomic_dec(&irq_context->in_used); return num_completed; } } @@ -3630,9 +3642,35 @@ complete_cmd_fusion(struct megasas_instance *instance, u32 MSIxIndex, instance->reply_post_host_index_addr[0]); megasas_check_and_restore_queue_depth(instance); } + + if (irq_context) + atomic_dec(&irq_context->in_used); + return num_completed; } +int megasas_blk_mq_poll(struct Scsi_Host *shost, unsigned int queue_num) +{ + + struct megasas_instance *instance; + int num_entries = 0; + struct fusion_context *fusion; + + instance = (struct megasas_instance *)shost->hostdata; + + fusion = instance->ctrl_context; + + queue_num = queue_num + instance->low_latency_index_start; + + if (!atomic_add_unless(&fusion->busy_mq_poll[queue_num], 1, 1)) + return 0; + + num_entries = complete_cmd_fusion(instance, queue_num, NULL); + atomic_dec(&fusion->busy_mq_poll[queue_num]); + + return num_entries; +} + /** * megasas_enable_irq_poll() - enable irqpoll * @instance: Adapter soft state @@ -4163,6 +4201,8 @@ void megasas_reset_reply_desc(struct megasas_instance *instance) fusion = instance->ctrl_context; count = instance->msix_vectors > 0 ? instance->msix_vectors : 1; + count += instance->iopoll_q_count; + for (i = 0 ; i < count ; i++) { fusion->last_reply_idx[i] = 0; reply_desc = fusion->reply_frames_desc[i]; diff --git a/drivers/scsi/megaraid/megaraid_sas_fusion.h b/drivers/scsi/megaraid/megaraid_sas_fusion.h index 30de4b01f703..ce84f811e5e1 100644 --- a/drivers/scsi/megaraid/megaraid_sas_fusion.h +++ b/drivers/scsi/megaraid/megaraid_sas_fusion.h @@ -1303,6 +1303,8 @@ struct fusion_context { u8 *sense; dma_addr_t sense_phys_addr; + atomic_t busy_mq_poll[MAX_MSIX_QUEUES_FUSION]; + dma_addr_t reply_frames_desc_phys[MAX_MSIX_QUEUES_FUSION]; union MPI2_REPLY_DESCRIPTORS_UNION *reply_frames_desc[MAX_MSIX_QUEUES_FUSION]; struct rdpq_alloc_detail rdpq_tracker[RDPQ_MAX_CHUNK_COUNT]; From patchwork Mon Feb 15 07:40:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kashyap Desai X-Patchwork-Id: 12088463 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DATE_IN_PAST_06_12, DKIMWL_WL_HIGH,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MIME_HEADER_CTYPE_ONLY,SPF_HELO_NONE,SPF_PASS, T_TVD_MIME_NO_HEADERS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 632D9C433E0 for ; Mon, 15 Feb 2021 15:43:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2FF5264E8D for ; Mon, 15 Feb 2021 15:43:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230414AbhBOPnk (ORCPT ); Mon, 15 Feb 2021 10:43:40 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41072 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230496AbhBOPnY (ORCPT ); Mon, 15 Feb 2021 10:43:24 -0500 Received: from mail-pj1-x102f.google.com (mail-pj1-x102f.google.com [IPv6:2607:f8b0:4864:20::102f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DEC6AC061794 for ; Mon, 15 Feb 2021 07:40:31 -0800 (PST) Received: by mail-pj1-x102f.google.com with SMTP id l18so4132954pji.3 for ; Mon, 15 Feb 2021 07:40:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=GdEPuflmCaUwNfeT5D/naLgr9JqBfyr7jIYdwFAwq0s=; b=AQVzsxig1h59R7AFHPBPe9EqrAmdKOIjiEHwVhyQ2QSiKV5ONhOY98R0m4GuyAKFnh 50GJGNhCmuI7cUfpiMWugbIOf1r1OIFY5MEVfadI1Jk7sN3lS4Zp4w7SN9wqE1tu5ixb S5fdZZCFUEkeuEpyx3r396Ij2oaPoGHFqxd30= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=GdEPuflmCaUwNfeT5D/naLgr9JqBfyr7jIYdwFAwq0s=; b=ZcxA3Bk15l+2MGl9nfjkmCpEi7e1jv68ehpcHhnjcGOwqrIHj6M97y2vPQEJH/14Ss qNsvP5ITsGs1+lwhWDqnBRXeqSHcDmq/PchvSQw7ktS3SUUCQKuLyegQxsF/ZCv1Cfvu 5dkjjhE7HqJNX9pNOcyAU886ZcCZC2Jpg1Qwv4j2SQwIGMUsFK/O0KSJEMG7t/eQnsfI 2vGqRbbTt0QxcOHIOCIpho5tKr0NNpRgMdp2uk3u66+AtuSWhUHdTXEK1ijzT0zTJ6IL s73brRqGud55yisZAjhcffUOZnHp/+5ys7pxxQM/z8JkoF1MejcQdi6Y7ZyvY7rncy3A b3/w== X-Gm-Message-State: AOAM532IP6ml1khrI3nhQE+WQ2RdJOD+N1RVvs0MEL0Qybr08BghhLYF Ot8xofBN66NcMwuXBs1QLOs+WYneC83luFw8vNRHE+JtjanRyeIuZct0Rli/lrvyzGuNbtdwv4e 8EhovpdrteyQWZJuAyMi0Xn/FDqrbhmQecxWE8Yjm0ti6W0jw3cRK8UVW/QdF53APMITrELw3ci SO4UltKNBI X-Google-Smtp-Source: ABdhPJyjanhJfPLpz0gydaRnFwqFA7jxpasPnAvt4Woj44ITRya3pJJNq1N6LLuXoU8o1e/5PrDabQ== X-Received: by 2002:a17:90a:6947:: with SMTP id j7mr16595788pjm.151.1613403631047; Mon, 15 Feb 2021 07:40:31 -0800 (PST) Received: from drv-bst-rhel8.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id m4sm18031341pgu.4.2021.02.15.07.40.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Feb 2021 07:40:30 -0800 (PST) From: Kashyap Desai To: linux-scsi@vger.kernel.org Cc: Kashyap Desai , dgilbert@interlog.com, linux-block@vger.kernel.org Subject: [PATCH v4 3/5] scsi_debug : iouring iopoll support Date: Mon, 15 Feb 2021 13:10:46 +0530 Message-Id: <20210215074048.19424-4-kashyap.desai@broadcom.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20210215074048.19424-1-kashyap.desai@broadcom.com> References: <20210215074048.19424-1-kashyap.desai@broadcom.com> Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org Add support of iouring iopoll interface in scsi_debug. This feature requires shared hosttag support in kernel and driver. Signed-off-by: Kashyap Desai Acked-by: Douglas Gilbert Tested-by: Douglas Gilbert Reviewed-by: Hannes Reinecke Cc: dgilbert@interlog.com Cc: linux-block@vger.kernel.org --- drivers/scsi/scsi_debug.c | 130 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 130 insertions(+) diff --git a/drivers/scsi/scsi_debug.c b/drivers/scsi/scsi_debug.c index d1b0cbe1b5f1..746eec521f79 100644 --- a/drivers/scsi/scsi_debug.c +++ b/drivers/scsi/scsi_debug.c @@ -829,6 +829,7 @@ static int sdeb_zbc_max_open = DEF_ZBC_MAX_OPEN_ZONES; static int sdeb_zbc_nr_conv = DEF_ZBC_NR_CONV_ZONES; static int submit_queues = DEF_SUBMIT_QUEUES; /* > 1 for multi-queue (mq) */ +static int poll_queues; /* iouring iopoll interface.*/ static struct sdebug_queue *sdebug_q_arr; /* ptr to array of submit queues */ static DEFINE_RWLOCK(atomic_rw); @@ -5432,6 +5433,14 @@ static int schedule_resp(struct scsi_cmnd *cmnd, struct sdebug_dev_info *devip, cmnd->host_scribble = (unsigned char *)sqcp; sd_dp = sqcp->sd_dp; spin_unlock_irqrestore(&sqp->qc_lock, iflags); + + /* Do not complete IO from default completion path. + * Let it to be on queue. + * Completion should happen from mq_poll interface. + */ + if ((sqp - sdebug_q_arr) >= (submit_queues - poll_queues)) + return 0; + if (!sd_dp) { sd_dp = kzalloc(sizeof(*sd_dp), GFP_ATOMIC); if (!sd_dp) { @@ -5615,6 +5624,7 @@ module_param_named(sector_size, sdebug_sector_size, int, S_IRUGO); module_param_named(statistics, sdebug_statistics, bool, S_IRUGO | S_IWUSR); module_param_named(strict, sdebug_strict, bool, S_IRUGO | S_IWUSR); module_param_named(submit_queues, submit_queues, int, S_IRUGO); +module_param_named(poll_queues, poll_queues, int, S_IRUGO); module_param_named(tur_ms_to_ready, sdeb_tur_ms_to_ready, int, S_IRUGO); module_param_named(unmap_alignment, sdebug_unmap_alignment, int, S_IRUGO); module_param_named(unmap_granularity, sdebug_unmap_granularity, int, S_IRUGO); @@ -5677,6 +5687,7 @@ MODULE_PARM_DESC(opt_xferlen_exp, "optimal transfer length granularity exponent MODULE_PARM_DESC(opts, "1->noise, 2->medium_err, 4->timeout, 8->recovered_err... (def=0)"); MODULE_PARM_DESC(per_host_store, "If set, next positive add_host will get new store (def=0)"); MODULE_PARM_DESC(physblk_exp, "physical block exponent (def=0)"); +MODULE_PARM_DESC(poll_queues, "support for iouring iopoll queues (1 to max(submit_queues - 1)"); MODULE_PARM_DESC(ptype, "SCSI peripheral type(def=0[disk])"); MODULE_PARM_DESC(random, "If set, uniformly randomize command duration between 0 and delay_in_ns"); MODULE_PARM_DESC(removable, "claim to have removable media (def=0)"); @@ -7201,6 +7212,104 @@ static int resp_not_ready(struct scsi_cmnd *scp, struct sdebug_dev_info *devip) return check_condition_result; } +static int sdebug_map_queues(struct Scsi_Host *shost) +{ + int i, qoff; + + if (shost->nr_hw_queues == 1) + return 0; + + for (i = 0, qoff = 0; i < HCTX_MAX_TYPES; i++) { + struct blk_mq_queue_map *map = &shost->tag_set.map[i]; + + map->nr_queues = 0; + + if (i == HCTX_TYPE_DEFAULT) + map->nr_queues = submit_queues - poll_queues; + else if (i == HCTX_TYPE_POLL) + map->nr_queues = poll_queues; + + if (!map->nr_queues) { + BUG_ON(i == HCTX_TYPE_DEFAULT); + continue; + } + + map->queue_offset = qoff; + blk_mq_map_queues(map); + + qoff += map->nr_queues; + } + + return 0; + +} + +static int sdebug_blk_mq_poll(struct Scsi_Host *shost, unsigned int queue_num) +{ + int qc_idx; + int retiring = 0; + unsigned long iflags; + struct sdebug_queue *sqp; + struct sdebug_queued_cmd *sqcp; + struct scsi_cmnd *scp; + struct sdebug_dev_info *devip; + int num_entries = 0; + + sqp = sdebug_q_arr + queue_num; + + do { + spin_lock_irqsave(&sqp->qc_lock, iflags); + qc_idx = find_first_bit(sqp->in_use_bm, sdebug_max_queue); + if (unlikely((qc_idx < 0) || (qc_idx >= sdebug_max_queue))) + goto out; + + sqcp = &sqp->qc_arr[qc_idx]; + scp = sqcp->a_cmnd; + if (unlikely(scp == NULL)) { + pr_err("scp is NULL, queue_num=%d, qc_idx=%d from %s\n", + queue_num, qc_idx, __func__); + goto out; + } + devip = (struct sdebug_dev_info *)scp->device->hostdata; + if (likely(devip)) + atomic_dec(&devip->num_in_q); + else + pr_err("devip=NULL from %s\n", __func__); + if (unlikely(atomic_read(&retired_max_queue) > 0)) + retiring = 1; + + sqcp->a_cmnd = NULL; + if (unlikely(!test_and_clear_bit(qc_idx, sqp->in_use_bm))) { + pr_err("Unexpected completion sqp %p queue_num=%d qc_idx=%d from %s\n", + sqp, queue_num, qc_idx, __func__); + goto out; + } + + if (unlikely(retiring)) { /* user has reduced max_queue */ + int k, retval; + + retval = atomic_read(&retired_max_queue); + if (qc_idx >= retval) { + pr_err("index %d too large\n", retval); + goto out; + } + k = find_last_bit(sqp->in_use_bm, retval); + if ((k < sdebug_max_queue) || (k == retval)) + atomic_set(&retired_max_queue, 0); + else + atomic_set(&retired_max_queue, k + 1); + } + spin_unlock_irqrestore(&sqp->qc_lock, iflags); + scp->scsi_done(scp); /* callback to mid level */ + num_entries++; + } while (1); + +out: + spin_unlock_irqrestore(&sqp->qc_lock, iflags); + return num_entries; +} + + static int scsi_debug_queuecommand(struct Scsi_Host *shost, struct scsi_cmnd *scp) { @@ -7380,6 +7489,8 @@ static struct scsi_host_template sdebug_driver_template = { .ioctl = scsi_debug_ioctl, .queuecommand = scsi_debug_queuecommand, .change_queue_depth = sdebug_change_qdepth, + .map_queues = sdebug_map_queues, + .mq_poll = sdebug_blk_mq_poll, .eh_abort_handler = scsi_debug_abort, .eh_device_reset_handler = scsi_debug_device_reset, .eh_target_reset_handler = scsi_debug_target_reset, @@ -7427,6 +7538,25 @@ static int sdebug_driver_probe(struct device *dev) if (sdebug_host_max_queue) hpnt->host_tagset = 1; + /* poll queues are possible for nr_hw_queues > 1 */ + if (hpnt->nr_hw_queues == 1 || (poll_queues < 1)) { + pr_warn("%s: trim poll_queues to 0. poll_q/nr_hw = (%d/%d)\n", + my_name, poll_queues, hpnt->nr_hw_queues); + poll_queues = 0; + } + + /* + * Poll queues don't need interrupts, but we need at least one I/O queue + * left over for non-polled I/O. + * If condition not met, trim poll_queues to 1 (just for simplicity). + */ + if (poll_queues >= submit_queues) { + pr_warn("%s: trim poll_queues to 1\n", my_name); + poll_queues = 1; + } + if (poll_queues) + hpnt->nr_maps = 3; + sdbg_host->shost = hpnt; *((struct sdebug_host_info **)hpnt->hostdata) = sdbg_host; if ((hpnt->this_id >= 0) && (sdebug_num_tgts > hpnt->this_id)) From patchwork Mon Feb 15 07:40:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kashyap Desai X-Patchwork-Id: 12088465 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DATE_IN_PAST_06_12, DKIMWL_WL_HIGH,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MIME_HEADER_CTYPE_ONLY,SPF_HELO_NONE,SPF_PASS, T_TVD_MIME_NO_HEADERS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 550CCC433E6 for ; Mon, 15 Feb 2021 15:44:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2603564E8E for ; Mon, 15 Feb 2021 15:44:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230520AbhBOPnt (ORCPT ); Mon, 15 Feb 2021 10:43:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41074 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231476AbhBOPnY (ORCPT ); Mon, 15 Feb 2021 10:43:24 -0500 Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C5601C061797 for ; Mon, 15 Feb 2021 07:40:33 -0800 (PST) Received: by mail-pj1-x102e.google.com with SMTP id cv23so3933552pjb.5 for ; Mon, 15 Feb 2021 07:40:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=TNweTJAKjAFpD6e4hnHJJDvDmzaZBh3DS1A4WdeyZEo=; b=OcVVaOnudTUBWGg1yTtq0/CbYIRDdcu0N/akscFOsI5Xi/MEs4P17RD7ua5OXZNU1t 8Wh6PaoRQSo4K2qAswA/oyz4bdlhy5yfeCRwj4/+UbkwCoCAibkDZ+bqyhif1YZZeBrH 132sTACNLxdw5au4jgbHpFL2aQjozLbVHdw8o= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=TNweTJAKjAFpD6e4hnHJJDvDmzaZBh3DS1A4WdeyZEo=; b=h8XIJ7CaUZWvdWQ4gpzW90vxbKpPWdLuYfIKi1Q5XMx+o5kRbSm0qJm9bqeRjSAXXu VJt5ID8XGDvObL5G2S/uHSH4D5d6fYjAoiljOH1XKMIusMmO3+dig0/f3Tc+fR70JU+9 iqQj6BJ+bYLNnkZzP0sL87cKJJ2PrHlJm8hi6zKfPOijUE60ITzkCr6lp/USPlg2ep2B SHYMakrKN1JgDjOlYjejNTVo/vDLy4DVnJsj9PhfQM2TsFmt8/+8ZowNdlxgZ/gxbgx5 IX6bMPd0/ar8XOoAfTwgPyg+2Q+B++yFSBUsmNgZm4WakNgUqC6Cg5Rp5AEqz040XEUY rm3w== X-Gm-Message-State: AOAM532E7+VVBzk6/YCIoasiowccmbtTNbRHIRbKPQ/JAbTl5PZgCTct EQl0SBEKgQWCZHB4Y0nejtmOb/ApnZrv7y66U+JGcclFPR166quTO09ZM7opNB+1dCloDN25U2O D6/3+ronGpTgAflFlXkMTAI7YAPFdK6jfsn7ZTkXuA7f0U62dTPJGW//jfvBPkHkLarXo8u5DFF WpbPiMwx23 X-Google-Smtp-Source: ABdhPJwAr3DAUiDoYyKCq/qWDNuI04dGiqwnusxvSp1pw2yp/yXyVBt0JKIKShppzm65g/IqryODvg== X-Received: by 2002:a17:90a:ac14:: with SMTP id o20mr11994661pjq.171.1613403632815; Mon, 15 Feb 2021 07:40:32 -0800 (PST) Received: from drv-bst-rhel8.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id m4sm18031341pgu.4.2021.02.15.07.40.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Feb 2021 07:40:32 -0800 (PST) From: Kashyap Desai To: linux-scsi@vger.kernel.org Cc: Kashyap Desai , Douglas Gilbert Subject: [PATCH v4 4/5] scsi_debug: add new defer type for mq poll Date: Mon, 15 Feb 2021 13:10:47 +0530 Message-Id: <20210215074048.19424-5-kashyap.desai@broadcom.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20210215074048.19424-1-kashyap.desai@broadcom.com> References: <20210215074048.19424-1-kashyap.desai@broadcom.com> Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org From: Douglas Gilbert Add a new sdeb_defer_type enumeration: SDEB_DEFER_POLL for requests that have REQ_HIPRI set in cmd_flags field. It is expected that these requests will be polled via the mq_poll entry point which is driven by calls to blk_poll() in the block layer. Therefore timer events are not 'wired up' in the normal fashion. There are still cases with short delays (e.g. < 10 microseconds) where by the time the command response processing occurs, the delay is already exceeded in which case the code calls scsi_done() directly. In such cases there is no window for mq_poll() to be called. Add 'mq_polls' counter that increments on each scsi_done() called via the mq_poll entry point. Can be used to show (with 'cat /proc/scsi/scsi_debug/') that blk_poll() is causing completions rather than some other mechanism. This patch is improvement over previous patch "scsi_debug: iouring iopoll support" Changes since version 3 - Fix IO hang issue. Do not return from schedule_resp. Use new defer type for mq poll to queue the REQ_HIPRI IOs. Changes since version 2 [sent 20210206 to linux-scsi list] - the sdebug_blk_mq_poll() callback didn't cope with the uncommon case where sqcp->sd_dp is NULL. Fix. Changes since version 1 [sent 20210201 to linux-scsi list] - harden SDEB_DEFER_POLL which broke under testing - add mq_polls counter for debug output Signed-off-by: Douglas Gilbert Tested-by: Kashyap Desai --- drivers/scsi/scsi_debug.c | 148 ++++++++++++++++++++++++-------------- 1 file changed, 94 insertions(+), 54 deletions(-) diff --git a/drivers/scsi/scsi_debug.c b/drivers/scsi/scsi_debug.c index 746eec521f79..c50de49a2c2f 100644 --- a/drivers/scsi/scsi_debug.c +++ b/drivers/scsi/scsi_debug.c @@ -322,17 +322,19 @@ struct sdeb_store_info { container_of(d, struct sdebug_host_info, dev) enum sdeb_defer_type {SDEB_DEFER_NONE = 0, SDEB_DEFER_HRT = 1, - SDEB_DEFER_WQ = 2}; + SDEB_DEFER_WQ = 2, SDEB_DEFER_POLL = 3}; struct sdebug_defer { struct hrtimer hrt; struct execute_work ew; + ktime_t cmpl_ts;/* time since boot to complete this cmd */ int sqa_idx; /* index of sdebug_queue array */ int qc_idx; /* index of sdebug_queued_cmd array within sqa_idx */ int hc_idx; /* hostwide tag index */ int issuing_cpu; bool init_hrt; bool init_wq; + bool init_poll; bool aborted; /* true when blk_abort_request() already called */ enum sdeb_defer_type defer_t; }; @@ -357,6 +359,7 @@ static atomic_t sdebug_completions; /* count of deferred completions */ static atomic_t sdebug_miss_cpus; /* submission + completion cpus differ */ static atomic_t sdebug_a_tsf; /* 'almost task set full' counter */ static atomic_t sdeb_inject_pending; +static atomic_t sdeb_mq_poll_count; /* bumped when mq_poll returns > 0 */ struct opcode_info_t { u8 num_attached; /* 0 if this is it (i.e. a leaf); use 0xff */ @@ -4730,7 +4733,6 @@ static void sdebug_q_cmd_complete(struct sdebug_defer *sd_dp) struct scsi_cmnd *scp; struct sdebug_dev_info *devip; - sd_dp->defer_t = SDEB_DEFER_NONE; if (unlikely(aborted)) sd_dp->aborted = false; qc_idx = sd_dp->qc_idx; @@ -4745,6 +4747,7 @@ static void sdebug_q_cmd_complete(struct sdebug_defer *sd_dp) return; } spin_lock_irqsave(&sqp->qc_lock, iflags); + sd_dp->defer_t = SDEB_DEFER_NONE; sqcp = &sqp->qc_arr[qc_idx]; scp = sqcp->a_cmnd; if (unlikely(scp == NULL)) { @@ -5434,13 +5437,6 @@ static int schedule_resp(struct scsi_cmnd *cmnd, struct sdebug_dev_info *devip, sd_dp = sqcp->sd_dp; spin_unlock_irqrestore(&sqp->qc_lock, iflags); - /* Do not complete IO from default completion path. - * Let it to be on queue. - * Completion should happen from mq_poll interface. - */ - if ((sqp - sdebug_q_arr) >= (submit_queues - poll_queues)) - return 0; - if (!sd_dp) { sd_dp = kzalloc(sizeof(*sd_dp), GFP_ATOMIC); if (!sd_dp) { @@ -5517,40 +5513,66 @@ static int schedule_resp(struct scsi_cmnd *cmnd, struct sdebug_dev_info *devip, kt -= d; } } - if (!sd_dp->init_hrt) { - sd_dp->init_hrt = true; - sqcp->sd_dp = sd_dp; - hrtimer_init(&sd_dp->hrt, CLOCK_MONOTONIC, - HRTIMER_MODE_REL_PINNED); - sd_dp->hrt.function = sdebug_q_cmd_hrt_complete; - sd_dp->sqa_idx = sqp - sdebug_q_arr; - sd_dp->qc_idx = k; + sd_dp->cmpl_ts = ktime_add(ns_to_ktime(ns_from_boot), kt); + if (cmnd->request->cmd_flags & REQ_HIPRI) { + spin_lock_irqsave(&sqp->qc_lock, iflags); + if (!sd_dp->init_poll) { + sd_dp->init_poll = true; + sqcp->sd_dp = sd_dp; + sd_dp->sqa_idx = sqp - sdebug_q_arr; + sd_dp->qc_idx = k; + } + sd_dp->defer_t = SDEB_DEFER_POLL; + spin_unlock_irqrestore(&sqp->qc_lock, iflags); + } else { + if (!sd_dp->init_hrt) { + sd_dp->init_hrt = true; + sqcp->sd_dp = sd_dp; + hrtimer_init(&sd_dp->hrt, CLOCK_MONOTONIC, + HRTIMER_MODE_REL_PINNED); + sd_dp->hrt.function = sdebug_q_cmd_hrt_complete; + sd_dp->sqa_idx = sqp - sdebug_q_arr; + sd_dp->qc_idx = k; + } + sd_dp->defer_t = SDEB_DEFER_HRT; + /* schedule the invocation of scsi_done() for a later time */ + hrtimer_start(&sd_dp->hrt, kt, HRTIMER_MODE_REL_PINNED); } if (sdebug_statistics) sd_dp->issuing_cpu = raw_smp_processor_id(); - sd_dp->defer_t = SDEB_DEFER_HRT; - /* schedule the invocation of scsi_done() for a later time */ - hrtimer_start(&sd_dp->hrt, kt, HRTIMER_MODE_REL_PINNED); } else { /* jdelay < 0, use work queue */ - if (!sd_dp->init_wq) { - sd_dp->init_wq = true; - sqcp->sd_dp = sd_dp; - sd_dp->sqa_idx = sqp - sdebug_q_arr; - sd_dp->qc_idx = k; - INIT_WORK(&sd_dp->ew.work, sdebug_q_cmd_wq_complete); - } - if (sdebug_statistics) - sd_dp->issuing_cpu = raw_smp_processor_id(); - sd_dp->defer_t = SDEB_DEFER_WQ; if (unlikely((sdebug_opts & SDEBUG_OPT_CMD_ABORT) && atomic_read(&sdeb_inject_pending))) sd_dp->aborted = true; - schedule_work(&sd_dp->ew.work); - if (unlikely((sdebug_opts & SDEBUG_OPT_CMD_ABORT) && - atomic_read(&sdeb_inject_pending))) { + sd_dp->cmpl_ts = ns_to_ktime(ns_from_boot); + if (cmnd->request->cmd_flags & REQ_HIPRI) { + spin_lock_irqsave(&sqp->qc_lock, iflags); + if (!sd_dp->init_poll) { + sd_dp->init_poll = true; + sqcp->sd_dp = sd_dp; + sd_dp->sqa_idx = sqp - sdebug_q_arr; + sd_dp->qc_idx = k; + } + sd_dp->defer_t = SDEB_DEFER_POLL; + spin_unlock_irqrestore(&sqp->qc_lock, iflags); + } else { + if (!sd_dp->init_wq) { + sd_dp->init_wq = true; + sqcp->sd_dp = sd_dp; + sd_dp->sqa_idx = sqp - sdebug_q_arr; + sd_dp->qc_idx = k; + INIT_WORK(&sd_dp->ew.work, sdebug_q_cmd_wq_complete); + } + sd_dp->defer_t = SDEB_DEFER_WQ; + schedule_work(&sd_dp->ew.work); + } + if (sdebug_statistics) + sd_dp->issuing_cpu = raw_smp_processor_id(); + if (unlikely(sd_dp->aborted)) { sdev_printk(KERN_INFO, sdp, "abort request tag %d\n", cmnd->request->tag); blk_abort_request(cmnd->request); atomic_set(&sdeb_inject_pending, 0); + sd_dp->aborted = false; } } if (unlikely((SDEBUG_OPT_Q_NOISE & sdebug_opts) && scsi_result == device_qfull_result)) @@ -5779,11 +5801,12 @@ static int scsi_debug_show_info(struct seq_file *m, struct Scsi_Host *host) dix_reads, dix_writes, dif_errors); seq_printf(m, "usec_in_jiffy=%lu, statistics=%d\n", TICK_NSEC / 1000, sdebug_statistics); - seq_printf(m, "cmnd_count=%d, completions=%d, %s=%d, a_tsf=%d\n", + seq_printf(m, "cmnd_count=%d, completions=%d, %s=%d, a_tsf=%d, mq_polls=%d\n", atomic_read(&sdebug_cmnd_count), atomic_read(&sdebug_completions), "miss_cpus", atomic_read(&sdebug_miss_cpus), - atomic_read(&sdebug_a_tsf)); + atomic_read(&sdebug_a_tsf), + atomic_read(&sdeb_mq_poll_count)); seq_printf(m, "submit_queues=%d\n", submit_queues); for (j = 0, sqp = sdebug_q_arr; j < submit_queues; ++j, ++sqp) { @@ -7246,52 +7269,68 @@ static int sdebug_map_queues(struct Scsi_Host *shost) static int sdebug_blk_mq_poll(struct Scsi_Host *shost, unsigned int queue_num) { - int qc_idx; - int retiring = 0; + bool first; + bool retiring = false; + int num_entries = 0; + unsigned int qc_idx = 0; unsigned long iflags; + ktime_t kt_from_boot = ktime_get_boottime(); struct sdebug_queue *sqp; struct sdebug_queued_cmd *sqcp; struct scsi_cmnd *scp; struct sdebug_dev_info *devip; - int num_entries = 0; + struct sdebug_defer *sd_dp; sqp = sdebug_q_arr + queue_num; + spin_lock_irqsave(&sqp->qc_lock, iflags); - do { - spin_lock_irqsave(&sqp->qc_lock, iflags); - qc_idx = find_first_bit(sqp->in_use_bm, sdebug_max_queue); - if (unlikely((qc_idx < 0) || (qc_idx >= sdebug_max_queue))) - goto out; + for (first = true; first || qc_idx + 1 < sdebug_max_queue; ) { + if (first) { + qc_idx = find_first_bit(sqp->in_use_bm, sdebug_max_queue); + first = false; + } else { + qc_idx = find_next_bit(sqp->in_use_bm, sdebug_max_queue, qc_idx + 1); + } + if (unlikely(qc_idx >= sdebug_max_queue)) + break; sqcp = &sqp->qc_arr[qc_idx]; + sd_dp = sqcp->sd_dp; + if (unlikely(!sd_dp)) + continue; scp = sqcp->a_cmnd; if (unlikely(scp == NULL)) { - pr_err("scp is NULL, queue_num=%d, qc_idx=%d from %s\n", + pr_err("scp is NULL, queue_num=%d, qc_idx=%u from %s\n", queue_num, qc_idx, __func__); - goto out; + break; } + if (sd_dp->defer_t == SDEB_DEFER_POLL) { + if (kt_from_boot < sd_dp->cmpl_ts) + continue; + + } else /* ignoring non REQ_HIPRI requests */ + continue; devip = (struct sdebug_dev_info *)scp->device->hostdata; if (likely(devip)) atomic_dec(&devip->num_in_q); else pr_err("devip=NULL from %s\n", __func__); if (unlikely(atomic_read(&retired_max_queue) > 0)) - retiring = 1; + retiring = true; sqcp->a_cmnd = NULL; if (unlikely(!test_and_clear_bit(qc_idx, sqp->in_use_bm))) { - pr_err("Unexpected completion sqp %p queue_num=%d qc_idx=%d from %s\n", + pr_err("Unexpected completion sqp %p queue_num=%d qc_idx=%u from %s\n", sqp, queue_num, qc_idx, __func__); - goto out; + break; } - if (unlikely(retiring)) { /* user has reduced max_queue */ int k, retval; retval = atomic_read(&retired_max_queue); if (qc_idx >= retval) { pr_err("index %d too large\n", retval); - goto out; + break; } k = find_last_bit(sqp->in_use_bm, retval); if ((k < sdebug_max_queue) || (k == retval)) @@ -7299,17 +7338,18 @@ static int sdebug_blk_mq_poll(struct Scsi_Host *shost, unsigned int queue_num) else atomic_set(&retired_max_queue, k + 1); } + sd_dp->defer_t = SDEB_DEFER_NONE; spin_unlock_irqrestore(&sqp->qc_lock, iflags); scp->scsi_done(scp); /* callback to mid level */ + spin_lock_irqsave(&sqp->qc_lock, iflags); num_entries++; - } while (1); - -out: + } spin_unlock_irqrestore(&sqp->qc_lock, iflags); + if (num_entries > 0) + atomic_add(num_entries, &sdeb_mq_poll_count); return num_entries; } - static int scsi_debug_queuecommand(struct Scsi_Host *shost, struct scsi_cmnd *scp) { From patchwork Mon Feb 15 07:40:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kashyap Desai X-Patchwork-Id: 12088467 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DATE_IN_PAST_06_12, DKIMWL_WL_HIGH,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MIME_HEADER_CTYPE_ONLY,SPF_HELO_NONE,SPF_PASS, T_TVD_MIME_NO_HEADERS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 08FC6C433E0 for ; Mon, 15 Feb 2021 15:44:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D1A3664E91 for ; Mon, 15 Feb 2021 15:44:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231285AbhBOPoC (ORCPT ); Mon, 15 Feb 2021 10:44:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41132 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231422AbhBOPnZ (ORCPT ); Mon, 15 Feb 2021 10:43:25 -0500 Received: from mail-pj1-x102f.google.com (mail-pj1-x102f.google.com [IPv6:2607:f8b0:4864:20::102f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 488CFC0617A7 for ; Mon, 15 Feb 2021 07:40:35 -0800 (PST) Received: by mail-pj1-x102f.google.com with SMTP id e9so4146418pjj.0 for ; Mon, 15 Feb 2021 07:40:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ghAL0+h3u1cX9VyWJd2C9M5FvR34M42h2tqveRBuqus=; b=M4S4achrnOe/5b9+HVprlhmTffhmiWsOUFewAcv+hIpD+WYa1b2AHcCQM+7BI5+uhe W+8B0eXV0Wvm3i+mW+m5diTJPI+hgO3jvhq/lNtRUxlHQ+fn9qGK+pNmHPnO/biBRkxr ymiHtGvB9DHV8wtFyLfoJnjJzRxLcCM3epji8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ghAL0+h3u1cX9VyWJd2C9M5FvR34M42h2tqveRBuqus=; b=FZovrbp0xjt3CF1dvoQYPfTrMVTr5Aviq8HOEzlufFts+t1emCgoxcpFcx22iqeCRC zarViCHwcvCnpVcq7x/hof1eizQp5h7X/W+VpIb1kl2B8vZLnMh/aQz5tmeKz4Q4nk0W pq837DhgPUpX7gHxKoKDAAa9Zeo3adgpY+AvowJIFlUd1XuEfmqhLV5cyG12DhetlFeN wEDtM3iryMxxC+gq7Ynvt+VcoUEhrqFSn3jJ0W3TnakugS/8YeNVk/oocIfZc93mQgjh UnU9I3NTPB9wtFMTUIctdIO+ZlNddg3+nZljevmNQsG4FSyhfoSjWtcDmwnIF5BIkUW9 upyQ== X-Gm-Message-State: AOAM531t4SHBOAJj7x82MgFEwoZXLVOFAxvkf6H5VeNeEC67GW1pXARZ Cu5SCGbYiHX1ZdRC+1GtlsQe6GyfCfqUgJEjGon+SCluWA5Tj1RHlZWgWwy7oimD7iD94tRE0u0 J2YM1qOWUuXCU2U/LhIkZlJMtoJ2LWn7qUFpfgp2SeM77Z0pPiSxBmpdPdGmegLJl7cORSrPwLT RUnLO0XZL5 X-Google-Smtp-Source: ABdhPJz4z+vS2l8D/WxfuaWIKgXTCPuM75quW8qVK5TZMY85DO50sU4akhCvlsVZ/uDW/16zaw6XjQ== X-Received: by 2002:a17:90a:517:: with SMTP id h23mr16827138pjh.108.1613403634434; Mon, 15 Feb 2021 07:40:34 -0800 (PST) Received: from drv-bst-rhel8.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id m4sm18031341pgu.4.2021.02.15.07.40.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Feb 2021 07:40:34 -0800 (PST) From: Kashyap Desai To: linux-scsi@vger.kernel.org Cc: Kashyap Desai Subject: [PATCH v4 5/5] scsi: set shost as hctx driver_data Date: Mon, 15 Feb 2021 13:10:48 +0530 Message-Id: <20210215074048.19424-6-kashyap.desai@broadcom.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20210215074048.19424-1-kashyap.desai@broadcom.com> References: <20210215074048.19424-1-kashyap.desai@broadcom.com> Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org hctx->driver_data is not set for SCSI currently. Separately set hctx->driver_data = shost. Suggested-by: John Garry Signed-off-by: Kashyap Desai Reviewed-by: John Garry Reviewed-by: Hannes Reinecke --- drivers/scsi/scsi_lib.c | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c index 8c29bf0e4cfd..f661c50f3b88 100644 --- a/drivers/scsi/scsi_lib.c +++ b/drivers/scsi/scsi_lib.c @@ -1792,9 +1792,7 @@ static void scsi_mq_exit_request(struct blk_mq_tag_set *set, struct request *rq, static int scsi_mq_poll(struct blk_mq_hw_ctx *hctx) { - struct request_queue *q = hctx->queue; - struct scsi_device *sdev = q->queuedata; - struct Scsi_Host *shost = sdev->host; + struct Scsi_Host *shost = hctx->driver_data; if (shost->hostt->mq_poll) return shost->hostt->mq_poll(shost, hctx->queue_num); @@ -1802,6 +1800,15 @@ static int scsi_mq_poll(struct blk_mq_hw_ctx *hctx) return 0; } +static int scsi_init_hctx(struct blk_mq_hw_ctx *hctx, void *data, + unsigned int hctx_idx) +{ + struct Scsi_Host *shost = data; + + hctx->driver_data = shost; + return 0; +} + static int scsi_map_queues(struct blk_mq_tag_set *set) { struct Scsi_Host *shost = container_of(set, struct Scsi_Host, tag_set); @@ -1869,15 +1876,14 @@ static const struct blk_mq_ops scsi_mq_ops_no_commit = { .cleanup_rq = scsi_cleanup_rq, .busy = scsi_mq_lld_busy, .map_queues = scsi_map_queues, + .init_hctx = scsi_init_hctx, .poll = scsi_mq_poll, }; static void scsi_commit_rqs(struct blk_mq_hw_ctx *hctx) { - struct request_queue *q = hctx->queue; - struct scsi_device *sdev = q->queuedata; - struct Scsi_Host *shost = sdev->host; + struct Scsi_Host *shost = hctx->driver_data; shost->hostt->commit_rqs(shost, hctx->queue_num); } @@ -1898,6 +1904,7 @@ static const struct blk_mq_ops scsi_mq_ops = { .cleanup_rq = scsi_cleanup_rq, .busy = scsi_mq_lld_busy, .map_queues = scsi_map_queues, + .init_hctx = scsi_init_hctx, .poll = scsi_mq_poll, };