From patchwork Fri Jan 25 18:34:50 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 10781893 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8B91C14E5 for ; Fri, 25 Jan 2019 18:35:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7A5B0299B5 for ; Fri, 25 Jan 2019 18:35:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6EA902D40C; Fri, 25 Jan 2019 18:35:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C506D299B5 for ; Fri, 25 Jan 2019 18:35:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728974AbfAYSft (ORCPT ); Fri, 25 Jan 2019 13:35:49 -0500 Received: from out002.mailprotect.be ([83.217.72.86]:38615 "EHLO out002.mailprotect.be" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726689AbfAYSft (ORCPT ); Fri, 25 Jan 2019 13:35:49 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mailprotect.be; s=mail; h=Content-Transfer-Encoding:MIME-Version:References :In-Reply-To:Message-Id:Date:Subject:Cc:To:From:reply-to:sender:bcc: content-type; bh=C2DofYuZePEuHZfcAD7TOy8xPBfdFgPlEtC6zojvl7g=; b=DvT9VkPxp7CF fwMJwxqXW1phSLzdlZY1g7MpAUfhGsm2wdLRvLnpWLWmW9ffX/eV0IaBxwlcllYrm+HB7Fs8s6TgP Rq15KtW5FR8BnrGM6ZW3I44whSL8EXJcJteOvvPIPd0DsHpxiYKDACHZPZgx8pDM/1hOIh6iKesXj SEP8Kqtt8nbjg4EDbImtDbeCBsy6YpvDOWX4+JPuMtIcatxM94Z3StSjd7Huz9aedVcsGO8ha6tV9 iB+Z5GIlJE9KOXs53iMyuOCE6qThe3ctdH7umdvP6p5h/1mXCE2b+yy6DvmiJCN7kO1ecMERe+dm+ s0rmHkzJmW8O3JvquViMJQ==; Received: from smtp-auth.mailprotect.be ([178.208.39.155]) by com-mpt-out002.mailprotect.be with esmtp (Exim 4.89) (envelope-from ) id 1gn6K7-00041F-5b; Fri, 25 Jan 2019 19:35:27 +0100 Received: from desktop-bart.svl.corp.google.com (unknown [104.133.8.89]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp-auth.mailprotect.be (Postfix) with ESMTPSA id 154D3C05E8; Fri, 25 Jan 2019 19:35:22 +0100 (CET) From: Bart Van Assche To: "Martin K . Petersen" Cc: Christoph Hellwig , target-devel@vger.kernel.org, Bart Van Assche , Doug Ledford , Jason Gunthorpe , Nicholas Bellinger , Mike Christie , Hannes Reinecke Subject: [PATCH 08/16] RDMA/srpt: Rework I/O context allocation Date: Fri, 25 Jan 2019 10:34:50 -0800 Message-Id: <20190125183458.220477-9-bvanassche@acm.org> X-Mailer: git-send-email 2.20.1.495.gaa96b0ce6b-goog In-Reply-To: <20190125183458.220477-1-bvanassche@acm.org> References: <20190125183458.220477-1-bvanassche@acm.org> MIME-Version: 1.0 X-Originating-IP: 178.208.39.155 X-SpamExperts-Domain: mailprotect.be X-SpamExperts-Username: 178.208.39.128/27 Authentication-Results: mailprotect.be; auth=pass smtp.auth=178.208.39.128/27@mailprotect.be X-SpamExperts-Outgoing-Class: ham X-SpamExperts-Outgoing-Evidence: Combined (0.01) X-Recommended-Action: accept X-Filter-ID: EX5BVjFpneJeBchSMxfU5gFPQmS7yB0UGkfYRuQNzMZ602E9L7XzfQH6nu9C/Fh9KJzpNe6xgvOx q3u0UDjvO1tLifGj39bI0bcPyaJsYTagCTS5H5Pie2sCQ3gGSzr06cM7U7xDGmYwNG93LVnxwxjk yGQYCd08t3lHobjxX/OigDGQOIiVUN5rGO3B5zfV6lLB7UkASj5+Q7Z1jo1wYoGcQZDc/NA+AYRc xIJwNJDswajEtdTx0z97RFosJ+0XX4J8p9+Z4Omt11SsRVhZc5O0+hI4m8XEgLG9kXesazurVxXh H80moATcJz9k4m0jSta6vzX6sWvOVdfPU9Q907I7wX4PVM8uRZ9JrPUGzdHL+nwxyTuzU6NHiyz3 2CjgPash4POzTXhXrXtCoyUxYnL7bykX8cT9ReKnRim+apKC0G2y+Mr2ale9nnzXJ+o1UDSGW782 k6dpc2+siniZEaBBQD2I5QH2BdbXmK3rYa8ppXum6KZe4eMUTLjXn3BvjnQnhcJex8V1sw+C2xqM rAbZykneW87JZ7jdpAlXQVADDgYNo2L2ikikIJmXA+EJKJ9CDjyJqwMkv6BJB2lztCOV7QtxJNU+ igS6jZCzGqpJmuDNJ3/L9Q9wLc4aqLRUvPEObNwbSIpq9S/jgd899z3dKxLhoxcmaInYbR5vlqFZ JlLyK2ygHNTfy41Cttvo7sn//RcLEIwmfxtejjVQPRdR7J54uCh7y/cyXn4gNizEl4GT+izmPxps 5i3VIDAVOJYVhKWF91egvasFoFaoK60jiD6XqsJZtjQxlyCdseyHOMuZZG++RlG2AEFbDFHRYfQQ D0IVT5dlkCOLok5OEpM+qhtGEAXCDCj6buXQqievXHVFggc0157Ysxlq5Z3dsksKeFRmKOHI/40C LjPihwIeoj9zwqNSJz8SHmCZ+bxCLHLD8M1AAidHW08kKeoFJwcvz6VMYEhUT3FsSRfv2/sMSIEg vIcW6VO3wxn7YUe2npEB56Tu+3WZdyLKI7D3A2hEbVRTTjTXJKZ3p5pgpyG7X+t1TW39Ja77LGPp OwAj6IMP28aPMtl2u6Kx+4+Huxzv1d6/j7MDIN1SScQedH2YWYgPyAz1BkOLTUiGJowwNFwKBhAh 7NeKYzwwwYGp+p2okElJG6VbPeUnGhV3sd4rxYOjbs6j0mefY0b4ffxz8Ne0ck6P4ivKdWmRJx+g ZBY48FVjzYzwQJ8NPI1L8nQzI24Mi0RodUDIJ7TXkAZ8Lg3hgpwJj9Y7nkVKceR7 X-Report-Abuse-To: spam@com-mpt-mgt001.mailprotect.be Sender: target-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: target-devel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Instead of maintaining a list of free I/O contexts, use an sbitmap data structure to track which I/O contexts are in use and which are free. This makes the ib_srpt driver more consistent with other LIO drivers. Cc: Doug Ledford Cc: Jason Gunthorpe Cc: Nicholas Bellinger Cc: Mike Christie Cc: Hannes Reinecke Cc: Christoph Hellwig Signed-off-by: Bart Van Assche --- drivers/infiniband/ulp/srpt/ib_srpt.c | 47 +++++++++++---------------- drivers/infiniband/ulp/srpt/ib_srpt.h | 4 --- 2 files changed, 19 insertions(+), 32 deletions(-) diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c index 0dc2df925f76..d426e01905f9 100644 --- a/drivers/infiniband/ulp/srpt/ib_srpt.c +++ b/drivers/infiniband/ulp/srpt/ib_srpt.c @@ -1217,22 +1217,15 @@ static int srpt_ch_qp_err(struct srpt_rdma_ch *ch) static struct srpt_send_ioctx *srpt_get_send_ioctx(struct srpt_rdma_ch *ch) { struct srpt_send_ioctx *ioctx; - unsigned long flags; + int tag, cpu; BUG_ON(!ch); - ioctx = NULL; - spin_lock_irqsave(&ch->spinlock, flags); - if (!list_empty(&ch->free_list)) { - ioctx = list_first_entry(&ch->free_list, - struct srpt_send_ioctx, free_list); - list_del(&ioctx->free_list); - } - spin_unlock_irqrestore(&ch->spinlock, flags); - - if (!ioctx) - return ioctx; + tag = sbitmap_queue_get(&ch->sess->sess_tag_pool, &cpu); + if (tag < 0) + return NULL; + ioctx = ch->ioctx_ring[tag]; BUG_ON(ioctx->ch != ch); ioctx->state = SRPT_STATE_NEW; WARN_ON_ONCE(ioctx->recv_ioctx); @@ -1245,6 +1238,8 @@ static struct srpt_send_ioctx *srpt_get_send_ioctx(struct srpt_rdma_ch *ch) */ memset(&ioctx->cmd, 0, sizeof(ioctx->cmd)); memset(&ioctx->sense_data, 0, sizeof(ioctx->sense_data)); + ioctx->cmd.map_tag = tag; + ioctx->cmd.map_cpu = cpu; return ioctx; } @@ -2148,7 +2143,7 @@ static int srpt_cm_req_recv(struct srpt_device *const sdev, struct srpt_rdma_ch *ch = NULL; char i_port_id[36]; u32 it_iu_len; - int i, ret; + int i, tag_num, tag_size, ret; WARN_ON_ONCE(irqs_disabled()); @@ -2248,11 +2243,8 @@ static int srpt_cm_req_recv(struct srpt_device *const sdev, goto free_rsp_cache; } - INIT_LIST_HEAD(&ch->free_list); - for (i = 0; i < ch->rq_size; i++) { + for (i = 0; i < ch->rq_size; i++) ch->ioctx_ring[i]->ch = ch; - list_add_tail(&ch->ioctx_ring[i]->free_list, &ch->free_list); - } if (!sdev->use_srq) { u16 imm_data_offset = req->req_flags & SRP_IMMED_REQUESTED ? be16_to_cpu(req->imm_data_offset) : 0; @@ -2306,18 +2298,20 @@ static int srpt_cm_req_recv(struct srpt_device *const sdev, pr_debug("registering session %s\n", ch->sess_name); + tag_num = ch->rq_size; + tag_size = 1; /* ib_srpt does not use se_sess->sess_cmd_map */ if (sport->port_guid_tpg.se_tpg_wwn) - ch->sess = target_setup_session(&sport->port_guid_tpg, 0, 0, - TARGET_PROT_NORMAL, + ch->sess = target_setup_session(&sport->port_guid_tpg, tag_num, + tag_size, TARGET_PROT_NORMAL, ch->sess_name, ch, NULL); if (sport->port_gid_tpg.se_tpg_wwn && IS_ERR_OR_NULL(ch->sess)) - ch->sess = target_setup_session(&sport->port_gid_tpg, 0, 0, - TARGET_PROT_NORMAL, i_port_id, ch, - NULL); + ch->sess = target_setup_session(&sport->port_gid_tpg, tag_num, + tag_size, TARGET_PROT_NORMAL, i_port_id, + ch, NULL); /* Retry without leading "0x" */ if (sport->port_gid_tpg.se_tpg_wwn && IS_ERR_OR_NULL(ch->sess)) - ch->sess = target_setup_session(&sport->port_gid_tpg, 0, 0, - TARGET_PROT_NORMAL, + ch->sess = target_setup_session(&sport->port_gid_tpg, tag_num, + tag_size, TARGET_PROT_NORMAL, i_port_id + 2, ch, NULL); if (IS_ERR_OR_NULL(ch->sess)) { WARN_ON_ONCE(ch->sess == NULL); @@ -3279,7 +3273,6 @@ static void srpt_release_cmd(struct se_cmd *se_cmd) struct srpt_send_ioctx, cmd); struct srpt_rdma_ch *ch = ioctx->ch; struct srpt_recv_ioctx *recv_ioctx = ioctx->recv_ioctx; - unsigned long flags; WARN_ON_ONCE(ioctx->state != SRPT_STATE_DONE && !(ioctx->cmd.transport_state & CMD_T_ABORTED)); @@ -3295,9 +3288,7 @@ static void srpt_release_cmd(struct se_cmd *se_cmd) ioctx->n_rw_ctx = 0; } - spin_lock_irqsave(&ch->spinlock, flags); - list_add(&ioctx->free_list, &ch->free_list); - spin_unlock_irqrestore(&ch->spinlock, flags); + target_free_tag(se_cmd->se_sess, se_cmd); } /** diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.h b/drivers/infiniband/ulp/srpt/ib_srpt.h index 39b3e50baf3d..ee9f20e9177a 100644 --- a/drivers/infiniband/ulp/srpt/ib_srpt.h +++ b/drivers/infiniband/ulp/srpt/ib_srpt.h @@ -207,7 +207,6 @@ struct srpt_rw_ctx { * @rw_ctxs: RDMA read/write contexts. * @imm_sg: Scatterlist for immediate data. * @rdma_cqe: RDMA completion queue element. - * @free_list: Node in srpt_rdma_ch.free_list. * @state: I/O context state. * @cmd: Target core command data structure. * @sense_data: SCSI sense data. @@ -227,7 +226,6 @@ struct srpt_send_ioctx { struct scatterlist imm_sg; struct ib_cqe rdma_cqe; - struct list_head free_list; enum srpt_command_state state; struct se_cmd cmd; u8 n_rdma; @@ -277,7 +275,6 @@ enum rdma_ch_state { * @req_lim_delta: Number of credits not yet sent back to the initiator. * @imm_data_offset: Offset from start of SRP_CMD for immediate data. * @spinlock: Protects free_list and state. - * @free_list: Head of list with free send I/O contexts. * @state: channel state. See also enum rdma_ch_state. * @using_rdma_cm: Whether the RDMA/CM or IB/CM is used for this channel. * @processing_wait_list: Whether or not cmd_wait_list is being processed. @@ -318,7 +315,6 @@ struct srpt_rdma_ch { atomic_t req_lim_delta; u16 imm_data_offset; spinlock_t spinlock; - struct list_head free_list; enum rdma_ch_state state; struct kmem_cache *rsp_buf_cache; struct srpt_send_ioctx **ioctx_ring;