From patchwork Mon Jun 4 12:30:03 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Pen X-Patchwork-Id: 10446515 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id DD275603D7 for ; Mon, 4 Jun 2018 12:30:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C106E28E58 for ; Mon, 4 Jun 2018 12:30:37 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BDBBC28EC4; Mon, 4 Jun 2018 12:30:37 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CB6E328E76 for ; Mon, 4 Jun 2018 12:30:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752646AbeFDMa2 (ORCPT ); Mon, 4 Jun 2018 08:30:28 -0400 Received: from mail-wr0-f194.google.com ([209.85.128.194]:34097 "EHLO mail-wr0-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752628AbeFDMa0 (ORCPT ); Mon, 4 Jun 2018 08:30:26 -0400 Received: by mail-wr0-f194.google.com with SMTP id a12-v6so8117740wro.1 for ; Mon, 04 Jun 2018 05:30:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=profitbricks-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=EzinHy88ekJeHVHwlgX5wzTMi3OymvCBCXHA/0CvPc4=; b=exrGRHhZuKOBDOtf/NlkmN9F8KURkAdx2JbAZUM3MRxQUJpt2l2wViJHPF4sGuFr6V lsY2z7G39rVePknnn4e4ysAhHajTvNpbAy34UEZiNVt4CVm65WtJ7mQre7DVwn08ysiE waWvQQFlHVkN2bHmPsb0y9wvu8QBQLnI9D8MB9HY3AGMVs09QMN1lmBsaolXPFlrN/Ct tt7UXbc0LJAzR13XfhlB25tDKAsyvNPtN811FMf2sLaKZxHY0jsUWDJDMUNXPSRn7m9b XCETEVYOXGIzkJ6xxob8E60Bt51zbvwyC7tETeFdZZZ6OrQ2cF1DO0Hf782fYRA0pkff 9L+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=EzinHy88ekJeHVHwlgX5wzTMi3OymvCBCXHA/0CvPc4=; b=p7yQqOkwYlXFQKkhVjOF8A1xyx2h8f98EJiRRoz5/JyRJzODxRSsq5GORvTPlno5jd tKvOzTUWW4irs9bgMEjlB0xqpdYYZQLHCfl5se/mDpAGGVC1T3DwR1Bratr9S2HtBR8F LE9ZWjCht9f45FmEU4hG80sNail4RVxMLbiyA1wznC+eMFNZWkdpJ2SgwG8nK+jHc5e1 3WPqbbVoZ3breO1ev3tJXb1dJZkJHrUWR/ggVadx/XAEDPu8eJJJjiBWA5XUo8rWZ5MD U8/qrj9o3rpjKvVdKYc/0wF8gRcKlx/AlRptaAf1oc+lfTidc9bKQuhFPZuj1KUHInXm pxtA== X-Gm-Message-State: ALKqPwffIjwml/2UQ5S2aizzPnOmA/2M9koTkZqKo7YaC10pwC36m6e/ tdM7Wn+LaxAWveCsD+m5nAvrJb5eXiQ= X-Google-Smtp-Source: ADUXVKJwdWCTTS+zcuvDRnkJubNddIrtEwdPuN168VoL8pqLx7vpUa8FI7r/8Z4DJpA6x5Jt74q/kA== X-Received: by 2002:adf:afe3:: with SMTP id y35-v6mr10153429wrd.176.1528115425138; Mon, 04 Jun 2018 05:30:25 -0700 (PDT) Received: from pb.pb.local ([62.217.45.26]) by smtp.gmail.com with ESMTPSA id s9-v6sm70578wmc.34.2018.06.04.05.30.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 04 Jun 2018 05:30:24 -0700 (PDT) From: Roman Pen To: linux-rdma@vger.kernel.org Cc: Roman Pen , Roman Pen , Christoph Hellwig , Steve Wise , Bart Van Assche , Sagi Grimberg , Doug Ledford , target-devel@vger.kernel.org Subject: [PATCH v2 8/8] IB/isert: use ib_client API to wrap ib_device Date: Mon, 4 Jun 2018 14:30:03 +0200 Message-Id: <20180604123003.24748-9-roman.penyaev@profitbricks.com> X-Mailer: git-send-email 2.13.1 In-Reply-To: <20180604123003.24748-1-roman.penyaev@profitbricks.com> References: <20180604123003.24748-1-roman.penyaev@profitbricks.com> Sender: target-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: target-devel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP ib_client API provides a way to wrap an ib_device with a specific ULP structure. Using that API local lists and mutexes can be completely avoided and allocation/removal paths become a bit cleaner. Signed-off-by: Roman Pen Cc: Christoph Hellwig Cc: Steve Wise Cc: Bart Van Assche Cc: Sagi Grimberg Cc: Doug Ledford Cc: target-devel@vger.kernel.org --- drivers/infiniband/ulp/isert/ib_isert.c | 96 ++++++++++++++++++++++----------- 1 file changed, 65 insertions(+), 31 deletions(-) diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c index 08dc9d27e5b1..c80f31573919 100644 --- a/drivers/infiniband/ulp/isert/ib_isert.c +++ b/drivers/infiniband/ulp/isert/ib_isert.c @@ -42,8 +42,7 @@ static int isert_debug_level; module_param_named(debug_level, isert_debug_level, int, 0644); MODULE_PARM_DESC(debug_level, "Enable debug tracing if > 0 (default:0)"); -static DEFINE_MUTEX(device_list_mutex); -static LIST_HEAD(device_list); +static struct ib_client isert_rdma_ib_client; static struct workqueue_struct *isert_comp_wq; static struct workqueue_struct *isert_release_wq; @@ -341,7 +340,6 @@ isert_free_device_ib_res(struct isert_device *device) static void isert_device_put(struct isert_device *device) { - mutex_lock(&device_list_mutex); isert_info("device %p refcount %d\n", device, refcount_read(&device->refcount)); if (refcount_dec_and_test(&device->refcount)) { @@ -349,48 +347,44 @@ isert_device_put(struct isert_device *device) list_del(&device->dev_node); kfree(device); } - mutex_unlock(&device_list_mutex); } static struct isert_device * isert_device_get(struct rdma_cm_id *cma_id) { struct isert_device *device; - int ret; - mutex_lock(&device_list_mutex); - list_for_each_entry(device, &device_list, dev_node) { - if (device->ib_device->node_guid == cma_id->device->node_guid) { - refcount_inc(&device->refcount); - isert_info("Found iser device %p refcount %d\n", - device, refcount_read(&device->refcount)); - mutex_unlock(&device_list_mutex); - return device; - } - } + /* Paired with isert_ib_client_remove_one() */ + rcu_read_lock(); + device = ib_get_client_data(cma_id->device, &isert_rdma_ib_client); + if (device && WARN_ON(!refcount_inc_not_zero(&device->refcount))) + device = NULL; + rcu_read_unlock(); - device = kzalloc(sizeof(struct isert_device), GFP_KERNEL); - if (!device) { - mutex_unlock(&device_list_mutex); - return ERR_PTR(-ENOMEM); - } + return device; +} + +static struct isert_device * +isert_device_alloc(struct ib_device *ib_device) +{ + struct isert_device *device; + int ret; + + device = kzalloc(sizeof(*device), GFP_KERNEL); + if (unlikely(!device)) + return NULL; INIT_LIST_HEAD(&device->dev_node); mutex_init(&device->comp_mutex); - - device->ib_device = cma_id->device; + refcount_set(&device->refcount, 1); + device->ib_device = ib_device; ret = isert_create_device_ib_res(device); if (ret) { kfree(device); - mutex_unlock(&device_list_mutex); - return ERR_PTR(ret); + return NULL; } - - refcount_inc(&device->refcount); - list_add_tail(&device->dev_node, &device_list); isert_info("Created a new iser device %p refcount %d\n", device, refcount_read(&device->refcount)); - mutex_unlock(&device_list_mutex); return device; } @@ -530,8 +524,8 @@ isert_connect_request(struct rdma_cm_id *cma_id, struct rdma_cm_event *event) goto out; device = isert_device_get(cma_id); - if (IS_ERR(device)) { - ret = PTR_ERR(device); + if (unlikely(!device)) { + ret = -ENODEV; goto out_rsp_dma_map; } isert_conn->device = device; @@ -2681,15 +2675,52 @@ static struct iscsit_transport iser_target_transport = { .iscsit_get_sup_prot_ops = isert_get_sup_prot_ops, }; +static void isert_ib_client_add_one(struct ib_device *ib_device) +{ + struct isert_device *device; + + device = isert_device_alloc(ib_device); + if (likely(device)) + ib_set_client_data(ib_device, &isert_rdma_ib_client, device); + else + pr_err("Allocattion of a device failed.\n"); +} + +static void isert_ib_client_remove_one(struct ib_device *ib_device, + void *client_data) +{ + struct isert_device *device = client_data; + + if (unlikely(!device)) + return; + + ib_set_client_data(ib_device, &isert_rdma_ib_client, NULL); + /* Paired with isert_device_get() */ + synchronize_rcu(); + isert_device_put(device); +} + +static struct ib_client isert_rdma_ib_client = { + .name = "isert_client_ib", + .add = isert_ib_client_add_one, + .remove = isert_ib_client_remove_one +}; + static int __init isert_init(void) { int ret; + ret = ib_register_client(&isert_rdma_ib_client); + if (unlikely(ret)) { + isert_err("ib_register_client(): %d\n", ret); + return ret; + } isert_comp_wq = alloc_workqueue("isert_comp_wq", WQ_UNBOUND | WQ_HIGHPRI, 0); if (!isert_comp_wq) { isert_err("Unable to allocate isert_comp_wq\n"); - return -ENOMEM; + ret = -ENOMEM; + goto unregister_client; } isert_release_wq = alloc_workqueue("isert_release_wq", WQ_UNBOUND, @@ -2707,6 +2738,8 @@ static int __init isert_init(void) destroy_comp_wq: destroy_workqueue(isert_comp_wq); +unregister_client: + ib_unregister_client(&isert_rdma_ib_client); return ret; } @@ -2717,6 +2750,7 @@ static void __exit isert_exit(void) destroy_workqueue(isert_release_wq); destroy_workqueue(isert_comp_wq); iscsit_unregister_transport(&iser_target_transport); + ib_unregister_client(&isert_rdma_ib_client); isert_info("iSER_TARGET[0] - Released iser_target_transport\n"); }