From patchwork Fri Oct 22 19:18:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12578943 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8716AC433FE for ; Fri, 22 Oct 2021 19:19:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6D02561040 for ; Fri, 22 Oct 2021 19:19:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232443AbhJVTVv (ORCPT ); Fri, 22 Oct 2021 15:21:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35742 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232747AbhJVTVt (ORCPT ); Fri, 22 Oct 2021 15:21:49 -0400 Received: from mail-ot1-x332.google.com (mail-ot1-x332.google.com [IPv6:2607:f8b0:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DF084C061766 for ; Fri, 22 Oct 2021 12:19:31 -0700 (PDT) Received: by mail-ot1-x332.google.com with SMTP id s18-20020a0568301e1200b0054e77a16651so5599936otr.7 for ; Fri, 22 Oct 2021 12:19:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=f8W28ffYZk4hShSglpsETM7Rcy0s2YqTOcOdvMh3sWY=; b=MbVEAwZCas7+RV/k/4FRF1UcLx7/nkgzTYN7z1ToyYu2CXy6gG/9ogTMoFWgVjmB2f 1WWKqA6CnQpix9U5wQkPWWRpVBoNqSlfRt8PSNlhdnOyeqC1KHhs5GHDd+KNX8xgreT2 JMv3mQkdos5K4LytJ6KJpGC97b3NEAmYf8e19XRBb8bdMQpW4h4M+0fiRPPzltkfbBva 2ZS6UzCHXUTv/ZV+EdUUgddcyRC2O2nSr9iHTysl5FEs1RzWE4ZGh6S7gUi3tbZN/wDl 3s82JQ7xlbdFulbIu0E5WW+Jp1sGhkOMgVC4lVBOAzmAYTENgY6a1wLjt7vqnxME2EHN 1yZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=f8W28ffYZk4hShSglpsETM7Rcy0s2YqTOcOdvMh3sWY=; b=aNJPEAReOOIDot7TqU4iz3P3tGaRCb7U4LYK4dyDKjee1/ukGAkOsmvLeG+Sm1SjOH ejQ2/d2RLA1LA3lxS3yb7oaxR5zPkaHdJx9tj2BpcKZOGjjY9krHCSVlVU4sOG+5uNiV BJE+1EmyPVefqV7vQblAwSHR9jLiN0rZBO2sw+4yr3MnQxZjsFVaNH48ycFGGuhzhbdd o1yWXJhAcXj6ZQMbRKSkplTfMfiCWsxI6yE+Q26yiHFiNxCo+osncXk+oX9ItSyuU83d MEXy7SWT5yS7utAcfxeEutTnwzhIl2jOD2H3DbkvaTz3rCdUOGESqT+nEFsP+wUXAL0i dpoA== X-Gm-Message-State: AOAM530NE3WOiYYrBQCg1Fd61bXFaY4Jbk9wERFW5P848B6+fMkA6AtP 8q1yiJ4OVJO2XIwwDzy0G0M= X-Google-Smtp-Source: ABdhPJzUDGFVRo5zbnFaLb3l5393RvMc7w0cHMCnjWWO1Eo0K8HYfy1UZWuc3VnAlYtzwWRjjMIf0w== X-Received: by 2002:a05:6830:348f:: with SMTP id c15mr1414788otu.257.1634930371314; Fri, 22 Oct 2021 12:19:31 -0700 (PDT) Received: from ubunto-21.tx.rr.com (2603-8081-140c-1a00-bfc7-2889-1b58-2997.res6.spectrum.com. [2603:8081:140c:1a00:bfc7:2889:1b58:2997]) by smtp.gmail.com with ESMTPSA id bf3sm2246594oib.34.2021.10.22.12.19.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 Oct 2021 12:19:31 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v3 01/10] RDMA/rxe: Make rxe_alloc() take pool lock Date: Fri, 22 Oct 2021 14:18:16 -0500 Message-Id: <20211022191824.18307-2-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211022191824.18307-1-rpearsonhpe@gmail.com> References: <20211022191824.18307-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org In rxe_pool.c there are two separate pool APIs for creating a new object rxe_alloc() and rxe_alloc_locked(). Currently they are identical except for GFP_KERNEL vs GFP_ATOMIC. Make rxe_alloc() take the pool lock which is in line with the other APIs in the library and was the original intent. Keep kzalloc() outside of the lock to avoid using GFP_ATOMIC. Make similar change to rxe_add_to_pool. The code checking the pool limit is potentially racy without a lock. The lock around finishing the pool element provides a memory barrier before 'publishing' the object. Signed-off-by: Bob Pearson --- v3 Kept rxe_alloc and rxe_alloc_locked separate to allow use of GFP_KERNEL in rxe_alloc drivers/infiniband/sw/rxe/rxe_pool.c | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_pool.c b/drivers/infiniband/sw/rxe/rxe_pool.c index 7b4cb46edfd9..8138e459b6c1 100644 --- a/drivers/infiniband/sw/rxe/rxe_pool.c +++ b/drivers/infiniband/sw/rxe/rxe_pool.c @@ -356,39 +356,51 @@ void *rxe_alloc(struct rxe_pool *pool) { struct rxe_type_info *info = &rxe_type_info[pool->type]; struct rxe_pool_entry *elem; + unsigned long flags; u8 *obj; + write_lock_irqsave(&pool->pool_lock, flags); if (atomic_inc_return(&pool->num_elem) > pool->max_elem) goto out_cnt; + write_unlock_irqrestore(&pool->pool_lock, flags); obj = kzalloc(info->size, GFP_KERNEL); - if (!obj) - goto out_cnt; + if (!obj) { + atomic_dec(&pool->num_elem); + return NULL; + } + write_lock_irqsave(&pool->pool_lock, flags); elem = (struct rxe_pool_entry *)(obj + info->elem_offset); - elem->pool = pool; kref_init(&elem->ref_cnt); + write_unlock_irqrestore(&pool->pool_lock, flags); return obj; out_cnt: atomic_dec(&pool->num_elem); + write_unlock_irqrestore(&pool->pool_lock, flags); return NULL; } int __rxe_add_to_pool(struct rxe_pool *pool, struct rxe_pool_entry *elem) { + unsigned long flags; + + write_lock_irqsave(&pool->pool_lock, flags); if (atomic_inc_return(&pool->num_elem) > pool->max_elem) goto out_cnt; elem->pool = pool; kref_init(&elem->ref_cnt); + write_unlock_irqrestore(&pool->pool_lock, flags); return 0; out_cnt: atomic_dec(&pool->num_elem); + write_unlock_irqrestore(&pool->pool_lock, flags); return -EINVAL; } From patchwork Fri Oct 22 19:18:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12578939 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E50AC433EF for ; Fri, 22 Oct 2021 19:19:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 08E5A60E95 for ; Fri, 22 Oct 2021 19:19:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232504AbhJVTVu (ORCPT ); Fri, 22 Oct 2021 15:21:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35746 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232443AbhJVTVu (ORCPT ); Fri, 22 Oct 2021 15:21:50 -0400 Received: from mail-oi1-x232.google.com (mail-oi1-x232.google.com [IPv6:2607:f8b0:4864:20::232]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 77787C061767 for ; Fri, 22 Oct 2021 12:19:32 -0700 (PDT) Received: by mail-oi1-x232.google.com with SMTP id t4so6288346oie.5 for ; Fri, 22 Oct 2021 12:19:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=VbcglwRPTMfZy8xKDggfbfFmuRg3N89aN8c8BXW7tu8=; b=B6/QqTSY34FwR8srnCt3dPKnUil3tWoQCn9A8Q/SSX+tk/ScPt52/0N5+gRVWBcec+ x5xhg3irsIXPKWwjyZ69V2xY6TjrcGLg3tSQle5h6NapBKY5FXOEFUae5roZOVidN5SA ff0Rb7193+xUFqCngC+DrxVGRwzAp1Pqg6+sQ6KWXvPmDvDalxO6WKR5x9nhDjevlVej aHly8bn+e03U7+Gn7HmetIuw9HicOcZrYeF/Z7FvcAI817plP9YL1418Lt4jfjfyWdR7 0GwHI5ddzly+JrkM7eICotCkPESnZmsAOJD4uKuiu8gvyfGfAB0MRidUXip5sWLVpNfX h8dg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=VbcglwRPTMfZy8xKDggfbfFmuRg3N89aN8c8BXW7tu8=; b=3pSyhnxNGwA+yKfMENlCsTuaZM+iTdXY9Wnlb53JIjSNc45EoxNVTtBXba/TOiwDU2 ppGk1ZwSCQQtZu4mS4/829yJhsMhmJZG1kqsIlQUHiayVdy+fGxid548zcXcTQvW496/ nrJyYr3rcjAnYI2mcH3nC+UVsfPbEptERcKRGPVQqjU3YQMvGIV70z7k36WW6Awe4g84 gcTsstJVFWlvXBxlXy0i/rk0I+DX4CSGAJ3oddqv5qyq1JvNHEIsCcw4Bmvyy12J5Ry/ IzO5ehMq/vuNKgF8LNIU6b415AMI6/4IlBJWiyU49HnYXFSxmLmwiEB5Vy1zy4Z0I8n0 NfdQ== X-Gm-Message-State: AOAM530/KLbY95wClH6R4A3LzbXNOPky4M/ehSp0aPDCh6rso0Wgc3sc 5ysAZ6yfn7q8HayZ2eal+i1AtWkMv5k= X-Google-Smtp-Source: ABdhPJwla6IIzVzdv9uUhmHNjAf7iVDXam53Mqguio9ZOZimSvBS9yps1o68JcxCRvP8HLtE2rxSkQ== X-Received: by 2002:aca:aa03:: with SMTP id t3mr1150519oie.47.1634930371863; Fri, 22 Oct 2021 12:19:31 -0700 (PDT) Received: from ubunto-21.tx.rr.com (2603-8081-140c-1a00-bfc7-2889-1b58-2997.res6.spectrum.com. [2603:8081:140c:1a00:bfc7:2889:1b58:2997]) by smtp.gmail.com with ESMTPSA id bf3sm2246594oib.34.2021.10.22.12.19.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 Oct 2021 12:19:31 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v3 02/10] RDMA/rxe: Copy setup parameters into rxe_pool Date: Fri, 22 Oct 2021 14:18:17 -0500 Message-Id: <20211022191824.18307-3-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211022191824.18307-1-rpearsonhpe@gmail.com> References: <20211022191824.18307-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org In rxe_pool.c copy remaining pool setup parameters from rxe_pool_info into rxe_pool. This saves looking up rxe_pool_info in the performance path. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_pool.c | 54 ++++++++++++---------------- drivers/infiniband/sw/rxe/rxe_pool.h | 6 ++-- 2 files changed, 25 insertions(+), 35 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_pool.c b/drivers/infiniband/sw/rxe/rxe_pool.c index 8138e459b6c1..86251145705f 100644 --- a/drivers/infiniband/sw/rxe/rxe_pool.c +++ b/drivers/infiniband/sw/rxe/rxe_pool.c @@ -7,9 +7,8 @@ #include "rxe.h" #include "rxe_loc.h" -/* info about object pools - */ -struct rxe_type_info rxe_type_info[RXE_NUM_TYPES] = { +/* info about object pools */ +static const struct rxe_type_info rxe_type_info[RXE_NUM_TYPES] = { [RXE_TYPE_UC] = { .name = "rxe-uc", .size = sizeof(struct rxe_ucontext), @@ -88,11 +87,6 @@ struct rxe_type_info rxe_type_info[RXE_NUM_TYPES] = { }, }; -static inline const char *pool_name(struct rxe_pool *pool) -{ - return rxe_type_info[pool->type].name; -} - static int rxe_pool_init_index(struct rxe_pool *pool, u32 max, u32 min) { int err = 0; @@ -127,35 +121,36 @@ int rxe_pool_init( enum rxe_elem_type type, unsigned int max_elem) { + const struct rxe_type_info *info = &rxe_type_info[type]; int err = 0; - size_t size = rxe_type_info[type].size; memset(pool, 0, sizeof(*pool)); pool->rxe = rxe; + pool->name = info->name; pool->type = type; pool->max_elem = max_elem; - pool->elem_size = ALIGN(size, RXE_POOL_ALIGN); - pool->flags = rxe_type_info[type].flags; + pool->elem_size = ALIGN(info->size, RXE_POOL_ALIGN); + pool->elem_offset = info->elem_offset; + pool->flags = info->flags; pool->index.tree = RB_ROOT; pool->key.tree = RB_ROOT; - pool->cleanup = rxe_type_info[type].cleanup; + pool->cleanup = info->cleanup; atomic_set(&pool->num_elem, 0); rwlock_init(&pool->pool_lock); - if (rxe_type_info[type].flags & RXE_POOL_INDEX) { - err = rxe_pool_init_index(pool, - rxe_type_info[type].max_index, - rxe_type_info[type].min_index); + if (info->flags & RXE_POOL_INDEX) { + err = rxe_pool_init_index(pool, info->max_index, + info->min_index); if (err) goto out; } - if (rxe_type_info[type].flags & RXE_POOL_KEY) { - pool->key.key_offset = rxe_type_info[type].key_offset; - pool->key.key_size = rxe_type_info[type].key_size; + if (info->flags & RXE_POOL_KEY) { + pool->key.key_offset = info->key_offset; + pool->key.key_size = info->key_size; } out: @@ -166,7 +161,7 @@ void rxe_pool_cleanup(struct rxe_pool *pool) { if (atomic_read(&pool->num_elem) > 0) pr_warn("%s pool destroyed with unfree'd elem\n", - pool_name(pool)); + pool->name); kfree(pool->index.table); } @@ -329,18 +324,17 @@ void __rxe_drop_index(struct rxe_pool_entry *elem) void *rxe_alloc_locked(struct rxe_pool *pool) { - struct rxe_type_info *info = &rxe_type_info[pool->type]; struct rxe_pool_entry *elem; u8 *obj; if (atomic_inc_return(&pool->num_elem) > pool->max_elem) goto out_cnt; - obj = kzalloc(info->size, GFP_ATOMIC); + obj = kzalloc(pool->elem_size, GFP_ATOMIC); if (!obj) goto out_cnt; - elem = (struct rxe_pool_entry *)(obj + info->elem_offset); + elem = (struct rxe_pool_entry *)(obj + pool->elem_offset); elem->pool = pool; kref_init(&elem->ref_cnt); @@ -354,7 +348,6 @@ void *rxe_alloc_locked(struct rxe_pool *pool) void *rxe_alloc(struct rxe_pool *pool) { - struct rxe_type_info *info = &rxe_type_info[pool->type]; struct rxe_pool_entry *elem; unsigned long flags; u8 *obj; @@ -364,14 +357,14 @@ void *rxe_alloc(struct rxe_pool *pool) goto out_cnt; write_unlock_irqrestore(&pool->pool_lock, flags); - obj = kzalloc(info->size, GFP_KERNEL); + obj = kzalloc(pool->elem_size, GFP_KERNEL); if (!obj) { atomic_dec(&pool->num_elem); return NULL; } write_lock_irqsave(&pool->pool_lock, flags); - elem = (struct rxe_pool_entry *)(obj + info->elem_offset); + elem = (struct rxe_pool_entry *)(obj + pool->elem_offset); elem->pool = pool; kref_init(&elem->ref_cnt); write_unlock_irqrestore(&pool->pool_lock, flags); @@ -409,14 +402,13 @@ void rxe_elem_release(struct kref *kref) struct rxe_pool_entry *elem = container_of(kref, struct rxe_pool_entry, ref_cnt); struct rxe_pool *pool = elem->pool; - struct rxe_type_info *info = &rxe_type_info[pool->type]; u8 *obj; if (pool->cleanup) pool->cleanup(elem); if (!(pool->flags & RXE_POOL_NO_ALLOC)) { - obj = (u8 *)elem - info->elem_offset; + obj = (u8 *)elem - pool->elem_offset; kfree(obj); } @@ -425,7 +417,6 @@ void rxe_elem_release(struct kref *kref) void *rxe_pool_get_index_locked(struct rxe_pool *pool, u32 index) { - struct rxe_type_info *info = &rxe_type_info[pool->type]; struct rb_node *node; struct rxe_pool_entry *elem; u8 *obj; @@ -445,7 +436,7 @@ void *rxe_pool_get_index_locked(struct rxe_pool *pool, u32 index) if (node) { kref_get(&elem->ref_cnt); - obj = (u8 *)elem - info->elem_offset; + obj = (u8 *)elem - pool->elem_offset; } else { obj = NULL; } @@ -467,7 +458,6 @@ void *rxe_pool_get_index(struct rxe_pool *pool, u32 index) void *rxe_pool_get_key_locked(struct rxe_pool *pool, void *key) { - struct rxe_type_info *info = &rxe_type_info[pool->type]; struct rb_node *node; struct rxe_pool_entry *elem; u8 *obj; @@ -491,7 +481,7 @@ void *rxe_pool_get_key_locked(struct rxe_pool *pool, void *key) if (node) { kref_get(&elem->ref_cnt); - obj = (u8 *)elem - info->elem_offset; + obj = (u8 *)elem - pool->elem_offset; } else { obj = NULL; } diff --git a/drivers/infiniband/sw/rxe/rxe_pool.h b/drivers/infiniband/sw/rxe/rxe_pool.h index 1feca1bffced..fb10e0098415 100644 --- a/drivers/infiniband/sw/rxe/rxe_pool.h +++ b/drivers/infiniband/sw/rxe/rxe_pool.h @@ -44,8 +44,6 @@ struct rxe_type_info { size_t key_size; }; -extern struct rxe_type_info rxe_type_info[]; - struct rxe_pool_entry { struct rxe_pool *pool; struct kref ref_cnt; @@ -61,14 +59,16 @@ struct rxe_pool_entry { struct rxe_pool { struct rxe_dev *rxe; + const char *name; rwlock_t pool_lock; /* protects pool add/del/search */ - size_t elem_size; void (*cleanup)(struct rxe_pool_entry *obj); enum rxe_pool_flags flags; enum rxe_elem_type type; unsigned int max_elem; atomic_t num_elem; + size_t elem_size; + size_t elem_offset; /* only used if indexed */ struct { From patchwork Fri Oct 22 19:18:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12578941 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5EB8C4332F for ; Fri, 22 Oct 2021 19:19:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CE02560E95 for ; Fri, 22 Oct 2021 19:19:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232747AbhJVTVv (ORCPT ); Fri, 22 Oct 2021 15:21:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35748 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232380AbhJVTVu (ORCPT ); Fri, 22 Oct 2021 15:21:50 -0400 Received: from mail-oi1-x22c.google.com (mail-oi1-x22c.google.com [IPv6:2607:f8b0:4864:20::22c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 14FC9C061764 for ; Fri, 22 Oct 2021 12:19:33 -0700 (PDT) Received: by mail-oi1-x22c.google.com with SMTP id g125so6281299oif.9 for ; Fri, 22 Oct 2021 12:19:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=DGV8bQQ2OeynsaVnpL6jsvyDeqqscqJWP4zU/ZdYmB0=; b=UsO2TLwMxvZFdopqGcg6YOOHbmdAnOWA3JQ291o5hZxef3hcGL3MupF+NzKSNKSqP6 +JEHL/BoETn7fer6o1f1g5hWwN5KqpwTKOuqVTA690kLx5PToaOjenuwpZEQlNB2p4Qp fO2XhFx9hSwnJLERKTuQfP5DXKoW63aUNGXuNfcLa8UWqBPiQyiBSvc4FLe1uuaxlFp2 0NCz28HyIg7obKv5RRA/lqQCusr+xSL4ga9NA1fD+YH2+rxjx/nwOmL6GlsMSEtq0sUH 48g9ZFG66cawWCTu+RaNVZhklFuiz/M+mucQ1RFcqWT+yNrOxkNvll2Nbv34gnhsIZ9t dO6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=DGV8bQQ2OeynsaVnpL6jsvyDeqqscqJWP4zU/ZdYmB0=; b=hG1XY3PyExQl2rKeUv06G9zUQFsMxNjGHLgZ+ew/qbv6zisQQBb6Qahxct7Uz5ZcsS VOfgkKgoR0ib0qNsAYwDJ7RBqUQ/kCsd/75iz5prpy3y4GBtHOCKOGeSR8sT4hphOGi6 sGKSdEAF6VKXmnFTI++PT19XttUj53vCysLfRvqR5gBthzJ0STLzC6FC4ttduLAXFJ6G 151CtP7ZtzNj5Owqd3fVL/hSK5QzdzUAkCDU5sCqGRpyYsTPgNXmOtptm8RrZpqk4h4O IvHIft7fv/HEOM1bSRBPhuw6qlzhzUpAWbvDn7RH1xhoQuhLr5JLAf9ZY8X4giCWADYT iJyw== X-Gm-Message-State: AOAM532nL6y2X7yZOoTamsYBhovPEvKlo7rlAtIO/TQgZYUEGQnGdhpu N6ToPz4dgwGDSmmoalA9nYM= X-Google-Smtp-Source: ABdhPJwNSAOd3iKJsC1z5WQeO31WMrYzWPtOjah5zdra59I11pz3KgKwy+AAt+2s6piThtR3xEKQ0g== X-Received: by 2002:a05:6808:1444:: with SMTP id x4mr1150191oiv.157.1634930372491; Fri, 22 Oct 2021 12:19:32 -0700 (PDT) Received: from ubunto-21.tx.rr.com (2603-8081-140c-1a00-bfc7-2889-1b58-2997.res6.spectrum.com. [2603:8081:140c:1a00:bfc7:2889:1b58:2997]) by smtp.gmail.com with ESMTPSA id bf3sm2246594oib.34.2021.10.22.12.19.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 Oct 2021 12:19:32 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v3 03/10] RDMA/rxe: Save object pointer in pool element Date: Fri, 22 Oct 2021 14:18:18 -0500 Message-Id: <20211022191824.18307-4-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211022191824.18307-1-rpearsonhpe@gmail.com> References: <20211022191824.18307-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org In rxe_pool.c currently there are many cases where it is necessary to compute the offset from a pool element struct to the object containing it in a type independent way where the offset is different for each type. By saving a pointer to the object when they are created extra work can be saved. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_pool.c | 30 ++++++++++++++++------------ drivers/infiniband/sw/rxe/rxe_pool.h | 1 + 2 files changed, 18 insertions(+), 13 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_pool.c b/drivers/infiniband/sw/rxe/rxe_pool.c index 86251145705f..2655bd372f59 100644 --- a/drivers/infiniband/sw/rxe/rxe_pool.c +++ b/drivers/infiniband/sw/rxe/rxe_pool.c @@ -220,7 +220,8 @@ static int rxe_insert_key(struct rxe_pool *pool, struct rxe_pool_entry *new) elem = rb_entry(parent, struct rxe_pool_entry, key_node); cmp = memcmp((u8 *)elem + pool->key.key_offset, - (u8 *)new + pool->key.key_offset, pool->key.key_size); + (u8 *)new + pool->key.key_offset, + pool->key.key_size); if (cmp == 0) { pr_warn("key already exists!\n"); @@ -325,7 +326,7 @@ void __rxe_drop_index(struct rxe_pool_entry *elem) void *rxe_alloc_locked(struct rxe_pool *pool) { struct rxe_pool_entry *elem; - u8 *obj; + void *obj; if (atomic_inc_return(&pool->num_elem) > pool->max_elem) goto out_cnt; @@ -334,9 +335,10 @@ void *rxe_alloc_locked(struct rxe_pool *pool) if (!obj) goto out_cnt; - elem = (struct rxe_pool_entry *)(obj + pool->elem_offset); + elem = (struct rxe_pool_entry *)((u8 *)obj + pool->elem_offset); elem->pool = pool; + elem->obj = obj; kref_init(&elem->ref_cnt); return obj; @@ -350,7 +352,7 @@ void *rxe_alloc(struct rxe_pool *pool) { struct rxe_pool_entry *elem; unsigned long flags; - u8 *obj; + void *obj; write_lock_irqsave(&pool->pool_lock, flags); if (atomic_inc_return(&pool->num_elem) > pool->max_elem) @@ -364,8 +366,9 @@ void *rxe_alloc(struct rxe_pool *pool) } write_lock_irqsave(&pool->pool_lock, flags); - elem = (struct rxe_pool_entry *)(obj + pool->elem_offset); + elem = (struct rxe_pool_entry *)((u8 *)obj + pool->elem_offset); elem->pool = pool; + elem->obj = obj; kref_init(&elem->ref_cnt); write_unlock_irqrestore(&pool->pool_lock, flags); @@ -386,6 +389,7 @@ int __rxe_add_to_pool(struct rxe_pool *pool, struct rxe_pool_entry *elem) goto out_cnt; elem->pool = pool; + elem->obj = (u8 *)elem - pool->elem_offset; kref_init(&elem->ref_cnt); write_unlock_irqrestore(&pool->pool_lock, flags); @@ -402,13 +406,13 @@ void rxe_elem_release(struct kref *kref) struct rxe_pool_entry *elem = container_of(kref, struct rxe_pool_entry, ref_cnt); struct rxe_pool *pool = elem->pool; - u8 *obj; + void *obj; if (pool->cleanup) pool->cleanup(elem); if (!(pool->flags & RXE_POOL_NO_ALLOC)) { - obj = (u8 *)elem - pool->elem_offset; + obj = elem->obj; kfree(obj); } @@ -419,7 +423,7 @@ void *rxe_pool_get_index_locked(struct rxe_pool *pool, u32 index) { struct rb_node *node; struct rxe_pool_entry *elem; - u8 *obj; + void *obj; node = pool->index.tree.rb_node; @@ -436,7 +440,7 @@ void *rxe_pool_get_index_locked(struct rxe_pool *pool, u32 index) if (node) { kref_get(&elem->ref_cnt); - obj = (u8 *)elem - pool->elem_offset; + obj = elem->obj; } else { obj = NULL; } @@ -446,7 +450,7 @@ void *rxe_pool_get_index_locked(struct rxe_pool *pool, u32 index) void *rxe_pool_get_index(struct rxe_pool *pool, u32 index) { - u8 *obj; + void *obj; unsigned long flags; read_lock_irqsave(&pool->pool_lock, flags); @@ -460,7 +464,7 @@ void *rxe_pool_get_key_locked(struct rxe_pool *pool, void *key) { struct rb_node *node; struct rxe_pool_entry *elem; - u8 *obj; + void *obj; int cmp; node = pool->key.tree.rb_node; @@ -481,7 +485,7 @@ void *rxe_pool_get_key_locked(struct rxe_pool *pool, void *key) if (node) { kref_get(&elem->ref_cnt); - obj = (u8 *)elem - pool->elem_offset; + obj = elem->obj; } else { obj = NULL; } @@ -491,7 +495,7 @@ void *rxe_pool_get_key_locked(struct rxe_pool *pool, void *key) void *rxe_pool_get_key(struct rxe_pool *pool, void *key) { - u8 *obj; + void *obj; unsigned long flags; read_lock_irqsave(&pool->pool_lock, flags); diff --git a/drivers/infiniband/sw/rxe/rxe_pool.h b/drivers/infiniband/sw/rxe/rxe_pool.h index fb10e0098415..e9bda4b14f86 100644 --- a/drivers/infiniband/sw/rxe/rxe_pool.h +++ b/drivers/infiniband/sw/rxe/rxe_pool.h @@ -46,6 +46,7 @@ struct rxe_type_info { struct rxe_pool_entry { struct rxe_pool *pool; + void *obj; struct kref ref_cnt; struct list_head list; From patchwork Fri Oct 22 19:18:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12578947 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93477C43217 for ; Fri, 22 Oct 2021 19:19:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 71D0861040 for ; Fri, 22 Oct 2021 19:19:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233657AbhJVTVw (ORCPT ); Fri, 22 Oct 2021 15:21:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35756 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233564AbhJVTVv (ORCPT ); Fri, 22 Oct 2021 15:21:51 -0400 Received: from mail-ot1-x32a.google.com (mail-ot1-x32a.google.com [IPv6:2607:f8b0:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B1DEEC061764 for ; Fri, 22 Oct 2021 12:19:33 -0700 (PDT) Received: by mail-ot1-x32a.google.com with SMTP id l24-20020a9d1c98000000b00552a5c6b23cso5588616ota.9 for ; Fri, 22 Oct 2021 12:19:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=vCSiWNX7UzpWNu1lCclCKSJe3izPc3P9KkbPQuhLTo4=; b=ekegE/yjr1OTBMIesjGRVHx+ogHZj5XCjNdAVwA7+MDrcV5vyQU8YqVmDwb5iltQns /ih9+w+nQQ6fiZR4zy+bhpArzZvvWC3bNJPRvGpIJEPTYN3zg+PAdd/k+SOAs4IsyPJR euoi88SDEcqRyc/tGVSO6gHzR0gyRG6f2Yaan3mn9zYQz2kOOmQk6Tn51tMEkT6wk/IP cjNlObVKLUk7wZcPSc6/61MWDQE6GC3nncLs1Z1yQvpMqCwnGV38yCgTA4oy32tpRvo8 tekGpJF8vX6//UI0S7q6jJgVmoSi/DkRa941ajAJw+VKtMxHM4cSsZTrPyphLz9qDBpP zaFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=vCSiWNX7UzpWNu1lCclCKSJe3izPc3P9KkbPQuhLTo4=; b=REaLAaOy9MjdkKrsYSkwGNBA6flmCbVMeu7IrE1m4XE8xFbzasH+HuKC1AhlLNhzxc zj0FCb46F3mdBA/QXhcCPQRj21rikA4bnPqVziFeCMtEdJXh8NaDgxI1jeeOf/c3p5gq bj32cF31mNJH76ayFDSgIsLTlNr12cyGJc2ECw3tF7D+RQtQuvh+q9yALrYAlMLyDEGl WLP3ARyB3MLeZsf5+2bdfKZ8dvt0MPtII7XbG2+3YjeOf3m4XXiWTCT2mupDhdXofzhw d+ZgJtg/lZikV9SHGytw7vz5ad3sdUroxDJHvvyCKu037MNzBuguoxgrzJbZ3AVqk5A5 e42A== X-Gm-Message-State: AOAM5333OvlQz37HAMtiYslRIp72E1LjfoEsZaoTWbOVfFgLJq98QpPk 1VbKrx2+clKWHz172hR+p+Q= X-Google-Smtp-Source: ABdhPJyJQTobpapQSO53Aajh07c3Lly8X7mPTLECW04x/qkahzXnSisZWPZkIbUv2hW0FkcheweIiQ== X-Received: by 2002:a05:6830:96:: with SMTP id a22mr1427828oto.249.1634930373081; Fri, 22 Oct 2021 12:19:33 -0700 (PDT) Received: from ubunto-21.tx.rr.com (2603-8081-140c-1a00-bfc7-2889-1b58-2997.res6.spectrum.com. [2603:8081:140c:1a00:bfc7:2889:1b58:2997]) by smtp.gmail.com with ESMTPSA id bf3sm2246594oib.34.2021.10.22.12.19.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 Oct 2021 12:19:32 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v3 04/10] RDMA/rxe: Combine rxe_add_index with rxe_alloc Date: Fri, 22 Oct 2021 14:18:19 -0500 Message-Id: <20211022191824.18307-5-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211022191824.18307-1-rpearsonhpe@gmail.com> References: <20211022191824.18307-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Currently rxe objects which have an index require adding and dropping the indices in a separate API call from allocating and freeing the object. These are always performed together so this patch combines them in a single operation. By taking a single pool lock around allocating the object and adding the index metadata or dropping the index metadata and releasing the object the possibility of a race condition where the metadata is not consistent with the state of the object is removed. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_mr.c | 1 - drivers/infiniband/sw/rxe/rxe_mw.c | 5 +- drivers/infiniband/sw/rxe/rxe_pool.c | 67 ++++++++++++++++----------- drivers/infiniband/sw/rxe/rxe_pool.h | 22 --------- drivers/infiniband/sw/rxe/rxe_verbs.c | 13 ------ 5 files changed, 41 insertions(+), 67 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c index 53271df10e47..6e71f67ccfe9 100644 --- a/drivers/infiniband/sw/rxe/rxe_mr.c +++ b/drivers/infiniband/sw/rxe/rxe_mr.c @@ -693,7 +693,6 @@ int rxe_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata) mr->state = RXE_MR_STATE_INVALID; rxe_drop_ref(mr_pd(mr)); - rxe_drop_index(mr); rxe_drop_ref(mr); return 0; diff --git a/drivers/infiniband/sw/rxe/rxe_mw.c b/drivers/infiniband/sw/rxe/rxe_mw.c index 9534a7fe1a98..854d0c283521 100644 --- a/drivers/infiniband/sw/rxe/rxe_mw.c +++ b/drivers/infiniband/sw/rxe/rxe_mw.c @@ -20,7 +20,6 @@ int rxe_alloc_mw(struct ib_mw *ibmw, struct ib_udata *udata) return ret; } - rxe_add_index(mw); mw->rkey = ibmw->rkey = (mw->pelem.index << 8) | rxe_get_next_key(-1); mw->state = (mw->ibmw.type == IB_MW_TYPE_2) ? RXE_MW_STATE_FREE : RXE_MW_STATE_VALID; @@ -335,7 +334,5 @@ struct rxe_mw *rxe_lookup_mw(struct rxe_qp *qp, int access, u32 rkey) void rxe_mw_cleanup(struct rxe_pool_entry *elem) { - struct rxe_mw *mw = container_of(elem, typeof(*mw), pelem); - - rxe_drop_index(mw); + /* nothing to do currently */ } diff --git a/drivers/infiniband/sw/rxe/rxe_pool.c b/drivers/infiniband/sw/rxe/rxe_pool.c index 2655bd372f59..bfed7bb48cd1 100644 --- a/drivers/infiniband/sw/rxe/rxe_pool.c +++ b/drivers/infiniband/sw/rxe/rxe_pool.c @@ -166,12 +166,16 @@ void rxe_pool_cleanup(struct rxe_pool *pool) kfree(pool->index.table); } +/* should never fail because there are at least as many indices as + * max objects + */ static u32 alloc_index(struct rxe_pool *pool) { u32 index; u32 range = pool->index.max_index - pool->index.min_index + 1; - index = find_next_zero_bit(pool->index.table, range, pool->index.last); + index = find_next_zero_bit(pool->index.table, range, + pool->index.last); if (index >= range) index = find_first_zero_bit(pool->index.table, range); @@ -192,7 +196,8 @@ static int rxe_insert_index(struct rxe_pool *pool, struct rxe_pool_entry *new) elem = rb_entry(parent, struct rxe_pool_entry, index_node); if (elem->index == new->index) { - pr_warn("element already exists!\n"); + pr_warn("element with index = 0x%x already exists!\n", + new->index); return -EINVAL; } @@ -281,31 +286,21 @@ void __rxe_drop_key(struct rxe_pool_entry *elem) write_unlock_irqrestore(&pool->pool_lock, flags); } -int __rxe_add_index_locked(struct rxe_pool_entry *elem) +static int rxe_add_index(struct rxe_pool_entry *elem) { struct rxe_pool *pool = elem->pool; int err; elem->index = alloc_index(pool); err = rxe_insert_index(pool, elem); + if (err) + clear_bit(elem->index - pool->index.min_index, + pool->index.table); return err; } -int __rxe_add_index(struct rxe_pool_entry *elem) -{ - struct rxe_pool *pool = elem->pool; - unsigned long flags; - int err; - - write_lock_irqsave(&pool->pool_lock, flags); - err = __rxe_add_index_locked(elem); - write_unlock_irqrestore(&pool->pool_lock, flags); - - return err; -} - -void __rxe_drop_index_locked(struct rxe_pool_entry *elem) +static void rxe_drop_index(struct rxe_pool_entry *elem) { struct rxe_pool *pool = elem->pool; @@ -313,20 +308,11 @@ void __rxe_drop_index_locked(struct rxe_pool_entry *elem) rb_erase(&elem->index_node, &pool->index.tree); } -void __rxe_drop_index(struct rxe_pool_entry *elem) -{ - struct rxe_pool *pool = elem->pool; - unsigned long flags; - - write_lock_irqsave(&pool->pool_lock, flags); - __rxe_drop_index_locked(elem); - write_unlock_irqrestore(&pool->pool_lock, flags); -} - void *rxe_alloc_locked(struct rxe_pool *pool) { struct rxe_pool_entry *elem; void *obj; + int err; if (atomic_inc_return(&pool->num_elem) > pool->max_elem) goto out_cnt; @@ -341,6 +327,14 @@ void *rxe_alloc_locked(struct rxe_pool *pool) elem->obj = obj; kref_init(&elem->ref_cnt); + if (pool->flags & RXE_POOL_INDEX) { + err = rxe_add_index(elem); + if (err) { + kfree(elem); + goto out_cnt; + } + } + return obj; out_cnt: @@ -353,6 +347,7 @@ void *rxe_alloc(struct rxe_pool *pool) struct rxe_pool_entry *elem; unsigned long flags; void *obj; + int err; write_lock_irqsave(&pool->pool_lock, flags); if (atomic_inc_return(&pool->num_elem) > pool->max_elem) @@ -370,6 +365,14 @@ void *rxe_alloc(struct rxe_pool *pool) elem->pool = pool; elem->obj = obj; kref_init(&elem->ref_cnt); + + if (pool->flags & RXE_POOL_INDEX) { + err = rxe_add_index(elem); + if (err) { + kfree(elem); + goto out_cnt; + } + } write_unlock_irqrestore(&pool->pool_lock, flags); return obj; @@ -383,6 +386,7 @@ void *rxe_alloc(struct rxe_pool *pool) int __rxe_add_to_pool(struct rxe_pool *pool, struct rxe_pool_entry *elem) { unsigned long flags; + int err; write_lock_irqsave(&pool->pool_lock, flags); if (atomic_inc_return(&pool->num_elem) > pool->max_elem) @@ -391,6 +395,12 @@ int __rxe_add_to_pool(struct rxe_pool *pool, struct rxe_pool_entry *elem) elem->pool = pool; elem->obj = (u8 *)elem - pool->elem_offset; kref_init(&elem->ref_cnt); + + if (pool->flags & RXE_POOL_INDEX) { + err = rxe_add_index(elem); + if (err) + goto out_cnt; + } write_unlock_irqrestore(&pool->pool_lock, flags); return 0; @@ -411,6 +421,9 @@ void rxe_elem_release(struct kref *kref) if (pool->cleanup) pool->cleanup(elem); + if (pool->flags & RXE_POOL_INDEX) + rxe_drop_index(elem); + if (!(pool->flags & RXE_POOL_NO_ALLOC)) { obj = elem->obj; kfree(obj); diff --git a/drivers/infiniband/sw/rxe/rxe_pool.h b/drivers/infiniband/sw/rxe/rxe_pool.h index e9bda4b14f86..f76addf87b4a 100644 --- a/drivers/infiniband/sw/rxe/rxe_pool.h +++ b/drivers/infiniband/sw/rxe/rxe_pool.h @@ -109,28 +109,6 @@ int __rxe_add_to_pool(struct rxe_pool *pool, struct rxe_pool_entry *elem); #define rxe_add_to_pool(pool, obj) __rxe_add_to_pool(pool, &(obj)->pelem) -/* assign an index to an indexed object and insert object into - * pool's rb tree holding and not holding the pool_lock - */ -int __rxe_add_index_locked(struct rxe_pool_entry *elem); - -#define rxe_add_index_locked(obj) __rxe_add_index_locked(&(obj)->pelem) - -int __rxe_add_index(struct rxe_pool_entry *elem); - -#define rxe_add_index(obj) __rxe_add_index(&(obj)->pelem) - -/* drop an index and remove object from rb tree - * holding and not holding the pool_lock - */ -void __rxe_drop_index_locked(struct rxe_pool_entry *elem); - -#define rxe_drop_index_locked(obj) __rxe_drop_index_locked(&(obj)->pelem) - -void __rxe_drop_index(struct rxe_pool_entry *elem); - -#define rxe_drop_index(obj) __rxe_drop_index(&(obj)->pelem) - /* assign a key to a keyed object and insert object into * pool's rb tree holding and not holding pool_lock */ diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c index 0aa0d7e52773..84ea03bc6a26 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.c +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c @@ -181,7 +181,6 @@ static int rxe_create_ah(struct ib_ah *ibah, return err; /* create index > 0 */ - rxe_add_index(ah); ah->ah_num = ah->pelem.index; if (uresp) { @@ -189,7 +188,6 @@ static int rxe_create_ah(struct ib_ah *ibah, err = copy_to_user(&uresp->ah_num, &ah->ah_num, sizeof(uresp->ah_num)); if (err) { - rxe_drop_index(ah); rxe_drop_ref(ah); return -EFAULT; } @@ -230,7 +228,6 @@ static int rxe_destroy_ah(struct ib_ah *ibah, u32 flags) { struct rxe_ah *ah = to_rah(ibah); - rxe_drop_index(ah); rxe_drop_ref(ah); return 0; } @@ -438,7 +435,6 @@ static int rxe_create_qp(struct ib_qp *ibqp, struct ib_qp_init_attr *init, if (err) return err; - rxe_add_index(qp); err = rxe_qp_from_init(rxe, qp, pd, init, uresp, ibqp->pd, udata); if (err) goto qp_init; @@ -446,7 +442,6 @@ static int rxe_create_qp(struct ib_qp *ibqp, struct ib_qp_init_attr *init, return 0; qp_init: - rxe_drop_index(qp); rxe_drop_ref(qp); return err; } @@ -491,7 +486,6 @@ static int rxe_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata) struct rxe_qp *qp = to_rqp(ibqp); rxe_qp_destroy(qp); - rxe_drop_index(qp); rxe_drop_ref(qp); return 0; } @@ -898,7 +892,6 @@ static struct ib_mr *rxe_get_dma_mr(struct ib_pd *ibpd, int access) if (!mr) return ERR_PTR(-ENOMEM); - rxe_add_index(mr); rxe_add_ref(pd); rxe_mr_init_dma(pd, access, mr); @@ -922,8 +915,6 @@ static struct ib_mr *rxe_reg_user_mr(struct ib_pd *ibpd, goto err2; } - rxe_add_index(mr); - rxe_add_ref(pd); err = rxe_mr_init_user(pd, start, length, iova, access, mr); @@ -934,7 +925,6 @@ static struct ib_mr *rxe_reg_user_mr(struct ib_pd *ibpd, err3: rxe_drop_ref(pd); - rxe_drop_index(mr); rxe_drop_ref(mr); err2: return ERR_PTR(err); @@ -957,8 +947,6 @@ static struct ib_mr *rxe_alloc_mr(struct ib_pd *ibpd, enum ib_mr_type mr_type, goto err1; } - rxe_add_index(mr); - rxe_add_ref(pd); err = rxe_mr_init_fast(pd, max_num_sg, mr); @@ -969,7 +957,6 @@ static struct ib_mr *rxe_alloc_mr(struct ib_pd *ibpd, enum ib_mr_type mr_type, err2: rxe_drop_ref(pd); - rxe_drop_index(mr); rxe_drop_ref(mr); err1: return ERR_PTR(err); From patchwork Fri Oct 22 19:18:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12578945 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1047C433F5 for ; Fri, 22 Oct 2021 19:19:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B8E0C60E95 for ; Fri, 22 Oct 2021 19:19:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233690AbhJVTVw (ORCPT ); Fri, 22 Oct 2021 15:21:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35758 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232380AbhJVTVw (ORCPT ); Fri, 22 Oct 2021 15:21:52 -0400 Received: from mail-oi1-x233.google.com (mail-oi1-x233.google.com [IPv6:2607:f8b0:4864:20::233]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 329D8C061766 for ; Fri, 22 Oct 2021 12:19:34 -0700 (PDT) Received: by mail-oi1-x233.google.com with SMTP id u69so6301471oie.3 for ; Fri, 22 Oct 2021 12:19:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2S1efil6ZsqFNqoJw8GG9YOttb//lRdzihRS6xToN5k=; b=YYKgjGuPQsLW1TCPwys6dasH1RP6wn9LTx4sXFkJ0HOJHl+7rQI/GPE0/f2LKDDnrD d3MkJQw6ur1++KmsHtwb/DJNUOsBhgur+sBVrS9VoTsBKQqibw0fNoO0axVZE6RkRzU7 jvNb0dbhYSoDPgnEPXk9X+lyPYYAkMAWS6VZ4JHU9QIQvMp6rx8TvvniR7dn8Jjy/uGA 8KYDiJPBw3EfLV/tPBAvZpFopNr//+hbnTYEoDJ92nvzxNt/eMutjyXsscTDhJSKoauF EpWUS/x9h0+JBupX3kYMCA7XUmB3EuLsyy0Em+hgLInw8GjRVsNhPLxSyUX698mgJbsE abrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2S1efil6ZsqFNqoJw8GG9YOttb//lRdzihRS6xToN5k=; b=I3bNJ28qpz5yyWqqARiBaAyPZbFX8Li7d5595SfTjUZm9hCPra3DFZN8r0gcOIk+1T nm/N38vesC2mUQ9XYL4lQuFVgk1iX35iJtEZAcwzw0lBbvhGu/2s3YosE7tPv0jHoK1d KSYmENJvU4va5+fRR7s27Nglh7yUetsGI+hF6ilJE/C+bJnqDRDE3obYTGXTvFIAbDgg Evlsjz5OkvCJO8XKUcxowh7HY/FFvmDO2nflGn70AoWB1VH94zJVri/65IYDjyhOne/6 dvUa5vaqe0u9DJwfOwsq8hOozQyE48YROs0Mxd2/6AXigK8dePKtYJlG+fFuHqJmch5Q 4VBw== X-Gm-Message-State: AOAM530PUjRHALy96O60fieSOvIMdy+gb+DUQhyANx57viGq8q70+UI7 13dl1aJ9Uy5hOFOX4cWpOPk= X-Google-Smtp-Source: ABdhPJzMjWFE9onplRcQtoBL4ueg5DLB/7dFRj+VdJqCe4SByQ/4OJvH/IyTxmtX57qna9Hq9ZNaDw== X-Received: by 2002:a05:6808:13d4:: with SMTP id d20mr8893661oiw.154.1634930373632; Fri, 22 Oct 2021 12:19:33 -0700 (PDT) Received: from ubunto-21.tx.rr.com (2603-8081-140c-1a00-bfc7-2889-1b58-2997.res6.spectrum.com. [2603:8081:140c:1a00:bfc7:2889:1b58:2997]) by smtp.gmail.com with ESMTPSA id bf3sm2246594oib.34.2021.10.22.12.19.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 Oct 2021 12:19:33 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v3 05/10] RDMA/rxe: Combine rxe_add_key with rxe_alloc Date: Fri, 22 Oct 2021 14:18:20 -0500 Message-Id: <20211022191824.18307-6-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211022191824.18307-1-rpearsonhpe@gmail.com> References: <20211022191824.18307-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Currently adding and dropping a key from a rxe object requires separate API calls from allocating and freeing the object but these are always performed together. This patch combines these into single APIs. This requires adding new rxe_allocate_with_key(_locked) APIs. By combining allocating an object and adding key metadata inside a single locked sequence and dropping the key metadata and releasing the object the possibility of a race condition where the object state and key metadata state are inconsistent is removed. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_mcast.c | 5 +- drivers/infiniband/sw/rxe/rxe_pool.c | 81 +++++++++++++-------------- drivers/infiniband/sw/rxe/rxe_pool.h | 24 ++------ 3 files changed, 45 insertions(+), 65 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_mcast.c b/drivers/infiniband/sw/rxe/rxe_mcast.c index 1c1d1b53312d..337dc2c68051 100644 --- a/drivers/infiniband/sw/rxe/rxe_mcast.c +++ b/drivers/infiniband/sw/rxe/rxe_mcast.c @@ -15,18 +15,16 @@ static struct rxe_mc_grp *create_grp(struct rxe_dev *rxe, int err; struct rxe_mc_grp *grp; - grp = rxe_alloc_locked(&rxe->mc_grp_pool); + grp = rxe_alloc_with_key_locked(&rxe->mc_grp_pool, mgid); if (!grp) return ERR_PTR(-ENOMEM); INIT_LIST_HEAD(&grp->qp_list); spin_lock_init(&grp->mcg_lock); grp->rxe = rxe; - rxe_add_key_locked(grp, mgid); err = rxe_mcast_add(rxe, mgid); if (unlikely(err)) { - rxe_drop_key_locked(grp); rxe_drop_ref(grp); return ERR_PTR(err); } @@ -174,6 +172,5 @@ void rxe_mc_cleanup(struct rxe_pool_entry *arg) struct rxe_mc_grp *grp = container_of(arg, typeof(*grp), pelem); struct rxe_dev *rxe = grp->rxe; - rxe_drop_key(grp); rxe_mcast_delete(rxe, &grp->mgid); } diff --git a/drivers/infiniband/sw/rxe/rxe_pool.c b/drivers/infiniband/sw/rxe/rxe_pool.c index bfed7bb48cd1..7fd5543ef1ae 100644 --- a/drivers/infiniband/sw/rxe/rxe_pool.c +++ b/drivers/infiniband/sw/rxe/rxe_pool.c @@ -245,47 +245,6 @@ static int rxe_insert_key(struct rxe_pool *pool, struct rxe_pool_entry *new) return 0; } -int __rxe_add_key_locked(struct rxe_pool_entry *elem, void *key) -{ - struct rxe_pool *pool = elem->pool; - int err; - - memcpy((u8 *)elem + pool->key.key_offset, key, pool->key.key_size); - err = rxe_insert_key(pool, elem); - - return err; -} - -int __rxe_add_key(struct rxe_pool_entry *elem, void *key) -{ - struct rxe_pool *pool = elem->pool; - unsigned long flags; - int err; - - write_lock_irqsave(&pool->pool_lock, flags); - err = __rxe_add_key_locked(elem, key); - write_unlock_irqrestore(&pool->pool_lock, flags); - - return err; -} - -void __rxe_drop_key_locked(struct rxe_pool_entry *elem) -{ - struct rxe_pool *pool = elem->pool; - - rb_erase(&elem->key_node, &pool->key.tree); -} - -void __rxe_drop_key(struct rxe_pool_entry *elem) -{ - struct rxe_pool *pool = elem->pool; - unsigned long flags; - - write_lock_irqsave(&pool->pool_lock, flags); - __rxe_drop_key_locked(elem); - write_unlock_irqrestore(&pool->pool_lock, flags); -} - static int rxe_add_index(struct rxe_pool_entry *elem) { struct rxe_pool *pool = elem->pool; @@ -342,6 +301,31 @@ void *rxe_alloc_locked(struct rxe_pool *pool) return NULL; } +void *rxe_alloc_with_key_locked(struct rxe_pool *pool, void *key) +{ + struct rxe_pool_entry *elem; + u8 *obj; + int err; + + obj = rxe_alloc_locked(pool); + if (!obj) + return NULL; + + elem = (struct rxe_pool_entry *)(obj + pool->elem_offset); + memcpy((u8 *)elem + pool->key.key_offset, key, pool->key.key_size); + err = rxe_insert_key(pool, elem); + if (err) { + kfree(obj); + goto out_cnt; + } + + return obj; + +out_cnt: + atomic_dec(&pool->num_elem); + return NULL; +} + void *rxe_alloc(struct rxe_pool *pool) { struct rxe_pool_entry *elem; @@ -383,6 +367,18 @@ void *rxe_alloc(struct rxe_pool *pool) return NULL; } +void *rxe_alloc_with_key(struct rxe_pool *pool, void *key) +{ + unsigned long flags; + void *obj; + + write_lock_irqsave(&pool->pool_lock, flags); + obj = rxe_alloc_with_key_locked(pool, key); + write_unlock_irqrestore(&pool->pool_lock, flags); + + return obj; +} + int __rxe_add_to_pool(struct rxe_pool *pool, struct rxe_pool_entry *elem) { unsigned long flags; @@ -424,6 +420,9 @@ void rxe_elem_release(struct kref *kref) if (pool->flags & RXE_POOL_INDEX) rxe_drop_index(elem); + if (pool->flags & RXE_POOL_KEY) + rb_erase(&elem->key_node, &pool->key.tree); + if (!(pool->flags & RXE_POOL_NO_ALLOC)) { obj = elem->obj; kfree(obj); diff --git a/drivers/infiniband/sw/rxe/rxe_pool.h b/drivers/infiniband/sw/rxe/rxe_pool.h index f76addf87b4a..e0242d488cc8 100644 --- a/drivers/infiniband/sw/rxe/rxe_pool.h +++ b/drivers/infiniband/sw/rxe/rxe_pool.h @@ -104,31 +104,15 @@ void *rxe_alloc_locked(struct rxe_pool *pool); void *rxe_alloc(struct rxe_pool *pool); +void *rxe_alloc_with_key_locked(struct rxe_pool *pool, void *key); + +void *rxe_alloc_with_key(struct rxe_pool *pool, void *key); + /* connect already allocated object to pool */ int __rxe_add_to_pool(struct rxe_pool *pool, struct rxe_pool_entry *elem); #define rxe_add_to_pool(pool, obj) __rxe_add_to_pool(pool, &(obj)->pelem) -/* assign a key to a keyed object and insert object into - * pool's rb tree holding and not holding pool_lock - */ -int __rxe_add_key_locked(struct rxe_pool_entry *elem, void *key); - -#define rxe_add_key_locked(obj, key) __rxe_add_key_locked(&(obj)->pelem, key) - -int __rxe_add_key(struct rxe_pool_entry *elem, void *key); - -#define rxe_add_key(obj, key) __rxe_add_key(&(obj)->pelem, key) - -/* remove elem from rb tree holding and not holding the pool_lock */ -void __rxe_drop_key_locked(struct rxe_pool_entry *elem); - -#define rxe_drop_key_locked(obj) __rxe_drop_key_locked(&(obj)->pelem) - -void __rxe_drop_key(struct rxe_pool_entry *elem); - -#define rxe_drop_key(obj) __rxe_drop_key(&(obj)->pelem) - /* lookup an indexed object from index holding and not holding the pool_lock. * takes a reference on object */ From patchwork Fri Oct 22 19:18:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12578949 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE9A1C4332F for ; Fri, 22 Oct 2021 19:19:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C056660E95 for ; Fri, 22 Oct 2021 19:19:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233793AbhJVTVx (ORCPT ); Fri, 22 Oct 2021 15:21:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35762 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233748AbhJVTVw (ORCPT ); Fri, 22 Oct 2021 15:21:52 -0400 Received: from mail-oi1-x230.google.com (mail-oi1-x230.google.com [IPv6:2607:f8b0:4864:20::230]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F2DBFC061764 for ; Fri, 22 Oct 2021 12:19:34 -0700 (PDT) Received: by mail-oi1-x230.google.com with SMTP id s9so6290603oiw.6 for ; Fri, 22 Oct 2021 12:19:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=v8y78ZaV1B/wQ9a1MLTFolvspCOSODCAFf3Dp4TXSvs=; b=mVQmBX6cPwnZWQDQU9dqf1BcoeuHla/ZnDuLH4jjaEFECXeCACNsUjoBDQFrRCI3NO esguPiPB4v8VN8E9R9WRwS4sgrnrYN58KdS45iuIwOe9wBFzDt+CF9mGCpxPGXSJcY9q xFtwZbLeWxMmTJr7Oy1OjTRurTlMrU+nm08AsMrjPm4ghv3QOKmIb5lQt5P/5OVZarej RE1Vj+cbwOXySJkkuTiyfqWvXbMuj8GgBESofuK6fSJZVpsxeRVtjCdwfo6XoW4P6h9/ nZEinNkoFfrlIbFyMPXkrPDIlkV8JFXMcvGfPgdpds4yrfj4BFNffV0o9LrftcbeasIr cHig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=v8y78ZaV1B/wQ9a1MLTFolvspCOSODCAFf3Dp4TXSvs=; b=0skJozMiTYEChEIaNN9tm6Kjx/Daa7wF7v0GRAK6WnLQrEgYtP0mu0/DwJHXkeg7Fy UOoks9RONzuDzH25IjvEEoohMt3DOyhe9YIKtM9S6DHqjVzGTyrD6OF+x7A9d/3qsSvy 5w971+rI7QcVbDudjI47ZJiRKbcIV2OMIgxkIkPrb7tItnB0BgZ2PcKDigWIKmReLu2M Z2/aiL6UTiikVBwaSYsDjpIWVMg+mN+uT13Sh8TZQnIY3dQ8SDgIsTcx8Z9PxlLLlWic 3Fbf914XUtoQ5gj2LW1IbOicEJDIbCFPzTkzUrjEb5FoJqoWeubeIxpKPJgJeS+v9xG1 JuOQ== X-Gm-Message-State: AOAM533ZXaoHZGZ9rXamkBVA3og0ESoS93cevOKRa9Pert3sepVRnqKH WRhPCii5QEyGrk345p72/eC5n1LoLZg= X-Google-Smtp-Source: ABdhPJwp4Zd0RX6bX56PqovmrU4HIegwdcD1cZD3VnoraR7eynFognr+1+Q8fgOAQlP7lQwlvnDk+Q== X-Received: by 2002:aca:bac3:: with SMTP id k186mr11478995oif.82.1634930374167; Fri, 22 Oct 2021 12:19:34 -0700 (PDT) Received: from ubunto-21.tx.rr.com (2603-8081-140c-1a00-bfc7-2889-1b58-2997.res6.spectrum.com. [2603:8081:140c:1a00:bfc7:2889:1b58:2997]) by smtp.gmail.com with ESMTPSA id bf3sm2246594oib.34.2021.10.22.12.19.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 Oct 2021 12:19:33 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v3 06/10] RDMA/rxe: Separate out last rxe_drop_ref Date: Fri, 22 Oct 2021 14:18:21 -0500 Message-Id: <20211022191824.18307-7-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211022191824.18307-1-rpearsonhpe@gmail.com> References: <20211022191824.18307-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Currently rxe_drop_ref() can be called from a destroy or deallocate verb or from someone releasing a reference protecting a pointer. This leads to the possibility that the object will be destroyed after rdma-core thinks it has. This was the original intent of using kref's but can cause other problems. This patch also separates out the last rxe_drop_ref as rxe_fini_ref() which can return an error if the object is not ready to be destroyed and changes rxe_drop_ref() to return an error if the object has already been destroyed. Now the destroy verbs will return -EBUSY if the object cannot be destroyed and other add/drop references will return -EINVAL if the object has already or would be destroyed. Correct programs should not normally return these values. Locking is added so that cleanup is executed atomically. Some drop references have their order changed so references are dropped after the pointer they protect has been successfully removed. This change requires that rxe_qp_destroy() be moved from the object cleanup routine to rxe_destroy_qp() the main verb API for destroying the QP so that the first stage of cleanup can operate while the QP reference count is still positive. This patch exposes some referencing errors which are addressed in the following patches. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe.h | 1 + drivers/infiniband/sw/rxe/rxe_av.c | 1 + drivers/infiniband/sw/rxe/rxe_cq.c | 9 +- drivers/infiniband/sw/rxe/rxe_loc.h | 3 +- drivers/infiniband/sw/rxe/rxe_mr.c | 10 +- drivers/infiniband/sw/rxe/rxe_mw.c | 23 +-- drivers/infiniband/sw/rxe/rxe_pool.c | 233 +++++++++++++++++++------- drivers/infiniband/sw/rxe/rxe_pool.h | 32 +++- drivers/infiniband/sw/rxe/rxe_srq.c | 8 + drivers/infiniband/sw/rxe/rxe_verbs.c | 42 +++-- 10 files changed, 253 insertions(+), 109 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe.h b/drivers/infiniband/sw/rxe/rxe.h index 1bb3fb618bf5..adecf146d999 100644 --- a/drivers/infiniband/sw/rxe/rxe.h +++ b/drivers/infiniband/sw/rxe/rxe.h @@ -14,6 +14,7 @@ #include #include +#include #include #include diff --git a/drivers/infiniband/sw/rxe/rxe_av.c b/drivers/infiniband/sw/rxe/rxe_av.c index 38c7b6fb39d7..5084841a7cd0 100644 --- a/drivers/infiniband/sw/rxe/rxe_av.c +++ b/drivers/infiniband/sw/rxe/rxe_av.c @@ -117,6 +117,7 @@ struct rxe_av *rxe_get_av(struct rxe_pkt_info *pkt) if (ah_num) { /* only new user provider or kernel client */ ah = rxe_pool_get_index(&pkt->rxe->ah_pool, ah_num); + rxe_drop_ref(ah); if (!ah || ah->ah_num != ah_num || rxe_ah_pd(ah) != pkt->qp->pd) { pr_warn("Unable to find AH matching ah_num\n"); return NULL; diff --git a/drivers/infiniband/sw/rxe/rxe_cq.c b/drivers/infiniband/sw/rxe/rxe_cq.c index 6848426c074f..0c05d612ae63 100644 --- a/drivers/infiniband/sw/rxe/rxe_cq.c +++ b/drivers/infiniband/sw/rxe/rxe_cq.c @@ -141,18 +141,15 @@ int rxe_cq_post(struct rxe_cq *cq, struct rxe_cqe *cqe, int solicited) return 0; } -void rxe_cq_disable(struct rxe_cq *cq) +void rxe_cq_cleanup(struct rxe_pool_entry *arg) { + struct rxe_cq *cq = container_of(arg, typeof(*cq), pelem); unsigned long flags; + /* TODO get rid of this */ spin_lock_irqsave(&cq->cq_lock, flags); cq->is_dying = true; spin_unlock_irqrestore(&cq->cq_lock, flags); -} - -void rxe_cq_cleanup(struct rxe_pool_entry *arg) -{ - struct rxe_cq *cq = container_of(arg, typeof(*cq), pelem); if (cq->queue) rxe_queue_cleanup(cq->queue); diff --git a/drivers/infiniband/sw/rxe/rxe_loc.h b/drivers/infiniband/sw/rxe/rxe_loc.h index 1ca43b859d80..a25d1c9f6adb 100644 --- a/drivers/infiniband/sw/rxe/rxe_loc.h +++ b/drivers/infiniband/sw/rxe/rxe_loc.h @@ -35,8 +35,6 @@ int rxe_cq_resize_queue(struct rxe_cq *cq, int new_cqe, int rxe_cq_post(struct rxe_cq *cq, struct rxe_cqe *cqe, int solicited); -void rxe_cq_disable(struct rxe_cq *cq); - void rxe_cq_cleanup(struct rxe_pool_entry *arg); /* rxe_mcast.c */ @@ -187,6 +185,7 @@ int rxe_srq_from_init(struct rxe_dev *rxe, struct rxe_srq *srq, int rxe_srq_from_attr(struct rxe_dev *rxe, struct rxe_srq *srq, struct ib_srq_attr *attr, enum ib_srq_attr_mask mask, struct rxe_modify_srq_cmd *ucmd, struct ib_udata *udata); +void rxe_srq_cleanup(struct rxe_pool_entry *arg); void rxe_dealloc(struct ib_device *ib_dev); diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c index 6e71f67ccfe9..6c50c8562fd8 100644 --- a/drivers/infiniband/sw/rxe/rxe_mr.c +++ b/drivers/infiniband/sw/rxe/rxe_mr.c @@ -684,6 +684,8 @@ int rxe_mr_set_page(struct ib_mr *ibmr, u64 addr) int rxe_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata) { struct rxe_mr *mr = to_rmr(ibmr); + struct rxe_pd *pd = mr_pd(mr); + int err; if (atomic_read(&mr->num_mw) > 0) { pr_warn("%s: Attempt to deregister an MR while bound to MWs\n", @@ -692,8 +694,12 @@ int rxe_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata) } mr->state = RXE_MR_STATE_INVALID; - rxe_drop_ref(mr_pd(mr)); - rxe_drop_ref(mr); + + err = rxe_fini_ref(mr); + if (err) + return err; + + rxe_drop_ref(pd); return 0; } diff --git a/drivers/infiniband/sw/rxe/rxe_mw.c b/drivers/infiniband/sw/rxe/rxe_mw.c index 854d0c283521..599699f93332 100644 --- a/drivers/infiniband/sw/rxe/rxe_mw.c +++ b/drivers/infiniband/sw/rxe/rxe_mw.c @@ -56,12 +56,15 @@ int rxe_dealloc_mw(struct ib_mw *ibmw) struct rxe_mw *mw = to_rmw(ibmw); struct rxe_pd *pd = to_rpd(ibmw->pd); unsigned long flags; + int err; spin_lock_irqsave(&mw->lock, flags); rxe_do_dealloc_mw(mw); spin_unlock_irqrestore(&mw->lock, flags); - rxe_drop_ref(mw); + err = rxe_fini_ref(mw); + if (err) + return err; rxe_drop_ref(pd); return 0; @@ -177,9 +180,9 @@ static void rxe_do_bind_mw(struct rxe_qp *qp, struct rxe_send_wqe *wqe, } if (mw->length) { + /* take over ref on mr from caller */ mw->mr = mr; atomic_inc(&mr->num_mw); - rxe_add_ref(mr); } if (mw->ibmw.type == IB_MW_TYPE_2) { @@ -192,7 +195,7 @@ int rxe_bind_mw(struct rxe_qp *qp, struct rxe_send_wqe *wqe) { int ret; struct rxe_mw *mw; - struct rxe_mr *mr; + struct rxe_mr *mr = NULL; struct rxe_dev *rxe = to_rdev(qp->ibqp.device); u32 mw_rkey = wqe->wr.wr.mw.mw_rkey; u32 mr_lkey = wqe->wr.wr.mw.mr_lkey; @@ -217,25 +220,23 @@ int rxe_bind_mw(struct rxe_qp *qp, struct rxe_send_wqe *wqe) } if (unlikely(mr->lkey != mr_lkey)) { + rxe_drop_ref(mr); ret = -EINVAL; - goto err_drop_mr; + goto err_drop_mw; } - } else { - mr = NULL; } spin_lock_irqsave(&mw->lock, flags); - ret = rxe_check_bind_mw(qp, wqe, mw, mr); - if (ret) + if (ret) { + if (mr) + rxe_drop_ref(mr); goto err_unlock; + } rxe_do_bind_mw(qp, wqe, mw, mr); err_unlock: spin_unlock_irqrestore(&mw->lock, flags); -err_drop_mr: - if (mr) - rxe_drop_ref(mr); err_drop_mw: rxe_drop_ref(mw); err: diff --git a/drivers/infiniband/sw/rxe/rxe_pool.c b/drivers/infiniband/sw/rxe/rxe_pool.c index 7fd5543ef1ae..4be43ae58219 100644 --- a/drivers/infiniband/sw/rxe/rxe_pool.c +++ b/drivers/infiniband/sw/rxe/rxe_pool.c @@ -33,6 +33,7 @@ static const struct rxe_type_info rxe_type_info[RXE_NUM_TYPES] = { .name = "rxe-srq", .size = sizeof(struct rxe_srq), .elem_offset = offsetof(struct rxe_srq, pelem), + .cleanup = rxe_srq_cleanup, .flags = RXE_POOL_INDEX | RXE_POOL_NO_ALLOC, .min_index = RXE_MIN_SRQ_INDEX, .max_index = RXE_MAX_SRQ_INDEX, @@ -195,9 +196,13 @@ static int rxe_insert_index(struct rxe_pool *pool, struct rxe_pool_entry *new) parent = *link; elem = rb_entry(parent, struct rxe_pool_entry, index_node); - if (elem->index == new->index) { - pr_warn("element with index = 0x%x already exists!\n", - new->index); + /* this can happen if memory was recycled and/or the + * old object was not deleted from the pool index + */ + if (unlikely(elem == new || elem->index == new->index)) { + pr_warn("%s#%d rf=%d: already in pool\n", + pool->name, new->index, + refcount_read(&new->refcnt)); return -EINVAL; } @@ -281,21 +286,20 @@ void *rxe_alloc_locked(struct rxe_pool *pool) goto out_cnt; elem = (struct rxe_pool_entry *)((u8 *)obj + pool->elem_offset); - elem->pool = pool; elem->obj = obj; - kref_init(&elem->ref_cnt); + refcount_set(&elem->refcnt, 1); if (pool->flags & RXE_POOL_INDEX) { err = rxe_add_index(elem); - if (err) { - kfree(elem); - goto out_cnt; - } + if (err) + goto out_free; } return obj; +out_free: + kfree(elem); out_cnt: atomic_dec(&pool->num_elem); return NULL; @@ -307,20 +311,33 @@ void *rxe_alloc_with_key_locked(struct rxe_pool *pool, void *key) u8 *obj; int err; - obj = rxe_alloc_locked(pool); + if (atomic_inc_return(&pool->num_elem) > pool->max_elem) + goto out_cnt; + + obj = kzalloc(pool->elem_size, GFP_ATOMIC); if (!obj) - return NULL; + goto out_cnt; + + elem = (struct rxe_pool_entry *)((u8 *)obj + pool->elem_offset); + elem->pool = pool; + elem->obj = obj; + refcount_set(&elem->refcnt, 1); + + if (pool->flags & RXE_POOL_INDEX) { + err = rxe_add_index(elem); + if (err) + goto out_free; + } - elem = (struct rxe_pool_entry *)(obj + pool->elem_offset); memcpy((u8 *)elem + pool->key.key_offset, key, pool->key.key_size); err = rxe_insert_key(pool, elem); - if (err) { - kfree(obj); - goto out_cnt; - } + if (err) + goto out_free; return obj; +out_free: + kfree(obj); out_cnt: atomic_dec(&pool->num_elem); return NULL; @@ -348,19 +365,19 @@ void *rxe_alloc(struct rxe_pool *pool) elem = (struct rxe_pool_entry *)((u8 *)obj + pool->elem_offset); elem->pool = pool; elem->obj = obj; - kref_init(&elem->ref_cnt); + refcount_set(&elem->refcnt, 1); if (pool->flags & RXE_POOL_INDEX) { err = rxe_add_index(elem); - if (err) { - kfree(elem); - goto out_cnt; - } + if (err) + goto out_free; } write_unlock_irqrestore(&pool->pool_lock, flags); return obj; +out_free: + kfree(elem); out_cnt: atomic_dec(&pool->num_elem); write_unlock_irqrestore(&pool->pool_lock, flags); @@ -369,14 +386,48 @@ void *rxe_alloc(struct rxe_pool *pool) void *rxe_alloc_with_key(struct rxe_pool *pool, void *key) { + struct rxe_pool_entry *elem; unsigned long flags; - void *obj; + u8 *obj; + int err; + + write_lock_irqsave(&pool->pool_lock, flags); + if (atomic_inc_return(&pool->num_elem) > pool->max_elem) + goto out_cnt; + write_unlock_irqrestore(&pool->pool_lock, flags); + + obj = kzalloc(pool->elem_size, GFP_KERNEL); + if (!obj) { + atomic_dec(&pool->num_elem); + return NULL; + } write_lock_irqsave(&pool->pool_lock, flags); - obj = rxe_alloc_with_key_locked(pool, key); + elem = (struct rxe_pool_entry *)((u8 *)obj + pool->elem_offset); + elem->pool = pool; + elem->obj = obj; + refcount_set(&elem->refcnt, 1); + + if (pool->flags & RXE_POOL_INDEX) { + err = rxe_add_index(elem); + if (err) + goto out_free; + } + + memcpy((u8 *)elem + pool->key.key_offset, key, pool->key.key_size); + err = rxe_insert_key(pool, elem); + if (err) + goto out_free; write_unlock_irqrestore(&pool->pool_lock, flags); return obj; + +out_free: + kfree(elem); +out_cnt: + atomic_dec(&pool->num_elem); + write_unlock_irqrestore(&pool->pool_lock, flags); + return NULL; } int __rxe_add_to_pool(struct rxe_pool *pool, struct rxe_pool_entry *elem) @@ -390,7 +441,7 @@ int __rxe_add_to_pool(struct rxe_pool *pool, struct rxe_pool_entry *elem) elem->pool = pool; elem->obj = (u8 *)elem - pool->elem_offset; - kref_init(&elem->ref_cnt); + refcount_set(&elem->refcnt, 1); if (pool->flags & RXE_POOL_INDEX) { err = rxe_add_index(elem); @@ -407,35 +458,11 @@ int __rxe_add_to_pool(struct rxe_pool *pool, struct rxe_pool_entry *elem) return -EINVAL; } -void rxe_elem_release(struct kref *kref) -{ - struct rxe_pool_entry *elem = - container_of(kref, struct rxe_pool_entry, ref_cnt); - struct rxe_pool *pool = elem->pool; - void *obj; - - if (pool->cleanup) - pool->cleanup(elem); - - if (pool->flags & RXE_POOL_INDEX) - rxe_drop_index(elem); - - if (pool->flags & RXE_POOL_KEY) - rb_erase(&elem->key_node, &pool->key.tree); - - if (!(pool->flags & RXE_POOL_NO_ALLOC)) { - obj = elem->obj; - kfree(obj); - } - - atomic_dec(&pool->num_elem); -} - void *rxe_pool_get_index_locked(struct rxe_pool *pool, u32 index) { struct rb_node *node; struct rxe_pool_entry *elem; - void *obj; + void *obj = NULL; node = pool->index.tree.rb_node; @@ -450,12 +477,8 @@ void *rxe_pool_get_index_locked(struct rxe_pool *pool, u32 index) break; } - if (node) { - kref_get(&elem->ref_cnt); + if (node && refcount_inc_not_zero(&elem->refcnt)) obj = elem->obj; - } else { - obj = NULL; - } return obj; } @@ -476,7 +499,7 @@ void *rxe_pool_get_key_locked(struct rxe_pool *pool, void *key) { struct rb_node *node; struct rxe_pool_entry *elem; - void *obj; + void *obj = NULL; int cmp; node = pool->key.tree.rb_node; @@ -495,12 +518,8 @@ void *rxe_pool_get_key_locked(struct rxe_pool *pool, void *key) break; } - if (node) { - kref_get(&elem->ref_cnt); + if (node && refcount_inc_not_zero(&elem->refcnt)) obj = elem->obj; - } else { - obj = NULL; - } return obj; } @@ -516,3 +535,97 @@ void *rxe_pool_get_key(struct rxe_pool *pool, void *key) return obj; } + +int __rxe_add_ref_locked(struct rxe_pool_entry *elem) +{ + return refcount_inc_not_zero(&elem->refcnt) ? 0 : -EINVAL; +} + +int __rxe_add_ref(struct rxe_pool_entry *elem) +{ + struct rxe_pool *pool = elem->pool; + unsigned long flags; + int ret; + + read_lock_irqsave(&pool->pool_lock, flags); + ret = __rxe_add_ref_locked(elem); + read_unlock_irqrestore(&pool->pool_lock, flags); + + return ret; +} + +int __rxe_drop_ref_locked(struct rxe_pool_entry *elem) +{ + return refcount_dec_not_one(&elem->refcnt) ? 0 : -EINVAL; +} + +int __rxe_drop_ref(struct rxe_pool_entry *elem) +{ + struct rxe_pool *pool = elem->pool; + unsigned long flags; + int ret; + + read_lock_irqsave(&pool->pool_lock, flags); + ret = __rxe_drop_ref_locked(elem); + read_unlock_irqrestore(&pool->pool_lock, flags); + + return ret; +} + +static int __rxe_fini(struct rxe_pool_entry *elem) +{ + struct rxe_pool *pool = elem->pool; + int done; + + done = refcount_dec_if_one(&elem->refcnt); + if (done) { + if (pool->flags & RXE_POOL_INDEX) + rxe_drop_index(elem); + if (pool->flags & RXE_POOL_KEY) + rb_erase(&elem->key_node, &pool->key.tree); + atomic_dec(&pool->num_elem); + return 0; + } else { + return -EBUSY; + } +} + +/* can only be used by pools that have a cleanup + * routine that can run while holding a spinlock + */ +int __rxe_fini_ref_locked(struct rxe_pool_entry *elem) +{ + struct rxe_pool *pool = elem->pool; + int ret; + + ret = __rxe_fini(elem); + + if (!ret) { + if (pool->cleanup) + pool->cleanup(elem); + if (!(pool->flags & RXE_POOL_NO_ALLOC)) + kfree(elem->obj); + } + + return ret; +} + +int __rxe_fini_ref(struct rxe_pool_entry *elem) +{ + struct rxe_pool *pool = elem->pool; + unsigned long flags; + int ret; + + read_lock_irqsave(&pool->pool_lock, flags); + ret = __rxe_fini(elem); + read_unlock_irqrestore(&pool->pool_lock, flags); + + if (!ret) { + if (pool->cleanup) + pool->cleanup(elem); + if (!(pool->flags & RXE_POOL_NO_ALLOC)) + kfree(elem->obj); + } + + return ret; +} diff --git a/drivers/infiniband/sw/rxe/rxe_pool.h b/drivers/infiniband/sw/rxe/rxe_pool.h index e0242d488cc8..7df52d34e306 100644 --- a/drivers/infiniband/sw/rxe/rxe_pool.h +++ b/drivers/infiniband/sw/rxe/rxe_pool.h @@ -47,7 +47,7 @@ struct rxe_type_info { struct rxe_pool_entry { struct rxe_pool *pool; void *obj; - struct kref ref_cnt; + refcount_t refcnt; struct list_head list; /* only used if keyed */ @@ -127,13 +127,33 @@ void *rxe_pool_get_key_locked(struct rxe_pool *pool, void *key); void *rxe_pool_get_key(struct rxe_pool *pool, void *key); -/* cleanup an object when all references are dropped */ -void rxe_elem_release(struct kref *kref); - /* take a reference on an object */ -#define rxe_add_ref(elem) kref_get(&(elem)->pelem.ref_cnt) +int __rxe_add_ref_locked(struct rxe_pool_entry *elem); + +#define rxe_add_ref_locked(obj) __rxe_add_ref_locked(&(obj)->pelem) + +int __rxe_add_ref(struct rxe_pool_entry *elem); + +#define rxe_add_ref(obj) __rxe_add_ref(&(obj)->pelem) /* drop a reference on an object */ -#define rxe_drop_ref(elem) kref_put(&(elem)->pelem.ref_cnt, rxe_elem_release) +int __rxe_drop_ref_locked(struct rxe_pool_entry *elem); + +#define rxe_drop_ref_locked(obj) __rxe_drop_ref_locked(&(obj)->pelem) + +int __rxe_drop_ref(struct rxe_pool_entry *elem); + +#define rxe_drop_ref(obj) __rxe_drop_ref(&(obj)->pelem) + +/* drop last reference on an object */ +int __rxe_fini_ref_locked(struct rxe_pool_entry *elem); + +#define rxe_fini_ref_locked(obj) __rxe_fini_ref_locked(&(obj)->pelem) + +int __rxe_fini_ref(struct rxe_pool_entry *elem); + +#define rxe_fini_ref(obj) __rxe_fini_ref(&(obj)->pelem) + +#define rxe_read_ref(obj) refcount_read(&(obj)->pelem.refcnt) #endif /* RXE_POOL_H */ diff --git a/drivers/infiniband/sw/rxe/rxe_srq.c b/drivers/infiniband/sw/rxe/rxe_srq.c index eb1c4c3b3a78..bb00643a2929 100644 --- a/drivers/infiniband/sw/rxe/rxe_srq.c +++ b/drivers/infiniband/sw/rxe/rxe_srq.c @@ -154,3 +154,11 @@ int rxe_srq_from_attr(struct rxe_dev *rxe, struct rxe_srq *srq, srq->rq.queue = NULL; return err; } + +void rxe_srq_cleanup(struct rxe_pool_entry *arg) +{ + struct rxe_srq *srq = container_of(arg, typeof(*srq), pelem); + + if (srq->rq.queue) + rxe_queue_cleanup(srq->rq.queue); +} diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c index 84ea03bc6a26..61fa633775f3 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.c +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c @@ -115,7 +115,7 @@ static void rxe_dealloc_ucontext(struct ib_ucontext *ibuc) { struct rxe_ucontext *uc = to_ruc(ibuc); - rxe_drop_ref(uc); + rxe_fini_ref(uc); } static int rxe_port_immutable(struct ib_device *dev, u32 port_num, @@ -149,8 +149,7 @@ static int rxe_dealloc_pd(struct ib_pd *ibpd, struct ib_udata *udata) { struct rxe_pd *pd = to_rpd(ibpd); - rxe_drop_ref(pd); - return 0; + return rxe_fini_ref(pd); } static int rxe_create_ah(struct ib_ah *ibah, @@ -188,7 +187,7 @@ static int rxe_create_ah(struct ib_ah *ibah, err = copy_to_user(&uresp->ah_num, &ah->ah_num, sizeof(uresp->ah_num)); if (err) { - rxe_drop_ref(ah); + rxe_fini_ref(ah); return -EFAULT; } } else if (ah->is_user) { @@ -228,8 +227,7 @@ static int rxe_destroy_ah(struct ib_ah *ibah, u32 flags) { struct rxe_ah *ah = to_rah(ibah); - rxe_drop_ref(ah); - return 0; + return rxe_fini_ref(ah); } static int post_one_recv(struct rxe_rq *rq, const struct ib_recv_wr *ibwr) @@ -313,8 +311,8 @@ static int rxe_create_srq(struct ib_srq *ibsrq, struct ib_srq_init_attr *init, return 0; err2: + rxe_fini_ref(srq); rxe_drop_ref(pd); - rxe_drop_ref(srq); err1: return err; } @@ -367,12 +365,15 @@ static int rxe_query_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr) static int rxe_destroy_srq(struct ib_srq *ibsrq, struct ib_udata *udata) { struct rxe_srq *srq = to_rsrq(ibsrq); + struct rxe_pd *pd = srq->pd; + int err; + + err = rxe_fini_ref(srq); + if (err) + return err; - if (srq->rq.queue) - rxe_queue_cleanup(srq->rq.queue); + rxe_drop_ref(pd); - rxe_drop_ref(srq->pd); - rxe_drop_ref(srq); return 0; } @@ -437,12 +438,12 @@ static int rxe_create_qp(struct ib_qp *ibqp, struct ib_qp_init_attr *init, err = rxe_qp_from_init(rxe, qp, pd, init, uresp, ibqp->pd, udata); if (err) - goto qp_init; + goto err_fini; return 0; -qp_init: - rxe_drop_ref(qp); +err_fini: + rxe_fini_ref(qp); return err; } @@ -486,8 +487,8 @@ static int rxe_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata) struct rxe_qp *qp = to_rqp(ibqp); rxe_qp_destroy(qp); - rxe_drop_ref(qp); - return 0; + + return rxe_fini_ref(qp); } static int validate_send_wr(struct rxe_qp *qp, const struct ib_send_wr *ibwr, @@ -797,10 +798,7 @@ static int rxe_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata) { struct rxe_cq *cq = to_rcq(ibcq); - rxe_cq_disable(cq); - - rxe_drop_ref(cq); - return 0; + return rxe_fini_ref(cq); } static int rxe_resize_cq(struct ib_cq *ibcq, int cqe, struct ib_udata *udata) @@ -924,8 +922,8 @@ static struct ib_mr *rxe_reg_user_mr(struct ib_pd *ibpd, return &mr->ibmr; err3: + rxe_fini_ref(mr); rxe_drop_ref(pd); - rxe_drop_ref(mr); err2: return ERR_PTR(err); } @@ -956,8 +954,8 @@ static struct ib_mr *rxe_alloc_mr(struct ib_pd *ibpd, enum ib_mr_type mr_type, return &mr->ibmr; err2: + rxe_fini_ref(mr); rxe_drop_ref(pd); - rxe_drop_ref(mr); err1: return ERR_PTR(err); } From patchwork Fri Oct 22 19:18:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12578951 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 365BCC433EF for ; Fri, 22 Oct 2021 19:19:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1D70361040 for ; Fri, 22 Oct 2021 19:19:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233926AbhJVTVx (ORCPT ); Fri, 22 Oct 2021 15:21:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35764 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232380AbhJVTVx (ORCPT ); Fri, 22 Oct 2021 15:21:53 -0400 Received: from mail-ot1-x336.google.com (mail-ot1-x336.google.com [IPv6:2607:f8b0:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4B6C9C061766 for ; Fri, 22 Oct 2021 12:19:35 -0700 (PDT) Received: by mail-ot1-x336.google.com with SMTP id b4-20020a9d7544000000b00552ab826e3aso5627958otl.4 for ; Fri, 22 Oct 2021 12:19:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ZIxAlVkKkqFaS0fEq4fOlQ/3/WsrfxnPjjypX5irPF0=; b=WixR/9FwyRiOUQoELANuQf8k0fkVCPooETaKFexO6pDkH/lexDDZCU3wm6zd/nIzzd Fu+lQ9OhHTFIX6BFKYkzutNbKxRJyVITU7fkWw2i1L1WTlblMhWLY6Vk3FpGuSvgFR2c VdrZL4bNzauWW+PqVWA3v4MZWgWURedE+xSpEYAW7P6n1rM7pwrOzDf1uP/SDMzt1s3e fvPu+WMVT81618iSR/pMEQQ03SiAXRfMW/YVoC4Ub5AXL6Qsb3m7NbbP2mWoliNFE9Pc d8qI+CiBwaAqsmupaYxiyQDONlj+nGLHGlo/LnnYeFStmjzBcSxrQ5uuMxEll0UlyQbQ UlgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ZIxAlVkKkqFaS0fEq4fOlQ/3/WsrfxnPjjypX5irPF0=; b=V4J7iCIl58JZi+nBynBEMnlLMaTWilDBtfnigeJlrX6WQifr3ykTcYzfsiaZ4s8XEu ZtbeHF40P6CbJopV8x8waGf3UQ4GdfXiSqMCDjhQjEdV/HljrpZTMQMqlFsoUrLwPalP ZEtAfUdnrby290A1aUEn2s6aFDH3ZjAxoRUd6hXlPCxm2EwsNDkmWCB5PllfveBd32ge +d6rmsqMPDoefPESTF1ji1ob3a/U6QlrUbf8GJZ8FPS0NfZEUGh6fpkoCnl11okqcfGD cpzQ5+abIf+sUs1NYOxfhUEoB3zIkcowTFc49h6WHgmVhlBg6GpOZ0Pk3eewh8LaXOmw sqtQ== X-Gm-Message-State: AOAM533YRuH0gKimq6V2cCt+/J7Ywy0mIOFyB+1tmGg3CBC8RW8XhVdz yhUmfe1X7Cn0DDsDUiDqpb4= X-Google-Smtp-Source: ABdhPJxOkFpaBNRCnUfJNn8Q7MwcLXZFCktEJqvM1qciC/tmtzccOewq7x78J8jDO9ZyuXukjB5j0w== X-Received: by 2002:a05:6830:410d:: with SMTP id w13mr1462040ott.292.1634930374671; Fri, 22 Oct 2021 12:19:34 -0700 (PDT) Received: from ubunto-21.tx.rr.com (2603-8081-140c-1a00-bfc7-2889-1b58-2997.res6.spectrum.com. [2603:8081:140c:1a00:bfc7:2889:1b58:2997]) by smtp.gmail.com with ESMTPSA id bf3sm2246594oib.34.2021.10.22.12.19.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 Oct 2021 12:19:34 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v3 07/10] RDMA/rxe: Rewrite rxe_mcast.c Date: Fri, 22 Oct 2021 14:18:22 -0500 Message-Id: <20211022191824.18307-8-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211022191824.18307-1-rpearsonhpe@gmail.com> References: <20211022191824.18307-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Rewrite rxe_mcast to support rxe_fini_ref and clean up referencing. Drop mc_elem as rxe pool objects and just use kmalloc/kfree. Take qp references as well as mc_grp references to force all mc groups to be de-attached before destroying QPs. Simplify the api to rxe_verbs. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe.c | 8 - drivers/infiniband/sw/rxe/rxe_loc.h | 11 +- drivers/infiniband/sw/rxe/rxe_mcast.c | 205 +++++++++++++++++--------- drivers/infiniband/sw/rxe/rxe_net.c | 22 --- drivers/infiniband/sw/rxe/rxe_pool.c | 6 - drivers/infiniband/sw/rxe/rxe_pool.h | 1 - drivers/infiniband/sw/rxe/rxe_verbs.c | 12 +- drivers/infiniband/sw/rxe/rxe_verbs.h | 3 +- 8 files changed, 143 insertions(+), 125 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe.c b/drivers/infiniband/sw/rxe/rxe.c index 8e0f9c489cab..4298a1d20ad5 100644 --- a/drivers/infiniband/sw/rxe/rxe.c +++ b/drivers/infiniband/sw/rxe/rxe.c @@ -31,7 +31,6 @@ void rxe_dealloc(struct ib_device *ib_dev) rxe_pool_cleanup(&rxe->mr_pool); rxe_pool_cleanup(&rxe->mw_pool); rxe_pool_cleanup(&rxe->mc_grp_pool); - rxe_pool_cleanup(&rxe->mc_elem_pool); if (rxe->tfm) crypto_free_shash(rxe->tfm); @@ -165,15 +164,8 @@ static int rxe_init_pools(struct rxe_dev *rxe) if (err) goto err9; - err = rxe_pool_init(rxe, &rxe->mc_elem_pool, RXE_TYPE_MC_ELEM, - rxe->attr.max_total_mcast_qp_attach); - if (err) - goto err10; - return 0; -err10: - rxe_pool_cleanup(&rxe->mc_grp_pool); err9: rxe_pool_cleanup(&rxe->mw_pool); err8: diff --git a/drivers/infiniband/sw/rxe/rxe_loc.h b/drivers/infiniband/sw/rxe/rxe_loc.h index a25d1c9f6adb..78312df8eea3 100644 --- a/drivers/infiniband/sw/rxe/rxe_loc.h +++ b/drivers/infiniband/sw/rxe/rxe_loc.h @@ -38,19 +38,12 @@ int rxe_cq_post(struct rxe_cq *cq, struct rxe_cqe *cqe, int solicited); void rxe_cq_cleanup(struct rxe_pool_entry *arg); /* rxe_mcast.c */ -int rxe_mcast_get_grp(struct rxe_dev *rxe, union ib_gid *mgid, - struct rxe_mc_grp **grp_p); - int rxe_mcast_add_grp_elem(struct rxe_dev *rxe, struct rxe_qp *qp, - struct rxe_mc_grp *grp); - + union ib_gid *mgid); int rxe_mcast_drop_grp_elem(struct rxe_dev *rxe, struct rxe_qp *qp, union ib_gid *mgid); - void rxe_drop_all_mcast_groups(struct rxe_qp *qp); -void rxe_mc_cleanup(struct rxe_pool_entry *arg); - /* rxe_mmap.c */ struct rxe_mmap_info { struct list_head pending_mmaps; @@ -104,8 +97,6 @@ int rxe_prepare(struct rxe_pkt_info *pkt, struct sk_buff *skb); int rxe_xmit_packet(struct rxe_qp *qp, struct rxe_pkt_info *pkt, struct sk_buff *skb); const char *rxe_parent_name(struct rxe_dev *rxe, unsigned int port_num); -int rxe_mcast_add(struct rxe_dev *rxe, union ib_gid *mgid); -int rxe_mcast_delete(struct rxe_dev *rxe, union ib_gid *mgid); /* rxe_qp.c */ int rxe_qp_chk_init(struct rxe_dev *rxe, struct ib_qp_init_attr *init); diff --git a/drivers/infiniband/sw/rxe/rxe_mcast.c b/drivers/infiniband/sw/rxe/rxe_mcast.c index 337dc2c68051..685440a20669 100644 --- a/drivers/infiniband/sw/rxe/rxe_mcast.c +++ b/drivers/infiniband/sw/rxe/rxe_mcast.c @@ -5,116 +5,192 @@ */ #include "rxe.h" -#include "rxe_loc.h" -/* caller should hold mc_grp_pool->pool_lock */ -static struct rxe_mc_grp *create_grp(struct rxe_dev *rxe, - struct rxe_pool *pool, - union ib_gid *mgid) +static int rxe_mcast_add(struct rxe_dev *rxe, union ib_gid *mgid) { int err; + unsigned char ll_addr[ETH_ALEN]; + + ipv6_eth_mc_map((struct in6_addr *)mgid->raw, ll_addr); + err = dev_mc_add(rxe->ndev, ll_addr); + + return err; +} + +static int rxe_mcast_delete(struct rxe_dev *rxe, union ib_gid *mgid) +{ + int err; + unsigned char ll_addr[ETH_ALEN]; + + ipv6_eth_mc_map((struct in6_addr *)mgid->raw, ll_addr); + err = dev_mc_del(rxe->ndev, ll_addr); + + return err; +} + +static int rxe_mcast_get_grp(struct rxe_dev *rxe, union ib_gid *mgid, + struct rxe_mc_grp **grp_p) +{ + struct rxe_pool *pool = &rxe->mc_grp_pool; struct rxe_mc_grp *grp; + unsigned long flags; + int err = 0; + + /* Perform this while holding the mc_grp_pool lock + * to prevent races where two coincident calls fail to lookup the + * same group and then both create the same group. + */ + write_lock_irqsave(&pool->pool_lock, flags); + grp = rxe_pool_get_key_locked(pool, mgid); + if (grp) + goto done; grp = rxe_alloc_with_key_locked(&rxe->mc_grp_pool, mgid); - if (!grp) - return ERR_PTR(-ENOMEM); + if (!grp) { + err = -ENOMEM; + goto done; + } INIT_LIST_HEAD(&grp->qp_list); spin_lock_init(&grp->mcg_lock); grp->rxe = rxe; err = rxe_mcast_add(rxe, mgid); - if (unlikely(err)) { - rxe_drop_ref(grp); - return ERR_PTR(err); + if (err) { + rxe_fini_ref_locked(grp); + grp = NULL; + goto done; } - return grp; + /* match the reference taken by get_key */ + rxe_add_ref_locked(grp); +done: + *grp_p = grp; + write_unlock_irqrestore(&pool->pool_lock, flags); + + return err; } -int rxe_mcast_get_grp(struct rxe_dev *rxe, union ib_gid *mgid, - struct rxe_mc_grp **grp_p) +static void rxe_mcast_put_grp(struct rxe_mc_grp *grp) { - int err; - struct rxe_mc_grp *grp; + struct rxe_dev *rxe = grp->rxe; struct rxe_pool *pool = &rxe->mc_grp_pool; unsigned long flags; - if (rxe->attr.max_mcast_qp_attach == 0) - return -EINVAL; - write_lock_irqsave(&pool->pool_lock, flags); - grp = rxe_pool_get_key_locked(pool, mgid); - if (grp) - goto done; + rxe_drop_ref_locked(grp); - grp = create_grp(rxe, pool, mgid); - if (IS_ERR(grp)) { - write_unlock_irqrestore(&pool->pool_lock, flags); - err = PTR_ERR(grp); - return err; + if (rxe_read_ref(grp) == 1) { + rxe_mcast_delete(rxe, &grp->mgid); + rxe_fini_ref_locked(grp); } -done: write_unlock_irqrestore(&pool->pool_lock, flags); - *grp_p = grp; - return 0; } +/** + * rxe_mcast_add_grp_elem() - Associate a multicast address with a QP + * @rxe: the rxe device + * @qp: the QP + * @mgid: the multicast address + * + * Each multicast group can be associated with one or more QPs and + * each QP can be associated with zero or more multicast groups. + * Between each multicast group associated with a QP there is a + * rxe_mc_elem struct which has two list head structs and is joined + * both to a list of QPs on the multicast group and a list of groups + * on the QP. The elem has pointers to the group and the QP and + * takes a reference for each one. + * + * Return: 0 on success or an error on failure. + */ int rxe_mcast_add_grp_elem(struct rxe_dev *rxe, struct rxe_qp *qp, - struct rxe_mc_grp *grp) + union ib_gid *mgid) { - int err; struct rxe_mc_elem *elem; + struct rxe_mc_grp *grp; + int err; + + if (rxe->attr.max_mcast_qp_attach == 0) + return -EINVAL; + + /* takes a ref on grp if successful */ + err = rxe_mcast_get_grp(rxe, mgid, &grp); + if (err) + return err; - /* check to see of the qp is already a member of the group */ spin_lock_bh(&qp->grp_lock); spin_lock_bh(&grp->mcg_lock); + + /* check to see if the qp is already a member of the group */ list_for_each_entry(elem, &grp->qp_list, qp_list) { - if (elem->qp == qp) { - err = 0; - goto out; - } + if (elem->qp == qp) + goto drop_ref; } if (grp->num_qp >= rxe->attr.max_mcast_qp_attach) { err = -ENOMEM; - goto out; + goto drop_ref; } - elem = rxe_alloc_locked(&rxe->mc_elem_pool); - if (!elem) { + if (atomic_read(&rxe->total_mcast_qp_attach) >= + rxe->attr.max_total_mcast_qp_attach) { err = -ENOMEM; - goto out; + goto drop_ref; } - /* each qp holds a ref on the grp */ - rxe_add_ref(grp); + elem = kmalloc(sizeof(*elem), GFP_KERNEL); + if (!elem) { + err = -ENOMEM; + goto drop_ref; + } + atomic_inc(&rxe->total_mcast_qp_attach); grp->num_qp++; + rxe_add_ref(qp); elem->qp = qp; + /* still holding a ref on grp */ elem->grp = grp; list_add(&elem->qp_list, &grp->qp_list); list_add(&elem->grp_list, &qp->grp_list); - err = 0; -out: + goto done; + +drop_ref: + rxe_drop_ref(grp); + +done: spin_unlock_bh(&grp->mcg_lock); spin_unlock_bh(&qp->grp_lock); + return err; } +/** + * rxe_mcast_drop_grp_elem() - Dissociate multicast address and QP + * @rxe: the rxe device + * @qp: the QP + * @mgid: the multicast group + * + * Walk the list of group elements to find one which matches QP + * Then delete from group and qp lists and free pointers and the elem. + * Check to see if we have removed the last qp from group and delete + * it if so. + * + * Return: 0 on success else an error on failure + */ int rxe_mcast_drop_grp_elem(struct rxe_dev *rxe, struct rxe_qp *qp, union ib_gid *mgid) { - struct rxe_mc_grp *grp; struct rxe_mc_elem *elem, *tmp; + struct rxe_mc_grp *grp; + int err = 0; grp = rxe_pool_get_key(&rxe->mc_grp_pool, mgid); if (!grp) - goto err1; + return -EINVAL; spin_lock_bh(&qp->grp_lock); spin_lock_bh(&grp->mcg_lock); @@ -123,26 +199,28 @@ int rxe_mcast_drop_grp_elem(struct rxe_dev *rxe, struct rxe_qp *qp, if (elem->qp == qp) { list_del(&elem->qp_list); list_del(&elem->grp_list); + rxe_drop_ref(grp); + rxe_drop_ref(qp); grp->num_qp--; - - spin_unlock_bh(&grp->mcg_lock); - spin_unlock_bh(&qp->grp_lock); - rxe_drop_ref(elem); - rxe_drop_ref(grp); /* ref held by QP */ - rxe_drop_ref(grp); /* ref from get_key */ - return 0; + kfree(elem); + atomic_dec(&rxe->total_mcast_qp_attach); + goto done; } } + err = -EINVAL; +done: spin_unlock_bh(&grp->mcg_lock); spin_unlock_bh(&qp->grp_lock); - rxe_drop_ref(grp); /* ref from get_key */ -err1: - return -EINVAL; + + rxe_mcast_put_grp(grp); + + return err; } void rxe_drop_all_mcast_groups(struct rxe_qp *qp) { + struct rxe_dev *rxe = to_rdev(qp->ibqp.device); struct rxe_mc_grp *grp; struct rxe_mc_elem *elem; @@ -162,15 +240,10 @@ void rxe_drop_all_mcast_groups(struct rxe_qp *qp) list_del(&elem->qp_list); grp->num_qp--; spin_unlock_bh(&grp->mcg_lock); - rxe_drop_ref(grp); - rxe_drop_ref(elem); - } -} -void rxe_mc_cleanup(struct rxe_pool_entry *arg) -{ - struct rxe_mc_grp *grp = container_of(arg, typeof(*grp), pelem); - struct rxe_dev *rxe = grp->rxe; - - rxe_mcast_delete(rxe, &grp->mgid); + kfree(elem); + atomic_dec(&rxe->total_mcast_qp_attach); + rxe_drop_ref(qp); + rxe_mcast_put_grp(grp); + } } diff --git a/drivers/infiniband/sw/rxe/rxe_net.c b/drivers/infiniband/sw/rxe/rxe_net.c index 2cb810cb890a..fcdf998ec896 100644 --- a/drivers/infiniband/sw/rxe/rxe_net.c +++ b/drivers/infiniband/sw/rxe/rxe_net.c @@ -20,28 +20,6 @@ static struct rxe_recv_sockets recv_sockets; -int rxe_mcast_add(struct rxe_dev *rxe, union ib_gid *mgid) -{ - int err; - unsigned char ll_addr[ETH_ALEN]; - - ipv6_eth_mc_map((struct in6_addr *)mgid->raw, ll_addr); - err = dev_mc_add(rxe->ndev, ll_addr); - - return err; -} - -int rxe_mcast_delete(struct rxe_dev *rxe, union ib_gid *mgid) -{ - int err; - unsigned char ll_addr[ETH_ALEN]; - - ipv6_eth_mc_map((struct in6_addr *)mgid->raw, ll_addr); - err = dev_mc_del(rxe->ndev, ll_addr); - - return err; -} - static struct dst_entry *rxe_find_route4(struct net_device *ndev, struct in_addr *saddr, struct in_addr *daddr) diff --git a/drivers/infiniband/sw/rxe/rxe_pool.c b/drivers/infiniband/sw/rxe/rxe_pool.c index 4be43ae58219..2cd4d8803a0e 100644 --- a/drivers/infiniband/sw/rxe/rxe_pool.c +++ b/drivers/infiniband/sw/rxe/rxe_pool.c @@ -76,16 +76,10 @@ static const struct rxe_type_info rxe_type_info[RXE_NUM_TYPES] = { .name = "rxe-mc_grp", .size = sizeof(struct rxe_mc_grp), .elem_offset = offsetof(struct rxe_mc_grp, pelem), - .cleanup = rxe_mc_cleanup, .flags = RXE_POOL_KEY, .key_offset = offsetof(struct rxe_mc_grp, mgid), .key_size = sizeof(union ib_gid), }, - [RXE_TYPE_MC_ELEM] = { - .name = "rxe-mc_elem", - .size = sizeof(struct rxe_mc_elem), - .elem_offset = offsetof(struct rxe_mc_elem, pelem), - }, }; static int rxe_pool_init_index(struct rxe_pool *pool, u32 max, u32 min) diff --git a/drivers/infiniband/sw/rxe/rxe_pool.h b/drivers/infiniband/sw/rxe/rxe_pool.h index 7df52d34e306..071e5c557d31 100644 --- a/drivers/infiniband/sw/rxe/rxe_pool.h +++ b/drivers/infiniband/sw/rxe/rxe_pool.h @@ -26,7 +26,6 @@ enum rxe_elem_type { RXE_TYPE_MR, RXE_TYPE_MW, RXE_TYPE_MC_GRP, - RXE_TYPE_MC_ELEM, RXE_NUM_TYPES, /* keep me last */ }; diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c index 61fa633775f3..a475fc04e05b 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.c +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c @@ -986,20 +986,10 @@ static int rxe_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg, static int rxe_attach_mcast(struct ib_qp *ibqp, union ib_gid *mgid, u16 mlid) { - int err; struct rxe_dev *rxe = to_rdev(ibqp->device); struct rxe_qp *qp = to_rqp(ibqp); - struct rxe_mc_grp *grp; - - /* takes a ref on grp if successful */ - err = rxe_mcast_get_grp(rxe, mgid, &grp); - if (err) - return err; - - err = rxe_mcast_add_grp_elem(rxe, qp, grp); - rxe_drop_ref(grp); - return err; + return rxe_mcast_add_grp_elem(rxe, qp, mgid); } static int rxe_detach_mcast(struct ib_qp *ibqp, union ib_gid *mgid, u16 mlid) diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h index 35e041450090..4f1d7777f755 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.h +++ b/drivers/infiniband/sw/rxe/rxe_verbs.h @@ -403,7 +403,8 @@ struct rxe_dev { struct rxe_pool mr_pool; struct rxe_pool mw_pool; struct rxe_pool mc_grp_pool; - struct rxe_pool mc_elem_pool; + + atomic_t total_mcast_qp_attach; spinlock_t pending_lock; /* guard pending_mmaps */ struct list_head pending_mmaps; From patchwork Fri Oct 22 19:18:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12578953 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6967DC433FE for ; Fri, 22 Oct 2021 19:19:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 51B2C61040 for ; Fri, 22 Oct 2021 19:19:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233872AbhJVTVz (ORCPT ); Fri, 22 Oct 2021 15:21:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35778 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233906AbhJVTVx (ORCPT ); Fri, 22 Oct 2021 15:21:53 -0400 Received: from mail-oi1-x22c.google.com (mail-oi1-x22c.google.com [IPv6:2607:f8b0:4864:20::22c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DDBA6C061243 for ; Fri, 22 Oct 2021 12:19:35 -0700 (PDT) Received: by mail-oi1-x22c.google.com with SMTP id z126so6248469oiz.12 for ; Fri, 22 Oct 2021 12:19:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=UGGwP3wF8+ozx6BrBF/pq8Xc6cwQIjZdghUwSFwGZSU=; b=VZ9wMdmLvB35yQFvDAR4EqOZjhoZL5lwAzSTUf732gCbGk6zt6/yxX4SEhunlt42Xd MmuIQXwd8BeUPNeyP1/0BIJIXCpGkBFkpVhsopMQ1h8un9nr42gXif3DpAAZM8gG2/+L M4WOmGKVoGsmzCy1+fXFpqorg25hUYd97/aA0XUOT+LcI+aELqre6oBbQMXn7rLGcCOA mGnmKTu8TvmPm4SINwy7lCjEPTmQmHfWcSLVhBOFt8OtD0uk2mMCizcCz4ZHZBaBI50y Ybs97CePRcrf2njvvsFqtyuBRcYAJQ42oPs7e70yhkGMDko6kRk0IJ1numymgx/YDUdb WSOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=UGGwP3wF8+ozx6BrBF/pq8Xc6cwQIjZdghUwSFwGZSU=; b=ArwBnqdqsqzklqRPNMwBPEyIAyMkzwZ6Z1/dk8GHw09qvTpha3aZWxRHhdiOFv/zaL tCkHBRE+YPBV4NHnWHJ07E9h/obdISUdUYdkDGMLIAqRQRY5qn6uMoIzAhulTfkPBWkD 0UtoAOdkwqBH0lDIfKHlP4Oj+Xr3TF3OSy2r3U7938U7Pf5gXGnMGqHyOTqkWG6V8s6B pUidqn0jjan2PhxosoB7O5/AQBQNB3cX6ka+Y9kc/vHkJWBPHIIlIr86Qy090cZSQetp boC2antOTiGAteAExOafR2Q/dgKMRlW4XcwBayWXKX0uoopo+WsIDUtQQ4lgOcrG/5QO 3aGg== X-Gm-Message-State: AOAM5300EF+CDuKbZrnBHKiDN45RrNEaouCyW/Nxr3MzEuiVTAno6mHr XY80H4oipbhSnXlFO4eC7z0xMK0dbZg= X-Google-Smtp-Source: ABdhPJw1mzQ28M3qerGzLxGMxCuWb1ASHo7WE+kmsm8Y4HWTb74yOulikbykl4tZTWwjvsQhZnitzw== X-Received: by 2002:a05:6808:148e:: with SMTP id e14mr11481242oiw.172.1634930375273; Fri, 22 Oct 2021 12:19:35 -0700 (PDT) Received: from ubunto-21.tx.rr.com (2603-8081-140c-1a00-bfc7-2889-1b58-2997.res6.spectrum.com. [2603:8081:140c:1a00:bfc7:2889:1b58:2997]) by smtp.gmail.com with ESMTPSA id bf3sm2246594oib.34.2021.10.22.12.19.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 Oct 2021 12:19:34 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v3 08/10] RDMA/rxe: Fix ref error in rxe_av.c Date: Fri, 22 Oct 2021 14:18:23 -0500 Message-Id: <20211022191824.18307-9-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211022191824.18307-1-rpearsonhpe@gmail.com> References: <20211022191824.18307-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org The commit referenced below can take a reference to the AH which is never dropped. This only happens in the UD request path. This patch optionally passes that AH back to the caller so that it can hold the reference while the AV is being accessed and then drop it. Code to do this is added to rxe_req.c. The AV is also passed to rxe_prepare in rxe_net.c as an optimization. Fixes: e2fe06c90806 ("RDMA/rxe: Lookup kernel AH from ah index in UD WQEs") Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_av.c | 25 +++++++++++-- drivers/infiniband/sw/rxe/rxe_loc.h | 5 ++- drivers/infiniband/sw/rxe/rxe_net.c | 17 +++++---- drivers/infiniband/sw/rxe/rxe_req.c | 55 +++++++++++++++++----------- drivers/infiniband/sw/rxe/rxe_resp.c | 2 +- 5 files changed, 67 insertions(+), 37 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_av.c b/drivers/infiniband/sw/rxe/rxe_av.c index 5084841a7cd0..8a6910a01e66 100644 --- a/drivers/infiniband/sw/rxe/rxe_av.c +++ b/drivers/infiniband/sw/rxe/rxe_av.c @@ -99,11 +99,14 @@ void rxe_av_fill_ip_info(struct rxe_av *av, struct rdma_ah_attr *attr) av->network_type = type; } -struct rxe_av *rxe_get_av(struct rxe_pkt_info *pkt) +struct rxe_av *rxe_get_av(struct rxe_pkt_info *pkt, struct rxe_ah **ahp) { struct rxe_ah *ah; u32 ah_num; + if (ahp) + *ahp = NULL; + if (!pkt || !pkt->qp) return NULL; @@ -117,11 +120,25 @@ struct rxe_av *rxe_get_av(struct rxe_pkt_info *pkt) if (ah_num) { /* only new user provider or kernel client */ ah = rxe_pool_get_index(&pkt->rxe->ah_pool, ah_num); - rxe_drop_ref(ah); - if (!ah || ah->ah_num != ah_num || rxe_ah_pd(ah) != pkt->qp->pd) { - pr_warn("Unable to find AH matching ah_num\n"); + if (!ah) { + pr_warn("%s: Unable to find AH matching ah_num\n", + __func__); + return NULL; + } + + if (rxe_ah_pd(ah) != pkt->qp->pd) { + pr_warn("%s: PDs don't match for AH and QP\n", + __func__); + rxe_drop_ref(ah); return NULL; } + + /* let caller hold ref to ah */ + if (ahp) + *ahp = ah; + else + rxe_drop_ref(ah); + return &ah->av; } diff --git a/drivers/infiniband/sw/rxe/rxe_loc.h b/drivers/infiniband/sw/rxe/rxe_loc.h index 78312df8eea3..a689ee8386b8 100644 --- a/drivers/infiniband/sw/rxe/rxe_loc.h +++ b/drivers/infiniband/sw/rxe/rxe_loc.h @@ -19,7 +19,7 @@ void rxe_av_to_attr(struct rxe_av *av, struct rdma_ah_attr *attr); void rxe_av_fill_ip_info(struct rxe_av *av, struct rdma_ah_attr *attr); -struct rxe_av *rxe_get_av(struct rxe_pkt_info *pkt); +struct rxe_av *rxe_get_av(struct rxe_pkt_info *pkt, struct rxe_ah **ahp); /* rxe_cq.c */ int rxe_cq_chk_attr(struct rxe_dev *rxe, struct rxe_cq *cq, @@ -93,7 +93,8 @@ void rxe_mw_cleanup(struct rxe_pool_entry *arg); /* rxe_net.c */ struct sk_buff *rxe_init_packet(struct rxe_dev *rxe, struct rxe_av *av, int paylen, struct rxe_pkt_info *pkt); -int rxe_prepare(struct rxe_pkt_info *pkt, struct sk_buff *skb); +int rxe_prepare(struct rxe_av *av, struct rxe_pkt_info *pkt, + struct sk_buff *skb); int rxe_xmit_packet(struct rxe_qp *qp, struct rxe_pkt_info *pkt, struct sk_buff *skb); const char *rxe_parent_name(struct rxe_dev *rxe, unsigned int port_num); diff --git a/drivers/infiniband/sw/rxe/rxe_net.c b/drivers/infiniband/sw/rxe/rxe_net.c index fcdf998ec896..996aabd6e57b 100644 --- a/drivers/infiniband/sw/rxe/rxe_net.c +++ b/drivers/infiniband/sw/rxe/rxe_net.c @@ -271,13 +271,13 @@ static void prepare_ipv6_hdr(struct dst_entry *dst, struct sk_buff *skb, ip6h->payload_len = htons(skb->len - sizeof(*ip6h)); } -static int prepare4(struct rxe_pkt_info *pkt, struct sk_buff *skb) +static int prepare4(struct rxe_av *av, struct rxe_pkt_info *pkt, + struct sk_buff *skb) { struct rxe_qp *qp = pkt->qp; struct dst_entry *dst; bool xnet = false; __be16 df = htons(IP_DF); - struct rxe_av *av = rxe_get_av(pkt); struct in_addr *saddr = &av->sgid_addr._sockaddr_in.sin_addr; struct in_addr *daddr = &av->dgid_addr._sockaddr_in.sin_addr; @@ -297,11 +297,11 @@ static int prepare4(struct rxe_pkt_info *pkt, struct sk_buff *skb) return 0; } -static int prepare6(struct rxe_pkt_info *pkt, struct sk_buff *skb) +static int prepare6(struct rxe_av *av, struct rxe_pkt_info *pkt, + struct sk_buff *skb) { struct rxe_qp *qp = pkt->qp; struct dst_entry *dst; - struct rxe_av *av = rxe_get_av(pkt); struct in6_addr *saddr = &av->sgid_addr._sockaddr_in6.sin6_addr; struct in6_addr *daddr = &av->dgid_addr._sockaddr_in6.sin6_addr; @@ -322,16 +322,17 @@ static int prepare6(struct rxe_pkt_info *pkt, struct sk_buff *skb) return 0; } -int rxe_prepare(struct rxe_pkt_info *pkt, struct sk_buff *skb) +int rxe_prepare(struct rxe_av *av, struct rxe_pkt_info *pkt, + struct sk_buff *skb) { int err = 0; if (skb->protocol == htons(ETH_P_IP)) - err = prepare4(pkt, skb); + err = prepare4(av, pkt, skb); else if (skb->protocol == htons(ETH_P_IPV6)) - err = prepare6(pkt, skb); + err = prepare6(av, pkt, skb); - if (ether_addr_equal(skb->dev->dev_addr, rxe_get_av(pkt)->dmac)) + if (ether_addr_equal(skb->dev->dev_addr, av->dmac)) pkt->mask |= RXE_LOOPBACK_MASK; return err; diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rxe/rxe_req.c index 0c9d2af15f3d..891cf98c74a0 100644 --- a/drivers/infiniband/sw/rxe/rxe_req.c +++ b/drivers/infiniband/sw/rxe/rxe_req.c @@ -361,6 +361,7 @@ static inline int get_mtu(struct rxe_qp *qp) } static struct sk_buff *init_req_packet(struct rxe_qp *qp, + struct rxe_av *av, struct rxe_send_wqe *wqe, int opcode, int payload, struct rxe_pkt_info *pkt) @@ -368,7 +369,6 @@ static struct sk_buff *init_req_packet(struct rxe_qp *qp, struct rxe_dev *rxe = to_rdev(qp->ibqp.device); struct sk_buff *skb; struct rxe_send_wr *ibwr = &wqe->wr; - struct rxe_av *av; int pad = (-payload) & 0x3; int paylen; int solicited; @@ -378,21 +378,9 @@ static struct sk_buff *init_req_packet(struct rxe_qp *qp, /* length from start of bth to end of icrc */ paylen = rxe_opcode[opcode].length + payload + pad + RXE_ICRC_SIZE; - - /* pkt->hdr, port_num and mask are initialized in ifc layer */ - pkt->rxe = rxe; - pkt->opcode = opcode; - pkt->qp = qp; - pkt->psn = qp->req.psn; - pkt->mask = rxe_opcode[opcode].mask; - pkt->paylen = paylen; - pkt->wqe = wqe; + pkt->paylen = paylen; /* init skb */ - av = rxe_get_av(pkt); - if (!av) - return NULL; - skb = rxe_init_packet(rxe, av, paylen, pkt); if (unlikely(!skb)) return NULL; @@ -453,13 +441,13 @@ static struct sk_buff *init_req_packet(struct rxe_qp *qp, return skb; } -static int finish_packet(struct rxe_qp *qp, struct rxe_send_wqe *wqe, - struct rxe_pkt_info *pkt, struct sk_buff *skb, - int paylen) +static int finish_packet(struct rxe_qp *qp, struct rxe_av *av, + struct rxe_send_wqe *wqe, struct rxe_pkt_info *pkt, + struct sk_buff *skb, int paylen) { int err; - err = rxe_prepare(pkt, skb); + err = rxe_prepare(av, pkt, skb); if (err) return err; @@ -614,6 +602,7 @@ static int rxe_do_local_ops(struct rxe_qp *qp, struct rxe_send_wqe *wqe) int rxe_requester(void *arg) { struct rxe_qp *qp = (struct rxe_qp *)arg; + struct rxe_dev *rxe = to_rdev(qp->ibqp.device); struct rxe_pkt_info pkt; struct sk_buff *skb; struct rxe_send_wqe *wqe; @@ -625,6 +614,8 @@ int rxe_requester(void *arg) struct rxe_send_wqe rollback_wqe; u32 rollback_psn; struct rxe_queue *q = qp->sq.queue; + struct rxe_ah *ah; + struct rxe_av *av; rxe_add_ref(qp); @@ -711,14 +702,28 @@ int rxe_requester(void *arg) payload = mtu; } - skb = init_req_packet(qp, wqe, opcode, payload, &pkt); + pkt.rxe = rxe; + pkt.opcode = opcode; + pkt.qp = qp; + pkt.psn = qp->req.psn; + pkt.mask = rxe_opcode[opcode].mask; + pkt.wqe = wqe; + + av = rxe_get_av(&pkt, &ah); + if (unlikely(!av)) { + pr_err("qp#%d Failed no address vector\n", qp_num(qp)); + wqe->status = IB_WC_LOC_QP_OP_ERR; + goto err_drop_ah; + } + + skb = init_req_packet(qp, av, wqe, opcode, payload, &pkt); if (unlikely(!skb)) { pr_err("qp#%d Failed allocating skb\n", qp_num(qp)); wqe->status = IB_WC_LOC_QP_OP_ERR; - goto err; + goto err_drop_ah; } - ret = finish_packet(qp, wqe, &pkt, skb, payload); + ret = finish_packet(qp, av, wqe, &pkt, skb, payload); if (unlikely(ret)) { pr_debug("qp#%d Error during finish packet\n", qp_num(qp)); if (ret == -EFAULT) @@ -726,9 +731,12 @@ int rxe_requester(void *arg) else wqe->status = IB_WC_LOC_QP_OP_ERR; kfree_skb(skb); - goto err; + goto err_drop_ah; } + if (ah) + rxe_drop_ref(ah); + /* * To prevent a race on wqe access between requester and completer, * wqe members state and psn need to be set before calling @@ -757,6 +765,9 @@ int rxe_requester(void *arg) goto next_wqe; +err_drop_ah: + if (ah) + rxe_drop_ref(ah); err: wqe->state = wqe_state_error; __rxe_do_task(&qp->comp.task); diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c index e8f435fa6e4d..f589f4dde35c 100644 --- a/drivers/infiniband/sw/rxe/rxe_resp.c +++ b/drivers/infiniband/sw/rxe/rxe_resp.c @@ -632,7 +632,7 @@ static struct sk_buff *prepare_ack_packet(struct rxe_qp *qp, if (ack->mask & RXE_ATMACK_MASK) atmack_set_orig(ack, qp->resp.atomic_orig); - err = rxe_prepare(ack, skb); + err = rxe_prepare(&qp->pri_av, ack, skb); if (err) { kfree_skb(skb); return NULL; From patchwork Fri Oct 22 19:18:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12578957 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 02185C433F5 for ; Fri, 22 Oct 2021 19:19:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DDB2A60E95 for ; Fri, 22 Oct 2021 19:19:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233968AbhJVTVz (ORCPT ); Fri, 22 Oct 2021 15:21:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35786 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233971AbhJVTVy (ORCPT ); Fri, 22 Oct 2021 15:21:54 -0400 Received: from mail-ot1-x329.google.com (mail-ot1-x329.google.com [IPv6:2607:f8b0:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 570BAC061225 for ; Fri, 22 Oct 2021 12:19:36 -0700 (PDT) Received: by mail-ot1-x329.google.com with SMTP id d21-20020a9d4f15000000b0054e677e0ac5so5546856otl.11 for ; Fri, 22 Oct 2021 12:19:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=CnyjJEEj7ZUaSs283Gd6KN+gvimKgfk3zjicV3+PNmA=; b=ZP5YDrtlwXGsw4vjC57Q70hZe2SrJ3IqkUcSL751EP+ZLH9f2LjgZIigY6RWwErt9Y R2JVig0qlPDJ9C6IYKVb0k9sZdv+O91ooqwATpV9vNZ2jIHWacrnmSfOfawP0NrAstmw F4aP6xgAFz2JwTeaz1maPcNEWFErAaO7Dt1VcVAyB6nnICen3jpBltmuhXRqVOxdI5On eftbnQy8IF0B0ZPD2mFjTFQ3HDKguBnCnrnzn/qXam+iZCe9lP0ywE+mObT4u+XU1frI 5Svblf1Y9W41ra/p+DQSQjE/D5Ew86e41L2UszLAG97+xKzKOR6I7LcIP9rRy+NOaOVI UhXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=CnyjJEEj7ZUaSs283Gd6KN+gvimKgfk3zjicV3+PNmA=; b=RUYDao6rxNXJ3nh8Jt3tuqscs/xqkpeijBfTMPLx9i7FYIa0OqUJl/FpUMS4CkZSX+ pIJy6bAA6YwvuZryADKatFmGwAAw2RkSCbAYonnfJ5pf8n7gcV9/FB8T+7miRxbQ7gwZ BnME3MZcGxwbq/lMUpPwZyKj8O09l7Yc+199M2FYF5CvXTDLDQtCUeo3kl9MRU1NjzuE pYlEzjBXvPJAWuDs4oxnhR6Q4ef6RHqIBPbknake0mx7Lgt+RlyEQEMH+DgNeUNt9n21 wnmSwZ0CpGElYh4rlTkKymbs6tQIizpZEdAXCBAkFHBZyCZ+rGhxZG+SW1JrMcv3Hiq2 FSXw== X-Gm-Message-State: AOAM5304bnMHcOWZWkiyX8k+oT+AcpoKB48ZdSG6CWxwxmfruu9m0TzR ayuiCUmzkQcjoaB09apLp5I= X-Google-Smtp-Source: ABdhPJy7wIj2yNQdv89oph6txnQzGsR8gn4NXUhCfMD92lfakt39kksb3xQ0MoVbi2jbh+AvSURxlA== X-Received: by 2002:a05:6830:4c8:: with SMTP id s8mr1427723otd.359.1634930375743; Fri, 22 Oct 2021 12:19:35 -0700 (PDT) Received: from ubunto-21.tx.rr.com (2603-8081-140c-1a00-bfc7-2889-1b58-2997.res6.spectrum.com. [2603:8081:140c:1a00:bfc7:2889:1b58:2997]) by smtp.gmail.com with ESMTPSA id bf3sm2246594oib.34.2021.10.22.12.19.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 Oct 2021 12:19:35 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v3 09/10] RDMA/rxe: Replace mr by rkey in responder resources Date: Fri, 22 Oct 2021 14:18:24 -0500 Message-Id: <20211022191824.18307-10-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211022191824.18307-1-rpearsonhpe@gmail.com> References: <20211022191824.18307-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Currently rxe saves a copy of MR in responder resources for RDMA reads. Since the responder resources are never freed just over written if more are needed this MR may not have a reference freed until the QP is destroyed. This patch uses the rkey instead of the MR and on subsequent packets of a multipacket read reply message it looks up the MR from the rkey for each packet. This makes it possible for a user to deregister an MR or unbind a MW on the fly and get correct behaviour. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_qp.c | 10 +-- drivers/infiniband/sw/rxe/rxe_resp.c | 123 ++++++++++++++++++-------- drivers/infiniband/sw/rxe/rxe_verbs.h | 1 - 3 files changed, 87 insertions(+), 47 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_qp.c b/drivers/infiniband/sw/rxe/rxe_qp.c index 975321812c87..5b9273b171af 100644 --- a/drivers/infiniband/sw/rxe/rxe_qp.c +++ b/drivers/infiniband/sw/rxe/rxe_qp.c @@ -135,12 +135,8 @@ static void free_rd_atomic_resources(struct rxe_qp *qp) void free_rd_atomic_resource(struct rxe_qp *qp, struct resp_res *res) { - if (res->type == RXE_ATOMIC_MASK) { + if (res->type == RXE_ATOMIC_MASK) kfree_skb(res->atomic.skb); - } else if (res->type == RXE_READ_MASK) { - if (res->read.mr) - rxe_drop_ref(res->read.mr); - } res->type = 0; } @@ -816,10 +812,8 @@ static void rxe_qp_do_cleanup(struct work_struct *work) if (qp->pd) rxe_drop_ref(qp->pd); - if (qp->resp.mr) { + if (qp->resp.mr) rxe_drop_ref(qp->resp.mr); - qp->resp.mr = NULL; - } if (qp_type(qp) == IB_QPT_RC) sk_dst_reset(qp->sk->sk); diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c index f589f4dde35c..c776289842e5 100644 --- a/drivers/infiniband/sw/rxe/rxe_resp.c +++ b/drivers/infiniband/sw/rxe/rxe_resp.c @@ -641,6 +641,78 @@ static struct sk_buff *prepare_ack_packet(struct rxe_qp *qp, return skb; } +static struct resp_res *rxe_prepare_read_res(struct rxe_qp *qp, + struct rxe_pkt_info *pkt) +{ + struct resp_res *res; + u32 pkts; + + res = &qp->resp.resources[qp->resp.res_head]; + rxe_advance_resp_resource(qp); + free_rd_atomic_resource(qp, res); + + res->type = RXE_READ_MASK; + res->replay = 0; + res->read.va = qp->resp.va + qp->resp.offset; + res->read.va_org = qp->resp.va + qp->resp.offset; + res->read.resid = qp->resp.resid; + res->read.length = qp->resp.resid; + res->read.rkey = qp->resp.rkey; + + pkts = max_t(u32, (reth_len(pkt) + qp->mtu - 1)/qp->mtu, 1); + res->first_psn = pkt->psn; + res->cur_psn = pkt->psn; + res->last_psn = (pkt->psn + pkts - 1) & BTH_PSN_MASK; + + res->state = rdatm_res_state_new; + + return res; +} + +/** + * rxe_recheck_mr - revalidate MR from rkey and get a reference + * @qp: the qp + * @rkey: the rkey + * + * This code allows the MR to be invalidated or deregistered or + * the MW if one was used to be invalidated or deallocated. + * It is assumed that the access permissions if originally good + * are OK and the mappings to be unchanged. + * + * Return: mr on success else NULL + */ +static struct rxe_mr *rxe_recheck_mr(struct rxe_qp *qp, u32 rkey) +{ + struct rxe_dev *rxe = to_rdev(qp->ibqp.device); + struct rxe_mr *mr; + struct rxe_mw *mw; + + if (rkey_is_mw(rkey)) { + mw = rxe_pool_get_index(&rxe->mw_pool, rkey >> 8); + if (!mw || mw->rkey != rkey) + return NULL; + + if (mw->state != RXE_MW_STATE_VALID) { + rxe_drop_ref(mw); + return NULL; + } + + mr = mw->mr; + rxe_drop_ref(mw); + } else { + mr = rxe_pool_get_index(&rxe->mr_pool, rkey >> 8); + if (!mr || mr->rkey != rkey) + return NULL; + } + + if (mr->state != RXE_MR_STATE_VALID) { + rxe_drop_ref(mr); + return NULL; + } + + return mr; +} + /* RDMA read response. If res is not NULL, then we have a current RDMA request * being processed or replayed. */ @@ -655,53 +727,26 @@ static enum resp_states read_reply(struct rxe_qp *qp, int opcode; int err; struct resp_res *res = qp->resp.res; + struct rxe_mr *mr; if (!res) { - /* This is the first time we process that request. Get a - * resource - */ - res = &qp->resp.resources[qp->resp.res_head]; - - free_rd_atomic_resource(qp, res); - rxe_advance_resp_resource(qp); - - res->type = RXE_READ_MASK; - res->replay = 0; - - res->read.va = qp->resp.va + - qp->resp.offset; - res->read.va_org = qp->resp.va + - qp->resp.offset; - - res->first_psn = req_pkt->psn; - - if (reth_len(req_pkt)) { - res->last_psn = (req_pkt->psn + - (reth_len(req_pkt) + mtu - 1) / - mtu - 1) & BTH_PSN_MASK; - } else { - res->last_psn = res->first_psn; - } - res->cur_psn = req_pkt->psn; - - res->read.resid = qp->resp.resid; - res->read.length = qp->resp.resid; - res->read.rkey = qp->resp.rkey; - - /* note res inherits the reference to mr from qp */ - res->read.mr = qp->resp.mr; - qp->resp.mr = NULL; - - qp->resp.res = res; - res->state = rdatm_res_state_new; + res = rxe_prepare_read_res(qp, req_pkt); + qp->resp.res = res; } if (res->state == rdatm_res_state_new) { + mr = qp->resp.mr; + qp->resp.mr = NULL; + if (res->read.resid <= mtu) opcode = IB_OPCODE_RC_RDMA_READ_RESPONSE_ONLY; else opcode = IB_OPCODE_RC_RDMA_READ_RESPONSE_FIRST; } else { + mr = rxe_recheck_mr(qp, res->read.rkey); + if (!mr) + return RESPST_ERR_RKEY_VIOLATION; + if (res->read.resid > mtu) opcode = IB_OPCODE_RC_RDMA_READ_RESPONSE_MIDDLE; else @@ -717,10 +762,12 @@ static enum resp_states read_reply(struct rxe_qp *qp, if (!skb) return RESPST_ERR_RNR; - err = rxe_mr_copy(res->read.mr, res->read.va, payload_addr(&ack_pkt), + err = rxe_mr_copy(mr, res->read.va, payload_addr(&ack_pkt), payload, RXE_FROM_MR_OBJ); if (err) pr_err("Failed copying memory\n"); + if (mr) + rxe_drop_ref(mr); if (bth_pad(&ack_pkt)) { u8 *pad = payload_addr(&ack_pkt) + payload; diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h index 4f1d7777f755..0cfbef7a36c9 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.h +++ b/drivers/infiniband/sw/rxe/rxe_verbs.h @@ -157,7 +157,6 @@ struct resp_res { struct sk_buff *skb; } atomic; struct { - struct rxe_mr *mr; u64 va_org; u32 rkey; u32 length; From patchwork Fri Oct 22 19:18:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12578955 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88B34C4332F for ; Fri, 22 Oct 2021 19:19:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7242560E95 for ; Fri, 22 Oct 2021 19:19:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233971AbhJVTV4 (ORCPT ); Fri, 22 Oct 2021 15:21:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35782 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234017AbhJVTVy (ORCPT ); Fri, 22 Oct 2021 15:21:54 -0400 Received: from mail-ot1-x333.google.com (mail-ot1-x333.google.com [IPv6:2607:f8b0:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BE9E9C061227 for ; Fri, 22 Oct 2021 12:19:36 -0700 (PDT) Received: by mail-ot1-x333.google.com with SMTP id s18-20020a0568301e1200b0054e77a16651so5600086otr.7 for ; Fri, 22 Oct 2021 12:19:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=kmD6gv8SJzsbE3q8rg5qYGeTx7mtLGZwbKFDlrgFjiQ=; b=Vhx/o1T0eBwQfQ9C59SQEeLAshQkohP7CfSV2Nd2qre9WIw93PV9Xc+PZd9DdLgrgm xjnE90LKFqq8mHeSBimgDjnt3C/BKmwbibhX2HljxMjXe6hWoNWNmgbzWJiK0LIsL/kN IMe08t60nS+Ekd/l+a6Ihu4QKHd4B+aI1nPcZGAAWIJRF51ev8UCepDIFGmd3ek8wqd7 mctOubUhPm9dKsZHMY2LJ6l6VgdNBJ0nuSmUi7vM5gJZvsoFdfwpT9+M1fHPTVBBX0Im 5x/M4AzVYQ9JE3q6HF8p9CvpzEvTSARJzvPzJVCPgrvxJmYukUyGvUb0WPD7t3gMxnGK T6HQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kmD6gv8SJzsbE3q8rg5qYGeTx7mtLGZwbKFDlrgFjiQ=; b=CwWM4D34JzVc/x9RqCrxKwoyqNn85MNIHLznkrBkb7gvKsE8fKyXHilShIaS/NMejQ pUDut13856juv1d9UapNivHin5SFkhgZrEOFz/RMfVs17G9CgND9cpIIUwr3dfmTyS7a 0BmXfSC952vRP2UfPMrt5I9kdboiA2gTJ50ytp1tNi80fdNpCP+h7PhM8H0a5oP79unD TjuJstxUWM/cbTkLFfJyAMjH3ihiNMvvkw3MtAyUMlmQLerOslYuV42neEx3fy/O1nh8 Rb4hjHhbcCfEbxfT4joMVt/ivs4ynzy+LCe/w/sR9+AJ5ykhLUam+9IC42pDrBJKiNWc y2nQ== X-Gm-Message-State: AOAM533OVXn1Swh7I7/AjPdEsl/ibIe6I/iTUGKvzcsBdXY+M2Q0x4NE 8ipUv82X2V7iHUSphgDY13yhpNwCaoo= X-Google-Smtp-Source: ABdhPJxGCRK8k/7CKqNpIjiWLn+9LvaR+Ixsislt+aAGCKmJwdNytVMxF75StP3GkEuEAzy91UVbHw== X-Received: by 2002:a9d:4b8d:: with SMTP id k13mr1358942otf.103.1634930376229; Fri, 22 Oct 2021 12:19:36 -0700 (PDT) Received: from ubunto-21.tx.rr.com (2603-8081-140c-1a00-bfc7-2889-1b58-2997.res6.spectrum.com. [2603:8081:140c:1a00:bfc7:2889:1b58:2997]) by smtp.gmail.com with ESMTPSA id bf3sm2246594oib.34.2021.10.22.12.19.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 Oct 2021 12:19:36 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v3 10/10] RDMA/rxe: Minor cleanup in rxe_pool.c Date: Fri, 22 Oct 2021 14:18:25 -0500 Message-Id: <20211022191824.18307-11-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211022191824.18307-1-rpearsonhpe@gmail.com> References: <20211022191824.18307-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Change pr_warn() to pr_debug() Check if RXE_POOL_INDEX in rxe_pool_cleanup() Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_pool.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_pool.c b/drivers/infiniband/sw/rxe/rxe_pool.c index 2cd4d8803a0e..82efc4626bf6 100644 --- a/drivers/infiniband/sw/rxe/rxe_pool.c +++ b/drivers/infiniband/sw/rxe/rxe_pool.c @@ -88,7 +88,7 @@ static int rxe_pool_init_index(struct rxe_pool *pool, u32 max, u32 min) size_t size; if ((max - min + 1) < pool->max_elem) { - pr_warn("not enough indices for max_elem\n"); + pr_debug("not enough indices for max_elem\n"); err = -EINVAL; goto out; } @@ -155,10 +155,11 @@ int rxe_pool_init( void rxe_pool_cleanup(struct rxe_pool *pool) { if (atomic_read(&pool->num_elem) > 0) - pr_warn("%s pool destroyed with unfree'd elem\n", + pr_debug("%s pool destroyed with unfree'd elem\n", pool->name); - kfree(pool->index.table); + if (pool->flags & RXE_POOL_INDEX) + kfree(pool->index.table); } /* should never fail because there are at least as many indices as @@ -194,7 +195,7 @@ static int rxe_insert_index(struct rxe_pool *pool, struct rxe_pool_entry *new) * old object was not deleted from the pool index */ if (unlikely(elem == new || elem->index == new->index)) { - pr_warn("%s#%d rf=%d: already in pool\n", + pr_debug("%s#%d(%d): already in pool\n", pool->name, new->index, refcount_read(&new->refcnt)); return -EINVAL; @@ -228,7 +229,9 @@ static int rxe_insert_key(struct rxe_pool *pool, struct rxe_pool_entry *new) pool->key.key_size); if (cmp == 0) { - pr_warn("key already exists!\n"); + pr_debug("%s#%d(%d): key already in pool\n", + pool->name, new->index, + refcount_read(&new->refcnt)); return -EINVAL; }