From patchwork Wed Aug 21 11:22:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Haris Iqbal X-Patchwork-Id: 13771248 Received: from mail-ej1-f43.google.com (mail-ej1-f43.google.com [209.85.218.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 45A5D1B2EC4 for ; Wed, 21 Aug 2024 11:23:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724239401; cv=none; b=ewmtztCiiPbfuD6m0dXuL2yZIPEcyY4PDL+Bfv6vCHb1SlzmiAR83M46EWM70PgitFZt8jC4dBar3q9YPHBsYtZ02tP8srStK8sDddl2OqmKf2m5SHnZywtALjYXTdltaUh1nETByIGrksxrAi6eP28bpKDB6jZ3czFNWRgb+b4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724239401; c=relaxed/simple; bh=/cLqBgzuz+7z5BY2d9XvPUSzLLAqeRO8SPA9GHWbjCM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=AV8aHXj07tmRB0f2MPUndy6wPy1/u6NZtGbr4J3x2+IMpMRILdM71NGVUd5WLTMBOuYsbc5Ip3O7AFtRvavItwmxYiX1hBuZke4Q8jXJOGxwH7BPORYFGL/OvM8Z4Yk8+NcoArhM8uqHo9RnwcrfNaT5DdjtpL4XxVEjYAT6rxQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ionos.com; spf=pass smtp.mailfrom=ionos.com; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b=hQJjiGC3; arc=none smtp.client-ip=209.85.218.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ionos.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ionos.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b="hQJjiGC3" Received: by mail-ej1-f43.google.com with SMTP id a640c23a62f3a-a8679f534c3so52577266b.0 for ; Wed, 21 Aug 2024 04:23:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; t=1724239397; x=1724844197; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hoj06T1lLnrz7lemwh9MfKf9lHfPP+xBU9RJw8REbTc=; b=hQJjiGC3S9GRNvWkrPyuiFncdian6RGEgW6oEcq9NJ/yugdenjSqUDrBrk9L77U8/I YYZ+ZNwg5lItMjXlZ1Ugkaw3yTjxW5ULoRoMt/A33walTy44LxBj8jS3+SyuMrcS0bhh OaCjt38QvrrAGMJHlsrxLF5bJeTmikIjEu147jZpiYKg7xxCyU2DbdGOXZ7wQi1jMonJ JA3wBva0RoB5DFnzMqUaoy0xc1daTQD705SGIsUUHMGQmCgsDQ29jXZP9p5vXGVYBs6j DTqts88HcZobc+B45oPPD0vIFpEI432roA8765yfHIKE7MTntawoeEW1hnApCauwOKkd h+VA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724239397; x=1724844197; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hoj06T1lLnrz7lemwh9MfKf9lHfPP+xBU9RJw8REbTc=; b=PCmnVOBoW0cY9pkn93icqq4nD8aZHTV5t9UT9FSe91nL7Lp32WR2IHV4onTvcltjEm JznnNw0WKQZPjDISvGC+JBwcuYxJSUx33bMfbVHitKPAfLj76eoQmyouXDfLfuTkTw0a nICg8TeiET4uErbbndkV3DPrUOkfHQmmvJXy1/LnV15gRAweixVaAUkonVGPJylRrJMR +AuNl2cKFMWpm+ROQwnuHKllftGw3RxDBEazAPase/E1uyUKH2whxp1X7suzZlVlJfsD SFrkoqn/bd2+qsDRmisn2o/+vjgT1a71kEyONLjaTgLHGHa8eoe8ia1fM+5FG8qeE1HT pF5A== X-Gm-Message-State: AOJu0Yw8vNXbCIKLlO7dvBmHvmBHWvAl/JkKNxP2EbMXJXlQ9kT/nABo OoccNNOXl2DfTe13Q0PkRsMHscWCi6E+JamjbWJu1J0M5D5IYkFi8iCEE3KYsoBwN3e1UEUneLR ACGM= X-Google-Smtp-Source: AGHT+IFb9xoBpPiWxcZPQwvzY1vvNaFDW2JUcaxPfsLySIEhkbJ0plPU/Nub4R0zMJuJX1/+r1cKHw== X-Received: by 2002:a17:906:c106:b0:a7a:be06:d8dc with SMTP id a640c23a62f3a-a866f893f11mr136936066b.48.1724239396898; Wed, 21 Aug 2024 04:23:16 -0700 (PDT) Received: from lb01533.speedport.ip (p200300f00f051d5f269a60e7b8956185.dip0.t-ipconnect.de. [2003:f0:f05:1d5f:269a:60e7:b895:6185]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a83838c65e7sm887934066b.20.2024.08.21.04.23.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 04:23:16 -0700 (PDT) From: Md Haris Iqbal To: linux-rdma@vger.kernel.org Cc: leon@kernel.org, jgg@ziepe.ca, haris.iqbal@ionos.com, jinpu.wang@ionos.com, Grzegorz Prajsner Subject: [PATCH v2 for-next 01/11] RDMA/rtrs: For HB error add additional clt/srv specific logging Date: Wed, 21 Aug 2024 13:22:07 +0200 Message-Id: <20240821112217.41827-2-haris.iqbal@ionos.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240821112217.41827-1-haris.iqbal@ionos.com> References: <20240821112217.41827-1-haris.iqbal@ionos.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 In case of HB error, we need to know the specific path on which it happened, for better debugging. Since the clt/srv path structures are not available in rtrs.c, it needs to be done in the individual HB error handler. This commit add those loging. A sample kernel log output after this commit: rtrs_core L357: : HB missed max reached. rtrs_server L717: : HB err handler for path=ip:x.x.x.x@ip:x.x.x.x . . rtrs_core L357: : HB missed max reached. rtrs_client L1519: : HB err handler for path=ip:x.x.x.x@ip:x.x.x.x Signed-off-by: Md Haris Iqbal Reviewed-by: Jack Wang Signed-off-by: Grzegorz Prajsner --- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 2 ++ drivers/infiniband/ulp/rtrs/rtrs-srv.c | 4 ++++ 2 files changed, 6 insertions(+) diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c index 88106cf5ce55..66ac4dba990f 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c @@ -1494,7 +1494,9 @@ static bool rtrs_clt_change_state_get_old(struct rtrs_clt_path *clt_path, static void rtrs_clt_hb_err_handler(struct rtrs_con *c) { struct rtrs_clt_con *con = container_of(c, typeof(*con), c); + struct rtrs_clt_path *clt_path = to_clt_path(con->c.path); + rtrs_err(con->c.path, "HB err handler for path=%s\n", kobject_name(&clt_path->kobj)); rtrs_rdma_error_recovery(con); } diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c index 1d33efb8fb03..f76d483c3784 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c @@ -672,6 +672,10 @@ static int map_cont_bufs(struct rtrs_srv_path *srv_path) static void rtrs_srv_hb_err_handler(struct rtrs_con *c) { + struct rtrs_srv_con *con = container_of(c, typeof(*con), c); + struct rtrs_srv_path *srv_path = to_srv_path(con->c.path); + + rtrs_err(con->c.path, "HB err handler for path=%s\n", kobject_name(&srv_path->kobj)); close_path(to_srv_path(c->path)); } From patchwork Wed Aug 21 11:22:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Haris Iqbal X-Patchwork-Id: 13771249 Received: from mail-ej1-f44.google.com (mail-ej1-f44.google.com [209.85.218.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D96A81A284D for ; Wed, 21 Aug 2024 11:23:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724239402; cv=none; b=M5sGl/f8/PbC23WOlBxD+x9jtfzP8EAm9k7IIdDD9w9NX3sKUXi6VGv6asQSJ1WvYoa8PZymhm5zo94Z1HoPoqePBfNulNcgYm+rKcWRcMkwoogDwpM+PpY5357GLXzuEU8aY4OZoRq0F0HzoGpvZNnPifp7OXZyP+31fbCvDZs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724239402; c=relaxed/simple; bh=aPdeol5t6wwSSHxVfJDH5JF9MYqzC9pPQihHi5HNRHY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Q7lO46Wv8/ecGjJ1y2lxJSDE44sokl8hTSbQcJL3Ti3TA+lGK7nyRfpi1Iym2eVl0ismvXwnQSAlP1Oq2m77SPM3CaEHir04d/6WFO9jKiU6a6cxgDzgMxyBeqHyD+nkwSD99LV97RK78MxmUBZ6qxrvsIxjw1WAXwmx4syBUCI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ionos.com; spf=pass smtp.mailfrom=ionos.com; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b=MVHtzwWJ; arc=none smtp.client-ip=209.85.218.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ionos.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ionos.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b="MVHtzwWJ" Received: by mail-ej1-f44.google.com with SMTP id a640c23a62f3a-a7d2a9a23d9so762913666b.3 for ; Wed, 21 Aug 2024 04:23:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; t=1724239398; x=1724844198; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4qJ3V7YJGTaGdVFsVivjHiBFvmyNekMesA0jctJoY0g=; b=MVHtzwWJ2QrpQ0kwR2Ll5f5O5B6SP9LRpM4+xUAuBH9Ohh+ofcqrUg9giHEp/TxqUs YTOFIGDgto4YkJznqt4X2sMX6iqOb08wobMXiP6yxFhvyeMi8KIgGW088f2gMqJuv4mk FWkxJedPHYpFKb+Ts2VryoI35LdfIB1ggHtHAKoGxZvkNooXncIQ6TJbA2+FKssEP3fk pYm1Bph7cP1DJKNJ5bXebw/c6DH9pk9mYBJzTL54zDPB8z/O5e588s76gGCjCggtZ+uh +oC/bTaQ8S3OG7RWeogHtDdtCoYaGLRdI/Rfy1JC3x/3qL/Gwxu/OEfTE9A2LkqwX7wM W1yA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724239398; x=1724844198; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4qJ3V7YJGTaGdVFsVivjHiBFvmyNekMesA0jctJoY0g=; b=JJZx9LH3DQ9wS2oocNk7ulNuT/AoiSgzkZ/hHvlStwKYDd49EOhNxcpdxxyH4/sE7N 6JPxRlMAvGCbUPeAa6GD9+W6hfKW67A5RTKzutgCqS28OW/PiTNipoL72nmqw03lGmax YXinvtD6N2HWYzKO/AFl7isdlvSkmrhvbeLNTbvnmWoPRXRk9n376p+1UjLllE5w8fvF 3F0YhuOTrByZAmwOg65KFGy8J6qVp91EAOKpTeo+Ge6aIlWcpNPvlN1wZz4XcWuEurH9 Z3QpqGi7FR+eE76IBAwNZt7aZ5+TifDI+ifTKmEzsuBQPHGCBbjdcLUDSRUEdbhS67jl vnJg== X-Gm-Message-State: AOJu0YxY+E/RRItALAo1gZ4NkeA/YUyQfRscHzVKlHeBLyBDAShAwvnJ npjM0FoeWcsXtuYXAohffghJxsu0ewn8hdi/0h+0qtMLwxSbcGIM5T0IN5DFxUJSm8jDpYCSHKe 9iZA= X-Google-Smtp-Source: AGHT+IFo3xr/2wRjH9B573sMM/AsBH1F846a16X87+m+NYa2uTqYpAyZHAHIfbC6knOzxIbj+tbz2g== X-Received: by 2002:a17:907:9706:b0:a7a:a06b:eec9 with SMTP id a640c23a62f3a-a866f11c1a5mr142347266b.4.1724239397581; Wed, 21 Aug 2024 04:23:17 -0700 (PDT) Received: from lb01533.speedport.ip (p200300f00f051d5f269a60e7b8956185.dip0.t-ipconnect.de. [2003:f0:f05:1d5f:269a:60e7:b895:6185]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a83838c65e7sm887934066b.20.2024.08.21.04.23.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 04:23:17 -0700 (PDT) From: Md Haris Iqbal To: linux-rdma@vger.kernel.org Cc: leon@kernel.org, jgg@ziepe.ca, haris.iqbal@ionos.com, jinpu.wang@ionos.com, Grzegorz Prajsner Subject: [PATCH v2 for-next 02/11] RDMA/rtrs-clt: Fix need_inv setting in error case Date: Wed, 21 Aug 2024 13:22:08 +0200 Message-Id: <20240821112217.41827-3-haris.iqbal@ionos.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240821112217.41827-1-haris.iqbal@ionos.com> References: <20240821112217.41827-1-haris.iqbal@ionos.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Jack Wang In some cases need_inv can be missed for write requests, additionally driver has to handle missing invalidates for write requests. While at it, remove the else case from write invalidate path as it is possible to reach there. Signed-off-by: Jack Wang Signed-off-by: Md Haris Iqbal Signed-off-by: Grzegorz Prajsner --- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c index 66ac4dba990f..d09018c11ece 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c @@ -391,11 +391,12 @@ static void complete_rdma_req(struct rtrs_clt_io_req *req, int errno, clt_path = to_clt_path(con->c.path); if (req->sg_cnt) { - if (req->dir == DMA_FROM_DEVICE && req->need_inv) { + if (req->need_inv) { /* - * We are here to invalidate read requests + * We are here to invalidate read/write requests * ourselves. In normal scenario server should - * send INV for all read requests, but + * send INV for all read requests, we do chained local + * invalidate for write requests, but * we are here, thus two things could happen: * * 1. this is failover, when errno != 0 @@ -422,14 +423,6 @@ static void complete_rdma_req(struct rtrs_clt_io_req *req, int errno, req->mr->rkey, err); } else if (can_wait) { wait_for_completion(&req->inv_comp); - } else { - /* - * Something went wrong, so request will be - * completed from INV callback. - */ - WARN_ON_ONCE(1); - - return; } if (!refcount_dec_and_test(&req->ref)) return; @@ -1146,6 +1139,7 @@ static int rtrs_clt_write_req(struct rtrs_clt_io_req *req) }; wr = &rwr.wr; fr_en = true; + req->need_inv = true; refcount_inc(&req->ref); } /* @@ -1164,6 +1158,10 @@ static int rtrs_clt_write_req(struct rtrs_clt_io_req *req) clt_path->hca_port); if (req->mp_policy == MP_POLICY_MIN_INFLIGHT) atomic_dec(&clt_path->stats->inflight); + if (req->need_inv) { + req->need_inv = false; + refcount_dec(&req->ref); + } if (req->sg_cnt) ib_dma_unmap_sg(clt_path->s.dev->ib_dev, req->sglist, req->sg_cnt, req->dir); From patchwork Wed Aug 21 11:22:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Haris Iqbal X-Patchwork-Id: 13771250 Received: from mail-ej1-f50.google.com (mail-ej1-f50.google.com [209.85.218.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 94AB11A4B8D for ; Wed, 21 Aug 2024 11:23:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724239402; cv=none; b=qBBRnfjHWghYJjbs7HN4sLUXoscwLDSxyJb5RoitxAyiRJ6jmFMOaJ6OkKVJUnPLK/NyjEgNq1Pj1u9F9oUKHzq32PUlPJxYjy+mKg8uEpA8fr1LyWkZtu5CDsZneCmSCm96AX2/z3AaJX233LRFYfydepN3nyw9el9wPdFnvEE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724239402; c=relaxed/simple; bh=wSG8mJWVwK1/KFG56H2h9FYqiJNGfBvw1cfDruXjyG8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=VGRvXl+YX3KpsPLEyM/rVoXIqXmmzgLi/zPkxDCPkSImH0XMcVA7N9I6cv9qpWWaXD4SUMGrIvVSqWwh39O8slB1rNiJOkFVS7HOjINFugJEVP0BQdUShM0IEVyFm5+YFkqFA0o4iP2lzxImVx5O3KMFD8ztqPnrvNjE0hUBoZs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ionos.com; spf=pass smtp.mailfrom=ionos.com; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b=S5UN039s; arc=none smtp.client-ip=209.85.218.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ionos.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ionos.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b="S5UN039s" Received: by mail-ej1-f50.google.com with SMTP id a640c23a62f3a-a7a94478a4eso122995066b.1 for ; Wed, 21 Aug 2024 04:23:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; t=1724239399; x=1724844199; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3fjw2Ta8LUJKcy0S2Y9A72sgh/iD12aMXAxb+1vnFQ4=; b=S5UN039sr2v0SnLoolvFf/EG5KJ1d1vd6qoZjEPECaAwDoaLMBCIutuG8U3JVIWVvO Z1imB/txczL4hqTQVtUieYttTFIpp9sBsbpw4IffbONG+9xEqG+wwR0BHdqM6tjVTMBg Y58K91aGmcRW0lQwFALzCteRGtsu+YSNjKryj8z7+4Wai6osG3HyKL7up0d+a95gR9m3 ty5TKialSyHV9Kx5qnBeawpasYKIIRgVKN1M9vKRJbTrM2Ux8ckVwbOu7Yoq7Jrr7L0x WU/o9arkOhw7DQcXGrxhmn3B2npe3Hv2l/EhXyr/w8xG4hgoJM0RPmRSFmaMqE2k/qRt vpFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724239399; x=1724844199; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3fjw2Ta8LUJKcy0S2Y9A72sgh/iD12aMXAxb+1vnFQ4=; b=HW6HWJs9YELnvccsLkIAJps6mjZw6jATy5KNG9HcquHe5hLkxf7/+RMQAnczwSpARz 2ZooOdB6CPqaKJ0rhJgeRpND1Mf0xPPC/hJ90olTC0iebH5GH4iW6HMbtaAEwunZH8bT GEnYKUdQuORhnOXOgcxBzvXNcnNos2gMAsViDwMwbr2ABXy1DzODps5Uw5oUX1kZkj2z f4Jyp5MSxbgENOzbE2bO8OLd9S8iU7EQDBBO9EP+gIC0xgGiS91xTSXgC/nDNuTExBTO S89mxjMzUZJEmV8EPxTDQw3Ski8nLn8MpErt51DBmTJiK0K5HBY5mwNaQbECvqvNp7W3 QDKg== X-Gm-Message-State: AOJu0YyFHHuIl06jBDGLoBb2i5a0NezY0N4ITGdTlG9iRV2vf7kITMUg hT+mYH8ASHVagggrlUsgHqxctjlqPs2ZGNZgDDFloLCBR2yJT+YUgV+15jGbZYP+gZOjVYBvGVJ V08o= X-Google-Smtp-Source: AGHT+IEsUfqne2r3nsaYjTyU8ED4E0/ZaBTF222Wk0s44ci9jA13M+pFm08VBzXvMhXp4Kv8lLfHtg== X-Received: by 2002:a17:907:6d24:b0:a7a:b561:358e with SMTP id a640c23a62f3a-a8670183ea0mr178400466b.33.1724239398245; Wed, 21 Aug 2024 04:23:18 -0700 (PDT) Received: from lb01533.speedport.ip (p200300f00f051d5f269a60e7b8956185.dip0.t-ipconnect.de. [2003:f0:f05:1d5f:269a:60e7:b895:6185]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a83838c65e7sm887934066b.20.2024.08.21.04.23.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 04:23:17 -0700 (PDT) From: Md Haris Iqbal To: linux-rdma@vger.kernel.org Cc: leon@kernel.org, jgg@ziepe.ca, haris.iqbal@ionos.com, jinpu.wang@ionos.com, Grzegorz Prajsner Subject: [PATCH v2 for-next 03/11] RDMA/rtrs-clt: Rate limit errors in IO path Date: Wed, 21 Aug 2024 13:22:09 +0200 Message-Id: <20240821112217.41827-4-haris.iqbal@ionos.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240821112217.41827-1-haris.iqbal@ionos.com> References: <20240821112217.41827-1-haris.iqbal@ionos.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Jack Wang On network errors, a large number of these logs are printed due to all the inflight IOs, rate limit them so they do not clutter kernel log. Signed-off-by: Jack Wang Signed-off-by: Md Haris Iqbal Signed-off-by: Grzegorz Prajsner --- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c index d09018c11ece..b34eb4908185 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c @@ -331,7 +331,7 @@ static void rtrs_clt_fast_reg_done(struct ib_cq *cq, struct ib_wc *wc) struct rtrs_clt_con *con = to_clt_con(wc->qp->qp_context); if (wc->status != IB_WC_SUCCESS) { - rtrs_err(con->c.path, "Failed IB_WR_REG_MR: %s\n", + rtrs_err_rl(con->c.path, "Failed IB_WR_REG_MR: %s\n", ib_wc_status_msg(wc->status)); rtrs_rdma_error_recovery(con); } @@ -351,7 +351,7 @@ static void rtrs_clt_inv_rkey_done(struct ib_cq *cq, struct ib_wc *wc) struct rtrs_clt_con *con = to_clt_con(wc->qp->qp_context); if (wc->status != IB_WC_SUCCESS) { - rtrs_err(con->c.path, "Failed IB_WR_LOCAL_INV: %s\n", + rtrs_err_rl(con->c.path, "Failed IB_WR_LOCAL_INV: %s\n", ib_wc_status_msg(wc->status)); rtrs_rdma_error_recovery(con); } @@ -419,7 +419,7 @@ static void complete_rdma_req(struct rtrs_clt_io_req *req, int errno, refcount_inc(&req->ref); err = rtrs_inv_rkey(req); if (err) { - rtrs_err(con->c.path, "Send INV WR key=%#x: %d\n", + rtrs_err_rl(con->c.path, "Send INV WR key=%#x: %d\n", req->mr->rkey, err); } else if (can_wait) { wait_for_completion(&req->inv_comp); From patchwork Wed Aug 21 11:22:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Haris Iqbal X-Patchwork-Id: 13771251 Received: from mail-ej1-f49.google.com (mail-ej1-f49.google.com [209.85.218.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0C947158544 for ; Wed, 21 Aug 2024 11:23:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724239402; cv=none; b=L0AGTMrymIa3nvSneLlpuaJ5YSvqAgf//X68xBpOUXRFqM9lgrMzRIvOPALAFriy2ei8UD8jSIHKiNgzjxlv28fZ2zf9eWonrlYXFOtvZkUXnsMZpoQAb1pxKxlmwajuaapupW0r0OVl/DYD1WWuIFRM6g02lreChXVLC46hRtM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724239402; c=relaxed/simple; bh=/6OItEvvxYPlrrVGtyxJUaH8vTuT08SMFkHPMeVL9XA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=E1rrsDpcJ6PxzKNM6QhPkmGdaahSjLb0qHdyoYhTRcM3XXWIS7mG4xBLHRp0LSBDen6Ug1XNojRj4EBcwBAydKeeGB7sm+sDwrmXg8ZwZn9nuGR2Gwk586DbrjA9yKP/PQE01nK1NcaB8cMHX7aexqrWqb8zP0mPzMQ9wUXiZBs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ionos.com; spf=pass smtp.mailfrom=ionos.com; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b=CUqTXRrL; arc=none smtp.client-ip=209.85.218.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ionos.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ionos.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b="CUqTXRrL" Received: by mail-ej1-f49.google.com with SMTP id a640c23a62f3a-a7aada2358fso106518766b.0 for ; Wed, 21 Aug 2024 04:23:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; t=1724239399; x=1724844199; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pE5YcoXaJNOQSKQWGfiOqDhkdWpVQdl73GYH6YX7gvI=; b=CUqTXRrL7kukfzg05dWjnkul+uhnH1tO9KVzJAWwnzJadoE0eS8uKfvap+gdxbxCww hj4zeJhV3o+dmcMrI+y3FdSgTy9tXdECWFoI9fwCvJHUr1fOAoUt+QFWIin/yGWmY3SH 0PVs2B893C4eNIqCJdKaArHYcd8O1bM95bn/OQ961mvngBYCnhTZZPqjo980pxTCUJrS H8E7qBEHIqrRJBXh27A5b1M5XXQAFmG6Wuezww2ciGuRxhNleOxcXaVMTeNLOwPNlX8M viNpr7guLF8wlNflakWdJn+R0OVKEIQ4dt+EjdLkdhW3eBVkPYXn5CYIFk/xFDCB75zx l8Zg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724239399; x=1724844199; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pE5YcoXaJNOQSKQWGfiOqDhkdWpVQdl73GYH6YX7gvI=; b=LbcsfZcxHlKefJfTGk50e7JdDCe/pFJYfNAmeYdiDoaud9wOBvojCexkzQocvKB6fK V+RQ2DVSpQ2xuq0h0/roIn5ivuLCBwmFuSZWeD//I6iOWukegaJOSJnkSV11m85HAOwv iovuaRELc/IZg3a/Nga2nWKVxguWhyGo1lkM2TZaRqq94ozUmHZ++SO6TEQdg2iJjWHb OQBmVxQrrqmebhaH0N8ezFMYxXmaC/eJkGmjSvvn3LlWG9oLRMh57fVU30LqH4z/ROwI TFp4wCvl6/fezU0cerxrLhJwk/tW4OkAEVnHuvaZc1xPbQP/tRBFlFZOUX7yLQilVCOJ NIRg== X-Gm-Message-State: AOJu0Yxwvz6Utn1bMotwGvKN+3yLPwnuv5+mh61AF2FEqLuAk053Yptq py2PpUxprInXMnkf4jaIGgTKs/s1BUIiQQ4/45TOJraSwFT8xnsUcGHfKxtesGJqLd70zsA6Zaz sDMM= X-Google-Smtp-Source: AGHT+IEx/uZ5Zjlhi7tgtAwgfiJWMI54Vgp0Qk4v+J08s4AHJI6sKYUAB2OjVJHvC88qyP2JgzV0mw== X-Received: by 2002:a17:907:940d:b0:a7d:c382:bcdf with SMTP id a640c23a62f3a-a866ff4be1bmr179248666b.10.1724239398926; Wed, 21 Aug 2024 04:23:18 -0700 (PDT) Received: from lb01533.speedport.ip (p200300f00f051d5f269a60e7b8956185.dip0.t-ipconnect.de. [2003:f0:f05:1d5f:269a:60e7:b895:6185]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a83838c65e7sm887934066b.20.2024.08.21.04.23.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 04:23:18 -0700 (PDT) From: Md Haris Iqbal To: linux-rdma@vger.kernel.org Cc: leon@kernel.org, jgg@ziepe.ca, haris.iqbal@ionos.com, jinpu.wang@ionos.com, Grzegorz Prajsner Subject: [PATCH v2 for-next 04/11] RDMA/rtrs: Reset hb_missed_cnt after receiving other traffic from peer Date: Wed, 21 Aug 2024 13:22:10 +0200 Message-Id: <20240821112217.41827-5-haris.iqbal@ionos.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240821112217.41827-1-haris.iqbal@ionos.com> References: <20240821112217.41827-1-haris.iqbal@ionos.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Jack Wang Reset hb_missed_cnt after receiving traffic from other peer, so hb is more robust again high load on host or network. Fixes: 6a98d71daea1 ("RDMA/rtrs: client: main functionality") Signed-off-by: Jack Wang Signed-off-by: Md Haris Iqbal Signed-off-by: Grzegorz Prajsner --- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 3 ++- drivers/infiniband/ulp/rtrs/rtrs-srv.c | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c index b34eb4908185..c1bca8972015 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c @@ -619,6 +619,7 @@ static void rtrs_clt_rdma_done(struct ib_cq *cq, struct ib_wc *wc) */ if (WARN_ON(wc->wr_cqe->done != rtrs_clt_rdma_done)) return; + clt_path->s.hb_missed_cnt = 0; rtrs_from_imm(be32_to_cpu(wc->ex.imm_data), &imm_type, &imm_payload); if (imm_type == RTRS_IO_RSP_IMM || @@ -636,7 +637,6 @@ static void rtrs_clt_rdma_done(struct ib_cq *cq, struct ib_wc *wc) return rtrs_clt_recv_done(con, wc); } else if (imm_type == RTRS_HB_ACK_IMM) { WARN_ON(con->c.cid); - clt_path->s.hb_missed_cnt = 0; clt_path->s.hb_cur_latency = ktime_sub(ktime_get(), clt_path->s.hb_last_sent); if (clt_path->flags & RTRS_MSG_NEW_RKEY_F) @@ -663,6 +663,7 @@ static void rtrs_clt_rdma_done(struct ib_cq *cq, struct ib_wc *wc) /* * Key invalidations from server side */ + clt_path->s.hb_missed_cnt = 0; WARN_ON(!(wc->wc_flags & IB_WC_WITH_INVALIDATE || wc->wc_flags & IB_WC_WITH_IMM)); WARN_ON(wc->wr_cqe->done != rtrs_clt_rdma_done); diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c index f76d483c3784..ffd3e80596d0 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c @@ -1233,6 +1233,7 @@ static void rtrs_srv_rdma_done(struct ib_cq *cq, struct ib_wc *wc) */ if (WARN_ON(wc->wr_cqe != &io_comp_cqe)) return; + srv_path->s.hb_missed_cnt = 0; err = rtrs_post_recv_empty(&con->c, &io_comp_cqe); if (err) { rtrs_err(s, "rtrs_post_recv(), err: %d\n", err); From patchwork Wed Aug 21 11:22:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Haris Iqbal X-Patchwork-Id: 13771252 Received: from mail-ej1-f46.google.com (mail-ej1-f46.google.com [209.85.218.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B5FDB1C4ECE for ; Wed, 21 Aug 2024 11:23:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724239403; cv=none; b=OwFDuEyfxPQySSOTXckmAovI9EPybNnPKjsPXnhnbRZ0sqMOgR1LsY0gGK4TtALm6WhXH+yb28KdApDiJ1IxsqMb1t1l+DFui0Gjj+6viqJY7nLhIPKsEgCy0JDV89GK3X3DH4Tyo7Xgxk2JJzmehVfZEfXXXQ7xdoH7qouK5fo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724239403; c=relaxed/simple; bh=r4XLSg/aoYdYcbzTL1SRlVWjZce8no+6qF2vTB5o4I8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=sDK7luUb6rHzQ7iguV+16Sz6mJuZiE62uthTYwCX3vcUaMEIUjZhW3/dz8pbTsLCVNsha1koeppwSbWatzzzUbn87H0tLeMgZibGKuwL8VAzZ8dtiIerC47/ML+uNGIF1n3kp6rrKw2av1zW1CM5Mxuy/VhMBb/P8Q0FD4RX4jI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ionos.com; spf=pass smtp.mailfrom=ionos.com; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b=igy5NGVZ; arc=none smtp.client-ip=209.85.218.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ionos.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ionos.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b="igy5NGVZ" Received: by mail-ej1-f46.google.com with SMTP id a640c23a62f3a-a7aa086b077so682958766b.0 for ; Wed, 21 Aug 2024 04:23:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; t=1724239400; x=1724844200; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Fx7gqp5M1vTXN5tlxGBDKn1NzWyNR+YNEd/cHoZlL1U=; b=igy5NGVZlTzB7eYg0d3C8EOwaUkOx3H+5YvVmsw3cha9Nobj7KdWKhRPQF/Hrj6qNm N3dvkw9qnrVnpDL3lCHrUNzQhleDxVGRoPyQ8wpbn77OnLBfpFBPk8HZm0cn/ONkOZ6o T0rNBYG3R7GJu/YLNdiJ5wikSQxZHTD3swhGF4NV/sb6Z46WtD1D3KKntvApoc+aguI6 ia0HdEGcuMW6AdLWj2z3O/FVbXZf9maYaan3OPe6YA9LDwizFzUZvfrRKiDLp7LRAWr6 SBsiA4pO7TNxncG+ihgcmhaYIpskr9XvaS1Qb4c17CKHXSedrx3Szk+GHYKNAU2K9xWV 6ySA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724239400; x=1724844200; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Fx7gqp5M1vTXN5tlxGBDKn1NzWyNR+YNEd/cHoZlL1U=; b=WqSJV0BUYC9JrYe+lAyVj2cM7pcCtM7BUrBP7dDyDavdP6crDIIt9NtypDF8Ncpk3C CEYN0yyuAiuxrqEz6bjXMBM5nNXhfFrARcms5LR04Qe2H0FaNzuhlyGzi15bZWiRX75B /dS6LbYN+obvnJbgZw6iSHTblLH1VQu+helTAeUNwu3rC/LM/6a30QR7DIkOobjWeraG 0+gN6u19YEkGpmHdzZyoBYackFVp41zHfRn5LZwH3EnIgHmbRAYIvCr8wsc091gpkW8u psmH2s3ODiY+Gr3zuJSlKxpGCIXVtKZfcQU6LqsrTmSClY6VYzZJWtiln0tHqK/6NZ4n DdlA== X-Gm-Message-State: AOJu0YxLC2QSlA7vcj/mxUeV197M8ICsjWsQsRvJqCxjGdTTGJMkw/zC w2nqaG/oBOodlpy2DSkz/5BseJrBjLucBLx8GQVU+Lm0Fn3r/URspOy1eFFQpuNHcwPxTWs0PEf 5frc= X-Google-Smtp-Source: AGHT+IFHmMTOF93YmzhXM3930QSYwL2ehrLDW7HclzipsPwBFtzXLRh+iLKrgKGe61pw28WC6nRg3A== X-Received: by 2002:a17:907:7212:b0:a7a:b070:92cc with SMTP id a640c23a62f3a-a866f8f45bfmr110236366b.45.1724239399656; Wed, 21 Aug 2024 04:23:19 -0700 (PDT) Received: from lb01533.speedport.ip (p200300f00f051d5f269a60e7b8956185.dip0.t-ipconnect.de. [2003:f0:f05:1d5f:269a:60e7:b895:6185]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a83838c65e7sm887934066b.20.2024.08.21.04.23.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 04:23:19 -0700 (PDT) From: Md Haris Iqbal To: linux-rdma@vger.kernel.org Cc: leon@kernel.org, jgg@ziepe.ca, haris.iqbal@ionos.com, jinpu.wang@ionos.com, Grzegorz Prajsner Subject: [PATCH v2 for-next 05/11] RDMA/rtrs-clt: Reuse need_inval from mr Date: Wed, 21 Aug 2024 13:22:11 +0200 Message-Id: <20240821112217.41827-6-haris.iqbal@ionos.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240821112217.41827-1-haris.iqbal@ionos.com> References: <20240821112217.41827-1-haris.iqbal@ionos.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Jack Wang mr has a member need_inval, which can be used to indicate if local invalidate is needed, switch to it and remove need_inv from rtrs_clt_io_req. Signed-off-by: Jack Wang Signed-off-by: Md Haris Iqbal Signed-off-by: Grzegorz Prajsner --- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 18 +++++++++--------- drivers/infiniband/ulp/rtrs/rtrs-clt.h | 1 - 2 files changed, 9 insertions(+), 10 deletions(-) diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c index c1bca8972015..e1557b0cda05 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c @@ -355,7 +355,7 @@ static void rtrs_clt_inv_rkey_done(struct ib_cq *cq, struct ib_wc *wc) ib_wc_status_msg(wc->status)); rtrs_rdma_error_recovery(con); } - req->need_inv = false; + req->mr->need_inval = false; if (req->need_inv_comp) complete(&req->inv_comp); else @@ -391,7 +391,7 @@ static void complete_rdma_req(struct rtrs_clt_io_req *req, int errno, clt_path = to_clt_path(con->c.path); if (req->sg_cnt) { - if (req->need_inv) { + if (req->mr->need_inval) { /* * We are here to invalidate read/write requests * ourselves. In normal scenario server should @@ -494,7 +494,7 @@ static void process_io_rsp(struct rtrs_clt_path *clt_path, u32 msg_id, req = &clt_path->reqs[msg_id]; /* Drop need_inv if server responded with send with invalidation */ - req->need_inv &= !w_inval; + req->mr->need_inval &= !w_inval; complete_rdma_req(req, errno, true, false); } @@ -961,7 +961,7 @@ static void rtrs_clt_init_req(struct rtrs_clt_io_req *req, req->dir = dir; req->con = rtrs_permit_to_clt_con(clt_path, permit); req->conf = conf; - req->need_inv = false; + req->mr->need_inval = false; req->need_inv_comp = false; req->inv_errno = 0; refcount_set(&req->ref, 1); @@ -1140,8 +1140,8 @@ static int rtrs_clt_write_req(struct rtrs_clt_io_req *req) }; wr = &rwr.wr; fr_en = true; - req->need_inv = true; refcount_inc(&req->ref); + req->mr->need_inval = true; } /* * Update stats now, after request is successfully sent it is not @@ -1159,8 +1159,8 @@ static int rtrs_clt_write_req(struct rtrs_clt_io_req *req) clt_path->hca_port); if (req->mp_policy == MP_POLICY_MIN_INFLIGHT) atomic_dec(&clt_path->stats->inflight); - if (req->need_inv) { - req->need_inv = false; + if (req->mr->need_inval) { + req->mr->need_inval = false; refcount_dec(&req->ref); } if (req->sg_cnt) @@ -1236,7 +1236,7 @@ static int rtrs_clt_read_req(struct rtrs_clt_io_req *req) msg->desc[0].len = cpu_to_le32(req->mr->length); /* Further invalidation is required */ - req->need_inv = !!RTRS_MSG_NEED_INVAL_F; + req->mr->need_inval = !!RTRS_MSG_NEED_INVAL_F; } else { msg->sg_cnt = 0; @@ -1269,7 +1269,7 @@ static int rtrs_clt_read_req(struct rtrs_clt_io_req *req) clt_path->hca_port); if (req->mp_policy == MP_POLICY_MIN_INFLIGHT) atomic_dec(&clt_path->stats->inflight); - req->need_inv = false; + req->mr->need_inval = false; if (req->sg_cnt) ib_dma_unmap_sg(dev->ib_dev, req->sglist, req->sg_cnt, req->dir); diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.h b/drivers/infiniband/ulp/rtrs/rtrs-clt.h index f848c0392d98..45dac15825f4 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-clt.h +++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.h @@ -115,7 +115,6 @@ struct rtrs_clt_io_req { struct completion inv_comp; int inv_errno; bool need_inv_comp; - bool need_inv; refcount_t ref; }; From patchwork Wed Aug 21 11:22:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Haris Iqbal X-Patchwork-Id: 13771253 Received: from mail-ej1-f49.google.com (mail-ej1-f49.google.com [209.85.218.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E1C291C4EE2 for ; Wed, 21 Aug 2024 11:23:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724239404; cv=none; b=KMjGNUoUOodSZg0BVfUfflO+vAFSOEtvK481YklvKqFP2oWMlX31KsqWXri/4tDDlw56fhBuQ6BFi0xQZhhU1XJNfx2TK15GYrh5uuH/tD/EcFJVWZWm1dPI/NGeJnEBlzrLmxFER5dUFlKazoJjt54dIXig45nJeZxCpsLQwF4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724239404; c=relaxed/simple; bh=WRV7DY/vIWFiZ3gU6V1V+cCtOFglJblnZJhmSfbRmuY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=K322U0XdrKWfskFgPFKo76uk3zHzQa4ndWNcchpqCoHherFajUJWFyDyxqeF2qGgDkanpPk0XPGYQ4+Z6ouxac6l61Au+dXAvMcJYPWLJuhwqpYqoEKzS3295QRBM9JCHa1zCXeY3YlEtyt6PYCBZX/C97Q+G+fUIz+KT9qp1jw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ionos.com; spf=pass smtp.mailfrom=ionos.com; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b=OP2VLynZ; arc=none smtp.client-ip=209.85.218.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ionos.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ionos.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b="OP2VLynZ" Received: by mail-ej1-f49.google.com with SMTP id a640c23a62f3a-a8677ae5a35so64722166b.0 for ; Wed, 21 Aug 2024 04:23:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; t=1724239401; x=1724844201; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+XXHfR6ei5DWieETKO7ps9vaiG5tNzbNEBSlnAsZkFM=; b=OP2VLynZgCKoPYBSaNt7cUVTB7XEusobHswovkPv0NrHiz3sMOJS2jSVmJ42zFO/oH yCYR8WSmo4oBOPtZ0Wr8EK2S+s7lTqjZaOfUCqyiyrZZacChjj2ObHtGXsvXokb0AM7l +pTUW36wb9uHsZ/gy/OBYLFM1o1Bq1shDlcBXieYCtcFqCNlAqJ4TBbjK6SIIEa6oHRi HDQnnfzMjTWSY3bT9EsyJ0KXEornHw/RV5MK4jG5IAADPA4eKJKfAPVgkmvTQK6nuLar un61AsAXGroyp+dN5n7vKivxTLFx1pW37BCPv5AGYRHRQ5oq8B/CWCp3fh88UEbaW3fA A4JQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724239401; x=1724844201; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+XXHfR6ei5DWieETKO7ps9vaiG5tNzbNEBSlnAsZkFM=; b=HYMRD3RRDPndGv3mHUVOk+wbWH9aLW8K0w489W99KScxDWUWyk8qFDedMbAbu7DKO7 +qC3gyL2gjiCtZQ/eudMdvlEtvdjxrLJJ27WAonIqFOx+incYvP5l0AcP3CLzjyZBP3K KnEXSjI51mSctj11u4kpiNcVNe+PqaYfx47lrGy06/dKTPjoBt1FwNlRAu/P86MIoKN/ gUKgkbtegavYfTg54CLkDpU+tWaODdfHNUAV9Eies1D96g6uhYgU1FJt5w5B/qKmHnV6 mCkSsqnRqHhwIfCk9MAUotykp1WbHwpxqoRrJd231L3+rXrGC6osH6q410xgENeEzu+g lfwQ== X-Gm-Message-State: AOJu0YzHDgceuKnrGkl/cxQ82+BdgBfRPHqj55gabInnMpwnQhTbsIyW lX5fmAwAQKr0LQ4mxEHKhM02VdgQLpE6NGKXYQt/uj0Do0ubCEEyMefLLTVI89gub07bZvvWgH7 p0Fo= X-Google-Smtp-Source: AGHT+IG9O4A5niIGiT8o7ZDBERKjHd7T4hEqfA2o+YTkUqqA6o5GYuasd/6blKgVWsXmDxrmMpK/jA== X-Received: by 2002:a17:907:1c22:b0:a7a:bece:6222 with SMTP id a640c23a62f3a-a866f14777bmr118932366b.10.1724239400773; Wed, 21 Aug 2024 04:23:20 -0700 (PDT) Received: from lb01533.speedport.ip (p200300f00f051d5f269a60e7b8956185.dip0.t-ipconnect.de. [2003:f0:f05:1d5f:269a:60e7:b895:6185]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a83838c65e7sm887934066b.20.2024.08.21.04.23.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 04:23:20 -0700 (PDT) From: Md Haris Iqbal To: linux-rdma@vger.kernel.org Cc: leon@kernel.org, jgg@ziepe.ca, haris.iqbal@ionos.com, jinpu.wang@ionos.com, Grzegorz Prajsner Subject: [PATCH v2 for-next 06/11] RDMA/rtrs-clt: Reset cid to con_num - 1 to stay in bounds Date: Wed, 21 Aug 2024 13:22:12 +0200 Message-Id: <20240821112217.41827-7-haris.iqbal@ionos.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240821112217.41827-1-haris.iqbal@ionos.com> References: <20240821112217.41827-1-haris.iqbal@ionos.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 In the function init_conns(), after the create_con() and create_cm() for loop if something fails. In the cleanup for loop after the destroy tag, we access out of bound memory because cid is set to clt_path->s.con_num. This commits resets the cid to clt_path->s.con_num - 1, to stay in bounds in the cleanup loop later. Fixes: 6a98d71daea1 ("RDMA/rtrs: client: main functionality") Signed-off-by: Md Haris Iqbal Signed-off-by: Jack Wang Signed-off-by: Grzegorz Prajsner --- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c index e1557b0cda05..777f8e52ed7c 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c @@ -2347,6 +2347,12 @@ static int init_conns(struct rtrs_clt_path *clt_path) if (err) goto destroy; } + + /* + * Set the cid to con_num - 1, since if we fail later, we want to stay in bounds. + */ + cid = clt_path->s.con_num - 1; + err = alloc_path_reqs(clt_path); if (err) goto destroy; From patchwork Wed Aug 21 11:22:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Haris Iqbal X-Patchwork-Id: 13771254 Received: from mail-ej1-f41.google.com (mail-ej1-f41.google.com [209.85.218.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6D4B91C4EF0 for ; Wed, 21 Aug 2024 11:23:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724239405; cv=none; b=ORWt7yHnr92ykz5OqLxNvakAcRePNMz8yhl4zlLDEgnBAE03qqPsVwpOPMhvwQxAXLif0PQnRQr5Q1vGpPFBSNbTcCJHzziFcWTsOAsUI/SuvQ/SRLAf3ZFJomq/OinWg4DH9gOxTRTXAbwd6RASuHJdgAdJE7d/meYF2vhuZY0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724239405; c=relaxed/simple; bh=E4Tkz+NNXf0Rqt+KeIW7YV0+OPjkqtWHuBsqB6QLKqw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=MN5tOlPHd8JL19nivl+Xjr9n9aVG2skg/PkMspyDYsY8rDaWVilo0qUdxdSjxvnH7BKsLDVFuegPscj18Skgqks3B4VG2UgIIRAqiIy7SNggKO5gBjtsp8EDcUUb0nq8j230MST/ZxtSNOe66iYLeP0S9w1seQSTKVI+UfIuU0A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ionos.com; spf=pass smtp.mailfrom=ionos.com; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b=VdSCh6cn; arc=none smtp.client-ip=209.85.218.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ionos.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ionos.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b="VdSCh6cn" Received: by mail-ej1-f41.google.com with SMTP id a640c23a62f3a-a83597ce5beso111434366b.1 for ; Wed, 21 Aug 2024 04:23:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; t=1724239402; x=1724844202; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HC3jHGsjdZENIhkaZnf9KhbQHlXlkHO+680dBNxyyNY=; b=VdSCh6cnEx1ef554tal8TDZPItXtG1Ad5J/TsMBDzSq2/mM4mCdmae4UUzqUOCHPG0 jeFiDpY0Fry+XAdrhJCW5dLMf65u8gtWctlGS5fr73khEGc6AGZZxkI4QIYqblvtMPJ+ REAdSHo49v2Se9e/SEq3Wg0qvhKzKjBUkcwD3T6QXHKuYza+EJMLCVgDN3j15yJHhZKr N1jJyHHRhKZ+HM8Ho27nN53QUcS3E/H64PMV2XeQS69jCrlai8yXCzC/xGaHp8rUY80H Qvfvlmy1hk1BEBIK79TjUSnIhNFIR00ThbSEA2g9UAE+kQClldYJAmYKA4MSbuSnmDQG AwYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724239402; x=1724844202; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HC3jHGsjdZENIhkaZnf9KhbQHlXlkHO+680dBNxyyNY=; b=t2BNddOnE+dch2RQi8QNSyD1QAwlmz0rKNIcEV45Z3t0EfUb7Bc+T9rxDBvPLV7mJL /mK8lf8YXUbpcoD6w8eN9Cfr/NvGxE5M20hJtUaDWGIyjfq+Ibzod1D8RU6btoPJp1cC 85fN0qC/p3LbbKEvWxeuHH+avVLXKzTgqB+m4iB2kgb/8Y8+Jt+l0CN0K/+Rb3E9hLYG L3BmbRhDCUmZeAnmgWwZw4Y4RIXoMBhEPSkqfB41/jh4GS1sdmRlKk3Fbwl5Dzve7RuM GtMekOAclET47Jfx5GRmZYk9bk5FHCCgOMKX3GGXUW9mYy4+obOuNaGvoBT8zAQzZU9A yBNg== X-Gm-Message-State: AOJu0YwVHKV8ly6CZIX20BO6WHK+kuKM2itLD/Ktyo/2t9NUpEFfoe35 jnkbuAXzkallmFgopidRbxZx4RJHoksI51RrZizZPf00ZpGN51+iKBAnxBFBQ1WA/SJCDKA3TYw nCMs= X-Google-Smtp-Source: AGHT+IHVtOtKjKEe2nw4L+xnQyvBeribB4AtIzAm+LIzBCmSeGBnSCHvaWZIgSbDeZCm4xw77OPcGQ== X-Received: by 2002:a17:907:a4c:b0:a7d:ab62:c317 with SMTP id a640c23a62f3a-a8670180e3bmr176885966b.30.1724239401395; Wed, 21 Aug 2024 04:23:21 -0700 (PDT) Received: from lb01533.speedport.ip (p200300f00f051d5f269a60e7b8956185.dip0.t-ipconnect.de. [2003:f0:f05:1d5f:269a:60e7:b895:6185]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a83838c65e7sm887934066b.20.2024.08.21.04.23.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 04:23:21 -0700 (PDT) From: Md Haris Iqbal To: linux-rdma@vger.kernel.org Cc: leon@kernel.org, jgg@ziepe.ca, haris.iqbal@ionos.com, jinpu.wang@ionos.com, Grzegorz Prajsner Subject: [PATCH v2 for-next 07/11] RDMA/rtrs-clt: Print request type for errors Date: Wed, 21 Aug 2024 13:22:13 +0200 Message-Id: <20240821112217.41827-8-haris.iqbal@ionos.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240821112217.41827-1-haris.iqbal@ionos.com> References: <20240821112217.41827-1-haris.iqbal@ionos.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Jack Wang Extend the output to print also the request type. Signed-off-by: Jack Wang Signed-off-by: Grzegorz Prajsner Signed-off-by: Md Haris Iqbal --- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c index 777f8e52ed7c..7c6d40380638 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c @@ -439,8 +439,10 @@ static void complete_rdma_req(struct rtrs_clt_io_req *req, int errno, req->con = NULL; if (errno) { - rtrs_err_rl(con->c.path, "IO request failed: error=%d path=%s [%s:%u] notify=%d\n", - errno, kobject_name(&clt_path->kobj), clt_path->hca_name, + rtrs_err_rl(con->c.path, + "IO %s request failed: error=%d path=%s [%s:%u] notify=%d\n", + req->dir == DMA_TO_DEVICE ? "write" : "read", errno, + kobject_name(&clt_path->kobj), clt_path->hca_name, clt_path->hca_port, notify); } From patchwork Wed Aug 21 11:22:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Haris Iqbal X-Patchwork-Id: 13771255 Received: from mail-lf1-f46.google.com (mail-lf1-f46.google.com [209.85.167.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D88C01C57B5 for ; Wed, 21 Aug 2024 11:23:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724239406; cv=none; b=UjP7nuhl41UM9Cn+IHW93Y7tqtSNwWjhZoL35Vh8JWKkH1fNf7wYqts/vDPATCXWF5BWzoYUWJj80lMA6GPp0UNu0BE0yH46i0jueUXoxE8cX1cTpr8Ux5+w3QbmAdFYNCaWqPIRxtsg7oju7zCkwM/+iRrKNyShTBuIGXyoIjE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724239406; c=relaxed/simple; bh=1LhXg5xWDvGPRaEsg0/S2HfUyvu2JUtIJZgNQ284l3k=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=dhqfb86s4R1Cl1yHTSN7birsvKXxD+iJJLoQnGNhSjXMp4jLFqUlkm3aLwPe1YcTZVx+4QRaMTlXATDDnCZqbi/Rb7gxvQl/Etn8DfxE52roclQf6VaHCAKCGJD61HX5IAqu/IAarYEB63u40sX8fFPWlc8+rGMAnX5fykwpKLY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ionos.com; spf=pass smtp.mailfrom=ionos.com; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b=Y2njDS6l; arc=none smtp.client-ip=209.85.167.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ionos.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ionos.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b="Y2njDS6l" Received: by mail-lf1-f46.google.com with SMTP id 2adb3069b0e04-5333b2fbedaso4150829e87.0 for ; Wed, 21 Aug 2024 04:23:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; t=1724239403; x=1724844203; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QlZP5HgK2ScBazOgpen0WVZnjmuyRXH0hLPE2FocaPA=; b=Y2njDS6l4iUQNeslCIXb3MGD0dYrG1eq4uxeRgHDmi5TWDXoih3kKWVZDq0IBegWCD edlbg5Fqz5CqoM1etmBzufyWPnHRJFc2q+8nS8TRySDHVbB+gS2jf69s2P02Ijf+Ug52 hujUEoTV1eu/hqT5FATZmuTwJvvwxUQEg/Js74wM7Ef6DMw0DWgFutaUsyFhO3r+9jvB ncmCpC24X8JFur0utyGfCU9RwN6S8YorSAVFIzqZlt8/+7MvawByHthq4f9p0VFNm8z5 iTPu0xQCJRKXGzTBc+rrJJsOVljLNqzZCk8damyxflpLCNtq1g1AZDjNALlJbUvGNtxk 6Vdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724239403; x=1724844203; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QlZP5HgK2ScBazOgpen0WVZnjmuyRXH0hLPE2FocaPA=; b=pqH0BZGEkpeygWxdK4ONJPer7o/8HXe3Lg28uwTnKN5D1rUxKdZMd8iaBmSqQotPbw VQ1PjrtChWTW9krwIx/H/fWSQ4+Wgzy0CCQO6IcnGpL5GXXZCdTcfttU6Ywu0unduaU4 kD3v+OH9QGElJUFXVM1jGLMFcaOcoegZst5omhVY6mFpzf7LjbI9KMmHPGwpiGfZN/tU JI0/yX9H5/YyBPJoO1Bz5CxlqQ1wxt8ixB91cheS1KOqRwecx3rp93zeXQqpVaqVMFiu 42dvuhegt3NVSWprR0NT7mNGa7KDYoYn9q3TJ2RZ3Ov7aymI0Y2pJRgjrN2sIB82ASjP YhLQ== X-Gm-Message-State: AOJu0YxH2+Mrd+HDXx5Pftfmer6UJHLVV8Sa0GxK2BF99TMSzP9bqSNU FY4IGCnFM0HBCN7Li0/GbJbgaS8MXBszpGcb7zLJKF+eoe7+i8wPYxRfo0I6RFp2K7v5e4lOOas GiDk= X-Google-Smtp-Source: AGHT+IEYtXLEE3wuknGU0Iq7743MyCEhVpKlPm073YUOdKVLgFs+BKlBLI3YieWSItAQUYRDjA9qlQ== X-Received: by 2002:a05:6512:2308:b0:52e:be1f:bf7f with SMTP id 2adb3069b0e04-533485c05d7mr1078649e87.27.1724239402180; Wed, 21 Aug 2024 04:23:22 -0700 (PDT) Received: from lb01533.speedport.ip (p200300f00f051d5f269a60e7b8956185.dip0.t-ipconnect.de. [2003:f0:f05:1d5f:269a:60e7:b895:6185]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a83838c65e7sm887934066b.20.2024.08.21.04.23.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 04:23:21 -0700 (PDT) From: Md Haris Iqbal To: linux-rdma@vger.kernel.org Cc: leon@kernel.org, jgg@ziepe.ca, haris.iqbal@ionos.com, jinpu.wang@ionos.com, Grzegorz Prajsner Subject: [PATCH v2 for-next 08/11] RDMA/rtrs-srv: Avoid null pointer deref during path establishment Date: Wed, 21 Aug 2024 13:22:14 +0200 Message-Id: <20240821112217.41827-9-haris.iqbal@ionos.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240821112217.41827-1-haris.iqbal@ionos.com> References: <20240821112217.41827-1-haris.iqbal@ionos.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 For RTRS path establishment, RTRS client initiates and completes con_num of connections. After establishing all its connections, the information is exchanged between the client and server through the info_req message. During this exchange, it is essential that all connections have been established, and the state of the RTRS srv path is CONNECTED. So add these sanity checks, to make sure we detect and abort process in error scenarios to avoid null pointer deref. Signed-off-by: Md Haris Iqbal Signed-off-by: Jack Wang Signed-off-by: Grzegorz Prajsner --- drivers/infiniband/ulp/rtrs/rtrs-srv.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c index ffd3e80596d0..05d15ff074bb 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c @@ -935,12 +935,11 @@ static void rtrs_srv_info_req_done(struct ib_cq *cq, struct ib_wc *wc) if (err) goto close; -out: rtrs_iu_free(iu, srv_path->s.dev->ib_dev, 1); return; close: + rtrs_iu_free(iu, srv_path->s.dev->ib_dev, 1); close_path(srv_path); - goto out; } static int post_recv_info_req(struct rtrs_srv_con *con) @@ -991,6 +990,16 @@ static int post_recv_path(struct rtrs_srv_path *srv_path) q_size = SERVICE_CON_QUEUE_DEPTH; else q_size = srv->queue_depth; + if (srv_path->state != RTRS_SRV_CONNECTING) { + rtrs_err(s, "Path state invalid. state %s\n", + rtrs_srv_state_str(srv_path->state)); + return -EIO; + } + + if (!srv_path->s.con[cid]) { + rtrs_err(s, "Conn not set for %d\n", cid); + return -EIO; + } err = post_recv_io(to_srv_con(srv_path->s.con[cid]), q_size); if (err) { From patchwork Wed Aug 21 11:22:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Haris Iqbal X-Patchwork-Id: 13771256 Received: from mail-ej1-f46.google.com (mail-ej1-f46.google.com [209.85.218.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E9BDE1C4EEB for ; Wed, 21 Aug 2024 11:23:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724239406; cv=none; b=JQqAJVzI5Q+i0ZCCNsBfErO9Ml4fWnXbnnIAaK6a2ueQM+b4PNHP9+VvrFdKkePsPDQGT5cQSMNX5VBfK4gIXbosr7AT0mSOSosByO5h/+K1BBP3td5lls1R7QbZmaqp2JBOHVTZkEVYD5CIq9qkj4B+vOQZyqVX62rISmw7Feo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724239406; c=relaxed/simple; bh=ToPpPF8PgRne+NZN4bTaf9yJyCjnXa/jwywqeKB/wYY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=hhNXtRHomXum2EfckNQa5Q+uQvDvrPPEqMEP6mDc7CQKfKNPxyltIXBOePFv+nt5LLRbRGF6E1LfYDMBQtAOVuWmwowWmQJNv/IfGx5GOyPdG8k5wsbBXqv7Bd24EC3k5soVz73S95iryg56RuitKSu4pGVb8vB3b5/+kiP+zJE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ionos.com; spf=pass smtp.mailfrom=ionos.com; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b=RtIe1ng0; arc=none smtp.client-ip=209.85.218.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ionos.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ionos.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b="RtIe1ng0" Received: by mail-ej1-f46.google.com with SMTP id a640c23a62f3a-a7d26c2297eso751824966b.2 for ; Wed, 21 Aug 2024 04:23:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; t=1724239403; x=1724844203; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vj3WjtP4ex7lKBF1Z85c4VX/qcuEjWxgusEnVCWcG+Q=; b=RtIe1ng0Zv3HD4SjYvXUZwfrNzDCvrwNa46qHpuZN/KajHp+evGrF07NdBQXh/nEWV XfJ2qAdaDkywmkhSlYJj8jillExdMIjadQruHPOlsu8/rKqLQVyWtBHkW/UdM6HOce9x nMZfFdOa0jcpSNSZRadzUYesi6uV6Z50uERtPpuCv6w6BLA2COY6QTBzEyKhj4B2fO8f bU1Tp5FmDkcPhBKbCOi4ikUfIH9wnjmPJdHcO1aLzKImqpmOztIaJzSxRNw4YetoMTn6 Q2iinN8TOXd3t3VBnbI6V2GvyMqAF5sqrXHHhK+87ZHIVqvjaqU2USKK14rnrN3xxo0j TAQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724239403; x=1724844203; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vj3WjtP4ex7lKBF1Z85c4VX/qcuEjWxgusEnVCWcG+Q=; b=vfd2NGf/Jn1EzU6cbJtVUqTT8Bj9d/wVt/NiUefsoRNp98NmwHvHpjyCne8BVAdfcj yjN78muCRVGZHQm4n2cd5m8oOCa1H2kzCq2ntiFFrcnRxR1jzHH83y5hzHiJyM8F2FdO w01ytF9g1Xc7jw0QRVD1vCs14XB9fYqI2k9kTVOif52CatkUihhJMVVjJLRnuDZZu9aO QTGuvrhvKkMXl/TlACIhCmay71rZHNY95TIXoTmLfPvw22Rma5M6y7zfxzB+yJfpNgV+ irv8aFODwKnYcu3dO61QtJOiSd4i+1eVdshrs/bt6qUsS92rcdmM6mC5xiocE3theDbF Y7sg== X-Gm-Message-State: AOJu0YyJJVBoswmWJ2C/lCYifP/HG1lkan8XahuiQZr80/3hei+4gIh1 awXduqFkbeU4pMtHKvBYtChVky7GFFnudiM/tJbsQYALIrP3NzO0YT0NBdMEKc4C+vizP85xs8X rMec= X-Google-Smtp-Source: AGHT+IHqz/9asCXG6QtYx3roh9DDIgCUKc+6JEJHPqQq76ekzQM2x2jvp1NNsUDoYi9C7xL16hPy6g== X-Received: by 2002:a17:907:3e8e:b0:a6f:1443:1e24 with SMTP id a640c23a62f3a-a866f342950mr170585766b.34.1724239402947; Wed, 21 Aug 2024 04:23:22 -0700 (PDT) Received: from lb01533.speedport.ip (p200300f00f051d5f269a60e7b8956185.dip0.t-ipconnect.de. [2003:f0:f05:1d5f:269a:60e7:b895:6185]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a83838c65e7sm887934066b.20.2024.08.21.04.23.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 04:23:22 -0700 (PDT) From: Md Haris Iqbal To: linux-rdma@vger.kernel.org Cc: leon@kernel.org, jgg@ziepe.ca, haris.iqbal@ionos.com, jinpu.wang@ionos.com, Grzegorz Prajsner Subject: [PATCH v2 for-next 09/11] RDMA/rtrs: register ib event handler Date: Wed, 21 Aug 2024 13:22:15 +0200 Message-Id: <20240821112217.41827-10-haris.iqbal@ionos.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240821112217.41827-1-haris.iqbal@ionos.com> References: <20240821112217.41827-1-haris.iqbal@ionos.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Grzegorz Prajsner Use ib_register_event_handler() to register event handlers for both client and server side. For now, all those handlers do, is to print type of incoming event. Signed-off-by: Grzegorz Prajsner Signed-off-by: Jack Wang Signed-off-by: Md Haris Iqbal --- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 21 +++++++++++++++- drivers/infiniband/ulp/rtrs/rtrs-clt.h | 2 ++ drivers/infiniband/ulp/rtrs/rtrs-pri.h | 2 ++ drivers/infiniband/ulp/rtrs/rtrs-srv.c | 33 +++++++++++++++++++++++++- drivers/infiniband/ulp/rtrs/rtrs-srv.h | 2 ++ 5 files changed, 58 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c index 7c6d40380638..230e5f6c8c90 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c @@ -3149,8 +3149,20 @@ int rtrs_clt_create_path_from_sysfs(struct rtrs_clt_sess *clt, return err; } +void rtrs_clt_ib_event_handler(struct ib_event_handler *handler, + struct ib_event *ibevent) +{ + pr_info("Handling event: %s (%d).\n", ib_event_msg(ibevent->event), + ibevent->event); +} + + static int rtrs_clt_ib_dev_init(struct rtrs_ib_dev *dev) { + INIT_IB_EVENT_HANDLER(&dev->event_handler, dev->ib_dev, + rtrs_clt_ib_event_handler); + ib_register_event_handler(&dev->event_handler); + if (!(dev->ib_dev->attrs.device_cap_flags & IB_DEVICE_MEM_MGT_EXTENSIONS)) { pr_err("Memory registrations not supported.\n"); @@ -3160,8 +3172,15 @@ static int rtrs_clt_ib_dev_init(struct rtrs_ib_dev *dev) return 0; } +static void rtrs_clt_ib_dev_deinit(struct rtrs_ib_dev *dev) +{ + ib_unregister_event_handler(&dev->event_handler); +} + + static const struct rtrs_rdma_dev_pd_ops dev_pd_ops = { - .init = rtrs_clt_ib_dev_init + .init = rtrs_clt_ib_dev_init, + .deinit = rtrs_clt_ib_dev_deinit }; static int __init rtrs_client_init(void) diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.h b/drivers/infiniband/ulp/rtrs/rtrs-clt.h index 45dac15825f4..0f57759b3080 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-clt.h +++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.h @@ -212,6 +212,8 @@ int rtrs_clt_remove_path_from_sysfs(struct rtrs_clt_path *path, void rtrs_clt_set_max_reconnect_attempts(struct rtrs_clt_sess *clt, int value); int rtrs_clt_get_max_reconnect_attempts(const struct rtrs_clt_sess *clt); void free_path(struct rtrs_clt_path *clt_path); +void rtrs_clt_ib_event_handler(struct ib_event_handler *handler, + struct ib_event *ibevent); /* rtrs-clt-stats.c */ diff --git a/drivers/infiniband/ulp/rtrs/rtrs-pri.h b/drivers/infiniband/ulp/rtrs/rtrs-pri.h index ab25619261d2..ef29bd483b5a 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-pri.h +++ b/drivers/infiniband/ulp/rtrs/rtrs-pri.h @@ -69,6 +69,7 @@ struct rtrs_ib_dev; struct rtrs_rdma_dev_pd_ops { int (*init)(struct rtrs_ib_dev *dev); + void (*deinit)(struct rtrs_ib_dev *dev); }; struct rtrs_rdma_dev_pd { @@ -84,6 +85,7 @@ struct rtrs_ib_dev { struct kref ref; struct list_head entry; struct rtrs_rdma_dev_pd *pool; + struct ib_event_handler event_handler; }; struct rtrs_con { diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c index 05d15ff074bb..e83d95647852 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c @@ -26,7 +26,10 @@ MODULE_LICENSE("GPL"); #define DEFAULT_SESS_QUEUE_DEPTH 512 #define MAX_HDR_SIZE PAGE_SIZE -static struct rtrs_rdma_dev_pd dev_pd; +static const struct rtrs_rdma_dev_pd_ops dev_pd_ops; +static struct rtrs_rdma_dev_pd dev_pd = { + .ops = &dev_pd_ops +}; const struct class rtrs_dev_class = { .name = "rtrs-server", }; @@ -2269,6 +2272,34 @@ static int check_module_params(void) return 0; } +void rtrs_srv_ib_event_handler(struct ib_event_handler *handler, + struct ib_event *ibevent) +{ + pr_info("Handling event: %s (%d).\n", ib_event_msg(ibevent->event), + ibevent->event); +} + +static int rtrs_srv_ib_dev_init(struct rtrs_ib_dev *dev) +{ + INIT_IB_EVENT_HANDLER(&dev->event_handler, dev->ib_dev, + rtrs_srv_ib_event_handler); + ib_register_event_handler(&dev->event_handler); + + return 0; +} + +static void rtrs_srv_ib_dev_deinit(struct rtrs_ib_dev *dev) +{ + ib_unregister_event_handler(&dev->event_handler); +} + + +static const struct rtrs_rdma_dev_pd_ops dev_pd_ops = { + .init = rtrs_srv_ib_dev_init, + .deinit = rtrs_srv_ib_dev_deinit +}; + + static int __init rtrs_server_init(void) { int err; diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.h b/drivers/infiniband/ulp/rtrs/rtrs-srv.h index 5e325b82ff33..014f85681f37 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.h +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.h @@ -132,6 +132,8 @@ struct rtrs_srv_ib_ctx { extern const struct class rtrs_dev_class; void close_path(struct rtrs_srv_path *srv_path); +void rtrs_srv_ib_event_handler(struct ib_event_handler *handler, + struct ib_event *ibevent); static inline void rtrs_srv_update_rdma_stats(struct rtrs_srv_stats *s, size_t size, int d) From patchwork Wed Aug 21 11:22:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Haris Iqbal X-Patchwork-Id: 13771257 Received: from mail-ed1-f41.google.com (mail-ed1-f41.google.com [209.85.208.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BE29F1CBEB7 for ; Wed, 21 Aug 2024 11:23:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724239407; cv=none; b=Fz1u60YehcJus4AmUgX7N+hCvKiRHZxvhgB/pg446mnOy5KOqP+vG0tAN08fsZM3f7w4vEqvKu8UaKHNdh77fFF89wbzRQuuhzo3rLGTERkKBhyYcqBGe44uRMXdW65sJNO9eDOXLWbSmvodEJsNQT5HOobnGSDNSteIIomE0AM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724239407; c=relaxed/simple; bh=xLcPXGIQYb8rDHlaDSABgBgt2ojUJwEMbr395zl0B7I=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=HrQPZqh3IRoPPB817vCBg6qWFBC22uC3N/GYFjEpi4JvLHEQS9yvKAwm/oKbu7RE7aBeUh2mJJRvPSEJwKQeWIOJIuZjqB390nl3sFHIe1EeRueTZoqDHNqUs0tIEikGic0EAWYOjlvM0mb8eG10MxeacIPPKrYAX+meCqlPhwY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ionos.com; spf=pass smtp.mailfrom=ionos.com; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b=XNsEvX1/; arc=none smtp.client-ip=209.85.208.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ionos.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ionos.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b="XNsEvX1/" Received: by mail-ed1-f41.google.com with SMTP id 4fb4d7f45d1cf-5bed83488b6so5660210a12.2 for ; Wed, 21 Aug 2024 04:23:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; t=1724239404; x=1724844204; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GQMKt+Z6oU8k718oVYzHw1t/snarDjAZbdeYqhgVzX8=; b=XNsEvX1/G0b6/Z1OHuIfGrO2K1NadShV2KmmrTFXZfDHEDHY0D9JOyqI5x3Ogam/ia tpG/aNLjlKHoUmo3biplRRF219dqjIUWuZax901tWMRPyImpbOkMeFQXQYheOriqtfAq Ry6Yb1TdL0OYk9ZpZ50pUFlIaZFAU0T0FPNnR205Uakf2Urox7seM3Sl2NIw5KY8lSK/ a2054mq148U9fH8cDiJ3BTc90qJu8Ktppit8xtP4vsvafAAG6eV1AQUqMQiOqMN+ljI9 PtN6B1BFDKt7vdKn9HegapbnledzQe26M7qoaT59QSZuEqNJTVG7rlPGOzI79L5tmpsr U9SQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724239404; x=1724844204; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GQMKt+Z6oU8k718oVYzHw1t/snarDjAZbdeYqhgVzX8=; b=itPPw9VIF4thZHJaZbhmz9H1HsDikSwT+r6dkWNKgcbbxReLMjZjbrvEHKsZhk/BMq dU129IoultNGa80MXHp7d+xwn+IXSCDh2m5tBg3hFmkHmOaEewHXV14W2zJYQj7YmDUh gelNt5BUkQnlna1DCOR4T2Fh6B+h0D7rcc8Lemddg6gHUMD+L0aitpyXQNImIkfd2pTA D2Ot/9F5ELbEt1YBiVETZR3gn6d/PNX8iRUhgLAjqVTlELnwSS2ft3EVeT8MT1jxMLvb 22VRwa4wEWKGeqmpcvpjyPQgIaQZQXMTlPco8+C7suYGSxidljE2iSg8exuZnqZVL5tj Dgcw== X-Gm-Message-State: AOJu0YzyZ4PsOU6C7A23QZdWp3pMQS7ZVu7ew9nZdpUrURazneEc6nys 5I5FPAX9ss+e+DRoanx8Ffca4UnVw9yMgD2+3R7bwUpUH8sUhg0WzFoZrYlo9D/kl8hxC6If62u r50E= X-Google-Smtp-Source: AGHT+IGDXTTnoykTio+IMFeSLV7X87G4m3Oowqfq3oV3Ef+KLkLk+zJIaxobu3fb59/DVsmz53UY1Q== X-Received: by 2002:a17:907:f1a2:b0:a7a:387c:23f8 with SMTP id a640c23a62f3a-a866f125490mr148115866b.3.1724239403641; Wed, 21 Aug 2024 04:23:23 -0700 (PDT) Received: from lb01533.speedport.ip (p200300f00f051d5f269a60e7b8956185.dip0.t-ipconnect.de. [2003:f0:f05:1d5f:269a:60e7:b895:6185]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a83838c65e7sm887934066b.20.2024.08.21.04.23.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 04:23:23 -0700 (PDT) From: Md Haris Iqbal To: linux-rdma@vger.kernel.org Cc: leon@kernel.org, jgg@ziepe.ca, haris.iqbal@ionos.com, jinpu.wang@ionos.com, Grzegorz Prajsner Subject: [PATCH v2 for-next 10/11] RDMA/rtrs-clt: Do local invalidate after write io completion Date: Wed, 21 Aug 2024 13:22:16 +0200 Message-Id: <20240821112217.41827-11-haris.iqbal@ionos.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240821112217.41827-1-haris.iqbal@ionos.com> References: <20240821112217.41827-1-haris.iqbal@ionos.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Jack Wang Switch local invalidate after write io completion avoid the chain usage of WR, this fixed the local protection error on LOCAL INVALIDATE WR. Signed-off-by: Jack Wang Signed-off-by: Md Haris Iqbal Signed-off-by: Grzegorz Prajsner --- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 20 +++++++------------- 1 file changed, 7 insertions(+), 13 deletions(-) diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c index 230e5f6c8c90..fb548d6a0aae 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c @@ -395,9 +395,9 @@ static void complete_rdma_req(struct rtrs_clt_io_req *req, int errno, /* * We are here to invalidate read/write requests * ourselves. In normal scenario server should - * send INV for all read requests, we do chained local - * invalidate for write requests, but - * we are here, thus two things could happen: + * send INV for all read requests, we do local + * invalidate for write requests ourselves, but + * we are here, thus three things could happen: * * 1. this is failover, when errno != 0 * and can_wait == 1, @@ -405,6 +405,9 @@ static void complete_rdma_req(struct rtrs_clt_io_req *req, int errno, * 2. something totally bad happened and * server forgot to send INV, so we * should do that ourselves. + * + * 3. write request finishes, we need to do local + * invalidate */ if (can_wait) { @@ -1085,7 +1088,6 @@ static int rtrs_clt_write_req(struct rtrs_clt_io_req *req) int ret, count = 0; u32 imm, buf_id; struct ib_reg_wr rwr; - struct ib_send_wr inv_wr; struct ib_send_wr *wr = NULL; bool fr_en = false; @@ -1126,13 +1128,6 @@ static int rtrs_clt_write_req(struct rtrs_clt_io_req *req) req->sg_cnt, req->dir); return ret; } - inv_wr = (struct ib_send_wr) { - .opcode = IB_WR_LOCAL_INV, - .wr_cqe = &req->inv_cqe, - .send_flags = IB_SEND_SIGNALED, - .ex.invalidate_rkey = req->mr->rkey, - }; - req->inv_cqe.done = rtrs_clt_inv_rkey_done; rwr = (struct ib_reg_wr) { .wr.opcode = IB_WR_REG_MR, .wr.wr_cqe = &fast_reg_cqe, @@ -1142,7 +1137,6 @@ static int rtrs_clt_write_req(struct rtrs_clt_io_req *req) }; wr = &rwr.wr; fr_en = true; - refcount_inc(&req->ref); req->mr->need_inval = true; } /* @@ -1153,7 +1147,7 @@ static int rtrs_clt_write_req(struct rtrs_clt_io_req *req) ret = rtrs_post_rdma_write_sg(req->con, req, rbuf, fr_en, count, req->usr_len + sizeof(*msg), - imm, wr, &inv_wr); + imm, wr, NULL); if (ret) { rtrs_err_rl(s, "Write request failed: error=%d path=%s [%s:%u]\n", From patchwork Wed Aug 21 11:22:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Haris Iqbal X-Patchwork-Id: 13771258 Received: from mail-ej1-f42.google.com (mail-ej1-f42.google.com [209.85.218.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6782C1C4EDD for ; Wed, 21 Aug 2024 11:23:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724239408; cv=none; b=Z7+ov0FuspJ3nM+pVkfXAoJfxwWkWDuic6fOvmcCpps7pIn2I/MtEIwTuc4OFReuSqdPRN8HBDfu1v5vrxnC3+ErNNjeELrLaeY9I9ftKTnFy2p0yMEfyJj2xN2CwYxzfhOh7JKPcDUSSj1Zoxsf/hEtDdSzR9bhZJ+CN9Vq+LU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724239408; c=relaxed/simple; bh=9Mhk0G5eTjvUVrGb1C1f4tU+Cv91RNdEbk4On7wexo8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=PWo2MuT67EmPJZwIvZR7xUdA3ogRpCOueCR5imV6Fg+qAwGn5UYDKuiDVc8imZjKtP53ZXGX5xbl9yqglqFetS0YKRzOtW2gMMXKFBgp/l82IoEr1jx3gnMRKaAbexIZ49573+GF3xoKXHyMGNTl0ojnG8e0ONJ4etmAi7LMbds= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ionos.com; spf=pass smtp.mailfrom=ionos.com; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b=gkYT60eX; arc=none smtp.client-ip=209.85.218.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ionos.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ionos.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b="gkYT60eX" Received: by mail-ej1-f42.google.com with SMTP id a640c23a62f3a-a8666734767so102953066b.1 for ; Wed, 21 Aug 2024 04:23:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; t=1724239405; x=1724844205; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nAw+RT5l4HLDJKlC26jq7Gm/uP6e5GmxJClzdU5sLnM=; b=gkYT60eX1/t8VOrskmE9FZj1It3+g6ud1rEwUcYFyyk7A+MxgRuwMZeUr+ZQQYkgW6 /MpQdoUPX/jq3wILODXJp74wdgWGq9ggQnZ1xRBQ4vPSmsUzMyb5Cr+FYeDoSu7qZPuF LkWw5FxGiqvMJ0P13Cr+u6FbcnAxGfCiz9rYr0Hs+liazYWphHJfWGC9Y6skmyWXm4W/ H6HLq+FIZ2aJr+rs/kbkxavCEQd80bYd9cIGBE9b8BDd3sGN2RN5lX56LgaNNdLu2pk/ uhfo/q0p9uztByrHMqox+Yb1VBf0y/gdS5fdDzFmfeegF7jHa7hwTDiQvWpOtf297Y6T 4RCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724239405; x=1724844205; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nAw+RT5l4HLDJKlC26jq7Gm/uP6e5GmxJClzdU5sLnM=; b=DY8sEHoKgVcIVWKYxat/XTUN2iDmbueqyh1tPzVKfSy2Mmzg5bXq/WANbDGclyUh2D Ngm4pfkAOZ8GccL5b5oSvDbuDaUtRHNakhwpw7NGzS0ib+lmrRVBuvGY1+2btEj1hdUI nIW/XsgBIwTYEpK38aJkTjTO+8L0PuyU/FV3iMr4iSewjDYLfBaaxJs2vAkRExwy2xr+ xX/CcNqDdsTVRrS69GUBex+eTlKdqN4jkPcu5XU0Ml0iuVv9Y1LIvR4P8DjVybzkQ1Ao LIdDCQGQ+cK/nUwzLJUy2DewVRwbhjzaHTIEVEcDEcUqhgyesOe0e6JHa/2hFLnM23PG SBuA== X-Gm-Message-State: AOJu0YzgQnDCtZVETJS9i4NA7jGcLcrRZzqapFm3VlGjvd4B0ruzpJ5r 8jgX3+TT1JEF+LumpK34Q598IeS8sebe/4e52bE/Iiv9nt15kVFVxjCqlQnpAzjs5ySdFevIcMx CLvA= X-Google-Smtp-Source: AGHT+IEt0aFD92I2QWlq8G4DDeCfTkuH67HgtPhfVcMKsFPTifLXzo+XYy8VgIEWJlW/b0MDKnnbCg== X-Received: by 2002:a17:907:1c1d:b0:a86:4649:28e6 with SMTP id a640c23a62f3a-a866f8acf7cmr131214566b.57.1724239404327; Wed, 21 Aug 2024 04:23:24 -0700 (PDT) Received: from lb01533.speedport.ip (p200300f00f051d5f269a60e7b8956185.dip0.t-ipconnect.de. [2003:f0:f05:1d5f:269a:60e7:b895:6185]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a83838c65e7sm887934066b.20.2024.08.21.04.23.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 04:23:24 -0700 (PDT) From: Md Haris Iqbal To: linux-rdma@vger.kernel.org Cc: leon@kernel.org, jgg@ziepe.ca, haris.iqbal@ionos.com, jinpu.wang@ionos.com, Alexei Pastuchov , Grzegorz Prajsner Subject: [PATCH v2 for-next 11/11] RDMA/rtrs-clt: Remove an extra space Date: Wed, 21 Aug 2024 13:22:17 +0200 Message-Id: <20240821112217.41827-12-haris.iqbal@ionos.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240821112217.41827-1-haris.iqbal@ionos.com> References: <20240821112217.41827-1-haris.iqbal@ionos.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Jack Wang No functional changes. Signed-off-by: Jack Wang Signed-off-by: Alexei Pastuchov Signed-off-by: Grzegorz Prajsner Signed-off-by: Md Haris Iqbal --- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c index fb548d6a0aae..71387811b281 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c @@ -1208,7 +1208,7 @@ static int rtrs_clt_read_req(struct rtrs_clt_io_req *req) ret = rtrs_map_sg_fr(req, count); if (ret < 0) { rtrs_err_rl(s, - "Read request failed, failed to map fast reg. data, err: %d\n", + "Read request failed, failed to map fast reg. data, err: %d\n", ret); ib_dma_unmap_sg(dev->ib_dev, req->sglist, req->sg_cnt, req->dir);