From patchwork Fri Jul 1 06:10:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Zhijian Li (Fujitsu)" X-Patchwork-Id: 12902742 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC1E0C43334 for ; Fri, 1 Jul 2022 06:10:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234600AbiGAGKi (ORCPT ); Fri, 1 Jul 2022 02:10:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43562 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234614AbiGAGKe (ORCPT ); Fri, 1 Jul 2022 02:10:34 -0400 Received: from esa10.fujitsucc.c3s2.iphmx.com (esa10.fujitsucc.c3s2.iphmx.com [68.232.159.247]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 85E2113DED for ; Thu, 30 Jun 2022 23:10:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=fujitsu.com; i=@fujitsu.com; q=dns/txt; s=fj1; t=1656655832; x=1688191832; h=from:to:subject:date:message-id:references:in-reply-to: content-transfer-encoding:mime-version; bh=QvdLTCjo5zlbS2jZ7uRlObpl8qbBKPB+I5xiTw592tw=; b=yZthBsqiF+cgP+uuJCCi2S+JUuHQmer7YARjNiv/QxfgeSNys0CwZdMo M3z+PWtT+FI86otpznlhVo18dFTyK1/Br/crIq4LRiKa7icDkvuesxeZ2 VmWyOrByWPbVCELrFIpmd7CXhuqm7jq71VSE/CcOum9ntW81XLZjXEcIg zxdRa4WGKYPjOhw7hY3iXAqYsWKvMlec4WS1xe9ktdgxHVlYkXaNtszho m9wfO6h4fKvXvDh66gIKkT5tNNxvKvj/dDvUlVIj7SEDtpW/glAphnnNw Dqn3oHVGHJF3Z2e+mibM1rEv03tCyNf7A3xZIEPNBs4xhGI1DBCbkuKp5 Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10394"; a="59519676" X-IronPort-AV: E=Sophos;i="5.92,236,1650898800"; d="scan'208";a="59519676" Received: from mail-tycjpn01lp2177.outbound.protection.outlook.com (HELO JPN01-TYC-obe.outbound.protection.outlook.com) ([104.47.23.177]) by ob1.fujitsucc.c3s2.iphmx.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Jul 2022 15:10:29 +0900 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HHRcNd1YOwJNBC0a3c+FdRftFivwdYOlRkA99fajCp2pLLfbktX0j7lLEd5gxXxgHRq1l2sdlT6e9hkZobAiIIIqF/ivLMAjwe2kEK/OTIo1jYDbc36ye1dW+FCPTLlR2VXxXfyiOYamzPGoSXo4GXTXnbwESY66PseEjYv2c2KxbAZeTCtzigOOlmeJFBE9Oum0Ndpg8j87P7HbFKmJknDETr9wxgDCai7U8f4Z/CZVn9QZP/IoDzMbcAJni8FbAJ3RsrC6igxcPRj7JF8kpeZ8KZf49mO+bYH2au/XBlBVvxIygwtAyA+/oVuhBGl91x7sPiUzrUYXppaJXknlSQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QvdLTCjo5zlbS2jZ7uRlObpl8qbBKPB+I5xiTw592tw=; b=mIpn3dZrFpysoLiTf12w0fTXRPfsectEDUqS4M23We/4REJWoRO5mOUDylQ5HFrNz/zXcYi1beQkBqVIPLt0G07t9o5oEcq4wUBfBfuelwFxOoGNyVymtvpv9Yql7E9vxU1iGD1ZMwmwHsoVb0XfkKx7qknRx5YtAH5hDwqR4F4ZdupWVS12UVq7thP3jhVs3fLf88YGnjpuFVGywzSNHRIWnP6ycy68nxfbE/Hou1jAs5VIux6L6Jvk3bLn2YKsRrOk7VScsSZIknIFkVOsVkyI+KoXsHLFy/68g1sKjTj93lSyAFHVmX1aIU2kzCY27L4fgzxHexX2RZQqAq1vNA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=fujitsu.com; dmarc=pass action=none header.from=fujitsu.com; dkim=pass header.d=fujitsu.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fujitsu.onmicrosoft.com; s=selector2-fujitsu-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QvdLTCjo5zlbS2jZ7uRlObpl8qbBKPB+I5xiTw592tw=; b=eDmUflcKXTMUv64v9WHliZbUVY5J0/aRQz1s9g+32gWGcWX2uH0Qbe3zJzdSaqtGZ07AI7mpyyqCo2LtJE2nBRAGWaPBgJiEAXacwxv7VPOBLQIhZPOsJxX9N0DfgIdgZToErf8g7m+cSgal7N1t6oDprKCPQabZT+fd8/42oTA= Received: from TYCPR01MB9305.jpnprd01.prod.outlook.com (2603:1096:400:196::10) by OS3PR01MB6626.jpnprd01.prod.outlook.com (2603:1096:604:10c::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5395.14; Fri, 1 Jul 2022 06:10:22 +0000 Received: from TYCPR01MB9305.jpnprd01.prod.outlook.com ([fe80::8d7a:baa8:3b18:cd93]) by TYCPR01MB9305.jpnprd01.prod.outlook.com ([fe80::8d7a:baa8:3b18:cd93%6]) with mapi id 15.20.5395.015; Fri, 1 Jul 2022 06:10:22 +0000 From: "lizhijian@fujitsu.com" To: Yanjun Zhu , Jason Gunthorpe , Haakon Bugge , "linux-rdma@vger.kernel.org" , Bob Pearson Subject: [PATCH v4 3/3] RDMA/rxe: Split qp state for requester and completer Thread-Topic: [PATCH v4 3/3] RDMA/rxe: Split qp state for requester and completer Thread-Index: AQHYjRE/TP5Oo7sEp0ak3lQusoVWIA== Date: Fri, 1 Jul 2022 06:10:22 +0000 Message-ID: <20220701061731.1582399-4-lizhijian@fujitsu.com> References: <20220701061731.1582399-1-lizhijian@fujitsu.com> In-Reply-To: <20220701061731.1582399-1-lizhijian@fujitsu.com> Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.31.1 authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=fujitsu.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: f3628d7f-6831-41ff-23c2-08da5b2861b3 x-ms-traffictypediagnostic: OS3PR01MB6626:EE_ x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: EW8wvLW+2U2CCVKAeJ2nKGKvUnbK39zWKcOxYGRaVuQzkntCO/+cPNvXTEaNwvpnlwaYpsuAylHKBAZvS+HAx6Ootx1ztj8IpWWiosTxrEyj26ebHgPhnRKlBQB+xeNS9REB6SIq3omsR5tcCcNHSwwoJwpF+pIWwvPjlcEqzTDDYbsDw9+km/bRcMRXlRHWbyWngn8KKg+afpSFWnQaUVwxG5hmKk5a7WkitaGURw8lHaPPjkHKfrFg6rzDsmvdOX5Ge1rscN4zHYE/Wgj3xwmQQY8Ni0Hx0RtFPOOBUMn2E2vvXiG3nD8zfLdx2oujNvtJj5+wjFvRtlxFA0O9GpfUHDd9itYxPxxNuF0/s4v49vPs8InuplOvDWNS3ODuRezXx3oxlkWi1tiZIx93bgfgxEm90e14uVVCDpMIbd+dMUkfrkxT/FVmWQE3qLijcxQIKpcGyb5KUUoMKnZl/6Tl0pAoKc6XX8v8XeGVD0pQ1fpOJN5AmiEag7LE8F3ph7fvFHGk+GOofaGb7GpZ6b9KKN6JOCyYrs0Cmu8XsqOrRBQGX3fH+HIskMZqgHc8Kywk+cVrZMHHDmM7BGIAgpREo7sYJGqP+EJZ9O/voX8d2YwnKs5SCF3SeJKfymHVQRPb4stS9ykg6/FKR+4zBf39q2+ZzdbpWPAgJr0SYG+2oBREJ8rvaXsTenWh5IK5WeC8FkRPT5muEAyiQ3oYctfRoyEAKGLPVNj80S3FZ0WGjEBkI80U5FO96FS+K1TCUqofr0/D4Eec96Y7a1ObW+OokTLYPrpQVxlUhPTVylG+2RBfOIuv9CfrtEIxOfD0 x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:TYCPR01MB9305.jpnprd01.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230016)(4636009)(346002)(366004)(396003)(376002)(136003)(39860400002)(5660300002)(36756003)(64756008)(110136005)(8676002)(122000001)(8936002)(2906002)(38070700005)(66446008)(91956017)(186003)(41300700001)(83380400001)(85182001)(66476007)(316002)(76116006)(66946007)(6486002)(71200400001)(66556008)(82960400001)(38100700002)(478600001)(86362001)(1076003)(6512007)(26005)(2616005)(6506007);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?eucgb2312_cn?b?M0pBNTkzL0RJNU1CMHlXTlov?= =?eucgb2312_cn?b?VFZ0WlhKYkRBd2UyN0E3RU5tS0FzVWNsVHFIWC9GQU1MaTlVWUh0bFMyS2ZKMVVz?= =?eucgb2312_cn?b?RmQwSUJOLzBXY2NkZUUvMUVBY2JCdDFtRHFVQkkvT3dQbXdpUXdpbDRmNmtCbjlX?= =?eucgb2312_cn?b?ckJvTHJsc2NHR0lyWG90TFhkRHpNTWFnUm5DcGtqMFlTOWRrcllCZGE4MlhTWHRz?= =?eucgb2312_cn?b?Uml5a2dFNXpkc3VMNjVYbUQrMmxMNFAxd0Zyb2pJTDVrb3hZQ0xJOGVlRzFFdkZR?= =?eucgb2312_cn?b?RHdkUVBWQlZiOUlKQ2Z0MUJoeUMvSlp2YXpDcXBkNHFMZ0hGNHZETmFRTGVWMkpX?= =?eucgb2312_cn?b?cFZyem1UL2lZUHZSaUJYZUFFZ1N1OFNreGk2cmZrbFVlc0plQVkxbFVMWEtNZktt?= =?eucgb2312_cn?b?OUk3VDJJUHlzcUVYcnpmOWFzVUVZQjBOSThjcFlzMkNQb1VQa2syZnhlMU0wQUI5?= =?eucgb2312_cn?b?aUJualpSQmtKNzEwMW8wUjhyTk96L0ZxOGt1aU92Vkxub0tMejFDQlc3WU9HRkdR?= =?eucgb2312_cn?b?cDZUeVlaYTlPM1IyanMrV2JpTXVNdWwwd0wwT0E1Ri9QTTVVVWRZbjJYd3JzS0E4?= =?eucgb2312_cn?b?ZEFaZzEwWnEvOHlFUkgySUxMSmVLbUczclU5OGdKUXdSMU8zMm8rMG1SU1RFTEVi?= =?eucgb2312_cn?b?VzNFb3BRdnJwRnZrY0UzRTdEajdoSFJ0UHpyTHM0SUJmYUE1cGppclNmamM2NHdz?= =?eucgb2312_cn?b?bDgrQWVTRlVTdkh5eWhUbCtvY2k0eXlTVE4wR1hSL2lwcW8reXQvbm45cllFS0JN?= =?eucgb2312_cn?b?UHBoc0Z0L3dSVGxXL0hMMVBXUmgxQmFHMFI1bGRVa3NxaG1LVTFTUFlTb0JMOFJD?= =?eucgb2312_cn?b?ZXhmVDFEK2FTeXphajVrUXFmaDc3VldSLzZUSjM2dEZ3bDlFL1hFRzhsOExER3g0?= =?eucgb2312_cn?b?UUhVYXFaSHVoYjN2bGtGZlY0WUs4Z3BtNTYvdzJ2bVlaZjUvTDNPRFhPclYveFNl?= =?eucgb2312_cn?b?RFZFMlVwWFR4K2dFandKL1dOUGd6bTZxbVBaRytDemZKeCtKckRxSW9vQm9lVnB6?= =?eucgb2312_cn?b?OWxVTi8vaysrbllmTlZlL2l5UlNkYlJZTUxobHozcmFLRmN0QnQ5djZxWVcraW1C?= =?eucgb2312_cn?b?THFJdDBpMFdqemRZOFpXRThIOEJGYU1yM21keTdJWWIybUVUTVRqc04rSW8ydlBS?= =?eucgb2312_cn?b?RDNPQjR2TDJCRmttWlgzNlQvcWtFMXB1bW50dGx0VlhQZ3VkTW80VVBNQXB2TkV5?= =?eucgb2312_cn?b?Y2U5Vm9uUTY5NUpSeVA3d1FpdTgrYzBYZm56NkVPNWRheDZESnVZK1B5eTBnSDJm?= =?eucgb2312_cn?b?cHpjeWRJb1JoZFFUYzMvWndyY2d4TEdaL1psS3JzZU80L2plNkVBYnNQQVpXQmJs?= =?eucgb2312_cn?b?NHZqSFBLMmV6SmJNMHNkdnRENjJhRElYYWFDRlRXbUVObDBCS0pyajFTYm1vdU4x?= =?eucgb2312_cn?b?eUc3c0ZFbEpqcXJDNmtGMGV6RlYxeEZMTGdLeUdPNE94SGZ0dDNpUXYwYVppQXdL?= =?eucgb2312_cn?b?RHJvUWdGQnhXcG0reHlWaHZWOXdHeEttMnV5TnVSUkJ4ZHR5L2ZKdXR6QnE2eDFI?= =?eucgb2312_cn?b?WEx0WDk4bENqWG5jbTNGbGVFUC9rZEJVeFVMTXFvRU1ScHhCdHZvK1JLVktZTk9B?= =?eucgb2312_cn?b?WmIxTmxiUlpjODhXKzY5UWFwZDVaWFZ5MExpZ3BSNFBjYkczanBQRVA5M1BRWGlG?= =?eucgb2312_cn?b?MmZtT1NjQWZubkpNTE1WOWlZUWd3WjhobEM3NnBDRDNCODRURXVKWmlJdnNvZ2pl?= =?eucgb2312_cn?b?Z2xQbWYxS0VoNkdpMWtWZCtSb1NOL291ak4xdXl5YVZoa1hsQm5OWStFMmdpcGZQ?= =?eucgb2312_cn?b?ZnZmN1JraitJT0ZMSVo1UGIrbXQyczNYalB3RnRYaGhWVktBR1d6enpQYVQxTzFu?= =?eucgb2312_cn?b?MGpoY0Zobkl3VVpNM052Mi9NRzM2VDdVUGlRWXRZL2kvcE5OaDFoeVNxajVrKzhE?= =?eucgb2312_cn?b?cnJlelB2RCt1Zzl2Z2g5cEt6V1NwUWhDdkNxaUdMNWNDRkJVVDVnR1VGbnpyUWxY?= =?eucgb2312_cn?b?SVphWUVTSGIyUm5CZUJtcExzakRxWml2OWozYzVET0pyREJVUE5WVXdQR3M1NWZN?= =?eucgb2312_cn?b?NmM3QS9Ea1cyN0pZVWhuUnk5L0VoL1NQVXZZSGROdWNSZUhMZmdBR0NiT05TOGFU?= =?eucgb2312_cn?b?R0gvWXZoWUZqMXgzY2Z4RURhV3c9PQ==?= MIME-Version: 1.0 X-OriginatorOrg: fujitsu.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: TYCPR01MB9305.jpnprd01.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: f3628d7f-6831-41ff-23c2-08da5b2861b3 X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Jul 2022 06:10:22.1514 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a19f121d-81e1-4858-a9d8-736e267fd4c7 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 02BLG0nQ5JRCqmHy/RCVnTQnLA2Yx7FV+xg7jviXAs2NCgGpXESbLmSh+ytO8iHKl1JbdnjisAcqAGMH1Fe61g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: OS3PR01MB6626 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Bob Pearson Currently the requester can continue to process send wqes after an local qp operation error is detected because the setting of the qp state to the error state is deferred until later. This patch splits the qp state for the completer and requester into two separate states and sets qp->req.state = QP_STATE_ERROR as soon as the error is detected before another wqe can be executed. Signed-off-by: Bob Pearson --- V4: new patch --- drivers/infiniband/sw/rxe/rxe_comp.c | 6 +++--- drivers/infiniband/sw/rxe/rxe_qp.c | 5 +++++ drivers/infiniband/sw/rxe/rxe_req.c | 1 + drivers/infiniband/sw/rxe/rxe_verbs.h | 1 + 4 files changed, 10 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_comp.c b/drivers/infiniband/sw/rxe/rxe_comp.c index da3a398053b8..0b68630a3e49 100644 --- a/drivers/infiniband/sw/rxe/rxe_comp.c +++ b/drivers/infiniband/sw/rxe/rxe_comp.c @@ -565,10 +565,10 @@ int rxe_completer(void *arg) if (!rxe_get(qp)) return -EAGAIN; - if (!qp->valid || qp->req.state == QP_STATE_ERROR || - qp->req.state == QP_STATE_RESET) { + if (!qp->valid || qp->comp.state == QP_STATE_ERROR || + qp->comp.state == QP_STATE_RESET) { rxe_drain_resp_pkts(qp, qp->valid && - qp->req.state == QP_STATE_ERROR); + qp->comp.state == QP_STATE_ERROR); ret = -EAGAIN; goto done; } diff --git a/drivers/infiniband/sw/rxe/rxe_qp.c b/drivers/infiniband/sw/rxe/rxe_qp.c index 22e9b85344c3..a95d3b49ae20 100644 --- a/drivers/infiniband/sw/rxe/rxe_qp.c +++ b/drivers/infiniband/sw/rxe/rxe_qp.c @@ -230,6 +230,7 @@ static int rxe_qp_init_req(struct rxe_dev *rxe, struct rxe_qp *qp, QUEUE_TYPE_FROM_CLIENT); qp->req.state = QP_STATE_RESET; + qp->comp.state = QP_STATE_RESET; qp->req.opcode = -1; qp->comp.opcode = -1; @@ -490,6 +491,7 @@ static void rxe_qp_reset(struct rxe_qp *qp) /* move qp to the reset state */ qp->req.state = QP_STATE_RESET; + qp->comp.state = QP_STATE_RESET; qp->resp.state = QP_STATE_RESET; /* let state machines reset themselves drain work and packet queues @@ -552,6 +554,7 @@ void rxe_qp_error(struct rxe_qp *qp) { qp->req.state = QP_STATE_ERROR; qp->resp.state = QP_STATE_ERROR; + qp->comp.state = QP_STATE_ERROR; qp->attr.qp_state = IB_QPS_ERR; /* drain work and packet queues */ @@ -689,6 +692,7 @@ int rxe_qp_from_attr(struct rxe_qp *qp, struct ib_qp_attr *attr, int mask, pr_debug("qp#%d state -> INIT\n", qp_num(qp)); qp->req.state = QP_STATE_INIT; qp->resp.state = QP_STATE_INIT; + qp->comp.state = QP_STATE_INIT; break; case IB_QPS_RTR: @@ -699,6 +703,7 @@ int rxe_qp_from_attr(struct rxe_qp *qp, struct ib_qp_attr *attr, int mask, case IB_QPS_RTS: pr_debug("qp#%d state -> RTS\n", qp_num(qp)); qp->req.state = QP_STATE_READY; + qp->comp.state = QP_STATE_READY; break; case IB_QPS_SQD: diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rxe/rxe_req.c index 7fdc8e6bf738..492332533892 100644 --- a/drivers/infiniband/sw/rxe/rxe_req.c +++ b/drivers/infiniband/sw/rxe/rxe_req.c @@ -775,6 +775,7 @@ int rxe_requester(void *arg) /* update wqe_index for each wqe completion */ qp->req.wqe_index = queue_next_index(qp->sq.queue, qp->req.wqe_index); wqe->state = wqe_state_error; + qp->req.state = QP_STATE_ERROR; __rxe_do_task(&qp->comp.task); exit: diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h index ac464e68c923..bbfffe243fd6 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.h +++ b/drivers/infiniband/sw/rxe/rxe_verbs.h @@ -129,6 +129,7 @@ struct rxe_req_info { }; struct rxe_comp_info { + enum rxe_qp_state state; u32 psn; int opcode; int timeout;