From patchwork Sat Oct 15 06:37:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xiao Yang X-Patchwork-Id: 13007608 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89486C433FE for ; Sat, 15 Oct 2022 07:00:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229454AbiJOHAc (ORCPT ); Sat, 15 Oct 2022 03:00:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58132 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229550AbiJOHA3 (ORCPT ); Sat, 15 Oct 2022 03:00:29 -0400 Received: from esa10.fujitsucc.c3s2.iphmx.com (esa10.fujitsucc.c3s2.iphmx.com [68.232.159.247]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0E94031359 for ; Sat, 15 Oct 2022 00:00:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=fujitsu.com; i=@fujitsu.com; q=dns/txt; s=fj1; t=1665817227; x=1697353227; h=from:to:cc:subject:date:message-id: content-transfer-encoding:mime-version; bh=uSt7brgoF+lCK5YiTsEm/CMCapYTRkTOwdDaK1BWu/o=; b=KOPSrj1xrcnQ9R7pM6JxAz1MbTYCfcfBQiApfkJpR2gZxaWj2QdTeggc Yb+ZlsD8Lck/y06h5CHvV90wmtqyjJ/E9YQkUymPs+f/3n5g0C2w82oNj A/T2Se9P8czFrgcQynEHncZtTXpTap4wyyO4lUTw/MT7wB9WoMznAf8LW 8Ndj8Ug9YszmjKdtG+kExSpNBgDtOLR2gr/wL9jSkbLQ416gfLRquLogJ EFT/Qm0BMU7KPuy7G0TAPiMFTAeriVBXPt3DTOHOkEnKmWwAGZ67aBnhn jXUcupKCSgVQKIQbA0PgCOXaB+wyZvkGW7YeZW8DaaMUgDL9Bp4Te4z1i A==; X-IronPort-AV: E=McAfee;i="6500,9779,10500"; a="67689746" X-IronPort-AV: E=Sophos;i="5.95,186,1661785200"; d="scan'208";a="67689746" Received: from mail-tycjpn01lp2170.outbound.protection.outlook.com (HELO JPN01-TYC-obe.outbound.protection.outlook.com) ([104.47.23.170]) by ob1.fujitsucc.c3s2.iphmx.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Oct 2022 15:37:07 +0900 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=al6Ar50kPC/0k0IWGww38OfkakpZKVNbPABWkTrFAAGlDAYqrFYO2X5XrS3Lb2jQNO/75x99zQS0KTWBF9ZQJ7AYsYyamBAOUsQ9XgiC9Sgwj52FKfXWyub4VmZ/KgHvZ5LBKbvG/Fh1dc6Maft78UExtsXVaiur39E9NosoEYSKI51t1hYzX1Hrtl3OyoGR0PvuLpRspmsYQLV/7nTzffQvbDr5aJs/OmsT6CF9CtiIupPR410/qC4TfYbG8gU/AaZ7U36/uWvPo1zm9hQZCLoI7bFN/OedY29AfsvoHdkmqj/kAen7iVvrVE0+iq4QK87WSVj5oErIZlBpPD+csw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=uSt7brgoF+lCK5YiTsEm/CMCapYTRkTOwdDaK1BWu/o=; b=btDru/cW0kV1Mkg5OnxrP6qmIvP7TMt0Nb2Ic4DIoDL3uQmzjYCbETa/ezepODjv9NCUqQq3hqI3dpNBPNgAXJTMOasgeM/FTe0jE+pDzpRCtqGAYKCsgBOaxkrtpevLO6neWJKfjYn8Ovqkku9NBVecgIg4D790UKu1GqGlAewRwyntmfmK3ZJx+L0QtmOEpgn/WlFKqYXxsuxuR0kdRQzkuOnXeswdDL5SQMkOLYjl+hrJdjcCaoYCN7D6YdJN0fwZsyeU+8+JABwNK7mrFJ0hEf2YNWacKf9OoWnLETx0kFhHeAqLAB+G9EiRjkOuhs8h9CIEgDMqIV4pk25DHQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=fujitsu.com; dmarc=pass action=none header.from=fujitsu.com; dkim=pass header.d=fujitsu.com; arc=none Received: from OS3PR01MB9499.jpnprd01.prod.outlook.com (2603:1096:604:1c8::5) by OSZPR01MB8661.jpnprd01.prod.outlook.com (2603:1096:604:185::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5723.29; Sat, 15 Oct 2022 06:37:04 +0000 Received: from OS3PR01MB9499.jpnprd01.prod.outlook.com ([fe80::5989:c90d:abf4:e100]) by OS3PR01MB9499.jpnprd01.prod.outlook.com ([fe80::5989:c90d:abf4:e100%6]) with mapi id 15.20.5723.029; Sat, 15 Oct 2022 06:37:04 +0000 From: "yangx.jy@fujitsu.com" To: "linux-rdma@vger.kernel.org" , "jgg@nvidia.com" , "rpearsonhpe@gmail.com" CC: "leon@kernel.org" , "lizhijian@fujitsu.com" , "y-goto@fujitsu.com" , "zyjzyj2000@gmail.com" , "yangx.jy@fujitsu.com" Subject: [PATCH v6 0/8] RDMA/rxe: Add atomic write operation Thread-Topic: [PATCH v6 0/8] RDMA/rxe: Add atomic write operation Thread-Index: AQHY4GCJKHXhw3vt20eBf1hwuOqTxw== Date: Sat, 15 Oct 2022 06:37:03 +0000 Message-ID: <20221015063648.52285-1-yangx.jy@fujitsu.com> Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.34.1 authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=fujitsu.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: OS3PR01MB9499:EE_|OSZPR01MB8661:EE_ x-ms-office365-filtering-correlation-id: 30a52455-766c-47ac-bdd4-08daae77ac2c x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: blvdf3rCaL2vng8hROiU4uOVNsI/U12DW/XNTrhSgCCCvVvhdyrPkTijahOx88KkVDlp1cTXdYOCU1rJuCX9T+/sSoxqE5igTY76LrE0K3KjQ6aNbttfpD6/WQIgm0efvc2snbZm4SLMDsiXlzpEkLKYM6J+QSi/IsAZb/8kstjNuV7YGnDVIaogNgvuv2q5b6VZNDHZT/cRMTgoxMqW/q3NAAAMKW1mUXNxNSLrRSoBfSWb8kVxJoRMT44jfAVA3/XBXsxAzMk6yquAuRNKFRxcBATsxAX0We1iPWpOIOF/fFI9YEsHtfBY2n9vIxwAe8X4jHJtk+OvRqa+9jGCJUDgAo4mA1RoXgQHNrT4ARuxuOXXCGVN8iAji53DnnpGirFUUMvVBswcwzaAqk9k6h7RTm5ZXeyQzLzm/EpVHPdEEb+x2XlaksAe6N6zyt0MQdCj+vpjA6JXJDvFufDz4wM+v35BYv7/caI82I97tb4h9y+aicymll751GM5xos/C44Kq2fPA+7tH/cDBzmGsS2Ry/EN3iRWiJiiULwCH0GtvCQ78WlUUDRFvVJtnueWPdjT+MAY+ce2GWLGaGLSOnBzKtfVIjAwDdhyar9vtDLOxoObJp5B7fVCp/zcqq7hkn9OAwONM7d5S8f/FzznnfwycCAzv+M2eaCs93Qwjn1SLNI6WwWoTcEd5jgBhoCx7oiOj8lBQwFSAO9ikaVfZCATRZjnntSOio2QrcuPAEHbmlgOh1+McFlc6W++oz9vQKZtISiNsg2ue8iifz4GKh9xYH+GyvN0zceYVehELW7Ld8iqZbDSqaGmZDIy6SUoTytXA88REez74g6F6E/unMx+rrkd3Lg/LsQwDImKrDYbHPfw7vlDRzIq1zP1Hmscxdm2U+xDlv/zLbgpRdaY3w== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:OS3PR01MB9499.jpnprd01.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(39860400002)(136003)(346002)(376002)(366004)(396003)(451199015)(1590799012)(6486002)(85182001)(316002)(86362001)(966005)(38100700002)(36756003)(5660300002)(110136005)(26005)(6512007)(54906003)(478600001)(122000001)(83380400001)(8936002)(82960400001)(91956017)(1580799009)(66946007)(76116006)(2616005)(107886003)(66556008)(66476007)(38070700005)(41300700001)(71200400001)(6506007)(66446008)(64756008)(8676002)(2906002)(4326008)(1076003)(186003)(43620500001)(15398625002);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?eucgb2312_cn?b?dUx2SDBQRzRna3VVRHdXek5i?= =?eucgb2312_cn?b?N2lPZE5hVkM4WWcxTTl5L2Jsb3ZsZCtuaSswQkV1Q1VsSXlqdWE0K01ZVlJYcmRh?= =?eucgb2312_cn?b?OXZ1aFByUjhJTzN2aXB2VnBBTWZyUExUeDNlRVBSS01hN1ozL0c4a0hqdVFzS1lZ?= =?eucgb2312_cn?b?dU1aR093eGM3MnQxdzVTWWV6dW5rREZON0ZPdmlobzUzdTliZXphQU1XY3d5VG1B?= =?eucgb2312_cn?b?RHlYMmVEVXpYclJYNHBHOUpCZWFFb0h1cWpBcUY0R05vQUF1Z2ZBVlFWU0x6TFE1?= =?eucgb2312_cn?b?OENZUlhGTk9NZ1hSTk9RYi8vMmlFL1RTZnNub0NUR2h3bUpZUm5GdmtOQnprTmVD?= =?eucgb2312_cn?b?YVBTZTlZNVVBY251L3BPTzBxNGtqY0cwZkRrdXphSElQWXN3MDNsZEU5S1Fqb2Vn?= =?eucgb2312_cn?b?MnkzaWptWDBXeGJKVUVvQk0yVkQ1cWxUNUEvZVZIcm5KWkdvWW9wdVl4akZaTVYr?= =?eucgb2312_cn?b?Q2FsNlp6VUVRbmdZemdtU09CNlZwYmZjc0V0SlpRaENmZDRZL0R2clhpZzh2N2lr?= =?eucgb2312_cn?b?bWY1NE5CVTlXUVhwd2VzZVVJckVaZW12TTU1dURzalZCVlhnOXFlTEZkZFluMGlo?= =?eucgb2312_cn?b?SnhVZHNobWRJVWpkQzk2UGpPMnIvZ1dQOUlaTUJ2M1Y2cnhPbUlSeERoT3NvbVNj?= =?eucgb2312_cn?b?MzhnRmZoSWpRUXNTbDl4WFF1bG5YMGh3Z0R1OGNrR3kxT01wSzBWbGtwMXRsNFNG?= =?eucgb2312_cn?b?K1dldjQ5ZURLSzd4b3VMYnNMaUxDK0UwdGNCUGZvcW93Q1VnWTJsdy9ONDdwSTFF?= =?eucgb2312_cn?b?N1hLR2MrY2FTa0tsVDRGMUJwSWdQbFBVVXFpNzFIeFQ3ZklFUkc1aGt6QzhIR2x2?= =?eucgb2312_cn?b?aWZ2dzJZWk12bnJudHFpN3Z3L3orRFRKVDc0SWpkdGlFbktwS1dxNTlXSGZ2blU2?= =?eucgb2312_cn?b?UVlaWk12RmIwemZPMjdMd0RpZ3JXcHQrQStTdWZSV09Ka0RPcHgwMzluNU1BQW8w?= =?eucgb2312_cn?b?OVFqNTRSWi9ZSzg4N0hoazNqMDlIN040M2NjU3kycWorNTFKV3hFb3R2NmZUZUth?= =?eucgb2312_cn?b?RllLRVhqMGNaN1hmdmZnWFhlQ092NjFaTlk1dnFkd25BaU1xb0JwNzNQb21JVVJm?= =?eucgb2312_cn?b?S21Jb3BzNjdBV3NOam9qcTlOeFdpenpRdEFsMkp5V1ZMR0tPckcra2IrZ1M3QmUy?= =?eucgb2312_cn?b?Vk9WVFJKQzRPVk01aTVXOGZTQ1BiaFZUbGxnN0NEVWgwTXVneUYzbDhnSzZMZURm?= =?eucgb2312_cn?b?WEJrVUhJNzBTS3hkbkR5MmlyLzRFbzJVZTFZT2xLc3FLZ2hraGdtN0ZVanlUM2pO?= =?eucgb2312_cn?b?aUFNVGVyUW9jZHo0ZkJDZGFwT2xQR3dlWGl1cHhLZjFqTExGazNPRjhGVEJZL0Y3?= =?eucgb2312_cn?b?RjZuTzZ4VllTZnNCSFA1dUpXdjRNMmVjRTNBKzRCclltLy9McnVQWWxlaVZFalNK?= =?eucgb2312_cn?b?MEtURGpwSTluZjU1MGljYnJJZVgzZVRxTmtRZ0U1SnlRVkpBV25icS93anV6OEFm?= =?eucgb2312_cn?b?OUtTY2J3dU55MXFIb0o1M0hRanhDMmI4TmlaeFJRQUZZVjJlSlhuWUhpU3RXMHRH?= =?eucgb2312_cn?b?U1Y0bWZDOUxnUUVFdmcvdXZ0N25UaTFNSi96dUdXcTNDN0hRT2x4MjJqU1ZHZzR2?= =?eucgb2312_cn?b?NE9zNVRtbUF6ZWRrendxVWFhVmJvYVcrdEJRamU2RG1OaU9qSGdkZUVwQUx4cTg2?= =?eucgb2312_cn?b?SjAzRXk0L0g0Sk0weXdWWmIwa0UwT3RyK1Fta1hBSWRNWndIZ0JQZzUxMGliMEJk?= =?eucgb2312_cn?b?UUFkWDFiUzJud1ZFYlBoNXZKeStSNG1FdGhzMnU2MkVIWnBWczZDeVpkcFBIL0tZ?= =?eucgb2312_cn?b?bHYxOGVGV2JmN3ZObTg4U0YwOGV4clJqamFSZFF4d2wrUDlBTlFrZTk3eGh1YlZv?= =?eucgb2312_cn?b?T3NSbGU3OHAvemJid0J4Z1E2Zmh3Y1U4ZXBicVNLa0o0Y3ZONUY2aWVFcnRaZGxL?= =?eucgb2312_cn?b?eWtwaURpMVlkSDdWUjhOUW1RYzBES3BKYVhVQnh3aUJpd3hhcXFJVnlsSnZ0Zmdp?= =?eucgb2312_cn?b?OUFGYnZ4bG4zcG9KQ2VnNHBRWU9HNnZFQXVzZmZCdDZpTDlTTkhwZkRDU2UvTnBF?= =?eucgb2312_cn?b?SG0vZytrcjF0N3hiRVBFVnpoUHg3aTJCamUwMFhMQWs3L0htcDZ3R0RYR1EwNGpu?= =?eucgb2312_cn?b?WUllbzJFdDBiamtQYURXNXVXdXc9PQ==?= MIME-Version: 1.0 X-OriginatorOrg: fujitsu.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: OS3PR01MB9499.jpnprd01.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 30a52455-766c-47ac-bdd4-08daae77ac2c X-MS-Exchange-CrossTenant-originalarrivaltime: 15 Oct 2022 06:37:03.8832 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a19f121d-81e1-4858-a9d8-736e267fd4c7 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 4QHv7PDYQA+KQhkMtNjc3CW2/rI11VCn5zEZokW0dPwLa2wvM5Q4vdjw46mK+2Mq0GZ2a3UChCxryYLl9LrKfg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: OSZPR01MB8661 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org The IB SPEC v1.5[1] defined new atomic write operation. This patchset makes SoftRoCE support new atomic write on RC service. On my rdma-core repository[2], I have introduced atomic write API for libibverbs and Pyverbs. I also have provided a rdma_atomic_write example and test_qp_ex_rc_atomic_write python test to verify the patchset. The steps to run the rdma_atomic_write example: server: $ ./rdma_atomic_write_server -s [server_address] -p [port_number] client: $ ./rdma_atomic_write_client -s [server_address] -p [port_number] The steps to run test_qp_ex_rc_atomic_write test: run_tests.py --dev rxe_enp0s3 --gid 1 -v test_qpex.QpExTestCase.test_qp_ex_rc_atomic_write test_qp_ex_rc_atomic_write (tests.test_qpex.QpExTestCase) ... ok ---------------------------------------------------------------------- Ran 1 test in 0.008s OK [1]: https://www.infinibandta.org/wp-content/uploads/2021/08/IBTA-Overview-of-IBTA-Volume-1-Release-1.5-and-MPE-2021-08-17-Secure.pptx [2]: https://github.com/yangx-jy/rdma-core/tree/new_api_with_point v5->v6: 1) Rebase on current for-next 2) Split the implementation of atomic write into 7 patches 3) Replace all "RDMA Atomic Write" with "atomic write" 4) Save 8-byte value in struct rxe_dma_info instead 5) Remove the print in atomic_write_reply() v4->v5: 1) Rebase on current wip/jgg-for-next 2) Rewrite the implementation on responder v3->v4: 1) Rebase on current wip/jgg-for-next 2) Fix a compiler error on 32-bit arch (e.g. parisc) by disabling RDMA Atomic Write 3) Replace 64-bit value with 8-byte array for atomic write V2->V3: 1) Rebase 2) Add RDMA Atomic Write attribute for rxe device V1->V2: 1) Set IB_OPCODE_RDMA_ATOMIC_WRITE to 0x1D 2) Add rdma.atomic_wr in struct rxe_send_wr and use it to pass the atomic write value 3) Use smp_store_release() to ensure that all prior operations have completed Xiao Yang (8): RDMA: Extend RDMA user ABI to support atomic write RDMA: Extend RDMA kernel ABI to support atomic write RDMA/rxe: Extend rxe user ABI to support atomic write RDMA/rxe: Extend rxe packet format to support atomic write RDMA/rxe: Make requester support atomic write on RC service RDMA/rxe: Make responder support atomic write on RC service RDMA/rxe: Implement atomic write completion RDMA/rxe: Enable atomic write capability for rxe device drivers/infiniband/sw/rxe/rxe_comp.c | 4 ++ drivers/infiniband/sw/rxe/rxe_opcode.c | 18 ++++++ drivers/infiniband/sw/rxe/rxe_opcode.h | 3 + drivers/infiniband/sw/rxe/rxe_param.h | 5 ++ drivers/infiniband/sw/rxe/rxe_req.c | 15 ++++- drivers/infiniband/sw/rxe/rxe_resp.c | 84 ++++++++++++++++++++++++-- include/rdma/ib_pack.h | 2 + include/rdma/ib_verbs.h | 3 + include/uapi/rdma/ib_user_verbs.h | 4 ++ include/uapi/rdma/rdma_user_rxe.h | 1 + 10 files changed, 132 insertions(+), 7 deletions(-)