From patchwork Fri Jul 8 04:02:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xiao Yang X-Patchwork-Id: 12910518 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3EF80C433EF for ; Fri, 8 Jul 2022 04:02:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237098AbiGHECo (ORCPT ); Fri, 8 Jul 2022 00:02:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46706 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237097AbiGHECn (ORCPT ); Fri, 8 Jul 2022 00:02:43 -0400 Received: from esa19.fujitsucc.c3s2.iphmx.com (esa19.fujitsucc.c3s2.iphmx.com [216.71.158.62]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 105A674DD6 for ; Thu, 7 Jul 2022 21:02:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=fujitsu.com; i=@fujitsu.com; q=dns/txt; s=fj1; t=1657252963; x=1688788963; h=from:to:cc:subject:date:message-id: content-transfer-encoding:mime-version; bh=XZjDyJW2sYfgkPGOnaGu+0Ztu5m7VuZr7SPlasGWH90=; b=LTO+SHkR7mP+nHRU/f6WbRVNVLbG633TZ6F7GPOnq9+HDZXNDCvWI1ZZ 6BrLm761fNrK+J8Rz1Sn2bK6YBBZ5qAFj6nveALDn/DpPoBtXMaFJHJbm YDGeVorQGGRHN237NtMAwteo08C9v06yNmsLbic2nO1x/bGsAVOunIhVu mnfhie7d3uJrwtOgAG3e+63UiJKBuqJCvs3DIWAnwbdXhH6TpeyDgXFTv Q12NkWnRpBAiNppex3tJs/rO/UMlOMfAGergvwRFMwlUN815BcjGWS13S y0Zk6Le5xMObTCKbLSh1VKijZYBLDuZ6EFHCtB3ZOGZ57b204n6ddrWj2 Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10401"; a="59574128" X-IronPort-AV: E=Sophos;i="5.92,254,1650898800"; d="scan'208";a="59574128" Received: from mail-os0jpn01lp2111.outbound.protection.outlook.com (HELO JPN01-OS0-obe.outbound.protection.outlook.com) ([104.47.23.111]) by ob1.fujitsucc.c3s2.iphmx.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jul 2022 13:02:39 +0900 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=MEaJ1MANsLtXtGpBGn/y+JfCasSGe8wSJYB65j+mjO9Wj0Jog/k3L0auoHHe2PjersRuZNo9k9pkp2XP49j0YeMmL1HOAa0Dhx7d+tzjYGUgMvp5DG+4xs8/9kUp5e77Z89qCbgdjXzWfRYsJ1feu41cLAY527jQr7Xw5JqEaseqG7WGYlJ12B2QUbLFkrfWXB4D+4NCNPE+39bqccogcSTeO7JCpCuqyBAzYt6VFtk62ZnR4dwjodt7s3RG7PvI2UqkuZ+gGbn5Duxa8JAV9hs4OQKVjPG1YWVPqTEXjcJOOeiaHhqFUOHEQWxiyTFe1NrYMgfDtjTRIToR+/Ce3g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=XZjDyJW2sYfgkPGOnaGu+0Ztu5m7VuZr7SPlasGWH90=; b=YHebnfzWfS8T/9gZtXeVHlw3kNVtj0/RRRBPDaiR8+OJq8VIpRy4Y2FeCvfomq9g8vWxrNrak71jsBeuBROLbf9DWQl7INv8a9OUa21mD4FpJknnCbEUO0qnGyY5YOfeF/aCWzwdgsp8CNbs3/iXVZ7GzqoPuHAflD2MK6wN90F7djwBEprBDA8gNpwUtZLYOEeEhi1r4Dz0gpsHJnIb17VDnv5nlQzx9yc9QGkeO+ZoAhICTZdqyAbH7est0s5M9k0bXHw8Duo6XPCvbVVkZhFpOde9uGXkCjqX/FVxqdmX+2Zl80NdwlXzMLftsdYPDmn78o0CgBh91ctUdfupZw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=fujitsu.com; dmarc=pass action=none header.from=fujitsu.com; dkim=pass header.d=fujitsu.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fujitsu.onmicrosoft.com; s=selector2-fujitsu-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XZjDyJW2sYfgkPGOnaGu+0Ztu5m7VuZr7SPlasGWH90=; b=PBjmD0e68lJLkl4Ft/8KCrNJamr6u63uYItTxOc8KWcMSA6mQXCsWRnjuqJhJ1u2zmLPPieGh4/Hh1eNewuuy2mZD/EdTu1pBU9L3lK6ZfkRYfygFgA0UWLBGnrSIqCUOXO9kKIjvKECLpsMQA3TY2uOd4jgWQ5FAkJvzLJgauk= Received: from OS3PR01MB9499.jpnprd01.prod.outlook.com (2603:1096:604:1c8::5) by TYWPR01MB10346.jpnprd01.prod.outlook.com (2603:1096:400:24a::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5417.16; Fri, 8 Jul 2022 04:02:35 +0000 Received: from OS3PR01MB9499.jpnprd01.prod.outlook.com ([fe80::d8d2:d66f:2f3:846c]) by OS3PR01MB9499.jpnprd01.prod.outlook.com ([fe80::d8d2:d66f:2f3:846c%7]) with mapi id 15.20.5395.022; Fri, 8 Jul 2022 04:02:35 +0000 From: "yangx.jy@fujitsu.com" To: "linux-rdma@vger.kernel.org" CC: "leon@kernel.org" , "jgg@ziepe.ca" , "rpearsonhpe@gmail.com" , "zyjzyj2000@gmail.com" , "lizhijian@fujitsu.com" , "yangx.jy@fujitsu.com" Subject: [RESEND PATCH v5 0/2] RDMA/rxe: Add RDMA Atomic Write operation Thread-Topic: [RESEND PATCH v5 0/2] RDMA/rxe: Add RDMA Atomic Write operation Thread-Index: AQHYkn+OvgSlKAPHREqgV/9WP2Vh0g== Date: Fri, 8 Jul 2022 04:02:35 +0000 Message-ID: <20220708040228.6703-1-yangx.jy@fujitsu.com> Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.34.1 authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=fujitsu.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 36807fab-efe0-4040-249a-08da6096b0cb x-ms-traffictypediagnostic: TYWPR01MB10346:EE_ x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 9ijsDgz7qBCUkozVNPhqrneC2NfEAiC/ML2mcYSZiLKUepOXXmOgQEP8r1UKBoXHt9MWAp/ETE97dmaKcIBtG6E6p2OFddesW0yz9sFIUtSjl8uvyKd/8IJMyksPReVjrbRwpfCNRKpEjX4CJzyr7RI9S4wGexUYbk+pL5QrfwYDwMMXr56J8eiY5nd0a9LhiMD3yvQedWNzkwJ/nnLaBgEAMaBsZsYTV4hniq82PJfeLPu9rreaNp6ts6tRz5DAlhwcAnA7mGcnRJeiOFPPyn4ZXvrOHynIcLXbIf0T7vkAX2hxESsmD1DB01q/OzWQl3T8CTSiMGuJd7/zp+JSjWPEdosErRxpOiheQFKSsuxleFi/tGz8eJhnHTYqUjlDHeUmMiHnNd9b1/t41Y34kzFlr+Bu4RdQKiPWUgYdHsYmg88I+N1GHRnRwmKEqectsYnd2/m6U28H2bOb/uNOLFZ6RmY01Efaln882e5FEKRJ+Qdl6m+7yc2XemKAmq2FJtq9/tCy2PZFKcGt7CofvsyXcKqNFwIJ/K13PcDrZjyihh+MT505uI/lG3GH3EkDOuJ2FjlHezKRmozfcLB3rv32gqmAM3xiO8lG+a2fZD4wjsP9sHPYAfR7LZX4aCCuOOA6aIiCHvzNwQWr3LLEcP0Yo2tp8Zbsmr7f3Ce6GCncbmOx1FHRV4SbrH8qhs81qrFxpYySl+4AS3XRwbU8gQ4Hbl3NuxEMzB1ONzesiya5ESFKHwKUCxiNZGuZeZ96PbNz0amoDD0g8eLurV6xWWGQW9fpGAErlHWl/Wu3wZYcDIlA6zHkQuhC4G8Q54SOKDmSDpFoU4hJQYXy9uptxJ2azCc4rfnr/o1yZQ6dYOdSa543hHWkcm5I6m2TtsvdzNZ8zAdejw5+DO0NB543jpJ2PT97hZhCSKcDJKLGW1q7rkzbDGKA8ycjNBMoS6I9 x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:OS3PR01MB9499.jpnprd01.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230016)(4636009)(376002)(346002)(396003)(136003)(39860400002)(366004)(107886003)(38070700005)(6512007)(8676002)(83380400001)(316002)(1076003)(54906003)(186003)(6916009)(4326008)(2616005)(36756003)(5660300002)(41300700001)(6506007)(478600001)(26005)(91956017)(6486002)(82960400001)(66946007)(66556008)(86362001)(2906002)(71200400001)(122000001)(966005)(8936002)(66446008)(76116006)(66476007)(64756008)(85182001)(38100700002)(15398625002)(43620500001);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?eucgb2312_cn?b?NUdPNlprQnlDbEVQQVBPMkFJ?= =?eucgb2312_cn?b?TGd1S2wreElCZWtTNkpORDlBS2JsVlhMcjJBbXlpRHY5b3k3bkZFTWl4T0tCUlZz?= =?eucgb2312_cn?b?ZU1tclZDTjdiSjhQUzR1aEhHa1F4NDJMbkxmb1g4dzhnc1FSMHdpVHB4Z2JpbHZU?= =?eucgb2312_cn?b?OXgrb092cW5mZWtBdTF2MG9nSXZIUHlkSmJkZ1FmcmRxQkJVSWhHZFdkaEl1aTlE?= =?eucgb2312_cn?b?ZGJlSkY3UXhuaHVVbFJhL2tXakxCaG1yU2pRdy9GNVBZTjhJUXBMWFBEU1VKRnVH?= =?eucgb2312_cn?b?V2ZESUN4eGNQdGtvQnBVbHA0dXMyeE9UTjBERHA3cTJyVFVMOFk0R2hJaEpwYkw4?= =?eucgb2312_cn?b?NmFscDhCVXpucTBhckV5Y3JIRkpEZ2JpWlFCdjdFbDk1Yk01dHplNWYraE5rcW5Y?= =?eucgb2312_cn?b?UGRvTnd0R25BbFpmV1Vzd3A0UmpOd0YrbDBXSkIrMDU3UEtIMmFGb3JLa3BYMzV4?= =?eucgb2312_cn?b?M1RTSGhHS0Njb0tlcUdaaTlTNUkxbVVhVmRhUU5tbTJGVW5uQnRYdXhyMVVHcXJ4?= =?eucgb2312_cn?b?WWp5LzlaSWwrMFF0RmNCVVBIdXhDbjdrM0FXeFAvdnVnbjZnUmlUVDlSa2VBdC9p?= =?eucgb2312_cn?b?TTN4YURhY3cvMHlHaytzZ1c3UGJPWExjUjBCdFBZS24wVFc0MUpJRnFwa1EzL1gx?= =?eucgb2312_cn?b?cFNxWFhqanl0Z1NiNkM2TUdjMGZsUjNuRktnYUIzODZvOHA0NGJqRllMWkRFdUk2?= =?eucgb2312_cn?b?QTEwT2xEZjQ5MUpyZ1hOZHh3djNhVURSTnNwTDNzZEcvVTJ5SjBYanZocWR6T0xu?= =?eucgb2312_cn?b?NERESGp3VWpZaUlVNWVtNEJwUHRFWEJ6ZnNLckpvQ0loTDA0RkVaNGVzdXNHTzBD?= =?eucgb2312_cn?b?cXI3MUNUTFBvbVRLakdocklqTkVUU0FQTmgyVTlaajFmUTBmWmd0eFQ1Y2pMaUZU?= =?eucgb2312_cn?b?Z2ljaGNLeVRHN0t1a3lxRTAyTUt2S1JkNU9MTTJpSEtZL1FxSmh6M0dSMFdkSzB0?= =?eucgb2312_cn?b?YVA1WHphQnZ6WVdKRHo4NkhFY0JpbGxUeUVRaTE1dEdnZ2d3d2luY3lwTGJIR3I5?= =?eucgb2312_cn?b?S3JDaFZCU0M4UDZFV2hkUVVVaHliZUpCRnQ4UUhLZWlQNVVIeVJTUlJZaVdYVXdS?= =?eucgb2312_cn?b?anhGWlFZSmhJekFIUkpHbjQrdlpNSGM3aWUvbFVUQmtVL09yZlo3TkRsTUhEVUwr?= =?eucgb2312_cn?b?aFhOTzBtUXBGS3laeUFqREUwQ2ZWS2tFVGYvMW4xaURHQjVWeDUyNzRzazZyMlIy?= =?eucgb2312_cn?b?UW8zcFdlMGE5MmNqVmE2dmhUNW5PL01QakNPU2g2WnIzWmVvd290SkJsSnhUWksw?= =?eucgb2312_cn?b?OW9udXZ1Tnpud05hNlJTMm50ZjRuRlJLZUlka0o4N1ZzNjR0ZFlNaUwxZElIVmky?= =?eucgb2312_cn?b?YVhIZS9Delh3OFM2WHgwNmREbitDenZtWW9UNUR5bEs5dk5HL0NKODdYZmUwV0pl?= =?eucgb2312_cn?b?ZUZTQVd6d2twY2tXd2ZodUZtL2gyazRaYkFCeGhWNWJZT1lrMFV5aE1HbXFkM1Nt?= =?eucgb2312_cn?b?K2I4RzhuK25xTjlzVzhtMFBja05NeTZKNm5XSTRNbWlVUGFOKzE3MHptQnF3ZElD?= =?eucgb2312_cn?b?dlluWWlHNFZtUGQ3OXFnb0hQWlQ4b0t3V2xjN0FDWkZJTkc5VFA4Q3NrNEVzN2VF?= =?eucgb2312_cn?b?clhzbTlyeWprZGMreVZ4UHVvQmZNUUdZRjRKbVFmR2ZqU2REbGZIdkw5Q3BETXBs?= =?eucgb2312_cn?b?b3g1dUI5VjhiQlh2UHlURFFzQU5LQmMvN3FIVWYxTHpXTXlSdDRIOHpFRXRmWUQ4?= =?eucgb2312_cn?b?b0tVbmN3WGViOEs1QzlXV2gvOVMrL3dkM2NWNUVLODZRSVdhL2dqQ2FOcDV6UnJF?= =?eucgb2312_cn?b?SWNxYllEMzEzc3pKdzZ3RjRSMWpoTkpDQlE0NlNwUlVQLy9Dc2VwNUdqK0YvZDlS?= =?eucgb2312_cn?b?TDJYSDE1bmxTRSt0aEU3dmJ1c2p3dkNHbXA4Wis3QzVXTSs1T2hlSWtmNHRFcFpV?= =?eucgb2312_cn?b?TnJnbmxrWW5remh2MjRMaHhnNFY1andiazQ2T1V6a0xjNkFLTzBsQnhIdWVuSFd1?= =?eucgb2312_cn?b?Q3F3R3ArMGJjK1N0TXpFUERYT2xnSzExRitIQzNKQ2JJcnNjTTBINm1aUW85UHpO?= =?eucgb2312_cn?b?K2pVbGhXbU00NVpLYUpiQXFtTVNycWZUaURhNGg5cStVZGpFNkFSUFp0MG5zSW1Q?= =?eucgb2312_cn?b?VGlrSjVjYVQ2UFNKM29Yb2Vsc1E9PQ==?= MIME-Version: 1.0 X-OriginatorOrg: fujitsu.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: OS3PR01MB9499.jpnprd01.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 36807fab-efe0-4040-249a-08da6096b0cb X-MS-Exchange-CrossTenant-originalarrivaltime: 08 Jul 2022 04:02:35.3654 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a19f121d-81e1-4858-a9d8-736e267fd4c7 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: AvtGFQdJ/vnrHuqP7Vefn7ElCISnPSaxxUBaHqLo2L/rqCrXQsaTq86cf8a1W+WqMosKSVN39SwJkWJKROx0Cw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: TYWPR01MB10346 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org The IB SPEC v1.5[1] defined new RDMA Atomic Write operation. This patchset makes SoftRoCE support new RDMA Atomic Write on RC service. On my rdma-core repository[2], I have introduced RDMA Atomic Write API for libibverbs and Pyverbs. I also have provided a rdma_atomic_write example and test_qp_ex_rc_rdma_atomic_write python test to verify the patchset. The steps to run the rdma_atomic_write example: server: $ ./rdma_atomic_write_server -s [server_address] -p [port_number] client: $ ./rdma_atomic_write_client -s [server_address] -p [port_number] The steps to run test_qp_ex_rc_rdma_atomic_write test: run_tests.py --dev rxe_enp0s3 --gid 1 -v test_qpex.QpExTestCase.test_qp_ex_rc_rdma_atomic_write test_qp_ex_rc_rdma_atomic_write (tests.test_qpex.QpExTestCase) ... ok ---------------------------------------------------------------------- Ran 1 test in 0.008s OK [1]: https://www.infinibandta.org/wp-content/uploads/2021/08/IBTA-Overview-of-IBTA-Volume-1-Release-1.5-and-MPE-2021-08-17-Secure.pptx [2]: https://github.com/yangx-jy/rdma-core/tree/new_api_with_point Note: This patch set depends on the following patch set: https://lore.kernel.org/linux-rdma/20220705145212.12014-1-yangx.jy@fujitsu.com/T/#t v4->v5: 1) Rebase on current wip/jgg-for-next 2) Rewrite the implementation on responder v3->v4: 1) Rebase on current wip/jgg-for-next 2) Fix a compiler error on 32-bit arch (e.g. parisc) by disabling RDMA Atomic Write 3) Replace 64-bit value with 8-byte array for RDMA Atomic Write V2->V3: 1) Rebase 2) Add RDMA Atomic Write attribute for rxe device V1->V2: 1) Set IB_OPCODE_RDMA_ATOMIC_WRITE to 0x1D 2) Add rdma.atomic_wr in struct rxe_send_wr and use it to pass the atomic write value 3) Use smp_store_release() to ensure that all prior operations have completed Xiao Yang (2): RDMA/rxe: Support RDMA Atomic Write operation RDMA/rxe: Add RDMA Atomic Write attribute for rxe device drivers/infiniband/sw/rxe/rxe_comp.c | 4 ++ drivers/infiniband/sw/rxe/rxe_opcode.c | 18 +++++ drivers/infiniband/sw/rxe/rxe_opcode.h | 3 + drivers/infiniband/sw/rxe/rxe_param.h | 5 ++ drivers/infiniband/sw/rxe/rxe_req.c | 15 +++- drivers/infiniband/sw/rxe/rxe_resp.c | 94 ++++++++++++++++++++++++-- include/rdma/ib_pack.h | 2 + include/rdma/ib_verbs.h | 3 + include/uapi/rdma/ib_user_verbs.h | 4 ++ include/uapi/rdma/rdma_user_rxe.h | 1 + 10 files changed, 142 insertions(+), 7 deletions(-)