From patchwork Fri Jul 8 04:02:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xiao Yang X-Patchwork-Id: 12910520 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60B4CC43334 for ; Fri, 8 Jul 2022 04:02:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237099AbiGHECr (ORCPT ); Fri, 8 Jul 2022 00:02:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46732 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237097AbiGHECp (ORCPT ); Fri, 8 Jul 2022 00:02:45 -0400 Received: from esa19.fujitsucc.c3s2.iphmx.com (esa19.fujitsucc.c3s2.iphmx.com [216.71.158.62]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9FD1874DE9 for ; Thu, 7 Jul 2022 21:02:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=fujitsu.com; i=@fujitsu.com; q=dns/txt; s=fj1; t=1657252964; x=1688788964; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=eK4YSR+U/mpIn4UKoSFdmRRJg2KRDOxpHRxz+bvFx0g=; b=NzzBQ0X7eyEfq9CjFDdYpECdVmsSIZ4B4zLTCkqGLD3UKadWtDBnWLaA rzNq9c4Z8lhIz/A6IW22aAH4AuwAtMhQfBzPzopKBokBt0nF0dC/uipP4 OV9VdHuBizg1dSyjwAyFCtzg0kOvmt7w8czvmnCiEeZ7gO7ek0vZDNl/P Fvre/0esPMQ88vL1QagH/6UQJqBXaTM6cJGquEy/sd/4Om3yxoPgsDu+6 R+KoQQ6V/SilQDV9h+KAYxh6zQ2H1k5RfOWx1zeG5LgiFYYfJ40gIyqRO V/NyVUDcBABDkB3uKpIVG2P3+cs77b05LFD5BM1u829ySu/Y3VLlzA+KK w==; X-IronPort-AV: E=McAfee;i="6400,9594,10401"; a="59574129" X-IronPort-AV: E=Sophos;i="5.92,254,1650898800"; d="scan'208";a="59574129" Received: from mail-os0jpn01lp2111.outbound.protection.outlook.com (HELO JPN01-OS0-obe.outbound.protection.outlook.com) ([104.47.23.111]) by ob1.fujitsucc.c3s2.iphmx.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jul 2022 13:02:41 +0900 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=JSnqfHWTkQcpc9l8qOANETuJLg9QIAYboTPnqqOKxBnjsk8JyfUMJKIARDffu7kg/yHq3dE0a9QLi1IV2bSnAVDVTO968uGL2wBFo879l0AQCsqMo5+VeSStk/sTzlsM9pHW3TzxJzL/Z6mD7S0WyBAyJvzYJMaz2woQLbajt1pD7UlT/LDftXP6iIEkcbe128O1VyADltP3UUtCyD928qS0mmrSjc3KhsTcWsRcGc8SD3iAxnuOc5VNhv62+mLLKaVEfKPWX6F65Jqml4ogkblFdtDHg2HmOuUjxLnefw3/XOpqIeep/+52r5dJhvI0f2acrJ7OoBSRU/1OzVuaWw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=eK4YSR+U/mpIn4UKoSFdmRRJg2KRDOxpHRxz+bvFx0g=; b=KiTOUIHw8pdnURgSVxykqcO0IbSWPFNzyPdD/s0pn8Sjy8bp1ODdjflo093mD7TSvPEfDCjVOk+X8HnnB5SxHfP6dENN/h1thc5fHHKApxk8IMRcmmd+eZuE8RCzxv+HQhM1hLdAo4SWt3mCMpu4tIFcxq8IdZipy+HnJ4GkKDvBYsfHp0m88qMsP0NhOB2MKVXYFOGcuirYpH9ICH+1DLqL3xvlJKxsKZjFXEBC+AnBfOWl1BwFS0G5cYoA99dK03RBx/dB69EOX7PXDNcUwvfw3d4wmLsq5+F6CU1+E/rIT6gsuQFjYSyjlskDlCexjG/wLciPsrUyCszLGzFn6Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=fujitsu.com; dmarc=pass action=none header.from=fujitsu.com; dkim=pass header.d=fujitsu.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fujitsu.onmicrosoft.com; s=selector2-fujitsu-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eK4YSR+U/mpIn4UKoSFdmRRJg2KRDOxpHRxz+bvFx0g=; b=d/jMogCJmTVFmI6a33NVUzs2Co4L/Kgsv2mzs6lV8FG757/CZUaf4pycoBXvjquG1m/q0B5+KeTyl5x8BazJQ8JLx3uiSJ7z8vv9NLfPGd/dCadwLKNkatchF8AaaU+mSHCgP2Bt+zPD2DNNjWS0UQ1fosthqmLqUDMfNpmUfH0= Received: from OS3PR01MB9499.jpnprd01.prod.outlook.com (2603:1096:604:1c8::5) by TYWPR01MB10346.jpnprd01.prod.outlook.com (2603:1096:400:24a::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5417.16; Fri, 8 Jul 2022 04:02:36 +0000 Received: from OS3PR01MB9499.jpnprd01.prod.outlook.com ([fe80::d8d2:d66f:2f3:846c]) by OS3PR01MB9499.jpnprd01.prod.outlook.com ([fe80::d8d2:d66f:2f3:846c%7]) with mapi id 15.20.5395.022; Fri, 8 Jul 2022 04:02:36 +0000 From: "yangx.jy@fujitsu.com" To: "linux-rdma@vger.kernel.org" CC: "leon@kernel.org" , "jgg@ziepe.ca" , "rpearsonhpe@gmail.com" , "zyjzyj2000@gmail.com" , "lizhijian@fujitsu.com" , "yangx.jy@fujitsu.com" Subject: [RESEND PATCH v5 1/2] RDMA/rxe: Support RDMA Atomic Write operation Thread-Topic: [RESEND PATCH v5 1/2] RDMA/rxe: Support RDMA Atomic Write operation Thread-Index: AQHYkn+ONFoqruKJZkGfwPAb3txSTA== Date: Fri, 8 Jul 2022 04:02:36 +0000 Message-ID: <20220708040228.6703-2-yangx.jy@fujitsu.com> References: <20220708040228.6703-1-yangx.jy@fujitsu.com> In-Reply-To: <20220708040228.6703-1-yangx.jy@fujitsu.com> Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.34.1 authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=fujitsu.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 8f8833ff-d288-441c-48f2-08da6096b147 x-ms-traffictypediagnostic: TYWPR01MB10346:EE_ x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: yTnl+ma+qQXLtAsIyr2jpyhhxqH9NpvylN5k4MqOxCiG8Lyd4sWuzrwgzo3eZhbUAn42ig9rc7XULtHv0Yf/YmovjS9oZmB+FLgDM1csdBi7Dzdh6THH8BjsyRSOWU3D2+GUoh7Xfst6/khGQw2unr4Vd82qZJow3lCe0FlskbvCij/eahvJWmyD87PY4GDI9fQN1EmNV8LDK6LcDdskWonOtr2pJ4g97OIwZivqktFiH1msOiNWoudz/TubuDyqb92pQh5iq6KsgeuBBa+ehh3ivA2YLf8HOPEfePuhWm21OquoPnwZze8vYfpomJMCVccFMCs7IlGideVniO9VXupN75xfE85Gm0oy29Othku97xs62lfwLLoLL8Yrla8KroEssFwsV3l8axkf98LvafNgdaGXlVp4uUc/sgMtJvDCJsunnISuf8rDuco9izj07ByDHv1VZ712moahQ1i+RRtNxKjKKT1u34Gm69TT4TbyLTRgGGnrMVrmHi+YVhQdg9LzQ8/iFTYisLhEBDoiwkLtdHMqFkX5mbRKz3afRZCTRWyJbhzCtFBD1h11oNp2DA7wUuGRYqnh2DX1djPZIyQ+4MuAKwr5ifaa3V/PT0PrTrlh2wDvAVrB+W+FRSDeg6f+CDogSeC2i5Bzb58H9l8H4zcGFUcMxcHJy/BS+0FN2OX3V1zuiPX9dbIC07tizzFgMESwSpamGIf2AV/ogixgWrE3Od1zal3ZeGEKVh6JAaANUsdJKYBdkbOzgOekXNlpIuq4DrIsYd75OWW9A1d1loayy3ZCjDNGHBq0KoJTxQjnCFJGh99QAG3kSacx/s4krPqQEQ5ncLP3PXABnysqrzhufaIWmOfyyekcF4U= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:OS3PR01MB9499.jpnprd01.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230016)(4636009)(376002)(346002)(396003)(136003)(39860400002)(366004)(107886003)(38070700005)(6512007)(8676002)(83380400001)(316002)(1076003)(54906003)(45080400002)(186003)(6916009)(4326008)(2616005)(36756003)(5660300002)(41300700001)(6506007)(30864003)(478600001)(26005)(91956017)(6486002)(82960400001)(66946007)(66556008)(86362001)(2906002)(71200400001)(122000001)(8936002)(66446008)(76116006)(66476007)(64756008)(85182001)(38100700002)(375104004);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?eucgb2312_cn?b?YjVzNkFVWlRZYm5VQUpYNlpO?= =?eucgb2312_cn?b?ZzdXSTNXWFhONjJMQ0pIeVo4a3l1Z0VSN054VW1FN0Rsck1KNVpQN3puSitwRzJX?= =?eucgb2312_cn?b?QitIT3F0KzdBOWo4Y0l5emhIcmlOb2FhbTNDZGp6U2ZiREREM2xqemgwejQwZ1ZK?= =?eucgb2312_cn?b?NzNXTTArYVlWeVJDbm5RSEU3MlQ4VWptdERxRjVoaGZxTzJzOEIwY3l6d3JmOUJr?= =?eucgb2312_cn?b?TExrRFNVdHRmVHVWS1EyVzNLTDg5bUdrcG5uNzFvZWcxYzBNQk1LQ3phRmRRSDZO?= =?eucgb2312_cn?b?SWNBVXJDMERFa2pOT3RvUy9XU0ttZ2RybnFtMU1lWHh6MytJaWVTbW9uSm9tK1Nz?= =?eucgb2312_cn?b?a3RRN2V6RGhQa3QwVHVuNzQ4cS9GTytEQWtyK2NUSnRXSDY3SHhjSFJ2cDBmalBT?= =?eucgb2312_cn?b?ZVlXVldFbTBMS043Z3htOE42QkVpeGI3NjgyMmkrYVZHNzVYRkhlRG1pdSt4TmxI?= =?eucgb2312_cn?b?aHJyM0tlenJwZkFIK2RhR0FZUnA0S3ovN2NBVDZlL01KcTM2TEdQeW84TThxSWRh?= =?eucgb2312_cn?b?aFhrbEJvZVFPQ3huRHZwMG9DTE9mQlJzSW1FVWVjUjhjU0M0UWRFYXZicVo2NFdU?= =?eucgb2312_cn?b?Z2o4ZURCckpTVGFEYlppV3doazYzTTFJQlFPSTRsd1hsQkFqaHJ2Ym04VEtNbHBL?= =?eucgb2312_cn?b?MzI4WXZheWw4a0lncFZGOW8xU0RGYTFKOGFsbjV5MU1qUy82N3pwU2JCVyt3bGpr?= =?eucgb2312_cn?b?NFNFeitsUEFQN3JFRnhId25pNUNacG1qczFKWkI5TWkrV3FDaWF0aEhFUUtFd0tC?= =?eucgb2312_cn?b?VUxMWWw5Y2JjWDhwckpSQStQOUpSYlEzeXZEc2tkTGlXemlPUzd0enUzMzEzQ1Mx?= =?eucgb2312_cn?b?WkVjUGtzWVdiWCtIbnpZdDRhQlRlN0orVGVQNXFsTTZYVmcxc0pZbUpZcXhPOElS?= =?eucgb2312_cn?b?TTdLeWwyMmd4Z1J2K2JWZnBQejlibUxmODFRQXZrcTRiN2hqS2ZGcm5ibWxDRUVp?= =?eucgb2312_cn?b?K0xMaDkwSHhKdXY1NWtlSGpKWjRhaG0xWS9adnovUm5sZ3FwWjVNSnJJZkowL1VI?= =?eucgb2312_cn?b?MVMyeWgyMkVQUXl4dklzMzAza1M2MmtENWFlNmJtdFB3M1ZwUFg2OFZmRzhzVFpK?= =?eucgb2312_cn?b?b1ZWTUNBQzBKSVhjYlhxMGVGTXFXZ1JDdnVveXBMK3hBNUdwZjdMdmtkT3dJbGNB?= =?eucgb2312_cn?b?UkdDZEVjVXhvRUFEa2cyaTZFSTFLb1llOXFwOTRUaHhuaUJteHZydmdSancrQWJW?= =?eucgb2312_cn?b?WS9LZ2hDVDlwLzFXc0FyZHdBaVl5RG55YU9FQUhVU1NkcElWNnZSY3JhNnE1RmRK?= =?eucgb2312_cn?b?ekFjdkR4NXNiTm5qMVRMU3drb09rYmI2RUtXeDJ6c3NLU3hhR1g3M0pZL2haWlVj?= =?eucgb2312_cn?b?MS9PL0FlVXBYNWFzN25CNHFpZFA3cDZGZmFOTnUzTlRneUdyK0Z3NGRVZ1hnQ01C?= =?eucgb2312_cn?b?bkJGcDBxTnRkVWpVbC92TmpSVDJFUE1LUjRxbmxYWkNReDNsYk9VNWRtSHNrdkJk?= =?eucgb2312_cn?b?UXpSOWgxK3RPNzl3ZGFMV3VDOWdTQjhtekFvRnNua1AxSThpaHNxZGtCQ3lxRG1J?= =?eucgb2312_cn?b?ZFp6eDlJM0dzbGo3WHVlbVlJOHNxRXQxRG84Tmwyc0czS3lJMzVuR0xhRFYvOC9w?= =?eucgb2312_cn?b?ZGpYTTVPTE1DemVGcjNwWC9JUWtYaW1VK2MzaS9tblQ3ZTVDS1drVjFNRytrSFlu?= =?eucgb2312_cn?b?YUVLd1REUDZzTTB0SlpFdkprYy9QQ3pKQlFSUnd4SjlZOHlaNU1wVHFMUGhWV1pu?= =?eucgb2312_cn?b?QUtnOTBpcXd1L3oyRlVGbStHZVp2MEV6VnpwclZJeDVZVkxSV0JIUmw1Z0JpM2Er?= =?eucgb2312_cn?b?ZEVON09rNEtUM2lVT0VtNXJnazYxdThJT2tmYmszZ1lPeWF2V3EwbzJDL3ovUTdu?= =?eucgb2312_cn?b?WHJrdDVYNlM0dmVudmc4MkpQejZVSXNZWjE5MS9EamZFWUpFZ3hvSDZuclBYR0cz?= =?eucgb2312_cn?b?L1g2NlBqdEdnOVdpaGlCSm1MVDc3YnZMbCthTWlGQ0VLL1FldGRMUjQ0VVk4SHRx?= =?eucgb2312_cn?b?Z0s3MkFEWTlObGF3YkMzVXB5STY0ekV1c05mTU1OZnE0c0YxQmRCMkFuemFrNUpB?= =?eucgb2312_cn?b?V09TTWdDcVBHamVkMlRtQkhPL1AyeXdINTBOc1plY2RHZ3Bsam1vVldEcDlrdkxo?= =?eucgb2312_cn?b?b3VMVitBQmQ2UHVxcHV4b0JLRkE9PQ==?= MIME-Version: 1.0 X-OriginatorOrg: fujitsu.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: OS3PR01MB9499.jpnprd01.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8f8833ff-d288-441c-48f2-08da6096b147 X-MS-Exchange-CrossTenant-originalarrivaltime: 08 Jul 2022 04:02:36.1321 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a19f121d-81e1-4858-a9d8-736e267fd4c7 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 2uLkmD83G2Ub83DZH7Tac+dzLDcY45Z/CE1JPOy84NWHxGMlHPhcZ1rcx5Gpg5N8xkhkzdpxMHJg/76TGLtLiQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: TYWPR01MB10346 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org This patch implements RDMA Atomic Write operation for RC service. Signed-off-by: Xiao Yang --- drivers/infiniband/sw/rxe/rxe_comp.c | 4 ++ drivers/infiniband/sw/rxe/rxe_opcode.c | 18 +++++ drivers/infiniband/sw/rxe/rxe_opcode.h | 3 + drivers/infiniband/sw/rxe/rxe_req.c | 15 +++- drivers/infiniband/sw/rxe/rxe_resp.c | 94 ++++++++++++++++++++++++-- include/rdma/ib_pack.h | 2 + include/rdma/ib_verbs.h | 2 + include/uapi/rdma/ib_user_verbs.h | 2 + include/uapi/rdma/rdma_user_rxe.h | 1 + 9 files changed, 134 insertions(+), 7 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_comp.c b/drivers/infiniband/sw/rxe/rxe_comp.c index da3a398053b8..16b90d68d2cb 100644 --- a/drivers/infiniband/sw/rxe/rxe_comp.c +++ b/drivers/infiniband/sw/rxe/rxe_comp.c @@ -104,6 +104,7 @@ static enum ib_wc_opcode wr_to_wc_opcode(enum ib_wr_opcode opcode) case IB_WR_LOCAL_INV: return IB_WC_LOCAL_INV; case IB_WR_REG_MR: return IB_WC_REG_MR; case IB_WR_BIND_MW: return IB_WC_BIND_MW; + case IB_WR_RDMA_ATOMIC_WRITE: return IB_WC_RDMA_ATOMIC_WRITE; default: return 0xff; @@ -256,6 +257,9 @@ static inline enum comp_state check_ack(struct rxe_qp *qp, if ((syn & AETH_TYPE_MASK) != AETH_ACK) return COMPST_ERROR; + if (wqe->wr.opcode == IB_WR_RDMA_ATOMIC_WRITE) + return COMPST_WRITE_SEND; + fallthrough; /* (IB_OPCODE_RC_RDMA_READ_RESPONSE_MIDDLE doesn't have an AETH) */ diff --git a/drivers/infiniband/sw/rxe/rxe_opcode.c b/drivers/infiniband/sw/rxe/rxe_opcode.c index d4ba4d506f17..d284fa8798c3 100644 --- a/drivers/infiniband/sw/rxe/rxe_opcode.c +++ b/drivers/infiniband/sw/rxe/rxe_opcode.c @@ -101,6 +101,12 @@ struct rxe_wr_opcode_info rxe_wr_opcode_info[] = { [IB_QPT_UC] = WR_LOCAL_OP_MASK, }, }, + [IB_WR_RDMA_ATOMIC_WRITE] = { + .name = "IB_WR_RDMA_ATOMIC_WRITE", + .mask = { + [IB_QPT_RC] = WR_ATOMIC_WRITE_MASK, + }, + }, }; struct rxe_opcode_info rxe_opcode[RXE_NUM_OPCODE] = { @@ -378,6 +384,18 @@ struct rxe_opcode_info rxe_opcode[RXE_NUM_OPCODE] = { RXE_IETH_BYTES, } }, + [IB_OPCODE_RC_RDMA_ATOMIC_WRITE] = { + .name = "IB_OPCODE_RC_RDMA_ATOMIC_WRITE", + .mask = RXE_RETH_MASK | RXE_PAYLOAD_MASK | RXE_REQ_MASK | + RXE_ATOMIC_WRITE_MASK | RXE_START_MASK | + RXE_END_MASK, + .length = RXE_BTH_BYTES + RXE_RETH_BYTES, + .offset = { + [RXE_BTH] = 0, + [RXE_RETH] = RXE_BTH_BYTES, + [RXE_PAYLOAD] = RXE_BTH_BYTES + RXE_RETH_BYTES, + } + }, /* UC */ [IB_OPCODE_UC_SEND_FIRST] = { diff --git a/drivers/infiniband/sw/rxe/rxe_opcode.h b/drivers/infiniband/sw/rxe/rxe_opcode.h index 8f9aaaf260f2..5962f5fc66a6 100644 --- a/drivers/infiniband/sw/rxe/rxe_opcode.h +++ b/drivers/infiniband/sw/rxe/rxe_opcode.h @@ -20,6 +20,7 @@ enum rxe_wr_mask { WR_READ_MASK = BIT(3), WR_WRITE_MASK = BIT(4), WR_LOCAL_OP_MASK = BIT(5), + WR_ATOMIC_WRITE_MASK = BIT(7), WR_READ_OR_WRITE_MASK = WR_READ_MASK | WR_WRITE_MASK, WR_WRITE_OR_SEND_MASK = WR_WRITE_MASK | WR_SEND_MASK, @@ -81,6 +82,8 @@ enum rxe_hdr_mask { RXE_LOOPBACK_MASK = BIT(NUM_HDR_TYPES + 12), + RXE_ATOMIC_WRITE_MASK = BIT(NUM_HDR_TYPES + 14), + RXE_READ_OR_ATOMIC_MASK = (RXE_READ_MASK | RXE_ATOMIC_MASK), RXE_WRITE_OR_SEND_MASK = (RXE_WRITE_MASK | RXE_SEND_MASK), RXE_READ_OR_WRITE_MASK = (RXE_READ_MASK | RXE_WRITE_MASK), diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rxe/rxe_req.c index 15fefc689ca3..613c7031f562 100644 --- a/drivers/infiniband/sw/rxe/rxe_req.c +++ b/drivers/infiniband/sw/rxe/rxe_req.c @@ -235,6 +235,10 @@ static int next_opcode_rc(struct rxe_qp *qp, u32 opcode, int fits) else return fits ? IB_OPCODE_RC_SEND_ONLY_WITH_INVALIDATE : IB_OPCODE_RC_SEND_FIRST; + + case IB_WR_RDMA_ATOMIC_WRITE: + return IB_OPCODE_RC_RDMA_ATOMIC_WRITE; + case IB_WR_REG_MR: case IB_WR_LOCAL_INV: return opcode; @@ -463,6 +467,11 @@ static int finish_packet(struct rxe_qp *qp, struct rxe_av *av, } } + if (pkt->mask & RXE_ATOMIC_WRITE_MASK) { + memcpy(payload_addr(pkt), wqe->wr.wr.rdma.atomic_wr, payload); + wqe->dma.resid -= payload; + } + return 0; } @@ -663,13 +672,15 @@ int rxe_requester(void *arg) } mask = rxe_opcode[opcode].mask; - if (unlikely(mask & RXE_READ_OR_ATOMIC_MASK)) { + if (unlikely(mask & (RXE_READ_OR_ATOMIC_MASK | + RXE_ATOMIC_WRITE_MASK))) { if (check_init_depth(qp, wqe)) goto exit; } mtu = get_mtu(qp); - payload = (mask & RXE_WRITE_OR_SEND_MASK) ? wqe->dma.resid : 0; + payload = (mask & (RXE_WRITE_OR_SEND_MASK | RXE_ATOMIC_WRITE_MASK)) ? + wqe->dma.resid : 0; if (payload > mtu) { if (qp_type(qp) == IB_QPT_UD) { /* C10-93.1.1: If the total sum of all the buffer lengths specified for a diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c index 28033849d404..2cf544abe0dc 100644 --- a/drivers/infiniband/sw/rxe/rxe_resp.c +++ b/drivers/infiniband/sw/rxe/rxe_resp.c @@ -22,6 +22,7 @@ enum resp_states { RESPST_EXECUTE, RESPST_READ_REPLY, RESPST_ATOMIC_REPLY, + RESPST_ATOMIC_WRITE_REPLY, RESPST_COMPLETE, RESPST_ACKNOWLEDGE, RESPST_CLEANUP, @@ -57,6 +58,7 @@ static char *resp_state_name[] = { [RESPST_EXECUTE] = "EXECUTE", [RESPST_READ_REPLY] = "READ_REPLY", [RESPST_ATOMIC_REPLY] = "ATOMIC_REPLY", + [RESPST_ATOMIC_WRITE_REPLY] = "ATOMIC_WRITE_REPLY", [RESPST_COMPLETE] = "COMPLETE", [RESPST_ACKNOWLEDGE] = "ACKNOWLEDGE", [RESPST_CLEANUP] = "CLEANUP", @@ -260,7 +262,7 @@ static enum resp_states check_op_valid(struct rxe_qp *qp, case IB_QPT_RC: if (((pkt->mask & RXE_READ_MASK) && !(qp->attr.qp_access_flags & IB_ACCESS_REMOTE_READ)) || - ((pkt->mask & RXE_WRITE_MASK) && + ((pkt->mask & (RXE_WRITE_MASK | RXE_ATOMIC_WRITE_MASK)) && !(qp->attr.qp_access_flags & IB_ACCESS_REMOTE_WRITE)) || ((pkt->mask & RXE_ATOMIC_MASK) && !(qp->attr.qp_access_flags & IB_ACCESS_REMOTE_ATOMIC))) { @@ -364,7 +366,7 @@ static enum resp_states check_resource(struct rxe_qp *qp, } } - if (pkt->mask & RXE_READ_OR_ATOMIC_MASK) { + if (pkt->mask & (RXE_READ_OR_ATOMIC_MASK | RXE_ATOMIC_WRITE_MASK)) { /* it is the requesters job to not send * too many read/atomic ops, we just * recycle the responder resource queue @@ -415,7 +417,7 @@ static enum resp_states check_rkey(struct rxe_qp *qp, enum resp_states state; int access; - if (pkt->mask & RXE_READ_OR_WRITE_MASK) { + if (pkt->mask & (RXE_READ_OR_WRITE_MASK | RXE_ATOMIC_WRITE_MASK)) { if (pkt->mask & RXE_RETH_MASK) { qp->resp.va = reth_va(pkt); qp->resp.offset = 0; @@ -483,7 +485,7 @@ static enum resp_states check_rkey(struct rxe_qp *qp, goto err; } - if (pkt->mask & RXE_WRITE_MASK) { + if (pkt->mask & (RXE_WRITE_MASK | RXE_ATOMIC_WRITE_MASK)) { if (resid > mtu) { if (pktlen != mtu || bth_pad(pkt)) { state = RESPST_ERR_LENGTH; @@ -583,6 +585,7 @@ static struct resp_res *rxe_prepare_res(struct rxe_qp *qp, res->state = rdatm_res_state_new; break; case RXE_ATOMIC_MASK: + case RXE_ATOMIC_WRITE_MASK: res->first_psn = pkt->psn; res->last_psn = pkt->psn; res->cur_psn = pkt->psn; @@ -652,6 +655,53 @@ static enum resp_states atomic_reply(struct rxe_qp *qp, return ret; } +static enum resp_states atomic_write_reply(struct rxe_qp *qp, + struct rxe_pkt_info *pkt) +{ + u64 src, *dst; + struct resp_res *res = qp->resp.res; + struct rxe_mr *mr = qp->resp.mr; + int payload = payload_size(pkt); + + if (!res) { + res = rxe_prepare_res(qp, pkt, RXE_ATOMIC_WRITE_MASK); + qp->resp.res = res; + } + + if (!res->replay) { +#ifdef CONFIG_64BIT + memcpy(&src, payload_addr(pkt), payload); + + dst = iova_to_vaddr(mr, qp->resp.va + qp->resp.offset, payload); + /* check vaddr is 8 bytes aligned. */ + if (!dst || (uintptr_t)dst & 7) + return RESPST_ERR_MISALIGNED_ATOMIC; + + /* Do atomic write after all prior operations have completed */ + smp_store_release(dst, src); + + /* decrease resp.resid to zero */ + qp->resp.resid -= sizeof(payload); + + qp->resp.msn++; + + /* next expected psn, read handles this separately */ + qp->resp.psn = (pkt->psn + 1) & BTH_PSN_MASK; + qp->resp.ack_psn = qp->resp.psn; + + qp->resp.opcode = pkt->opcode; + qp->resp.status = IB_WC_SUCCESS; + + return RESPST_ACKNOWLEDGE; +#else + pr_err("32-bit arch doesn't support 8-byte atomic write\n"); + return RESPST_ERR_UNSUPPORTED_OPCODE; +#endif /* CONFIG_64BIT */ + } + + return RESPST_ACKNOWLEDGE; +} + static struct sk_buff *prepare_ack_packet(struct rxe_qp *qp, struct rxe_pkt_info *ack, int opcode, @@ -892,6 +942,8 @@ static enum resp_states execute(struct rxe_qp *qp, struct rxe_pkt_info *pkt) return RESPST_READ_REPLY; } else if (pkt->mask & RXE_ATOMIC_MASK) { return RESPST_ATOMIC_REPLY; + } else if (pkt->mask & RXE_ATOMIC_WRITE_MASK) { + return RESPST_ATOMIC_WRITE_REPLY; } else { /* Unreachable */ WARN_ON_ONCE(1); @@ -1074,6 +1126,31 @@ static int send_atomic_ack(struct rxe_qp *qp, u8 syndrome, u32 psn) return err; } +static int send_read_response(struct rxe_qp *qp, u8 syndrome, u32 psn) +{ + int err = 0; + struct rxe_pkt_info ack_pkt; + struct sk_buff *skb; + + skb = prepare_ack_packet(qp, &ack_pkt, IB_OPCODE_RC_RDMA_READ_RESPONSE_ONLY, + 0, psn, syndrome); + if (!skb) { + err = -ENOMEM; + goto out; + } + + err = rxe_xmit_packet(qp, &ack_pkt, skb); + if (err) + pr_err_ratelimited("Failed sending read response\n"); + + /* have to clear this since it is used to trigger + * long read replies + */ + qp->resp.res = NULL; +out: + return err; +} + static enum resp_states acknowledge(struct rxe_qp *qp, struct rxe_pkt_info *pkt) { @@ -1084,6 +1161,8 @@ static enum resp_states acknowledge(struct rxe_qp *qp, send_ack(qp, qp->resp.aeth_syndrome, pkt->psn); else if (pkt->mask & RXE_ATOMIC_MASK) send_atomic_ack(qp, AETH_ACK_UNLIMITED, pkt->psn); + else if (pkt->mask & RXE_ATOMIC_WRITE_MASK) + send_read_response(qp, AETH_ACK_UNLIMITED, pkt->psn); else if (bth_ack(pkt)) send_ack(qp, AETH_ACK_UNLIMITED, pkt->psn); @@ -1195,7 +1274,9 @@ static enum resp_states duplicate_request(struct rxe_qp *qp, res->replay = 1; res->cur_psn = pkt->psn; qp->resp.res = res; - rc = RESPST_ATOMIC_REPLY; + rc = pkt->mask & RXE_ATOMIC_MASK ? + RESPST_ATOMIC_REPLY : + RESPST_ATOMIC_WRITE_REPLY; goto out; } @@ -1335,6 +1416,9 @@ int rxe_responder(void *arg) case RESPST_ATOMIC_REPLY: state = atomic_reply(qp, pkt); break; + case RESPST_ATOMIC_WRITE_REPLY: + state = atomic_write_reply(qp, pkt); + break; case RESPST_ACKNOWLEDGE: state = acknowledge(qp, pkt); break; diff --git a/include/rdma/ib_pack.h b/include/rdma/ib_pack.h index a9162f25beaf..519ec6b841e7 100644 --- a/include/rdma/ib_pack.h +++ b/include/rdma/ib_pack.h @@ -84,6 +84,7 @@ enum { /* opcode 0x15 is reserved */ IB_OPCODE_SEND_LAST_WITH_INVALIDATE = 0x16, IB_OPCODE_SEND_ONLY_WITH_INVALIDATE = 0x17, + IB_OPCODE_RDMA_ATOMIC_WRITE = 0x1D, /* real constants follow -- see comment about above IB_OPCODE() macro for more details */ @@ -112,6 +113,7 @@ enum { IB_OPCODE(RC, FETCH_ADD), IB_OPCODE(RC, SEND_LAST_WITH_INVALIDATE), IB_OPCODE(RC, SEND_ONLY_WITH_INVALIDATE), + IB_OPCODE(RC, RDMA_ATOMIC_WRITE), /* UC */ IB_OPCODE(UC, SEND_FIRST), diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index 9c6317cf80d5..7834285c8498 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -985,6 +985,7 @@ enum ib_wc_opcode { IB_WC_REG_MR, IB_WC_MASKED_COMP_SWAP, IB_WC_MASKED_FETCH_ADD, + IB_WC_RDMA_ATOMIC_WRITE = IB_UVERBS_WC_RDMA_ATOMIC_WRITE, /* * Set value of IB_WC_RECV so consumers can test if a completion is a * receive by testing (opcode & IB_WC_RECV). @@ -1325,6 +1326,7 @@ enum ib_wr_opcode { IB_UVERBS_WR_MASKED_ATOMIC_CMP_AND_SWP, IB_WR_MASKED_ATOMIC_FETCH_AND_ADD = IB_UVERBS_WR_MASKED_ATOMIC_FETCH_AND_ADD, + IB_WR_RDMA_ATOMIC_WRITE = IB_UVERBS_WR_RDMA_ATOMIC_WRITE, /* These are kernel only and can not be issued by userspace */ IB_WR_REG_MR = 0x20, diff --git a/include/uapi/rdma/ib_user_verbs.h b/include/uapi/rdma/ib_user_verbs.h index 7dd903d932e5..175ade79e358 100644 --- a/include/uapi/rdma/ib_user_verbs.h +++ b/include/uapi/rdma/ib_user_verbs.h @@ -466,6 +466,7 @@ enum ib_uverbs_wc_opcode { IB_UVERBS_WC_BIND_MW = 5, IB_UVERBS_WC_LOCAL_INV = 6, IB_UVERBS_WC_TSO = 7, + IB_UVERBS_WC_RDMA_ATOMIC_WRITE = 9, }; struct ib_uverbs_wc { @@ -784,6 +785,7 @@ enum ib_uverbs_wr_opcode { IB_UVERBS_WR_RDMA_READ_WITH_INV = 11, IB_UVERBS_WR_MASKED_ATOMIC_CMP_AND_SWP = 12, IB_UVERBS_WR_MASKED_ATOMIC_FETCH_AND_ADD = 13, + IB_UVERBS_WR_RDMA_ATOMIC_WRITE = 15, /* Review enum ib_wr_opcode before modifying this */ }; diff --git a/include/uapi/rdma/rdma_user_rxe.h b/include/uapi/rdma/rdma_user_rxe.h index f09c5c9e3dd5..845da9cb04fd 100644 --- a/include/uapi/rdma/rdma_user_rxe.h +++ b/include/uapi/rdma/rdma_user_rxe.h @@ -86,6 +86,7 @@ struct rxe_send_wr { __aligned_u64 remote_addr; __u32 rkey; __u32 reserved; + __u8 atomic_wr[8]; } rdma; struct { __aligned_u64 remote_addr;