From patchwork Wed Jan 24 09:05:19 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yury Norov X-Patchwork-Id: 10181949 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id A6604602B7 for ; Wed, 24 Jan 2018 09:06:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9800D286FF for ; Wed, 24 Jan 2018 09:06:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8BAF32871E; Wed, 24 Jan 2018 09:06:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAD_ENC_HEADER,BAYES_00, DKIM_SIGNED, DKIM_VALID, RCVD_IN_DNSWL_MED autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 163E1286FF for ; Wed, 24 Jan 2018 09:06:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=w3WoDSfTG9OfxgazDXL7AE4WFAWySNt0UuR6xQwr+EE=; b=Sfg4C5lo1Ksrf8 pfi29/dieYZ+3Su4MHlA5Fkib3KGY8jnqoMdrz0kNc31f/AougANxSe82D7aSF+uqZqaVVT6/8x+a KP7ZNvqvTdnb4BmRXZqWHWZfUdLE+Y8F40d7eBIb6oNoYHYUIePTbv4KB4X715CZCAKnneqBWuJKk ExFnDtCsaCn7JZ9CB3YOcgze/wOkaWFKQrp3EdvaVvLT84IJ/DCGfgrLMwSGmAIOz6KwKPTT1uj7T iD/RA6Uknw9F3nOHu7rgfUxNdSNEtFD/xmpDk8x0SQvZF28fffBb7eMPkuHOUKesIDl/xRG4vj02A WOLRn0hdvHvxhxhtjZOQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.89 #1 (Red Hat Linux)) id 1eeH1B-0001D9-J2; Wed, 24 Jan 2018 09:06:53 +0000 Received: from mail-sn1nam01on0600.outbound.protection.outlook.com ([2a01:111:f400:fe40::600] helo=NAM01-SN1-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.89 #1 (Red Hat Linux)) id 1eeH0n-0000hd-R7 for linux-arm-kernel@lists.infradead.org; Wed, 24 Jan 2018 09:06:36 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=qwhuUzTM4UlNVLxRAhnJ4WsIDxqF6tqfen4PNzEo55I=; b=R/7ocBsGzlMfdm4wxccmgW5ppX98SvQR4LOq3w1jgFG4OU29JRHfSh3Mt+ozyhymKXKo1kPzX/w09wQmGe3c+9g/Illr0Zh3UQLxGOHsm7Jkx3rUi/QXDZ9Pl2i+TPThO3Gt4DGV0qEDudKscc1BHvSj4ZkbGVPcdqoeObelZ6U= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Yuri.Norov@cavium.com; Received: from localhost (42.106.18.247) by BN6PR07MB2898.namprd07.prod.outlook.com (10.173.28.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.428.17; Wed, 24 Jan 2018 09:06:17 +0000 From: Yury Norov To: linux-arm-kernel@lists.infradead.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org Subject: [PATCH 3/3] arm64: enable 128-bit memory read/write support Date: Wed, 24 Jan 2018 12:05:19 +0300 Message-Id: <20180124090519.6680-4-ynorov@caviumnetworks.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180124090519.6680-1-ynorov@caviumnetworks.com> References: <20180124090519.6680-1-ynorov@caviumnetworks.com> MIME-Version: 1.0 X-Originating-IP: [42.106.18.247] X-ClientProxiedBy: DB3PR0202CA0025.eurprd02.prod.outlook.com (52.134.64.166) To BN6PR07MB2898.namprd07.prod.outlook.com (10.173.28.144) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ef9042ce-8694-44b9-a1ea-08d56309baaa X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(4534165)(4627221)(201703031133081)(201702281549075)(5600026)(4604075)(2017052603307)(7153060)(7193020); SRVR:BN6PR07MB2898; X-Microsoft-Exchange-Diagnostics: 1; BN6PR07MB2898; 3:uybUIkqTCkU+CDpD1E8c93yIjeIzGPbIbZSP2tGTimC4NYf6Kq5Nm94kq/vZfSq8N0yENRzvInaNpx7f1aPoXemBdOWGwnQWlW+PIo3zKkz/6m6W6gH5DjnSkJ63mX8ARfWH3z16RIzO2kCL+DjcjM7bXSDoBNDTHzHd1+Q+j2HboFTdW4AdZWMtTLVzaIwUw2v8/+gWofmiW7ovhAdt1xmI59ViOzWS2iyq8NlxPNLDVWLTn5aqEUt90EtYnMvw; 25:A+/3BRiBHjKuty+ODNuC0ReNBgYnrTOI2WULXdW7MuUdwon1oDORi7luiQoQxdcozo65+I6uKXGcsG4nYYC+8qSQcxhb6Dy/NfUB+640s9cXetbvFgLzrsGkIRdQCIjSvLC10Vse2LM44T0be1dmFt6W7SKMCI7fwv7mUhyp4Vs/P0UmHYJCDRKyLeTvy4c0bqCTg9bgqHPnFy3m9ZjW5AzIfDJuG73eInO8bTkzdOSgSOrvDjLHj/+0ad4qI+PtOL4IZgphFa3vS0Kz2r9nBCCBe0Q1MaHIUo/n/VZm1BoNd//80Wr2atvMCaaaMDmrOgqLGXlR8ksCNx2QPnbJHg==; 31:0eiWmqUaOWzBt24T5Q4ZfmFKyEP1KExnJEJDwG8qVj/v47bRk+GNALzpNAbcWBGUbJAw4DcajRIb+CtdWnJPSzpt87j6h/gBb/fUexswBZUpAadAi52hVudgeT+5F3VA98KLNYJraNJTZmaZzju8JZlkZ4QNdMoyw/hlHRmpBVf5ctuEUFnmCCJlj2D/g6Hwy3kS9xafwQCs8xNLVNyZktJ3rVirFkvuvaCu01WpAxo= X-MS-TrafficTypeDiagnostic: BN6PR07MB2898: X-Microsoft-Exchange-Diagnostics: 1; BN6PR07MB2898; 20:xxwyaBAiJfuAWatWeusswnSbuu9wuTIe7yQU8LtMNDFKrf0R+09LoMsgBFG/h4mxASB14IRnQl1C//e1hdKlwuDURPTkpsunu9XH5Go4qDJ+3vlg/gWKRVlVhXTzU3oXCH1IzCDQkKzlg7oOH4a07OJXl88BWiR22vgcV/ICltMYe4ta3LIsbcBDQ8kEXYHyoL6KO5gWK6ZJpW9KSP4Eq6CLImTIL1bUuDcXd59KVPgmDPAP552AEqHmeT5MD5b6bJvxhjf/+KyOiOfEHBYrWl4nI6/sCKH+9aCF+qB8UPKJz24JRghGTWRSknwOctehRqRgHW0t6A7DGtkWNmmjWcMi88Pl/E/b2dJF8lRHPu2Hc+AFcsD7zwBU1oDr921FoCMHaejX3lu7nYJHRr/GDN6vW/JKKx/9ml9hAlPeC0lhLRWuPrT8aCJlHm6LNb4Il0RXcifAhlHugT/38Jx9stNlA86e+fOY3nDwJBe5x9IiraBK8sZQf0G7G5X/nP+bmkmcMspWKTft2RSaRROu/zjEOfdEw4B8yW1Xcwpy4GNg/UWEVbetTrakS8dJ66QlDabfgJxD0EKiIq2+0w0ZuqhzIMQ+wCzGn1qtgHMCpBU=; 4:h0I8+HSjnOg56i+7qeRhoGizGP2CqKjNgck9rblmc4iwtsF9acRp52Z3Vt12i27LCbimXkjoeW2AxwZFaORVRTaoigPyTueUEFW9+hcBBQPIoo2VB08jja40TIY9ZbEArvkVdTW+lZI9BSmZXC6/0BzsmPgZ6iVs78RhJwrCntFtgre47EnJoRtQ1gBrXNaUwdPCqOU8fH5gvXTp4bYQgO5xPaPZpoTv75X3FDVabE8LK+GAwd7WLMbEvn+EvqYe2fxg3LRtCyi4IwjUiuS9NQ== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040501)(2401047)(8121501046)(5005006)(93006095)(10201501046)(3002001)(3231023)(2400081)(944501161)(6041288)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123562045)(20161123564045)(20161123558120)(20161123560045)(6072148)(201708071742011); SRVR:BN6PR07MB2898; BCL:0; PCL:0; RULEID:; SRVR:BN6PR07MB2898; X-Forefront-PRVS: 056297E276 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6069001)(39860400002)(396003)(346002)(366004)(376002)(39380400002)(199004)(189003)(48376002)(316002)(8676002)(76506005)(25786009)(8936002)(4326008)(72206003)(39060400002)(50466002)(6496006)(81156014)(53936002)(50226002)(6486002)(305945005)(16526018)(7416002)(105586002)(81166006)(97736004)(5009440100003)(59450400001)(26005)(66066001)(386003)(47776003)(7736002)(54906003)(5660300001)(51416003)(52116002)(3846002)(6116002)(42882006)(2950100002)(16586007)(68736007)(1076002)(575784001)(76176011)(36756003)(478600001)(106356001)(2906002); DIR:OUT; SFP:1101; SCL:1; SRVR:BN6PR07MB2898; H:localhost; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BN6PR07MB2898; 23:jKvuqa9nGgx2+vMApTS62mfiXpf70nOTxA5PpjoRB?= =?us-ascii?Q?K0zPrSn1cPguUiiaKEoSeW+0PzRNH/2hg3Bx6C7f0LvMNd3kpwQQ3/jGKbsI?= =?us-ascii?Q?I47xLB767Mw/rSObHa9WydGOD9O19RAML2UhB0WX5UgGgdYJootDieg52yeW?= =?us-ascii?Q?MyFNHEjijpf4YZ4q68KCg6ZXLi8fNOCH0qsv6UhMlw3nctP3NCuCPGavvdy/?= =?us-ascii?Q?zE6jgLdVo1ckWACg86BGfJXOPTcjoUDJjFKoclb1n6Tis/N662FjfBF+49x9?= =?us-ascii?Q?DA6hXwxnlVDwQi6NAoRKJuT462FmunN14/KTZqvgUPQjAinwshRJ2NJcMurD?= =?us-ascii?Q?Vdy95B+YbDuO9mqiNeV5Gv5ngDa5WsoJ1F6pUUtkTJqBkI3lqHbT0SUqgUPi?= =?us-ascii?Q?jJemDVT24xdIS1pXGR4kRURS61vX1A9LBEJ0YRUhPMCcdOtcu8dxtYwwe/zV?= =?us-ascii?Q?+UMg3Eq3LknXuqZlQHEsfOy72AAadXTV4Ikd2zg9MlhZR9BOGDagvzqSvKGQ?= =?us-ascii?Q?GurBNFd3yYTzN4tTDGftJB0p0no7mibIWEYs5rJyT0l45J0/AXhkUUYL2u5p?= =?us-ascii?Q?9BYPQo5vEtLKtMLatmQPw+7gcT1i9E+za9bmGc76WROZb6baWO8StcwutJPG?= =?us-ascii?Q?HVqNi5Prj/F0lIPofe6c6uRdHAZ222uM/ujiqE9U1ffFyrsudimbpp8PhKgX?= =?us-ascii?Q?CHvN7fOr0BocvArl4X1j4bRbuBMjz57SfvczTMgI6BxWKEJFmphpDZuk56Fa?= =?us-ascii?Q?HOuFfsn/wxESQEawsHc6u1GWmMHtctsiqi7HvNTALgLXie2q/HI09kMPdcBv?= =?us-ascii?Q?00GxGhEyXbmTwZxYv4UoeIXnc6rAxOTFiyPnwf4iHPUqmCzkpr2gXoKtk/Sn?= =?us-ascii?Q?A5XiHu3xfIxDFtXcD92rx39eFP1IztjAt6qrhLfQZc3Ac01eU8mkwTaUORvH?= =?us-ascii?Q?mQBGHrI0Ln94hK47pdX5j5CRRKFi7pO5FKRPIZ8OgNVpTu/Pq1dbvSAegCfS?= =?us-ascii?Q?Q8X+hrD+y4CTBVW0ZoTmpnMfDSe+JWSXjLxHSaHeQXYPvP8ie7SEeuTeMWLl?= =?us-ascii?Q?3NfSdr7HBz1XropbuspESVObmwrcFKJw+0PEciCORdR9rsTLl91BgWw1C5gL?= =?us-ascii?Q?RAWIUuC9VoIWU1ytfeRv/InizUS5BF4IB+7ue1hgHd282mJ8zdTIdWQdgP3v?= =?us-ascii?Q?Uu+x/UO1mEyjLoQArdwAm7NA/k+YEZcC8S+iAHddSp8PVbiT5ZfnL8Iki6R0?= =?us-ascii?Q?20/DDulXMpToany70s=3D?= X-Microsoft-Exchange-Diagnostics: 1; BN6PR07MB2898; 6:0Ov7Mw7ymk/f+ZmCVxl/gA9teLCNeC4gKSD9fqCoHpjAHyMRCH1cZDKp33UFYzNsRfxeVTXE0H2g6Ep8/FJSDlPeaWKgXzSO/nxZ84f5QirGc/JBZt/7tR555zk+rhM+4YeITgBkbjDdLydGN7nkEtBXwag41sb1nDOphhhrqAhUfqK9skOca8SZvU7vm6s65P4VBg3BiyNTA9sa7M9epHwVMArPCF3Rv79npBTaDjo2ySIG8/K0g8fnKQhoszElKzQkzhfFls1iSJnoz3bhbRZK3FwIAk2TXfHI7r+JduotwrDBFmF7xsezLWbM+zrzGJnEMxo3+gOrHunmnvsxMw8EmCJLTP9LHF6aIJSUoZo=; 5:xwUsDVwB9gRTjhaMEFqwFT6G64Twp/mKnYcl3H92QF24rZwp0Uxn8HN7atPfTQTZeIDmbKG3k4YW3l7ioy4fGrtsIqD7dHJOq+xnReD/Ok3bHLLxnMsCTR9Tgsla8MltT1oA0sFE+dYPV1xJnlSBmQB83oQ75fCS7tfYEW2Pigk=; 24:6zKKbrabk2B4zOZb9lsM1eyLZbwnHA0tASoHAsruIbQl6NS1WwTrgDe/R1hXXRQG2aOS9vTQ86tJiLMiJuEkmsRy27q6azhbUKbokgOFwAk=; 7:pLDVKdV5xzMwAVVVxLmgF34WN0Rzqg8OOf1EVpeSZqOFbuA15PlpDFhE7Gmr04DzqQYhrE/axqt9bXXVsXGfOjbfm1EjIW3K9RkLI6+AAJ5XgrdTbEZAARBPFu9aD0/8hWRD98ZweyEGbugr4DzPPdgx5CYEjSyWxONG9jWi0+ncXHDjH/m0Ifp2XDhU/V3U4rJW2xxQkipu/P+Hk+6qgO5mf9mw2NHIrfrHG774HFcHhpHQvjYk+9MyOPNcNmLX SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jan 2018 09:06:17.1487 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ef9042ce-8694-44b9-a1ea-08d56309baaa X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR07MB2898 X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Andrew Pinski , Sunil Goutham , Kees Cook , Arnd Bergmann , Catalin Marinas , Will Deacon , Nicholas Piggin , Ingo Molnar , Geethasowjanya Akula , Yury Norov , Al Viro , Greg Kroah-Hartman , Andrew Morton , Laura Abbott , "David S . Miller" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Introduce __raw_writeo(), __raw_reado() and other arch-specific RW functions for 128-bit memory access, and enable it for arm64. 128-bit I/O is required for example by Octeon TX2 device to access some registers. According to Hardware Reference Manual: A 128-bit write to the OP_FREE0/1 registers frees a pointer into a given [...] pool. All other accesses to these registers (e.g. reads and 64-bit writes) are RAZ/WI. Starting from ARMv8.4, stp and ldp instructions become atomic, and API for 128-bit access would be helpful for core code. Signed-off-by: Yury Norov --- arch/Kconfig | 7 +++++++ arch/arm64/include/asm/io.h | 31 +++++++++++++++++++++++++++++++ 2 files changed, 38 insertions(+) diff --git a/arch/Kconfig b/arch/Kconfig index 76c0b54443b1..2baff7de405d 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -116,6 +116,13 @@ config UPROBES managed by the kernel and kept transparent to the probed application. ) +config HAVE_128BIT_ACCESS + def_bool ARM64 + help + Architectures having 128-bit access require corresponding APIs, + like reado() and writeo(), which stands for reading and writing + the octet of bytes at once. + config HAVE_64BIT_ALIGNED_ACCESS def_bool 64BIT && !HAVE_EFFICIENT_UNALIGNED_ACCESS help diff --git a/arch/arm64/include/asm/io.h b/arch/arm64/include/asm/io.h index 35b2e50f17fb..7c5d834abfd8 100644 --- a/arch/arm64/include/asm/io.h +++ b/arch/arm64/include/asm/io.h @@ -60,6 +60,18 @@ static inline void __raw_writeq(u64 val, volatile void __iomem *addr) asm volatile("str %x0, [%1]" : : "rZ" (val), "r" (addr)); } +#define __raw_writeo __raw_writeo +static inline void __raw_writeo(__uint128_t val, volatile void __iomem *addr) +{ + u64 l = (u64) val; + u64 h = (u64) (val >> 64); + __uint128_t *__addr = (__uint128_t *) addr; + + asm volatile("stp %x[x0], %x[x1], %x[p1]" + : [p1]"=Ump"(*__addr) + : [x0]"r"(l), [x1]"r"(h)); +} + #define __raw_readb __raw_readb static inline u8 __raw_readb(const volatile void __iomem *addr) { @@ -105,6 +117,19 @@ static inline u64 __raw_readq(const volatile void __iomem *addr) return val; } +#define __raw_reado __raw_reado +static inline __uint128_t __raw_reado(const volatile void __iomem *addr) +{ + u64 l, h; + __uint128_t *__addr = (__uint128_t *) addr; + + asm volatile("ldp %x[x0], %x[x1], %x[p1]" + : [x0]"=r"(l), [x1]"=r"(h) + : [p1]"Ump"(*__addr)); + + return (__uint128_t) l | ((__uint128_t) h) << 64; +} + /* IO barriers */ #define __iormb() rmb() #define __iowmb() wmb() @@ -120,11 +145,13 @@ static inline u64 __raw_readq(const volatile void __iomem *addr) #define readw_relaxed(c) ({ u16 __r = le16_to_cpu((__force __le16)__raw_readw(c)); __r; }) #define readl_relaxed(c) ({ u32 __r = le32_to_cpu((__force __le32)__raw_readl(c)); __r; }) #define readq_relaxed(c) ({ u64 __r = le64_to_cpu((__force __le64)__raw_readq(c)); __r; }) +#define reado_relaxed(c) ({ __uint128_t __r = le128_to_cpu((__force __le128)__raw_reado(c)); __r; }) #define writeb_relaxed(v,c) ((void)__raw_writeb((v),(c))) #define writew_relaxed(v,c) ((void)__raw_writew((__force u16)cpu_to_le16(v),(c))) #define writel_relaxed(v,c) ((void)__raw_writel((__force u32)cpu_to_le32(v),(c))) #define writeq_relaxed(v,c) ((void)__raw_writeq((__force u64)cpu_to_le64(v),(c))) +#define writeo_relaxed(v,c) ((void)__raw_writeo((__force __uint128_t)cpu_to_le128(v),(c))) /* * I/O memory access primitives. Reads are ordered relative to any @@ -135,11 +162,13 @@ static inline u64 __raw_readq(const volatile void __iomem *addr) #define readw(c) ({ u16 __v = readw_relaxed(c); __iormb(); __v; }) #define readl(c) ({ u32 __v = readl_relaxed(c); __iormb(); __v; }) #define readq(c) ({ u64 __v = readq_relaxed(c); __iormb(); __v; }) +#define reado(c) ({ __uint128_t __v = reado_relaxed(c); __iormb(); __v; }) #define writeb(v,c) ({ __iowmb(); writeb_relaxed((v),(c)); }) #define writew(v,c) ({ __iowmb(); writew_relaxed((v),(c)); }) #define writel(v,c) ({ __iowmb(); writel_relaxed((v),(c)); }) #define writeq(v,c) ({ __iowmb(); writeq_relaxed((v),(c)); }) +#define writeo(v,c) ({ __iowmb(); writeo_relaxed((v),(c)); }) /* * I/O port access primitives. @@ -188,10 +217,12 @@ extern void __iomem *ioremap_cache(phys_addr_t phys_addr, size_t size); #define ioread16be(p) ({ __u16 __v = be16_to_cpu((__force __be16)__raw_readw(p)); __iormb(); __v; }) #define ioread32be(p) ({ __u32 __v = be32_to_cpu((__force __be32)__raw_readl(p)); __iormb(); __v; }) #define ioread64be(p) ({ __u64 __v = be64_to_cpu((__force __be64)__raw_readq(p)); __iormb(); __v; }) +#define ioread128be(p) ({ __uint128_t __v = be128_to_cpu((__force __be128)__raw_readq(p)); __iormb(); __v; }) #define iowrite16be(v,p) ({ __iowmb(); __raw_writew((__force __u16)cpu_to_be16(v), p); }) #define iowrite32be(v,p) ({ __iowmb(); __raw_writel((__force __u32)cpu_to_be32(v), p); }) #define iowrite64be(v,p) ({ __iowmb(); __raw_writeq((__force __u64)cpu_to_be64(v), p); }) +#define iowrite128be(v,p) ({ __iowmb(); __raw_writeo((__force __u128)cpu_to_be128(v), p); }) #include