From patchwork Wed Jan 24 09:05:18 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yury Norov X-Patchwork-Id: 10181953 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 67C6F602B7 for ; Wed, 24 Jan 2018 09:07:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5A3DF286E6 for ; Wed, 24 Jan 2018 09:07:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4C8C82871B; Wed, 24 Jan 2018 09:07:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 989E8286E6 for ; Wed, 24 Jan 2018 09:07:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932758AbeAXJHA (ORCPT ); Wed, 24 Jan 2018 04:07:00 -0500 Received: from mail-by2nam01on0080.outbound.protection.outlook.com ([104.47.34.80]:56221 "EHLO NAM01-BY2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S932669AbeAXJGC (ORCPT ); Wed, 24 Jan 2018 04:06:02 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=H07m8OD2Jn5Z2EERxjjLjkQBf4h/XvBqbcsSNXRsnbo=; b=llx8AdBRGFaNuaU35A7d9dqyGGuyMp28mFt4JtLDXW6YqIy1BN5f5E9B6vyyYW9F0bxUpean/V37w30UH8fla5x2Zw0GyVVxfImJ1K2wz/bmErDneW6c0O3sQJaCNc8aXcQP3WJxnqyzIdWPzRVi90KpEdlzKUZ7cFcIhLmdYNg= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Yuri.Norov@cavium.com; Received: from localhost (42.106.18.247) by BN6PR07MB2898.namprd07.prod.outlook.com (10.173.28.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.428.17; Wed, 24 Jan 2018 09:06:00 +0000 From: Yury Norov To: linux-arm-kernel@lists.infradead.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org Cc: Yury Norov , Al Viro , Andrew Morton , Andrew Pinski , Arnd Bergmann , Catalin Marinas , "David S . Miller" , Geethasowjanya Akula , Greg Kroah-Hartman , Ingo Molnar , Kees Cook , Laura Abbott , Nicholas Piggin , Sunil Goutham , Will Deacon Subject: [PATCH 2/3] asm-generic/io.h: API for 128-bit memory accessors Date: Wed, 24 Jan 2018 12:05:18 +0300 Message-Id: <20180124090519.6680-3-ynorov@caviumnetworks.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180124090519.6680-1-ynorov@caviumnetworks.com> References: <20180124090519.6680-1-ynorov@caviumnetworks.com> MIME-Version: 1.0 X-Originating-IP: [42.106.18.247] X-ClientProxiedBy: DB3PR0202CA0030.eurprd02.prod.outlook.com (52.134.64.171) To BN6PR07MB2898.namprd07.prod.outlook.com (10.173.28.144) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 18a33b42-9f59-4883-9904-08d56309b0a7 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(4534165)(4627221)(201703031133081)(201702281549075)(5600026)(4604075)(2017052603307)(7153060)(7193020); SRVR:BN6PR07MB2898; X-Microsoft-Exchange-Diagnostics: 1; BN6PR07MB2898; 3:pR3mgXcBaC724VcHsQtuOVYqpQ0CcGrYxasiyu91MHv7fBBZ8WswotuyTxfAGcZAkk/2zqI2yz7VllC4esqF9x4w8j8ZShH85bU4QmVVIUIZAG5OUxtu7xXvdNv2Uj1PRp3gNDEjEDjwOB7gwbhEUo+J+D3IsLgA8MeHIaJxIYR6tcV+qnJD6+cUO/MUVkpX3tnLByJRSeApNc4iFYIZeSjIK27v1cwCyJ/uc71sYBoXse36ZAF3zv44XGSBQZy6; 25:xOfSowblPnSnTbOGXEu8aDWOwIx0CGe15xG+7oSVr4UPaUBoAoY3T6VxKVqF8d2YzkXIseCwB+5dalyMjwW2xSkkORBkTAdNTtZeO7mv3mJHrq1AE7MTSgX9pqfPC9dFJZvKwCA1IsGBBbOtNt9RndTFLcrPzH7oD2ESJXH5uZr+Thyv0sMKv+0S/PeO+ZGyAyH9mWrLsgDAXkx6GcDKZ/mcE6Xyn6bN4IVLwljhhyRcnZk1UdZB/EyhlJ/SnR0+1b9qB0mOPVQ3HrDHclP3I3pP0Hzc1MZrhBUBwjdgX5nAWIuZ7I2xFV4J1MA1IPBo7Zt4WKJf8os9vnGg9KYCVg==; 31:diavsGHzKY9ma5pqsctnbvfxRmczp56o+DhjxS8uviTTih9tOrMfCZwOzHcRC04osL92tclPLK0i1jTf1LTSUnrZIGmAJuretqxyT/qir11EeXUxNDOp+Tk0E26/UAIFhLA4Tq+5/5q7/D0WVioKLatCAxsXMRQl02eC5qJkJcXN3NKUl1P9uEuTOx+D6B4yYhHNG9lkjW2em4Fik5WO5SttFTs44/icoM9EqsLRc1c= X-MS-TrafficTypeDiagnostic: BN6PR07MB2898: X-Microsoft-Exchange-Diagnostics: 1; BN6PR07MB2898; 20:RrwDbaUA9rGRAF9/aTsqIYdm4oU3sXja4Pkdwj/786Sauyzn4CfWFIQqUtfVrCm4qpVKaOwu04I6JphavEQ8z5B71LPincx2+Ro4kQHmJtx5+y5F1HGpmOFwkQphfFCkE4tBneH2BXfMyyXWqsw9kQUrFNJHBMHZ4IAszcsAsMINy40PrIK1+iFW/f+qTf2LNAfecSrCk+qudEyBZKiJlvtEk4M8LjPJY+mrPZXFiC2TKcnQgmkMHnuYdJDsV6fpK1k1dDhDFjAxLE80q9YRc8PrZBKSBsZAXFngZpZ9RhruKXHg+fRF7L4zzMALfj9VaOl4O1uFtLYz2BEOIYOwIqLvS08tEJ6DWmvB3HpZ6NgEL6SLFLwHKOWfTgWgTAsQnN7mKTeVoHsZzYFeWiBWzfjzsbBO8TIESVnWE5OJ+RBnyGFTxXRbImbqUj91o/HbMTmkOm8R+e7AGY8oaUQAHZtQoOgeuwcPychbNAFomMb/n47BLNBr6uwZrC3ZnzI5nj6T4d3oW/gJG4F+ecQ5rgMobvDtFbTrPzVmdrd+4EmJMSN7aT+DawzvlNqpYPL/d+ZtE2XlOil8Xj2WU9/xRBF4WDZ1jJ+gA7Bkf1RZ6DI=; 4:vXp5gghSkJiGVFfGtT8UCe97Y+ZaSdW9RNMaR80yq8ysBccMUVlS4/rPoOD2H/nIM9oN+WzeNnn77iY+ZKtFeEWOsmujWQAQ46biCMUqooyJi5515RaXZpp88fVIrk3Djjckv0Y1FITX9vS+XxxZOfS2tApVu+RxpJYK1e+RviQipzdoPRMZ+gF9dJDjFgcZnIfLQfPeI61IkkRFXOmETcZTC5yD+lzg5RCLBFkMyad0nMg6SLwBk5bpqATfmmwLPsuHbL9Ly8sQWRmRrqtExA== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040501)(2401047)(8121501046)(5005006)(93006095)(10201501046)(3002001)(3231023)(2400081)(944501161)(6041288)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123562045)(20161123564045)(20161123558120)(20161123560045)(6072148)(201708071742011); SRVR:BN6PR07MB2898; BCL:0; PCL:0; RULEID:; SRVR:BN6PR07MB2898; X-Forefront-PRVS: 056297E276 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6069001)(39860400002)(396003)(346002)(366004)(376002)(39380400002)(199004)(189003)(48376002)(316002)(8676002)(76506005)(25786009)(8936002)(4326008)(72206003)(39060400002)(50466002)(6496006)(81156014)(53936002)(50226002)(6486002)(305945005)(16526018)(7416002)(105586002)(81166006)(97736004)(5009440100003)(59450400001)(26005)(66066001)(386003)(47776003)(7736002)(54906003)(5660300001)(51416003)(52116002)(3846002)(6116002)(6666003)(42882006)(2950100002)(16586007)(68736007)(1076002)(76176011)(36756003)(478600001)(106356001)(2906002)(41533002); DIR:OUT; SFP:1101; SCL:1; SRVR:BN6PR07MB2898; H:localhost; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BN6PR07MB2898; 23:bYtGkBuWfYVK4up5/kpXdq4zW83v57ERddGijYc1E?= =?us-ascii?Q?4cBYv/S+Uv2L1MwgEyEiBucOeUVp3rUuzefIwORnWG2l1DflYyorRu3PB+yj?= =?us-ascii?Q?leNcX9PzapPTaqab1pfQzY/TpwkuRhmcKUWwDuOOY47H2suOIj/EUYplPu6D?= =?us-ascii?Q?bGADdV2WZKysxyVkRxg9l92gQaD7QOlsbf8gYl3J2eWs/hZ7KRob/MYazGSG?= =?us-ascii?Q?IGarOm4is0zg0vA6BwVMQxDRV1++gtqTrgFi+AzEofSfLwCalW9E/+euP7kB?= =?us-ascii?Q?PuRPn8Ie3xl2EaXdoXZ2nQxO8rLziRIO2vjeIJYxrudt18lVJB65xHBRjKip?= =?us-ascii?Q?qtynzy+Nmnb/DkoDNl7xJWY74LBtwVoRPKe5a414d16S3DDllE4IyALs/vy6?= =?us-ascii?Q?SABRCOc7CgbtyX6BTY+B15+mFYmOcYcyqKpYxpBnvE/YdTXUCl4TAzHi/lVI?= =?us-ascii?Q?bDoqoIIWBA9KN/iOEPE04MoJ5h44nAP2oB223GOBgvXxU36Kb9hMcq703FBQ?= =?us-ascii?Q?ke2MLeHR0ZYH5kf7QjrKATQYYgqVZIIcbfGqdALnjs2AXl+r3xm/ptghqcWZ?= =?us-ascii?Q?dm7J7Il3hf0Wb6LhsQm1upg8Hsz9QPt9zIy3cgFYkN+lZbYFIRzqmsFmGtkh?= =?us-ascii?Q?ayAD+MPg4mMpUmLmbWvM4XRM+rGR1MWY6fp73QMUGUKs+48oRFJ5Zy3tRcmU?= =?us-ascii?Q?X4L/m7QH65MSTjx4gpDXJHonMC/f1l971TFbgQE0xsFpxXLmv+4q85W4219s?= =?us-ascii?Q?3efM3ym7JutwGy2V/99db1y3FX590Pt0rR0xKyZHPqwviE9KH/KB1S/IMLQa?= =?us-ascii?Q?QZ+cjmLS4WTFHdmlLRF1vSokOoxCYI9ZhfjHDEYOrsRTsmoRCb2IDoz5V3SY?= =?us-ascii?Q?YbsgswZ/gKFtlnIuC65gdeJ+KYyNM10xRVCSVQz3i6eiiJxKLejXecgmkwrd?= =?us-ascii?Q?xvbD1A8w2xAEb90OacFDtVEmyMvB8dG0Rj/MNUSFU/uTjFqe+E+6JprLb7UU?= =?us-ascii?Q?En9BIx/G2qNQHIzay3RqHDJSyr9uliIsO9edeyPsmc23uDVMhPmxJkgKOosi?= =?us-ascii?Q?5DLAqGWz0GhfMb/5nVTsXgSs/5CvF7cpHzryJPh6g3CQkAXQtscDD6dGS8Y4?= =?us-ascii?Q?Z7R/f9Mzkq1pLkJTxMGDfAvkzacu7sO0kGK0iEIZfYW5aQ0N255/0nDtgjJB?= =?us-ascii?Q?0WK802xUWex/zTLePMjqG5II9dECe4DjfZNbVI/KWmFX1WgjJYyR5H8fSJYZ?= =?us-ascii?Q?AlayuVU7uBoIIOvMhs8KGZzcYbDtW+LzVbnV1Yy?= X-Microsoft-Exchange-Diagnostics: 1; BN6PR07MB2898; 6:ILl5luyCUGSNMfJTrGHD3ScitDECN/6jyRzNtJ8bHhfehPe9W0KschUq++8CxEOIVgHL9mDz4y1xlGlRtdKZalSR9oL2HsD5FNQX2+sI96gNq0Bz+w90Vcj1r56TC7L8nCKhsP/VYJRzEQ7+bvKy5zLleCT0mfz+TqWunm5L/JkcUVpQaz2aSykq0JqltPYFv1PHQ+B0oN4dKowUWGeGHOy6z/ZIcHKZ+L7Dr6b6qNvPD+CGZPDwe3yDChqUOOTww1soDYEF6DBBhoPgJPMSUaeDikxhdPMXLZWi1R4sVhXEZoHVrb7WRKgUtNVHUteZZJ1fSqgXCsflv5AfUN5bhbg/Xxl919Vl78hD53sWjBc=; 5:19mtmRqyMrdkqgTdnCMlYonyzqFaxlPz60qb+i/7N76pWcsYJlfRd9zEnTFbwq6ClBu0mRJ6zb95HFjA2qc6Te0Z3/IYGuS98Tutdq3TeCH7Ab3Hs/Es2sP+GY87HDOAWXJ4QxsYfa49UR6CqdY89/n1zdeLNkIRQ08Dvwc6rzQ=; 24:eQd8vztPIWhPYaIZUmmfJbDkAON2387G3ey9QWmF3f3RsutnDv6vR0rR85JpXh1xCE25Z+bjw8da2vfoZdi72rV1W8Irsa38252bOKII5+Q=; 7:vvmUtAU9ArmFDPuxyVQk7sDmIpNUVCitxN508wMQhH0SmpR4ojk1gJPt3y4DIKTzJNyWSJJw3kA2GP1L2ub5/E92hQpqojG+CrkCAwhhatTfmy4ziROKAs62lomJaDPhQTBO/d6GfNgVzJ8Tu9Pcs6bKybwcZYmn/q9P3FWYLCC792vd8IsfYUKsd1Np3v6mnKV21qE8ZD8B4gKqDP/ea7u8mHGuuTKl3lQNB40E/+kwYibbNHPaN33/Oz+WN9ta SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jan 2018 09:06:00.3203 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 18a33b42-9f59-4883-9904-08d56309b0a7 X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR07MB2898 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Some architectures, like arm64, support 128-bit memory access. For ARM64 - using load/store pair instructions. This patch introduces reado() and writeo() functions family, where suffix 'o' stands for reading and writing the octet of bytes at once. Signed-off-by: Yury Norov --- include/asm-generic/io.h | 147 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 147 insertions(+) diff --git a/include/asm-generic/io.h b/include/asm-generic/io.h index b4531e3b2120..ac4a4de69efc 100644 --- a/include/asm-generic/io.h +++ b/include/asm-generic/io.h @@ -67,6 +67,16 @@ static inline u64 __raw_readq(const volatile void __iomem *addr) #endif #endif /* CONFIG_64BIT */ +#ifdef CONFIG_HAVE_128BIT_ACCESS +#ifndef __raw_reado +#define __raw_reado __raw_reado +static inline __uint128_t __raw_reado(const volatile void __iomem *addr) +{ + return *(const volatile __uint128_t __force *)addr; +} +#endif +#endif /* CONFIG_HAVE_128BIT_ACCESS */ + #ifndef __raw_writeb #define __raw_writeb __raw_writeb static inline void __raw_writeb(u8 value, volatile void __iomem *addr) @@ -101,6 +111,16 @@ static inline void __raw_writeq(u64 value, volatile void __iomem *addr) #endif #endif /* CONFIG_64BIT */ +#ifdef CONFIG_HAVE_128BIT_ACCESS +#ifndef __raw_writeo +#define __raw_writeo __raw_writeo +static inline void __raw_writeo(__uint128_t value, volatile void __iomem *addr) +{ + *(volatile __uint128_t __force *)addr = value; +} +#endif +#endif /* CONFIG_HAVE_128BIT_ACCESS */ + /* * {read,write}{b,w,l,q}() access little endian memory and return result in * native endianness. @@ -140,6 +160,16 @@ static inline u64 readq(const volatile void __iomem *addr) #endif #endif /* CONFIG_64BIT */ +#ifdef CONFIG_HAVE_128BIT_ACCESS +#ifndef reado +#define reado reado +static inline __uint128_t reado(const volatile void __iomem *addr) +{ + return __le128_to_cpu(__raw_reado(addr)); +} +#endif +#endif /* CONFIG_HAVE_128BIT_ACCESS */ + #ifndef writeb #define writeb writeb static inline void writeb(u8 value, volatile void __iomem *addr) @@ -174,6 +204,16 @@ static inline void writeq(u64 value, volatile void __iomem *addr) #endif #endif /* CONFIG_64BIT */ +#ifdef CONFIG_HAVE_128BIT_ACCESS +#ifndef writeo +#define writeo writeo +static inline void writeo(__uint128_t value, volatile void __iomem *addr) +{ + __raw_writeo(__cpu_to_le128(value), addr); +} +#endif +#endif /* CONFIG_HAVE_128BIT_ACCESS */ + /* * {read,write}{b,w,l,q}_relaxed() are like the regular version, but * are not guaranteed to provide ordering against spinlocks or memory @@ -195,6 +235,10 @@ static inline void writeq(u64 value, volatile void __iomem *addr) #define readq_relaxed readq #endif +#if defined(reado) && !defined(reado_relaxed) +#define reado_relaxed reado +#endif + #ifndef writeb_relaxed #define writeb_relaxed writeb #endif @@ -211,6 +255,10 @@ static inline void writeq(u64 value, volatile void __iomem *addr) #define writeq_relaxed writeq #endif +#if defined(writeo) && !defined(writeo_relaxed) +#define writeo_relaxed writeo +#endif + /* * {read,write}s{b,w,l,q}() repeatedly access the same memory address in * native endianness in 8-, 16-, 32- or 64-bit chunks (@count times). @@ -281,6 +329,24 @@ static inline void readsq(const volatile void __iomem *addr, void *buffer, #endif #endif /* CONFIG_64BIT */ +#ifdef CONFIG_HAVE_128BIT_ACCESS +#ifndef readso +#define readso readso +static inline void readso(const volatile void __iomem *addr, void *buffer, + unsigned int count) +{ + if (count) { + __uint128_t *buf = buffer; + + do { + __uint128_t x = __raw_reado(addr); + *buf++ = x; + } while (--count); + } +} +#endif +#endif /* CONFIG_HAVE_128BIT_ACCESS */ + #ifndef writesb #define writesb writesb static inline void writesb(volatile void __iomem *addr, const void *buffer, @@ -343,6 +409,23 @@ static inline void writesq(volatile void __iomem *addr, const void *buffer, #endif #endif /* CONFIG_64BIT */ +#ifdef CONFIG_HAVE_128BIT_ACCESS +#ifndef writeso +#define writeso writeso +static inline void writeso(volatile void __iomem *addr, const void *buffer, + unsigned int count) +{ + if (count) { + const __uint128_t *buf = buffer; + + do { + __raw_writeo(*buf++, addr); + } while (--count); + } +} +#endif +#endif /* CONFIG_HAVE_128BIT_ACCESS */ + #ifndef PCI_IOBASE #define PCI_IOBASE ((void __iomem *)0) #endif @@ -595,6 +678,16 @@ static inline u64 ioread64(const volatile void __iomem *addr) #endif #endif /* CONFIG_64BIT */ +#ifdef CONFIG_HAVE_128BIT_ACCESS +#ifndef ioread128 +#define ioread128 ioread128 +static inline __uint128_t ioread128(const volatile void __iomem *addr) +{ + return reado(addr); +} +#endif +#endif /* CONFIG_HAVE_128BIT_ACCESS */ + #ifndef iowrite8 #define iowrite8 iowrite8 static inline void iowrite8(u8 value, volatile void __iomem *addr) @@ -629,6 +722,16 @@ static inline void iowrite64(u64 value, volatile void __iomem *addr) #endif #endif /* CONFIG_64BIT */ +#ifdef CONFIG_HAVE_128BIT_ACCESS +#ifndef iowrite128 +#define iowrite128 iowrite128 +static inline void iowrite128(__uint128_t value, volatile void __iomem *addr) +{ + writeo(value, addr); +} +#endif +#endif /* CONFIG_HAVE_128BIT_ACCESS */ + #ifndef ioread16be #define ioread16be ioread16be static inline u16 ioread16be(const volatile void __iomem *addr) @@ -655,6 +758,16 @@ static inline u64 ioread64be(const volatile void __iomem *addr) #endif #endif /* CONFIG_64BIT */ +#ifdef CONFIG_HAVE_128BIT_ACCESS +#ifndef ioread128be +#define ioread128be ioread128be +static inline __uint128_t ioread128be(const volatile void __iomem *addr) +{ + return swab128(reado(addr)); +} +#endif +#endif /* CONFIG_HAVE_128BIT_ACCESS */ + #ifndef iowrite16be #define iowrite16be iowrite16be static inline void iowrite16be(u16 value, void volatile __iomem *addr) @@ -681,6 +794,16 @@ static inline void iowrite64be(u64 value, volatile void __iomem *addr) #endif #endif /* CONFIG_64BIT */ +#ifdef CONFIG_HAVE_128BIT_ACCESS +#ifndef iowrite128be +#define iowrite128be iowrite128be +static inline void iowrite128be(__uint128_t value, volatile void __iomem *addr) +{ + writeo(swab128(value), addr); +} +#endif +#endif /* CONFIG_HAVE_128BIT_ACCESS */ + #ifndef ioread8_rep #define ioread8_rep ioread8_rep static inline void ioread8_rep(const volatile void __iomem *addr, void *buffer, @@ -719,6 +842,17 @@ static inline void ioread64_rep(const volatile void __iomem *addr, #endif #endif /* CONFIG_64BIT */ +#ifdef CONFIG_HAVE_128BIT_ACCESS +#ifndef ioread128_rep +#define ioread128_rep ioread128_rep +static inline void ioread128_rep(const volatile void __iomem *addr, + void *buffer, unsigned int count) +{ + readso(addr, buffer, count); +} +#endif +#endif /* CONFIG_HAVE_128BIT_ACCESS */ + #ifndef iowrite8_rep #define iowrite8_rep iowrite8_rep static inline void iowrite8_rep(volatile void __iomem *addr, @@ -760,6 +894,19 @@ static inline void iowrite64_rep(volatile void __iomem *addr, } #endif #endif /* CONFIG_64BIT */ + +#ifdef CONFIG_HAVE_128BIT_ACCESS +#ifndef iowrite128_rep +#define iowrite128_rep iowrite128_rep +static inline void iowrite128_rep(volatile void __iomem *addr, + const void *buffer, + unsigned int count) +{ + writeso(addr, buffer, count); +} +#endif +#endif /* CONFIG_HAVE_128BIT_ACCESS */ + #endif /* CONFIG_GENERIC_IOMAP */ #ifdef __KERNEL__