From patchwork Thu Apr 11 16:46:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13626421 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E142CC04FF6 for ; Thu, 11 Apr 2024 16:47:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=qXUajg3undbStCXsAshu2F4PfPghKomsWk1QFKr2hxM=; b=kkFYBEIiLQi45/ j0YuaouCiGUcEmIwYQV3HEcaAccFpZzAA6pfsGHHiFQmW/dQ89lz+/X95s2tZIN+bTyqVcphdHEvq 6EY2cRbGsV3vkzYrcjCTVy1vGaYtCuf1B6b0EvyE4YSabKEdW7wk724i1hnnvXz1RrGcteJ4I6Kv+ l6HKpY0bdV6y3B7/u0ZZcElbPG3oFLJ2dhDqEqyKds0Oj8bksvW4tjBh8fomKg3LDS/hv89bs3XGF TIZ05bJTwOI3mhWm5nnWxC+cxW//g+57OleIHx3AuNntp04Ka2ktQEWLgaGd+R6Pp9EtdaB27bBW2 3CmUM7Rgu13BHdCf/QIQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1ruxZh-0000000D9uD-0zyh; Thu, 11 Apr 2024 16:46:57 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1ruxZZ-0000000D9pd-2EGj for linux-arm-kernel@bombadil.infradead.org; Thu, 11 Apr 2024 16:46:49 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=MIME-Version:Content-Type: Content-Transfer-Encoding:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Sender:Reply-To:Content-ID:Content-Description; bh=3gPpaTPtg1xHqEwWLlT6hLXfYBWsyZWBrYOYzEf2EXU=; b=cteC0SId0k1Q8G4ZUpG9JLkeyk D06/xDOIzLDvA+Dzn7blsW1BbhYNjqjeQ4qKD5d0SpFA76gzKRSB7LIm/z+2tVD6osk/ld/loRmUl PxKlds3ddz9OeaoTg6W6eyDxYbL+PCe7qLjEWsJl4EDj2uLFA0YwnZbuBtCoLuw6C9nu+3eSoSrub 1oxsAx6Hkr4PUWdnoYECSDhi16YlKvF/oSJaR2bGDqV/7Xa5JjnucMjiGy7p/D5MEqwtEEWwoUlYH AFzuKX9U+q7DNPUwVaAN8zdALxeLnC51TLF6z9oY9ijYBLYB9d+qkV1LwsSABjsFSMpVq1JweDmDi rOL5p7/w==; Received: from mail-bn8nam12on20601.outbound.protection.outlook.com ([2a01:111:f403:2418::601] helo=NAM12-BN8-obe.outbound.protection.outlook.com) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1ruxZW-00000008iaj-209t for linux-arm-kernel@lists.infradead.org; Thu, 11 Apr 2024 16:46:48 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=DB9VDzG8p/ueiEuC7nw9ZIZ8lmrzeblEMsBJXtxYkCwNLeeb/WqOSbCaAoYAgIvDV4k6rsGeRZ2BGJBv0DcFZ85511PiYFicR04fswPJWWWZhZXKdQWr/yvl9AqFnwXFFMvljSsloldrIGN8vrNzlnrIi4fCFciE9u5TQQzpSrDVFcvuzoaZy8y2PUnTcLLBcDQXiWtQelA4evIR13yMfAyHyaNS2FV6vre6dkCfIn9/wLPCnuitIHA5r7ptOxvL9bymjzhjAk5VHonEJhZ65YRVdFtDngOdWIxaC91wYD5Jq1aCbzw9MzzxGbHPakBjRiao47VWxJ/rOCtXQnrVFA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=3gPpaTPtg1xHqEwWLlT6hLXfYBWsyZWBrYOYzEf2EXU=; b=ntbtklFo7HOxmuueCXtpCkPSeswGHpNShcjtB6eiT6+GkyXMBH4VSVx0C0k/wMQiNMyClgzBMUQrkxGQrBKT1CDEYJg1xI85/GdQAajPFLyP8fnRF9lbdcrJPj/1SgNyX4z8p85Cs3//s6T6vfuBWkynzif1kBp773FP/l4Yqw0Gfy+YMBKDWTORVIZZsV10yAXR+L4aQlm8uCdtamaPOwwzSmUMO9AfiPxv2HjRSdGWQ7GTr+DW8VBIrLFhJ29R7vD85awDESRkS4NOx8hcSFSybf2HwNTsbZDFgkU3qUscCd4qvP4I3Zf0RKPqF5J+3/FU+eXFeYpUBtRMir9sCw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3gPpaTPtg1xHqEwWLlT6hLXfYBWsyZWBrYOYzEf2EXU=; b=H/N5Z7rIrxMcLVyGToQM8H8QaxYxl/xYpS55Gbaj0ItEvcY1U2uWaJE1wJh0F/IT3+VHYdg2Wx0f3XNWuNJO3OSWj0UJBc0zIzAjby3cVA+j7gRJ0EncvKko5xA4dn6w3WELLeQiZvbHpt0W503wi1tf2UXXQdwjn8CBy4GX4ku6Rg8Lr90RecLVQ7XTEE/Ubi0QaZ/2ZUKky0wqNxrJrdlDHRq7xG4HCnHTjuxEybKJXDkorW0I9VRfm4WUN9aqsNXG0gfTxdqcudb+da+/4wlMkV5jTB0JfQo7hNZjJe6PGGJTnfTNNWjoQjXzCVfLS5Fx9IKWym010hNESqaDHw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by SJ2PR12MB8064.namprd12.prod.outlook.com (2603:10b6:a03:4cc::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.46; Thu, 11 Apr 2024 16:46:23 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.053; Thu, 11 Apr 2024 16:46:23 +0000 From: Jason Gunthorpe To: Alexander Gordeev , Andrew Morton , Christian Borntraeger , Borislav Petkov , Dave Hansen , "David S. Miller" , Eric Dumazet , Gerald Schaefer , Vasily Gorbik , Heiko Carstens , "H. Peter Anvin" , Justin Stitt , Jakub Kicinski , Leon Romanovsky , linux-rdma@vger.kernel.org, linux-s390@vger.kernel.org, llvm@lists.linux.dev, Ingo Molnar , Bill Wendling , Nathan Chancellor , Nick Desaulniers , netdev@vger.kernel.org, Paolo Abeni , Salil Mehta , Sven Schnelle , Thomas Gleixner , x86@kernel.org, Yisen Zhuang Cc: Arnd Bergmann , Catalin Marinas , Leon Romanovsky , linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Mark Rutland , Michael Guralnik , patches@lists.linux.dev, Niklas Schnelle , Jijie Shao , Will Deacon Subject: [PATCH v3 1/6] x86: Stop using weak symbols for __iowrite32_copy() Date: Thu, 11 Apr 2024 13:46:14 -0300 Message-ID: <1-v3-1893cd8b9369+1925-mlx5_arm_wc_jgg@nvidia.com> In-Reply-To: <0-v3-1893cd8b9369+1925-mlx5_arm_wc_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR07CA0026.namprd07.prod.outlook.com (2603:10b6:208:1a0::36) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|SJ2PR12MB8064:EE_ X-MS-Office365-Filtering-Correlation-Id: f32dd621-6a5c-425c-9cc4-08dc5a46eace X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: OU7S7Hl+VBNhs9Wa8WCFye8hSOeQ4H02MR1sEZ9H3/0TOKntdBY4VfIbNv+TFFus2t3Iwzu1Ci/sYHJMIl5BZCdq3VBLAUhjGEb0amxb4lP2/8LUscLJVrASIfeccbISmOZ7F14NpmYlPkvXM/tPT9noKnA/uJA6J7VnxY1IiFTwl5nEVVUN68Hb3n5GTtefv12NhUU3ywLtqJjHhnzPAVUdpGitdybIcGSyZ7cSTkz9A09FJ3DDTJv3nIh6aMs1VohosHayjYYt8PXrGjB8ec4JLuT4ApE4g+JLUgY3J/nYQYOiJqwzIiJUnygSWDPsO/lG6t8duH7K1NCzoREcLLMpGKxScD5c7c0n8sTJAfuUk8/X6TbdlWd0PWPr2emmjmbUnNLSd3WHKhViNN0gwyAcohXaF2BxoK2rHpL6h8hkwdVMJY5nA8I1oZw2PhGyw15gq0Bl7nLuKdKY0p1IY1kNOS8QCXsf7dsWcU49XqV8z8c22T5ulyf2WZeQJPPAC6SF3CR/rN5dmXX5rkFSPbaxFsozLZAj2Brbsro5aLd8hqJKxTdyQ9OAb8/L19SSahDVc7v5rLsDNnNKcL0szFvFUMFkZV1yjDhPhqTGEY1WuvbO+uEDYuvZAK9xGHVbi5NAEUBSIbXBJXJ//P15S+3svfPwIKBx/nauDRwW3Sbr0uCH/EbAUwgA62UgIV+ygEJd5x4fLcg0VikGD5TNsA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(1800799015)(376005)(7416005)(366007)(921011);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Txg6XFOW/9TquxGDKyNZXljB30YxpUzdYu5hAM7DraLSVT0ndSxqJsx1neLGyTUFPX00WOCTmFBOo2Rn3RiuS8SDuNiNVq1/sRP/8dK+HpAK1Ns+qtZW9LFsdyIEg3/hXQLl3btwP0WBeCANLRmSQVwYzBaqbGlZLOSb46LwGt8zldnUgVXYI+eNHZxaa7W0iYN2UKSaKslCeCE+339PKgQNRk5HmgfL0lrB+UpUzYO7r0pWMCrRXYWsATwRVASG6RduY2HketyBKk2dSOXZwcHFE37BkVTnwkgKAGLfiWdL6arZszUx9xutdPomNJHZC6A2WpgwdAc1lDwAWEKLX0yDyb750C/orz9d43eMKl9yo91Y1yc8k0qVXuTN3gZx/Vpjc1EOB3k2cB3LkExBWKLwBhLT+1FQmB0I2/cv2WfTQnoT+AhFEVKfVqPdNeQUJL4wrhVD03jf24wnpxkVHOteQ0kQvgo4jHJN7UT1f4stfcuOQ/REjlDEHDSH6TtHnoe9n1F1xgWW0dw2x48zHYpHG9pZDQYhLMnOlWF+gsRQTID9nH57oC0GzSZF0kjsKPAPBmbiqPE4Yh2PIKEmaWz9il4qqDnXvMwd2aluD+ORJ4IMxSq5BWGGpT3xLOUKLJoU5vrfD2iFP91fFo6fDj07Ed37lkKu7F5aPMUSxK0g63cwOY7fhPk1vVFEWHompo2sADbWWc7iHR0d4VCPmj8aPxT4CMLRzebcyy4ZZotXpiVSkb16KnuiMCU+E9Evm4UAwjxEBDhObIXcPqBlmLa1HsEqN0zmGNfmERmHrI1GrrS3ILDZh1cSmRLKp8Z4ZhClLjZ8KfSVJ3cfZngdHmnn+4JzIGZ5whEw4zs9GGXeFU3zQlKw+osoqmV37ZsLNxMli0+ULwETaHXK8ftWUU3iintFo8Fn8ccLj0D51YT2W3yokrVXG34EUiwEbQepkGDemuuJKXk9YEXcO5z0fSNJIT5WGmDKu2IQFyVvPxNLS0SLT3lBpOwYxAFKbdh6JPVQeEb8VWFHRv+Pvcdd0tVjx6gvlhpwE5xVn/AZceE1fwvS2c/MYRq1cX0dAlwtwPYsN29aRYBPYnvNM+MYJkPwMeEDBoBHbKJzFZOY69e2lK4ZQtRcFO4JNMG0cmLYb4AIyVb0nTAMsHCBhjuxG9p9xH6lAdYRDFZd54xk4H+quXsTOk0tBI3HVXArxygSL1kHymnNKp7hv8051x2i14YNCNrIwYBB4Diq2JA2Z0o7wVBjWS6vH7nL+ciaxWftwg4X1pDypIUwTZ+0y0OwdDI2u9T4f4bZ3TLW+oWAM4TPA3Z4NAs2VCeepQJUU5gqk/5RGEf98MrpSi0p7ZAz0hFYf1I8i1SpU/7/Gos5UU8TFEe4rLsTSs0GwYP+xmeCoFztefFj1qdv9Lhyv5rOPG8/BxeMoQ+/QgZrmdByFSh0sTTScRUfOzOs8ZhnO7ClF/2J6cz+A1J1tLFjD0VRkyH2E6dx3nQuZW6QXuki443HdYgSmsg/cYwErB7qi9z60s0NvxZawvyXr4UYPzCwY1A2rbJX0zV4pFPvWMpgTTPCa9zQXaWquZI17qUNFJGC X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: f32dd621-6a5c-425c-9cc4-08dc5a46eace X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Apr 2024 16:46:21.4794 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: kdCiKkIt5BwwTn8c2VYAXmiqGIPWwW6B1wLC+j+bGnFl+i34PF2akqQepPwAn50Y X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB8064 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240411_174646_679060_4F31A473 X-CRM114-Status: GOOD ( 16.81 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Start switching iomap_copy routines over to use #define and arch provided inline/macro functions instead of weak symbols. Inline functions allow more compiler optimization and this is often a driver hot path. x86 has the only weak implementation for __iowrite32_copy(), so replace it with a static inline containing the same single instruction inline assembly. The compiler will generate the "mov edx,ecx" in a more optimal way. Remove iomap_copy_64.S Signed-off-by: Jason Gunthorpe Acked-by: Arnd Bergmann --- arch/x86/include/asm/io.h | 17 +++++++++++++++++ arch/x86/lib/Makefile | 1 - arch/x86/lib/iomap_copy_64.S | 15 --------------- include/linux/io.h | 5 ++++- lib/iomap_copy.c | 6 +++--- 5 files changed, 24 insertions(+), 20 deletions(-) delete mode 100644 arch/x86/lib/iomap_copy_64.S diff --git a/arch/x86/include/asm/io.h b/arch/x86/include/asm/io.h index 294cd2a4081812..4b99ed326b1748 100644 --- a/arch/x86/include/asm/io.h +++ b/arch/x86/include/asm/io.h @@ -209,6 +209,23 @@ void memset_io(volatile void __iomem *, int, size_t); #define memcpy_toio memcpy_toio #define memset_io memset_io +#ifdef CONFIG_X86_64 +/* + * Commit 0f07496144c2 ("[PATCH] Add faster __iowrite32_copy routine for + * x86_64") says that circa 2006 rep movsl is noticeably faster than a copy + * loop. + */ +static inline void __iowrite32_copy(void __iomem *to, const void *from, + size_t count) +{ + asm volatile("rep ; movsl" + : "=&c"(count), "=&D"(to), "=&S"(from) + : "0"(count), "1"(to), "2"(from) + : "memory"); +} +#define __iowrite32_copy __iowrite32_copy +#endif + /* * ISA space is 'always mapped' on a typical x86 system, no need to * explicitly ioremap() it. The fact that the ISA IO space is mapped diff --git a/arch/x86/lib/Makefile b/arch/x86/lib/Makefile index 6da73513f02668..98583a9dbab337 100644 --- a/arch/x86/lib/Makefile +++ b/arch/x86/lib/Makefile @@ -53,7 +53,6 @@ ifneq ($(CONFIG_X86_CMPXCHG64),y) lib-y += atomic64_386_32.o endif else - obj-y += iomap_copy_64.o ifneq ($(CONFIG_GENERIC_CSUM),y) lib-y += csum-partial_64.o csum-copy_64.o csum-wrappers_64.o endif diff --git a/arch/x86/lib/iomap_copy_64.S b/arch/x86/lib/iomap_copy_64.S deleted file mode 100644 index 6ff2f56cb0f71a..00000000000000 --- a/arch/x86/lib/iomap_copy_64.S +++ /dev/null @@ -1,15 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-only */ -/* - * Copyright 2006 PathScale, Inc. All Rights Reserved. - */ - -#include - -/* - * override generic version in lib/iomap_copy.c - */ -SYM_FUNC_START(__iowrite32_copy) - movl %edx,%ecx - rep movsl - RET -SYM_FUNC_END(__iowrite32_copy) diff --git a/include/linux/io.h b/include/linux/io.h index 235ba7d80a8f0d..ce86120ce9d526 100644 --- a/include/linux/io.h +++ b/include/linux/io.h @@ -16,7 +16,10 @@ struct device; struct resource; -__visible void __iowrite32_copy(void __iomem *to, const void *from, size_t count); +#ifndef __iowrite32_copy +void __iowrite32_copy(void __iomem *to, const void *from, size_t count); +#endif + void __ioread32_copy(void *to, const void __iomem *from, size_t count); void __iowrite64_copy(void __iomem *to, const void *from, size_t count); diff --git a/lib/iomap_copy.c b/lib/iomap_copy.c index 5de7c04e05ef56..8ddcbb53507dfe 100644 --- a/lib/iomap_copy.c +++ b/lib/iomap_copy.c @@ -16,9 +16,8 @@ * time. Order of access is not guaranteed, nor is a memory barrier * performed afterwards. */ -void __attribute__((weak)) __iowrite32_copy(void __iomem *to, - const void *from, - size_t count) +#ifndef __iowrite32_copy +void __iowrite32_copy(void __iomem *to, const void *from, size_t count) { u32 __iomem *dst = to; const u32 *src = from; @@ -28,6 +27,7 @@ void __attribute__((weak)) __iowrite32_copy(void __iomem *to, __raw_writel(*src++, dst++); } EXPORT_SYMBOL_GPL(__iowrite32_copy); +#endif /** * __ioread32_copy - copy data from MMIO space, in 32-bit units From patchwork Thu Apr 11 16:46:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13626418 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AC0A2C4345F for ; Thu, 11 Apr 2024 16:47:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=wUXGYijAs24nHgVJacM/drOHCOc/6HjG4Px23VYn2XM=; b=k2gOzFrKOlqUbf 2Ms9LwOoguhHqfHJouNOPJACYyo2Oi0GwdibNwOotV2tKvJuvowl866H53EeX4CaUy2e/M0WEs+/a lMAOtSnMKkjylgFDagbJkY5XwaQ1kQRHVZ695Cpyfp3JCzOrvdSs8dKuJdkLSSWKaUEH2jNgjKzpN grqtcfWJdBwzgKpFG4R6K+NTyKU0Un/Yc9Jrev7kvBQ7/L/i/TAQMSBMPYCfDF8MbS15Gx/ZE7Vu0 rbSL3ugBWAn3V+CAgPCtYXDiThJFP6jB0i00B8j/zudZSZEZH5GhJSdWLvqDlkSQUI1/BnN8PQbk3 GtEjPLxpRtu4XlNXvPeQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1ruxZd-0000000D9rd-1acZ; Thu, 11 Apr 2024 16:46:53 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1ruxZT-0000000D9n0-2Cx4 for linux-arm-kernel@bombadil.infradead.org; Thu, 11 Apr 2024 16:46:43 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=MIME-Version:Content-Type: Content-Transfer-Encoding:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Sender:Reply-To:Content-ID:Content-Description; bh=PRPh4dhsyZVmxVVomsPWCtm81IAZr2FMm5Xc6ZHkaQc=; b=VVj01t3pijnqrtKOHIntpMjdlu GntHPhzUzcOKmLJuZwLWerXV5IrJkozsp0NECJcPt493V2t7BS5Qu9Bq3wcd5dzYomD5G1tFVVTT7 el+mVgPjoaDlasbPCmYnNSg+3P/Nfrxlhu2Q7RIkwCsxz4779SUfqmY6OJqXSOXrva8/6ED/RJ0a+ cu7H3m17x0T1AGnTmKhPyyr+8Mf7be73lXaRQx2lmFSkzgkDQwKvb5OaWLmi0KTUSL1NBdIP1OMNM vU0j9shSu3/S0B/voyLZigJzYSVjCjn1/SafhYM0HvN30p59I/kH/h8XQqjhJJGikXVUh6j4b7Ulb gNwDnTDw==; Received: from mail-bn8nam12on20601.outbound.protection.outlook.com ([2a01:111:f403:2418::601] helo=NAM12-BN8-obe.outbound.protection.outlook.com) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1ruxZP-00000008iaj-3RPR for linux-arm-kernel@lists.infradead.org; Thu, 11 Apr 2024 16:46:42 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=QVmFYM0hSCAgGZWHj7nU1V2dIk+WtG0QT2F2Xo5uJPacGomujQ/MlIp40a4WuL08T56OgYt32V3V3MHkn/dyygnKwEfgSMq+tyHo6KaL73ldu7lx573XMQvK0RQ43gPgIgPVz/ZweytuevYj/iyY8LiJOIwatNaqpfy7QN4f5LC88wH8CZ3WlccX0Cq3ZTYK3EjdiZBnswsywf/X8Sk1QZvipSkVE8MC/6VF8yXe+mwSfcSUylt77KWY/d6chJApYg/jF7WqgmkeHCR/2kPinaH5LWPDBDUx3RlGBs8KUxNBV6sLBw5j14Ti40l6dfcFZj30FpK108zlFu/UVZhMTg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PRPh4dhsyZVmxVVomsPWCtm81IAZr2FMm5Xc6ZHkaQc=; b=kUyTAXQSUoeDGPWYuvbJGsu0vHfg37vQrMtTFm+DXsaMQRUpQDY+iQblBwABmT3xjeKOC3xFaPwP92eE6YHrVN/DxJBKQ3bkE9/QYYRLSvDvuodRBXhxHf7fRJT940MTSz5tnv1lPw7aS7IBmpbXkinl5sxv3biH32N4S5v5pZS9Y9IwIbB05ggOJFCYGJ5ls31DtD8ceOVRRDd0zH7KdvXEYlXYxf1me6B/F6N1hpuFqc3AZqeRBNxLzMV69A6NLA66wGftna5pE8IwitQGUXNaU4LxXOV8xgflWp6C4OSX5xj3CkAJYEHqD+5CJmEaYIWwddMcewxJty8uaTxAxg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PRPh4dhsyZVmxVVomsPWCtm81IAZr2FMm5Xc6ZHkaQc=; b=S85+0jqmnugPP6n4ZY8lRDAtefp0lT07vp5JmNZq45jh56K0nlogCLHG8i4qAZLcyEUdIH2pc3wTHGJGecGFneoDfPPWHPi2N8fZXoaCsfDbW9IcUQ/pf5oAabiWQ0IXpp585hluKyAfJl+9mhlkmtvyqGAZlZLwtiGEAPf4DXTb/UlJXcuPC9Uig9WNuZOAYSiJFdKyjqmBu2bUr0KuSPobsYRuy1TzSs4iIZ6ypLZB4iOERQJixh6v617gUzEEP+w1j+5lfpuoq6WqKRgz4LN7MioRKW9gju2+M3gCqkTcJH0Ag5GQHH36Dn7k1C6qM4xfhblqlwqD9cSm5TpWCg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by SJ2PR12MB8064.namprd12.prod.outlook.com (2603:10b6:a03:4cc::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.46; Thu, 11 Apr 2024 16:46:22 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.053; Thu, 11 Apr 2024 16:46:22 +0000 From: Jason Gunthorpe To: Alexander Gordeev , Andrew Morton , Christian Borntraeger , Borislav Petkov , Dave Hansen , "David S. Miller" , Eric Dumazet , Gerald Schaefer , Vasily Gorbik , Heiko Carstens , "H. Peter Anvin" , Justin Stitt , Jakub Kicinski , Leon Romanovsky , linux-rdma@vger.kernel.org, linux-s390@vger.kernel.org, llvm@lists.linux.dev, Ingo Molnar , Bill Wendling , Nathan Chancellor , Nick Desaulniers , netdev@vger.kernel.org, Paolo Abeni , Salil Mehta , Sven Schnelle , Thomas Gleixner , x86@kernel.org, Yisen Zhuang Cc: Arnd Bergmann , Catalin Marinas , Leon Romanovsky , linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Mark Rutland , Michael Guralnik , patches@lists.linux.dev, Niklas Schnelle , Jijie Shao , Will Deacon Subject: [PATCH v3 2/6] s390: Implement __iowrite32_copy() Date: Thu, 11 Apr 2024 13:46:15 -0300 Message-ID: <2-v3-1893cd8b9369+1925-mlx5_arm_wc_jgg@nvidia.com> In-Reply-To: <0-v3-1893cd8b9369+1925-mlx5_arm_wc_jgg@nvidia.com> References: X-ClientProxiedBy: BL1PR13CA0264.namprd13.prod.outlook.com (2603:10b6:208:2ba::29) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|SJ2PR12MB8064:EE_ X-MS-Office365-Filtering-Correlation-Id: 276b625d-16cd-4755-020a-08dc5a46eaa5 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 0z+KsGAxioAC5gG7TV1cZ+Z9lvKbtsTo28Hf7uyMUlY2HskjDQKH7qKBUFklCXfgiBV0GyUc5ueU8dhPWq7X+/H9jHTAqg+ZO+80B4groWEFyp7e6hQbc84fefhNkfIKK+KGC8jjs8z1KLPRnC+xx9+P35/2Fd5/3QA74eaDls8IJLSy3kK0ohRRFiLBmdNcPMVgYLyMbBkr7zFjEeuJzUXO6DdrCwpifoeioO/IxaWTHHZZ2dBtLmaFU8wRQs+eqWsquFL8R1ZPpcuzQ3sgekW48ODKyxjvBhANWK/puny7BAOobMnbfFsPYCoFjTtXFo9dDRfyxrDpKaxpydLmOvbZHXy+gGF/IeWqQiV9nDlqacizA0Q6opdtdhpCwlatq3hYFvfRhTHiqTQa3CRKlOWupd33w/akUOkh87RtFzryfeOhZWUK4sKdiuhZZ/h0WHJH+vCJLC+uv8sYaQsjgZDblUivQo0knn6aQYp6rTjCCRkcCk7SYgvQB8OrUo1AuikKAfqG9MozM7ZZw3tnWKuSo7DC0bfnDbD27SvIvjmHJycU5Oko6/N4V246l/9fgh2Ron/L5yB9jD7ORMp/zsCpuqcpmzaZzJ96z552AlNJ7OQJse2Xr9/ZNncxOKLeTDIw7ORGIxLp+8W7KltM4TOTO+fqhjCFRblQAXYxj7qfyurZFOwBSoakAu/YqiT1xU/1OQWU5pZPE3JmAINtuA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(1800799015)(376005)(7416005)(366007)(921011);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: i7lXgCdif+kwxcoaKncQ+KlIN8n2nlbjbUAok+EWcO2iMQ2pnaCZQ2UfQ1RRjSwdhNfo/0pzuS0DsmEzPVUjGYMZBqB2EoiKxAv/c9GD5/BuqXjrUg8ZILas0S8cdPZ8N/Q7TCZg0i/eWn9RH7xMWvEDlXQTTE0Xu19Ly5EuXdDQglVaxgt2M+gbYmqMcIC1blUkC+qq4HsvGKLBHMNvzRSSO2zQQnXkZqhE/SXCVU4IWddCGYy7pvJoNvr6kfbcE2vkH1NE65Wc7ITGr/afjVLIv6DQa7x07uHs38F6PGXFiY/74aQN+6ngLE4is98hPOPzYmxePcWz2y3sFjRIVJhCxIZbv4RwU+YJvhnxs97HeYco07X1tId3qVGQ2jBHx5vFN0648MKo9EKFX6hmDeZHsRJWfRZiEL3IfzihHdTGhSAoTSqEYi3wND71PFk4WGLYo8gr1S/sWPMW/CTwR/GF1uVG+SFUJuuTHILx5tGM7HYCm9llJdE6023F1TwIZCDP1+3GiTzSO4R555vrqH81sxHmDcEbt2Vjh5NPX30SD/3/13+DYxbsTs7CwDFn9SkFoqEGQ+ZUiAwmjb9IFce6cBKQb7CqG8Cy223EDDb4eyY5NQyPLN2ve91VCxepAExIUMfMXh9gLiuLDS3Psvo7p1OlgSMl+Csy7Dc1qfVc8SVsKfx9jzUewKLvCQ70MFO8nXbosbFTFLLXv/TjQ4CTDiBtBb+NrA+Ob8qiP5mtLxMcXIiLsOrylgukEEDTMZrygEvjoEBsm1f9h8PU+WfxkuxqQNnooDTdbXaH7Yp9IKxYRpwVwoPdnrcYDZeKQAcFPo7++5jZ3VoH5VSkKd8dNOVfNrYxtkfkvnmbsSTSToSTNdx4U/ywgk7qp23iXIYSCO3arvDtU/oJxcKC6u0F8dZSA2S25+UgeGzzXvrUykwKvMMRSSDrf3mJeEZ8iBG7Pb3LK8XaCsOP5yeu0rRQf5fDno6aifBwX4jXAqk1e1/XoRNJGVOwm8erCnCkfaMJKLSLb165pJqV6uPRoL3iInInGRVOvAYhFxJI+BfCOM7itXJ46K2fVQmGgyJuCkVZmAOX6C6tW2B8i3oMSCa7qB9qHsnYWOrFDdoSlPdwtNp7bifPta3rPNGV8MgnGHciidR3vuG3SR3GGuPkLVJww5CVl4/iJwftppzkT7kFDv9Ux90/YdygJV1aatUm7qQVhRRFiSev3XdIjDgMLvVbZCg+kRwKPTFhL4sGdx+DNYEh1+1+lexuYqfLiIApJfT5oIGS1Xtm+shl2pCLzgbf85Hvd19LUofqaSXrIRGSN/Y/QxPe2c8tyIJd/rBYidVr7KVv8XZzKF3lseps4szrTm5xy1aeOtXkHNwiYlLF/RTemPzGnXvCt2BiefnnYs2Vniz2jCWU66Sq/Eke9roNoXVS3WJDNxxkt784rY7VisCpZpciNIWIPpqupjx/fNcSQjiK+Dj/yKT/En7N7lvQQUagRpWabOEnpcufAJp1nhb4ICHnE6SnMCMxj6UvXYD+Tiv8CfGRLk/QgmVDkXNNMKX9oLu4Zg+m2X1lTBif5ZRhlpgYPtAx6vN3ToPJ X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 276b625d-16cd-4755-020a-08dc5a46eaa5 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Apr 2024 16:46:21.1873 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: sDOJi1NGltfsyrIP6cojeXtmR+KV8RlY70GOwHhCZSJ+GUNVihQTNT/ZLw+OG12O X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB8064 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240411_174640_040252_BBB39744 X-CRM114-Status: UNSURE ( 9.52 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org It is trivial to implement an inline to do this, so provide it in the s390 headers. Like the 64 bit version it should just invoke zpci_memcpy_toio() with the correct size. Acked-by: Niklas Schnelle Signed-off-by: Jason Gunthorpe --- arch/s390/include/asm/io.h | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/arch/s390/include/asm/io.h b/arch/s390/include/asm/io.h index 4453ad7c11aced..00704fc8a54b30 100644 --- a/arch/s390/include/asm/io.h +++ b/arch/s390/include/asm/io.h @@ -73,6 +73,14 @@ static inline void ioport_unmap(void __iomem *p) #define __raw_writel zpci_write_u32 #define __raw_writeq zpci_write_u64 +/* combine single writes by using store-block insn */ +static inline void __iowrite32_copy(void __iomem *to, const void *from, + size_t count) +{ + zpci_memcpy_toio(to, from, count * 4); +} +#define __iowrite32_copy __iowrite32_copy + #endif /* CONFIG_PCI */ #include From patchwork Thu Apr 11 16:46:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13626422 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C85D3C4345F for ; Thu, 11 Apr 2024 16:47:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=RBkh5mcvFBlihx/+4EnEig5GrNZ9jrkMXBIeljKX/JE=; b=f+0FgVCSteNnCa Yqsq2WIsE2vRhH80Xr8HoHIrxGBmT68JzzoMEvWQEIgPr4U05nzYPgVcDWQ8vRRh3OVnzQrosrLJ8 nL5Yam+DRH/Q4e2AtcMU7wkKQuZAoFZ7OQJMJH5I5CxwbGMA8Ca5oImIWCIoQ8Uj9HF4C9OI5Q5JP SOC8UmH66nLHLs/pPunboTGHmzkZIbqlIDUO2H0rWbHXBBaJ2lXESwcsh82aZ6yGPTzXDZSrUn42p bZrrPQmlycCeVFPXhDF/fC7zuiK1ecfF/197KxsEwV7wTWct+8ziQ2n5NWJp4mRZVp69SvtzldqI/ UdgcLfG4IZ4ZhnOXfP5g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1ruxZi-0000000D9uu-0Dv3; Thu, 11 Apr 2024 16:46:58 +0000 Received: from mail-co1nam11on20600.outbound.protection.outlook.com ([2a01:111:f403:2416::600] helo=NAM11-CO1-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1ruxZZ-0000000D9hG-31Eg for linux-arm-kernel@lists.infradead.org; Thu, 11 Apr 2024 16:46:51 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=T+oQNLGQa/NbW6tJ37APPnOvjTVVe+iHEeWg04fNIXtWJJrQcR/NBHjqwjUWjjUP4fAJq9l5rZd1p5eRriuaQuLMRcH55gJupA80G2Pl/R09OtTutWpYwwwCdPNhKkXQ0VmjjFlymrDpQ2tzZK5WKIPvKMngKPl60vMjYfGUzkJ77OEteC+Rxd2KeV5HaylOGJgKCVUYpGozIqQxZbfliShA2Zi2jhCVmHZavaFu0sb5JBrbwI8KbPJM9MXiZ9XdhuJz6Kl7Cuz2HxAICHcT9KhtZCG256kQ+dxzCZ4i+vU+r40Vdw20iDe8Mbt1gCm9cy9g6RkYkcB4CF4z/WdBPw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=0NrN9Dwl70wQUv0Iu7xVFQtRrGzGbWKrTpmtmyKfgvc=; b=jxvlnCDd2ZKlgbO5AW9FKOHzatHVeEEcZYY5kuaFmSZ/xXrXV5CJFBr8RTVSQk9YB9GjZkSxPYMPEwYHB0KRd8EIQR74jw23NoEKiDMLciGoyPoPGIFMjzww1gt8B3f0EHvYC5CPsjTv/9Gk3o5nSQYn/oCqNtFu6fAqqGgFIQEsSFrSmvsrV/guTwktkAobCLcsLr/dVl5AcdAWMFafKT1Rhxm11tbjpH/rqL80i9iBJ2cA4vkD8wYB2M/MzRFPloztbUogipHKXsTaR1HrSq/cs3dllHwm5L+fzUetsbc/a3E3HlQWRZq70Wy72EgAvqqx7hDVM6g3BjUaY4fF+w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0NrN9Dwl70wQUv0Iu7xVFQtRrGzGbWKrTpmtmyKfgvc=; b=HVLy/bM132ITWpF3knAxICOesuDINLarM5PcDtjPmCJxNHbbSxjJNX2BOAQtjjGhh2J6yPIiNqPPaAuZqhcbxMKoOsvRBHmA9ErJEuQqaTFPeZA4nFzARtEB7JTLBjGcbUFRuC6GB5ODcffoaFbCHh3HD1WUsEv4vDNvtoYMESn537I/6xae7UPYGVxks4ULoDBbBsdf2P0jOH61hb9FQQfpmYI6zTqUL1AZIoYDdz3PiyzQgUBDUsz2XhWkZQGzzsCtpAxXHQoydhJMi9CBhHLBYiLWrU7IGHf27kcUh8q7rlkYGPYavHnFn9ejnjIjiNZ+8mBWHbwwwGfY5A9Zhw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by SJ2PR12MB8064.namprd12.prod.outlook.com (2603:10b6:a03:4cc::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.46; Thu, 11 Apr 2024 16:46:30 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.053; Thu, 11 Apr 2024 16:46:30 +0000 From: Jason Gunthorpe To: Alexander Gordeev , Andrew Morton , Christian Borntraeger , Borislav Petkov , Dave Hansen , "David S. Miller" , Eric Dumazet , Gerald Schaefer , Vasily Gorbik , Heiko Carstens , "H. Peter Anvin" , Justin Stitt , Jakub Kicinski , Leon Romanovsky , linux-rdma@vger.kernel.org, linux-s390@vger.kernel.org, llvm@lists.linux.dev, Ingo Molnar , Bill Wendling , Nathan Chancellor , Nick Desaulniers , netdev@vger.kernel.org, Paolo Abeni , Salil Mehta , Sven Schnelle , Thomas Gleixner , x86@kernel.org, Yisen Zhuang Cc: Arnd Bergmann , Catalin Marinas , Leon Romanovsky , linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Mark Rutland , Michael Guralnik , patches@lists.linux.dev, Niklas Schnelle , Jijie Shao , Will Deacon Subject: [PATCH v3 3/6] s390: Stop using weak symbols for __iowrite64_copy() Date: Thu, 11 Apr 2024 13:46:16 -0300 Message-ID: <3-v3-1893cd8b9369+1925-mlx5_arm_wc_jgg@nvidia.com> In-Reply-To: <0-v3-1893cd8b9369+1925-mlx5_arm_wc_jgg@nvidia.com> References: X-ClientProxiedBy: BL1PR13CA0249.namprd13.prod.outlook.com (2603:10b6:208:2ba::14) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|SJ2PR12MB8064:EE_ X-MS-Office365-Filtering-Correlation-Id: c7e18661-fd39-4a1a-89fe-08dc5a46eba3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: KlfCzdrYJw0UEuCsLzFWUViNz0Atr7Qx3yp8rN9mlnkVJImYi4CCpjFR0MCeWHvm2gEfZOlW2aqtNTqvQHb4zus5Q1yjpZDq2SOtVSsDZipAYkpoGRT3lOLN26oFi67SUt72RNwDkjQXbrqM2shg9FlvwD38TlIHFxhAQYKD1M9OF4ymgibhfmfybfB/6lnNSGuZ1oBmTZkK2gYT7bIkFzPPXqkzcU/8KUBn61A5n27cGbh943LRT1mfgzApOLEiNvIVx7ynTyHv5CmSxatjgVkKgZJQB0+Bgl/RA7PzqDNadFTiG12erE/xX9F3wBOt104C7OqztHLyqafAhminsw7SZTN3H1nCnOPyfyaU4yW1WGeMCf6XNWv1CC1dBlTP375PIDnHU7hzsuuqeKpDx6tYyGSnNW7OVmhgZhXOEEeDlrjljd9LS08jxnBw1mi1TlbWKgXo8pc3i3VRyhMSMwMD8hHmCMQk3jcFgjK8G+q/v8KW5ZBtHqudkQcEDYu0tR24AqcojR3WnyUHZ0LYYqJE0EXSnOhzk/0uaokOEr9nsBDeD4IO7sStUKw+5F3vYcnUBWjzh78/RmoxXernnYgT6esvzdx8q39XjFYFvQHN7+PMbvZwZjLsD5ip13OyWYi2yOy5pCvPTUZNkVsttfb/0BQkO4g+j4YNoFJRf5IM2PolzwAHs2+r+bJcvNMRtu+PvCW4AvsO7cuKkSpEOw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(1800799015)(376005)(7416005)(366007)(921011);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: uL8NMwQjLrc6AX25m86UiRvzHhp8hmcrYiEXeqBcC57cJTbfuFR+8T44OEeYCx7tdGV1d/HAYFkr+UZe+bU93JFBKHIBr519CY7slxbYCHkAvR0hkK1QVjSn535nxBr6btGwNg/H/SqpwUNvtJkTfMiGH9daIyxMo7Xtwxp/eswLhdeAsWkZHjc69Pxrzxk7ABWxrtSsthihjKUlBruAUxAubaJjaDqxLbI2rGqy75Uwsi0ZqoIGHltEuNGgJUOTJhqnBAyO60fycUVGQY8Fp+aLfy6sYPGB1t/7xPmn70hjDEnYsI4Zd7g2tfAuV4/V3e5j584Byyaw6VM7LOjoeOnsdF9h2+3PGpc5fSNKiIAaD+DnfowH/WCXvxZwbjBr+JMGUQZ+jnX+KxfgbxmtJERsEOPb0YYSvx8MKTTQPaDBgDVpxx7t2W3xTp5R0W1fRMeAQTMv/nNs1PpVGR6rvXKA5BCnlCn+KB0pXaPJ9VVFHqJeWP3/WBGHaX8Xy+AkKa4OGmgMFAdtC1Vzftbv7u4Oa8oWivEdWuGdZApaHGfg44DOqBVfOVhMkbCUVApbJ3xvjpbqDv3WBNAiiz7ZQo3AhK6Be/3MrJhNHLTO6MFkhklpZ1lg3PeECXNpQFV8qgt6E2VH6lrq1CAZaXPzH1IUKGHbMAYGVgKNaYaeZ2ppIK6NQMIESRkc7MnQv0eWh2BBrryNMdZL1bqG4FptOTwdFp9mvQgS2Df1JRZ1iIfgGzo4KL0NyiMmItM2eTHxfcwINjOK6oAf8Z2y7mxgmcfuYfhukwX4mDv1zANdfCumoiPL5/n2PPCKjUwGEiIy9LCE7+WKh4+yg3Q8JuTxJgi6XLd+J3Rek3OnJKoE4rOvxugjUTBpYWS32KGPFkDSBga+Rk3qlk/T/FdINcGEcL6fihRM0Np0g4BJEJbGD4sno3U69S8uXLLxoHhPsX6Zq5GgpdsGHf3h2wPxmACYO+R5ijvKE6q3GCR7vRVhE0/AYNhU+7SV8tzK86bqpaX7Hreg4wKlPeATHu702pxGdza+Clb1PyzmklsHMkKh1o9rruNH7wSHj8zi3QFuFd/HEU3/qiBkGohuy/Em2bnogLxAZ7CHPASZxNys6nVTPw8edwMuqQOkc+67qJigkd6Pk2xMh/gxToaTbus3bf7t5xStlzI8sPZNMhvqHOSAgVPmnxxvkXXefYlHiPFWgNDbLlohkhx4xo1KEpt4LhjQlwvgO/bz+a0zUJjQYqa2y2fVvxugjE9sfcrZwikMCzZ622lFaRVyXo2grtIHZE9xgKuOll72HuyPduil7jZ4uCK2jCsWSL+sFtRghtbmql8BReTkIdU0myZDdvt9ojG7YD8T57KqLx9n6WqeaQtVG5DDyF/kOJQ4WQKygvX/SY5+Vqo1lAYK8PeRQYK6P+FNxlpt9csna+4zJYLiraO30PCBcB/myB2SU9GMdWmuM1qA1qrDKKzuc6JuIN6g+OCXvuj6rnMec6d9y0TLXWVY/GdYYTAcC78ZQSLWhZ4osj6GHTmv5oEHilRdq5NIszXLQ32kl0ALntWy3f0ogeFACQGhkGQXrlhvtM0QFrxuIjd2 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: c7e18661-fd39-4a1a-89fe-08dc5a46eba3 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Apr 2024 16:46:22.8471 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: FMIENZXDu35ZSoupm9JlXaDrxm9gEzo5m+94C+//A7yeYNvlId6XS51RoRP92YfA X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB8064 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240411_094650_091528_1C9F863E X-CRM114-Status: GOOD ( 11.61 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Complete switching the __iowriteXX_copy() routines over to use #define and arch provided inline/macro functions instead of weak symbols. S390 has an implementation that simply calls another memcpy function. Inline this so the callers don't have to do two jumps. Acked-by: Niklas Schnelle Signed-off-by: Jason Gunthorpe Acked-by: Arnd Bergmann --- arch/s390/include/asm/io.h | 7 +++++++ arch/s390/pci/pci.c | 6 ------ include/linux/io.h | 3 +++ lib/iomap_copy.c | 7 +++---- 4 files changed, 13 insertions(+), 10 deletions(-) diff --git a/arch/s390/include/asm/io.h b/arch/s390/include/asm/io.h index 00704fc8a54b30..0fbc992d7a5ea7 100644 --- a/arch/s390/include/asm/io.h +++ b/arch/s390/include/asm/io.h @@ -81,6 +81,13 @@ static inline void __iowrite32_copy(void __iomem *to, const void *from, } #define __iowrite32_copy __iowrite32_copy +static inline void __iowrite64_copy(void __iomem *to, const void *from, + size_t count) +{ + zpci_memcpy_toio(to, from, count * 8); +} +#define __iowrite64_copy __iowrite64_copy + #endif /* CONFIG_PCI */ #include diff --git a/arch/s390/pci/pci.c b/arch/s390/pci/pci.c index 26afde0d1ed34c..0de0f6e405b51e 100644 --- a/arch/s390/pci/pci.c +++ b/arch/s390/pci/pci.c @@ -250,12 +250,6 @@ resource_size_t pcibios_align_resource(void *data, const struct resource *res, return 0; } -/* combine single writes by using store-block insn */ -void __iowrite64_copy(void __iomem *to, const void *from, size_t count) -{ - zpci_memcpy_toio(to, from, count * 8); -} - void __iomem *ioremap_prot(phys_addr_t phys_addr, size_t size, unsigned long prot) { diff --git a/include/linux/io.h b/include/linux/io.h index ce86120ce9d526..42e132808f0035 100644 --- a/include/linux/io.h +++ b/include/linux/io.h @@ -21,7 +21,10 @@ void __iowrite32_copy(void __iomem *to, const void *from, size_t count); #endif void __ioread32_copy(void *to, const void __iomem *from, size_t count); + +#ifndef __iowrite64_copy void __iowrite64_copy(void __iomem *to, const void *from, size_t count); +#endif #ifdef CONFIG_MMU int ioremap_page_range(unsigned long addr, unsigned long end, diff --git a/lib/iomap_copy.c b/lib/iomap_copy.c index 8ddcbb53507dfe..2fd5712fb7c02b 100644 --- a/lib/iomap_copy.c +++ b/lib/iomap_copy.c @@ -60,9 +60,8 @@ EXPORT_SYMBOL_GPL(__ioread32_copy); * time. Order of access is not guaranteed, nor is a memory barrier * performed afterwards. */ -void __attribute__((weak)) __iowrite64_copy(void __iomem *to, - const void *from, - size_t count) +#ifndef __iowrite64_copy +void __iowrite64_copy(void __iomem *to, const void *from, size_t count) { #ifdef CONFIG_64BIT u64 __iomem *dst = to; @@ -75,5 +74,5 @@ void __attribute__((weak)) __iowrite64_copy(void __iomem *to, __iowrite32_copy(to, from, count * 2); #endif } - EXPORT_SYMBOL_GPL(__iowrite64_copy); +#endif From patchwork Thu Apr 11 16:46:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13626419 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9447FC4345F for ; Thu, 11 Apr 2024 16:47:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=yyJujdwwjaDEB3f3rhjV8/HMPaDHXgghLPpAk5UJsmQ=; b=vouvTiCUWaz13T NkCcG9Xu8071tm3qxM8IdD1g1etTqbpFwz+fFZbKDAz+gNqfw0W41bbLlKV62+AJ1hcULjLMBOpTx BttC7rSiFrr/Ox3pxwQVfclamtFWt4aQX7fPb25BlZ2PVLnqBB5JUZbO93yUMsVlTEpvUuVOsfmD+ jo7FLr+hmIyaT8tJ7wQtgAmEXHY8NDtGrUDsOElRyvAMCaNej6dXly4TGK/R7IoTjK2DsSNXcLBhU RDmxdBINkljpvqHMVtFHQHfXSoqhXm6xwy6CKz9ugC3J41RB7y+Owenly6yyhh2ppT7UnJdAWCaqV OJEwTeD2BfYFY4aHPtyg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1ruxZe-0000000D9sQ-2ToC; Thu, 11 Apr 2024 16:46:54 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1ruxZX-0000000D9oh-1Jgx for linux-arm-kernel@bombadil.infradead.org; Thu, 11 Apr 2024 16:46:47 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=MIME-Version:Content-Type: Content-Transfer-Encoding:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Sender:Reply-To:Content-ID:Content-Description; bh=QdkM2UeXpLcuC+zg1KbbyqRpkGfk4PfuMlT6xAklK9E=; b=OXy9VXysnmBEGCIKJrFAQVUVPV vsJq9uwvJwvu4mr+tAbZL9Yiyuio6xcFgQCVAF/0tu8guSJAP9GniVHgsrw4wd6q0cUzKvtTiJoV/ Ay24HUWvk60iaWr1i+zLrZib048hItu5ipdqTaNcrS0N3GAnTGhKDZ/LJbsBRdIryReOjgjLqGigB FXAQc979RGJ+qz3u3BHgHVOmEcfkk+iGy0y90590AADNjnY4wA89zUFEtSxCCvxKF9MYxSR5/Uui1 GBJXDwNFZ1yrQZjbIKyGd8AaYcbEh8918Z+wVmsgy1Kg42Nqn0eCBecnMKs4fLos47jWrBc9s2uIZ +rui/aSw==; Received: from mail-bn8nam12on20601.outbound.protection.outlook.com ([2a01:111:f403:2418::601] helo=NAM12-BN8-obe.outbound.protection.outlook.com) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1ruxZS-00000008iaj-3BND for linux-arm-kernel@lists.infradead.org; Thu, 11 Apr 2024 16:46:45 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TJEMRd1Tx0CtllvCZUIu6q7Fl4LV4yJCj7nzJuH8p+EzuH/jMriR/JT3rq0zAx7OnOS0ZXGU3lgYQKYQkzvYv3L5Je5n3e7uUGbXn4tp12fG55ypLj736xhjjau9FCsSyQxSlseVvw+7HtGy3yqaYeOWzskauPr1bLVjvRqTzN4wJ4tpaDgcZYR4LgrpwOMwCCxOORXqxaZZPx+e+Y0ibJ9GOXWFx/GCekE7phMB8kMg0xz/B/oHJSwVbTBGV8wfpfSUB8EZ5v6jcJdP1hwd4SwYF4AM8dqY8DbHxIAv2opR76GOd8Kk1LQaNqal//R9//8uMLn4H+ZE4EUv5zJHxw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QdkM2UeXpLcuC+zg1KbbyqRpkGfk4PfuMlT6xAklK9E=; b=Y1kL+svoKTmo08Ir8vy2mQ6rUWm1Q4n7WYM7c4oEfGz1X8vfXGrN6NclA9fhlPu47KMbVYsv34Hv9hOdxHO7tL6AkG1SoHTRVuNTyb1K0f+kzEixUVUfemgVIxkulMwwKCHkMD3CLgrGGCDxYKa0bXOWi7tjP0zWUxH5LZXFMdUnC8VQRux1Jguc3CC1S+qkDnUQRehFwUAbbRQ4Rhh0VEysYYmZBgxYfkhLrU2GcXOLVq31NNie6IokR1LX05rZiJKR1myFFPoGRjznyANKqffzfj1fhBPHbDS0GAz8CczJtSY9fXQIAwMlK8hLlfR8Fm/dEFLo415XcLQbS6B8jw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QdkM2UeXpLcuC+zg1KbbyqRpkGfk4PfuMlT6xAklK9E=; b=d5b3yzMwZp2PfMPIDS5x7ziKGCf4mR8hJ+OGbldd51xr2V2qPfTgm7T2ACF6oCQjsEmvW8i2/Ic6AlcmWyzf8qpIIohGoovG7SiiffvBPzlLWy9aLI8+5SiYoIcUiK/1Xd6VFJwp64PPaiRjIPrIb9flM/qmuRB0X3antaRhKjpCppBhZLx0gMBQChzm0b7FgDD21cMxrmFrupigbqrY7SlWFkOfHwTQYAg0yQa2X3cUB7QK+20GrndVdvQlRBuk9j8IogxYrHwW0HUravuz9t7Jgba3inzsBdfwVNBtcr2ep3t86gs1nH4DRikmjid4oMbqdyltgAGnugGynPd5tA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by SJ2PR12MB8064.namprd12.prod.outlook.com (2603:10b6:a03:4cc::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.46; Thu, 11 Apr 2024 16:46:21 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.053; Thu, 11 Apr 2024 16:46:21 +0000 From: Jason Gunthorpe To: Alexander Gordeev , Andrew Morton , Christian Borntraeger , Borislav Petkov , Dave Hansen , "David S. Miller" , Eric Dumazet , Gerald Schaefer , Vasily Gorbik , Heiko Carstens , "H. Peter Anvin" , Justin Stitt , Jakub Kicinski , Leon Romanovsky , linux-rdma@vger.kernel.org, linux-s390@vger.kernel.org, llvm@lists.linux.dev, Ingo Molnar , Bill Wendling , Nathan Chancellor , Nick Desaulniers , netdev@vger.kernel.org, Paolo Abeni , Salil Mehta , Sven Schnelle , Thomas Gleixner , x86@kernel.org, Yisen Zhuang Cc: Arnd Bergmann , Catalin Marinas , Leon Romanovsky , linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Mark Rutland , Michael Guralnik , patches@lists.linux.dev, Niklas Schnelle , Jijie Shao , Will Deacon Subject: [PATCH v3 4/6] arm64/io: Provide a WC friendly __iowriteXX_copy() Date: Thu, 11 Apr 2024 13:46:17 -0300 Message-ID: <4-v3-1893cd8b9369+1925-mlx5_arm_wc_jgg@nvidia.com> In-Reply-To: <0-v3-1893cd8b9369+1925-mlx5_arm_wc_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR07CA0027.namprd07.prod.outlook.com (2603:10b6:208:1a0::37) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|SJ2PR12MB8064:EE_ X-MS-Office365-Filtering-Correlation-Id: c64ce94c-f38d-4ded-fb14-08dc5a46ea88 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: FrMUx/egIULNisnlgtTRQjHeUD1fYlXWH3Fr+pttxLHUwtbUq0OXKk/91UfBuXlJjr/mUtrkpdZtgcpxBEsHievATJhHWv/YyU+b0PIkqBtx3B0ACFWsNOlyVZ2rW1ysIUfg8eHuRdAozlUtyzOb4J3bpFDThnsmf3s1AKsnDrzivNTeJHKbjl72pXAW3LxLpYY6ZjAWpXI5LL65bs0FgCHU/INZj82XQAjiQ9qR8nhUCnrL28KLHNye7j9lQDM1Fia1gj2C3LWDxbUF/gsvoykek8ay2OI0HaMCXA4x34SG5ptWQ4ukdrIx/T293QlI7Tt4lMsvnP3HZ0kHaWp5MiyDv/QkeIAGI0SIaYaN9qeVYAZuVmqoyS80/WmgJxE0OtClX4el1KuBUVXazfNidHJvHBMbGTcm4gmo/pDlmkwk6YgnjbwAm7pxoKzDkf/UU+nyu+0g5X6cRQhu7t7/mfIMU2sTuC8OydamA8JlTt4tw0swapwNi/uAetrmHQr+O1V/NhhGNwox5MiCYKK+CwiDBH6+B7ABb5tcBVyvVUzZuNP+iBda15lnWQ93uyMSRmWKPUJ4C3WUXleJZ7kr0dJE+Us3F2lOncQkp0ULAJiiGbzwn7WDfuJWOBAlxWkX0sLggsN4UoPmEsGYrKYWaWQiFup+fKjX+aNA2qIOG3Lt4dvt34Yx5Vu/BDLmQlaCMsEz3+hmD1+eMSV+ljSHaA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(1800799015)(376005)(7416005)(366007)(921011);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Krto3W5Rf/xWe+ysl1hDaR34+oixAHavmGDs5wLMd0ZMa4niOyJURnoAqahjmJMSHarihLYcUwKaULISaCgS/iNTIoAhzAgNn6weeEeLX0ZizeibithMpXHXDiwNykXWYeuCg1PzTPIZgdoD7xZ/Qamm5crxTE24FOlg8XOtL4tQoxFzIAYGe9PT8JbmXeReVm0mRbtbTpUKEiLcMZ7CMP5/ffjf0JwsV/pXyOeBt74fP1EJ8jzWPNmafggVRfXh8qc82jxl/okACa49tmFZsiir0ToQzTOAEcGdk02apxvuwpdSQvNvKE46wNxwYb5rijbAtI/B8RvHCVAAcWKQtHBZ7rMhR4eGzXxUe3Reb0Paoy635dX4zRtIAWk5rGDoL9kP7Da/1LGqAMAbjl3OORNxR0ErIj3XDuWPE8VsGc6NgC9ZO459JtEOOlDb4Hlp7rGU/pnP+VVUDJXUh1Oz+Zvwq2g6J5gQtf9toQvTKQha0Jn2ixoUbNFbYsYtOsRR2OXrhKo6n+jURNHR1csXpqjVQVXSMqPtUQ++BY/i713oZasIom6ORZiVuJunR5y20gEukUwIdxbW7eDJa/voSZg77NUjW1GAL5Hr0tMWkl1LeEY8JFrVv3gN3inqrSik2Pd8Onz1rUoPB/p8yHfb1R0fJAvNJIpXgoSdMNB+uYzHujAiZEix39ZWrmIDRk4uE7f+7Ptr9/VELcvCVDsYHX+4YCIuydBiVDUl5upv2HsxyaYEd3gBETxaaaNQn6y5B8sVP5GlgKfQ8V+exsCkrvcKqp9P79tK40fawz2yT4OX/w1gOdU89GKyxTK2HHiQ3oyL//rKtziMirnGgtxq+1k2vX6WZU3xzKA6edVrVjkgEnQrJNPS2BBFB1dhEeP0lpoglxVY0PpAIZ9MDMxgQAwaUVRdNp6YlhK1cJLede5e0RGtZEP9zc/7XPZ/igsVzQuhK54CfP5YdA5uiDXsv9qiY8nOdxlUDzzSCrZ2Z9uZCnKuRpCbZ6z/8t6TG1IKCM45TAMKq7Qvz0ul6bA1G+A5V67wyO0Tw2WSUIGqZCjr8qvhxEn0b69sSp2lq/QOjRWWRgjsNeg2rpNbtl8O/CJKdeVkTBJKvQSIVYunfRgYcGUTYxscejD2XundUAaAiFnY31XHweJs6yd1kKc+MqJqcHKAfdKJw8MuOZlC9Yh3I8GzaIaY6LhTTZBbv2O1jh4KSNs4+1UYcb9LMFG+MfodUAT4CRUe72OBTEUN9tlIyeQEmWSXf263yy8Az4tDdAAacwHcPIZkgFRZsBX0kj8tCVDm5ybjToIyql3iiBsgfZYeZFGupAmR6WyaS1O0MaJJUD85nrVKsleQKxjcuUVbS1ceUGcbTQD0SL7DF7nAHN+wdlw6oBOu0VuUcP/jKiJPXgYnfkFF1Mu0pY+hbEPDMoZFeQ8Yq77FtjsSWPxbPhSMxqxNCQl3z5hMLTr5EIyZ/uHXOHS681mct4UnhlTsLukpLaAtYVbhVJ0Hoj8FGkXxGiaWbcnPHrzjDIEDSh3h2sBeWaCettXLRM9eDLKFaDJPipo1XqeX9n706nZ+EolJux7tSvBNyNQND+vp X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: c64ce94c-f38d-4ded-fb14-08dc5a46ea88 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Apr 2024 16:46:21.0969 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: gi1ybZbh//lOL0sZX9HUsV/9B1EQ+Exz9d3YRFh0s+znvjJKJCAlW/IzA/7aT8wJ X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB8064 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240411_174643_041047_A3772FEC X-CRM114-Status: GOOD ( 17.84 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The kernel provides driver support for using write combining IO memory through the __iowriteXX_copy() API which is commonly used as an optional optimization to generate 16/32/64 byte MemWr TLPs in a PCIe environment. iomap_copy.c provides a generic implementation as a simple 4/8 byte at a time copy loop that has worked well with past ARM64 CPUs, giving a high frequency of large TLPs being successfully formed. However modern ARM64 CPUs are quite sensitive to how the write combining CPU HW is operated and a compiler generated loop with intermixed load/store is not sufficient to frequently generate a large TLP. The CPUs would like to see the entire TLP generated by consecutive store instructions from registers. Compilers like gcc tend to intermix loads and stores and have poor code generation, in part, due to the ARM64 situation that writeq() does not codegen anything other than "[xN]". However even with that resolved compilers like clang still do not have good code generation. This means on modern ARM64 CPUs the rate at which __iowriteXX_copy() successfully generates large TLPs is very small (less than 1 in 10,000) tries), to the point that the use of WC is pointless. Implement __iowrite32/64_copy() specifically for ARM64 and use inline assembly to build consecutive blocks of STR instructions. Provide direct support for 64/32/16 large TLP generation in this manner. Optimize for common constant lengths so that the compiler can directly inline the store blocks. This brings the frequency of large TLP generation up to a high level that is comparable with older CPU generations. As the __iowriteXX_copy() family of APIs is intended for use with WC incorporate the DGH hint directly into the function. Cc: Arnd Bergmann Cc: Catalin Marinas Cc: Will Deacon Cc: Mark Rutland Cc: linux-arch@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org Reviewed-by: Catalin Marinas Signed-off-by: Jason Gunthorpe --- arch/arm64/include/asm/io.h | 132 ++++++++++++++++++++++++++++++++++++ arch/arm64/kernel/io.c | 42 ++++++++++++ 2 files changed, 174 insertions(+) diff --git a/arch/arm64/include/asm/io.h b/arch/arm64/include/asm/io.h index 8d825522c55c84..4ff0ae3f6d6690 100644 --- a/arch/arm64/include/asm/io.h +++ b/arch/arm64/include/asm/io.h @@ -139,6 +139,138 @@ extern void __memset_io(volatile void __iomem *, int, size_t); #define memcpy_fromio(a,c,l) __memcpy_fromio((a),(c),(l)) #define memcpy_toio(c,a,l) __memcpy_toio((c),(a),(l)) +/* + * The ARM64 iowrite implementation is intended to support drivers that want to + * use write combining. For instance PCI drivers using write combining with a 64 + * byte __iowrite64_copy() expect to get a 64 byte MemWr TLP on the PCIe bus. + * + * Newer ARM core have sensitive write combining buffers, it is important that + * the stores be contiguous blocks of store instructions. Normal memcpy + * approaches have a very low chance to generate write combining. + * + * Since this is the only API on ARM64 that should be used with write combining + * it also integrates the DGH hint which is supposed to lower the latency to + * emit the large TLP from the CPU. + */ + +static inline void __const_memcpy_toio_aligned32(volatile u32 __iomem *to, + const u32 *from, size_t count) +{ + switch (count) { + case 8: + asm volatile("str %w0, [%8, #4 * 0]\n" + "str %w1, [%8, #4 * 1]\n" + "str %w2, [%8, #4 * 2]\n" + "str %w3, [%8, #4 * 3]\n" + "str %w4, [%8, #4 * 4]\n" + "str %w5, [%8, #4 * 5]\n" + "str %w6, [%8, #4 * 6]\n" + "str %w7, [%8, #4 * 7]\n" + : + : "rZ"(from[0]), "rZ"(from[1]), "rZ"(from[2]), + "rZ"(from[3]), "rZ"(from[4]), "rZ"(from[5]), + "rZ"(from[6]), "rZ"(from[7]), "r"(to)); + break; + case 4: + asm volatile("str %w0, [%4, #4 * 0]\n" + "str %w1, [%4, #4 * 1]\n" + "str %w2, [%4, #4 * 2]\n" + "str %w3, [%4, #4 * 3]\n" + : + : "rZ"(from[0]), "rZ"(from[1]), "rZ"(from[2]), + "rZ"(from[3]), "r"(to)); + break; + case 2: + asm volatile("str %w0, [%2, #4 * 0]\n" + "str %w1, [%2, #4 * 1]\n" + : + : "rZ"(from[0]), "rZ"(from[1]), "r"(to)); + break; + case 1: + __raw_writel(*from, to); + break; + default: + BUILD_BUG(); + } +} + +void __iowrite32_copy_full(void __iomem *to, const void *from, size_t count); + +static inline void __const_iowrite32_copy(void __iomem *to, const void *from, + size_t count) +{ + if (count == 8 || count == 4 || count == 2 || count == 1) { + __const_memcpy_toio_aligned32(to, from, count); + dgh(); + } else { + __iowrite32_copy_full(to, from, count); + } +} + +#define __iowrite32_copy(to, from, count) \ + (__builtin_constant_p(count) ? \ + __const_iowrite32_copy(to, from, count) : \ + __iowrite32_copy_full(to, from, count)) + +static inline void __const_memcpy_toio_aligned64(volatile u64 __iomem *to, + const u64 *from, size_t count) +{ + switch (count) { + case 8: + asm volatile("str %x0, [%8, #8 * 0]\n" + "str %x1, [%8, #8 * 1]\n" + "str %x2, [%8, #8 * 2]\n" + "str %x3, [%8, #8 * 3]\n" + "str %x4, [%8, #8 * 4]\n" + "str %x5, [%8, #8 * 5]\n" + "str %x6, [%8, #8 * 6]\n" + "str %x7, [%8, #8 * 7]\n" + : + : "rZ"(from[0]), "rZ"(from[1]), "rZ"(from[2]), + "rZ"(from[3]), "rZ"(from[4]), "rZ"(from[5]), + "rZ"(from[6]), "rZ"(from[7]), "r"(to)); + break; + case 4: + asm volatile("str %x0, [%4, #8 * 0]\n" + "str %x1, [%4, #8 * 1]\n" + "str %x2, [%4, #8 * 2]\n" + "str %x3, [%4, #8 * 3]\n" + : + : "rZ"(from[0]), "rZ"(from[1]), "rZ"(from[2]), + "rZ"(from[3]), "r"(to)); + break; + case 2: + asm volatile("str %x0, [%2, #8 * 0]\n" + "str %x1, [%2, #8 * 1]\n" + : + : "rZ"(from[0]), "rZ"(from[1]), "r"(to)); + break; + case 1: + __raw_writeq(*from, to); + break; + default: + BUILD_BUG(); + } +} + +void __iowrite64_copy_full(void __iomem *to, const void *from, size_t count); + +static inline void __const_iowrite64_copy(void __iomem *to, const void *from, + size_t count) +{ + if (count == 8 || count == 4 || count == 2 || count == 1) { + __const_memcpy_toio_aligned64(to, from, count); + dgh(); + } else { + __iowrite64_copy_full(to, from, count); + } +} + +#define __iowrite64_copy(to, from, count) \ + (__builtin_constant_p(count) ? \ + __const_iowrite64_copy(to, from, count) : \ + __iowrite64_copy_full(to, from, count)) + /* * I/O memory mapping functions. */ diff --git a/arch/arm64/kernel/io.c b/arch/arm64/kernel/io.c index aa7a4ec6a3ae6f..ef48089fbfe1a4 100644 --- a/arch/arm64/kernel/io.c +++ b/arch/arm64/kernel/io.c @@ -37,6 +37,48 @@ void __memcpy_fromio(void *to, const volatile void __iomem *from, size_t count) } EXPORT_SYMBOL(__memcpy_fromio); +/* + * This generates a memcpy that works on a from/to address which is aligned to + * bits. Count is in terms of the number of bits sized quantities to copy. It + * optimizes to use the STR groupings when possible so that it is WC friendly. + */ +#define memcpy_toio_aligned(to, from, count, bits) \ + ({ \ + volatile u##bits __iomem *_to = to; \ + const u##bits *_from = from; \ + size_t _count = count; \ + const u##bits *_end_from = _from + ALIGN_DOWN(_count, 8); \ + \ + for (; _from < _end_from; _from += 8, _to += 8) \ + __const_memcpy_toio_aligned##bits(_to, _from, 8); \ + if ((_count % 8) >= 4) { \ + __const_memcpy_toio_aligned##bits(_to, _from, 4); \ + _from += 4; \ + _to += 4; \ + } \ + if ((_count % 4) >= 2) { \ + __const_memcpy_toio_aligned##bits(_to, _from, 2); \ + _from += 2; \ + _to += 2; \ + } \ + if (_count % 2) \ + __const_memcpy_toio_aligned##bits(_to, _from, 1); \ + }) + +void __iowrite64_copy_full(void __iomem *to, const void *from, size_t count) +{ + memcpy_toio_aligned(to, from, count, 64); + dgh(); +} +EXPORT_SYMBOL(__iowrite64_copy_full); + +void __iowrite32_copy_full(void __iomem *to, const void *from, size_t count) +{ + memcpy_toio_aligned(to, from, count, 32); + dgh(); +} +EXPORT_SYMBOL(__iowrite32_copy_full); + /* * Copy data from "real" memory space to IO memory space. */ From patchwork Thu Apr 11 16:46:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13626417 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1B853C4345F for ; Thu, 11 Apr 2024 16:46:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=+1JG82+LTIVljqD+YtYqauViq+Yw+pkIxyb8ZwcYD4w=; b=4DyHZNTxCEWlC7 2pKEc7dZWkNeDUGuRTi5IqLhsOj/Xh1wWkbyRSDc9O+P44ho5tRx8GyveRVbu2wV/kp2NWMIDSdM6 Xv9RpVC7bsQKCOKOTjj1KY+jml8PUPeHZteQNimbUQrOSJxPzDOVZf0S9zgspcic9Yb5EEORIHH1A DA0m3pUAhC9FIq4oeGpgdYOfRt6ZfOz/2o4KveK/9i3D/OuxH4ZLOZID7x7dVkHjber9dEQtmTrtZ b7ZnURL53GdHQa5MbVge76VOvQFAE5JY0aspjCodScbBh3Y/N/Ziib+Qax+mc1/vnNLtzKzrnNEsD 2OhS16NRkIBf/aFVNo1Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1ruxZV-0000000D9o6-2LQH; Thu, 11 Apr 2024 16:46:45 +0000 Received: from mail-co1nam11on20600.outbound.protection.outlook.com ([2a01:111:f403:2416::600] helo=NAM11-CO1-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1ruxZP-0000000D9hG-1DTu for linux-arm-kernel@lists.infradead.org; Thu, 11 Apr 2024 16:46:42 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WfetvjP0bRsA7pApjR90PEzjVf5iqzjj4EwUMwx1GS0WYSxbAQIcMzvy1lG3hSBkCKsASjNCVfDwg4Hazwgvxlp7vzkemALmMOJILISl6F/czFDSqk6es0MGP2ohDSzVlVz7ELm0xJY++fz8wpYtLlQtOiB4/aPuRHa9kOkZTN8X/Z1zbkU3I/K2Vh5+MmUZM2r/MDXzdTRK49WHtHSyAvYb4nfv4NLgaJMegPbASkVqZ6QPVp4Ske7iPCQC+zS+mIk2LIeu1KARWBlLouQit0HVQysaWRjFSVqkFhzb/yVnG6rJ+WosRe5Bv9qS6xyexXJu7qTBWi7kEV3SY//KaA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=iL/+1Ovzzk5ip+yquiKZoDvD1L09FZj6JaZuh5HD2sM=; b=W56kkzgzr9xa90yAWcPTo4mDU5iV9gRBqTDmYgJVQArgWFddISXXwKw7E9vrBIoddEI2IFaqEGkWGQTSzsx4E4PqbgheUZO6q+cTP2OHrZR/bqLlDcqmNyFWXxQwe4EoZ1uncYLDuQcLAWqV+0c8J6qTAZTy5gnJzLjRX2WlS0vQ17hXM2FAHRIn9berYAJiIsIFlTHP2nDBRDdxoMXSBZi9KYLMDA8BKUxalMrTtdZ174ftYhgoqRdLyxEpdQ6SvzsgxeGfAwTkLIZ9knedWdN0epMGlT5Qon+iGHkh6gel+bUHoUn7tK/KUVLTHuNGwzc0CJV5zeaxQiZ6dFYEng== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iL/+1Ovzzk5ip+yquiKZoDvD1L09FZj6JaZuh5HD2sM=; b=WltAjyD4WBz27CJmwOp742fSbs8E3sJikV40Ffa0MXy1S+DP6iuKIByZ5I95ps4eJ0eCyEpIeXb3JEHDfDBKfsVevsEPnKQlcBpFTS0Gm+XTDWw1EBq4bEV+WaryC8atHfSlymMuycaekcC/Ar+d9v+AzPjobHMe8G7kTAcsySdAtSNvx74pm94Iy3+9cMZMkDtQZZEwskuB5vLYEBbaz81pL0SMYJTZMe9uY/UNO7KCG4KMmqIEmHaDz4Euotl/weUdBDIP6ZJQQwWgMKRfHZQ5BA4bL9ToyPbOCTKw7mfrPZf+hy1+o4ue4bQfZI4j919RdztTxznFYa2BWu9Skg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by SJ2PR12MB8064.namprd12.prod.outlook.com (2603:10b6:a03:4cc::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.46; Thu, 11 Apr 2024 16:46:22 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.053; Thu, 11 Apr 2024 16:46:22 +0000 From: Jason Gunthorpe To: Alexander Gordeev , Andrew Morton , Christian Borntraeger , Borislav Petkov , Dave Hansen , "David S. Miller" , Eric Dumazet , Gerald Schaefer , Vasily Gorbik , Heiko Carstens , "H. Peter Anvin" , Justin Stitt , Jakub Kicinski , Leon Romanovsky , linux-rdma@vger.kernel.org, linux-s390@vger.kernel.org, llvm@lists.linux.dev, Ingo Molnar , Bill Wendling , Nathan Chancellor , Nick Desaulniers , netdev@vger.kernel.org, Paolo Abeni , Salil Mehta , Sven Schnelle , Thomas Gleixner , x86@kernel.org, Yisen Zhuang Cc: Arnd Bergmann , Catalin Marinas , Leon Romanovsky , linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Mark Rutland , Michael Guralnik , patches@lists.linux.dev, Niklas Schnelle , Jijie Shao , Will Deacon Subject: [PATCH v3 5/6] net: hns3: Remove io_stop_wc() calls after __iowrite64_copy() Date: Thu, 11 Apr 2024 13:46:18 -0300 Message-ID: <5-v3-1893cd8b9369+1925-mlx5_arm_wc_jgg@nvidia.com> In-Reply-To: <0-v3-1893cd8b9369+1925-mlx5_arm_wc_jgg@nvidia.com> References: X-ClientProxiedBy: BLAPR03CA0071.namprd03.prod.outlook.com (2603:10b6:208:329::16) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|SJ2PR12MB8064:EE_ X-MS-Office365-Filtering-Correlation-Id: 162f02e4-4c68-4f16-202b-08dc5a46eac6 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 7JAzZB7dIhZczOp87EmeQ3lo4R0RBRwn9quRWo3Az8U065VXuJTfjNHCCWWt/2Kh0T9JPZNqMYTCbxmZi3cXDDuKr+jqIkJooQ9aJM7xzy8yts26tHkXzPU7xHzhWS89dZ66HeqcU5Z+OokWjnbuk4Beo4fvPAVfnKphT0qVQEiSa0S83gATAVlIoPeyzBfFrEXZdfczbbtpnuu68C2wmncUxuefAdK/J0R/XvJvVk+lGtqc0Xi4bssFDvLxKBEv1IeegF2n6fzsQ5OoVTlY+dPBJPbyzOV4v7KOn4EV0Lj4gkc+eg3nsq+VtL8O82gBF76eeoMjuz3tJ9GFTstYygc/4QnIQSqm80hZtRzIbwmSUaa0qN9jZKO0mv4XXqTSurfvYhhzdwaidzU1d3FCwfkN/ZelAY6I2rt6kmZIkeN9VkiQrNvFoCL35fp/YvIimn+YqATQOt7mU60ZIeEt1V+4RRMB6OZlBuRATGs9ychXd5yzUha5xWojWMH85A+RVX3SvoJbBzH1K5lj60ImuyDAb/R+H1Ag5d0UomlosmXBH7puyJzpjfVM1ft2zoEc3X3viXKP0vGj4OcOKheCvQHr9JKOyLBJJXiA6s4KSbPJHNJmxP7PCK9G2aGNDSus52jC0kkHrgAzFliiAAiRHf2Inh4wgfeaIU+wHFBzeFcNWOst0mvb0OqmbS/4x4ZU4Ph2uQ4gyjD3mi6Z4/Xt2Q== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(1800799015)(376005)(7416005)(366007)(921011);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: X6SvtBQW9hfgnqqofhbe4drSqngbPmUuplwZIDmBET/aj3Tdq3L8tT0gafPbDj91vcTOdzyFtdEonsJT9Q056tJWieVbV/JJOVWGKSSVlDvD+xliMeL7fhFaveEfiR0RjpBJ1Y3VxL8K+HqeejjsTEPgjOp2HtCIsnm2qbTiaaOxWqDfGaVIlnjwRMZlzVyGIRrgCDA5GhZccvjBUFimCKO72C4SuviLffJs2MIjAtsD749S4w0qT70vGqidR13rZ68rwi1ef90NRGAE10kD2SpsW/NIqx6tiQeGr1Li0GYQ90nPwxPFS4bAr724bt9v/e/4Q0rcxoSfR/HGb87f4Ap3Yly3SHPAqsHEYcX8jGFRFMzGug5VFs2JAMTpVYXPqSisq8rj62Vlvez/Mjnuop4WxnoGXDgneYtNbuDj68s4kB8L4ZDf9vBaMLGGXqFzW0GjTLtlbFKdM7gxlHBLuxSLhNzRvEDxFeWm3qpWfArRm79GNGdMJIiPZFXnK/xCraATVbtpBWWYNn5zJzw6uUUBS5M4BSZIsHMcVr0CVDBHju+CgrCDesc4f/lGuLaDoUHwu+skR8hTMs8PXT2hvs+KxpIq0CLaFodyciYRfHRVUPBQh2S0iJsAaNxX7LN/rR6BmQL9uDzyiP6vq9bOCPjUYAiokyjsDt8riyaRdQbhe9l5uIyatWS9FmlyJw96y79uBIiZ4+STBxZvqARu5kjfBIuheIUS7Xp1fDLjYC/qdTdLZ252IPVZ7APwYp3342ZcYxUhTIaRwKODbE9MuYduzWjWrGgnseGi6wvkIPuNbI782LbSgBPrALbzUFvacuP6Hg0cOn47qkusCFCob7dl3sSFC/+oCkPkyOH24ipj6JZrszG4id6r4y+jpneFN4rG5JOsOErapLclINz87WubDL4WjoQ6vkChlbDEdnMf5WrTCFrRlx4fT71ORB7fAXPZLyty4Qc0kWs6lGWtSKtvzu0FSc1+Jpsut65he+Hhw3OWEJENDGQQxWdZpM+zu7FSjBWCb3uJ08FR0Viyt8OC605gXRkfzHv3tRaoKSNv6IV80LJAwc7SfYJxfNnA/P0cee8voVu/UUMZuhiAOjy0r4CEh0Q21AQuYDtTvSDncwLtGauDlLggBD9vL5vCdxibkQiRwcebSZFB77QMcQircK46RfUgyjKfY+mWRNHLymva/CrZLlOURV5rxKj7T1hgKi34ciEpNBVV2SDecUZUZMPhI1Qgn5w+LbB26TTSb6MBs/+4FhfUqbZ7NUX+HdeWlA7SSk22KwdzFt6IWRBMKN0UtmamDeyZ864usH8qiLBLe3ZB/a5HvuXphWmMGE+CyJCkc82A7d4124yn7l1nrfc8fBkGUrZ11WHv1f85gT0ITbVhhDdUSATNhkp8AabJkn0azJIsl7LAAJyF310Dz03Pgnx39m4F1LoeV7sqKfmUCEXV4MwDYzcEcdiSiBOSBHeb6KE6d0WNIxxVew57KEmHUN4XIeGnhbKKPiSDlyEjcMd7rJMocHC3ZrLyG3c5EkjXTH4oevGHza3iKLvZFg3KQ8XkcNcNgJyvGq+QulfjojqsVbEUw63PSbaf X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 162f02e4-4c68-4f16-202b-08dc5a46eac6 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Apr 2024 16:46:21.3880 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: s4Vw9au7lLe47jv4H7gUn9+gsAiz82wDhVtCX7hsISQylfWL9cWHdbq4E0Y3Nkqa X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB8064 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240411_094639_506697_C48856E4 X-CRM114-Status: UNSURE ( 9.44 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Now that the ARM64 arch implementation does the DGH as part of __iowrite64_copy() there is no reason to open code this in drivers. Reviewed-by: Jijie Shao Signed-off-by: Jason Gunthorpe --- drivers/net/ethernet/hisilicon/hns3/hns3_enet.c | 4 ---- 1 file changed, 4 deletions(-) diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3_enet.c b/drivers/net/ethernet/hisilicon/hns3/hns3_enet.c index 19668a8d22f76a..04b9e86363f8fc 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hns3_enet.c +++ b/drivers/net/ethernet/hisilicon/hns3/hns3_enet.c @@ -2068,8 +2068,6 @@ static void hns3_tx_push_bd(struct hns3_enet_ring *ring, int num) __iowrite64_copy(ring->tqp->mem_base, desc, (sizeof(struct hns3_desc) * HNS3_MAX_PUSH_BD_NUM) / HNS3_BYTES_PER_64BIT); - - io_stop_wc(); } static void hns3_tx_mem_doorbell(struct hns3_enet_ring *ring) @@ -2088,8 +2086,6 @@ static void hns3_tx_mem_doorbell(struct hns3_enet_ring *ring) u64_stats_update_begin(&ring->syncp); ring->stats.tx_mem_doorbell += ring->pending_buf; u64_stats_update_end(&ring->syncp); - - io_stop_wc(); } static void hns3_tx_doorbell(struct hns3_enet_ring *ring, int num, From patchwork Thu Apr 11 16:46:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13626420 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EA659C001CC for ; Thu, 11 Apr 2024 16:47:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=kyOv+YTZOIom8LPjRnW/MgiN4Hpp2rwMOyyxw8rAQNI=; b=m+o/Mz8DTTd2wo dSOboZBvn9vsFvPHk2ojrEG7XGTPwMFKWkznIi2lLXXePWyU8Vv3JycfBhbo4jnnKrMTnSyaMna2M zp9gnIcmBtouVmvqsU2CpcXi8Y4Vos3ZUfrHs7hFdM4oOXoqV7E5edOUgBKQLbr4ydepzvy9KYtcu 53WcltjBszrMfUMKMsXVDIvns43rz3Xnv8xqhMStYHvfuwmg9k1pW0zVulZKkF/G+Js1bIqcJBw2w Nns0G1oDtqFTfFXNHJjV9nI3iBelZsYWWRmwr4cl0iDutu97mMhXyjc/EOG8vmINETLR+RcUHL3JA cW7/Ivt2AtCOEXKqnW/w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1ruxZg-0000000D9tG-09jC; Thu, 11 Apr 2024 16:46:56 +0000 Received: from mail-co1nam11on20600.outbound.protection.outlook.com ([2a01:111:f403:2416::600] helo=NAM11-CO1-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1ruxZS-0000000D9hG-2Ijk for linux-arm-kernel@lists.infradead.org; Thu, 11 Apr 2024 16:46:48 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ANyvqUK2sHBM55LTyq27gnMhGkTm7PIQGxUKbypxeyNdgpElh3Exw1Qii55G9peCwncz/vJqzZuhVEW+s/21mJlFb2bs8oW29cgdYPphM5Bv930x3NxT0GPq13oAW5dXklVO6M4LTDs0qrvKCTo6cOp/G/yrMsrHxfDOWFeepSX7okxR/as5fuhTnhipYrRceB95gtwsLGIXdmR/fUstBHGIRy/U/osvdsVUPf0G4Pvn5GOgEOUnB94gW8CTZjzokxfnIEBkj3MjvDtlKju58zZfwn7mteMqD4hwW99WggQCqN0WVFYrbKAOrXlGHkt8uPa2GdHbwghFfG1Z9CoHKA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=mCBJ6dLMHu5mQWHFWQoqDEbAiF7nN/N1UtwVHyaeXCw=; b=ZUhb8p8yYy8H1w0NC6iTkobJhiRurMgINoIQJSYSd039tIlFYaa+rhpNd2QF20kQzliVU4NUFF3WBUzbTtNrtLjip4IjIfia4o10XkPI4EclYAAC2RNui4a2r6txyvokekTdSgwlihiTcbWOFd2RS8DnJihpY8xKWlmn7uEpIvxBEGeA6L4QIwWu01Pw/mQ3dMTAzHRUEtt6K4zZqJNhzsz3Cmk96N2YrRKekh0uxAiQmOkrJVYIHjZpG7uxtHy/o62uO2Ux3lADOBSWSPZ9rdIU6zq6x4kAH9LLmeBwGLl5ECHhyeU/M44sO58DwZEGPcii0VBu3GR68i06Isi91Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mCBJ6dLMHu5mQWHFWQoqDEbAiF7nN/N1UtwVHyaeXCw=; b=RODdimYot7SvWfBj8A9+tRc2B6LuoqiMGSEtOqf+gjUUPMCb9dE3LH+G63yCynqMP7PEmY4ALZ6muoUNHM9Rb9uuz0IIzEqXonfe+AvTvFYwszCbSdnwSiO+GVr8kuYGWCicarmfXbabFntCK3WPMGDYIxKxdz5/Xe0QkSi59NOdeNyIZu5SR5N+NS/p0FEqSWvR88DaZKNYRlMfTuYf3Zq6tCSWW4hOzpg+WPTQy3tgeoh6z3G2Y/uTrnEo9Fz+VwimwiJHOdMIYEsjDtl9VA1m/O4Wl+whY4eij0e0ae6J4D9reBMi/JGQpXsAaC0cgO+hjzZ7TudPi5iTxqxbRQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by SJ2PR12MB8064.namprd12.prod.outlook.com (2603:10b6:a03:4cc::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.46; Thu, 11 Apr 2024 16:46:21 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7409.053; Thu, 11 Apr 2024 16:46:21 +0000 From: Jason Gunthorpe To: Alexander Gordeev , Andrew Morton , Christian Borntraeger , Borislav Petkov , Dave Hansen , "David S. Miller" , Eric Dumazet , Gerald Schaefer , Vasily Gorbik , Heiko Carstens , "H. Peter Anvin" , Justin Stitt , Jakub Kicinski , Leon Romanovsky , linux-rdma@vger.kernel.org, linux-s390@vger.kernel.org, llvm@lists.linux.dev, Ingo Molnar , Bill Wendling , Nathan Chancellor , Nick Desaulniers , netdev@vger.kernel.org, Paolo Abeni , Salil Mehta , Sven Schnelle , Thomas Gleixner , x86@kernel.org, Yisen Zhuang Cc: Arnd Bergmann , Catalin Marinas , Leon Romanovsky , linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Mark Rutland , Michael Guralnik , patches@lists.linux.dev, Niklas Schnelle , Jijie Shao , Will Deacon Subject: [PATCH v3 6/6] IB/mlx5: Use __iowrite64_copy() for write combining stores Date: Thu, 11 Apr 2024 13:46:19 -0300 Message-ID: <6-v3-1893cd8b9369+1925-mlx5_arm_wc_jgg@nvidia.com> In-Reply-To: <0-v3-1893cd8b9369+1925-mlx5_arm_wc_jgg@nvidia.com> References: X-ClientProxiedBy: BL1PR13CA0360.namprd13.prod.outlook.com (2603:10b6:208:2c6::35) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|SJ2PR12MB8064:EE_ X-MS-Office365-Filtering-Correlation-Id: a2d37bec-dfc8-4ea9-b9c5-08dc5a46ea89 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: bAbZtqgSLYoBHuexDrOmcxhNdNitRnYM5AsP/TQ/uHr681wIIQRskhYweCJLioORQmxze6aI5vesNYadFZ3MM+VDL2P/ZSZl9IYWcKhLQOq7z+VLc9gU4aL33pk5DdkpWdm6bMnp+oJPFCcs2n5eMmMZul0b4llYYggYTVrkPMuVGVrXZ0WUBoHF0jzv9So8ZyEWEmwolWe4bG0ePZ9tEGp/DPO7XIXSv9rn4hns1D5Y7wfz5yf0AdL9YveF4I1zA/qHhor0tqqiAH7p/eWQl+xzC5KwJhKIEd5V2SCqKte0M3BRNrhFJsIaOtYZPSJ946+2B8H7TrC8se09v2e3/Q9VXo+/3cZrsJr2h/SPQcZo27qIFEqhX6diNhuNNylpMhg2uytg6LtyOu3ZAlpjE3GE8ekydnaYCVCRbT3tDmSJR+oAbOpLUBZ/40zoqwfUBSikfZ7fMubWnJ2m+i3mX4cC9Dx5frriWsFBQs0LAfTAmPiYncBQpuKbTZDI+O7VEhQqfhIhMd+NHuqk2bhVwxCYBhQ1HlvtW9hNNK0o2hgpraHflobJdq5FXf4cfYYT19XE3K4lX/y2fS1oJiLXvqFmv+v9tA583Dago9NuDfd+aFBIXsk8EyL5SpLRX5mnQ9c17Hxgg7HmFB6ofSmRHHv+x532HQSaL8qiIpeByRDI6WwEtcu4ky/C2xDfZE0Dav2v9LS39HFP7RYrGrIxjA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(1800799015)(376005)(7416005)(366007)(921011);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 2mJ8dZJrhO1gnl+3xL0a5jYH7+vnn6zd2w1S3bqFK6RkSLm5am+GYCJh8L2Fw1QWLHZcWT2Zudt57JwA1rzms8VxVCe50JlX/RORDovLZEQ9SOEzwS0lBgFTmcUcNf4xZVcn/cQ6TpnoTHuTN/z6/xW7F3PymJuqaKbI0PlPBamlKUhrcAWO+5pNX90M4pMe2jDx1Fr6DdaYDn2sSvfZUouaiYBl/uKngPWBm4ofRVdK51cT/hZsP8STtH+Xr/KhWi5FFyhrMYwvrbRnIzZt8ijMH+7KaGE4Gq9vvfp8dikTcGmLuxswVK3DkSiKen3xchOFn+/f7KCz3oVILCbwnp+9c3e/PeR95rHWFF+pJF+DwajfkbKm3cBBzu9lXIE7JSni7sF95a2AMPtiGwoZm/mmJdQCp4pZlk592GGlZ8b+LUkuwdZLRxYsyELDJ/ntbEOjZMWgqLKNpteao4A4yWXjhUBy2MOPN5MejwyN5wHs+DYVeRexbCPkJLA942kBpCzVW5wpiI5EfnF4w5jY/EPT7N0lKGIyVtg9V70sHyFop0HgpnX12fMJJEorBIw33exceiWh4UKn8s5itXE6OP01oYJEIUa32pxPXadEI0+008rgN6eFl0gNu+bWCqE6h2jYlUJdt59wPtV/rIW5MdWv4KgIIFzJKkhDq8qnV8Rp2py4b2Vv4/9X0BOYvTt6sT9Bqz7DjpePgBxQszi1OXxkgswntMKP9eFxCQ0MUOpN9oYiJLJp3ilc5y3/kyt91gDnexg0mOZI1Z/FCpC1prhtNgnrKa9R5R+LckAnQ+YmeaJc87cbSikGqgNKR63v8uoxndqeioQqe0SsRGcQjt7DP3I1mxvOMgpz2AKeKQD88nrn3LaVBx6rBbg0Q332pa5ii1PieWAztToObdwjnDQVPOoqIYBpLvyLxVusAf6GM/lcIDTWdwMwYAPBn2PQFe8xX9Lf7Qe4D3dUj4i8hV54VDcqA2zlOx0aEhCFzns0xpLkEooqHuFkIVc51kvzdPiFxdDrJthAfR1RhCcA4E1tNG3pAmwCZ1m9bzEWpX+G9WcrUg4ZZryN16mPztFT517eJeOD5j2Gjxl0htGe/WpE7FOT220MTJqUw2QoiDyHf0Jjkwzj4qvef8b5Ggfh3WOMCHe7gs8nWd7n+LUgksCLzcWpZ8kH3mPdMJE8FlCM2mP8Pa9VbASreJFl/T20iSndV1TGvED8fjFBAOaBoIKpB5tS8+dbYYcwGyF5a/NYpumVcPLgvw8F2fc/a+fHLzC2pgwkPtqV0jA8ZyLh8pIgqmnzRHpi4SSs8gIbU66FSoqDjeVhbWU+vYSorkwLTv9UasZrCkqaawBe/nNEd5buUyI7dx2VzascxlqNQlngazAam09cFxZ94cTbtQDqMEe/7n/B+yZ6IJrB5AJVQh4rM2hmnMpbhAJzs0KWNSBYHFPzZSYMin3Z0nwSrax1FpIqdynsEwPlw+eRtRgNjIskvFnfI9T05og8oWLNXm3ci5uuwg9SlwefC3TN0vdWfuEY/pg14hhN51440dMhwJWPvbzdrbqsqaTCk7208sgab7fPAhpSFCiq4OSziA7z X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: a2d37bec-dfc8-4ea9-b9c5-08dc5a46ea89 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Apr 2024 16:46:21.0499 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: zfYX9S0grWr1lnmTmc8Lwrv4DMnlxUBjZCJlwsgl7GiBwUWRL2qE0Yx3c29brbA8 X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB8064 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240411_094642_771297_277CE2D1 X-CRM114-Status: GOOD ( 15.62 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org mlx5 has a built in self-test at driver startup to evaluate if the platform supports write combining to generate a 64 byte PCIe TLP or not. This has proven necessary because a lot of common scenarios end up with broken write combining (especially inside virtual machines) and there is other way to learn this information. This self test has been consistently failing on new ARM64 CPU designs (specifically with NVIDIA Grace's implementation of Neoverse V2). The C loop around writeq() generates some pretty terrible ARM64 assembly, but historically this has worked on a lot of existing ARM64 CPUs till now. We see it succeed about 1 time in 10,000 on the worst effected systems. The CPU architects speculate that the load instructions interspersed with the stores makes the WC buffers statistically flush too often and thus the generation of large TLPs becomes infrequent. This makes the boot up test unreliable in that it indicates no write-combining, however userspace would be fine since it uses a ST4 instruction. Further, S390 has similar issues where only the special zpci_memcpy_toio() will actually generate large TLPs, and the open coded loop does not trigger it at all. Fix both ARM64 and S390 by switching to __iowrite64_copy() which now provides architecture specific variants that have a high change of generating a large TLP with write combining. x86 continues to use a similar writeq loop in the generate __iowrite64_copy(). Fixes: 11f552e21755 ("IB/mlx5: Test write combining support") Tested-by: Niklas Schnelle Signed-off-by: Jason Gunthorpe Acked-by: Leon Romanovsky --- drivers/infiniband/hw/mlx5/mem.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/hw/mlx5/mem.c b/drivers/infiniband/hw/mlx5/mem.c index 96ffbbaf0a73d1..5a22be14d958f2 100644 --- a/drivers/infiniband/hw/mlx5/mem.c +++ b/drivers/infiniband/hw/mlx5/mem.c @@ -30,6 +30,7 @@ * SOFTWARE. */ +#include #include #include "mlx5_ib.h" #include @@ -108,7 +109,6 @@ static int post_send_nop(struct mlx5_ib_dev *dev, struct ib_qp *ibqp, u64 wr_id, __be32 mmio_wqe[16] = {}; unsigned long flags; unsigned int idx; - int i; if (unlikely(dev->mdev->state == MLX5_DEVICE_STATE_INTERNAL_ERROR)) return -EIO; @@ -148,10 +148,8 @@ static int post_send_nop(struct mlx5_ib_dev *dev, struct ib_qp *ibqp, u64 wr_id, * we hit doorbell */ wmb(); - for (i = 0; i < 8; i++) - mlx5_write64(&mmio_wqe[i * 2], - bf->bfreg->map + bf->offset + i * 8); - io_stop_wc(); + __iowrite64_copy(bf->bfreg->map + bf->offset, mmio_wqe, + sizeof(mmio_wqe) / 8); bf->offset ^= bf->buf_size;