From patchwork Mon Dec 19 15:35:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13076735 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51293C4167B for ; Mon, 19 Dec 2022 15:43:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 934D28E000A; Mon, 19 Dec 2022 10:43:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 829068E0006; Mon, 19 Dec 2022 10:43:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3E13B8E0003; Mon, 19 Dec 2022 10:43:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id F36308E0003 for ; Mon, 19 Dec 2022 10:43:44 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 99BAD80B3D for ; Mon, 19 Dec 2022 15:43:44 +0000 (UTC) X-FDA: 80259475968.12.E65D012 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) by imf21.hostedemail.com (Postfix) with ESMTP id AB3781C001B for ; Mon, 19 Dec 2022 15:43:42 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=XTpDGZcO; spf=none (imf21.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671464623; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=EGvEEhj67bxGYwjZSrZRnMP7nFcuILVLo/TJsg7wp70=; b=HT/6R5xyYAkH9yxUXASCIphlYtmaKLWQQyJsTkhhLUN9PWs37Ct+RsFAFB9oHVYBdSl9IE JCppJO+i0absb/3uC6hFV/UMzAYMJ0ySK6nbRVq7/o14qLlvGqJ6NqCvEBmKqgWSkYLk0Q iNdclioasQ3+YQX2rgYW+c6UD5zfV30= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=XTpDGZcO; spf=none (imf21.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671464623; a=rsa-sha256; cv=none; b=eUTk3psKCOCba8gttOUN4xtt5gz+LLjJ0Ywx4/G9vQbq4rE8U3v21nsWLdoiyZTuW+ZZ+p CNPdNjpKGdPiWLbgBREMAPPAUFt1dMb42UYTedaVEM3SVtBsDiI1TW5g+lxcoftxOJXxvV igdlYFylF59xSreQYtXSqNCjX7HHcYc= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=EGvEEhj67bxGYwjZSrZRnMP7nFcuILVLo/TJsg7wp70=; b=XTpDGZcOneyi2jU6pOMWNGxSrG Tdqw3sx7AG1vT1ne1fXNaPNp/c69+TrAS0VOUR9s5qT2xIXABRaivHmARLOPbnJq0jzG+7GWUEiE4 eO1FkjNo5cUP4B8MZTqFLdIr08eD9muQ0B3AXs0KOvBV3UbV3RCW3i/O7OXSbm86entb2bwh0I9fN D5sLZ8446vFcqxVt8ybKuHmZk9Lw7DXjhJh5Qg3IyC7Yy85C812g45O996HpRXURZMlMLfklLD21p LvziG6s731SVNOgX+uqOircga46FxYtSbIOBEecUmbyboFFKGW/xwGhoUzyXfAUnoDpZ3INrhuGol Y+kXACug==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1p7III-00CeDq-39; Mon, 19 Dec 2022 15:43:11 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 0EB5F303382; Mon, 19 Dec 2022 16:43:10 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 8676D20B0F899; Mon, 19 Dec 2022 16:43:06 +0100 (CET) Message-ID: <20221219154119.286760562@infradead.org> User-Agent: quilt/0.66 Date: Mon, 19 Dec 2022 16:35:32 +0100 From: Peter Zijlstra To: torvalds@linux-foundation.org Cc: corbet@lwn.net, will@kernel.org, peterz@infradead.org, boqun.feng@gmail.com, mark.rutland@arm.com, catalin.marinas@arm.com, dennis@kernel.org, tj@kernel.org, cl@linux.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, Herbert Xu , davem@davemloft.net, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, joro@8bytes.org, suravee.suthikulpanit@amd.com, robin.murphy@arm.com, dwmw2@infradead.org, baolu.lu@linux.intel.com, Arnd Bergmann , penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, Andrew Morton , vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-s390@vger.kernel.org, linux-crypto@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org Subject: [RFC][PATCH 07/12] percpu: Wire up cmpxchg128 References: <20221219153525.632521981@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: AB3781C001B X-Stat-Signature: fehd7d15kwzjemj8h4fc7pjy59un5g94 X-Rspam-User: X-HE-Tag: 1671464622-12550 X-HE-Meta: U2FsdGVkX1/S9/TeNqEfe5SixdCc+F5IUqA3NbGktErI1tVSl8Kd/sbpBzHEdrYveqhHt1kCo7HGc+Biw7Smk6YnG6NNyABYfP+cEyH5q7GKpDdMg+8FBhvQXvx7cy1MlYXPLkea5CqV/212bCGBQ5baY8c8ZODyu2DMad4q2nvuym70ym06YD09q3phw0JuJo+m32cx/EtcCCob2odCJEDSQCTGkiVgLX0tP5BvBgr/CwVstNQjQZVzP0UK/g/C04e3AskALzLhpqfp6/iWSICGPDf4xw0gBe6Wa//Gu0Inbiv8STrNYKmdrCeoOAMBmG60tdIRVKgMRNeLhXtko0l5Y/XGINphjpwGNZuMN4S8Za1ZBownMEuatjkUFLcJy2o4xoDnm43aKDl573LQjuvdKYf+Gx4M7OuVl8viG4u5N2f5JNgJ4C2cVxLPUJsfXeBnZf79m1pjoOojVEIO9NCujfofMatxaWil27xROjw5VecvfVRqgE35Kl41GIgwM/kzGvAxuW4nYPHsWIhU/YuNULm6QVI4SEnk4KQpDp/rG2K3wvj90Mgn87RF2jZkaSwmcc5viphBHWireoUevWK567PjV9J0+IS4TwLrnQJN+lRGFgjXzhZ+y4WYdW1bS9XgTCJ6SM2jX5C0hq/zBfxA3H427ARedgaZ0NirikYnDfFt2fHrCt8KHLx9QUNnTfrNlIEFVodxzSq3sYSwkc9dr3JmwZvla1XTxMF3p2pds0yoH4oult2kOb89m2yInwLZCcY1JYG9UPVMtBHcyopUW2dn+zw+rT2GbnNqTCZXJO/dLdOafc4fgbut0Cvy+Rkiv94O/zwLc66kdUW4f2svXyn6DE5KmBQ8Hs2BCyFbRFLezJPBBsszKHQdRVHHzImILnvBxSuciayrGorLq6Id9DdCDk6+kDLhVtkdY+IYexhXRIbFbAnSQWEHBrbh/hqIBNZmhAZy2WU+Alv tdg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In order to replace cmpxchg_double() with the newly minted cmpxchg128() family of functions, wire it up in this_cpu_cmpxchg(). Signed-off-by: Peter Zijlstra (Intel) --- arch/arm64/include/asm/percpu.h | 24 ++++++++++++++++++ arch/s390/include/asm/percpu.h | 20 +++++++++++++++ arch/x86/include/asm/percpu.h | 52 ++++++++++++++++++++++++++++++++++++++++ include/asm-generic/percpu.h | 8 ++++++ include/linux/percpu-defs.h | 20 +++++++++++++-- 5 files changed, 122 insertions(+), 2 deletions(-) --- a/arch/arm64/include/asm/percpu.h +++ b/arch/arm64/include/asm/percpu.h @@ -140,6 +140,10 @@ PERCPU_RET_OP(add, add, ldadd) * re-enabling preemption for preemptible kernels, but doing that in a way * which builds inside a module would mean messing directly with the preempt * count. If you do this, peterz and tglx will hunt you down. + * + * Not to mention it'll break the actual preemption model for missing a + * preemption point when TIF_NEED_RESCHED gets set while preemption is + * disabled. */ #define this_cpu_cmpxchg_double_8(ptr1, ptr2, o1, o2, n1, n2) \ ({ \ @@ -240,6 +244,26 @@ PERCPU_RET_OP(add, add, ldadd) #define this_cpu_cmpxchg_8(pcp, o, n) \ _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) +#define __pcpu_cast_128(_exp, _val) \ + _Generic((_exp), \ + u128: (_val), \ + s128: (_val), \ + default: (unsigned long)(_val)) + +#define this_cpu_cmpxchg_16(pcp, o, n) \ +({ \ + u128 old__ = __pcpu_cast_128((o), (o)); \ + u128 new__ = __pcpu_cast_128((n), (n)); \ + typedef typeof(pcp) pcp_op_T__; \ + pcp_op_T__ *ptr__; \ + u128 ret__; \ + preempt_disable_notrace(); \ + ptr__ = raw_cpu_ptr(&(pcp)); \ + ret__ = cmpxchg128_local((void *)ptr__, old__, new__); \ + preempt_enable_notrace(); \ + (typeof(pcp))__pcpu_cast_128(*ptr__, ret__); \ +}) + #ifdef __KVM_NVHE_HYPERVISOR__ extern unsigned long __hyp_per_cpu_offset(unsigned int cpu); #define __per_cpu_offset --- a/arch/s390/include/asm/percpu.h +++ b/arch/s390/include/asm/percpu.h @@ -148,6 +148,26 @@ #define this_cpu_cmpxchg_4(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) #define this_cpu_cmpxchg_8(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) +#define __pcpu_cast_128(_exp, _val) \ + _Generic((_exp), \ + u128: (_val), \ + s128: (_val), \ + default: (unsigned long)(_val)) + +#define this_cpu_cmpxchg_16(pcp, oval, nval) \ +({ \ + u128 old__ = __pcpu_cast_128((nval), (nval)); \ + u128 new__ = __pcpu_cast_128((oval), (oval)); \ + typedef typeof(pcp) pcp_op_T__; \ + pcp_op_T__ *ptr__; \ + u128 ret__; \ + preempt_disable_notrace(); \ + ptr__ = raw_cpu_ptr(&(pcp)); \ + ret__ = cmpxchg128((void *)ptr__, old__, new__); \ + preempt_enable_notrace(); \ + (typeof(pcp))__pcpu_cast_128(*ptr__, ret__); \ +}) + #define arch_this_cpu_xchg(pcp, nval) \ ({ \ typeof(pcp) *ptr__; \ --- a/arch/x86/include/asm/percpu.h +++ b/arch/x86/include/asm/percpu.h @@ -210,6 +210,58 @@ do { \ (typeof(_var))(unsigned long) pco_old__; \ }) +#if defined(CONFIG_X86_32) && defined(CONFIG_X86_CMPXCHG64) +#define __pcpu_cast_64(_exp, _val) \ + _Generic((_exp), \ + u64: (_val), \ + s64: (_val), \ + default: (unsigned long)(_val)) + +#define percpu_cmpxchg64_op(size, qual, _var, _oval, _nval) \ +({ \ + __pcpu_type_##size pco_old__ = __pcpu_cast_64((_oval), (_oval));\ + __pcpu_type_##size pco_new__ = __pcpu_cast_64((_nval), (_nval));\ + asm qual ("cmpxchg8b " __percpu_arg([var]) \ + : [var] "+m" (_var), \ + "+A" (pco_old__) \ + : "b" ((u32)pco_new__), "c" ((u32)(pco_new__ >> 32)) \ + : "memory"); \ + (typeof(_var))__pcpu_cast_64(_var, pco_old__); \ +}) + +#define raw_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg64_op(8, , pcp, oval, nval) +#define this_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg64_op(8, volatile, pcp, oval, nval) +#endif + +#ifdef CONFIG_X86_64 +#define __pcpu_cast_128(_exp, _val) \ + _Generic((_exp), \ + u128: (_val), \ + s128: (_val), \ + default: (unsigned long)(_val)) + +#define percpu_cmpxchg128_op(size, qual, _var, _oval, _nval) \ +({ \ + union __u128_halves pco_old__ = { \ + .full = __pcpu_cast_128((_oval), (_oval)) \ + }; \ + union __u128_halves pco_new__ = { \ + .full = __pcpu_cast_128((_nval), (_nval)) \ + }; \ + asm qual ("cmpxchg16b " __percpu_arg([var]) \ + : [var] "+m" (_var), \ + "+a" (pco_old__.low), \ + "+d" (pco_old__.high) \ + : "b" (pco_new__.low), \ + "c" (pco_new__.high) \ + : "memory"); \ + (typeof(_var))__pcpu_cast_128(_var, pco_old__.full); \ +}) + +#define raw_cpu_cmpxchg_16(pcp, oval, nval) percpu_cmpxchg128_op(16, , pcp, oval, nval) +#define this_cpu_cmpxchg_16(pcp, oval, nval) percpu_cmpxchg128_op(16, volatile, pcp, oval, nval) +#endif + /* * this_cpu_read() makes gcc load the percpu variable every time it is * accessed while this_cpu_read_stable() allows the value to be cached. --- a/include/asm-generic/percpu.h +++ b/include/asm-generic/percpu.h @@ -298,6 +298,10 @@ do { \ #define raw_cpu_cmpxchg_8(pcp, oval, nval) \ raw_cpu_generic_cmpxchg(pcp, oval, nval) #endif +#ifndef raw_cpu_cmpxchg_16 +#define raw_cpu_cmpxchg_16(pcp, oval, nval) \ + raw_cpu_generic_cmpxchg(pcp, oval, nval) +#endif #ifndef raw_cpu_cmpxchg_double_1 #define raw_cpu_cmpxchg_double_1(pcp1, pcp2, oval1, oval2, nval1, nval2) \ @@ -423,6 +427,10 @@ do { \ #define this_cpu_cmpxchg_8(pcp, oval, nval) \ this_cpu_generic_cmpxchg(pcp, oval, nval) #endif +#ifndef this_cpu_cmpxchg_16 +#define this_cpu_cmpxchg_16(pcp, oval, nval) \ + this_cpu_generic_cmpxchg(pcp, oval, nval) +#endif #ifndef this_cpu_cmpxchg_double_1 #define this_cpu_cmpxchg_double_1(pcp1, pcp2, oval1, oval2, nval1, nval2) \ --- a/include/linux/percpu-defs.h +++ b/include/linux/percpu-defs.h @@ -343,6 +343,22 @@ static inline void __this_cpu_preempt_ch pscr2_ret__; \ }) +#define __pcpu_size16_call_return2(stem, variable, ...) \ +({ \ + typeof(variable) pscr2_ret__; \ + __verify_pcpu_ptr(&(variable)); \ + switch(sizeof(variable)) { \ + case 1: pscr2_ret__ = stem##1(variable, __VA_ARGS__); break; \ + case 2: pscr2_ret__ = stem##2(variable, __VA_ARGS__); break; \ + case 4: pscr2_ret__ = stem##4(variable, __VA_ARGS__); break; \ + case 8: pscr2_ret__ = stem##8(variable, __VA_ARGS__); break; \ + case 16: pscr2_ret__ = stem##16(variable, __VA_ARGS__); break; \ + default: \ + __bad_size_call_parameter(); break; \ + } \ + pscr2_ret__; \ +}) + /* * Special handling for cmpxchg_double. cmpxchg_double is passed two * percpu variables. The first has to be aligned to a double word @@ -425,7 +441,7 @@ do { \ #define raw_cpu_add_return(pcp, val) __pcpu_size_call_return2(raw_cpu_add_return_, pcp, val) #define raw_cpu_xchg(pcp, nval) __pcpu_size_call_return2(raw_cpu_xchg_, pcp, nval) #define raw_cpu_cmpxchg(pcp, oval, nval) \ - __pcpu_size_call_return2(raw_cpu_cmpxchg_, pcp, oval, nval) + __pcpu_size16_call_return2(raw_cpu_cmpxchg_, pcp, oval, nval) #define raw_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \ __pcpu_double_call_return_bool(raw_cpu_cmpxchg_double_, pcp1, pcp2, oval1, oval2, nval1, nval2) @@ -512,7 +528,7 @@ do { \ #define this_cpu_add_return(pcp, val) __pcpu_size_call_return2(this_cpu_add_return_, pcp, val) #define this_cpu_xchg(pcp, nval) __pcpu_size_call_return2(this_cpu_xchg_, pcp, nval) #define this_cpu_cmpxchg(pcp, oval, nval) \ - __pcpu_size_call_return2(this_cpu_cmpxchg_, pcp, oval, nval) + __pcpu_size16_call_return2(this_cpu_cmpxchg_, pcp, oval, nval) #define this_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \ __pcpu_double_call_return_bool(this_cpu_cmpxchg_double_, pcp1, pcp2, oval1, oval2, nval1, nval2)