From patchwork Sat Oct 22 11:14:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13015933 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CAF88C433FE for ; Sat, 22 Oct 2022 11:48:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 108958E0002; Sat, 22 Oct 2022 07:48:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DACF68E0005; Sat, 22 Oct 2022 07:48:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BAFDE8E0001; Sat, 22 Oct 2022 07:48:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id A5E568E0002 for ; Sat, 22 Oct 2022 07:48:35 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 7681680A07 for ; Sat, 22 Oct 2022 11:48:35 +0000 (UTC) X-FDA: 80048412990.15.5A8311C Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) by imf02.hostedemail.com (Postfix) with ESMTP id 85B3180003 for ; Sat, 22 Oct 2022 11:48:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=B/ouY7blFSKbM/QG08uUBBh7FdO1AskbZkW9MoHfj6s=; b=S//P5/VxJDDNdFqdqGkA0ghIGL VinkdVqdm4OvZQrImBQTdKp1JukAyaG5YVjjy3e3UbDF00JPqap9zUb+NzaQ9P+zYX2Pis2cBtO0z EgXV49kD0T3r9+YUm/5EWRySgkDdgVD8o2tUfQDEEFP59xFAkC0pvHiU34HWdkAN3NmJAqwoaljai vOSad0sFEWshxHC1ZMIEuH56m4bKhF56jG0i4XooByga1cS3S1h7RrM/RW1O7j+nHFv3L1Cs38kQ9 GL7ZcQmZzyK2TEjFTPmNeE2yXiIj2VXTm5ImT10pf0FddSMPwgv3YCOyx9TMjBlEnjQrdOYxkwymX r5aanGyA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1omCzL-005XdF-G3; Sat, 22 Oct 2022 11:48:28 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 327EC300472; Sat, 22 Oct 2022 13:48:26 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 1C06028B8E50E; Sat, 22 Oct 2022 13:48:26 +0200 (CEST) Message-ID: <20221022114424.515572025@infradead.org> User-Agent: quilt/0.66 Date: Sat, 22 Oct 2022 13:14:04 +0200 From: Peter Zijlstra To: x86@kernel.org, willy@infradead.org, torvalds@linux-foundation.org, akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, linux-mm@kvack.org, aarcange@redhat.com, kirill.shutemov@linux.intel.com, jroedel@suse.de, ubizjak@gmail.com Subject: [PATCH 01/13] mm: Update ptep_get_lockless()s comment References: <20221022111403.531902164@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1666439315; a=rsa-sha256; cv=none; b=CLvsLrMZwdDwBOlTzAgkMO/21lGZufFbGpOH8SMbIxtIHoz5DW2ZbibLuI3tkEK3suS+9C y8tCY4ycfSkZybrfIod3M9JSvITXIhbPEUg89IbSMGtAHZUFRsylVC/Bp/klpIMtjZyUeX 57wRbKrMXZfc6RFRc1bAb2RhmgUapew= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b="S//P5/Vx"; spf=none (imf02.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1666439315; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=B/ouY7blFSKbM/QG08uUBBh7FdO1AskbZkW9MoHfj6s=; b=bG4t8IOK5e2Kas0yKAXcJhtYZB/iArl0AlqE69BuBeew2YllgUu+VfAZ+4m/RnrVSS5I1E kp2kcl2qXNee6o/Hym4bIGhHPPbA6xgXhneOaJM1PXnXjZen3k3j3ytoJM0sHsLuV0QQwt kRWXdN121IHDw0/MblThISmnhj3z+E8= X-Stat-Signature: 5bjjuun51wpgqpb5omdix6cisfahuc9e X-Rspamd-Queue-Id: 85B3180003 X-Rspam-User: X-Rspamd-Server: rspam03 Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b="S//P5/Vx"; spf=none (imf02.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none X-HE-Tag: 1666439314-681228 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Improve the comment. Suggested-by: Matthew Wilcox Signed-off-by: Peter Zijlstra (Intel) Acked-by: Peter Zijlstra (Intel) Acked-by: Gerald Schaefer # s390 --- include/linux/pgtable.h | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -260,15 +260,12 @@ static inline pte_t ptep_get(pte_t *ptep #ifdef CONFIG_GUP_GET_PTE_LOW_HIGH /* - * WARNING: only to be used in the get_user_pages_fast() implementation. - * - * With get_user_pages_fast(), we walk down the pagetables without taking any - * locks. For this we would like to load the pointers atomically, but sometimes - * that is not possible (e.g. without expensive cmpxchg8b on x86_32 PAE). What - * we do have is the guarantee that a PTE will only either go from not present - * to present, or present to not present or both -- it will not switch to a - * completely different present page without a TLB flush in between; something - * that we are blocking by holding interrupts off. + * For walking the pagetables without holding any locks. Some architectures + * (eg x86-32 PAE) cannot load the entries atomically without using expensive + * instructions. We are guaranteed that a PTE will only either go from not + * present to present, or present to not present -- it will not switch to a + * completely different present page without a TLB flush inbetween; which we + * are blocking by holding interrupts off. * * Setting ptes from not present to present goes: * From patchwork Sat Oct 22 11:14:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13015939 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8BB1FA3742 for ; Sat, 22 Oct 2022 11:48:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E49008E000C; Sat, 22 Oct 2022 07:48:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B3AB68E0001; Sat, 22 Oct 2022 07:48:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8A2D08E0008; Sat, 22 Oct 2022 07:48:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 202778E0008 for ; Sat, 22 Oct 2022 07:48:36 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id E9EF21206FB for ; Sat, 22 Oct 2022 11:48:35 +0000 (UTC) X-FDA: 80048412990.09.686969C Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) by imf04.hostedemail.com (Postfix) with ESMTP id 9C3994002E for ; Sat, 22 Oct 2022 11:48:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=f4g92HolWEIcUN5sHQre4J13ebXNQzotEK5t5/pJgYg=; b=fWHHwCBmHCqVGItBT3D823XpSB 9Dn0Fb6t5HnrtAN4U48A3Kztys6P91mnsAdPmGWnZ3YL+ZvkCkwUYv4Ial3sfGO9vpxYlV7hlxvyK 3S4ZVu9VS0hcs0IcRh4wPkHYNzxCF+xcAfF3mKxqWKNm5u3W3mE/zcxAUdZZ35p0u0vwdqmIL82rj HnNNEu1yDU0W/lGAtsf9OLSNIhvyVEARw22A89v685RQawGPHxFiOAqR9vB8sH2yL2Ka38J0NakbM ssY2mMFhQM/DSYKztlMPN7IRfdul4t4thrnD2FzWx4MShTDJuzc0hlc+KfdxZjC/RbZuES/YqJTQo zuzGHOWA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1omCzL-005XdE-G3; Sat, 22 Oct 2022 11:48:28 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 36A6630067F; Sat, 22 Oct 2022 13:48:26 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 1FFF728B8E50D; Sat, 22 Oct 2022 13:48:26 +0200 (CEST) Message-ID: <20221022114424.580310787@infradead.org> User-Agent: quilt/0.66 Date: Sat, 22 Oct 2022 13:14:05 +0200 From: Peter Zijlstra To: x86@kernel.org, willy@infradead.org, torvalds@linux-foundation.org, akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, linux-mm@kvack.org, aarcange@redhat.com, kirill.shutemov@linux.intel.com, jroedel@suse.de, ubizjak@gmail.com Subject: [PATCH 02/13] x86/mm/pae: Make pmd_t similar to pte_t References: <20221022111403.531902164@infradead.org> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=fWHHwCBm; spf=none (imf04.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1666439315; a=rsa-sha256; cv=none; b=sy1kRcZwIwmKTpebtgtEG7tFks07ltY4L8ZvA5T61JCC5VV6e5vQj5oezW0Y4r7k1TLC09 qcmw6WlFEd6vpPI0snim9dcjBZ2Sa3YeZRHh8rUbNUtyBfDnHe1JaXjddtbLPNfDeMALWu 0aPqD99pm/qOyR6Kr/IWKPtNUr8JW2M= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1666439315; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=f4g92HolWEIcUN5sHQre4J13ebXNQzotEK5t5/pJgYg=; b=PPyfpHnwqGb7N8zpJTnVbXl+ZdljIPQwLFDt3h36un4kZ2Ws9m5vWmoGDF1y0Q75J0GnhO O2VPX7DSvIHtcjd/P2RE870DCPtQLZGPFuDqpHQxn4PPz8L3vsj0BKYzxhG2ovOVf+IzRa tXjg5g7Yfh1A/zhyt6rC7pR/Sc8b5lQ= X-Stat-Signature: jycewn4no8f175ntpr5q6ty8twgm8egt X-Rspamd-Queue-Id: 9C3994002E X-Rspam-User: Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=fWHHwCBm; spf=none (imf04.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none X-Rspamd-Server: rspam06 X-HE-Tag: 1666439315-870387 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Instead of mucking about with at least 2 different ways of fudging it, do the same thing we do for pte_t. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/pgtable-3level.h | 42 +++++++++------------------- arch/x86/include/asm/pgtable-3level_types.h | 7 ++++ arch/x86/include/asm/pgtable_64_types.h | 1 arch/x86/include/asm/pgtable_types.h | 4 -- 4 files changed, 23 insertions(+), 31 deletions(-) --- a/arch/x86/include/asm/pgtable-3level.h +++ b/arch/x86/include/asm/pgtable-3level.h @@ -87,7 +87,7 @@ static inline pmd_t pmd_read_atomic(pmd_ ret |= ((pmdval_t)*(tmp + 1)) << 32; } - return (pmd_t) { ret }; + return (pmd_t) { .pmd = ret }; } static inline void native_set_pte_atomic(pte_t *ptep, pte_t pte) @@ -121,12 +121,11 @@ static inline void native_pte_clear(stru ptep->pte_high = 0; } -static inline void native_pmd_clear(pmd_t *pmd) +static inline void native_pmd_clear(pmd_t *pmdp) { - u32 *tmp = (u32 *)pmd; - *tmp = 0; + pmdp->pmd_low = 0; smp_wmb(); - *(tmp + 1) = 0; + pmdp->pmd_high = 0; } static inline void native_pud_clear(pud_t *pudp) @@ -162,25 +161,17 @@ static inline pte_t native_ptep_get_and_ #define native_ptep_get_and_clear(xp) native_local_ptep_get_and_clear(xp) #endif -union split_pmd { - struct { - u32 pmd_low; - u32 pmd_high; - }; - pmd_t pmd; -}; - #ifdef CONFIG_SMP static inline pmd_t native_pmdp_get_and_clear(pmd_t *pmdp) { - union split_pmd res, *orig = (union split_pmd *)pmdp; + pmd_t res; /* xchg acts as a barrier before setting of the high bits */ - res.pmd_low = xchg(&orig->pmd_low, 0); - res.pmd_high = orig->pmd_high; - orig->pmd_high = 0; + res.pmd_low = xchg(&pmdp->pmd_low, 0); + res.pmd_high = READ_ONCE(pmdp->pmd_high); + WRITE_ONCE(pmdp->pmd_high, 0); - return res.pmd; + return res; } #else #define native_pmdp_get_and_clear(xp) native_local_pmdp_get_and_clear(xp) @@ -199,17 +190,12 @@ static inline pmd_t pmdp_establish(struc * anybody. */ if (!(pmd_val(pmd) & _PAGE_PRESENT)) { - union split_pmd old, new, *ptr; - - ptr = (union split_pmd *)pmdp; - - new.pmd = pmd; - /* xchg acts as a barrier before setting of the high bits */ - old.pmd_low = xchg(&ptr->pmd_low, new.pmd_low); - old.pmd_high = ptr->pmd_high; - ptr->pmd_high = new.pmd_high; - return old.pmd; + old.pmd_low = xchg(&pmdp->pmd_low, pmd.pmd_low); + old.pmd_high = READ_ONCE(pmdp->pmd_high); + WRITE_ONCE(pmdp->pmd_high, pmd.pmd_high); + + return old; } do { --- a/arch/x86/include/asm/pgtable-3level_types.h +++ b/arch/x86/include/asm/pgtable-3level_types.h @@ -18,6 +18,13 @@ typedef union { }; pteval_t pte; } pte_t; + +typedef union { + struct { + unsigned long pmd_low, pmd_high; + }; + pmdval_t pmd; +} pmd_t; #endif /* !__ASSEMBLY__ */ #define SHARED_KERNEL_PMD (!static_cpu_has(X86_FEATURE_PTI)) --- a/arch/x86/include/asm/pgtable_64_types.h +++ b/arch/x86/include/asm/pgtable_64_types.h @@ -19,6 +19,7 @@ typedef unsigned long pgdval_t; typedef unsigned long pgprotval_t; typedef struct { pteval_t pte; } pte_t; +typedef struct { pmdval_t pmd; } pmd_t; #ifdef CONFIG_X86_5LEVEL extern unsigned int __pgtable_l5_enabled; --- a/arch/x86/include/asm/pgtable_types.h +++ b/arch/x86/include/asm/pgtable_types.h @@ -381,11 +381,9 @@ static inline pudval_t native_pud_val(pu #endif #if CONFIG_PGTABLE_LEVELS > 2 -typedef struct { pmdval_t pmd; } pmd_t; - static inline pmd_t native_make_pmd(pmdval_t val) { - return (pmd_t) { val }; + return (pmd_t) { .pmd = val }; } static inline pmdval_t native_pmd_val(pmd_t pmd) From patchwork Sat Oct 22 11:14:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13015934 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B93B4C04A95 for ; Sat, 22 Oct 2022 11:48:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D112D8E0003; Sat, 22 Oct 2022 07:48:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C99F08E0002; Sat, 22 Oct 2022 07:48:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B13908E0003; Sat, 22 Oct 2022 07:48:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 9CF848E0001 for ; Sat, 22 Oct 2022 07:48:35 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 73F77A08A4 for ; Sat, 22 Oct 2022 11:48:35 +0000 (UTC) X-FDA: 80048412990.17.0C7B53A Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf28.hostedemail.com (Postfix) with ESMTP id 5A377C000B for ; Sat, 22 Oct 2022 11:48:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=VsbQJE2gXZYIdS3+vUbgKcHjJGGPNYaddyEXb2iNKwA=; b=R/Cm2B1BOg/6TAQKIgkjPpWVyg aJqvcmGvaLHJIyjHUTTOC9NQOLfcdXPVFDNmrXjpOfj2HRuokxNbnDFv/xyauV8LYwlc6vtAvUbv8 qVx3drzMG8oj3vR8aj3v3JuvhjtGCrH6x8DIszHxZibmaxQOjKYKhniIsmx8QKAyVh8T0SzLH+L3N YIuC8UEvEF+f63BUsupCIiDX3G/ZGcBdpuKPmYfTMyl6VsHmFXx3eesedrDyRlojc94tKwrIsNqOg 7SCZNRhzklPUgvlZ2P4w8vhBw28FFxoMep0YjTvSE0ICbGaBrTrF7eV5JUiFFZy5fsqJBBxZu0Mu4 AbqXXtlw==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1omCzQ-00Dtgp-6K; Sat, 22 Oct 2022 11:48:32 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 397173006B1; Sat, 22 Oct 2022 13:48:26 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 241D428B8E50B; Sat, 22 Oct 2022 13:48:26 +0200 (CEST) Message-ID: <20221022114424.645657294@infradead.org> User-Agent: quilt/0.66 Date: Sat, 22 Oct 2022 13:14:06 +0200 From: Peter Zijlstra To: x86@kernel.org, willy@infradead.org, torvalds@linux-foundation.org, akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, linux-mm@kvack.org, aarcange@redhat.com, kirill.shutemov@linux.intel.com, jroedel@suse.de, ubizjak@gmail.com Subject: [PATCH 03/13] sh/mm: Make pmd_t similar to pte_t References: <20221022111403.531902164@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1666439315; a=rsa-sha256; cv=none; b=D9bPi556U225RThDouGCSsODBAfZdd3HAFRPJ9R7O3p+3zOoGwQwzuZ2a/IxvErD+LR/LM ArBu7FgXIzPA8PmJXCy98NHN/4jc+hb6tbz7iofQHdw4j2mfFMxjpc4v+Xm8SO/dtmkWeG UAsuqCHkyhulrJ9ZkOAbMZ3lDhNeHfQ= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="R/Cm2B1B"; spf=none (imf28.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1666439315; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=VsbQJE2gXZYIdS3+vUbgKcHjJGGPNYaddyEXb2iNKwA=; b=8nB3dGr1OqPByJNZcP9nmbqyHFNGkRP1SMGV4qX/XJihdRp+TVwfCvjFg+nNII6i2ERAwa 5MRE3uOkE/ybaGYMhXHx94Q/4mHj2NtceR8ykJa8ZPHf8SzAjQrN/v+/c3yZGJ4NwOkPrO IPzKtG693IscPFAZZuoLZPp7c3pyDWw= X-Stat-Signature: xokdarcyem39sekk8qrusdgp4oxygmm6 X-Rspamd-Queue-Id: 5A377C000B X-Rspam-User: X-Rspamd-Server: rspam03 Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="R/Cm2B1B"; spf=none (imf28.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org; dmarc=none X-HE-Tag: 1666439314-651688 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Just like 64bit pte_t, have a low/high split in pmd_t. Signed-off-by: Peter Zijlstra (Intel) --- arch/sh/include/asm/pgtable-3level.h | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) --- a/arch/sh/include/asm/pgtable-3level.h +++ b/arch/sh/include/asm/pgtable-3level.h @@ -28,9 +28,15 @@ #define pmd_ERROR(e) \ printk("%s:%d: bad pmd %016llx.\n", __FILE__, __LINE__, pmd_val(e)) -typedef struct { unsigned long long pmd; } pmd_t; +typedef struct { + struct { + unsigned long pmd_low; + unsigned long pmd_high; + }; + unsigned long long pmd; +} pmd_t; #define pmd_val(x) ((x).pmd) -#define __pmd(x) ((pmd_t) { (x) } ) +#define __pmd(x) ((pmd_t) { .pmd = (x) } ) static inline pmd_t *pud_pgtable(pud_t pud) { From patchwork Sat Oct 22 11:14:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13015935 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49A0AFA373D for ; Sat, 22 Oct 2022 11:48:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 921E88E0006; Sat, 22 Oct 2022 07:48:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 743098E0001; Sat, 22 Oct 2022 07:48:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 520718E0007; Sat, 22 Oct 2022 07:48:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id F1D9E8E0007 for ; Sat, 22 Oct 2022 07:48:35 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id BD18FA0AF4 for ; Sat, 22 Oct 2022 11:48:35 +0000 (UTC) X-FDA: 80048412990.14.32927D6 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) by imf05.hostedemail.com (Postfix) with ESMTP id 49960100010 for ; Sat, 22 Oct 2022 11:48:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=WrWjWAdXlYrZgI+2wdU5TCD9SMPt4h2Vfa3Cvok7+uU=; b=nwDkzw3R5O12iesOP1q8VwET/4 ZsL/dWqBoN6o0f0YpYhXVZ3YijSmTavj8IkPaWHQEqvefhwtLf1jemo/PGf5ldXjjwMRsIjSqacpe xst/jckJF8WzaoFNFKMklCC16WgqrSaV6BVJBi3IWWzqOi4gfDwqytHzaZmw3wjdmpWGiFZle50zi kPQ/2PGDg0ljw0gZjsm4HzQ5CICs1p+mnrJeR7VDEtP9QHcoJ/+q3IFu0klABPRfnUhNCxm6hhtrI gWNG3kd08LexHBzRJStQuirx3jBgrbnqxgJjQsfmcJzsW/K5l6IAmX2YsavdKyVGhUNAtUCo6j/TI v8um1bAQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1omCzL-005XdD-G3; Sat, 22 Oct 2022 11:48:28 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 3E33D300BBA; Sat, 22 Oct 2022 13:48:26 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 2841228B8E50F; Sat, 22 Oct 2022 13:48:26 +0200 (CEST) Message-ID: <20221022114424.711181252@infradead.org> User-Agent: quilt/0.66 Date: Sat, 22 Oct 2022 13:14:07 +0200 From: Peter Zijlstra To: x86@kernel.org, willy@infradead.org, torvalds@linux-foundation.org, akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, linux-mm@kvack.org, aarcange@redhat.com, kirill.shutemov@linux.intel.com, jroedel@suse.de, ubizjak@gmail.com Subject: [PATCH 04/13] mm: Fix pmd_read_atomic() References: <20221022111403.531902164@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1666439315; a=rsa-sha256; cv=none; b=jc7VfrB/W2lcV3Cnr0tVI/XhTrGOyOurJsf/D84a2bAMLXKfHIzR4nBSuhnnVsls7BPrjk HtLHCWxhNP22uzecYMnpsaAuN3ndlcSQVPV1taD2eQYYGcX0yLKONaz3j0x64+I0/xtnKv VPv2gBiHhUT55kQN5GKzYlmwC0fGLDQ= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=nwDkzw3R; spf=none (imf05.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1666439315; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=WrWjWAdXlYrZgI+2wdU5TCD9SMPt4h2Vfa3Cvok7+uU=; b=HoTxFW8InHijn8G4la520QZYqPASPB4uFPNKDPccpnMMcZ7DMZ0jfeQEjsC12XmFAHxa2V ROoM5CkOhrLFvihe+nv35X1MyEMsi7BbAHzrSEfZx2MKhiPNWhQki2Tkhm/v1vGCY3k43a 47L9erDVE38bISHr1FH5FM+dac7EbnQ= X-Stat-Signature: i54o67w8zx8mo94pn8xhhiaxbz5uy15y X-Rspamd-Queue-Id: 49960100010 Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=nwDkzw3R; spf=none (imf05.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1666439314-424366 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: AFAICT there's no reason to do anything different than what we do for PTEs. Make it so (also affects SH). Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/pgtable-3level.h | 56 ---------------------------------- include/linux/pgtable.h | 49 +++++++++++++++++++++++------ 2 files changed, 39 insertions(+), 66 deletions(-) --- a/arch/x86/include/asm/pgtable-3level.h +++ b/arch/x86/include/asm/pgtable-3level.h @@ -34,62 +34,6 @@ static inline void native_set_pte(pte_t ptep->pte_low = pte.pte_low; } -#define pmd_read_atomic pmd_read_atomic -/* - * pte_offset_map_lock() on 32-bit PAE kernels was reading the pmd_t with - * a "*pmdp" dereference done by GCC. Problem is, in certain places - * where pte_offset_map_lock() is called, concurrent page faults are - * allowed, if the mmap_lock is hold for reading. An example is mincore - * vs page faults vs MADV_DONTNEED. On the page fault side - * pmd_populate() rightfully does a set_64bit(), but if we're reading the - * pmd_t with a "*pmdp" on the mincore side, a SMP race can happen - * because GCC will not read the 64-bit value of the pmd atomically. - * - * To fix this all places running pte_offset_map_lock() while holding the - * mmap_lock in read mode, shall read the pmdp pointer using this - * function to know if the pmd is null or not, and in turn to know if - * they can run pte_offset_map_lock() or pmd_trans_huge() or other pmd - * operations. - * - * Without THP if the mmap_lock is held for reading, the pmd can only - * transition from null to not null while pmd_read_atomic() runs. So - * we can always return atomic pmd values with this function. - * - * With THP if the mmap_lock is held for reading, the pmd can become - * trans_huge or none or point to a pte (and in turn become "stable") - * at any time under pmd_read_atomic(). We could read it truly - * atomically here with an atomic64_read() for the THP enabled case (and - * it would be a whole lot simpler), but to avoid using cmpxchg8b we - * only return an atomic pmdval if the low part of the pmdval is later - * found to be stable (i.e. pointing to a pte). We are also returning a - * 'none' (zero) pmdval if the low part of the pmd is zero. - * - * In some cases the high and low part of the pmdval returned may not be - * consistent if THP is enabled (the low part may point to previously - * mapped hugepage, while the high part may point to a more recently - * mapped hugepage), but pmd_none_or_trans_huge_or_clear_bad() only - * needs the low part of the pmd to be read atomically to decide if the - * pmd is unstable or not, with the only exception when the low part - * of the pmd is zero, in which case we return a 'none' pmd. - */ -static inline pmd_t pmd_read_atomic(pmd_t *pmdp) -{ - pmdval_t ret; - u32 *tmp = (u32 *)pmdp; - - ret = (pmdval_t) (*tmp); - if (ret) { - /* - * If the low part is null, we must not read the high part - * or we can end up with a partial pmd. - */ - smp_rmb(); - ret |= ((pmdval_t)*(tmp + 1)) << 32; - } - - return (pmd_t) { .pmd = ret }; -} - static inline void native_set_pte_atomic(pte_t *ptep, pte_t pte) { set_64bit((unsigned long long *)(ptep), native_pte_val(pte)); --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -258,6 +258,13 @@ static inline pte_t ptep_get(pte_t *ptep } #endif +#ifndef __HAVE_ARCH_PMDP_GET +static inline pmd_t pmdp_get(pmd_t *pmdp) +{ + return READ_ONCE(*pmdp); +} +#endif + #ifdef CONFIG_GUP_GET_PTE_LOW_HIGH /* * For walking the pagetables without holding any locks. Some architectures @@ -302,15 +309,42 @@ static inline pte_t ptep_get_lockless(pt return pte; } -#else /* CONFIG_GUP_GET_PTE_LOW_HIGH */ +#define ptep_get_lockless ptep_get_lockless + +#if CONFIG_PGTABLE_LEVELS > 2 +static inline pmd_t pmdp_get_lockless(pmd_t *pmdp) +{ + pmd_t pmd; + + do { + pmd.pmd_low = pmdp->pmd_low; + smp_rmb(); + pmd.pmd_high = pmdp->pmd_high; + smp_rmb(); + } while (unlikely(pmd.pmd_low != pmdp->pmd_low)); + + return pmd; +} +#define pmdp_get_lockless pmdp_get_lockless +#endif /* CONFIG_PGTABLE_LEVELS > 2 */ +#endif /* CONFIG_GUP_GET_PTE_LOW_HIGH */ + /* * We require that the PTE can be read atomically. */ +#ifndef ptep_get_lockless static inline pte_t ptep_get_lockless(pte_t *ptep) { return ptep_get(ptep); } -#endif /* CONFIG_GUP_GET_PTE_LOW_HIGH */ +#endif + +#ifndef pmdp_get_lockless +static inline pmd_t pmdp_get_lockless(pmd_t *pmdp) +{ + return pmdp_get(pmdp); +} +#endif #ifdef CONFIG_TRANSPARENT_HUGEPAGE #ifndef __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR @@ -1211,17 +1247,10 @@ static inline int pud_trans_unstable(pud #endif } -#ifndef pmd_read_atomic static inline pmd_t pmd_read_atomic(pmd_t *pmdp) { - /* - * Depend on compiler for an atomic pmd read. NOTE: this is - * only going to work, if the pmdval_t isn't larger than - * an unsigned long. - */ - return *pmdp; + return pmdp_get_lockless(pmdp); } -#endif #ifndef arch_needs_pgtable_deposit #define arch_needs_pgtable_deposit() (false) From patchwork Sat Oct 22 11:14:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13015943 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C9B9C433FE for ; Sat, 22 Oct 2022 11:48:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1EF9D8E000D; Sat, 22 Oct 2022 07:48:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 177128E000A; Sat, 22 Oct 2022 07:48:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EBF248E000D; Sat, 22 Oct 2022 07:48:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D505B8E000A for ; Sat, 22 Oct 2022 07:48:39 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id AB5EC1A05FD for ; Sat, 22 Oct 2022 11:48:39 +0000 (UTC) X-FDA: 80048413158.11.5C0B732 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf24.hostedemail.com (Postfix) with ESMTP id 53C4A18000C for ; Sat, 22 Oct 2022 11:48:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=TK/Z6RlKcsjeIiMJAK610lvWrcZp9kpetOjgnaxEVyk=; b=BukeXEN+MIjiqbC4c+Me1GShNp 3NJEOydsKmBtXlCm9wZSBPlu7rh1Tpdlku9TEONhdRsvIEUUB+bqMHMSm1mvu7KAgQPtSwMg4xvVP Y8xYk+e0IhS0rasaGjclCuquJU+ICuftzGWoGhAr8Zz1A+LQFPCC3Vx3cw7UAs74Lh2KxPL1vNJnA 7F4+ebmeUEF0Y0T/0KGU0Qh4x75bo4Pi7iIXE/tgZgByH3FEN75c7yP7CWc522xWgxDQzT/j/YQII y60Lvw31VYz/F7wfeNBNBsmpc4sJJDb665REzL/H0+BzHuN2LAKPzZnF+fbzi2K4hiN5y/0t+Y58q da2+8wqw==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1omCzR-00Dtgu-Ja; Sat, 22 Oct 2022 11:48:33 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id BDB1E301FC4; Sat, 22 Oct 2022 13:48:26 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 2C25E28B8E510; Sat, 22 Oct 2022 13:48:26 +0200 (CEST) Message-ID: <20221022114424.776404066@infradead.org> User-Agent: quilt/0.66 Date: Sat, 22 Oct 2022 13:14:08 +0200 From: Peter Zijlstra To: x86@kernel.org, willy@infradead.org, torvalds@linux-foundation.org, akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, linux-mm@kvack.org, aarcange@redhat.com, kirill.shutemov@linux.intel.com, jroedel@suse.de, ubizjak@gmail.com Subject: [PATCH 05/13] mm: Rename GUP_GET_PTE_LOW_HIGH References: <20221022111403.531902164@infradead.org> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=BukeXEN+; spf=none (imf24.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1666439319; a=rsa-sha256; cv=none; b=0Pe3f4Vi5HfQ1MIopx2f27+WmUhctEEAKW51sEe5dEt9JbxBA+CeUhUVsCnfr5EE/1NZGw lgKmwTcBYXMkxnZsk00gSX9OCKYq/B6VSgApcZdLFz1odTaM1UIsvnmB0kG/GOGyRe0wiL B8ML4udNdlzrBveg2oxbNUgANwT7lDY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1666439319; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=TK/Z6RlKcsjeIiMJAK610lvWrcZp9kpetOjgnaxEVyk=; b=aEr+UuZ7DtmHNdKZE04NURysOVLIJudMg+5c1TAzcNhfVvJTTqIBSqGnwdHZfIAIAVQi3u msofh3essItMcmDA88SVc3PFATU3NZAEuWIH+AAAi8UIlnrFNYmvQ5MLSwfb4+ovCnnur0 Tqvti0UPXMwxwD+GLujdmzBY6Difs8c= X-Rspamd-Queue-Id: 53C4A18000C Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=BukeXEN+; spf=none (imf24.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org; dmarc=none X-Rspam-User: X-Rspamd-Server: rspam09 X-Stat-Signature: 7durjzsgj7tzgxxyb5zxijzemey3axqq X-HE-Tag: 1666439319-213791 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since it no longer applies to only PTEs, rename it to PXX. Suggested-by: Linus Torvalds Signed-off-by: Peter Zijlstra (Intel) --- arch/mips/Kconfig | 2 +- arch/sh/Kconfig | 2 +- arch/x86/Kconfig | 2 +- include/linux/pgtable.h | 4 ++-- mm/Kconfig | 2 +- 5 files changed, 6 insertions(+), 6 deletions(-) --- a/arch/mips/Kconfig +++ b/arch/mips/Kconfig @@ -46,7 +46,7 @@ config MIPS select GENERIC_SCHED_CLOCK if !CAVIUM_OCTEON_SOC select GENERIC_SMP_IDLE_THREAD select GENERIC_TIME_VSYSCALL - select GUP_GET_PTE_LOW_HIGH if CPU_MIPS32 && PHYS_ADDR_T_64BIT + select GUP_GET_PXX_LOW_HIGH if CPU_MIPS32 && PHYS_ADDR_T_64BIT select HAVE_ARCH_COMPILER_H select HAVE_ARCH_JUMP_LABEL select HAVE_ARCH_KGDB if MIPS_FP_SUPPORT --- a/arch/sh/Kconfig +++ b/arch/sh/Kconfig @@ -24,7 +24,7 @@ config SUPERH select GENERIC_PCI_IOMAP if PCI select GENERIC_SCHED_CLOCK select GENERIC_SMP_IDLE_THREAD - select GUP_GET_PTE_LOW_HIGH if X2TLB + select GUP_GET_PXX_LOW_HIGH if X2TLB select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_KGDB select HAVE_ARCH_SECCOMP_FILTER --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -157,7 +157,7 @@ config X86 select GENERIC_TIME_VSYSCALL select GENERIC_GETTIMEOFDAY select GENERIC_VDSO_TIME_NS - select GUP_GET_PTE_LOW_HIGH if X86_PAE + select GUP_GET_PXX_LOW_HIGH if X86_PAE select HARDIRQS_SW_RESEND select HARDLOCKUP_CHECK_TIMESTAMP if X86_64 select HAVE_ACPI_APEI if ACPI --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -305,7 +305,7 @@ static inline pmd_t pmdp_get(pmd_t *pmdp } #endif -#ifdef CONFIG_GUP_GET_PTE_LOW_HIGH +#ifdef CONFIG_GUP_GET_PXX_LOW_HIGH /* * For walking the pagetables without holding any locks. Some architectures * (eg x86-32 PAE) cannot load the entries atomically without using expensive @@ -365,7 +365,7 @@ static inline pmd_t pmdp_get_lockless(pm } #define pmdp_get_lockless pmdp_get_lockless #endif /* CONFIG_PGTABLE_LEVELS > 2 */ -#endif /* CONFIG_GUP_GET_PTE_LOW_HIGH */ +#endif /* CONFIG_GUP_GET_PXX_LOW_HIGH */ /* * We require that the PTE can be read atomically. --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1044,7 +1044,7 @@ config GUP_TEST comment "GUP_TEST needs to have DEBUG_FS enabled" depends on !GUP_TEST && !DEBUG_FS -config GUP_GET_PTE_LOW_HIGH +config GUP_GET_PXX_LOW_HIGH bool config ARCH_HAS_PTE_SPECIAL From patchwork Sat Oct 22 11:14:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13015938 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA977C3A59D for ; Sat, 22 Oct 2022 11:48:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 522C58E0009; Sat, 22 Oct 2022 07:48:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4D1248E0006; Sat, 22 Oct 2022 07:48:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D8F88E0009; Sat, 22 Oct 2022 07:48:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id EF32E8E0006 for ; Sat, 22 Oct 2022 07:48:35 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id C5947A0B9C for ; Sat, 22 Oct 2022 11:48:35 +0000 (UTC) X-FDA: 80048412990.14.7705002 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) by imf11.hostedemail.com (Postfix) with ESMTP id 68C3D40036 for ; Sat, 22 Oct 2022 11:48:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=lkZdGp83/TthtV3rxO0LKzlFIb0aX3v3XYJ5iZQQNzg=; b=DvKf19ngBUpdzl0ij7xBmZMRXh ui/RMKslNxqoOsZHJQEPa5zDXLSQ3lYvqgHe3zbQqojpn0U9BuDn4pJaoxbaK86/AQqbdKJhh3oYD ktC6an0vkVyWJJKA9MR4FG0zwzprr5Xgm+MolIvlAy29FCTpuSFLo1xrWIqCK/O8Qs5Gsck/i8qIQ 4mfjWuAYUdAynm0EIt1Pp3+QSrXSg1pT6whi7abx5v7Sj6NZsGkXKAvhdlLwSQjvnDw18bP07Zv4y Mc6ZlO9yHBlrAsiBQYNLFfQ3Yhfz13CO5XsuOaulhx5bKEzAJXiaIYGya4dd4IzKwJEJA1YcKJKVh 2tGUPmqA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1omCzM-005XdR-WF; Sat, 22 Oct 2022 11:48:29 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id BDB32302D62; Sat, 22 Oct 2022 13:48:26 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 3015B28B8E511; Sat, 22 Oct 2022 13:48:26 +0200 (CEST) Message-ID: <20221022114424.841277397@infradead.org> User-Agent: quilt/0.66 Date: Sat, 22 Oct 2022 13:14:09 +0200 From: Peter Zijlstra To: x86@kernel.org, willy@infradead.org, torvalds@linux-foundation.org, akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, linux-mm@kvack.org, aarcange@redhat.com, kirill.shutemov@linux.intel.com, jroedel@suse.de, ubizjak@gmail.com Subject: [PATCH 06/13] mm: Rename pmd_read_atomic() References: <20221022111403.531902164@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1666439315; a=rsa-sha256; cv=none; b=XARJsY+cm9F1xVlHMhYPJGP0HWM5+sIGccPnd30vzykN36aPDthBijM6VIHI+rKo6L1Iid ROnR8QN2RsBX18ZqFC42VAJ8l4A6W2rVyYKxSnfZZiuiaW/aSTqPAyHBu63k8yaV4yJMfB jpQx2+62s1J6xye4qkdh7B2Y5j3JjEs= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=DvKf19ng; spf=none (imf11.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1666439315; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=lkZdGp83/TthtV3rxO0LKzlFIb0aX3v3XYJ5iZQQNzg=; b=04jt9Jq7eucgm1x2CoQjU36a0F1vgHVftQ2ktKerUafiaacGRfkGApi274NS1yF5/5JLXY zL7DTa6Mk5HH+buh9LAChoAsMxLiTf8hYmpuVG2Hpn8PF7FQwk1cOFJHZLc6CMn56OToYe ZrM4CCD7T3Zdk5hb28fn5j0PenhJEOY= X-Stat-Signature: wyc9y75f9awb5hkg1rjwrko4d8u3oxwj X-Rspamd-Queue-Id: 68C3D40036 Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=DvKf19ng; spf=none (imf11.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none X-Rspam-User: X-Rspamd-Server: rspam05 X-HE-Tag: 1666439314-580811 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There's no point in having the identical routines for PTE/PMD have different names. Signed-off-by: Peter Zijlstra (Intel) --- include/linux/pgtable.h | 9 ++------- mm/hmm.c | 2 +- mm/khugepaged.c | 2 +- mm/mapping_dirty_helpers.c | 2 +- mm/mprotect.c | 2 +- mm/userfaultfd.c | 2 +- mm/vmscan.c | 4 ++-- 7 files changed, 9 insertions(+), 14 deletions(-) --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1352,11 +1352,6 @@ static inline int pud_trans_unstable(pud #endif } -static inline pmd_t pmd_read_atomic(pmd_t *pmdp) -{ - return pmdp_get_lockless(pmdp); -} - #ifndef arch_needs_pgtable_deposit #define arch_needs_pgtable_deposit() (false) #endif @@ -1383,13 +1378,13 @@ static inline pmd_t pmd_read_atomic(pmd_ */ static inline int pmd_none_or_trans_huge_or_clear_bad(pmd_t *pmd) { - pmd_t pmdval = pmd_read_atomic(pmd); + pmd_t pmdval = pmdp_get_lockless(pmd); /* * The barrier will stabilize the pmdval in a register or on * the stack so that it will stop changing under the code. * * When CONFIG_TRANSPARENT_HUGEPAGE=y on x86 32bit PAE, - * pmd_read_atomic is allowed to return a not atomic pmdval + * pmdp_get_lockless is allowed to return a not atomic pmdval * (for example pointing to an hugepage that has never been * mapped in the pmd). The below checks will only care about * the low part of the pmd with 32bit PAE x86 anyway, with the --- a/mm/hmm.c +++ b/mm/hmm.c @@ -361,7 +361,7 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, * huge or device mapping one and compute corresponding pfn * values. */ - pmd = pmd_read_atomic(pmdp); + pmd = pmdp_get_lockless(pmdp); barrier(); if (!pmd_devmap(pmd) && !pmd_trans_huge(pmd)) goto again; --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -862,7 +862,7 @@ static int find_pmd_or_thp_or_none(struc if (!*pmd) return SCAN_PMD_NULL; - pmde = pmd_read_atomic(*pmd); + pmde = pmdp_get_lockless(*pmd); #ifdef CONFIG_TRANSPARENT_HUGEPAGE /* See comments in pmd_none_or_trans_huge_or_clear_bad() */ --- a/mm/mapping_dirty_helpers.c +++ b/mm/mapping_dirty_helpers.c @@ -126,7 +126,7 @@ static int clean_record_pte(pte_t *pte, static int wp_clean_pmd_entry(pmd_t *pmd, unsigned long addr, unsigned long end, struct mm_walk *walk) { - pmd_t pmdval = pmd_read_atomic(pmd); + pmd_t pmdval = pmdp_get_lockless(pmd); if (!pmd_trans_unstable(&pmdval)) return 0; --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -292,7 +292,7 @@ static unsigned long change_pte_range(st */ static inline int pmd_none_or_clear_bad_unless_trans_huge(pmd_t *pmd) { - pmd_t pmdval = pmd_read_atomic(pmd); + pmd_t pmdval = pmdp_get_lockless(pmd); /* See pmd_none_or_trans_huge_or_clear_bad for info on barrier */ #ifdef CONFIG_TRANSPARENT_HUGEPAGE --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -613,7 +613,7 @@ static __always_inline ssize_t __mcopy_a break; } - dst_pmdval = pmd_read_atomic(dst_pmd); + dst_pmdval = pmdp_get_lockless(dst_pmd); /* * If the dst_pmd is mapped as THP don't * override it and just be strict. --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4039,9 +4039,9 @@ static void walk_pmd_range(pud_t *pud, u /* walk_pte_range() may call get_next_vma() */ vma = args->vma; for (i = pmd_index(start), addr = start; addr != end; i++, addr = next) { - pmd_t val = pmd_read_atomic(pmd + i); + pmd_t val = pmdp_get_lockless(pmd + i); - /* for pmd_read_atomic() */ + /* for pmdp_get_lockless() */ barrier(); next = pmd_addr_end(addr, end); From patchwork Sat Oct 22 11:14:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13015941 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE744C433FE for ; Sat, 22 Oct 2022 11:48:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 69F458E0001; Sat, 22 Oct 2022 07:48:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 39DA98E000B; Sat, 22 Oct 2022 07:48:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 154F08E0008; Sat, 22 Oct 2022 07:48:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D0FF18E000B for ; Sat, 22 Oct 2022 07:48:36 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id AEB28140791 for ; Sat, 22 Oct 2022 11:48:35 +0000 (UTC) X-FDA: 80048412990.13.466FD31 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) by imf03.hostedemail.com (Postfix) with ESMTP id 595E22001C for ; Sat, 22 Oct 2022 11:48:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=ZxLGyKaonAFGosmEJEXCJIuUc/p2pYnEnlwCFC//QMs=; b=d5RZJQiWmMamzGxaIKQnEOtUwD T9H/sWSomE2By/o8fhYFxcJnvZOZCOoyYsj4DPqrDTAEiVXHR6xyiOPOuV3LpQeQUbDuoAPcmVKGs DJSaXerbzdFXHI/CeMHGBW7cMxLNnTrKYaXjJZ3DmnuM+KA59Jo1ENMyp0WIE4NfWdcSJlN6hSA3v I0C+dfhtEKU3yRXt+wguStxjqcWSysKILzObipzuaBHkU3xB0q+bVaL3s37Udbo8ewcAHquPYv+98 QYAVXxD+sKkG/ZpZypq3uRMpr18Br1YlM8YBXYyqaiaLhGv+xeybW3Binc8LI7NIXuDe907gPs0vd qMBpHPRw==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1omCzM-005XdG-1V; Sat, 22 Oct 2022 11:48:28 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id BDB6F302D82; Sat, 22 Oct 2022 13:48:26 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 33F9C28B8E514; Sat, 22 Oct 2022 13:48:26 +0200 (CEST) Message-ID: <20221022114424.906110403@infradead.org> User-Agent: quilt/0.66 Date: Sat, 22 Oct 2022 13:14:10 +0200 From: Peter Zijlstra To: x86@kernel.org, willy@infradead.org, torvalds@linux-foundation.org, akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, linux-mm@kvack.org, aarcange@redhat.com, kirill.shutemov@linux.intel.com, jroedel@suse.de, ubizjak@gmail.com Subject: [PATCH 07/13] mm/gup: Fix the lockless PMD access References: <20221022111403.531902164@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1666439315; a=rsa-sha256; cv=none; b=aNhdWmCdfYIb3/k0+J7QjGj2ZRhI/5ssIlOAqT6BQguch7iNK/F1TdIw5TCQ05B/AZnW4j WdIPwWS0rFduNaqlaCM5C23cyAgpJIY6jGp3hKVfYI1+1NNqfrTHvUoQe1EfyrGnIR6GMY yPsQ6ERPdG0z+dMxbzmwuKpNFcx18k8= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=d5RZJQiW; spf=none (imf03.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1666439315; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=ZxLGyKaonAFGosmEJEXCJIuUc/p2pYnEnlwCFC//QMs=; b=7W5AM1QWokhltqV0tBVRPzqMrIaiA+LQ2aE8qo5LmEYqd5eoHH/aEfNIr58E2cQ1eikz20 jxNHyKoqxe47LpxUpCkdE2M75kD5zY3r1xiG+WoQYwMkDyeRYpyELzWiiJlQ4NASfSPz8q okEPEjDkqSFLnz1PTh/kOZqfI3N4tFE= X-Stat-Signature: arngc5t14decdrsbfx8p9cqzxmontxzs X-Rspamd-Queue-Id: 595E22001C X-Rspam-User: X-Rspamd-Server: rspam03 Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=d5RZJQiW; spf=none (imf03.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none X-HE-Tag: 1666439314-82807 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On architectures where the PTE/PMD is larger than the native word size (i386-PAE for example), READ_ONCE() can do the wrong thing. Use pmdp_get_lockless() just like we use ptep_get_lockless(). Signed-off-by: Peter Zijlstra (Intel) --- kernel/events/core.c | 2 +- mm/gup.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -7186,7 +7186,7 @@ static u64 perf_get_pgtable_size(struct return pud_leaf_size(pud); pmdp = pmd_offset_lockless(pudp, pud, addr); - pmd = READ_ONCE(*pmdp); + pmd = pmdp_get_lockless(pmdp); if (!pmd_present(pmd)) return 0; --- a/mm/gup.c +++ b/mm/gup.c @@ -2507,7 +2507,7 @@ static int gup_pmd_range(pud_t *pudp, pu pmdp = pmd_offset_lockless(pudp, pud, addr); do { - pmd_t pmd = READ_ONCE(*pmdp); + pmd_t pmd = pmdp_get_lockless(pmdp); next = pmd_addr_end(addr, end); if (!pmd_present(pmd)) From patchwork Sat Oct 22 11:14:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13015944 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 562C9C3A59D for ; Sat, 22 Oct 2022 11:48:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2D9748E000E; Sat, 22 Oct 2022 07:48:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 23A7A8E000A; Sat, 22 Oct 2022 07:48:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 040688E000E; Sat, 22 Oct 2022 07:48:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E5B9F8E000A for ; Sat, 22 Oct 2022 07:48:42 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id C9512140129 for ; Sat, 22 Oct 2022 11:48:42 +0000 (UTC) X-FDA: 80048413284.14.84D940A Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf17.hostedemail.com (Postfix) with ESMTP id 6C64F4001C for ; Sat, 22 Oct 2022 11:48:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=rlvYdOzEA1PYmv3xpXNUozerfLc4suXD0vdv08sNG6k=; b=gfnL7eWmkMJzCbyJLYVMNiLqdN Yy/XAKxgWJ1eol3mJOSAcMtqvfGvqoxIJjpKemQp81MaKf1+ntLPHeb8Tem7q/EMim9uN0vxFMDUB eXbMhDOcxXURog4p2Dz9JSV7iQMaS+NvXu2N+TIdLF2D1tj3LS6LRHUMXcL2NpUST9QLYcdvQ+L6m 0tSzURodRdHPX/bvpcDRKMHpJhdmSmNMA8PjoBlbBQTsxsSBpfIXNc3inYvAKWS+p0g8i/fZ3WPsc CtjByrke9HLUT+6Qs8yFq1wAEjWRrJtLKO2twxmWKmHgpKCQcsNt+RZg5B7IuCgeZG9FXhnWrtHSL GKlrjMlQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1omCzR-00Dtgt-IR; Sat, 22 Oct 2022 11:48:33 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id BDB42302D80; Sat, 22 Oct 2022 13:48:26 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 37BBE28B8E515; Sat, 22 Oct 2022 13:48:26 +0200 (CEST) Message-ID: <20221022114424.971450128@infradead.org> User-Agent: quilt/0.66 Date: Sat, 22 Oct 2022 13:14:11 +0200 From: Peter Zijlstra To: x86@kernel.org, willy@infradead.org, torvalds@linux-foundation.org, akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, linux-mm@kvack.org, aarcange@redhat.com, kirill.shutemov@linux.intel.com, jroedel@suse.de, ubizjak@gmail.com Subject: [PATCH 08/13] x86/mm/pae: Dont (ab)use atomic64 References: <20221022111403.531902164@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1666439322; a=rsa-sha256; cv=none; b=u7Oix2LzFAVL6bXjvh3GXIue32T8ZrJ78XjUG68/5jFDkHEiBCBp6ZSOema0YNAlLBkPyZ UZhj0zRnE69+lEneP7+7s6bcWzi5upjkE5hUVTbZ7vzWFBoGpBS5hkbHtr0FSizpuYDBBV yhX2vEMNrISbIwkrff8RV5csZ8NG8zA= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=gfnL7eWm; dmarc=none; spf=none (imf17.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1666439322; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=rlvYdOzEA1PYmv3xpXNUozerfLc4suXD0vdv08sNG6k=; b=zsrFkTtuJkuqh1iRgN36Q65y1IZCdnhoRZgvB8cawdWqBz81a1My0UWSjYVrfRFeckJa8x 8HB5xKoR+m04EGif0Ia1a32nbIfKtQzIHzjuTnbKPyoIb7mpo6bNWQIj719hlAIWiBTdTc 05suXBr2N6o/Q9tbHGpECysKqvCuurc= Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=gfnL7eWm; dmarc=none; spf=none (imf17.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org X-Rspamd-Server: rspam04 X-Rspam-User: X-Stat-Signature: ino8uwuit8fk1jgu61u7uw9h5jrr3m7m X-Rspamd-Queue-Id: 6C64F4001C X-HE-Tag: 1666439322-329997 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: PAE implies CX8, write readable code. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/pgtable-3level.h | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) --- a/arch/x86/include/asm/pgtable-3level.h +++ b/arch/x86/include/asm/pgtable-3level.h @@ -2,8 +2,6 @@ #ifndef _ASM_X86_PGTABLE_3LEVEL_H #define _ASM_X86_PGTABLE_3LEVEL_H -#include - /* * Intel Physical Address Extension (PAE) Mode - three-level page * tables on PPro+ CPUs. @@ -95,11 +93,12 @@ static inline void pud_clear(pud_t *pudp #ifdef CONFIG_SMP static inline pte_t native_ptep_get_and_clear(pte_t *ptep) { - pte_t res; + pte_t old = *ptep; - res.pte = (pteval_t)arch_atomic64_xchg((atomic64_t *)ptep, 0); + do { + } while (!try_cmpxchg64(&ptep->pte, &old.pte, 0ULL)); - return res; + return old; } #else #define native_ptep_get_and_clear(xp) native_local_ptep_get_and_clear(xp) From patchwork Sat Oct 22 11:14:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13015942 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6E0DFA373D for ; Sat, 22 Oct 2022 11:48:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 84B618E000B; Sat, 22 Oct 2022 07:48:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6377A8E000A; Sat, 22 Oct 2022 07:48:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1ECD98E0001; Sat, 22 Oct 2022 07:48:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id D82D48E0005 for ; Sat, 22 Oct 2022 07:48:36 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id B7BDFA08A4 for ; Sat, 22 Oct 2022 11:48:35 +0000 (UTC) X-FDA: 80048412990.11.E334195 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) by imf27.hostedemail.com (Postfix) with ESMTP id 4BD4040002 for ; Sat, 22 Oct 2022 11:48:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=21hb22ZOLPug1Y7rU1MItuB/yDHg1vIxzPuc1LZH5hI=; b=PdKGQbC12QBugaGBHXI+1OdOwW jIeT7sZU8f/cCzbk9SkQJaFP1b540Ku1IBJuuSIHSLzg9x2w8Wmc8o276uAHPOpXzXLG3d6TZq0tF ufch0SOHGlt6rJ9GjVJv7If3PI2CCLHZEcs/plOsipgGDnY0U9T2MARB4MNF2md4sW8rQWFDyYM2z ftKN36vAn858VDWdBe2+QF71E9pc0xQfcgKilAvXMfbC3SKqEsFNp+XKGyHGw+FG2kU27DviLQ2k2 Zgve+Gbb5vgj55/AffmFiNzVSPKfRJE8lA9PCBRfZ7JGWZYdwkJx6h4LWZv/Y1jswpVYaPfpTSW4b 7CjP2Teg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1omCzM-005XdH-1s; Sat, 22 Oct 2022 11:48:28 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id BDB8D302D91; Sat, 22 Oct 2022 13:48:26 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 3B49F28B8E516; Sat, 22 Oct 2022 13:48:26 +0200 (CEST) Message-ID: <20221022114425.038102604@infradead.org> User-Agent: quilt/0.66 Date: Sat, 22 Oct 2022 13:14:12 +0200 From: Peter Zijlstra To: x86@kernel.org, willy@infradead.org, torvalds@linux-foundation.org, akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, linux-mm@kvack.org, aarcange@redhat.com, kirill.shutemov@linux.intel.com, jroedel@suse.de, ubizjak@gmail.com Subject: [PATCH 09/13] x86/mm/pae: Use WRITE_ONCE() References: <20221022111403.531902164@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1666439315; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=21hb22ZOLPug1Y7rU1MItuB/yDHg1vIxzPuc1LZH5hI=; b=407ouvL8E43uAR6P+QpH09pSkU0Uczmu5r3HKKc+SAXkMg5DpeId4wctp/52U62IaAsQgy Z4Ab+vWxScjYlRqLhoiXwc578d+tjZ0OUIUi5qvgwCeetnJQQe9jmFsorwq7RLVQU9hlPv kBbuFR1Koiv23rgGDYysgH3ueQeSQ+k= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=PdKGQbC1; spf=none (imf27.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1666439315; a=rsa-sha256; cv=none; b=wr+p+R0G9DILJpWUe6BCGTL9BB13Cy0Ph2Y1XCZhDeWpneAfQoZgKekJsAblluf7HvPj3s ZIh4erRqN2QWiiCk2bzpEJnLpZT3OXpo1svT3n0zA85V8G93Kv1tMzYS8juH2/WKHjGjjf VbCf6SnO9/kJ4QpQP+y1HxHZj0o9AtY= X-Stat-Signature: u8hwhbat8nhmgg9qnrety589egx6wpfd X-Rspam-User: Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=PdKGQbC1; spf=none (imf27.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 4BD4040002 X-HE-Tag: 1666439314-709641 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Disallow write-tearing, that would be really unfortunate. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/pgtable-3level.h | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) --- a/arch/x86/include/asm/pgtable-3level.h +++ b/arch/x86/include/asm/pgtable-3level.h @@ -27,9 +27,9 @@ */ static inline void native_set_pte(pte_t *ptep, pte_t pte) { - ptep->pte_high = pte.pte_high; + WRITE_ONCE(ptep->pte_high, pte.pte_high); smp_wmb(); - ptep->pte_low = pte.pte_low; + WRITE_ONCE(ptep->pte_low, pte.pte_low); } static inline void native_set_pte_atomic(pte_t *ptep, pte_t pte) @@ -58,16 +58,16 @@ static inline void native_set_pud(pud_t static inline void native_pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { - ptep->pte_low = 0; + WRITE_ONCE(ptep->pte_low, 0); smp_wmb(); - ptep->pte_high = 0; + WRITE_ONCE(ptep->pte_high, 0); } static inline void native_pmd_clear(pmd_t *pmdp) { - pmdp->pmd_low = 0; + WRITE_ONCE(pmdp->pmd_low, 0); smp_wmb(); - pmdp->pmd_high = 0; + WRITE_ONCE(pmdp->pmd_high, 0); } static inline void native_pud_clear(pud_t *pudp) From patchwork Sat Oct 22 11:14:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13015956 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D07AC04A95 for ; Sat, 22 Oct 2022 12:38:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 17A888E0003; Sat, 22 Oct 2022 08:38:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 103CB8E0001; Sat, 22 Oct 2022 08:38:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E709C8E0003; Sat, 22 Oct 2022 08:38:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D7C438E0001 for ; Sat, 22 Oct 2022 08:38:11 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id B27061C5F27 for ; Sat, 22 Oct 2022 12:38:11 +0000 (UTC) X-FDA: 80048537982.16.5467DA2 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) by imf16.hostedemail.com (Postfix) with ESMTP id 2D3A4180033 for ; Sat, 22 Oct 2022 12:38:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=ooIZw4CypGPCb0zPum6oOMgggH2beVobr6aR5c3g6lI=; b=K1NZC28rbCpDx1lOpuyqqeOlu0 u3HYYfJW0hm7LNd8SPY5tyqEMpHWPxr11iKs9l58bdoPnVU6wHHL2BsM1n5/JdXsM0nZfADRbHFqz v8V6qBdgbc9YK9ivVKEpDExRf6QDkCdPgacqCcnSoo9/LIGBwfOTjWoso36OcEgniZ48URn3tfWKv 9W3APtgZ2AOVMvmrrjyx1tGJ6tYWebVeBHPYVBttdim6OSimso7Cn3gFaTTlAa6QuNq13n8bEp1WD JUaT8tW6KC84dQ4vZgwnBJuYXBP8WtAmVqc1uEfDt3bFBi0IyCgb1WGcB4XpIzmQjGYlCvhnqr0X6 FlQvWLug==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1omCzM-005XdS-Vs; Sat, 22 Oct 2022 11:48:29 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id C79DA302E97; Sat, 22 Oct 2022 13:48:26 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 3EEA728B8E517; Sat, 22 Oct 2022 13:48:26 +0200 (CEST) Message-ID: <20221022114425.103392961@infradead.org> User-Agent: quilt/0.66 Date: Sat, 22 Oct 2022 13:14:13 +0200 From: Peter Zijlstra To: x86@kernel.org, willy@infradead.org, torvalds@linux-foundation.org, akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, linux-mm@kvack.org, aarcange@redhat.com, kirill.shutemov@linux.intel.com, jroedel@suse.de, ubizjak@gmail.com Subject: [PATCH 10/13] x86/mm/pae: Be consistent with pXXp_get_and_clear() References: <20221022111403.531902164@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1666442291; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=ooIZw4CypGPCb0zPum6oOMgggH2beVobr6aR5c3g6lI=; b=CE5fIJ6+p7lD6tjRDurtr8IRo1XwL3L6WDtrefV7SMbANZYcyilH4FD20rotsxHJX4u7ZX KWbXE0PPwja3IztsG6AxnWwly18PG69bKpYY6qqnytC3zHo7fvL2iIsfU+arKb8CBUk0Cq yK39NkcEtUqRQvNc84E5kaLl9d+xRrU= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=K1NZC28r; spf=none (imf16.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1666442291; a=rsa-sha256; cv=none; b=UuBZaRKXO2RvnRELfk9f9aqAUIfN1gs4HfT+1yQ8SZoiHgQrAOiQzmyxSaYdygGgftZFTz wgMX6wyjt4KJEvpa8ElLfI1oE/lyKP6JFwldxhAMSk/i/FwdeqZKnrszQQy3s9A7Z4wcap bVg0SiLZmWM8FfJbbJY9Ns/y3NtEUoM= X-Stat-Signature: tsod4k6i1fp84xo4rpdxwh59r3kkx56b X-Rspam-User: Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=K1NZC28r; spf=none (imf16.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 2D3A4180033 X-HE-Tag: 1666442290-984239 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Given that ptep_get_and_clear() uses cmpxchg8b, and that should be by far the most common case, there's no point in having an optimized variant for pmd/pud. Introduce the pxx_xchg64() helper to implement the common logic once. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/pgtable-3level.h | 67 ++++++++-------------------------- 1 file changed, 17 insertions(+), 50 deletions(-) --- a/arch/x86/include/asm/pgtable-3level.h +++ b/arch/x86/include/asm/pgtable-3level.h @@ -90,34 +90,33 @@ static inline void pud_clear(pud_t *pudp */ } + +#define pxx_xchg64(_pxx, _ptr, _val) ({ \ + _pxx##val_t *_p = (_pxx##val_t *)_ptr; \ + _pxx##val_t _o = *_p; \ + do { } while (!try_cmpxchg64(_p, &_o, (_val))); \ + native_make_##_pxx(_o); \ +}) + #ifdef CONFIG_SMP static inline pte_t native_ptep_get_and_clear(pte_t *ptep) { - pte_t old = *ptep; - - do { - } while (!try_cmpxchg64(&ptep->pte, &old.pte, 0ULL)); - - return old; + return pxx_xchg64(pte, ptep, 0ULL); } -#else -#define native_ptep_get_and_clear(xp) native_local_ptep_get_and_clear(xp) -#endif -#ifdef CONFIG_SMP static inline pmd_t native_pmdp_get_and_clear(pmd_t *pmdp) { - pmd_t res; - - /* xchg acts as a barrier before setting of the high bits */ - res.pmd_low = xchg(&pmdp->pmd_low, 0); - res.pmd_high = READ_ONCE(pmdp->pmd_high); - WRITE_ONCE(pmdp->pmd_high, 0); + return pxx_xchg64(pmd, pmdp, 0ULL); +} - return res; +static inline pud_t native_pudp_get_and_clear(pud_t *pudp) +{ + return pxx_xchg64(pud, pudp, 0ULL); } #else +#define native_ptep_get_and_clear(xp) native_local_ptep_get_and_clear(xp) #define native_pmdp_get_and_clear(xp) native_local_pmdp_get_and_clear(xp) +#define native_pudp_get_and_clear(xp) native_local_pudp_get_and_clear(xp) #endif #ifndef pmdp_establish @@ -141,40 +140,8 @@ static inline pmd_t pmdp_establish(struc return old; } - do { - old = *pmdp; - } while (cmpxchg64(&pmdp->pmd, old.pmd, pmd.pmd) != old.pmd); - - return old; -} -#endif - -#ifdef CONFIG_SMP -union split_pud { - struct { - u32 pud_low; - u32 pud_high; - }; - pud_t pud; -}; - -static inline pud_t native_pudp_get_and_clear(pud_t *pudp) -{ - union split_pud res, *orig = (union split_pud *)pudp; - -#ifdef CONFIG_PAGE_TABLE_ISOLATION - pti_set_user_pgtbl(&pudp->p4d.pgd, __pgd(0)); -#endif - - /* xchg acts as a barrier before setting of the high bits */ - res.pud_low = xchg(&orig->pud_low, 0); - res.pud_high = orig->pud_high; - orig->pud_high = 0; - - return res.pud; + return pxx_xchg64(pmd, pmdp, pmd.pmd); } -#else -#define native_pudp_get_and_clear(xp) native_local_pudp_get_and_clear(xp) #endif /* Encode and de-code a swap entry */ From patchwork Sat Oct 22 11:14:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13015955 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96DA8C433FE for ; Sat, 22 Oct 2022 12:38:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EE7A28E0002; Sat, 22 Oct 2022 08:38:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E973C8E0001; Sat, 22 Oct 2022 08:38:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D85EC8E0002; Sat, 22 Oct 2022 08:38:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id C86E78E0001 for ; Sat, 22 Oct 2022 08:38:10 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 9E2721A0583 for ; Sat, 22 Oct 2022 12:38:10 +0000 (UTC) X-FDA: 80048537940.23.2255D7E Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) by imf16.hostedemail.com (Postfix) with ESMTP id B24CF180031 for ; Sat, 22 Oct 2022 12:38:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=kfJUOBrrmjy4JsdmvOUplpu2MGBZGnH8F3TsabZggcA=; b=TZtejNuXM7kl3MOXWKy8aaU/tk e118lDcxQsCXKrtcjfNaX3hhzfxtkXwSWz763aZgaWRoymxnOariKlXTCOTFQ1zOXOH9zc1OTUFRv 8azjLpfaOOQEKL6SRwBny2sFE0p0l69+LTKxpFmQ79VJ+ySU5VFdBl2ClPmUNW6KfJP7IiEb2xjC9 tsNtJRffbNAX/xOBgv9twdHzjXxpXTWw7dPoKQlb47QCW08TuhULthxwrfwPLHr8YayH5Eek3Idiz D6G/kMsRd5f8hFWCtPSwW5VwwMGGMZa4EtEoRiAGDeP8Ppb2+SjV6sDe0sQYqyLmn55ceyQHrM481 m9ajaoKQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1omCzM-005XdT-Vx; Sat, 22 Oct 2022 11:48:29 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id C85DC3030FD; Sat, 22 Oct 2022 13:48:26 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 4299A28B8E512; Sat, 22 Oct 2022 13:48:26 +0200 (CEST) Message-ID: <20221022114425.168036718@infradead.org> User-Agent: quilt/0.66 Date: Sat, 22 Oct 2022 13:14:14 +0200 From: Peter Zijlstra To: x86@kernel.org, willy@infradead.org, torvalds@linux-foundation.org, akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, linux-mm@kvack.org, aarcange@redhat.com, kirill.shutemov@linux.intel.com, jroedel@suse.de, ubizjak@gmail.com Subject: [PATCH 11/13] x86_64: Remove pointless set_64bit() usage References: <20221022111403.531902164@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1666442290; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=kfJUOBrrmjy4JsdmvOUplpu2MGBZGnH8F3TsabZggcA=; b=XLmkJsWP7C5xiGa1WPOBNVFqVaLEY4A1L7npaTD3EpG/6NMNMK7LgnQFq1Wyn5ZouBJma2 laHrp5FTe5RooLnDBqBhjnKAqRLF8NPG4lMF4s6Iu4WU3EC/eGlJxEin1v819INBwxncId ulDdMbdPHP3b5maqud2Nxbv965yvaK0= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=TZtejNuX; spf=none (imf16.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1666442290; a=rsa-sha256; cv=none; b=e55N+PLXWyKp5ahFG8KZv74sEAqBr6k1tjGpR5LlabumXRNxSUNgpU8DUKOC0naFyZyA0z 2TylFraBhqslZbw0ZaAUfN2Ckc3EFYz10pdKQdC4tI1lKw8WmTVXQCtaIhegSPdY624wmY WGNZA4K1H8OAp1JZ6rA1LWOSH7s5QyE= X-Stat-Signature: hfatq6upqen5ju7sca93scg4qq3tqggu X-Rspam-User: Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=TZtejNuX; spf=none (imf16.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: B24CF180031 X-HE-Tag: 1666442289-812412 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The use of set_64bit() in X86_64 only code is pretty pointless, seeing how it's a direct assignment. Remove all this nonsense. Additionally, since x86_64 unconditionally has HAVE_CMPXCHG_DOUBLE, there is no point in even having that fallback. Signed-off-by: Peter Zijlstra (Intel) --- arch/um/include/asm/pgtable-3level.h | 8 -------- arch/x86/include/asm/cmpxchg_64.h | 5 ----- drivers/iommu/intel/irq_remapping.c | 10 ++-------- 3 files changed, 2 insertions(+), 21 deletions(-) --- a/arch/um/include/asm/pgtable-3level.h +++ b/arch/um/include/asm/pgtable-3level.h @@ -58,11 +58,7 @@ #define pud_populate(mm, pud, pmd) \ set_pud(pud, __pud(_PAGE_TABLE + __pa(pmd))) -#ifdef CONFIG_64BIT -#define set_pud(pudptr, pudval) set_64bit((u64 *) (pudptr), pud_val(pudval)) -#else #define set_pud(pudptr, pudval) (*(pudptr) = (pudval)) -#endif static inline int pgd_newpage(pgd_t pgd) { @@ -71,11 +67,7 @@ static inline int pgd_newpage(pgd_t pgd) static inline void pgd_mkuptodate(pgd_t pgd) { pgd_val(pgd) &= ~_PAGE_NEWPAGE; } -#ifdef CONFIG_64BIT -#define set_pmd(pmdptr, pmdval) set_64bit((u64 *) (pmdptr), pmd_val(pmdval)) -#else #define set_pmd(pmdptr, pmdval) (*(pmdptr) = (pmdval)) -#endif static inline void pud_clear (pud_t *pud) { --- a/arch/x86/include/asm/cmpxchg_64.h +++ b/arch/x86/include/asm/cmpxchg_64.h @@ -2,11 +2,6 @@ #ifndef _ASM_X86_CMPXCHG_64_H #define _ASM_X86_CMPXCHG_64_H -static inline void set_64bit(volatile u64 *ptr, u64 val) -{ - *ptr = val; -} - #define arch_cmpxchg64(ptr, o, n) \ ({ \ BUILD_BUG_ON(sizeof(*(ptr)) != 8); \ --- a/drivers/iommu/intel/irq_remapping.c +++ b/drivers/iommu/intel/irq_remapping.c @@ -173,7 +173,6 @@ static int modify_irte(struct irq_2_iomm index = irq_iommu->irte_index + irq_iommu->sub_handle; irte = &iommu->ir_table->base[index]; -#if defined(CONFIG_HAVE_CMPXCHG_DOUBLE) if ((irte->pst == 1) || (irte_modified->pst == 1)) { bool ret; @@ -187,11 +186,6 @@ static int modify_irte(struct irq_2_iomm * same as the old value. */ WARN_ON(!ret); - } else -#endif - { - set_64bit(&irte->low, irte_modified->low); - set_64bit(&irte->high, irte_modified->high); } __iommu_flush_cache(iommu, irte, sizeof(*irte)); @@ -249,8 +243,8 @@ static int clear_entries(struct irq_2_io end = start + (1 << irq_iommu->irte_mask); for (entry = start; entry < end; entry++) { - set_64bit(&entry->low, 0); - set_64bit(&entry->high, 0); + WRITE_ONCE(entry->low, 0); + WRITE_ONCE(entry->high, 0); } bitmap_release_region(iommu->ir_table->bitmap, index, irq_iommu->irte_mask); From patchwork Sat Oct 22 11:14:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13015940 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7EE02C04A95 for ; Sat, 22 Oct 2022 11:48:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 461E28E0008; Sat, 22 Oct 2022 07:48:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 065728E000D; Sat, 22 Oct 2022 07:48:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CECE38E0008; Sat, 22 Oct 2022 07:48:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 800F08E0007 for ; Sat, 22 Oct 2022 07:48:36 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 61D901C3042 for ; Sat, 22 Oct 2022 11:48:36 +0000 (UTC) X-FDA: 80048413032.06.54E08D2 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf23.hostedemail.com (Postfix) with ESMTP id 0BAAD140002 for ; Sat, 22 Oct 2022 11:48:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=vmo888klJOYTK6NXdYxj1ylLTVw6+U7xUyde2QpBL2o=; b=bYr1reyz4HyDuluM2y+c4SgJM0 PBJN8uYkZxEcRHT3GePK91yBuUwfsfegkY9kOgusA3DzeACZX107P9rwRsGS+B5vmcl2jCFpRqPPq dY+JhH8JW06eGjLRNrjQfsUO/3WvAUW0Mnr0U8/5MItJ1KB06PnuMLz9AKqkVjcZ+3ImKa4B4PPbx MXfbYcCiAe9q7O28LJ1ssPhDG1tuqtRmAMRil+8BvHeMkUevTn5g569pTJJaTks3FEbOAnHuEgDx5 13zre+JP7+tEWJSrresAW1tKhUL7Mnjcjod2QJFzkI1sVeA4Id2JzSbvBWj8QCFumXkoCwc6YQj9L ka4hjhhw==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1omCzR-00Dtgv-LC; Sat, 22 Oct 2022 11:48:33 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id CA46F303106; Sat, 22 Oct 2022 13:48:26 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 46C8E28B8E519; Sat, 22 Oct 2022 13:48:26 +0200 (CEST) Message-ID: <20221022114425.233481884@infradead.org> User-Agent: quilt/0.66 Date: Sat, 22 Oct 2022 13:14:15 +0200 From: Peter Zijlstra To: x86@kernel.org, willy@infradead.org, torvalds@linux-foundation.org, akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, linux-mm@kvack.org, aarcange@redhat.com, kirill.shutemov@linux.intel.com, jroedel@suse.de, ubizjak@gmail.com Subject: [PATCH 12/13] x86/mm/pae: Get rid of set_64bit() References: <20221022111403.531902164@infradead.org> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=bYr1reyz; spf=none (imf23.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1666439316; a=rsa-sha256; cv=none; b=F2se0EfL/NPKsYwjfWUfNTqSeKfYxioasNqfzdtLKhu7zgVZvBReI+7TUnsMU7tVqoMyPQ aIpDWNjTW5p/SnYWySkWEogvcpgGJ2fJv3/PD06uOtp866vDCK6pPKQ8OetW3ULB2fBcR0 7OT2R+z+uIOey9jlyqhjV5ZnV8m8Tb0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1666439316; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=vmo888klJOYTK6NXdYxj1ylLTVw6+U7xUyde2QpBL2o=; b=uCWH19SKbxFOaZdLRVWf7wnSFeLYLszwi9ly6UgE/BrucOBqjCTN5fgAfSIZN9SJP+GELG a2Z5ywjGh4SdNk/HvLb4lSalmcrzDzK2DVCESHy5zjUgNyBfxquStWshaVGmam+iYqE4R/ zD+IpSzKiiQ07E6a8CQaU/L4zrdmV/A= Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=bYr1reyz; spf=none (imf23.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org; dmarc=none X-Stat-Signature: 7ic5fugjj5hstgpmdx9xgc3oikhx8rpu X-Rspamd-Queue-Id: 0BAAD140002 X-Rspamd-Server: rspam02 X-Rspam-User: X-HE-Tag: 1666439315-212092 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Recognise that set_64bit() is a special case of our previously introduced pxx_xchg64(), so use that and get rid of set_64bit(). Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/cmpxchg_32.h | 28 ---------------------------- arch/x86/include/asm/pgtable-3level.h | 23 ++++++++++++----------- 2 files changed, 12 insertions(+), 39 deletions(-) --- a/arch/x86/include/asm/cmpxchg_32.h +++ b/arch/x86/include/asm/cmpxchg_32.h @@ -7,34 +7,6 @@ * you need to test for the feature in boot_cpu_data. */ -/* - * CMPXCHG8B only writes to the target if we had the previous - * value in registers, otherwise it acts as a read and gives us the - * "new previous" value. That is why there is a loop. Preloading - * EDX:EAX is a performance optimization: in the common case it means - * we need only one locked operation. - * - * A SIMD/3DNOW!/MMX/FPU 64-bit store here would require at the very - * least an FPU save and/or %cr0.ts manipulation. - * - * cmpxchg8b must be used with the lock prefix here to allow the - * instruction to be executed atomically. We need to have the reader - * side to see the coherent 64bit value. - */ -static inline void set_64bit(volatile u64 *ptr, u64 value) -{ - u32 low = value; - u32 high = value >> 32; - u64 prev = *ptr; - - asm volatile("\n1:\t" - LOCK_PREFIX "cmpxchg8b %0\n\t" - "jnz 1b" - : "=m" (*ptr), "+A" (prev) - : "b" (low), "c" (high) - : "memory"); -} - #ifdef CONFIG_X86_CMPXCHG64 #define arch_cmpxchg64(ptr, o, n) \ ((__typeof__(*(ptr)))__cmpxchg64((ptr), (unsigned long long)(o), \ --- a/arch/x86/include/asm/pgtable-3level.h +++ b/arch/x86/include/asm/pgtable-3level.h @@ -19,7 +19,15 @@ pr_err("%s:%d: bad pgd %p(%016Lx)\n", \ __FILE__, __LINE__, &(e), pgd_val(e)) -/* Rules for using set_pte: the pte being assigned *must* be +#define pxx_xchg64(_pxx, _ptr, _val) ({ \ + _pxx##val_t *_p = (_pxx##val_t *)_ptr; \ + _pxx##val_t _o = *_p; \ + do { } while (!try_cmpxchg64(_p, &_o, (_val))); \ + native_make_##_pxx(_o); \ +}) + +/* + * Rules for using set_pte: the pte being assigned *must* be * either not present or in a state where the hardware will * not attempt to update the pte. In places where this is * not possible, use pte_get_and_clear to obtain the old pte @@ -34,12 +42,12 @@ static inline void native_set_pte(pte_t static inline void native_set_pte_atomic(pte_t *ptep, pte_t pte) { - set_64bit((unsigned long long *)(ptep), native_pte_val(pte)); + pxx_xchg64(pte, ptep, native_pte_val(pte)); } static inline void native_set_pmd(pmd_t *pmdp, pmd_t pmd) { - set_64bit((unsigned long long *)(pmdp), native_pmd_val(pmd)); + pxx_xchg64(pmd, pmdp, native_pmd_val(pmd)); } static inline void native_set_pud(pud_t *pudp, pud_t pud) @@ -47,7 +55,7 @@ static inline void native_set_pud(pud_t #ifdef CONFIG_PAGE_TABLE_ISOLATION pud.p4d.pgd = pti_set_user_pgtbl(&pudp->p4d.pgd, pud.p4d.pgd); #endif - set_64bit((unsigned long long *)(pudp), native_pud_val(pud)); + pxx_xchg64(pud, pudp, native_pud_val(pud)); } /* @@ -91,13 +99,6 @@ static inline void pud_clear(pud_t *pudp } -#define pxx_xchg64(_pxx, _ptr, _val) ({ \ - _pxx##val_t *_p = (_pxx##val_t *)_ptr; \ - _pxx##val_t _o = *_p; \ - do { } while (!try_cmpxchg64(_p, &_o, (_val))); \ - native_make_##_pxx(_o); \ -}) - #ifdef CONFIG_SMP static inline pte_t native_ptep_get_and_clear(pte_t *ptep) { From patchwork Sat Oct 22 11:14:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13015937 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2EC76FA3741 for ; Sat, 22 Oct 2022 11:48:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1F4F98E0005; Sat, 22 Oct 2022 07:48:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E21C28E000A; Sat, 22 Oct 2022 07:48:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 917B28E0005; Sat, 22 Oct 2022 07:48:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 2ED828E0005 for ; Sat, 22 Oct 2022 07:48:36 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 124801A05FD for ; Sat, 22 Oct 2022 11:48:36 +0000 (UTC) X-FDA: 80048413032.08.052CAA0 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) by imf29.hostedemail.com (Postfix) with ESMTP id AAB4712003C for ; Sat, 22 Oct 2022 11:48:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=RY5OJVKm7PHKfP6ch20ecx7TX/MGByzkwaP5zQXVje4=; b=OJUNT+dm8Xm8G4HKPUJTT88+5b tS7rS4Xte2tScbrsFmj1YbzEc81Fk6njpuY3V7B/fQSSRiowYqw06hkfp4K7lil+cYoEilAlK626c skvA6gh56x7NxdcMn/+qHQbgcePeOyPqteTa64FhrQD2XdvJRo7DyDyAcFyvMhNpo/ScLaXybnWgd N35tVDujd87+4UpkzlXOj/vEoKlK+9YprxpUkKPWJwSBL6UY46tD9qx15+E/rcO3cvGY6EFN4XohB 8Ot3mLfMet0JZV34mz6DbGldZ+616MqADT10saDPFs+Rw1ii0z/QnTUVeBnufCvLBhWNZ13DQWN5F 7HXKoxeQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1omCzM-005XdU-WF; Sat, 22 Oct 2022 11:48:29 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id D6086303109; Sat, 22 Oct 2022 13:48:26 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 518E428B8E518; Sat, 22 Oct 2022 13:48:26 +0200 (CEST) Message-ID: <20221022114425.298833095@infradead.org> User-Agent: quilt/0.66 Date: Sat, 22 Oct 2022 13:14:16 +0200 From: Peter Zijlstra To: x86@kernel.org, willy@infradead.org, torvalds@linux-foundation.org, akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, linux-mm@kvack.org, aarcange@redhat.com, kirill.shutemov@linux.intel.com, jroedel@suse.de, ubizjak@gmail.com Subject: [PATCH 13/13] mm: Remove pointless barrier() after pmdp_get_lockless() References: <20221022111403.531902164@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1666439315; a=rsa-sha256; cv=none; b=I90MGemekGBu+wvUj3gjvDCMe6Q5rPNc3FDGpTZ3pUeSSBPn+xUu3GhClG6a8dWgS6ET9N hFXkmn/jf+8T/ax5bO4ofGQe71CTfQsk9j5HwLRTjwmpfX2PFZCp+HB4r2naOpWC6r9iAn 1zqdRWsO2DZZif6pvBNS1w/QWnmBOi4= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=OJUNT+dm; spf=none (imf29.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1666439315; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=RY5OJVKm7PHKfP6ch20ecx7TX/MGByzkwaP5zQXVje4=; b=udg+jq1G2ii0REWLKN6B/W62Hb0B1vkbXSVpAk5DWOUd8YYg2M/1e0d/TumwTIdgpVKbZz vId1z9ivIXbWdLH6gmzPraZitrz8YRa7SDxJOmtFQ8QJg21LGtuPb1Pad0BVy1DebmVYgF 4PA8G/jhbKu8A/uAHUokqF/N43OxaDo= Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=OJUNT+dm; spf=none (imf29.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none X-Stat-Signature: kqjtmuquwzd33u5nhtfbggyrmwmxixtw X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: AAB4712003C X-Rspam-User: X-HE-Tag: 1666439315-871478 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: pmdp_get_lockless() should itself imply any ordering required. Signed-off-by: Peter Zijlstra (Intel) --- mm/hmm.c | 1 - mm/vmscan.c | 3 --- 2 files changed, 4 deletions(-) --- a/mm/hmm.c +++ b/mm/hmm.c @@ -362,7 +362,6 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, * values. */ pmd = pmdp_get_lockless(pmdp); - barrier(); if (!pmd_devmap(pmd) && !pmd_trans_huge(pmd)) goto again; --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4041,9 +4041,6 @@ static void walk_pmd_range(pud_t *pud, u for (i = pmd_index(start), addr = start; addr != end; i++, addr = next) { pmd_t val = pmdp_get_lockless(pmd + i); - /* for pmdp_get_lockless() */ - barrier(); - next = pmd_addr_end(addr, end); if (!pmd_present(val) || is_huge_zero_pmd(val)) {