From patchwork Mon Nov 30 11:27:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 11940257 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A6A5C64E7B for ; Mon, 30 Nov 2020 11:38:49 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8732A206C1 for ; Mon, 30 Nov 2020 11:38:33 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="2/ThSzwG" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8732A206C1 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 238A78D0001; Mon, 30 Nov 2020 06:38:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E1E698D0005; Mon, 30 Nov 2020 06:38:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B573F8D0008; Mon, 30 Nov 2020 06:38:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0004.hostedemail.com [216.40.44.4]) by kanga.kvack.org (Postfix) with ESMTP id 8DA958D0005 for ; Mon, 30 Nov 2020 06:38:25 -0500 (EST) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 50DE51EE6 for ; Mon, 30 Nov 2020 11:38:25 +0000 (UTC) X-FDA: 77540886570.12.train84_5f14ea4273a1 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin12.hostedemail.com (Postfix) with ESMTP id 215AD1801F0F0 for ; Mon, 30 Nov 2020 11:38:25 +0000 (UTC) X-HE-Tag: train84_5f14ea4273a1 X-Filterd-Recvd-Size: 3875 Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) by imf03.hostedemail.com (Postfix) with ESMTP for ; Mon, 30 Nov 2020 11:38:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=merlin.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=p4owENHTSGpOxSLxjnv01cC/b4aSY2KNSrKXKhQFpp0=; b=2/ThSzwGx8iKGWVBXcvf+IhJ95 tu7jvyTICS7JwwwSKX20c35Lpo6UlFWfAqTGCKkGGcO7izqEDpnTa3kioX8M1La2C2NwzGmtXHVnA bDptqOsODjFjZKrt4I/NY0vtaY8OZwAffyjLnPMKywcT5wJCwM5YhBkBgpx3tO8DHoAIEUTeE5xnx Jb9WaHWZYpt00qnD7JQMbSOPypOIXGgUd3NivAB4VHk3LuCH1venb1emG+bYDo2tPvmyuYjYrrVsy m5HLED4lbjB+Uq+TUbbkVr5jCVPTPxtStatP2fMSkYgDVIxzshiZSUq+vCHbGeIToei25DPYkYccD qoAKEj2w==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=noisy.programming.kicks-ass.net) by merlin.infradead.org with esmtpsa (Exim 4.92.3 #3 (Red Hat Linux)) id 1kjhVc-0006CV-Gb; Mon, 30 Nov 2020 11:38:20 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id E3C19302753; Mon, 30 Nov 2020 12:38:17 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id D10902B84F9AE; Mon, 30 Nov 2020 12:38:17 +0100 (CET) Message-ID: <20201130113602.841001656@infradead.org> User-Agent: quilt/0.66 Date: Mon, 30 Nov 2020 12:27:06 +0100 From: Peter Zijlstra To: x86@kernel.org, willy@infradead.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, aarcange@redhat.com, kirill.shutemov@linux.intel.com, jroedel@suse.de, peterz@infradead.org Subject: [RFC][PATCH 1/9] mm: Update ptep_get_lockless()s comment References: <20201130112705.900705277@infradead.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Improve the comment and add a lockdep assertion because nobody reads comments. Suggested-by: Matthew Wilcox Signed-off-by: Peter Zijlstra (Intel) --- include/linux/pgtable.h | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -260,15 +260,12 @@ static inline pte_t ptep_get(pte_t *ptep #ifdef CONFIG_GUP_GET_PTE_LOW_HIGH /* - * WARNING: only to be used in the get_user_pages_fast() implementation. - * - * With get_user_pages_fast(), we walk down the pagetables without taking any - * locks. For this we would like to load the pointers atomically, but sometimes - * that is not possible (e.g. without expensive cmpxchg8b on x86_32 PAE). What - * we do have is the guarantee that a PTE will only either go from not present - * to present, or present to not present or both -- it will not switch to a - * completely different present page without a TLB flush in between; something - * that we are blocking by holding interrupts off. + * For walking the pagetables without holding any locks. Some architectures + * (eg x86-32 PAE) cannot load the entries atomically without using expensive + * instructions. We are guaranteed that a PTE will only either go from not + * present to present, or present to not present -- it will not switch to a + * completely different present page without a TLB flush inbetween; which we + * are blocking by holding interrupts off. * * Setting ptes from not present to present goes: * @@ -294,6 +291,8 @@ static inline pte_t ptep_get_lockless(pt { pte_t pte; + lockdep_assert_irqs_disabled(); + do { pte.pte_low = ptep->pte_low; smp_rmb(); From patchwork Mon Nov 30 11:27:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 11940247 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E5BCC64E90 for ; Mon, 30 Nov 2020 11:38:28 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8D16A20870 for ; Mon, 30 Nov 2020 11:38:26 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="nVErVX4k" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8D16A20870 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4C7608D0006; Mon, 30 Nov 2020 06:38:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4866C8D0005; Mon, 30 Nov 2020 06:38:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 33D688D0005; Mon, 30 Nov 2020 06:38:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0190.hostedemail.com [216.40.44.190]) by kanga.kvack.org (Postfix) with ESMTP id 1CAEB8D0001 for ; Mon, 30 Nov 2020 06:38:25 -0500 (EST) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id DD6283634 for ; Mon, 30 Nov 2020 11:38:24 +0000 (UTC) X-FDA: 77540886528.25.fight19_251476a273a1 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin25.hostedemail.com (Postfix) with ESMTP id B984F1804E3BC for ; Mon, 30 Nov 2020 11:38:24 +0000 (UTC) X-HE-Tag: fight19_251476a273a1 X-Filterd-Recvd-Size: 5847 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf06.hostedemail.com (Postfix) with ESMTP for ; Mon, 30 Nov 2020 11:38:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=f4g92HolWEIcUN5sHQre4J13ebXNQzotEK5t5/pJgYg=; b=nVErVX4kx+o9OjH/bg2ppeww7q wtGoAs11AZd7qKjpkmAcJ3C4Ay0JfPzveCcIRguanHrKAAJM+oiSfvTwmS46L6fh70lCf4g27qUy/ PN5zperMQcjygmWj2pRcD91mVsCuOhq/Q0vWeMCxFipB+m4kXm2qzmiT+/SoUn7oU0pLHGQA1qx2j UF7W0KA9QvPnznSF6TvhEgq6gAZXa6xPqcC7GYdNbiJDy+DSHMzvBfnZJlrYIkiCQxLD3eI11sBe0 AydvPyfKm50jvfvJtGL8aWLNz6yXsNv1nytu7gf0ocWanej63lgwXkNL+tYUAsnggi/ArjsB0U/oP PV8CLKeQ==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.92.3 #3 (Red Hat Linux)) id 1kjhVa-0007Qd-Rm; Mon, 30 Nov 2020 11:38:19 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id E68FC30477A; Mon, 30 Nov 2020 12:38:17 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id D4FE02B84F9C4; Mon, 30 Nov 2020 12:38:17 +0100 (CET) Message-ID: <20201130113602.900215647@infradead.org> User-Agent: quilt/0.66 Date: Mon, 30 Nov 2020 12:27:07 +0100 From: Peter Zijlstra To: x86@kernel.org, willy@infradead.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, aarcange@redhat.com, kirill.shutemov@linux.intel.com, jroedel@suse.de, peterz@infradead.org Subject: [RFC][PATCH 2/9] x86/mm/pae: Make pmd_t similar to pte_t References: <20201130112705.900705277@infradead.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Instead of mucking about with at least 2 different ways of fudging it, do the same thing we do for pte_t. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/pgtable-3level.h | 42 +++++++++------------------- arch/x86/include/asm/pgtable-3level_types.h | 7 ++++ arch/x86/include/asm/pgtable_64_types.h | 1 arch/x86/include/asm/pgtable_types.h | 4 -- 4 files changed, 23 insertions(+), 31 deletions(-) --- a/arch/x86/include/asm/pgtable-3level.h +++ b/arch/x86/include/asm/pgtable-3level.h @@ -87,7 +87,7 @@ static inline pmd_t pmd_read_atomic(pmd_ ret |= ((pmdval_t)*(tmp + 1)) << 32; } - return (pmd_t) { ret }; + return (pmd_t) { .pmd = ret }; } static inline void native_set_pte_atomic(pte_t *ptep, pte_t pte) @@ -121,12 +121,11 @@ static inline void native_pte_clear(stru ptep->pte_high = 0; } -static inline void native_pmd_clear(pmd_t *pmd) +static inline void native_pmd_clear(pmd_t *pmdp) { - u32 *tmp = (u32 *)pmd; - *tmp = 0; + pmdp->pmd_low = 0; smp_wmb(); - *(tmp + 1) = 0; + pmdp->pmd_high = 0; } static inline void native_pud_clear(pud_t *pudp) @@ -162,25 +161,17 @@ static inline pte_t native_ptep_get_and_ #define native_ptep_get_and_clear(xp) native_local_ptep_get_and_clear(xp) #endif -union split_pmd { - struct { - u32 pmd_low; - u32 pmd_high; - }; - pmd_t pmd; -}; - #ifdef CONFIG_SMP static inline pmd_t native_pmdp_get_and_clear(pmd_t *pmdp) { - union split_pmd res, *orig = (union split_pmd *)pmdp; + pmd_t res; /* xchg acts as a barrier before setting of the high bits */ - res.pmd_low = xchg(&orig->pmd_low, 0); - res.pmd_high = orig->pmd_high; - orig->pmd_high = 0; + res.pmd_low = xchg(&pmdp->pmd_low, 0); + res.pmd_high = READ_ONCE(pmdp->pmd_high); + WRITE_ONCE(pmdp->pmd_high, 0); - return res.pmd; + return res; } #else #define native_pmdp_get_and_clear(xp) native_local_pmdp_get_and_clear(xp) @@ -199,17 +190,12 @@ static inline pmd_t pmdp_establish(struc * anybody. */ if (!(pmd_val(pmd) & _PAGE_PRESENT)) { - union split_pmd old, new, *ptr; - - ptr = (union split_pmd *)pmdp; - - new.pmd = pmd; - /* xchg acts as a barrier before setting of the high bits */ - old.pmd_low = xchg(&ptr->pmd_low, new.pmd_low); - old.pmd_high = ptr->pmd_high; - ptr->pmd_high = new.pmd_high; - return old.pmd; + old.pmd_low = xchg(&pmdp->pmd_low, pmd.pmd_low); + old.pmd_high = READ_ONCE(pmdp->pmd_high); + WRITE_ONCE(pmdp->pmd_high, pmd.pmd_high); + + return old; } do { --- a/arch/x86/include/asm/pgtable-3level_types.h +++ b/arch/x86/include/asm/pgtable-3level_types.h @@ -18,6 +18,13 @@ typedef union { }; pteval_t pte; } pte_t; + +typedef union { + struct { + unsigned long pmd_low, pmd_high; + }; + pmdval_t pmd; +} pmd_t; #endif /* !__ASSEMBLY__ */ #define SHARED_KERNEL_PMD (!static_cpu_has(X86_FEATURE_PTI)) --- a/arch/x86/include/asm/pgtable_64_types.h +++ b/arch/x86/include/asm/pgtable_64_types.h @@ -19,6 +19,7 @@ typedef unsigned long pgdval_t; typedef unsigned long pgprotval_t; typedef struct { pteval_t pte; } pte_t; +typedef struct { pmdval_t pmd; } pmd_t; #ifdef CONFIG_X86_5LEVEL extern unsigned int __pgtable_l5_enabled; --- a/arch/x86/include/asm/pgtable_types.h +++ b/arch/x86/include/asm/pgtable_types.h @@ -381,11 +381,9 @@ static inline pudval_t native_pud_val(pu #endif #if CONFIG_PGTABLE_LEVELS > 2 -typedef struct { pmdval_t pmd; } pmd_t; - static inline pmd_t native_make_pmd(pmdval_t val) { - return (pmd_t) { val }; + return (pmd_t) { .pmd = val }; } static inline pmdval_t native_pmd_val(pmd_t pmd) From patchwork Mon Nov 30 11:27:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 11940255 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D635DC64E7B for ; Mon, 30 Nov 2020 11:38:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5545220855 for ; Mon, 30 Nov 2020 11:38:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="v/A3yNF6" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5545220855 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 678C88D000D; Mon, 30 Nov 2020 06:38:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5B0F18D000A; Mon, 30 Nov 2020 06:38:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4A41B8D000C; Mon, 30 Nov 2020 06:38:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0086.hostedemail.com [216.40.44.86]) by kanga.kvack.org (Postfix) with ESMTP id 29F898D000A for ; Mon, 30 Nov 2020 06:38:31 -0500 (EST) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id E00F91EE6 for ; Mon, 30 Nov 2020 11:38:30 +0000 (UTC) X-FDA: 77540886780.28.cars59_1d03268273a1 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin28.hostedemail.com (Postfix) with ESMTP id BBE036C37 for ; Mon, 30 Nov 2020 11:38:30 +0000 (UTC) X-HE-Tag: cars59_1d03268273a1 X-Filterd-Recvd-Size: 2896 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf05.hostedemail.com (Postfix) with ESMTP for ; Mon, 30 Nov 2020 11:38:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=9rpmj7e+Hv4t7twTnPnStJvZhGql4xrwM1pM4f4qTOM=; b=v/A3yNF6sUJE5HpF7raY5uFFNv LPvE3P4LP5V9kkgmI+/QIcDyPpVT7Vh3gc5QRFwNaiWzqzaL5X2gi+khpdds4OqWrrWqvRTzZW77A 9rp3kp5SMnx2isXm7N/cf4A8FxPDVR3hPnZB8ZCGbB1pycC/C7iOsoNNJV2fsS705RBHS+VpqDBrl KovUEqHFj7y3yqT2EqGgpZrx+UocBkhosH13AEmesyHKZF+zploGxfnqYmYPEVGCBMZg+qCdB3Zu4 DKJZbFge63xNTPMMHmJ2sXWU1S4rGAwU1jgBF40LEiENRmdGXH8BlGqHK5kRGxaPlpQLRYLKJlGPa fli9mZ/g==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.92.3 #3 (Red Hat Linux)) id 1kjhVa-0007Qe-SJ; Mon, 30 Nov 2020 11:38:19 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id EA9FE3059DD; Mon, 30 Nov 2020 12:38:17 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id D90AE2B84F9C5; Mon, 30 Nov 2020 12:38:17 +0100 (CET) Message-ID: <20201130113602.959570640@infradead.org> User-Agent: quilt/0.66 Date: Mon, 30 Nov 2020 12:27:08 +0100 From: Peter Zijlstra To: x86@kernel.org, willy@infradead.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, aarcange@redhat.com, kirill.shutemov@linux.intel.com, jroedel@suse.de, peterz@infradead.org Subject: [RFC][PATCH 3/9] sh/mm: Make pmd_t similar to pte_t References: <20201130112705.900705277@infradead.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Just like 64bit pte_t, have a low/high split in pmd_t. Signed-off-by: Peter Zijlstra (Intel) --- arch/sh/include/asm/pgtable-3level.h | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) --- a/arch/sh/include/asm/pgtable-3level.h +++ b/arch/sh/include/asm/pgtable-3level.h @@ -28,9 +28,15 @@ #define pmd_ERROR(e) \ printk("%s:%d: bad pmd %016llx.\n", __FILE__, __LINE__, pmd_val(e)) -typedef struct { unsigned long long pmd; } pmd_t; +typedef struct { + struct { + unsigned long pmd_low; + unsigned long pmd_high; + }; + unsigned long long pmd; +} pmd_t; #define pmd_val(x) ((x).pmd) -#define __pmd(x) ((pmd_t) { (x) } ) +#define __pmd(x) ((pmd_t) { .pmd = (x) } ) static inline unsigned long pud_page_vaddr(pud_t pud) { From patchwork Mon Nov 30 11:27:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 11940261 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68958C64E90 for ; Mon, 30 Nov 2020 11:38:54 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D54E5206C0 for ; Mon, 30 Nov 2020 11:38:38 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="PY/ct2Y6" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D54E5206C0 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 255E48D000B; Mon, 30 Nov 2020 06:38:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1BA028D0005; Mon, 30 Nov 2020 06:38:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0099A8D000B; Mon, 30 Nov 2020 06:38:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0182.hostedemail.com [216.40.44.182]) by kanga.kvack.org (Postfix) with ESMTP id C56258D0005 for ; Mon, 30 Nov 2020 06:38:29 -0500 (EST) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 73D1D3634 for ; Mon, 30 Nov 2020 11:38:29 +0000 (UTC) X-FDA: 77540886738.06.hose29_541338a273a1 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin06.hostedemail.com (Postfix) with ESMTP id 4FCF310041A0F for ; Mon, 30 Nov 2020 11:38:29 +0000 (UTC) X-HE-Tag: hose29_541338a273a1 X-Filterd-Recvd-Size: 7326 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf42.hostedemail.com (Postfix) with ESMTP for ; Mon, 30 Nov 2020 11:38:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=Ie6Cf8JpAr3an+Bdhd3x1NiMOyVtEAcA4IqwZY52f4o=; b=PY/ct2Y6CM2kCapKoMcZEAgw2B XcM3BRjMXJqnyqWM1lDX4phuRr6CZvbT95tLtR/qluC3tfA/tI+Gh4d3pZ96BpofUig6TxJFzrU0R teH7GeIEC9bhUVwl6Fvfn2JeEOa48e6MQHiDED581yVgBjunDtX1iGzQCZu1kxmdi0CTy5TcXVLY0 649nuw6Z9uiIJZI0cn6MNk6Eio9aPR0/gsEO4EGLQ7kqCxBpbhefAExK8ikKXz/zm/crAwbDGsjre O1ej10L8/QaQOcN3Iyn4Niw2hCLK/2srVZhqFwMWFg6Cxpqa86N/IWvq0F3P3CZbRLwS/sh154RO3 FufOOQRw==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.92.3 #3 (Red Hat Linux)) id 1kjhVa-0007Qc-R9; Mon, 30 Nov 2020 11:38:19 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id EE795305C11; Mon, 30 Nov 2020 12:38:17 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id DD1AC2B84F9C6; Mon, 30 Nov 2020 12:38:17 +0100 (CET) Message-ID: <20201130113603.018510568@infradead.org> User-Agent: quilt/0.66 Date: Mon, 30 Nov 2020 12:27:09 +0100 From: Peter Zijlstra To: x86@kernel.org, willy@infradead.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, aarcange@redhat.com, kirill.shutemov@linux.intel.com, jroedel@suse.de, peterz@infradead.org Subject: [RFC][PATCH 4/9] mm: Fix pmd_read_atomic() References: <20201130112705.900705277@infradead.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: AFAICT there's no reason to do anything different than what we do for PTEs. Make it so (also affects SH). Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/pgtable-3level.h | 56 ---------------------------------- include/linux/pgtable.h | 49 +++++++++++++++++++++++------ 2 files changed, 39 insertions(+), 66 deletions(-) --- a/arch/x86/include/asm/pgtable-3level.h +++ b/arch/x86/include/asm/pgtable-3level.h @@ -34,62 +34,6 @@ static inline void native_set_pte(pte_t ptep->pte_low = pte.pte_low; } -#define pmd_read_atomic pmd_read_atomic -/* - * pte_offset_map_lock() on 32-bit PAE kernels was reading the pmd_t with - * a "*pmdp" dereference done by GCC. Problem is, in certain places - * where pte_offset_map_lock() is called, concurrent page faults are - * allowed, if the mmap_lock is hold for reading. An example is mincore - * vs page faults vs MADV_DONTNEED. On the page fault side - * pmd_populate() rightfully does a set_64bit(), but if we're reading the - * pmd_t with a "*pmdp" on the mincore side, a SMP race can happen - * because GCC will not read the 64-bit value of the pmd atomically. - * - * To fix this all places running pte_offset_map_lock() while holding the - * mmap_lock in read mode, shall read the pmdp pointer using this - * function to know if the pmd is null or not, and in turn to know if - * they can run pte_offset_map_lock() or pmd_trans_huge() or other pmd - * operations. - * - * Without THP if the mmap_lock is held for reading, the pmd can only - * transition from null to not null while pmd_read_atomic() runs. So - * we can always return atomic pmd values with this function. - * - * With THP if the mmap_lock is held for reading, the pmd can become - * trans_huge or none or point to a pte (and in turn become "stable") - * at any time under pmd_read_atomic(). We could read it truly - * atomically here with an atomic64_read() for the THP enabled case (and - * it would be a whole lot simpler), but to avoid using cmpxchg8b we - * only return an atomic pmdval if the low part of the pmdval is later - * found to be stable (i.e. pointing to a pte). We are also returning a - * 'none' (zero) pmdval if the low part of the pmd is zero. - * - * In some cases the high and low part of the pmdval returned may not be - * consistent if THP is enabled (the low part may point to previously - * mapped hugepage, while the high part may point to a more recently - * mapped hugepage), but pmd_none_or_trans_huge_or_clear_bad() only - * needs the low part of the pmd to be read atomically to decide if the - * pmd is unstable or not, with the only exception when the low part - * of the pmd is zero, in which case we return a 'none' pmd. - */ -static inline pmd_t pmd_read_atomic(pmd_t *pmdp) -{ - pmdval_t ret; - u32 *tmp = (u32 *)pmdp; - - ret = (pmdval_t) (*tmp); - if (ret) { - /* - * If the low part is null, we must not read the high part - * or we can end up with a partial pmd. - */ - smp_rmb(); - ret |= ((pmdval_t)*(tmp + 1)) << 32; - } - - return (pmd_t) { .pmd = ret }; -} - static inline void native_set_pte_atomic(pte_t *ptep, pte_t pte) { set_64bit((unsigned long long *)(ptep), native_pte_val(pte)); --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -258,6 +258,13 @@ static inline pte_t ptep_get(pte_t *ptep } #endif +#ifndef __HAVE_ARCH_PMDP_GET +static inline pmd_t pmdp_get(pmd_t *pmdp) +{ + return READ_ONCE(*pmdp); +} +#endif + #ifdef CONFIG_GUP_GET_PTE_LOW_HIGH /* * For walking the pagetables without holding any locks. Some architectures @@ -302,15 +309,44 @@ static inline pte_t ptep_get_lockless(pt return pte; } -#else /* CONFIG_GUP_GET_PTE_LOW_HIGH */ +#define ptep_get_lockless ptep_get_lockless + +#if CONFIG_PGTABLE_LEVELS > 2 +static inline pmd_t pmdp_get_lockless(pmd_t *pmdp) +{ + pmd_t pmd; + + lockdep_assert_irqs_disabled(); + + do { + pmd.pmd_low = pmdp->pmd_low; + smp_rmb(); + pmd.pmd_high = pmdp->pmd_high; + smp_rmb(); + } while (unlikely(pmd.pmd_low != pmdp->pmd_low)); + + return pmd; +} +#define pmdp_get_lockless pmdp_get_lockless +#endif /* CONFIG_PGTABLE_LEVELS > 2 */ +#endif /* CONFIG_GUP_GET_PTE_LOW_HIGH */ + /* * We require that the PTE can be read atomically. */ +#ifndef ptep_get_lockless static inline pte_t ptep_get_lockless(pte_t *ptep) { return ptep_get(ptep); } -#endif /* CONFIG_GUP_GET_PTE_LOW_HIGH */ +#endif + +#ifndef pmdp_get_lockless +static inline pmd_t pmdp_get_lockless(pmd_t *pmdp) +{ + return pmdp_get(pmdp); +} +#endif #ifdef CONFIG_TRANSPARENT_HUGEPAGE #ifndef __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR @@ -1211,17 +1247,10 @@ static inline int pud_trans_unstable(pud #endif } -#ifndef pmd_read_atomic static inline pmd_t pmd_read_atomic(pmd_t *pmdp) { - /* - * Depend on compiler for an atomic pmd read. NOTE: this is - * only going to work, if the pmdval_t isn't larger than - * an unsigned long. - */ - return *pmdp; + return pmdp_get_lockless(pmdp); } -#endif #ifndef arch_needs_pgtable_deposit #define arch_needs_pgtable_deposit() (false) From patchwork Mon Nov 30 11:27:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 11940253 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 26F82C64E8A for ; Mon, 30 Nov 2020 11:38:43 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 97997207BC for ; Mon, 30 Nov 2020 11:38:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="WSukwZUW" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 97997207BC Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5AE7C8D0005; Mon, 30 Nov 2020 06:38:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4F5C48D000C; Mon, 30 Nov 2020 06:38:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 253B58D000A; Mon, 30 Nov 2020 06:38:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0087.hostedemail.com [216.40.44.87]) by kanga.kvack.org (Postfix) with ESMTP id E91A58D000A for ; Mon, 30 Nov 2020 06:38:29 -0500 (EST) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id B34AD8249980 for ; Mon, 30 Nov 2020 11:38:29 +0000 (UTC) X-FDA: 77540886738.25.plate42_020cb12273a1 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin25.hostedemail.com (Postfix) with ESMTP id 9265C1804E3B2 for ; Mon, 30 Nov 2020 11:38:29 +0000 (UTC) X-HE-Tag: plate42_020cb12273a1 X-Filterd-Recvd-Size: 4591 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf02.hostedemail.com (Postfix) with ESMTP for ; Mon, 30 Nov 2020 11:38:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=DNbURhpRXVFm+D5xOcWGNkmJVhljKHRLoeBZiWAdDOY=; b=WSukwZUW2AUbYsK4fJL0oXkaxW odUBv4mi6iNIvKmegNq3IpgMdtuPQIkrzmlWcC75IN8eUldqGCS3nmGICMnBMR6IKqRIunmhrG38k XTaPsHKDJkJLZjK5jA57vQFLECmfq/KSb8E/RS5IBee4jcHYXTDexcp/Jq3eT5/PkB7x7SnOIyuns T5h78gjjuC4GKj4sEb+2zMOJzrzKmSRTwSrTww5zMsjxjjYYz8Bb/9EDKMWJSRJtwU4fgL93RSGWN /MtrBbmLQKzFk0/exYFgJ1UzxthgzgjWWKRr+A9ipBU9tU7eIU5Qwm6AoHMqd6fJPXQ/WOc42GqoM KOlmPxHg==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.92.3 #3 (Red Hat Linux)) id 1kjhVc-0007Qo-5f; Mon, 30 Nov 2020 11:38:20 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 35070305C22; Mon, 30 Nov 2020 12:38:18 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id E283D2B84F9C7; Mon, 30 Nov 2020 12:38:17 +0100 (CET) Message-ID: <20201130113603.079835817@infradead.org> User-Agent: quilt/0.66 Date: Mon, 30 Nov 2020 12:27:10 +0100 From: Peter Zijlstra To: x86@kernel.org, willy@infradead.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, aarcange@redhat.com, kirill.shutemov@linux.intel.com, jroedel@suse.de, peterz@infradead.org Subject: [RFC][PATCH 5/9] mm: Rename pmd_read_atomic() References: <20201130112705.900705277@infradead.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There's no point in having the identical routines for PTE/PMD have different names. Signed-off-by: Peter Zijlstra (Intel) --- include/linux/pgtable.h | 7 +------ mm/hmm.c | 2 +- mm/mapping_dirty_helpers.c | 2 +- mm/mprotect.c | 2 +- mm/userfaultfd.c | 2 +- 5 files changed, 5 insertions(+), 10 deletions(-) --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1244,11 +1244,6 @@ static inline int pud_trans_unstable(pud #endif } -static inline pmd_t pmd_read_atomic(pmd_t *pmdp) -{ - return pmdp_get_lockless(pmdp); -} - #ifndef arch_needs_pgtable_deposit #define arch_needs_pgtable_deposit() (false) #endif @@ -1275,7 +1270,7 @@ static inline pmd_t pmd_read_atomic(pmd_ */ static inline int pmd_none_or_trans_huge_or_clear_bad(pmd_t *pmd) { - pmd_t pmdval = pmd_read_atomic(pmd); + pmd_t pmdval = pmdp_get_lockless(pmd); /* * The barrier will stabilize the pmdval in a register or on * the stack so that it will stop changing under the code. --- a/mm/hmm.c +++ b/mm/hmm.c @@ -356,7 +356,7 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, * huge or device mapping one and compute corresponding pfn * values. */ - pmd = pmd_read_atomic(pmdp); + pmd = pmdp_get_lockless(pmdp); barrier(); if (!pmd_devmap(pmd) && !pmd_trans_huge(pmd)) goto again; --- a/mm/mapping_dirty_helpers.c +++ b/mm/mapping_dirty_helpers.c @@ -123,7 +123,7 @@ static int clean_record_pte(pte_t *pte, static int wp_clean_pmd_entry(pmd_t *pmd, unsigned long addr, unsigned long end, struct mm_walk *walk) { - pmd_t pmdval = pmd_read_atomic(pmd); + pmd_t pmdval = pmdp_get_lockless(pmd); if (!pmd_trans_unstable(&pmdval)) return 0; --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -190,7 +190,7 @@ static unsigned long change_pte_range(st */ static inline int pmd_none_or_clear_bad_unless_trans_huge(pmd_t *pmd) { - pmd_t pmdval = pmd_read_atomic(pmd); + pmd_t pmdval = pmdp_get_lockless(pmd); /* See pmd_none_or_trans_huge_or_clear_bad for info on barrier */ #ifdef CONFIG_TRANSPARENT_HUGEPAGE --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -553,7 +553,7 @@ static __always_inline ssize_t __mcopy_a break; } - dst_pmdval = pmd_read_atomic(dst_pmd); + dst_pmdval = pmdp_get_lockless(dst_pmd); /* * If the dst_pmd is mapped as THP don't * override it and just be strict. From patchwork Mon Nov 30 11:27:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 11940245 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B0C9C64E7B for ; Mon, 30 Nov 2020 11:38:26 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BD9D42087C for ; Mon, 30 Nov 2020 11:38:25 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="su+36IVy" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BD9D42087C Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D29678D0003; Mon, 30 Nov 2020 06:38:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CD9898D0001; Mon, 30 Nov 2020 06:38:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BF1CD8D0003; Mon, 30 Nov 2020 06:38:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0122.hostedemail.com [216.40.44.122]) by kanga.kvack.org (Postfix) with ESMTP id A93218D0001 for ; Mon, 30 Nov 2020 06:38:24 -0500 (EST) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 6F85D3634 for ; Mon, 30 Nov 2020 11:38:24 +0000 (UTC) X-FDA: 77540886528.06.loaf13_3515224273a1 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin06.hostedemail.com (Postfix) with ESMTP id 49B4310041A00 for ; Mon, 30 Nov 2020 11:38:24 +0000 (UTC) X-HE-Tag: loaf13_3515224273a1 X-Filterd-Recvd-Size: 3079 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf04.hostedemail.com (Postfix) with ESMTP for ; Mon, 30 Nov 2020 11:38:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=UiVGVeFLRq7zVPo40bMLNVgAJoLSCFMPhHigNy3P9ck=; b=su+36IVymoYyHjztNNWkljSkgM +0Crf3dx2ZnLdzrBXvO0ylXLyyl72BD55GrGQA1gkGoDBQy3e4fswH/PgmwajJNNXKEsgGM8MwCRe NuejYRToqRn3+BQINeTsw44xll9bOLVTWw8iQqEXF5aKJ3MOzZjXgSrnYIWBapz4ySRJ+/zgofBJL xVXbGyWYrRs7mTgev5RRdc9ROcTavZtbfJ56v7j3gIehKdJNjwaLeaTPoiT72+7ORYvYiVkK5OBYb EjCOGasOe2zX5Kpcn4vb2ZTQF6ceB9GIusmHZOoUNpebQQtLH2sImbfg7LwSA6cTAn6V/lfi1AIy4 M9n1SfQQ==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.92.3 #3 (Red Hat Linux)) id 1kjhVc-0007Qr-5Z; Mon, 30 Nov 2020 11:38:20 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 3508530769C; Mon, 30 Nov 2020 12:38:18 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id E46F42B84F9C3; Mon, 30 Nov 2020 12:38:17 +0100 (CET) Message-ID: <20201130113603.145310991@infradead.org> User-Agent: quilt/0.66 Date: Mon, 30 Nov 2020 12:27:11 +0100 From: Peter Zijlstra To: x86@kernel.org, willy@infradead.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, aarcange@redhat.com, kirill.shutemov@linux.intel.com, jroedel@suse.de, peterz@infradead.org Subject: [RFC][PATCH 6/9] mm/gup: Fix the lockless walkers References: <20201130112705.900705277@infradead.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On architectures where the PTE/PMD is larger than the native word size (i386-PAE for example), READ_ONCE() can do the wrong thing. Use pmdp_get_lockless() just like we use ptep_get_lockless(). Signed-off-by: Peter Zijlstra (Intel) --- kernel/events/core.c | 2 +- mm/gup.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -7041,7 +7041,7 @@ static u64 perf_get_pgtable_size(struct return pud_leaf_size(pud); pmdp = pmd_offset_lockless(pudp, pud, addr); - pmd = READ_ONCE(*pmdp); + pmd = pmdp_get_lockless(pmdp); if (!pmd_present(pmd)) return 0; --- a/mm/gup.c +++ b/mm/gup.c @@ -2467,7 +2467,7 @@ static int gup_pmd_range(pud_t *pudp, pu pmdp = pmd_offset_lockless(pudp, pud, addr); do { - pmd_t pmd = READ_ONCE(*pmdp); + pmd_t pmd = pmdp_get_lockless(pmdp); next = pmd_addr_end(addr, end); if (!pmd_present(pmd)) From patchwork Mon Nov 30 11:27:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 11940249 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E24AC64E7B for ; Mon, 30 Nov 2020 11:38:32 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id ADF1220855 for ; Mon, 30 Nov 2020 11:38:28 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="CGwc8xq0" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ADF1220855 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B5A3B8D0009; Mon, 30 Nov 2020 06:38:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A46998D0007; Mon, 30 Nov 2020 06:38:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 73F398D0008; Mon, 30 Nov 2020 06:38:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 4433D8D0001 for ; Mon, 30 Nov 2020 06:38:25 -0500 (EST) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 0DF7C180AD830 for ; Mon, 30 Nov 2020 11:38:25 +0000 (UTC) X-FDA: 77540886570.04.uncle84_230a500273a1 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin04.hostedemail.com (Postfix) with ESMTP id E866C800C89D for ; Mon, 30 Nov 2020 11:38:24 +0000 (UTC) X-HE-Tag: uncle84_230a500273a1 X-Filterd-Recvd-Size: 3083 Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) by imf07.hostedemail.com (Postfix) with ESMTP for ; Mon, 30 Nov 2020 11:38:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=merlin.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=/uoQQYbsRyNoJjZQKQhQxCD1OTCWXbPOIVU6csr5cic=; b=CGwc8xq0mLRcSMz/aOXXnK7TYg A2heKgcQG+T24YsWMXOGDcvBAeE8VacWZsbVXt0acpnu+OXthiOUr+O0fD0F6fsQZVHAThCi7QRaQ nzKGoTgxmD8/H0VuMiVBKYqOUjmyJcVoKNR7LTQeHJBeVv777uU9AHZs3OJ5E+DgIc7+ALuq6gUp1 wA68Golu7aKnkZP8goRA0SS2/t5N0DYpd/+M5Nmlt95Z5SofsIDIHAjZvld+eVDVsBC3JChqSkT2b GuRbLWBeMbyLUVWCXi336BxcKffs4F8logOqgGNRLmen/I35IgdiKaJ3767if4YeDAcad1+tG8g5A 8qk6Jgxg==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=noisy.programming.kicks-ass.net) by merlin.infradead.org with esmtpsa (Exim 4.92.3 #3 (Red Hat Linux)) id 1kjhVc-0006CX-TD; Mon, 30 Nov 2020 11:38:21 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 38C87307890; Mon, 30 Nov 2020 12:38:18 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id E9F922B84F9C9; Mon, 30 Nov 2020 12:38:17 +0100 (CET) Message-ID: <20201130113603.205586001@infradead.org> User-Agent: quilt/0.66 Date: Mon, 30 Nov 2020 12:27:12 +0100 From: Peter Zijlstra To: x86@kernel.org, willy@infradead.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, aarcange@redhat.com, kirill.shutemov@linux.intel.com, jroedel@suse.de, peterz@infradead.org Subject: [RFC][PATCH 7/9] x86/mm/pae: Dont (ab)use atomic64 References: <20201130112705.900705277@infradead.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: PAE implies CX8, write readable code. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/pgtable-3level.h | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) --- a/arch/x86/include/asm/pgtable-3level.h +++ b/arch/x86/include/asm/pgtable-3level.h @@ -2,8 +2,6 @@ #ifndef _ASM_X86_PGTABLE_3LEVEL_H #define _ASM_X86_PGTABLE_3LEVEL_H -#include - /* * Intel Physical Address Extension (PAE) Mode - three-level page * tables on PPro+ CPUs. @@ -96,11 +94,13 @@ static inline void pud_clear(pud_t *pudp #ifdef CONFIG_SMP static inline pte_t native_ptep_get_and_clear(pte_t *ptep) { - pte_t res; + pte_t old; - res.pte = (pteval_t)arch_atomic64_xchg((atomic64_t *)ptep, 0); + do { + old = *ptep; + } while (cmpxchg64(&ptep->pte, old.pte, 0ULL) != old.pte); - return res; + return old; } #else #define native_ptep_get_and_clear(xp) native_local_ptep_get_and_clear(xp) From patchwork Mon Nov 30 11:27:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 11940259 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 888E3C64E8A for ; Mon, 30 Nov 2020 11:38:53 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id F2BE12073C for ; Mon, 30 Nov 2020 11:38:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="IntvTR+I" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F2BE12073C Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AE51E8D000C; Mon, 30 Nov 2020 06:38:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A46DA8D000A; Mon, 30 Nov 2020 06:38:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 84DAA8D000E; Mon, 30 Nov 2020 06:38:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0122.hostedemail.com [216.40.44.122]) by kanga.kvack.org (Postfix) with ESMTP id 616CA8D000C for ; Mon, 30 Nov 2020 06:38:31 -0500 (EST) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 2B3E8181AC553 for ; Mon, 30 Nov 2020 11:38:31 +0000 (UTC) X-FDA: 77540886822.10.wave38_070974d273a1 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin10.hostedemail.com (Postfix) with ESMTP id 0A14A16A046 for ; Mon, 30 Nov 2020 11:38:31 +0000 (UTC) X-HE-Tag: wave38_070974d273a1 X-Filterd-Recvd-Size: 3142 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf18.hostedemail.com (Postfix) with ESMTP for ; Mon, 30 Nov 2020 11:38:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=V9QjUriZDqhHtbmhIGOJQ2OP24LyJ0914LVzwONP/5E=; b=IntvTR+I8tJ7fMvRMnYS9SHGv2 5QTX0u3Exk+2zImll++TftJj9uAShEAa0kGy23ztNLIE5FLk3maTTHXVEimUEglqS6mDzKC4xcGQp bhUOhy38oOdmsupqMBaKTZOsq6SQus4an4hbMDGS4mcZ2MRchReKPNvL/OqZm6aCcXOIFQUCPcYvt +Xtg7s+lMOa9IUwAZVpr9edcR+8VvE87kj3Ijt9X3qXUF/KlZZWMbHV80nym1hF4ijCzh86CdAjNg 0Vj0nMh5mt0b1zl/vabbQkoq1YMdppfUKs0uuqW9LosNXePQ9LuRYm0OR+S7MTNj8X7cLLD4p19/7 lpI2/QUQ==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.92.3 #3 (Red Hat Linux)) id 1kjhVc-0007Qq-5B; Mon, 30 Nov 2020 11:38:20 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 37EB430785A; Mon, 30 Nov 2020 12:38:18 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id EF3D92B84F9CA; Mon, 30 Nov 2020 12:38:17 +0100 (CET) Message-ID: <20201130113603.264918968@infradead.org> User-Agent: quilt/0.66 Date: Mon, 30 Nov 2020 12:27:13 +0100 From: Peter Zijlstra To: x86@kernel.org, willy@infradead.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, aarcange@redhat.com, kirill.shutemov@linux.intel.com, jroedel@suse.de, peterz@infradead.org Subject: [RFC][PATCH 8/9] x86/mm/pae: Use WRITE_ONCE() References: <20201130112705.900705277@infradead.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/pgtable-3level.h | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) --- a/arch/x86/include/asm/pgtable-3level.h +++ b/arch/x86/include/asm/pgtable-3level.h @@ -27,9 +27,9 @@ */ static inline void native_set_pte(pte_t *ptep, pte_t pte) { - ptep->pte_high = pte.pte_high; + WRITE_ONCE(ptep->pte_high, pte.pte_high); smp_wmb(); - ptep->pte_low = pte.pte_low; + WRITE_ONCE(ptep->pte_low, pte.pte_low); } static inline void native_set_pte_atomic(pte_t *ptep, pte_t pte) @@ -58,14 +58,14 @@ static inline void native_set_pud(pud_t static inline void native_pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { - ptep->pte_low = 0; + WRITE_ONCE(ptep->pte_low, 0); smp_wmb(); - ptep->pte_high = 0; + WRITE_ONCE(ptep->pte_high, 0); } static inline void native_pmd_clear(pmd_t *pmdp) { - pmdp->pmd_low = 0; + WRITE_ONCE(pmdp->pmd_low, 0); smp_wmb(); pmdp->pmd_high = 0; } From patchwork Mon Nov 30 11:27:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 11940251 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C67CC64E7B for ; Mon, 30 Nov 2020 11:38:40 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 48F4F2073C for ; Mon, 30 Nov 2020 11:38:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="sNy3qGAi" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 48F4F2073C Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3B5738D0008; Mon, 30 Nov 2020 06:38:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 367528D0005; Mon, 30 Nov 2020 06:38:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 280698D0008; Mon, 30 Nov 2020 06:38:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0047.hostedemail.com [216.40.44.47]) by kanga.kvack.org (Postfix) with ESMTP id 01DC48D0005 for ; Mon, 30 Nov 2020 06:38:26 -0500 (EST) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id C10B28249980 for ; Mon, 30 Nov 2020 11:38:26 +0000 (UTC) X-FDA: 77540886612.11.kite54_240b735273a1 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin11.hostedemail.com (Postfix) with ESMTP id A46E2180F8B80 for ; Mon, 30 Nov 2020 11:38:26 +0000 (UTC) X-HE-Tag: kite54_240b735273a1 X-Filterd-Recvd-Size: 3944 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf08.hostedemail.com (Postfix) with ESMTP for ; Mon, 30 Nov 2020 11:38:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=xH6cVi24iSh0Z6RI2Sd+J27ye9DAHC3TEE3Mr7hJfxo=; b=sNy3qGAiEjXLyHzpUioeA1iNVj kxI/wm2ONT9/EW9Zy5jZI/BdH6tiIDmN86xxtJv+KEpCwA3EtMI2Pq2kGJ/UClCZMFOPPU+/KQ+Qq 9VmH/LYfiWgUoyzrDHdHK9+Xq9Bllz0/IpZrZn4arQDnBBU+Xxv2GVPTz7vwkpXDR7k7M071B8Ewt 4AmGRWfggI2KBA2GifsgP2Um2Qtrf7x2OmeO6LWeHiCjDonbzYVM4SJURDcziUVKVMjxXvqkLmeir vCzPO2buys+Ka7Bh3R4k5iWhnZrhihoTQK+btdTaHWvEtwLsCp62RKwSMV7tfFaIY69bR81HrgbHl Zuxwguyg==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.92.3 #3 (Red Hat Linux)) id 1kjhVc-0007Qp-6R; Mon, 30 Nov 2020 11:38:20 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 37E6D3077E1; Mon, 30 Nov 2020 12:38:18 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 006072B84F9CB; Mon, 30 Nov 2020 12:38:17 +0100 (CET) Message-ID: <20201130113603.324154533@infradead.org> User-Agent: quilt/0.66 Date: Mon, 30 Nov 2020 12:27:14 +0100 From: Peter Zijlstra To: x86@kernel.org, willy@infradead.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, aarcange@redhat.com, kirill.shutemov@linux.intel.com, jroedel@suse.de, peterz@infradead.org Subject: [RFC][PATCH 9/9] x86/mm/pae: Be consistent with pXXp_get_and_clear() References: <20201130112705.900705277@infradead.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Given that ptep_get_and_clear() uses cmpxchg8b, and that should be by far the most common case, there's no point in having an optimized variant for pmd/pud. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/pgtable-3level.h | 34 ++++++++++------------------------ 1 file changed, 10 insertions(+), 24 deletions(-) --- a/arch/x86/include/asm/pgtable-3level.h +++ b/arch/x86/include/asm/pgtable-3level.h @@ -108,14 +108,13 @@ static inline pte_t native_ptep_get_and_ #ifdef CONFIG_SMP static inline pmd_t native_pmdp_get_and_clear(pmd_t *pmdp) { - pmd_t res; + pmd_t old; - /* xchg acts as a barrier before setting of the high bits */ - res.pmd_low = xchg(&pmdp->pmd_low, 0); - res.pmd_high = READ_ONCE(pmdp->pmd_high); - WRITE_ONCE(pmdp->pmd_high, 0); + do { + old = *pmdp; + } while (cmpxchg64(&pmdp->pmd, old.pmd, 0ULL) != old.pmd); - return res; + return old; } #else #define native_pmdp_get_and_clear(xp) native_local_pmdp_get_and_clear(xp) @@ -151,28 +150,15 @@ static inline pmd_t pmdp_establish(struc #endif #ifdef CONFIG_SMP -union split_pud { - struct { - u32 pud_low; - u32 pud_high; - }; - pud_t pud; -}; - static inline pud_t native_pudp_get_and_clear(pud_t *pudp) { - union split_pud res, *orig = (union split_pud *)pudp; + pud_t old; -#ifdef CONFIG_PAGE_TABLE_ISOLATION - pti_set_user_pgtbl(&pudp->p4d.pgd, __pgd(0)); -#endif - - /* xchg acts as a barrier before setting of the high bits */ - res.pud_low = xchg(&orig->pud_low, 0); - res.pud_high = orig->pud_high; - orig->pud_high = 0; + do { + old = *pudp; + } while (cmpxchg64(&pudp->pud, old.pud, 0ULL) != old.pud); - return res.pud; + return old; } #else #define native_pudp_get_and_clear(xp) native_local_pudp_get_and_clear(xp)