From patchwork Tue Jul 25 13:53:04 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Catalin Marinas X-Patchwork-Id: 9862067 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id EC5C9602B1 for ; Tue, 25 Jul 2017 13:55:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E35E1285C5 for ; Tue, 25 Jul 2017 13:55:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D7E0A28630; Tue, 25 Jul 2017 13:55:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id EFB84285C5 for ; Tue, 25 Jul 2017 13:55:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=CgAhHCVkG1M66KxkoJjf8Rera0kjOVfZsP8LpkC1SYE=; b=AXlPNehDGEzhy2QMVA2sYMU1Xb V0kO532OStG27LjRJ8l+htmd/wZePOmFlgX37LgMq3MqD2ee5jLzFkwZU9Gx2jkyRRFpOaaNSEfqA 6Cz6V68qxxKAk96KYhRIiYb2XBBcmWKfWKIBTksov0X4BVTk3/Zi9MWr6j/Byg8Wyv2UM5Jjr2qNw sLlPnZ7UV6dN3XZdit/fm45Ri6fa5XLJD8vCRnPS/rscYq/ajEkMTj9qDxcU3ZslgH3j4uZuapsD1 lZu8S1GUUG0Na2H7tVhp5A3PkqGfLankDM4OOPcGYXGw5XimO0iUm28hr0iz6K9CBLfdlmNObYSh1 SowrR8ug==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1da0Iu-0003QJ-4B; Tue, 25 Jul 2017 13:55:16 +0000 Received: from merlin.infradead.org ([2001:8b0:10b:1231::1]) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1da0HS-0001jc-JJ for linux-arm-kernel@bombadil.infradead.org; Tue, 25 Jul 2017 13:53:46 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=merlin.20170209; h=References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=EAQ0VjPov4yXcbnlfaZKnfclAgG0lE01hegmrovPtXQ=; b=xwTpu3vqHzMSdiePLXnO6UNQ0 KufAcigbUXuJFboVbHpzcLVLBBiPt1ydz0yiFS0q5M/05kL4Vavii1XTjjQoieDaTRLm2ypAkTNuY EmYP10/6XM3UVpCoEt4KHU9iDmbyB2qRJYLn77rTCODEUrI/HVcxIJjiHVmW6mliBw+jzRpYKizrD CPlqXd/UqnyIkELvntF8jpSil9dGY3Vf+x0oETFGUOm9+RoJYVN4KcWvMKz82lhB8DVVy+bpvrRv6 gbVMYc7PnjdqVHMYXIney06LNV43MKq5RDofO7d/fG70QbqORKPNQW8vXJa3L6LIIQW7SKLNTq8BD ZGMOzp6CA==; Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70] helo=foss.arm.com) by merlin.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1da0HM-0007DA-VW for linux-arm-kernel@lists.infradead.org; Tue, 25 Jul 2017 13:53:44 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7E25615BF; Tue, 25 Jul 2017 06:53:21 -0700 (PDT) Received: from armageddon.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 447B53F540; Tue, 25 Jul 2017 06:53:20 -0700 (PDT) From: Catalin Marinas To: linux-arm-kernel@lists.infradead.org Subject: [PATCH 2/6] arm64: Convert pte handling from inline asm to using (cmp)xchg Date: Tue, 25 Jul 2017 14:53:04 +0100 Message-Id: <20170725135308.18173-3-catalin.marinas@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20170725135308.18173-1-catalin.marinas@arm.com> References: <20170725135308.18173-1-catalin.marinas@arm.com> X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Marc Zyngier , Will Deacon , Christoffer Dall , Steve Capper MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP With the support for hardware updates of the access and dirty states, the following pte handling functions had to be implemented using exclusives: __ptep_test_and_clear_young(), ptep_get_and_clear(), ptep_set_wrprotect() and ptep_set_access_flags(). To take advantage of the LSE atomic instructions and also make the code cleaner, convert these pte functions to use the more generic cmpxchg()/xchg(). Cc: Will Deacon Acked-by: Mark Rutland Acked-by: Steve Capper Signed-off-by: Catalin Marinas --- arch/arm64/include/asm/pgtable.h | 67 +++++++++++++++++----------------------- arch/arm64/mm/fault.c | 23 ++++++-------- 2 files changed, 39 insertions(+), 51 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 6eae342ced6b..4580d5992d0c 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -39,6 +39,7 @@ #ifndef __ASSEMBLY__ +#include #include #include @@ -173,6 +174,11 @@ static inline pte_t pte_clear_rdonly(pte_t pte) return clear_pte_bit(pte, __pgprot(PTE_RDONLY)); } +static inline pte_t pte_set_rdonly(pte_t pte) +{ + return set_pte_bit(pte, __pgprot(PTE_RDONLY)); +} + static inline pte_t pte_mkpresent(pte_t pte) { return set_pte_bit(pte, __pgprot(PTE_VALID)); @@ -593,20 +599,15 @@ static inline int pmdp_set_access_flags(struct vm_area_struct *vma, #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG static inline int __ptep_test_and_clear_young(pte_t *ptep) { - pteval_t pteval; - unsigned int tmp, res; + pte_t old_pte, pte; - asm volatile("// __ptep_test_and_clear_young\n" - " prfm pstl1strm, %2\n" - "1: ldxr %0, %2\n" - " ubfx %w3, %w0, %5, #1 // extract PTE_AF (young)\n" - " and %0, %0, %4 // clear PTE_AF\n" - " stxr %w1, %0, %2\n" - " cbnz %w1, 1b\n" - : "=&r" (pteval), "=&r" (tmp), "+Q" (pte_val(*ptep)), "=&r" (res) - : "L" (~PTE_AF), "I" (ilog2(PTE_AF))); + do { + old_pte = READ_ONCE(*ptep); + pte = pte_mkold(old_pte); + } while (cmpxchg_relaxed(&pte_val(*ptep), pte_val(old_pte), + pte_val(pte)) != pte_val(old_pte)); - return res; + return pte_young(old_pte); } static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, @@ -630,17 +631,7 @@ static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address, pte_t *ptep) { - pteval_t old_pteval; - unsigned int tmp; - - asm volatile("// ptep_get_and_clear\n" - " prfm pstl1strm, %2\n" - "1: ldxr %0, %2\n" - " stxr %w1, xzr, %2\n" - " cbnz %w1, 1b\n" - : "=&r" (old_pteval), "=&r" (tmp), "+Q" (pte_val(*ptep))); - - return __pte(old_pteval); + return __pte(xchg_relaxed(&pte_val(*ptep), 0)); } #ifdef CONFIG_TRANSPARENT_HUGEPAGE @@ -659,21 +650,21 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, #define __HAVE_ARCH_PTEP_SET_WRPROTECT static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long address, pte_t *ptep) { - pteval_t pteval; - unsigned long tmp; - - asm volatile("// ptep_set_wrprotect\n" - " prfm pstl1strm, %2\n" - "1: ldxr %0, %2\n" - " tst %0, %4 // check for hw dirty (!PTE_RDONLY)\n" - " csel %1, %3, xzr, eq // set PTE_DIRTY|PTE_RDONLY if dirty\n" - " orr %0, %0, %1 // if !dirty, PTE_RDONLY is already set\n" - " and %0, %0, %5 // clear PTE_WRITE/PTE_DBM\n" - " stxr %w1, %0, %2\n" - " cbnz %w1, 1b\n" - : "=&r" (pteval), "=&r" (tmp), "+Q" (pte_val(*ptep)) - : "r" (PTE_DIRTY|PTE_RDONLY), "L" (PTE_RDONLY), "L" (~PTE_WRITE) - : "cc"); + pte_t old_pte, pte; + + do { + pte = old_pte = READ_ONCE(*ptep); + /* + * If hardware-dirty (PTE_WRITE/DBM bit set and PTE_RDONLY + * clear), set the PTE_DIRTY and PTE_RDONLY bits. + */ + if (pte_hw_dirty(pte)) { + pte = pte_mkdirty(pte); + pte = pte_set_rdonly(pte); + } + pte = pte_wrprotect(pte); + } while (cmpxchg_relaxed(&pte_val(*ptep), pte_val(old_pte), + pte_val(pte)) != pte_val(old_pte)); } #ifdef CONFIG_TRANSPARENT_HUGEPAGE diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 2509e4fe6992..65ed66bb2828 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -34,6 +34,7 @@ #include #include +#include #include #include #include @@ -154,8 +155,7 @@ int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address, pte_t *ptep, pte_t entry, int dirty) { - pteval_t old_pteval; - unsigned int tmp; + pteval_t old_pteval, pteval; if (pte_same(*ptep, entry)) return 0; @@ -165,7 +165,7 @@ int ptep_set_access_flags(struct vm_area_struct *vma, /* set PTE_RDONLY if actual read-only or clean PTE */ if (!pte_write(entry) || !pte_sw_dirty(entry)) - pte_val(entry) |= PTE_RDONLY; + entry = pte_set_rdonly(entry); /* * Setting the flags must be done atomically to avoid racing with the @@ -174,16 +174,13 @@ int ptep_set_access_flags(struct vm_area_struct *vma, * (calculated as: a & b == ~(~a | ~b)). */ pte_val(entry) ^= PTE_RDONLY; - asm volatile("// ptep_set_access_flags\n" - " prfm pstl1strm, %2\n" - "1: ldxr %0, %2\n" - " eor %0, %0, %3 // negate PTE_RDONLY in *ptep\n" - " orr %0, %0, %4 // set flags\n" - " eor %0, %0, %3 // negate final PTE_RDONLY\n" - " stxr %w1, %0, %2\n" - " cbnz %w1, 1b\n" - : "=&r" (old_pteval), "=&r" (tmp), "+Q" (pte_val(*ptep)) - : "L" (PTE_RDONLY), "r" (pte_val(entry))); + do { + old_pteval = READ_ONCE(pte_val(*ptep)); + pteval = old_pteval ^ PTE_RDONLY; + pteval |= pte_val(entry); + pteval ^= PTE_RDONLY; + } while (cmpxchg_relaxed(&pte_val(*ptep), old_pteval, pteval) != + old_pteval); flush_tlb_fix_spurious_fault(vma, address); return 1;