From patchwork Fri Aug 2 15:14:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13751626 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA9A6C52D6F for ; Fri, 2 Aug 2024 15:18:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 35BC96B0088; Fri, 2 Aug 2024 11:18:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 30C6B6B0092; Fri, 2 Aug 2024 11:18:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1D4816B0096; Fri, 2 Aug 2024 11:18:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id EDC0B6B0088 for ; Fri, 2 Aug 2024 11:18:42 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 877F31A1213 for ; Fri, 2 Aug 2024 15:18:42 +0000 (UTC) X-FDA: 82407662484.28.E24FF31 Received: from mail-lj1-f169.google.com (mail-lj1-f169.google.com [209.85.208.169]) by imf15.hostedemail.com (Postfix) with ESMTP id 8F577A002E for ; Fri, 2 Aug 2024 15:18:40 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=toj0CO6p; spf=pass (imf15.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.208.169 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1722611914; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AGB1X3sQQ3LKpfi0jpo5C6FpBS7I6wNMeoGCJ6BimlY=; b=WpvnviRH3nDmi+Z0XVG1tMz0kbcR4yiP/RDnWwUr3mNgLGfeqGAHG4MA02SSTw1y494foi 3L55YmUpzFCj7uun7KLz2tdsFhtuSMKNh1o7p8dTyvEb+ck6DC2v5e/qoMuitaragwAsbJ LkWZeAVWxzXmtNQMmCVaehZZEqN5g24= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=toj0CO6p; spf=pass (imf15.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.208.169 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1722611914; a=rsa-sha256; cv=none; b=3Dwlwd4hpsyYgzmR9rLGuudC1XV/jCWagcPqJyeETL/vpsOSk4cPvt9sWm2X7ipiL5tUUz otsJQ6cPRk3aPuTlc8yBikQR9h5ml2RNIeKk38fPn0NMiHO07Lv7ObhFVyPKvZo+ZJ3p2a nUkqiTxdZvo3QRaQUqzD4R1lydv+79I= Received: by mail-lj1-f169.google.com with SMTP id 38308e7fff4ca-2ef2d96164aso92152141fa.3 for ; Fri, 02 Aug 2024 08:18:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1722611919; x=1723216719; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AGB1X3sQQ3LKpfi0jpo5C6FpBS7I6wNMeoGCJ6BimlY=; b=toj0CO6pL+aN0OX/PWMUlb3kdOuOPT+VZ6kt3ublKqDgjR5Vw7rEG+qFe7iFWeLfb+ vjW9bPZTfCoI/2Jr6a+lNK+vLmqKVBHBX7kHsFQZOOll4NgUZXxlkT9/aurvNUAQxNNv 8PHTTIzddb6s8lgE+pH9Owi0u8GxrjVipOpVelPkUyJxngny4azlqkx54m2gPPpUoyP3 dbAOlJuRxiQrGEai7EJoq5CfQtDGr61LL/Fyyui/2dKI7Wls36MHnKb/IfDt9BlJ+oGV dEcqXIh1+2FyVxadwv7gYeZ+w//GkI0CkqnyAtWVBeNyKMWeC5h/UtVQ3MCTbig/496d iBhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722611919; x=1723216719; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AGB1X3sQQ3LKpfi0jpo5C6FpBS7I6wNMeoGCJ6BimlY=; b=wMLiwCcj1xpteGJh+bYX7kOEH9V8QpNlDr8YhKAosZEqXKZl9+Pi027Ma2zIEIBbeT ylLp/gmn3zXAzkpJc+MsF5oqWxcKML6k8/SwfVX+gVePw4DAN7wjN8+iAtZ/Z4rFp+01 9CWnlAWmWrfleWZdVIQqWIEJF0XC6TVhQVPvcQW3pbHlDb786KeNdrRiBtai214N82YI YtPyrrwBjjSdQgEbzhkjJwaOfff0jnP1hreeBQ5dWQAcSwbW3QSxkwCbRnNzMVNm1fxx iyQUmNiJfBcpUVRrtdwhTMsNB6guWQqGCC91BOJruNexzN0OSiwLJIKAjIHJh3XbiWjW EZIQ== X-Forwarded-Encrypted: i=1; AJvYcCX6f1J1plx2IGEn6EluLyb7cqDz9fdohJS5852D7ZxERXTl0rGp0v0R+E453Ob1WWL7zBbtMAdJs5AZi2KFstBtsZ0= X-Gm-Message-State: AOJu0Yw85WibDW1oHOlvqbNUil/eRiU7kl+JT3FI0bCPPoveG0PQDVNy VP8TW0yDxAFtUSZdLKeDcZyRRsUGoPa5H6/zjROoQi0n37dIRQFD06tt4N+57pg= X-Google-Smtp-Source: AGHT+IFLYojfl67dZHCnARw4zAZgsB8CSLUwEfcrwOW6i95dP1Od+Eh6wNfB4OrwDg1Qyouoqg9rsQ== X-Received: by 2002:a2e:b170:0:b0:2ef:1bd5:bac3 with SMTP id 38308e7fff4ca-2f15ab419camr28122861fa.41.1722611918703; Fri, 02 Aug 2024 08:18:38 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-42824af5410sm97336045e9.1.2024.08.02.08.18.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Aug 2024 08:18:38 -0700 (PDT) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH v3 4/9] mm: Use common set_huge_pte_at() function for riscv/arm64 Date: Fri, 2 Aug 2024 17:14:25 +0200 Message-Id: <20240802151430.99114-5-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240802151430.99114-1-alexghiti@rivosinc.com> References: <20240802151430.99114-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: g9igkdnkpsausasywc9ku58ae8pj4wg7 X-Rspamd-Queue-Id: 8F577A002E X-Rspamd-Server: rspam11 X-HE-Tag: 1722611920-481796 X-HE-Meta: U2FsdGVkX19ZfGsWsWufiPIs38MgqTX/DGYSYBjIhcXfuc61+b2414HjEwEK0tdIldWB6l1LjsvWL3skee2nVB8U5uRK+0DDXITPzxNZTVIlJM4JNnqMM3hVCluFXpzDnMOHy/cWMSHtc9XAk1bj5ohgHKfPajysXAovp/+MqR/Qy8oJhhxStupexT5nlJL1v1dK5+id6d76G+FoMD2nt1hx28OEL66LcXMdO82oY8vFwwvrH1s6XTCDsKX4oTOJmRP8pWbmjtRdm5BGnqOQQAvoT10sdMvDzFwaUg1YFiZJGWPEIiHA1CJSQv8LjLZh9cbTpuEdgEOOHS+2wsOa95Vxwy8k2Yb/c3xlPH9CUsi1YblCBKBkWe/E8gL1xO+O/Px7xe4ufw1AOA6u/oCLaCWo2dH/JUFWtCHewaJVJPAqya741we1HHv2dM4bYawz4xUTDOoWJO23mok6l6HD0xCpFSmOzYl3f366Nn6g1b0zdKNak2PMoWrJY+kduz2YLIyyijsknZEOVZIUlQhPLsZ8K3rQNvDunIQBTk9CldisvKviQGJ6/iq/uPmjXoVSf03WU1C4hRt1NGX+oW99nLVJq7SVKjp+HB1NmghTKeQEB9ZMgvDYm1DQByEq1gZV7D2zdjPlkRqGi8lPfDfYbHrOIe/Z9iVE31GOtpjWraHC3H+xKSkei66/KhG1vzzbPb9H0HyVQ1N+dmdW8Z6lLC/RRzFOj+MUYH68epMkXMs9Rm2UMYtM/XWNNW/JA9Sp6JtztdQjxqycJqNpoXb4BuXwNr1iz8cG65DEXeIaco6UOm3oFoZ2f5c7jsONqcLBbqlIIx+of8vGvpwGulOAvuwnurvVsoPI6HImyCFlLBe/pJqaN080y5cH42xPV/73HBnuGKfngTPbvsQqS87VsNjz5NvJcLlJgDwG9Gr9NNQlhpRyeAJ9P99pNd/eY+3kIcqfs4blVmUGOa1DwNO yvFJsetd n2p0rh6v5aaLRwip8i8Csn8mXg86bRU6xhEa9msmwXveeuy8s7Oc3ZmOXJnQKzZ3MBz7jpmgTJXOtq83EI39SbCDCmQkSSDE7cZcVO3wKKhCFx9jIODQCuXjNKUCYjDzam4msxIHF7Yw92Suu/nyLHD5iW4QhEQ2IMXeKo+rjCwX4EuiH8AN6fmwgZVulblRoVKVvw/E5CucNP7dJlxtopNnYf93Lyyv5yodqQSyL/shdWmgc38zF0E4UztSDxdIgRf9EvwaeApS2A1aMR3SSCgoiHf7UvMyGTyyJzGuRwu85fz2mslVdiuZUCiofhdzY4IX8ojtPlXpha9WULKZ+cKzQkYOXs0okUra7kiveSPw8ZaqwEPO93mZISTEiSZ7w3P4EYswzoZebls5yArWJhPr1HA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: After some adjustments, both architectures have the same implementation so move it to the generic code. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/hugetlb.h | 3 -- arch/arm64/mm/hugetlbpage.c | 56 ----------------------------- arch/riscv/include/asm/hugetlb.h | 5 --- arch/riscv/include/asm/pgtable.h | 8 +++-- arch/riscv/mm/hugetlbpage.c | 62 -------------------------------- include/linux/hugetlb_contpte.h | 5 +++ mm/hugetlb_contpte.c | 59 ++++++++++++++++++++++++++++++ 7 files changed, 69 insertions(+), 129 deletions(-) diff --git a/arch/arm64/include/asm/hugetlb.h b/arch/arm64/include/asm/hugetlb.h index 80d25b4eff25..d35093b7ab59 100644 --- a/arch/arm64/include/asm/hugetlb.h +++ b/arch/arm64/include/asm/hugetlb.h @@ -27,9 +27,6 @@ static inline void arch_clear_hugetlb_flags(struct folio *folio) pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags); #define arch_make_huge_pte arch_make_huge_pte -#define __HAVE_ARCH_HUGE_SET_HUGE_PTE_AT -extern void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pte, unsigned long sz); #define __HAVE_ARCH_HUGE_PTEP_SET_ACCESS_FLAGS extern int huge_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 58cb5e06dcb2..c7130d1f07c4 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -145,62 +145,6 @@ static pte_t get_clear_contig_flush(struct mm_struct *mm, return orig_pte; } -/* - * Changing some bits of contiguous entries requires us to follow a - * Break-Before-Make approach, breaking the whole contiguous set - * before we can change any entries. See ARM DDI 0487A.k_iss10775, - * "Misprogramming of the Contiguous bit", page D4-1762. - * - * This helper performs the break step for use cases where the - * original pte is not needed. - */ -static void clear_flush(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, - unsigned long pgsize, - unsigned long ncontig) -{ - struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); - unsigned long i, saddr = addr; - - for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) - __ptep_get_and_clear(mm, addr, ptep); - - flush_tlb_range(&vma, saddr, addr); -} - -void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pte, unsigned long sz) -{ - size_t pgsize; - int i; - int ncontig; - unsigned long pfn, dpfn; - pgprot_t hugeprot; - - ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); - - if (!pte_present(pte)) { - for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) - __set_ptes(mm, addr, ptep, pte, 1); - return; - } - - if (!pte_cont(pte)) { - __set_ptes(mm, addr, ptep, pte, 1); - return; - } - - pfn = pte_pfn(pte); - dpfn = pgsize >> PAGE_SHIFT; - hugeprot = pte_pgprot(pte); - - clear_flush(mm, addr, ptep, pgsize, ncontig); - - for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) - __set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); -} - pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr, unsigned long sz) { diff --git a/arch/riscv/include/asm/hugetlb.h b/arch/riscv/include/asm/hugetlb.h index d9f9bfb84908..28cbf5d761e1 100644 --- a/arch/riscv/include/asm/hugetlb.h +++ b/arch/riscv/include/asm/hugetlb.h @@ -24,11 +24,6 @@ bool arch_hugetlb_migration_supported(struct hstate *h); void huge_pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned long sz); -#define __HAVE_ARCH_HUGE_SET_HUGE_PTE_AT -void set_huge_pte_at(struct mm_struct *mm, - unsigned long addr, pte_t *ptep, pte_t pte, - unsigned long sz); - #define __HAVE_ARCH_HUGE_PTEP_GET_AND_CLEAR pte_t huge_ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep); diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 412ccebcdee9..46b409e558b3 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -628,9 +628,8 @@ extern int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addre extern int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pte_t *ptep); -#define __HAVE_ARCH_PTEP_GET_AND_CLEAR -static inline pte_t ptep_get_and_clear(struct mm_struct *mm, - unsigned long address, pte_t *ptep) +static inline pte_t __ptep_get_and_clear(struct mm_struct *mm, + unsigned long address, pte_t *ptep) { pte_t pte = __pte(atomic_long_xchg((atomic_long_t *)ptep, 0)); @@ -730,6 +729,9 @@ static inline pte_t ptep_get(pte_t *ptep) #define ptep_get __ptep_get #endif /* CONFIG_RISCV_ISA_SVNAPOT */ +#define __HAVE_ARCH_PTEP_GET_AND_CLEAR +#define ptep_get_and_clear __ptep_get_and_clear + #define pgprot_nx pgprot_nx static inline pgprot_t pgprot_nx(pgprot_t _prot) { diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index d51863824540..0ecb2846c3f0 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -173,68 +173,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) return entry; } -static void clear_flush(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, - unsigned long pgsize, - unsigned long ncontig) -{ - struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); - unsigned long i, saddr = addr; - - for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) - ptep_get_and_clear(mm, addr, ptep); - - flush_tlb_range(&vma, saddr, addr); -} - -/* - * When dealing with NAPOT mappings, the privileged specification indicates that - * "if an update needs to be made, the OS generally should first mark all of the - * PTEs invalid, then issue SFENCE.VMA instruction(s) covering all 4 KiB regions - * within the range, [...] then update the PTE(s), as described in Section - * 4.2.1.". That's the equivalent of the Break-Before-Make approach used by - * arm64. - */ -void set_huge_pte_at(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, - pte_t pte, - unsigned long sz) -{ - unsigned long hugepage_shift, pgsize; - int i, pte_num; - - if (sz >= PGDIR_SIZE) - hugepage_shift = PGDIR_SHIFT; - else if (sz >= P4D_SIZE) - hugepage_shift = P4D_SHIFT; - else if (sz >= PUD_SIZE) - hugepage_shift = PUD_SHIFT; - else if (sz >= PMD_SIZE) - hugepage_shift = PMD_SHIFT; - else - hugepage_shift = PAGE_SHIFT; - - pte_num = sz >> hugepage_shift; - pgsize = 1 << hugepage_shift; - - if (!pte_present(pte)) { - for (i = 0; i < pte_num; i++, ptep++, addr += pgsize) - set_ptes(mm, addr, ptep, pte, 1); - return; - } - - if (!pte_napot(pte)) { - set_ptes(mm, addr, ptep, pte, 1); - return; - } - - clear_flush(mm, addr, ptep, pgsize, pte_num); - - set_ptes(mm, addr, ptep, pte, pte_num); -} - int huge_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, diff --git a/include/linux/hugetlb_contpte.h b/include/linux/hugetlb_contpte.h index ec4189cd65b8..7acd734a75e8 100644 --- a/include/linux/hugetlb_contpte.h +++ b/include/linux/hugetlb_contpte.h @@ -9,4 +9,9 @@ #define __HAVE_ARCH_HUGE_PTEP_GET extern pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep); +#define __HAVE_ARCH_HUGE_SET_HUGE_PTE_AT +extern void set_huge_pte_at(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, pte_t pte, + unsigned long sz); + #endif /* _LINUX_HUGETLB_CONTPTE_H */ diff --git a/mm/hugetlb_contpte.c b/mm/hugetlb_contpte.c index 0e3ba6f97c58..9a3a376784b0 100644 --- a/mm/hugetlb_contpte.c +++ b/mm/hugetlb_contpte.c @@ -10,6 +10,8 @@ /* * Any arch that wants to use that needs to define: * - __ptep_get() + * - __set_ptes() + * - __ptep_get_and_clear() * - pte_cont() * - arch_contpte_get_num_contig() */ @@ -17,6 +19,7 @@ /* * This file implements the following contpte aware API: * - huge_ptep_get() + * - set_huge_pte_at() */ pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep) @@ -42,3 +45,59 @@ pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep) } return orig_pte; } + +/* + * ARM64: Changing some bits of contiguous entries requires us to follow a + * Break-Before-Make approach, breaking the whole contiguous set + * before we can change any entries. See ARM DDI 0487A.k_iss10775, + * "Misprogramming of the Contiguous bit", page D4-1762. + * + * RISCV: When dealing with NAPOT mappings, the privileged specification + * indicates that "if an update needs to be made, the OS generally should first + * mark all of the PTEs invalid, then issue SFENCE.VMA instruction(s) covering + * all 4 KiB regions within the range, [...] then update the PTE(s), as + * described in Section 4.2.1.". That's the equivalent of the Break-Before-Make + * approach used by arm64. + * + * This helper performs the break step for use cases where the + * original pte is not needed. + */ +static void clear_flush(struct mm_struct *mm, + unsigned long addr, + pte_t *ptep, + unsigned long pgsize, + unsigned long ncontig) +{ + struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); + unsigned long i, saddr = addr; + + for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) + __ptep_get_and_clear(mm, addr, ptep); + + flush_tlb_range(&vma, saddr, addr); +} + +void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned long sz) +{ + size_t pgsize; + int i; + int ncontig; + + ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); + + if (!pte_present(pte)) { + for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) + __set_ptes(mm, addr, ptep, pte, 1); + return; + } + + if (!pte_cont(pte)) { + __set_ptes(mm, addr, ptep, pte, 1); + return; + } + + clear_flush(mm, addr, ptep, pgsize, ncontig); + + set_contptes(mm, addr, ptep, pte, ncontig, pgsize); +}