From patchwork Thu Apr 15 17:18:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 12205829 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96A28C433ED for ; Thu, 15 Apr 2021 17:20:57 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1CCBD61074 for ; Thu, 15 Apr 2021 17:20:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1CCBD61074 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=csgroup.eu Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:MIME-Version:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Date:Cc:To:Subject:From:References:In-Reply-To: Message-Id:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=2c8Ho8EV9pyM94jI0Oa4Kg0HL8cALKOxHJkGPEX8GbM=; b=IkEhzBPJXeJ3bRfSURYzwkjR1n p/2a9AxFdAdmLkaodG49r7jzIgtQErwdDV0PbrFa6xkHCiv2dHL1g/bhCnBhldW074x8sSmrO1H6Y f83X/14+Ixdgd+mECHxoFGPQbSz1w2VzvEA3mH/9MTx8A/RV3DM2iogPwHfpyo2j4qX/ksCONvzPN nKZdAjhtgbax2DbRIWRtHd/4OIMQSsj7R1b4oPX91ckzdYGI0YDEoTLVv8irIgPg67LQ6XIYyqL8X yUPWoVbBt6jpwAOSydcTypcNOldTDtKeC5OhoMB4bv4toUosCKRLgaz1k6J9qQuADCdKQ9GFARfSf LZjz7Wbg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lX5dW-00GqFP-Ds; Thu, 15 Apr 2021 17:18:38 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lX5dE-00GqBY-Sp; Thu, 15 Apr 2021 17:18:21 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Date:Cc:To:Subject:From:References: In-Reply-To:Message-Id:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description; bh=Btm2cic9GFZuY/f1ULdFW+o09h/gz/ykpjirvAgRoaM=; b=cm4EbBiM6I67ZpKgY/3bFPIdxW 7eahfN7S4lwzVmIBA0U/Vm/0Q2Coylx9gi/T1Rjh3VH9f0SnLr1bLE8IHuJKX5gcVxicI1fgVn6BJ 1cAMVr7h7u2uIh9n52YWOwYTn8nNeja99PNfiinJLqSLXZ2pXs+f2hASt6SGm33ZiWXjCuhCHpZgS 1Egyb2bTow+TZo4qSojDSPpjif9/hPPZHU3fPLVzjcN7wqXmN/xpaiNsLgRdQXzkDskUUQcSeCruJ xgzdxT6K3dGzVsaCluZQFb1Gxgx+cGQqplYhTOB8TtL4fhibJvJAhAJOfuouug78UjTvbPSPIMjAG bGk4MWlg==; Received: from pegase1.c-s.fr ([93.17.236.30]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lX5dB-008llw-E7; Thu, 15 Apr 2021 17:18:19 +0000 Received: from localhost (mailhub1-int [192.168.12.234]) by localhost (Postfix) with ESMTP id 4FLmJR3NnVz9vBLn; Thu, 15 Apr 2021 19:18:15 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [192.168.12.234]) (amavisd-new, port 10024) with ESMTP id C9R3Cqc6HllM; Thu, 15 Apr 2021 19:18:15 +0200 (CEST) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 4FLmJR2HvLz9tyvC; Thu, 15 Apr 2021 19:18:15 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 2924B8B804; Thu, 15 Apr 2021 19:18:15 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id G8IFx-cy7Cpg; Thu, 15 Apr 2021 19:18:15 +0200 (CEST) Received: from po16121vm.idsi0.si.c-s.fr (unknown [192.168.4.90]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 9F20A8B7F6; Thu, 15 Apr 2021 19:18:14 +0200 (CEST) Received: by po16121vm.idsi0.si.c-s.fr (Postfix, from userid 0) id 7E3BA679F6; Thu, 15 Apr 2021 17:18:14 +0000 (UTC) Message-Id: <733408f48b1ed191f53518123ee6fc6d42287cc6.1618506910.git.christophe.leroy@csgroup.eu> In-Reply-To: References: From: Christophe Leroy Subject: [PATCH v1 1/5] mm: pagewalk: Fix walk for hugepage tables To: Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , Steven Price , akpm@linux-foundation.org Cc: linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-riscv@lists.infradead.org, x86@kernel.org, linux-mm@kvack.org Date: Thu, 15 Apr 2021 17:18:14 +0000 (UTC) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210415_101817_812111_FEC77E0A X-CRM114-Status: GOOD ( 12.06 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Pagewalk ignores hugepd entries and walk down the tables as if it was traditionnal entries, leading to crazy result. Add walk_hugepd_range() and use it to walk hugepage tables. Signed-off-by: Christophe Leroy --- mm/pagewalk.c | 54 +++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 48 insertions(+), 6 deletions(-) diff --git a/mm/pagewalk.c b/mm/pagewalk.c index e81640d9f177..410a9d8f7572 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -58,6 +58,32 @@ static int walk_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, return err; } +static int walk_hugepd_range(hugepd_t *phpd, unsigned long addr, + unsigned long end, struct mm_walk *walk, int pdshift) +{ + int err = 0; +#ifdef CONFIG_ARCH_HAS_HUGEPD + const struct mm_walk_ops *ops = walk->ops; + int shift = hugepd_shift(*phpd); + int page_size = 1 << shift; + + if (addr & (page_size - 1)) + return 0; + + for (;;) { + pte_t *pte = hugepte_offset(*phpd, addr, pdshift); + + err = ops->pte_entry(pte, addr, addr + page_size, walk); + if (err) + break; + if (addr >= end - page_size) + break; + addr += page_size; + } +#endif + return err; +} + static int walk_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, struct mm_walk *walk) { @@ -108,7 +134,10 @@ static int walk_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, goto again; } - err = walk_pte_range(pmd, addr, next, walk); + if (is_hugepd(__hugepd(pmd_val(*pmd)))) + err = walk_hugepd_range((hugepd_t *)pmd, addr, next, walk, PMD_SHIFT); + else + err = walk_pte_range(pmd, addr, next, walk); if (err) break; } while (pmd++, addr = next, addr != end); @@ -157,7 +186,10 @@ static int walk_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, if (pud_none(*pud)) goto again; - err = walk_pmd_range(pud, addr, next, walk); + if (is_hugepd(__hugepd(pud_val(*pud)))) + err = walk_hugepd_range((hugepd_t *)pud, addr, next, walk, PUD_SHIFT); + else + err = walk_pmd_range(pud, addr, next, walk); if (err) break; } while (pud++, addr = next, addr != end); @@ -189,8 +221,13 @@ static int walk_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, if (err) break; } - if (ops->pud_entry || ops->pmd_entry || ops->pte_entry) - err = walk_pud_range(p4d, addr, next, walk); + if (ops->pud_entry || ops->pmd_entry || ops->pte_entry) { + if (is_hugepd(__hugepd(p4d_val(*p4d)))) + err = walk_hugepd_range((hugepd_t *)p4d, addr, next, walk, + P4D_SHIFT); + else + err = walk_pud_range(p4d, addr, next, walk); + } if (err) break; } while (p4d++, addr = next, addr != end); @@ -225,8 +262,13 @@ static int walk_pgd_range(unsigned long addr, unsigned long end, break; } if (ops->p4d_entry || ops->pud_entry || ops->pmd_entry || - ops->pte_entry) - err = walk_p4d_range(pgd, addr, next, walk); + ops->pte_entry) { + if (is_hugepd(__hugepd(pgd_val(*pgd)))) + err = walk_hugepd_range((hugepd_t *)pgd, addr, next, walk, + PGDIR_SHIFT); + else + err = walk_p4d_range(pgd, addr, next, walk); + } if (err) break; } while (pgd++, addr = next, addr != end);