From patchwork Mon Oct 7 15:38:11 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Price X-Patchwork-Id: 11177925 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2A17417D4 for ; Mon, 7 Oct 2019 15:39:18 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D8FBB21655 for ; Mon, 7 Oct 2019 15:39:17 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D8FBB21655 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B20AC8E0012; Mon, 7 Oct 2019 11:39:16 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id AABEF8E0003; Mon, 7 Oct 2019 11:39:16 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 94A058E0012; Mon, 7 Oct 2019 11:39:16 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0157.hostedemail.com [216.40.44.157]) by kanga.kvack.org (Postfix) with ESMTP id 707C68E0003 for ; Mon, 7 Oct 2019 11:39:16 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with SMTP id 09CB1824CA3B for ; Mon, 7 Oct 2019 15:39:16 +0000 (UTC) X-FDA: 76017397512.04.trail67_6df360546143c X-Spam-Summary: 2,0,0,7802a1fd2b622245,d41d8cd98f00b204,steven.price@arm.com,::steven.price@arm.com:luto@kernel.org:ard.biesheuvel@linaro.org:arnd@arndb.de:bp@alien8.de:catalin.marinas@arm.com:dave.hansen@linux.intel.com:mingo@redhat.com:james.morse@arm.com:jglisse@redhat.com:peterz@infradead.org:tglx@linutronix.de:will@kernel.org:x86@kernel.org:hpa@zytor.com:linux-arm-kernel@lists.infradead.org:linux-kernel@vger.kernel.org:mark.rutland@arm.com:kan.liang@linux.intel.com:akpm@linux-foundation.org,RULES_HIT:41:69:355:379:541:800:960:968:973:988:989:1260:1261:1311:1314:1345:1359:1437:1515:1535:1544:1605:1711:1730:1747:1777:1792:2393:2553:2559:2562:2693:2730:2898:3138:3139:3140:3141:3142:3865:3866:3867:3868:3871:3872:3874:4117:4250:4321:4605:5007:6119:6261:6742:6743:7875:7903:8634:9592:10004:11026:11658:11914:12296:12297:12517:12519:12555:12895:13141:13180:13229:13230:13894:14096:14181:14394:14721:21080:21451:21627:30054:30070:30090,0,RBL:217.140.110.172:@arm.com:.lbl8.mailshell. net-62.2 X-HE-Tag: trail67_6df360546143c X-Filterd-Recvd-Size: 6144 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf28.hostedemail.com (Postfix) with ESMTP for ; Mon, 7 Oct 2019 15:39:15 +0000 (UTC) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8625919BF; Mon, 7 Oct 2019 08:39:14 -0700 (PDT) Received: from e112269-lin.arm.com (unknown [10.1.197.21]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id EF0633F6C4; Mon, 7 Oct 2019 08:39:11 -0700 (PDT) From: Steven Price To: linux-mm@kvack.org Cc: Steven Price , Andy Lutomirski , Ard Biesheuvel , Arnd Bergmann , Borislav Petkov , Catalin Marinas , Dave Hansen , Ingo Molnar , James Morse , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Peter Zijlstra , Thomas Gleixner , Will Deacon , x86@kernel.org, "H. Peter Anvin" , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Mark Rutland , "Liang, Kan" , Andrew Morton Subject: [PATCH v11 11/22] mm: pagewalk: Add p4d_entry() and pgd_entry() Date: Mon, 7 Oct 2019 16:38:11 +0100 Message-Id: <20191007153822.16518-12-steven.price@arm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191007153822.16518-1-steven.price@arm.com> References: <20191007153822.16518-1-steven.price@arm.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: pgd_entry() and pud_entry() were removed by commit 0b1fbfe50006c410 ("mm/pagewalk: remove pgd_entry() and pud_entry()") because there were no users. We're about to add users so reintroduce them, along with p4d_entry() as we now have 5 levels of tables. Note that commit a00cc7d9dd93d66a ("mm, x86: add support for PUD-sized transparent hugepages") already re-added pud_entry() but with different semantics to the other callbacks. Since there have never been upstream users of this, revert the semantics back to match the other callbacks. This means pud_entry() is called for all entries, not just transparent huge pages. Signed-off-by: Steven Price --- include/linux/pagewalk.h | 19 +++++++++++++------ mm/pagewalk.c | 27 ++++++++++++++++----------- 2 files changed, 29 insertions(+), 17 deletions(-) diff --git a/include/linux/pagewalk.h b/include/linux/pagewalk.h index bddd9759bab9..12004b097eae 100644 --- a/include/linux/pagewalk.h +++ b/include/linux/pagewalk.h @@ -8,15 +8,15 @@ struct mm_walk; /** * mm_walk_ops - callbacks for walk_page_range - * @pud_entry: if set, called for each non-empty PUD (2nd-level) entry - * this handler should only handle pud_trans_huge() puds. - * the pmd_entry or pte_entry callbacks will be used for - * regular PUDs. - * @pmd_entry: if set, called for each non-empty PMD (3rd-level) entry + * @pgd_entry: if set, called for each non-empty PGD (top-level) entry + * @p4d_entry: if set, called for each non-empty P4D entry + * @pud_entry: if set, called for each non-empty PUD entry + * @pmd_entry: if set, called for each non-empty PMD entry * this handler is required to be able to handle * pmd_trans_huge() pmds. They may simply choose to * split_huge_page() instead of handling it explicitly. - * @pte_entry: if set, called for each non-empty PTE (4th-level) entry + * @pte_entry: if set, called for each non-empty PTE (lowest-level) + * entry * @pte_hole: if set, called for each hole at all levels * @hugetlb_entry: if set, called for each hugetlb entry * @test_walk: caller specific callback function to determine whether @@ -24,8 +24,15 @@ struct mm_walk; * "do page table walk over the current vma", returning * a negative value means "abort current page table walk * right now" and returning 1 means "skip the current vma" + * + * p?d_entry callbacks are called even if those levels are folded on a + * particular architecture/configuration. */ struct mm_walk_ops { + int (*pgd_entry)(pgd_t *pgd, unsigned long addr, + unsigned long next, struct mm_walk *walk); + int (*p4d_entry)(p4d_t *p4d, unsigned long addr, + unsigned long next, struct mm_walk *walk); int (*pud_entry)(pud_t *pud, unsigned long addr, unsigned long next, struct mm_walk *walk); int (*pmd_entry)(pmd_t *pmd, unsigned long addr, diff --git a/mm/pagewalk.c b/mm/pagewalk.c index d48c2a986ea3..fc4d98a3a5a0 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -93,15 +93,9 @@ static int walk_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, } if (ops->pud_entry) { - spinlock_t *ptl = pud_trans_huge_lock(pud, walk->vma); - - if (ptl) { - err = ops->pud_entry(pud, addr, next, walk); - spin_unlock(ptl); - if (err) - break; - continue; - } + err = ops->pud_entry(pud, addr, next, walk); + if (err) + break; } split_huge_pud(walk->vma, pud, addr); @@ -135,7 +129,12 @@ static int walk_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, break; continue; } - if (ops->pmd_entry || ops->pte_entry) + if (ops->p4d_entry) { + err = ops->p4d_entry(p4d, addr, next, walk); + if (err) + break; + } + if (ops->pud_entry || ops->pmd_entry || ops->pte_entry) err = walk_pud_range(p4d, addr, next, walk); if (err) break; @@ -162,7 +161,13 @@ static int walk_pgd_range(unsigned long addr, unsigned long end, break; continue; } - if (ops->pmd_entry || ops->pte_entry) + if (ops->pgd_entry) { + err = ops->pgd_entry(pgd, addr, next, walk); + if (err) + break; + } + if (ops->p4d_entry || ops->pud_entry || ops->pmd_entry || + ops->pte_entry) err = walk_p4d_range(pgd, addr, next, walk); if (err) break;