From patchwork Fri Aug 27 15:04:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: SeongJae Park X-Patchwork-Id: 12462279 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84CA1C432BE for ; Fri, 27 Aug 2021 15:04:13 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 322DB60EFF for ; Fri, 27 Aug 2021 15:04:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 322DB60EFF Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id BA6A18D0001; Fri, 27 Aug 2021 11:04:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B57436B0071; Fri, 27 Aug 2021 11:04:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A1EED8D0001; Fri, 27 Aug 2021 11:04:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0057.hostedemail.com [216.40.44.57]) by kanga.kvack.org (Postfix) with ESMTP id 84CB06B006C for ; Fri, 27 Aug 2021 11:04:12 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 37A521818FCC4 for ; Fri, 27 Aug 2021 15:04:12 +0000 (UTC) X-FDA: 78521181144.22.E81914E Received: from mail-qv1-f45.google.com (mail-qv1-f45.google.com [209.85.219.45]) by imf15.hostedemail.com (Postfix) with ESMTP id E837AD0000A2 for ; Fri, 27 Aug 2021 15:04:11 +0000 (UTC) Received: by mail-qv1-f45.google.com with SMTP id a5so4206809qvq.0 for ; Fri, 27 Aug 2021 08:04:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=op2QVF7ofZpANc1CMzE9ciWUBXUVrosnUhzGCjoQSlU=; b=vbYlZelwKh8Sl1ByA94JNKLxNAbvyVPJNZYQdKIORBxqH8DA5iOQnbkBUbib2yX5dC if2PnA3BlvTyGKDHPo/hVZtsdhdbZpAzdkOEZmF16y6KCPcveFOhPXQkCfZ5olWQUpPi mDUdFgUsxacDJ5vjp3hK0dCIK8QrrbuXaTmVwxayQ5egHtd2MumLwiGVVMT2ilCv9yUH 3hG2OsSrPZjubhWYhoCJ83ORNj8YpjvJk2bg3OmkmKukqlpeHylQWSFpLyH9FOR5NmCT sW9XQvRBFLaPACgbS2bF514h9ZMdbtIvp9ERE3BVsQ3n0qLoqs29Xw85UrICj0NXNGXQ ITOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=op2QVF7ofZpANc1CMzE9ciWUBXUVrosnUhzGCjoQSlU=; b=HEJiWUG5t8mQJ0RCgFrMANoh6ypFb5hTOmX8aEstZYd+YuwtLs7OMp8nTZwLdQPKVo Flqy3AzMBSgnY36Le9h9b6IBHxWb7NPXPoz7LjJ0j9n0NYPjW3hgYCXFU62Vf5MpF2GR 4o3lA9frzUkX7ecTOPlBH0/tC4WZjWMJoz1QTdvW6q9oLw3fYp6DUAWZtNEdUs7WsdoG JiiFgw9NnzpmTRkbj9WQGdgyXci6KlBtP4AGZ5BLgSWI5RGlkSywHjjLluFUH1LamvX9 H6NvwY9qi3orAVx6jtPxuA+LPF//g7UOmHtcCSNPVB3zNGZoinU3Ma/JOh/x54Aw8BK3 0kmA== X-Gm-Message-State: AOAM530gHoLC6UGdDwQQwjKcjUxHQdLUIEEJrcQqX6asQUT/Fx1UErBO nT/09rctgZJZm0t7DpS3c1YdRY8K1/fOrg== X-Google-Smtp-Source: ABdhPJxLp0R/ieXX1YdB/sEdguqUzsVUVLdaa9tGsB4/JIL2IVGh0T3SCkZ4VyqCP7KJihUiiTheqQ== X-Received: by 2002:a05:6214:194b:: with SMTP id q11mr9954664qvk.33.1630076651299; Fri, 27 Aug 2021 08:04:11 -0700 (PDT) Received: from localhost.localdomain (ec2-35-169-212-159.compute-1.amazonaws.com. [35.169.212.159]) by smtp.gmail.com with ESMTPSA id m187sm4796385qkd.131.2021.08.27.08.04.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 27 Aug 2021 08:04:10 -0700 (PDT) From: SeongJae Park To: akpm@linux-foundation.org Cc: david@redhat.com, markubo@amazon.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, SeongJae Park Subject: [PATCH] mm/damon/vaddr: Safely walk page table Date: Fri, 27 Aug 2021 15:04:00 +0000 Message-Id: <20210827150400.6305-1-sj38.park@gmail.com> X-Mailer: git-send-email 2.17.1 Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=vbYlZelw; spf=pass (imf15.hostedemail.com: domain of sj38park@gmail.com designates 209.85.219.45 as permitted sender) smtp.mailfrom=sj38park@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: E837AD0000A2 X-Stat-Signature: 4175o5u9ife63oyogrrxzirugxus6x9d X-HE-Tag: 1630076651-467584 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: SeongJae Park Commit d7f647622761 ("mm/damon: implement primitives for the virtual memory address spaces") of linux-mm[1] tries to find PTE or PMD for arbitrary virtual address using 'follow_invalidate_pte()' without proper locking[2]. This commit fixes the issue by using another page table walk function for more general use case under proper locking. [1] https://github.com/hnaz/linux-mm/commit/d7f647622761 [2] https://lore.kernel.org/linux-mm/3b094493-9c1e-6024-bfd5-7eca66399b7e@redhat.com Fixes: d7f647622761 ("mm/damon: implement primitives for the virtual memory address spaces") Reported-by: David Hildenbrand Signed-off-by: SeongJae Park Reviewed-by: Markus Boehme --- mm/damon/vaddr.c | 81 +++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 74 insertions(+), 7 deletions(-) diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c index 230db7413278..b3677f2ef54b 100644 --- a/mm/damon/vaddr.c +++ b/mm/damon/vaddr.c @@ -8,10 +8,12 @@ #define pr_fmt(fmt) "damon-va: " fmt #include +#include #include #include #include #include +#include #include #include #include @@ -446,14 +448,69 @@ static void damon_pmdp_mkold(pmd_t *pmd, struct mm_struct *mm, #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ } +struct damon_walk_private { + pmd_t *pmd; + pte_t *pte; + spinlock_t *ptl; +}; + +static int damon_pmd_entry(pmd_t *pmd, unsigned long addr, unsigned long next, + struct mm_walk *walk) +{ + struct damon_walk_private *priv = walk->private; + + if (pmd_huge(*pmd)) { + priv->ptl = pmd_lock(walk->mm, pmd); + if (pmd_huge(*pmd)) { + priv->pmd = pmd; + return 0; + } + spin_unlock(priv->ptl); + } + + if (pmd_none(*pmd) || unlikely(pmd_bad(*pmd))) + return -EINVAL; + priv->pte = pte_offset_map_lock(walk->mm, pmd, addr, &priv->ptl); + if (!pte_present(*priv->pte)) { + pte_unmap_unlock(priv->pte, priv->ptl); + priv->pte = NULL; + return -EINVAL; + } + return 0; +} + +static struct mm_walk_ops damon_walk_ops = { + .pmd_entry = damon_pmd_entry, +}; + +int damon_follow_pte_pmd(struct mm_struct *mm, unsigned long addr, + struct damon_walk_private *private) +{ + int rc; + + private->pte = NULL; + private->pmd = NULL; + rc = walk_page_range(mm, addr, addr + 1, &damon_walk_ops, private); + if (!rc && !private->pte && !private->pmd) + return -EINVAL; + return rc; +} + static void damon_va_mkold(struct mm_struct *mm, unsigned long addr) { - pte_t *pte = NULL; - pmd_t *pmd = NULL; + struct damon_walk_private walk_result; + pte_t *pte; + pmd_t *pmd; spinlock_t *ptl; - if (follow_invalidate_pte(mm, addr, NULL, &pte, &pmd, &ptl)) + mmap_write_lock(mm); + if (damon_follow_pte_pmd(mm, addr, &walk_result)) { + mmap_write_unlock(mm); return; + } + pte = walk_result.pte; + pmd = walk_result.pmd; + ptl = walk_result.ptl; if (pte) { damon_ptep_mkold(pte, mm, addr); @@ -462,6 +519,7 @@ static void damon_va_mkold(struct mm_struct *mm, unsigned long addr) damon_pmdp_mkold(pmd, mm, addr); spin_unlock(ptl); } + mmap_write_unlock(mm); } /* @@ -495,14 +553,21 @@ void damon_va_prepare_access_checks(struct damon_ctx *ctx) static bool damon_va_young(struct mm_struct *mm, unsigned long addr, unsigned long *page_sz) { - pte_t *pte = NULL; - pmd_t *pmd = NULL; + struct damon_walk_private walk_result; + pte_t *pte; + pmd_t *pmd; spinlock_t *ptl; struct page *page; bool young = false; - if (follow_invalidate_pte(mm, addr, NULL, &pte, &pmd, &ptl)) + mmap_write_lock(mm); + if (damon_follow_pte_pmd(mm, addr, &walk_result)) { + mmap_write_unlock(mm); return false; + } + pte = walk_result.pte; + pmd = walk_result.pmd; + ptl = walk_result.ptl; *page_sz = PAGE_SIZE; if (pte) { @@ -513,7 +578,7 @@ static bool damon_va_young(struct mm_struct *mm, unsigned long addr, if (page) put_page(page); pte_unmap_unlock(pte, ptl); - return young; + goto out; } #ifdef CONFIG_TRANSPARENT_HUGEPAGE @@ -528,6 +593,8 @@ static bool damon_va_young(struct mm_struct *mm, unsigned long addr, *page_sz = ((1UL) << HPAGE_PMD_SHIFT); #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +out: + mmap_write_unlock(mm); return young; }