From patchwork Tue Feb 28 12:23:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yin Fengwei X-Patchwork-Id: 13154856 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85603C64EC7 for ; Tue, 28 Feb 2023 12:22:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 56CAC6B007B; Tue, 28 Feb 2023 07:22:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 47E4D6B0080; Tue, 28 Feb 2023 07:22:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 238966B007B; Tue, 28 Feb 2023 07:22:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id F3E006B007B for ; Tue, 28 Feb 2023 07:22:30 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id C84B0120321 for ; Tue, 28 Feb 2023 12:22:30 +0000 (UTC) X-FDA: 80516613660.28.F8F8AB6 Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by imf23.hostedemail.com (Postfix) with ESMTP id B6C23140004 for ; Tue, 28 Feb 2023 12:22:28 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=d200wGKY; spf=pass (imf23.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677586948; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LD1lWbFwW7ZVR5pqFBwcUmIBn+MRU/gnJ+vXlQHQPa4=; b=Z/MjZuvmDgfHaJmvwTisocKxvwYhHQrCorPqmpFILzc+ZREEccvmeol0FfuD/I3nUzM8uL VgvupeumPcZWonggUSShuDIaKuc+aHMnaUS9zXkBUn7ckL9JwE47DD3yOFtXD5ikUhsQTy 859EH5wHwAhsVkBoe9uh9keLHhY3sgI= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=d200wGKY; spf=pass (imf23.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677586949; a=rsa-sha256; cv=none; b=Xk8rwCJauzgp6HHaFw4I8bMBNjR0r6OpANu0aaLT/8zbo/XNLJqyGUOk91XD3Te4xZiemn ON+0z+KNOUkVULOmMYoDk9aJYFdQYoWu4bGHTrA1fd5Th337tU8WdEO0dZrsxb9eOq6J+H AsUXUmQ4IU7E/sK7aj9eFEyUavCT/aQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677586948; x=1709122948; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=VMMRS6YvSAo+8kzloSgN4rxmLa9+TgipJdVxggMXU3w=; b=d200wGKYxcbBj9bqj8O1+q8dLuoPCTzp8GIOftlynVHGeqo42v6ICt37 EEYD2TcE3CLZPlAJ3QoohyJWBD11cMagxPFE2oCuwglqasPl8401/yOAe fCnMytAbT8Cr2Sb3pJ63JP5EV5LTgBCCvOGRrLf/55M6WTkvLyKATVz2d TRZO89odGEf53kmgEfE5pa7ybIH22eqW2nQmpHYqReeajdsOZDTqsdNC7 zy7FLs4ks8ZGifY9cUAsBMMLMS5WlJhaSsc6lVRF5BUFdpaeDdNMUYVqS NVHA0+oTc+bxy0jTwYQ3NhQcM+/LPrFFyEJZtUXClDQ9tstebv8hLyMOG g==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="317921179" X-IronPort-AV: E=Sophos;i="5.98,221,1673942400"; d="scan'208";a="317921179" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Feb 2023 04:22:14 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="1003220782" X-IronPort-AV: E=Sophos;i="5.98,221,1673942400"; d="scan'208";a="1003220782" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by fmsmga005.fm.intel.com with ESMTP; 28 Feb 2023 04:22:05 -0800 From: Yin Fengwei To: linux-mm@kvack.org, akpm@linux-foundation.org, willy@infradead.org, sidhartha.kumar@oracle.com, mike.kravetz@oracle.com, jane.chu@oracle.com, naoya.horiguchi@nec.com Cc: fengwei.yin@intel.com Subject: [PATCH v2 4/5] rmap:addd folio_remove_rmap_range() Date: Tue, 28 Feb 2023 20:23:07 +0800 Message-Id: <20230228122308.2972219-5-fengwei.yin@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230228122308.2972219-1-fengwei.yin@intel.com> References: <20230228122308.2972219-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Stat-Signature: 3z7oz5yzp81sd348u9g4fjmmbw3aohbf X-Rspam-User: X-Rspamd-Queue-Id: B6C23140004 X-Rspamd-Server: rspam06 X-HE-Tag: 1677586948-603745 X-HE-Meta: U2FsdGVkX19yXyVMs2foEZhaKAoZnzR25NHNF/U0du7OrEiQgo66DlpC5mA0zlFQ5bQcTrkTA76AbOcjGU+JiEGqVwlZE77tvT829Uo6wJda1YaxtMrYzgTR0wgFdd6enVVX7LCRg74nzm55qytZ3VtJscJ9dwB/1M2R7rraCqk/iqiVx8rIwH7ZIsWgb6mmOZc8NfbRyGr8/6hg+t4jIrkDzDkeB/kUciY+BCAk3MNaUMu14ImDKRmH0Fbge4bOSzLyes7KkzpN57tedKbvq1oKNpnjl4+VuqsA3SBYjCeBho7q82RiPbejjwgTuQy2RrUhm8N4Vi5O+uQxepuot5+mbcP2yLHxwDy+6KdzDAa00mkkzUlRtsexhaRe6k3fyA94xUhjflnxMn812LsjzK3S90a6dsDfUh8PgvJ7iRxMD8zRDYcH4yc0RV5yAVOeZlejA2aWIYHp+iU2/SBPsTQCqrC0hhwEmgzLJ56vq3cCqiKWHXlH6wpy3lNLqHJifnz4PwsXljYufJzAJwJxK+22cSCzScSsRzFkqfF4hd+TkqMgRScV0lgdoN+lbtxR3EF/r/7nRvG9sQ+G6kETYDI/zQxiGPezR5MQtXAIn0h9L+LGxFSEH6hAyEw6Z+J8U1nE2dKUfDQ4/oMOMNDWmqSG2iFdO/smB0RG6s15iB+2NtfOxsE0x5G0AfrXCKRvhQ76ATxSH2MZBqKi2oRBaN1Y9buN1KIzqZaCiUMcngtiPLl02zw2G3vhhhbsOGtSIGfzU3VIJUaeUpgm0KmBZ9k2GjBcM6UKiAZU6hibgzHLb30DgL+cw1wPN7CRIRKp4JGOKptsDiaWqEovXPClFW0+QnDCvPo/RZQiXkyiF28Ep1WU+bCDu7Q/3PvhjKXENuw9z6+gE5Fnuz4EOyLze604JantXeSKx8k+CTQCGMCO3l9j99yHgUA5ueHIKbgQTp5JfQ0PHrnyUa1B92g mlm5rFlG i967vHjIyFV9o7VNwY4YS+jCLs7QlaeQ9yQOiADOZI+jDYIEq5pxHJWDJE1zdChvIwbqltWbHoLxF1FtN29lDINzv1hVvLYdzyFY0g6GKQaHfF6sWl9GL5U2aC9wuEG+u7vxL5I0nIim/sFAm+YRLKjA5eCSdfzXFXC2xS52D2k2uZ1+fmD4ZOAJ+udaXHaiU2mN5ChqKQW9/dNIiwfoWMd8rCMYu/4JLvVI9n9WJidEvdwtiEuDc7uOHz8pQ+ha1CaSU X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: folio_remove_rmap_range() allows to take down the pte mapping to a specific range of folio. Comparing to page_remove_rmap(), it batched updates __lruvec_stat for large folio. Signed-off-by: Yin Fengwei --- include/linux/rmap.h | 4 +++ mm/rmap.c | 58 +++++++++++++++++++++++++++++++++----------- 2 files changed, 48 insertions(+), 14 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index b87d01660412..d2569b42e21a 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -200,6 +200,10 @@ void page_add_file_rmap(struct page *, struct vm_area_struct *, bool compound); void page_remove_rmap(struct page *, struct vm_area_struct *, bool compound); +void folio_remove_rmap_range(struct folio *, struct page *, + unsigned int nr_pages, struct vm_area_struct *, + bool compound); + void hugepage_add_anon_rmap(struct page *, struct vm_area_struct *, unsigned long address, rmap_t flags); diff --git a/mm/rmap.c b/mm/rmap.c index d243e557c6e4..fc02a8f9c59c 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1357,23 +1357,25 @@ void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, } /** - * page_remove_rmap - take down pte mapping from a page - * @page: page to remove mapping from + * folio_remove_rmap_range - take down pte mapping from a range of pages + * @folio: folio to remove mapping from + * @page: The first page to take down pte mapping + * @nr_pages: The number of pages which will be take down pte mapping * @vma: the vm area from which the mapping is removed * @compound: uncharge the page as compound or small page * * The caller needs to hold the pte lock. */ -void page_remove_rmap(struct page *page, struct vm_area_struct *vma, - bool compound) +void folio_remove_rmap_range(struct folio *folio, struct page *page, + unsigned int nr_pages, struct vm_area_struct *vma, + bool compound) { - struct folio *folio = page_folio(page); atomic_t *mapped = &folio->_nr_pages_mapped; - int nr = 0, nr_pmdmapped = 0; - bool last; + int nr = 0, nr_pmdmapped = 0, last; enum node_stat_item idx; - VM_BUG_ON_PAGE(compound && !PageHead(page), page); + VM_BUG_ON_FOLIO(compound && (nr_pages != folio_nr_pages(folio)), folio); + VM_BUG_ON_FOLIO(compound && (page != &folio->page), folio); /* Hugetlb pages are not counted in NR_*MAPPED */ if (unlikely(folio_test_hugetlb(folio))) { @@ -1384,12 +1386,16 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, /* Is page being unmapped by PTE? Is this its last map to be removed? */ if (likely(!compound)) { - last = atomic_add_negative(-1, &page->_mapcount); - nr = last; - if (last && folio_test_large(folio)) { - nr = atomic_dec_return_relaxed(mapped); - nr = (nr < COMPOUND_MAPPED); - } + do { + last = atomic_add_negative(-1, &page->_mapcount); + if (last && folio_test_large(folio)) { + last = atomic_dec_return_relaxed(mapped); + last = (last < COMPOUND_MAPPED); + } + + if (last) + nr++; + } while (page++, --nr_pages > 0); } else if (folio_test_pmd_mappable(folio)) { /* That test is redundant: it's for safety or to optimize out */ @@ -1443,6 +1449,30 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, munlock_vma_folio(folio, vma, compound); } +/** + * page_remove_rmap - take down pte mapping from a page + * @page: page to remove mapping from + * @vma: the vm area from which the mapping is removed + * @compound: uncharge the page as compound or small page + * + * The caller needs to hold the pte lock. + */ +void page_remove_rmap(struct page *page, struct vm_area_struct *vma, + bool compound) +{ + struct folio *folio = page_folio(page); + unsigned int nr_pages; + + VM_BUG_ON_FOLIO(compound && (page != &folio->page), folio); + + if (likely(!compound)) + nr_pages = 1; + else + nr_pages = folio_nr_pages(folio); + + folio_remove_rmap_range(folio, page, nr_pages, vma, compound); +} + static bool try_to_unmap_one_hugetlb(struct folio *folio, struct vm_area_struct *vma, struct mmu_notifier_range range, struct page_vma_mapped_walk pvmw, unsigned long address,