From patchwork Tue Feb 28 12:23:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yin Fengwei X-Patchwork-Id: 13154854 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93EE9C7EE32 for ; Tue, 28 Feb 2023 12:22:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A4EA16B0078; Tue, 28 Feb 2023 07:22:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 98A776B007B; Tue, 28 Feb 2023 07:22:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 828956B007E; Tue, 28 Feb 2023 07:22:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 6D7FE6B0078 for ; Tue, 28 Feb 2023 07:22:30 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 234F4160CF6 for ; Tue, 28 Feb 2023 12:22:30 +0000 (UTC) X-FDA: 80516613660.01.4B99F8A Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by imf14.hostedemail.com (Postfix) with ESMTP id 09330100002 for ; Tue, 28 Feb 2023 12:22:27 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=P3XIGi8r; spf=pass (imf14.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677586948; a=rsa-sha256; cv=none; b=CKT1Cl8fgoadlNNmJAZDf5MWnz2bzjmMlImP2n22fzVZSDHiX+9d6uXRBbSSru0pf16JG7 d/x4UhpwLeH6QLXLzr6wFba8Ca0ampUlUD7zyRUb6rLPkI05xh4cLkBU1+UR1Rl1OJ+dLO cxRRBDC8jPoziIcQlkec6fW6mDO+v5k= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=P3XIGi8r; spf=pass (imf14.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677586948; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8GuT8ZTejFpZQvJKAhqGky6Mm6hpqAjqKMe23Yn3ckk=; b=1X1acH/BmAgjNUwl40i8wLCkAAxIjKYAoJgdu9G/F9LeWZiNrgoo4iTDpCujkhyaAP9Q7G 3ITBby0TF0imUBRzMTJixabcySSJtONU4dT1CJQ0zFk9idjjOSKQT1P43rtZx874JHLbjr Ajdjk3gxqjsYLP9+Ns9GpLSFiAEzMSg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677586948; x=1709122948; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=UpSIPZ+9DCUWOZcnzjq7arbpb2lEc08dm0ksjgfOGqo=; b=P3XIGi8rjNLikbI0D7vyZgWPpTceK8YgP16yzZcel7byVcggy5Otxrvb AnznAcg1BscFRt/szL6ZPg/hVSgcey0xWu4uaAiNKVkEGMfL+PHadFTYe OTtE68KsfYKIBZJPa5qFxwM48vlI0cbYt2iVNh6gBHPRRDE3qD6zfrpVK sxCGOFAHlp+uhkCyEq0MeO4Qqf5g+hKeChIx7rP3oeamS2zOVVvO6avVD rHEGfs/F9Qds2mCqszf7vaCsSErcsmI+tEy04ONGKN+j0wo5o3IYAP272 RYFJ+EOk9cH9T8lJSi3x7Vt3K1L6lnN5dfbmiY5lP33ZqJPSxqNNp6zmy Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="317921184" X-IronPort-AV: E=Sophos;i="5.98,221,1673942400"; d="scan'208";a="317921184" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Feb 2023 04:22:14 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10634"; a="1003220784" X-IronPort-AV: E=Sophos;i="5.98,221,1673942400"; d="scan'208";a="1003220784" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by fmsmga005.fm.intel.com with ESMTP; 28 Feb 2023 04:22:07 -0800 From: Yin Fengwei To: linux-mm@kvack.org, akpm@linux-foundation.org, willy@infradead.org, sidhartha.kumar@oracle.com, mike.kravetz@oracle.com, jane.chu@oracle.com, naoya.horiguchi@nec.com Cc: fengwei.yin@intel.com Subject: [PATCH v2 5/5] try_to_unmap_one: batched remove rmap, update folio refcount Date: Tue, 28 Feb 2023 20:23:08 +0800 Message-Id: <20230228122308.2972219-6-fengwei.yin@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230228122308.2972219-1-fengwei.yin@intel.com> References: <20230228122308.2972219-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 09330100002 X-Rspamd-Server: rspam01 X-Stat-Signature: 9crht8wkkuaaifonebnh44dpj9r3ie38 X-HE-Tag: 1677586947-430073 X-HE-Meta: U2FsdGVkX18ijwOiJG3ucikeMYQm/uw+FvDE3bnNPxOxVr51gNohCmCaIrwDc5m8+sC1OlDUaIDmYmUsX28uIT3s1Y8ogargvjdEq4pfZhkkXTu0j+VLKKETeysa80QGXiFcLssSRkLCivdJNzFSGyyE7lRHRZwvSBo5zTL7ouhjmmtdG10awsMZJ8as6xDhBWcy9BoBjx6vRByhmENhay7abaHbTmLyXEsDhOSsOSgbfQPjy384u4vV26Oh2RiVT7+7pKPrTQFjTA4Zu1LkEtYmR1TzH/+2hO893Sl/ZEi1jvybywAD7e+4dt2T69O+SUrTpcRFepnLgj4nh8VKaYThzK624OMSRiNvtUuqGUu5+Cp05fSx50jWZZ9533obb3AIuF/3no5km+MZg9jbw41onNFPldMIeLx5R+HfnxJCE+VueyZOqsPsZOaLvCJ+labmv4Dlps/0K1Lkqd29bmRomG2obFVdfOqcUm8EqUuSXetpU7XdGvHAFZlGW9H8B5F2Axdez40mh2QWyhgoRfOoorbKso9Fp/uDaNDahuMnlDv5bCAyVwpiiZ77czNyKTw5mfHDMqOGLXg9j/1b28/AK+gSrHLMiRcgre3HEYV9EOtHh3DPpjgogGrMXDwAK4a4fznR6lFR8Kxi8VCgLZP5tNlzGu5sm6gZP8pv6tQ6flMuTbkQeIOUpkE8FQDgoI9gYX4DNbqpFsNOHsIVqvx+UvU+fvkL1mQBWin3MtKOlja4mCzC7ejlkY8ZpvyjbYsMJbSwQcPDyX8fACFQkHn8mki3icpYYq1QeX0YVW24kNFA7zS1/t024Mfi3lvQnoSYkyrbMlqPPFGyh4oX2CfQ+dDXeCEIIGXJuTjnlIZDOxZJqGCzQSabg9lp/K5LPB47mZu/Z5nphZbMUf79877Bb7lyAPjk60B3DimdVvJ2BnKz7kJkreo18Qm5a/bqj9gTlDblyL5iCLyVI6t 4jJpcEcT ApXUgb6VS5Ylf+61u2oDC01qeMpO0Rw61GHDZgyg6HLwBZQRe8eODk3w7K2XiW0/gfSuPN9+mtu0D3Sgnyh9nqfaAbKj05VI92iRRhxHcWE18QykFygHZNSE4VrLWtt3nqLKYjkMowNZ//eXQVgYuHudo+lSqBgnKxKbIMIU1NOPubPifqWpz5J3MoThDjkVIgsfVju4njKdBYv1fie3c2KWbmCwD5eyYkRu/ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If unmap one page fails, or the vma walk will skip next pte, or the vma walk will end on next pte, batched remove map, update folio refcount. Signed-off-by: Yin Fengwei --- include/linux/rmap.h | 1 + mm/page_vma_mapped.c | 30 +++++++++++++++++++++++++++ mm/rmap.c | 48 ++++++++++++++++++++++++++++++++++---------- 3 files changed, 68 insertions(+), 11 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index d2569b42e21a..18193d1d5a8e 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -424,6 +424,7 @@ static inline void page_vma_mapped_walk_done(struct page_vma_mapped_walk *pvmw) } bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw); +bool pvmw_walk_skip_or_end_on_next(struct page_vma_mapped_walk *pvmw); /* * Used by swapoff to help locate where page is expected in vma. diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 4e448cfbc6ef..19e997dfb5c6 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -291,6 +291,36 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) return false; } +/** + * pvmw_walk_skip_or_end_on_next - check if next pte will be skipped or + * end the walk + * @pvmw: pointer to struct page_vma_mapped_walk. + * + * This function can only be called with correct pte lock hold + */ +bool pvmw_walk_skip_or_end_on_next(struct page_vma_mapped_walk *pvmw) +{ + unsigned long address = pvmw->address + PAGE_SIZE; + + if (address >= vma_address_end(pvmw)) + return true; + + if ((address & (PMD_SIZE - PAGE_SIZE)) == 0) + return true; + + if (pte_none(*pvmw->pte)) + return true; + + pvmw->pte++; + if (!check_pte(pvmw)) { + pvmw->pte--; + return true; + } + pvmw->pte--; + + return false; +} + /** * page_mapped_in_vma - check whether a page is really mapped in a VMA * @page: the page to test diff --git a/mm/rmap.c b/mm/rmap.c index fc02a8f9c59c..a6ed95b89078 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1743,6 +1743,26 @@ static bool try_to_unmap_one_page(struct folio *folio, return false; } +static void folio_remove_rmap_and_update_count(struct folio *folio, + struct page *start, struct vm_area_struct *vma, int count) +{ + if (count == 0) + return; + + /* + * No need to call mmu_notifier_invalidate_range() it has be + * done above for all cases requiring it to happen under page + * table lock before mmu_notifier_invalidate_range_end() + * + * See Documentation/mm/mmu_notifier.rst + */ + folio_remove_rmap_range(folio, start, count, vma, + folio_test_hugetlb(folio)); + if (vma->vm_flags & VM_LOCKED) + mlock_drain_local(); + folio_ref_sub(folio, count); +} + /* * @arg: enum ttu_flags will be passed to this argument */ @@ -1750,10 +1770,11 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, unsigned long address, void *arg) { DEFINE_FOLIO_VMA_WALK(pvmw, folio, vma, address, 0); - struct page *subpage; + struct page *start = NULL; bool ret = true; struct mmu_notifier_range range; enum ttu_flags flags = (enum ttu_flags)(long)arg; + int count = 0; /* * When racing against e.g. zap_pte_range() on another cpu, @@ -1814,26 +1835,31 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, break; } - subpage = folio_page(folio, + if (!start) + start = folio_page(folio, pte_pfn(*pvmw.pte) - folio_pfn(folio)); ret = try_to_unmap_one_page(folio, vma, range, pvmw, address, flags); if (!ret) { + folio_remove_rmap_and_update_count(folio, + start, vma, count); page_vma_mapped_walk_done(&pvmw); break; } + count++; /* - * No need to call mmu_notifier_invalidate_range() it has be - * done above for all cases requiring it to happen under page - * table lock before mmu_notifier_invalidate_range_end() - * - * See Documentation/mm/mmu_notifier.rst + * If next pte will be skipped in page_vma_mapped_walk() or + * the walk will end at it, batched remove rmap and update + * page refcount. We can't do it after page_vma_mapped_walk() + * return false because the pte lock will not be hold. */ - page_remove_rmap(subpage, vma, false); - if (vma->vm_flags & VM_LOCKED) - mlock_drain_local(); - folio_put(folio); + if (pvmw_walk_skip_or_end_on_next(&pvmw)) { + folio_remove_rmap_and_update_count(folio, + start, vma, count); + count = 0; + start = NULL; + } } mmu_notifier_invalidate_range_end(&range);