From patchwork Thu Aug 29 16:56:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13783477 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76F6FC87FC8 for ; Thu, 29 Aug 2024 16:58:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0874B6B00A5; Thu, 29 Aug 2024 12:58:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 039026B00A6; Thu, 29 Aug 2024 12:58:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DF3246B00A7; Thu, 29 Aug 2024 12:58:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id BDAFE6B00A5 for ; Thu, 29 Aug 2024 12:58:51 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 796E4402A2 for ; Thu, 29 Aug 2024 16:58:51 +0000 (UTC) X-FDA: 82505892462.10.C40AB07 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf01.hostedemail.com (Postfix) with ESMTP id C6D4F40008 for ; Thu, 29 Aug 2024 16:58:49 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="K0rzA/mM"; spf=pass (imf01.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724950685; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Xlz/Hr/MLXsaGBmeoyntSanirDE8WH40+jKnaAIxWt4=; b=571agoWGTK0B8VZYugeG5crrCgHuiQlbhtAE1BB+ReQdvTVzyC7YZjMov1v3JqtqGnwUO4 dZQEcTw5tDPKSH6xx/OMuTQ01xU8MkRwofyz9EQz6ALKXewAdZ0AR6sW+nzzf/RunVpySk OxkkKMUrSViI6necZc0DWt4dg+DudKk= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="K0rzA/mM"; spf=pass (imf01.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724950685; a=rsa-sha256; cv=none; b=1E1puELzW5gFhDy30ZcbpDwqBgLhf6wBtdiPc8gLyrbPJuVnR2S0YaKtq48oauh7VvzwLX F1Mzc5iDmKyCldBvT0u5+WcZXiMCNRJXozLpzuB2NOG9G1JrXmahuzJENIcvsCoG+e5HxW BgALeUhB7ZSpafmTuRxKankWJEnc+Xc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724950729; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Xlz/Hr/MLXsaGBmeoyntSanirDE8WH40+jKnaAIxWt4=; b=K0rzA/mMYkREbPB/ZxPvtNTSq/BTkJ78ln5P4CVal1SJ/OnWNN70qityKS8WvxRK2yEyK4 BtJsJclEX4EfppDmcqq4fh3LTAo/zMFyvCTk29hBNa8JxQf0yi/ciefF3dJUks0Jx17dZ7 TCo47BBj2Q5jp6nJ7g7UuDGU9eSr3Vg= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-302-FFTH9f1IPp2QjlcNQc8UoQ-1; Thu, 29 Aug 2024 12:58:45 -0400 X-MC-Unique: FFTH9f1IPp2QjlcNQc8UoQ-1 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id A6ADE190308D; Thu, 29 Aug 2024 16:58:43 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.193.245]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 3F3A51955F21; Thu, 29 Aug 2024 16:58:31 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, cgroups@vger.kernel.org, x86@kernel.org, linux-fsdevel@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Johannes Weiner , =?utf-8?q?Michal_Koutn=C3=BD?= , Jonathan Corbet , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen Subject: [PATCH v1 12/17] mm: remove per-page mapcount dependency in folio_likely_mapped_shared() (CONFIG_NO_PAGE_MAPCOUNT) Date: Thu, 29 Aug 2024 18:56:15 +0200 Message-ID: <20240829165627.2256514-13-david@redhat.com> In-Reply-To: <20240829165627.2256514-1-david@redhat.com> References: <20240829165627.2256514-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: C6D4F40008 X-Stat-Signature: c61czwrtkfa9igbeau1ba5uspzzdxesq X-HE-Tag: 1724950729-191532 X-HE-Meta: U2FsdGVkX18f/hdTYdO/yaaWpl+r0EQ8PytUmBnmSktBtsyTA1Cz7nfvZiFGAwDZ4B+fwKdPXF0doK7U06j3nWrh6mLRpaTVqcO1L6WtXE8FJh9J8hP88gcojEio5nnFqQ2O1BZznOMW3phOgqjQYaniYlDShK5zo0Vs2P3vNQJLNfHMymtjvrgkL5W0YCx9gYQEa0TXSbg1QF7ET0MevlhpwDN/CzmWdQyfl94i4nqwX9yYnbAxGj+v8jR4H1UoLC6rIpNgyDUgSENsZANecpgVtUupKZoR93bOCgOTXC6uuJ5VRfR6VvL53F+Puen/hicc19mbOIrVivkY7+TkkkqyiE2KB1E0qBTL8xXedHkG46D2GdGKcg/nOyvbEJtnJUIM1bRmFHhl4wKky+buPfg0WYvknkoWGDZyOFwv8q+U+egewThSVvwyaRHSKZhAaUdNVSZxugK54BwBLUNB0JM0/aXht3e54YP+a6n4ePA82rowLHzb2Bme52kEpwVAoOTsxTVgHLehMpePQ4Rm0bgFM2hLOwDWKfifoc4+tMP7nGoOai2YkVpqC6hXk78TkkCtwwBq9A2unthD68nXyKUnDd1nAOPnPSjqd/8o4zeFzv/1agDGvw2W57HpnD1FiE9MQ5XQOVT+9tc9ZFXfLN7RhzBBDiw/iS3M4tt14QZqrcA9mIMRrQ6kMZnVI669z/b1jvLQuGhfKCO+Gyk8Xuv3O1Qf8d4nlUNiQAhvDptI5D7NDtLSLIDj3uMUG2L5J9uthP9RqtlDBVsSSQQIfHveZQ046gFJNH+LwQ3EdwW/xIW5Uof+LZOcEx4sjb2b0Ta08+hTrsQAZ75unZVCHHrcYXiYAN67MImYjtNErT51RSAIY0WwnV1+/Si9D19EOqXdjGhuQ4t7VliHHYUf3tUIRYGWcdW6xvV+4Df2TTNchrliiYraSb7tE1mwsfgErw8su7K/qG8tzm/TIM1 xU3Aqj9v 3MYPsiNNODviyowUQ/u+8wTvg8pJf6aMXSgbdkDb/x5rCRLzYU0P4BKs9RxehunSKmWNQeCJPIQQxeMON4PznOBhUUJHvhuvRMCAFbVfo29dPftMQgqi7GBZ5h8HYtFmm37V7d8nfQ7gQ02XM3h50qIi5J9UwVCUCGloJ7iyhateMj3xQ9WtEK9xgnWXOwdypP3JFnNz6O9SwpGEV5Q02mY7hU81AkhJTaejNGzW5tgvf4yDNzI23RN/Fnpn1/PSV18Pu908CPiGrHdUyUiiANOpHGKTpY+pbvR4+T7l+SdOeHCxwQlKcbo/zRpudF++PspMjuw1hxnJ1smDdrVIihGzxi2EQmi+BJj8jRB836Bo1Z6XKZLUviOXtWw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's remove the dependency on the mapcount of the first folio page in large folios and consequently any "false negatives" from folio_likely_mapped_shared(). In theory, we could implement this change only with CONFIG_MM_ID, without gluing it to another config option. But we'll be a bit careful for the time being, because folio_likely_mapped_shared() can now return "false positives" more frequently. Glue it to CONFIG_NO_PAGE_MAPCOUNT, which expresses the "EXPERIMENTAL" character for now. Let's reuse our new MM ownership tracking infrastructure for large folios. Thoroughly document the changed semantics. We might now detect that a folio as "mapped shared" although it no longer is -- this can only happen if more than two MMs mapped a folio at the same time, and neither of the first two is the last one mapping the folio. "false positives" in this context are certainly better than "false negatives" when it comes to enforcing policies (e.g., is process 1 allowed to migrate a folio that might also be used by another process?), but in an ideal world we wouldn't have these "false positives" either. It's worth noting that there will not be a change for small folios and hugetlb folios. In general, for PMD-mapped THP we don't expect a change, only for PTE-mapped THP. This will affect various users of folio_likely_mapped_shared(): (1) khugepaged counts PTEs that target shared folios towards the max_ptes_shared. With false positives we might collapse too little, with false negatives too much. (2) NUMA hinting: PROT_NONE NUMA protection will be skipped for shared folios in COW mappings. With false positives we skip too many, with false negatives we don't skip some we should be skipping. During NUMA hinting faults, we will set TNF_SHARED with shared folios in shared mappings. With false positives we set it too often, with false negatives not often enough. During NUMA hinting faults, we will reject to migrate shared folios in mappings with execute permissions (expectation: shared libraries). With false positives we reject to migrate some, with false negatives we migrate too many. (3) MADV_COLD / MADV_PAGEOUT / MADV_FREE will not try splitting PTE-mapped THPs that are considered shared but not fully covered by the requested range, consequently not processing them. With false positives we will not split+process some we could have processed, with false negatives we split some folios we probably shouldn't have split. (4) mbind() / migrate_pages() / move_pages() will refuse to migrate shared folios unless MPOL_MF_MOVE_ALL is effective (requires CAP_SYS_NICE). With false positives we reject to migrate some folios that could be migrated, with false negatives we migrate some folios that shouldn't have been migrated. (5) folio_referenced_one() will skip exclusive swapbacked folios in dying processes. Shared folios will not be skipped. With false positives we might skip this optimization, with false negatives we might apply this optimization wrongly. Likely (3) and (4) are not really used a lot on folios that are heavily shared among processes -- rather on anonymous memory (mostly from a single parent process) or almost-exclusively mmap'ed files. Similarly (1) is not expected to matter much in practice, and if so, only for long-running child processes after fork(). But even here, it's unlikely that it matters in practice. (5) is not expected to matter much at all, it's a new optimization either way. (2) is interesting: the expectation here is that for anon folios it might not make a big difference. For file-backed pages it might, we'll have to learn about that. Long story short: this paves the way for a complete CONFIG_NO_PAGE_MAPCOUNT implementation, but maybe we'll have to switch to another MM ownership tracking later. Signed-off-by: David Hildenbrand --- include/linux/mm.h | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 98411e53da916..b37f20b26776d 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2142,9 +2142,9 @@ static inline size_t folio_size(const struct folio *folio) * are independent. * * As precise information is not easily available for all folios, this function - * estimates the number of MMs ("sharers") that are currently mapping a folio - * using the number of times the first page of the folio is currently mapped - * into page tables. + * must sometimes estimate the number of MMs ("sharers") that are currently + * mapping a folio using the number of times the first page of the folio is + * currently mapped into page tables. * * For small anonymous folios and anonymous hugetlb folios, the return * value will be exactly correct: non-KSM folios can only be mapped at most once @@ -2152,13 +2152,21 @@ static inline size_t folio_size(const struct folio *folio) * considered shared even if mapped multiple times into the same MM. * * For other folios, the result can be fuzzy: - * #. For partially-mappable large folios (THP), the return value can wrongly - * indicate "mapped exclusively" (false negative) when the folio is - * only partially mapped into at least one MM. + * #. With CONFIG_PAGE_MAPCOUNT: For partially-mappable large folios (THP), + * the return value can wrongly indicate "mapped exclusively" (false + * negative) when the folio is only partially mapped into at least one MM. + * #. With CONFIG_NO_PAGE_MAPCOUNT: For partially-mappable large folios + * (THP), the return value can wrongly indicate "mapped shared" (false + * positive) in some scenarios. This can only happen if two MMs are + * already mapping a folio and a more MM starts mapping the folio. We + * would still the detect the folio as "mapped shared" after the first + * two MMs no longer map the folio. * #. For pagecache folios (including hugetlb), the return value can wrongly * indicate "mapped shared" (false positive) when two VMAs in the same MM * cover the same file range. * + * With CONFIG_MM_ID, this function will never return "false negatives". + * * Further, this function only considers current page table mappings that * are tracked using the folio mapcount(s). * @@ -2183,12 +2191,16 @@ static inline bool folio_likely_mapped_shared(struct folio *folio) if (mapcount <= 1) return false; +#ifdef CONFIG_PAGE_MAPCOUNT /* If any page is mapped more than once we treat it "mapped shared". */ if (folio_entire_mapcount(folio) || mapcount > folio_large_nr_pages(folio)) return true; /* Let's guess based on the first subpage. */ return atomic_read(&folio->_mapcount) > 0; +#else /* !CONFIG_PAGE_MAPCOUNT */ + return !folio_test_large_mapped_exclusively(folio); +#endif /* !CONFIG_PAGE_MAPCOUNT */ } #ifndef HAVE_ARCH_MAKE_FOLIO_ACCESSIBLE