From patchwork Tue Aug 13 12:02:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 13761885 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3ACE3C52D7B for ; Tue, 13 Aug 2024 12:03:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B53806B009E; Tue, 13 Aug 2024 08:03:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ADCDB6B009F; Tue, 13 Aug 2024 08:03:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 92E9F6B00A0; Tue, 13 Aug 2024 08:03:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 6FF0E6B009E for ; Tue, 13 Aug 2024 08:03:48 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 1653EA6FB1 for ; Tue, 13 Aug 2024 12:03:48 +0000 (UTC) X-FDA: 82447088136.16.8386F11 Received: from mail-qk1-f179.google.com (mail-qk1-f179.google.com [209.85.222.179]) by imf14.hostedemail.com (Postfix) with ESMTP id 33E3E100035 for ; Tue, 13 Aug 2024 12:03:45 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=R78+EhzQ; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf14.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.222.179 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723550613; a=rsa-sha256; cv=none; b=s4bSGh86YzDdKHUt25CMMrd/CMyIMN84NuCS0DjlQKQWBuskwZX86QSXFwpnjanSki4spP 0q3sWkN8F7RJe5JLsADiH+SMaEU2T1XzpikSgnT1NSIEW2HTs+KvI+Ox9zd60sg/KrmPI6 X8AFQYLvbm8VcJi1ARki9tm5ryJ034M= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=R78+EhzQ; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf14.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.222.179 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723550613; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0tRn9HldKKoz64Z505xjl60T+jy+qH2RDb/CoWbJ5vM=; b=Ey0CxSUvIjryWNFBQkLztcDTDQZeY7vpOQgwZh248w+EUgWIY9vcn/Io6mfcMonVS7MsdY eZw9kGDeL10BGuNYexLsQ2U+V8tlH8SqbD3hgdJHZ2KBfrG1XUBVQ4SA0b+VxRMU854PdE SfQDQ1qEMFER9AOfVMmmebNFFaF38Vg= Received: by mail-qk1-f179.google.com with SMTP id af79cd13be357-7a1d066a5daso339934985a.3 for ; Tue, 13 Aug 2024 05:03:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1723550624; x=1724155424; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0tRn9HldKKoz64Z505xjl60T+jy+qH2RDb/CoWbJ5vM=; b=R78+EhzQi/X99hWIBGSDAsm3p5jtLr9cGPj3J0xqFwzXEieJpDBDKtv4HME8Bp9mM1 xA5hnuBTTEPVrgQXoUixZoj0JyTAYUxcit2aFwRF1fUEEssDK1B6mr0XUKUy88k7D9rP Ob++fLgGkoiAh5X2MQ9tuh+3UJFCCVMziGOPVFP4j2LghDdRjJaYAe4P6bx7ulod+/Dn oqHbXO+HX0Q7bQaXY6j2QdrWAg2xT1sJ+JsoTrBmabLglRwgPfvhaxC7Z6SUZMWP0QM+ Gtu7xu6Xd9Fca2dNHUOnE9NyqAqFGJCN2MF03nqdwW+zkKtIiyx5xbZ0mMJyCcfMmJux snRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723550624; x=1724155424; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0tRn9HldKKoz64Z505xjl60T+jy+qH2RDb/CoWbJ5vM=; b=sOjOwmeOhKRZqMztH2V76YUbRcNyq6gknPY7Z3uaoLjsAMNz1qgRKcdnVm7JG+gfx2 1KCT8UqeE6TohORIWE2OCIz7e4DYYBdlnhYOl3PgYqziDDX1P1d+C0qDFuKPS7vIOQcs oJNfXTq8V653HoSyfN3UuJM/vKegBZkrZxXBpoJ3yYRdL3jvUbxofqnnpLwmcqM/bHDd D6gAScHToXfpVBNnYxP7p4gDDvmClfTGlp8xYINoodjE7iudwMM/ILRFjPadsGFmpJSU ssbIo/otvXiJtyawt0JSk0Q5/vOB8p8YUPvC26erODcnkB/bRqcwZyAXvxSB5BcE/pRt lsvg== X-Forwarded-Encrypted: i=1; AJvYcCXalZZ3Kyv/xNIXwDgDiP1/XX2HK2mwOv6wyMYi1Dkjr5T7eVgFu3uuIX3nT+Cvlea30GtPsmF0QQ==@kvack.org X-Gm-Message-State: AOJu0YwdMKeiCyWIv0M567lssIWV5wu1vzOS0944jlehi2IzhSOlb1F0 puLj/0gcl62D4e+li/yz/l9IwV8e/h5jEskcjJZ4bGwcvd56bBuU X-Google-Smtp-Source: AGHT+IHCyk5c5SE6e3z0DoyqupXKxHAXubnY9Py1uh22TPysRE3LzZi4VhhYcHmcgCKHwTje7P8Lig== X-Received: by 2002:a05:620a:4091:b0:79d:9102:554a with SMTP id af79cd13be357-7a4e15011femr407319085a.14.1723550624171; Tue, 13 Aug 2024 05:03:44 -0700 (PDT) Received: from localhost (fwdproxy-ash-112.fbsv.net. [2a03:2880:20ff:70::face:b00c]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-4531c26ec58sm31235081cf.60.2024.08.13.05.03.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Aug 2024 05:03:43 -0700 (PDT) From: Usama Arif To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, roman.gushchin@linux.dev, yuzhao@google.com, david@redhat.com, baohua@kernel.org, ryan.roberts@arm.com, rppt@kernel.org, willy@infradead.org, cerasuolodomenico@gmail.com, corbet@lwn.net, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kernel-team@meta.com, Shuang Zhai , Usama Arif Subject: [PATCH v3 1/6] mm: free zapped tail pages when splitting isolated thp Date: Tue, 13 Aug 2024 13:02:44 +0100 Message-ID: <20240813120328.1275952-2-usamaarif642@gmail.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20240813120328.1275952-1-usamaarif642@gmail.com> References: <20240813120328.1275952-1-usamaarif642@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 33E3E100035 X-Rspamd-Server: rspam01 X-Stat-Signature: b4st4c7jycwq7i1xs4wzgkawzn3nho9c X-HE-Tag: 1723550625-353487 X-HE-Meta: U2FsdGVkX1+tkCl/Uqe+Qah7s1Lx7NAD+sb20Sd+6U69ciwAfs2FyqsNEEgiFfGTBlRwcjxX8L6k4Ed92QcysjbeGt1HbQ3+NBfAqU8+VVGzbV9PoU7+PlHMKmaDMkipy+5U16KcceZhIFiyDvtuLzfK+50Lg95RmFGCEhnLy2tF6GEehmkRVShO3EQivCSLnKv3z5lvR6H4NrIm8gcf/EL25VtRFIrMrEPX8KP6YKp4Z24rA/C+GBLPlWxAU2gHPa5Ih72iMaF4dBd3+VYhaqfzceIhQ3VxqezDDNCKXL+/XeRlI95dbWErCrqlz3+RBSnM+GBDZrZK/vTq1Gpvk70MO6Tw15LApBxcX1q0CCX88Au5s2R3rEMxXT55IGdTZ1CjvzXAlraZ8Y6Lw3Qdu2A9CgxrHcMk2qHEfelzbJlOlDWEb2UyQ8ZTvbGgq53K49YLiY75Z9+YDzOQK9o9OuDHPa5FjCsLse5x6zo8Eq9ZtBH4FXe2OqMYFObzLyfO4sWc5TdugpuyzeFQZWPNXc+HYDsmKcp9hctrl+9lUxNmOt+F2Xt9lnpfodc3PiPhxy3LTHb/OaRj0U5ktT3IA1gOO4t5ykxorhsjqMAvrNxCCMW6qpgmvUzhqp5GWz28FaFMuaa7cR3w6k5vdfWKvdxRRbxl4OkqT4TeaudDj9tzowTiYjAK0m9K0l6wA4M1dsaBZmYhrRNcP2SNRjPvBUQAYjXJvxXL8ktGfiGYBICoI2Npt+eXu4KsOz+FVio3U1B/Zxhr2TVugGZ0qEQmUFDPYun/HhIHVsdqngkNh5wS2//Zjc1J8l0d7Q8cTS7x93Zx7QNydqwRT7Q8YmkIGuVPOGOw/lLorSG6u/TFFtFeg/tQSOn35PLDNjyw830O1zjrQ5rkKKdsFcnXfqSRAH6vKaZ3d0OQ0kgUNF+aWoU/jB8qqQXd+9xgmKZ/zUOnZ19uoUgZG7eHZ2Ol390 jM2JQ64K LI+IqRepDocWq2jirqfGNPpfzcT909+xNmY5vdehICTFEuugnp9k/9GBFWbGyz8dQ1VmXmQA9IWulGJb2GJ04jPAgyQ0XDNxHWOhPIsNjqpJy1CXlKS0QKeObGQd7hVRZYJ0v7bHv04xlY4wCMJdQ/Fns9djzfN6ywpRbHfUWiV1cLXW+7t5YrAqebS9bWliYS7FI4jWkCrK1Iqp+nB+VE1ntMacJkJxXRyQkplmU9aj/HnGjDOJTKwwP4+c2QU7Vjaie9I72FDQMhnYV9+RmLiKFbqKtYI2JoXt/JyQj4Y/iGdOnDiMStYTer0SZrc7ATPveyViVGe8Vijuk0xFjIQBD0mxWpVRGY8EQJC3oEa7L1NWChEZ65+je1cCC/LcH7dZSIrXaY+b477wciHo4dBnbOrZjre61y4LG X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Yu Zhao If a tail page has only two references left, one inherited from the isolation of its head and the other from lru_add_page_tail() which we are about to drop, it means this tail page was concurrently zapped. Then we can safely free it and save page reclaim or migration the trouble of trying it. Signed-off-by: Yu Zhao Tested-by: Shuang Zhai Signed-off-by: Usama Arif Acked-by: Johannes Weiner --- mm/huge_memory.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 04ee8abd6475..85a424e954be 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3059,7 +3059,9 @@ static void __split_huge_page(struct page *page, struct list_head *list, unsigned int new_nr = 1 << new_order; int order = folio_order(folio); unsigned int nr = 1 << order; + struct folio_batch free_folios; + folio_batch_init(&free_folios); /* complete memcg works before add pages to LRU */ split_page_memcg(head, order, new_order); @@ -3143,6 +3145,26 @@ static void __split_huge_page(struct page *page, struct list_head *list, if (subpage == page) continue; folio_unlock(new_folio); + /* + * If a folio has only two references left, one inherited + * from the isolation of its head and the other from + * lru_add_page_tail() which we are about to drop, it means this + * folio was concurrently zapped. Then we can safely free it + * and save page reclaim or migration the trouble of trying it. + */ + if (list && folio_ref_freeze(new_folio, 2)) { + VM_WARN_ON_ONCE_FOLIO(folio_test_lru(new_folio), new_folio); + VM_WARN_ON_ONCE_FOLIO(folio_test_large(new_folio), new_folio); + VM_WARN_ON_ONCE_FOLIO(folio_mapped(new_folio), new_folio); + + folio_clear_active(new_folio); + folio_clear_unevictable(new_folio); + if (!folio_batch_add(&free_folios, folio)) { + mem_cgroup_uncharge_folios(&free_folios); + free_unref_folios(&free_folios); + } + continue; + } /* * Subpages may be freed if there wasn't any mapping @@ -3153,6 +3175,11 @@ static void __split_huge_page(struct page *page, struct list_head *list, */ free_page_and_swap_cache(subpage); } + + if (free_folios.nr) { + mem_cgroup_uncharge_folios(&free_folios); + free_unref_folios(&free_folios); + } } /* Racy check whether the huge page can be split */ From patchwork Tue Aug 13 12:02:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 13761886 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 02A5FC531DC for ; Tue, 13 Aug 2024 12:03:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7AB016B009F; Tue, 13 Aug 2024 08:03:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 75D316B00A0; Tue, 13 Aug 2024 08:03:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4E8CB6B00A1; Tue, 13 Aug 2024 08:03:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 34FB06B009F for ; Tue, 13 Aug 2024 08:03:49 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id AC8AD1607BB for ; Tue, 13 Aug 2024 12:03:48 +0000 (UTC) X-FDA: 82447088136.23.D7A62B4 Received: from mail-qt1-f182.google.com (mail-qt1-f182.google.com [209.85.160.182]) by imf02.hostedemail.com (Postfix) with ESMTP id 9831F8000B for ; Tue, 13 Aug 2024 12:03:46 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=US3MwXRO; spf=pass (imf02.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.160.182 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723550556; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ArHqWK+pDk4THc6HGtgBtflIMEbd37YycfL2VsChgqA=; b=5d56RPUaTxrOb+FDwTVEM+eS2RncWN/cWDpc5grPxSEkcZr3JGsdH4dfeqF7m2qmYbvnsh SKBEiRzKXo4YseJ1fzX11fJmRAsY3pKtvO1E6cJ5zkCzFpEHBQg4ZVVqf5MLzolRQzqSX6 BzIHB3mcjvR0vOt4gtQ+UK2Q9eMjegI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723550556; a=rsa-sha256; cv=none; b=NqfRmxnq/0shEoqAUSrNZBqZPVcFwylVBkoEpFCaNTtQwAPG1QW96k11KTxEZ/iaa6AkrT qVbvstBaMHEY9Oo+wZug13oO3obnCdXxJ5IId2qfzKd3LFIytIjhV77GO5Qwf1c67fxDYn pm8v7Yxn8a2SyOs3WQnwM5lSHRGa8R4= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=US3MwXRO; spf=pass (imf02.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.160.182 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-qt1-f182.google.com with SMTP id d75a77b69052e-44fee8813c3so32332571cf.2 for ; Tue, 13 Aug 2024 05:03:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1723550625; x=1724155425; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ArHqWK+pDk4THc6HGtgBtflIMEbd37YycfL2VsChgqA=; b=US3MwXRObbMtwTfzygu5x3IsJfsxntPlucpAGOf6A3rAjqFd/tn/UjmfHFqkFglvLE rNz7XaxFbb3wBjmY0ihQpApXAs1ZDNDZ0rxV0tUDEBEdodYCefwmU4Q89pjsp+88gtuC 12VhQvXhAUAxSdLs/F16l/Vh0ER3i4SDUHP4FEQpd1/l5A5rfVOEa5ldG3FYrQVcSThI 5U6xATQr0nrVmArwq0esAR61rEXgqCFmZtWAC6KVfFrGTHq58XBKYMMb072MGtdjG+Sk rpQr3vN0rFMpIbho9XYLQlk9jBw+rTf+z+1Hxl6/mLuqE7J61FXcoAbDFDoeMzdHnTA4 BsFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723550625; x=1724155425; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ArHqWK+pDk4THc6HGtgBtflIMEbd37YycfL2VsChgqA=; b=DItgWVr3PmNP1t9XirLyqDrFMOfBqwdG0lejZliY+MP0y6Zbvy7teaSDX6DQ9QRYhn krl1w7VaI16+TLj3iKy17Onc8ID0bs6fx2Hvm2NBJf7D1fDa6AXKtnBWBl7rF6siXeBW 0DLuCOHcXeKiL55cIqhPXdB1fmU57I/X5hBEcS3kjQIBhXOf3dGb+DpRvDu4dMMSS6L1 TEbFhkIfFMtku2j3GdKq6LCcb6KLjhAI3Hmnox6TpnuFNILfyVNqkuQPx9Dc4M1L76C2 bC8cU6apDZd9tlvHDkKDIIsCeWXqZwYjg54TO8QshMwoCPMufPBo1SWQdKsVAFRzUCsR gTIQ== X-Forwarded-Encrypted: i=1; AJvYcCW1OzPsesfSWoggkrC3YoFiopVf6qxghJQz99LT0vVRrvqZKOR0v95Vbvp7+1iUL8EuoN+fsfwJ5GKoyzL/CwyQIzM= X-Gm-Message-State: AOJu0YwUprKavpQZVn1qI4K/KPQ/Ofu7ok6J1wPKF27qIsQA620SsHHP yukx9a9CoqGKo20v7mKORD5bNK8TnqPjI5nwNGQrY4u27fNyg8eZ X-Google-Smtp-Source: AGHT+IFmFyxDVIM6hJt06Re344Hp5hGhJ4dfTOoArazVB1tqefIgGdnCSzB4+w1duntTekqvxAqRhA== X-Received: by 2002:a05:622a:316:b0:453:4aaa:d585 with SMTP id d75a77b69052e-4534aaad5f5mr34502981cf.4.1723550625396; Tue, 13 Aug 2024 05:03:45 -0700 (PDT) Received: from localhost (fwdproxy-ash-000.fbsv.net. [2a03:2880:20ff::face:b00c]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-4531c1a7f12sm31519601cf.20.2024.08.13.05.03.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Aug 2024 05:03:45 -0700 (PDT) From: Usama Arif To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, roman.gushchin@linux.dev, yuzhao@google.com, david@redhat.com, baohua@kernel.org, ryan.roberts@arm.com, rppt@kernel.org, willy@infradead.org, cerasuolodomenico@gmail.com, corbet@lwn.net, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kernel-team@meta.com, Shuang Zhai , Usama Arif Subject: [PATCH v3 2/6] mm: remap unused subpages to shared zeropage when splitting isolated thp Date: Tue, 13 Aug 2024 13:02:45 +0100 Message-ID: <20240813120328.1275952-3-usamaarif642@gmail.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20240813120328.1275952-1-usamaarif642@gmail.com> References: <20240813120328.1275952-1-usamaarif642@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 9831F8000B X-Stat-Signature: rwe357nm5wnsbd76cppt86mgr3am1yez X-HE-Tag: 1723550626-146868 X-HE-Meta: U2FsdGVkX1+mmNz20JmwLUcTnKKddMwZzTs4lENMq4kBBtXbBNd/ml6JLh1q5OqsqJZp46Aishk5JoU+tu/tKS70emfMNr+LyoFIXEosWcKVvQql+49PL8ZIrUDQUvjJsfGMjqbT2Ij4/sD71NG/tepvwGwRKRRTMjAwAY2W4/NYEFGi7KZUE4WQ5Obp6Ou7gEYuvgsmEMygpETh+VbVEsAbgc5HrRbjau32NuFs6Ii76L3tBuIbIPiLH2PFixx7nwT+RiPNDUjNGNy7I/WOa8p0220GCKOXaZ4f2KYC2cCUtrVUpz1deF5tjbOX88RSUb1bcCjV5Yz2ZWReor5hj/Tx5LWGZNpp875zQQAbWbentGl0o+ZSPlHvjPyf26ya9JupZsWZZdjLBIIWyiaZL1MYfn1OejJDsEsPZwjA2BOd0hBbFwbqyecRgvsfJFJpl6tn45nJUHGrpWmpsGp6PPlSF4yfAPeV61FpdSJXsxK5C3n4aRoDR3rw5ZLB4Laf6PJPCu0zlb1Bi1kZSUoVZ2mG6AUWuiH8E7uWisw7tTYU1Pu1MQ6nvzOx2jE4iiKPSGucurCiW7T1uOm2le8jJPkQxPh9ktDvRE9dOSR6S1rEshTkvtOmpL25DEtysxuoQ9S/gKtAxv9sFQ81BUUKCUMUyXgh69cKmWY8h8KODCCDwKo/VpIbfLc83c+fNYkbWEFVDL04DfyJCko0bWD0xzSXY9fCJS0hT58Js+5vZhlpQeqOkLIsO6MiDhIniYoTx1uiWtGqZBU3ydUc3XxxpOzZXaojWtC29MZm4jdc1+ShkA0HA5HFmBerlqu+igEZN3e0xb2WKaqW4exCS/XqvMHGdy/f1rG62V9Vn+AbxasQWLaBrJR3txBSWz8dQELsWLUS7M+ScfNQmESXc0uhteRbTACABT10yfs13MP5jGCF8f6Fp9KsXQmziwnBAhir3TxvH3+dAzI7axT7ak3 kT7zrk5v oxDmUy1e2IRyHZzfatXl6SPQLy8erwmWlyp3m1rV14y1ds0QahxtjP3UFXubreyyAvYkhPBAHrm8I1Gf+lJFOcSL6qeZYrod1XrCDrmQ+GzXbdiRWPnT8bO3s2YFQcBOMRg04C28iYhKvq9mvqI4nxYi6fxAiLh9+PVnqsYczNetuLrK/bMdb99Xn/VAhuMZBv6lVdMpn8OXhau3AcoHDYR552H1rXqFDaUyYmdNK03X0+k9P6/u7mp4cviPRNcFZ8dMhL7rbC7n6EcjI3mNTQhrm3YqmHSASecLMW9clxecERNg2CEbFNJioKOG8vwN5w6bfnFhuipAs+jlAv6Qvl9uqw69xrd/J5IAa15yu8QdqmHU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Yu Zhao Here being unused means containing only zeros and inaccessible to userspace. When splitting an isolated thp under reclaim or migration, the unused subpages can be mapped to the shared zeropage, hence saving memory. This is particularly helpful when the internal fragmentation of a thp is high, i.e. it has many untouched subpages. This is also a prerequisite for THP low utilization shrinker which will be introduced in later patches, where underutilized THPs are split, and the zero-filled pages are freed saving memory. Signed-off-by: Yu Zhao Tested-by: Shuang Zhai Signed-off-by: Usama Arif --- include/linux/rmap.h | 7 ++++- mm/huge_memory.c | 8 ++--- mm/migrate.c | 71 ++++++++++++++++++++++++++++++++++++++------ mm/migrate_device.c | 4 +-- 4 files changed, 74 insertions(+), 16 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 0978c64f49d8..07854d1f9ad6 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -745,7 +745,12 @@ int folio_mkclean(struct folio *); int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages, pgoff_t pgoff, struct vm_area_struct *vma); -void remove_migration_ptes(struct folio *src, struct folio *dst, bool locked); +enum rmp_flags { + RMP_LOCKED = 1 << 0, + RMP_USE_SHARED_ZEROPAGE = 1 << 1, +}; + +void remove_migration_ptes(struct folio *src, struct folio *dst, int flags); /* * rmap_walk_control: To control rmap traversing for specific needs diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 85a424e954be..6df0e9f4f56c 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2911,7 +2911,7 @@ bool unmap_huge_pmd_locked(struct vm_area_struct *vma, unsigned long addr, return false; } -static void remap_page(struct folio *folio, unsigned long nr) +static void remap_page(struct folio *folio, unsigned long nr, int flags) { int i = 0; @@ -2919,7 +2919,7 @@ static void remap_page(struct folio *folio, unsigned long nr) if (!folio_test_anon(folio)) return; for (;;) { - remove_migration_ptes(folio, folio, true); + remove_migration_ptes(folio, folio, RMP_LOCKED | flags); i += folio_nr_pages(folio); if (i >= nr) break; @@ -3129,7 +3129,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, if (nr_dropped) shmem_uncharge(folio->mapping->host, nr_dropped); - remap_page(folio, nr); + remap_page(folio, nr, PageAnon(head) ? RMP_USE_SHARED_ZEROPAGE : 0); /* * set page to its compound_head when split to non order-0 pages, so @@ -3424,7 +3424,7 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, if (mapping) xas_unlock(&xas); local_irq_enable(); - remap_page(folio, folio_nr_pages(folio)); + remap_page(folio, folio_nr_pages(folio), 0); ret = -EAGAIN; } diff --git a/mm/migrate.c b/mm/migrate.c index 66a5f73ebfdf..3288ac041d03 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -178,13 +178,56 @@ void putback_movable_pages(struct list_head *l) } } +static bool try_to_map_unused_to_zeropage(struct page_vma_mapped_walk *pvmw, + struct folio *folio, + unsigned long idx) +{ + struct page *page = folio_page(folio, idx); + bool contains_data; + pte_t newpte; + void *addr; + + VM_BUG_ON_PAGE(PageCompound(page), page); + VM_BUG_ON_PAGE(!PageAnon(page), page); + VM_BUG_ON_PAGE(!PageLocked(page), page); + VM_BUG_ON_PAGE(pte_present(*pvmw->pte), page); + + if (PageMlocked(page) || (pvmw->vma->vm_flags & VM_LOCKED)) + return false; + + /* + * The pmd entry mapping the old thp was flushed and the pte mapping + * this subpage has been non present. If the subpage is only zero-filled + * then map it to the shared zeropage. + */ + addr = kmap_local_page(page); + contains_data = memchr_inv(addr, 0, PAGE_SIZE); + kunmap_local(addr); + + if (contains_data || mm_forbids_zeropage(pvmw->vma->vm_mm)) + return false; + + newpte = pte_mkspecial(pfn_pte(my_zero_pfn(pvmw->address), + pvmw->vma->vm_page_prot)); + set_pte_at(pvmw->vma->vm_mm, pvmw->address, pvmw->pte, newpte); + + dec_mm_counter(pvmw->vma->vm_mm, mm_counter(folio)); + return true; +} + +struct rmap_walk_arg { + struct folio *folio; + bool map_unused_to_zeropage; +}; + /* * Restore a potential migration pte to a working pte entry */ static bool remove_migration_pte(struct folio *folio, - struct vm_area_struct *vma, unsigned long addr, void *old) + struct vm_area_struct *vma, unsigned long addr, void *arg) { - DEFINE_FOLIO_VMA_WALK(pvmw, old, vma, addr, PVMW_SYNC | PVMW_MIGRATION); + struct rmap_walk_arg *rmap_walk_arg = arg; + DEFINE_FOLIO_VMA_WALK(pvmw, rmap_walk_arg->folio, vma, addr, PVMW_SYNC | PVMW_MIGRATION); while (page_vma_mapped_walk(&pvmw)) { rmap_t rmap_flags = RMAP_NONE; @@ -208,6 +251,9 @@ static bool remove_migration_pte(struct folio *folio, continue; } #endif + if (rmap_walk_arg->map_unused_to_zeropage && + try_to_map_unused_to_zeropage(&pvmw, folio, idx)) + continue; folio_get(folio); pte = mk_pte(new, READ_ONCE(vma->vm_page_prot)); @@ -286,14 +332,21 @@ static bool remove_migration_pte(struct folio *folio, * Get rid of all migration entries and replace them by * references to the indicated page. */ -void remove_migration_ptes(struct folio *src, struct folio *dst, bool locked) +void remove_migration_ptes(struct folio *src, struct folio *dst, int flags) { + struct rmap_walk_arg rmap_walk_arg = { + .folio = src, + .map_unused_to_zeropage = flags & RMP_USE_SHARED_ZEROPAGE, + }; + struct rmap_walk_control rwc = { .rmap_one = remove_migration_pte, - .arg = src, + .arg = &rmap_walk_arg, }; - if (locked) + VM_BUG_ON_FOLIO((flags & RMP_USE_SHARED_ZEROPAGE) && (src != dst), src); + + if (flags & RMP_LOCKED) rmap_walk_locked(dst, &rwc); else rmap_walk(dst, &rwc); @@ -903,7 +956,7 @@ static int writeout(struct address_space *mapping, struct folio *folio) * At this point we know that the migration attempt cannot * be successful. */ - remove_migration_ptes(folio, folio, false); + remove_migration_ptes(folio, folio, 0); rc = mapping->a_ops->writepage(&folio->page, &wbc); @@ -1067,7 +1120,7 @@ static void migrate_folio_undo_src(struct folio *src, struct list_head *ret) { if (page_was_mapped) - remove_migration_ptes(src, src, false); + remove_migration_ptes(src, src, 0); /* Drop an anon_vma reference if we took one */ if (anon_vma) put_anon_vma(anon_vma); @@ -1305,7 +1358,7 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, lru_add_drain(); if (old_page_state & PAGE_WAS_MAPPED) - remove_migration_ptes(src, dst, false); + remove_migration_ptes(src, dst, 0); out_unlock_both: folio_unlock(dst); @@ -1443,7 +1496,7 @@ static int unmap_and_move_huge_page(new_folio_t get_new_folio, if (page_was_mapped) remove_migration_ptes(src, - rc == MIGRATEPAGE_SUCCESS ? dst : src, false); + rc == MIGRATEPAGE_SUCCESS ? dst : src, 0); unlock_put_anon: folio_unlock(dst); diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 6d66dc1c6ffa..8f875636b35b 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -424,7 +424,7 @@ static unsigned long migrate_device_unmap(unsigned long *src_pfns, continue; folio = page_folio(page); - remove_migration_ptes(folio, folio, false); + remove_migration_ptes(folio, folio, 0); src_pfns[i] = 0; folio_unlock(folio); @@ -837,7 +837,7 @@ void migrate_device_finalize(unsigned long *src_pfns, src = page_folio(page); dst = page_folio(newpage); - remove_migration_ptes(src, dst, false); + remove_migration_ptes(src, dst, 0); folio_unlock(src); if (is_zone_device_page(page)) From patchwork Tue Aug 13 12:02:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 13761888 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F23CAC52D7B for ; Tue, 13 Aug 2024 12:03:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DA4386B00A1; Tue, 13 Aug 2024 08:03:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D4CF66B00A3; Tue, 13 Aug 2024 08:03:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A66B96B00A5; Tue, 13 Aug 2024 08:03:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 61A3A6B00A1 for ; Tue, 13 Aug 2024 08:03:51 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id E25C214078F for ; Tue, 13 Aug 2024 12:03:50 +0000 (UTC) X-FDA: 82447088220.06.B95A190 Received: from mail-qv1-f42.google.com (mail-qv1-f42.google.com [209.85.219.42]) by imf16.hostedemail.com (Postfix) with ESMTP id AF9D118000D for ; Tue, 13 Aug 2024 12:03:47 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=NL68yJXV; spf=pass (imf16.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.219.42 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723550556; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zqJ9Y6RR0iCOONporxiAWd0rvcRWYXEUM+u9cAIorwk=; b=DLCtg+DtEdXavh/ri8d89IZ0FHTlOSAczxW94oxUL4IIlhf4Af1Jl36AIibuvaxgIGeKHH yjQZAG+Pau4kETHmLUaoDau58auw78glpm7pcyks/JKXgLpfjj5tyFkVa5SpAGA7Mfk1bU OiOmMDZuhHClUODW5lTL8AXhkyuU8HU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723550556; a=rsa-sha256; cv=none; b=PkjGdnVcboatBzvTDzNzqIP5Ly2LIM5Nd2r1wAcwvX8SCkRFDDtglOhBDJ5ZEas66Oaw5d uBxzZSKCbDoXSka5SGzNfbweni1O57zWypX0eCN/0tWjyRZuPKVyG/In44PLsMPEYdrqGN fQaIs0Lpc6qAXz/ZYq4RWZssnkt83Cs= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=NL68yJXV; spf=pass (imf16.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.219.42 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-qv1-f42.google.com with SMTP id 6a1803df08f44-6b79b93a4c9so45030986d6.1 for ; Tue, 13 Aug 2024 05:03:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1723550627; x=1724155427; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=zqJ9Y6RR0iCOONporxiAWd0rvcRWYXEUM+u9cAIorwk=; b=NL68yJXVV593ASiA82+QymJHrmS2wvQupdxUbZTn6Yboem2HDVPeLKvT/wxpEEAlLM fErtPoGOtJk7/jIgrrTS+NFYeC8znlljUSd/jvqb5QLIRlyreYWfmjF7V/w5ibHrn1Kx DUvwq4zTL9thlJIbQUjrjTa9iRlAEqTg98m+fMoumbdqk+5BDzQk2jBtfOVFbpvStxD3 ZptOlGGQdu8xrOm7I6nJmB/MSC0lpUkEWJySR9yIUQTBqjjvNuKfYYVA1tM2cSggR6Y5 3NAgVbauIQYaudMNe1c/fua5+cRCqWn0DNWY5HXClu86s6yv4Xu1KjA2qUXJr6OJ7wW0 WqRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723550627; x=1724155427; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zqJ9Y6RR0iCOONporxiAWd0rvcRWYXEUM+u9cAIorwk=; b=sSNTMleuSyu6NOkfs0UNcp++zz6uyOuVJdICbtHYhJ94NfS35+30dGUYjfiUHSPMAG inuZx+bHW9+xY5vFNjV3o96JVZ/2AeCZNUZlR10YowtU1dVHPifcPTjTO4Pd+Qk+4xcY q9hPtYnC4DPYApsI33WBuagba8+K3oD19laeE5rQz7G/k6r5K7Zo+UcU8kW8fiOIAfjz OPr7/OOIQqpXVaN3DPeZITovYMUCyOsZnQYrSPUsF1v9dfU853gG0wDlJSeenjMhTtSU q1+Qu7iHonLF+YOLc+aLL55ngdm++33+y40olFRkCVMlmufd/WEw9dDUlptQfNXuTZUl U5lQ== X-Forwarded-Encrypted: i=1; AJvYcCXlOIh1ZcmMh/jisYprmBi2cvDH76PB6+uVzhdLvlaRsrA3mlDAi37R38ou8R3OhNaac3uhC6DoGm/BUziq7VNzIoo= X-Gm-Message-State: AOJu0YzG00CxkDITH7v2ypoMhHNEz5dZzp/G7vJqsb38Ye2dJ8SkE2mM OY4srgMQiAZJ/jizrcDMnPhKsIvh4y0RB4kGXOj91pm9gLUOCOT3 X-Google-Smtp-Source: AGHT+IFJ5i6wBfUkrpsXhs0Tc15ioa/fxXOSNWt+VVOIv2sQLq9LGVXdX8ciULa9jF+2pjwR9W6Tjw== X-Received: by 2002:a05:6214:2aab:b0:6bb:8b7b:c2df with SMTP id 6a1803df08f44-6bf50c6a12emr48629266d6.25.1723550626609; Tue, 13 Aug 2024 05:03:46 -0700 (PDT) Received: from localhost (fwdproxy-ash-112.fbsv.net. [2a03:2880:20ff:70::face:b00c]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6bd82e35785sm33439146d6.81.2024.08.13.05.03.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Aug 2024 05:03:46 -0700 (PDT) From: Usama Arif To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, roman.gushchin@linux.dev, yuzhao@google.com, david@redhat.com, baohua@kernel.org, ryan.roberts@arm.com, rppt@kernel.org, willy@infradead.org, cerasuolodomenico@gmail.com, corbet@lwn.net, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kernel-team@meta.com, Alexander Zhu , Usama Arif Subject: [PATCH v3 3/6] mm: selftest to verify zero-filled pages are mapped to zeropage Date: Tue, 13 Aug 2024 13:02:46 +0100 Message-ID: <20240813120328.1275952-4-usamaarif642@gmail.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20240813120328.1275952-1-usamaarif642@gmail.com> References: <20240813120328.1275952-1-usamaarif642@gmail.com> MIME-Version: 1.0 X-Stat-Signature: 5e467899ee9zoj7oqm4u579egqmy1k45 X-Rspamd-Queue-Id: AF9D118000D X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1723550627-332066 X-HE-Meta: U2FsdGVkX1/bAN7auqbFVSH8aKzWL+eUglhAfuk7wjXuBiH14ygP1/GjMqSKYnk61CAleHu+hS5zu5fMB9fVsUFys/y22qcskRZ/D7Jj23+DXaZB8VkthuIpgF7la1iBug9lIXMa5N+4SzWWUzIAOJO+ZvMjM2kx5pmT4BRcmWub0vLS+mHvkPhRW3klkQI9aYqa5qdNu5QhlrR3yXgAh8EG+euviGFA5MLp4DPz3cnl5jz/vvqWovJ89YUuohUQqSb2MrcRRyvxbycYONrYXk5X5GHCotXwtjvzzMR/OXAObxqXSRbKLV4PeajLBtvReqUJWqU4AklZYp8k4MsAoDt9r6tARJXJ7vhp6qns5/M7thq8jJEzJKv175GauOG6M1jF4+TMxCVVuMGwG7ahwoxZkCkgm0ogebcOQ7Ae2k103+MZ8DPl6+Vvy4awzfaeaOPc0rFQ3AnrviG1B1cRmEzmuRHcALzeI5R9wD7LkMlDpdy9aDGTTx7POVOIi9f77NRWq2Jv44/Rao+nVBHPIYVfUR3Xn6gz5d55L7qG5+N56AwbNyd3TODRC6e/Om7BR//jFNuDEb/PKsfSgF0sO8n6Nq/m1N4P1six2ZTuK0z3bBp+f6Fp1iv15XY4Ix4LY7Q3NlFi+BIksPu5vmSPPpbEHy+KJ00iCeSnzE1YafyzKExFMEJoT9Yjh8YKG+T8qZ/nk1Hdm2D7YD/cf7lamgXOJYoLMBxxDm2K6Ao+NA20FOFSezitMkuQNj6YSi1YtTwQ4gq8T8axWn/kzX66CdMEm09fPRqOkzRKQ7HL6qRxD8yWiGQLmc3xHbbWod8cNmrjzldR1IdrEDKncOu+y2sP/pa9jxyzuXtM9o2pNQoKmLZ4FyxLz8KlvFdTY4EFDsm5BCXAOz78aTWHIgOcAZ/SupdxgCgiJJqduSMnXSS+9x0IPWvbcu8sl+NVegb5xKppMeab+H0F9KeoZuR X8XJgh0y 6SD/u0kvJ0bU1HkWC0USJ3Te3T9YYTZ1dlKIP9caBjYaTrAHfPyNNBvjXqh8/2PqHwXKBHsNEjHNHR3NpC4dMq2DnJiDLucGp+9TAyF1QwFrrvOHwbb2lztm/qGIAwCUpPwrvqkcPKPv0qqLTpEj1jxf2rgB/BgHZByoefkunrl8op/me05lPMy9+dcD26sJAt+xa4P/6xcGHMv5OYOj7NpioqE5hCizTxQnHwhJiaDf7B24xwIz4i0sTE4nakFzHTcX09DRrVWSzH/Itm8UgB34yRU0aHV4yQ8xnKX+2nq7g48uE3b+2IZyTd7+ByGSJS3JY/Ds1cShQbI+GxkK/AgONEt5dK4dQGDOUmgLGUSSFayJFIF28yQW/fKLlqx20Q7chY7/GWUlE4sn+QRimnxWrq/dfjt3yoPMt3JB0ufyy3NxR1fQFMntr8fJf0tTrfN0zj0HBBg51Qfs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexander Zhu When a THP is split, any subpage that is zero-filled will be mapped to the shared zeropage, hence saving memory. Add selftest to verify this by allocating zero-filled THP and comparing RssAnon before and after split. Signed-off-by: Alexander Zhu Acked-by: Rik van Riel Signed-off-by: Usama Arif --- .../selftests/mm/split_huge_page_test.c | 71 +++++++++++++++++++ tools/testing/selftests/mm/vm_util.c | 22 ++++++ tools/testing/selftests/mm/vm_util.h | 1 + 3 files changed, 94 insertions(+) diff --git a/tools/testing/selftests/mm/split_huge_page_test.c b/tools/testing/selftests/mm/split_huge_page_test.c index e5e8dafc9d94..eb6d1b9fc362 100644 --- a/tools/testing/selftests/mm/split_huge_page_test.c +++ b/tools/testing/selftests/mm/split_huge_page_test.c @@ -84,6 +84,76 @@ static void write_debugfs(const char *fmt, ...) write_file(SPLIT_DEBUGFS, input, ret + 1); } +static char *allocate_zero_filled_hugepage(size_t len) +{ + char *result; + size_t i; + + result = memalign(pmd_pagesize, len); + if (!result) { + printf("Fail to allocate memory\n"); + exit(EXIT_FAILURE); + } + + madvise(result, len, MADV_HUGEPAGE); + + for (i = 0; i < len; i++) + result[i] = (char)0; + + return result; +} + +static void verify_rss_anon_split_huge_page_all_zeroes(char *one_page, int nr_hpages, size_t len) +{ + unsigned long rss_anon_before, rss_anon_after; + size_t i; + + if (!check_huge_anon(one_page, 4, pmd_pagesize)) { + printf("No THP is allocated\n"); + exit(EXIT_FAILURE); + } + + rss_anon_before = rss_anon(); + if (!rss_anon_before) { + printf("No RssAnon is allocated before split\n"); + exit(EXIT_FAILURE); + } + + /* split all THPs */ + write_debugfs(PID_FMT, getpid(), (uint64_t)one_page, + (uint64_t)one_page + len, 0); + + for (i = 0; i < len; i++) + if (one_page[i] != (char)0) { + printf("%ld byte corrupted\n", i); + exit(EXIT_FAILURE); + } + + if (!check_huge_anon(one_page, 0, pmd_pagesize)) { + printf("Still AnonHugePages not split\n"); + exit(EXIT_FAILURE); + } + + rss_anon_after = rss_anon(); + if (rss_anon_after >= rss_anon_before) { + printf("Incorrect RssAnon value. Before: %ld After: %ld\n", + rss_anon_before, rss_anon_after); + exit(EXIT_FAILURE); + } +} + +void split_pmd_zero_pages(void) +{ + char *one_page; + int nr_hpages = 4; + size_t len = nr_hpages * pmd_pagesize; + + one_page = allocate_zero_filled_hugepage(len); + verify_rss_anon_split_huge_page_all_zeroes(one_page, nr_hpages, len); + printf("Split zero filled huge pages successful\n"); + free(one_page); +} + void split_pmd_thp(void) { char *one_page; @@ -431,6 +501,7 @@ int main(int argc, char **argv) fd_size = 2 * pmd_pagesize; + split_pmd_zero_pages(); split_pmd_thp(); split_pte_mapped_thp(); split_file_backed_thp(); diff --git a/tools/testing/selftests/mm/vm_util.c b/tools/testing/selftests/mm/vm_util.c index 5a62530da3b5..d8d0cf04bb57 100644 --- a/tools/testing/selftests/mm/vm_util.c +++ b/tools/testing/selftests/mm/vm_util.c @@ -12,6 +12,7 @@ #define PMD_SIZE_FILE_PATH "/sys/kernel/mm/transparent_hugepage/hpage_pmd_size" #define SMAP_FILE_PATH "/proc/self/smaps" +#define STATUS_FILE_PATH "/proc/self/status" #define MAX_LINE_LENGTH 500 unsigned int __page_size; @@ -171,6 +172,27 @@ uint64_t read_pmd_pagesize(void) return strtoul(buf, NULL, 10); } +unsigned long rss_anon(void) +{ + unsigned long rss_anon = 0; + FILE *fp; + char buffer[MAX_LINE_LENGTH]; + + fp = fopen(STATUS_FILE_PATH, "r"); + if (!fp) + ksft_exit_fail_msg("%s: Failed to open file %s\n", __func__, STATUS_FILE_PATH); + + if (!check_for_pattern(fp, "RssAnon:", buffer, sizeof(buffer))) + goto err_out; + + if (sscanf(buffer, "RssAnon:%10lu kB", &rss_anon) != 1) + ksft_exit_fail_msg("Reading status error\n"); + +err_out: + fclose(fp); + return rss_anon; +} + bool __check_huge(void *addr, char *pattern, int nr_hpages, uint64_t hpage_size) { diff --git a/tools/testing/selftests/mm/vm_util.h b/tools/testing/selftests/mm/vm_util.h index 9007c420d52c..71b75429f4a5 100644 --- a/tools/testing/selftests/mm/vm_util.h +++ b/tools/testing/selftests/mm/vm_util.h @@ -39,6 +39,7 @@ unsigned long pagemap_get_pfn(int fd, char *start); void clear_softdirty(void); bool check_for_pattern(FILE *fp, const char *pattern, char *buf, size_t len); uint64_t read_pmd_pagesize(void); +uint64_t rss_anon(void); bool check_huge_anon(void *addr, int nr_hpages, uint64_t hpage_size); bool check_huge_file(void *addr, int nr_hpages, uint64_t hpage_size); bool check_huge_shmem(void *addr, int nr_hpages, uint64_t hpage_size); From patchwork Tue Aug 13 12:02:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 13761887 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 214AAC52D7C for ; Tue, 13 Aug 2024 12:03:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 90ACD6B00A2; Tue, 13 Aug 2024 08:03:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8BDF66B00A4; Tue, 13 Aug 2024 08:03:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5FE1D6B00A3; Tue, 13 Aug 2024 08:03:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 382176B00A1 for ; Tue, 13 Aug 2024 08:03:51 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id EB8E6120809 for ; Tue, 13 Aug 2024 12:03:50 +0000 (UTC) X-FDA: 82447088220.03.53E6A80 Received: from mail-qk1-f181.google.com (mail-qk1-f181.google.com [209.85.222.181]) by imf10.hostedemail.com (Postfix) with ESMTP id F0949C0031 for ; Tue, 13 Aug 2024 12:03:48 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=BhiIX1Cu; spf=pass (imf10.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.222.181 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723550594; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ml/fx4q0/GpQb3pUk4RTjdvQZB9z9qmUFwqEb9M1LyE=; b=TIitCocOsWbtJg59vTGDVV2ieRuioyfacEVCJLfY+aYAnPTUErw8nirmbyud3osJdfodMD 8reCRUj8BmLCM9ARfnLCQ+VpSYLiDvyiVHHXxvyqiH6xSuksywOEcaS3UJm6aSGtDS3MGs Uwf5uQU8BG5GNaU6xoAw/JuoNuqnl6M= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=BhiIX1Cu; spf=pass (imf10.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.222.181 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723550594; a=rsa-sha256; cv=none; b=Fb/LK5Qs90rVwXBw4++xQnyTsWvPE2NI7NXevlXUAzX/Xr4uJFxVY+Zfp06W/sgLpXfdSL h4VdzetQOb0ilw3Zstd87NOzLB2nxwiGQAGNScCx+1ViCNW6R571/L2eZeHGQT9hM+P8j4 6sTs8ewUwm1jQidShszgYH6DBhoV4+A= Received: by mail-qk1-f181.google.com with SMTP id af79cd13be357-7a4df9dc840so72733385a.2 for ; Tue, 13 Aug 2024 05:03:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1723550628; x=1724155428; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ml/fx4q0/GpQb3pUk4RTjdvQZB9z9qmUFwqEb9M1LyE=; b=BhiIX1CugPrbnDrXNn1Sj9pMhSb9wm3LT9O5hd/WtMMO/8IJgosrXTKFbmxtLDlXaO u/MTPnjlAT7cVIvEtOJHp7PyaBVq8AZycrfXlmooiYEU7mViiK4bmQBWERxYSnjeNkLD srkH75lo+DN81BIO6wCzibUheNXGg6PQDRhB6N8SH2cfZzw1i3pVu3gNB5Sh89HwAYoq YhKCoJamAl3Q/sOJVGnapRdcrp+6xGv0WCkyS3z6CJlIohlZhPW9FdZK8pALSLezz2Fh +o8RRE7BBzWjGNJVUFVHDWC7Y6UQNc0kEoKqiOPDkCQ+uHqCGAaWaf0USSdAPCwkbDmk XG2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723550628; x=1724155428; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ml/fx4q0/GpQb3pUk4RTjdvQZB9z9qmUFwqEb9M1LyE=; b=FJLOq6nNVECNbhxLBlgkfRMmDMmhF8WzrfDlPhpq5u2a0mVqr62cVTKooJF0II+wQW XYA/M0zAz0gxlFoC0ozhbU+0+I7Tj7a2+h2E+2P8toOBYY8zbX51HY4XQt4hjjoHarup kwgol+1JxJaCbXTLMbUCHlo4bxRLblep0LOQZ1ypXU7m75npBvXk9CTOfXRqSU5vwyYU RQbv1fpC0DDsS350MeOz4MY0sYcYzFbyrK+PCx570X5rJDMn+3c1XvcZbuSTzCzngU1i 72WFVEyt94S3rUtJH4vQ/DcMyhLbi2Tt3Q3DaIi5hZ6YAVGDWqLWV9U/IHXcteWBCHyk t7Iw== X-Forwarded-Encrypted: i=1; AJvYcCUpeNeN1v/gy7NyJDTwTDdsrjvjTkFHmCKWQqpovDlVnUhHuv9arAeSnbF5sM8YOCsWtnSRfpFYG0ywC1uitrp2caI= X-Gm-Message-State: AOJu0YyAH4zqHEoiEdoWbHwxHk0uNWocelK900nybt9FUpra9DeU7pwr aB8ftJI+1DbtXywDIiqUyRv+Xz7/QZPsWO7eTHLSfuAbG8VYI21U X-Google-Smtp-Source: AGHT+IGnLAxIgNZTQh4vIeUHubCTk+9Li27aBYorSBwfeBTlQEsQV26mOZEo4xLzaw1PA1P3cAHmZQ== X-Received: by 2002:a05:620a:31a6:b0:7a3:7920:5500 with SMTP id af79cd13be357-7a4e15537c4mr389032385a.32.1723550627906; Tue, 13 Aug 2024 05:03:47 -0700 (PDT) Received: from localhost (fwdproxy-ash-006.fbsv.net. [2a03:2880:20ff:6::face:b00c]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a4c7d62678sm333062785a.5.2024.08.13.05.03.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Aug 2024 05:03:47 -0700 (PDT) From: Usama Arif To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, roman.gushchin@linux.dev, yuzhao@google.com, david@redhat.com, baohua@kernel.org, ryan.roberts@arm.com, rppt@kernel.org, willy@infradead.org, cerasuolodomenico@gmail.com, corbet@lwn.net, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kernel-team@meta.com, Usama Arif Subject: [PATCH v3 4/6] mm: Introduce a pageflag for partially mapped folios Date: Tue, 13 Aug 2024 13:02:47 +0100 Message-ID: <20240813120328.1275952-5-usamaarif642@gmail.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20240813120328.1275952-1-usamaarif642@gmail.com> References: <20240813120328.1275952-1-usamaarif642@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: F0949C0031 X-Stat-Signature: 6iw3apj84ka4oyot7iff5tnz1nsohtbh X-HE-Tag: 1723550628-291014 X-HE-Meta: U2FsdGVkX18HEBb9NdGu8fPtdud3TrenzKBzgIVX4DKEbL9Udj+axcwRE0WlTK2Zf58tCWRiRdigzrEH8EyBncJ0s/zs1XLUYhaXvL7fU8KlLrN0OTsEibLb+/xnBV6XP+H3uBQwWaOl/W88iM8mrPXGqmgLUZTIkfJZ2TvB7ZgHKtKaClH5X+QAkvSAy//LgpGO704/RVChJ/9lx8nw9KAMkfKqrcU9JbyG7a34ASNH1ZXVlLHvpaS4eLsz+p/2ljC29Iab3X9EBvaH4qRu6ejW5bmbXYRX+bq1XWsTQeQ0Fir3WPapG5ek/QGYJ1M5TeuPinUgYO7LmKGPLQBPDmKgMUCV+HxwPCVwCAroUkJFgaWyIGwoApbgnk2g3gbDt5cY0VyZH1HbogRRHMVPo64fPSenKugiueca5h6iPX7GQxEpdAEEaKbCiuuuQr0Y8/GUDy4d8ZFfFDZWRoy3M4UO1hemxHX54d7qa5ijqGhR6zGMmFIlU04SbhAfJKOPsptJiQfQvwvilS6VHPmwnXpafLzSHr9+80Sb85WNTSYm7iOQ8vNRRrrtwK9zvsGEB7gaXpkYXtxg68RrldbFy0fyltKv3H2o7KkfVCC+I6H0lb2TJs3fN/l9eZ1e/v1ueLGP2+zKOxN+Ivr5edLFd8LlVPrGy9I4vT22H8oVNXvmMllpvdy7+RFJcHbuEJMw/scRrBG7epBOcwM6W00uc5GtmCzuFo3CF5owAtE2gQXXrBEiQlPi1YYc/CpjUpdkJKNJRxLKJatFzlD5DDfbZNsCCX1fX+UYdJ/jGk2O2gdr5xi7aEJ7zu2Q7n9m4FgDt0R+AkFySVeGgBfffsy1tkaELVugH0dwoic55N4/W2QqBNXh92auvFW+7bbAuGUATlAPFvc+zeCmd/Oaj7WKg2Nf0u5SmNakfjot541LDL9FcWSQ/3Ju3yICR9ZPfvNO/5po2+VUejZq+gGT+3M cRPwTeir qoZNnbohL3uuORaIbLZl9IEoFEczm2y/zzzwrauPheRK1yNA4EQn/lHuPjO8wn5yRjESfE18J3JxvGMzu+FKTA8O5A2hxnLVAuzJqEojD4FTno2E100+OYOX/VHDg1HstrvHFQDjyqCPUWI9mvphWYUF7VygqPmOhaeosP3KqhzorQla2dwRPPvA3WoRUrQpqwrQ957yX0LcdqBP43Y5BdZEHTdwjLLihK5jqdC/GAQFUD0oE1+AALF0RSGI6LY2b9xOvOMpvghfqmlY8QDAU6D1K8hK4qk/jLZ4MxWPbiE0DyqTatxDxnE5QyD5nu4+A906IriQkeJ9MauejMfoqGbRZjA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently folio->_deferred_list is used to keep track of partially_mapped folios that are going to be split under memory pressure. In the next patch, all THPs that are faulted in and collapsed by khugepaged are also going to be tracked using _deferred_list. This patch introduces a pageflag to be able to distinguish between partially mapped folios and others in the deferred_list at split time in deferred_split_scan. Its needed as __folio_remove_rmap decrements _mapcount, _large_mapcount and _entire_mapcount, hence it won't be possible to distinguish between partially mapped folios and others in deferred_split_scan. Eventhough it introduces an extra flag to track if the folio is partially mapped, there is no functional change intended with this patch and the flag is not useful in this patch itself, it will become useful in the next patch when _deferred_list has non partially mapped folios. Signed-off-by: Usama Arif --- include/linux/huge_mm.h | 4 ++-- include/linux/page-flags.h | 3 +++ mm/huge_memory.c | 21 +++++++++++++-------- mm/hugetlb.c | 1 + mm/internal.h | 4 +++- mm/memcontrol.c | 3 ++- mm/migrate.c | 3 ++- mm/page_alloc.c | 5 +++-- mm/rmap.c | 3 ++- mm/vmscan.c | 3 ++- 10 files changed, 33 insertions(+), 17 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 4c32058cacfe..969f11f360d2 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -321,7 +321,7 @@ static inline int split_huge_page(struct page *page) { return split_huge_page_to_list_to_order(page, NULL, 0); } -void deferred_split_folio(struct folio *folio); +void deferred_split_folio(struct folio *folio, bool partially_mapped); void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, unsigned long address, bool freeze, struct folio *folio); @@ -495,7 +495,7 @@ static inline int split_huge_page(struct page *page) { return 0; } -static inline void deferred_split_folio(struct folio *folio) {} +static inline void deferred_split_folio(struct folio *folio, bool partially_mapped) {} #define split_huge_pmd(__vma, __pmd, __address) \ do { } while (0) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index a0a29bd092f8..cecc1bad7910 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -182,6 +182,7 @@ enum pageflags { /* At least one page in this folio has the hwpoison flag set */ PG_has_hwpoisoned = PG_active, PG_large_rmappable = PG_workingset, /* anon or file-backed */ + PG_partially_mapped, /* was identified to be partially mapped */ }; #define PAGEFLAGS_MASK ((1UL << NR_PAGEFLAGS) - 1) @@ -861,8 +862,10 @@ static inline void ClearPageCompound(struct page *page) ClearPageHead(page); } FOLIO_FLAG(large_rmappable, FOLIO_SECOND_PAGE) +FOLIO_FLAG(partially_mapped, FOLIO_SECOND_PAGE) #else FOLIO_FLAG_FALSE(large_rmappable) +FOLIO_FLAG_FALSE(partially_mapped) #endif #define PG_head_mask ((1UL << PG_head)) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 6df0e9f4f56c..c024ab0f745c 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3397,6 +3397,7 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, * page_deferred_list. */ list_del_init(&folio->_deferred_list); + folio_clear_partially_mapped(folio); } spin_unlock(&ds_queue->split_queue_lock); if (mapping) { @@ -3453,11 +3454,12 @@ void __folio_undo_large_rmappable(struct folio *folio) if (!list_empty(&folio->_deferred_list)) { ds_queue->split_queue_len--; list_del_init(&folio->_deferred_list); + folio_clear_partially_mapped(folio); } spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); } -void deferred_split_folio(struct folio *folio) +void deferred_split_folio(struct folio *folio, bool partially_mapped) { struct deferred_split *ds_queue = get_deferred_split_queue(folio); #ifdef CONFIG_MEMCG @@ -3485,14 +3487,17 @@ void deferred_split_folio(struct folio *folio) if (folio_test_swapcache(folio)) return; - if (!list_empty(&folio->_deferred_list)) - return; - spin_lock_irqsave(&ds_queue->split_queue_lock, flags); + if (partially_mapped) + folio_set_partially_mapped(folio); + else + folio_clear_partially_mapped(folio); if (list_empty(&folio->_deferred_list)) { - if (folio_test_pmd_mappable(folio)) - count_vm_event(THP_DEFERRED_SPLIT_PAGE); - count_mthp_stat(folio_order(folio), MTHP_STAT_SPLIT_DEFERRED); + if (partially_mapped) { + if (folio_test_pmd_mappable(folio)) + count_vm_event(THP_DEFERRED_SPLIT_PAGE); + count_mthp_stat(folio_order(folio), MTHP_STAT_SPLIT_DEFERRED); + } list_add_tail(&folio->_deferred_list, &ds_queue->split_queue); ds_queue->split_queue_len++; #ifdef CONFIG_MEMCG @@ -3541,6 +3546,7 @@ static unsigned long deferred_split_scan(struct shrinker *shrink, } else { /* We lost race with folio_put() */ list_del_init(&folio->_deferred_list); + folio_clear_partially_mapped(folio); ds_queue->split_queue_len--; } if (!--sc->nr_to_scan) @@ -3558,7 +3564,6 @@ static unsigned long deferred_split_scan(struct shrinker *shrink, next: folio_put(folio); } - spin_lock_irqsave(&ds_queue->split_queue_lock, flags); list_splice_tail(&list, &ds_queue->split_queue); spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 1fdd9eab240c..2ae2d9a18e40 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1758,6 +1758,7 @@ static void __update_and_free_hugetlb_folio(struct hstate *h, free_gigantic_folio(folio, huge_page_order(h)); } else { INIT_LIST_HEAD(&folio->_deferred_list); + folio_clear_partially_mapped(folio); folio_put(folio); } } diff --git a/mm/internal.h b/mm/internal.h index 52f7fc4e8ac3..d64546b8d377 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -662,8 +662,10 @@ static inline void prep_compound_head(struct page *page, unsigned int order) atomic_set(&folio->_entire_mapcount, -1); atomic_set(&folio->_nr_pages_mapped, 0); atomic_set(&folio->_pincount, 0); - if (order > 1) + if (order > 1) { INIT_LIST_HEAD(&folio->_deferred_list); + folio_clear_partially_mapped(folio); + } } static inline void prep_compound_tail(struct page *head, int tail_idx) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index e1ffd2950393..0fd95daecf9a 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4669,7 +4669,8 @@ static void uncharge_folio(struct folio *folio, struct uncharge_gather *ug) VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); VM_BUG_ON_FOLIO(folio_order(folio) > 1 && !folio_test_hugetlb(folio) && - !list_empty(&folio->_deferred_list), folio); + !list_empty(&folio->_deferred_list) && + folio_test_partially_mapped(folio), folio); /* * Nobody should be changing or seriously looking at diff --git a/mm/migrate.c b/mm/migrate.c index 3288ac041d03..6e32098ac2dc 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1734,7 +1734,8 @@ static int migrate_pages_batch(struct list_head *from, * use _deferred_list. */ if (nr_pages > 2 && - !list_empty(&folio->_deferred_list)) { + !list_empty(&folio->_deferred_list) && + folio_test_partially_mapped(folio)) { if (!try_split_folio(folio, split_folios, mode)) { nr_failed++; stats->nr_thp_failed += is_thp; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 408ef3d25cf5..a145c550dd2a 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -957,8 +957,9 @@ static int free_tail_page_prepare(struct page *head_page, struct page *page) break; case 2: /* the second tail page: deferred_list overlaps ->mapping */ - if (unlikely(!list_empty(&folio->_deferred_list))) { - bad_page(page, "on deferred list"); + if (unlikely(!list_empty(&folio->_deferred_list) && + folio_test_partially_mapped(folio))) { + bad_page(page, "partially mapped folio on deferred list"); goto out; } break; diff --git a/mm/rmap.c b/mm/rmap.c index a6b9cd0b2b18..9ad558c2bad0 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1579,7 +1579,8 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, */ if (partially_mapped && folio_test_anon(folio) && list_empty(&folio->_deferred_list)) - deferred_split_folio(folio); + deferred_split_folio(folio, true); + __folio_mod_stat(folio, -nr, -nr_pmdmapped); /* diff --git a/mm/vmscan.c b/mm/vmscan.c index 25e43bb3b574..25f4e8403f41 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1233,7 +1233,8 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, * Split partially mapped folios right away. * We can free the unmapped pages without IO. */ - if (data_race(!list_empty(&folio->_deferred_list)) && + if (data_race(!list_empty(&folio->_deferred_list) && + folio_test_partially_mapped(folio)) && split_folio_to_list(folio, folio_list)) goto activate_locked; } From patchwork Tue Aug 13 12:02:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 13761889 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A0993C52D7B for ; Tue, 13 Aug 2024 12:03:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E889B6B00A4; Tue, 13 Aug 2024 08:03:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E39B66B00A5; Tue, 13 Aug 2024 08:03:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CDB216B00A6; Tue, 13 Aug 2024 08:03:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id A93246B00A4 for ; Tue, 13 Aug 2024 08:03:52 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 22D6B1607CC for ; Tue, 13 Aug 2024 12:03:52 +0000 (UTC) X-FDA: 82447088304.27.9DBFAAA Received: from mail-qk1-f173.google.com (mail-qk1-f173.google.com [209.85.222.173]) by imf05.hostedemail.com (Postfix) with ESMTP id 50F7010001D for ; Tue, 13 Aug 2024 12:03:50 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="S/VyARDi"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf05.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.222.173 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723550573; a=rsa-sha256; cv=none; b=nNHiuurPRVRXzbDD9dBHO0fWTifAF8CeeRxcRkOthw0dbJ0L746GTny5zASNLW8a7syxt3 CrFb/+KMO/iuX0rTb7HgnL0ReM/2C7AgzzMILOXCLAJtIV/mBFqDSK5UrIWoY3rXbZnNnX 6p2ppacuYoDkOUBVSVROcidc9TwgVrY= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="S/VyARDi"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf05.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.222.173 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723550573; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/3e8jqaD9j4AlkmfSijGsYDb40oSO+aTFas99401M0M=; b=Pw1kRA1EgtbTKX+bXXJ0gI0qAWBkgzoQOm4gWa0Md6OyV64c/isPduCCAUwW4IkTnN3EMU MIBbfpUUfw5ECN8VrgPwgC6u+5tvGLVWvIFC34RWKbL+OsDqtupxoaXA3ZxMerFfIwWIiI pwL+mPQVp6oT9EOXl8HVNotNp0hxHX8= Received: by mail-qk1-f173.google.com with SMTP id af79cd13be357-7a1dd2004e1so337486685a.3 for ; Tue, 13 Aug 2024 05:03:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1723550629; x=1724155429; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/3e8jqaD9j4AlkmfSijGsYDb40oSO+aTFas99401M0M=; b=S/VyARDigTW+Z58CjvJS+yVU3aGpPypSFHXr05T8E8mWd9jd3vH6Av3LQ4bJT8bKnP I+6KDhfpaeyA9mlZGs6zP8YzcmOeOcNnAo0IKTMFrKKHX7wQelC+dU5zYeNRMAScHg/w UcT/dOvEef0QiCE7k/cnG0d1x2ii5uu5LdAX6EdLQUYP1nDGTPW9FANqz4BM6ASNU0Al ZgxiHDCbBT+tHdysFUpPJ6GsyIihNggyBX2kK3WjSPggZ0JqwibQldmCxG1VqBBKuv// 7F19fiNDQnSpFS54LQXQI0kwTtz9yRwjo6HGKGemnWkN/Oewp+gC5nvRgOXprlDQ1pZV PcFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723550629; x=1724155429; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/3e8jqaD9j4AlkmfSijGsYDb40oSO+aTFas99401M0M=; b=KzoOPRwk0FBvxEUPz+NP4xVYupzNqjGkC5/3/wrTOgxDKGzWfd1HOm+4qyXhK9lD5f Ik+E8MGk0QZw5EgInwn3ecXg2M0QlIwg0DdxHbsNMkQQPDmrx68niaMBCN/+EbJQ56ND UPLeLYCtkcR482GaE2XNJt0kUJZpLUP+U5aymfzXsi7scxGVvcEuGcsK7ACF25jE4xxo PHbbYzHOo9beodFKwlI/UYSSoKaDKmvhaq3TeavOcvOcs++0+QCDAZmXn/K+aBuoGQUr +UnGrf9W8q58hurhTsGL4hKaS+SjZIe3KGhn0lZNbxkw/zNO7OGTcsi5EhF7zVjmLsTA upmw== X-Forwarded-Encrypted: i=1; AJvYcCXZAVqO0NzElrRvWPDkLcn7ChOYlm2mjxWfmgKa9ItRYS2dmTovAH2XLBE+Nza3z4T7/nbUcOMzq9EAHmUoSl6DXAU= X-Gm-Message-State: AOJu0YzgUyD04lL0b4wVzjDzd2Uo5ccG/XCoc82vxSJEYUeM6vrZc3/k gPbY8Z6dwMYv0t2bSZKBMle1qWO1sbvNqnCE5RD9CKk//BNLtXIN X-Google-Smtp-Source: AGHT+IEFfd2FYRK2Jx6dZup0/ifNGdrGzF8HSf/AxkPfSro8RvwQqrR7qIuXRhOvG6+PgXGqlDdltw== X-Received: by 2002:a05:620a:2441:b0:79f:b72:fb30 with SMTP id af79cd13be357-7a4e1667d8amr395537185a.59.1723550629311; Tue, 13 Aug 2024 05:03:49 -0700 (PDT) Received: from localhost (fwdproxy-ash-000.fbsv.net. [2a03:2880:20ff::face:b00c]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a4c7e040e0sm333193985a.101.2024.08.13.05.03.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Aug 2024 05:03:48 -0700 (PDT) From: Usama Arif To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, roman.gushchin@linux.dev, yuzhao@google.com, david@redhat.com, baohua@kernel.org, ryan.roberts@arm.com, rppt@kernel.org, willy@infradead.org, cerasuolodomenico@gmail.com, corbet@lwn.net, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kernel-team@meta.com, Usama Arif Subject: [PATCH v3 5/6] mm: split underutilized THPs Date: Tue, 13 Aug 2024 13:02:48 +0100 Message-ID: <20240813120328.1275952-6-usamaarif642@gmail.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20240813120328.1275952-1-usamaarif642@gmail.com> References: <20240813120328.1275952-1-usamaarif642@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 50F7010001D X-Stat-Signature: igb1gkzja5e995zbraixfqp8cjjxudw6 X-Rspam-User: X-HE-Tag: 1723550630-474449 X-HE-Meta: U2FsdGVkX19OZ1q4GaFbVuNJjJdBQyZ9yyti/a9l5Wam4pEu3+3fBnClt1Yb+llbjDl2ykVJEMTO9P19VKmERDxW7XrY5Hn9bjPoDmt4el+xu7bXqwiVMQhfFCguVvci/tNStydj8aUFkHmMwzkAKZlP35Ar4BUEWUq2OD73Z70T8Rfk7nGnE0hdRmddUGRCcRpbLnKntWaf1Tkh9ph7vor+0zSj5CutpamwfP4nbw/vVO+ApJU3F6Fux4G/JpVh1xGmS1/KS0fdKufiaxR+qotde5Ay+kuP9o7kzBnjsAS1c0dJFJQhhhvTIDE19BcPIhLb38HY/yG33QbQFEbGV80f8NA6J5PWmQbbSUFbxhgznlULoZrAMt2A/9dlCcxa+83vIHYOcaQu+niONlLWsIFcbm9B2sdHstKCqCzEi8MAzF2VyDygUl1dAEfVFT0M8qC7X90EAiU06nFDsjqzwXY0CUF53bJQp85M+t/JhgKRTKmEyCZrT4K4OQWRDnEes2XQa8KdU5FyBcanUKNOPHnAwarbXVQu0SCjTQ8dgQ9Am7haf67ppEpmEq5KRjVJWi3caEQRjquzHzI+KJUmA6A1jzvusRMS1KRLepRXlkbVmZR0VHO+Nk5OTF410SII3c3XinxqNb8DnoVUxhj5JRvdmFXKwdsL7anCHL3GoXMHbrknwVSgIS91ZHKCT3RAzX9BeFLiM/nbwo2EAHDHUTkNaLvPoBDEAlLC7Zb3SXSqGljWRc3szg4/ePhkLGex+gs3Z8JqUqxysVoKYGrkIemL28xNnmx9SHAHwmnldDaC2SIVzX8Qtgz/x0JTsKCyi3gZD/Ln0TbZ6CpdfDQ/lWQhLXDxW1tDI0zvGoq1rm5q4BdgZHqRJvQfaE6d2cjExGEjLlduMsJ2bc0Ohd6HiC17dRHPKqAEksGwrsYrCXKZtmZr4YlBGRUbarTy13BSgEHe1cdmZ8g5hSGhjw2 Vu0byNYJ m6STw1hp9mw6ObX4ufJrcdy0VGhhkvSxkzZvsyFwSsd17/0iMupamx9aCSXrIbu4RtjPUbBS6qB/yhcaw2ZgL0Gqx7ggaRLqXAx0OPbVVorqpSzDP8RNrHY7MghMv9ujshELfgRuVRPDVqKaXLj3bWWbypUMatQuWWnphPm7GjtsWFxBuNgxsOmoi5j7JoRoZlhCzs453M2/NV5avU/chU1I597PqqIGxbcCTV1v3iRj1CvNmnQ7xQ2D+kxwBvX23brAEOfPG84yc7gftJRnv9X0boW1C4qSiEo6z51RRYSyyKuY2ujo0NCNOvztBjrWGfULQ8XUIDLiiz43aAJo3xa6KAruPqc+2Zpc5UQawWNTylOfzREhgViW0CaOOGewmRwxkF3F8yPsdkJhjnaVRW39zkN3TeGE9wAJPSVnIQZF8tLju9NptE4GnuqI+FZnBVMppDepcwLuKIug= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is an attempt to mitigate the issue of running out of memory when THP is always enabled. During runtime whenever a THP is being faulted in (__do_huge_pmd_anonymous_page) or collapsed by khugepaged (collapse_huge_page), the THP is added to _deferred_list. Whenever memory reclaim happens in linux, the kernel runs the deferred_split shrinker which goes through the _deferred_list. If the folio was partially mapped, the shrinker attempts to split it. If the folio is not partially mapped, the shrinker checks if the THP was underutilized, i.e. how many of the base 4K pages of the entire THP were zero-filled. If this number goes above a certain threshold (decided by /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none), the shrinker will attempt to split that THP. Then at remap time, the pages that were zero-filled are mapped to the shared zeropage, hence saving memory. Suggested-by: Rik van Riel Co-authored-by: Johannes Weiner Signed-off-by: Usama Arif --- Documentation/admin-guide/mm/transhuge.rst | 6 ++ include/linux/khugepaged.h | 1 + include/linux/vm_event_item.h | 1 + mm/huge_memory.c | 76 ++++++++++++++++++++-- mm/khugepaged.c | 3 +- mm/vmstat.c | 1 + 6 files changed, 80 insertions(+), 8 deletions(-) diff --git a/Documentation/admin-guide/mm/transhuge.rst b/Documentation/admin-guide/mm/transhuge.rst index 058485daf186..60522f49178b 100644 --- a/Documentation/admin-guide/mm/transhuge.rst +++ b/Documentation/admin-guide/mm/transhuge.rst @@ -447,6 +447,12 @@ thp_deferred_split_page splitting it would free up some memory. Pages on split queue are going to be split under memory pressure. +thp_underutilized_split_page + is incremented when a huge page on the split queue was split + because it was underutilized. A THP is underutilized if the + number of zero pages in the THP is above a certain threshold + (/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none). + thp_split_pmd is incremented every time a PMD split into table of PTEs. This can happen, for instance, when application calls mprotect() or diff --git a/include/linux/khugepaged.h b/include/linux/khugepaged.h index f68865e19b0b..30baae91b225 100644 --- a/include/linux/khugepaged.h +++ b/include/linux/khugepaged.h @@ -4,6 +4,7 @@ #include /* MMF_VM_HUGEPAGE */ +extern unsigned int khugepaged_max_ptes_none __read_mostly; #ifdef CONFIG_TRANSPARENT_HUGEPAGE extern struct attribute_group khugepaged_attr_group; diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index aae5c7c5cfb4..bf1470a7a737 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -105,6 +105,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, THP_SPLIT_PAGE, THP_SPLIT_PAGE_FAILED, THP_DEFERRED_SPLIT_PAGE, + THP_UNDERUTILIZED_SPLIT_PAGE, THP_SPLIT_PMD, THP_SCAN_EXCEED_NONE_PTE, THP_SCAN_EXCEED_SWAP_PTE, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index c024ab0f745c..6b32b2d4ab1e 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1087,6 +1087,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, update_mmu_cache_pmd(vma, vmf->address, vmf->pmd); add_mm_counter(vma->vm_mm, MM_ANONPAGES, HPAGE_PMD_NR); mm_inc_nr_ptes(vma->vm_mm); + deferred_split_folio(folio, false); spin_unlock(vmf->ptl); count_vm_event(THP_FAULT_ALLOC); count_mthp_stat(HPAGE_PMD_ORDER, MTHP_STAT_ANON_FAULT_ALLOC); @@ -3522,6 +3523,39 @@ static unsigned long deferred_split_count(struct shrinker *shrink, return READ_ONCE(ds_queue->split_queue_len); } +static bool thp_underutilized(struct folio *folio) +{ + int num_zero_pages = 0, num_filled_pages = 0; + void *kaddr; + int i; + + if (khugepaged_max_ptes_none == HPAGE_PMD_NR - 1) + return false; + + for (i = 0; i < folio_nr_pages(folio); i++) { + kaddr = kmap_local_folio(folio, i * PAGE_SIZE); + if (!memchr_inv(kaddr, 0, PAGE_SIZE)) { + num_zero_pages++; + if (num_zero_pages > khugepaged_max_ptes_none) { + kunmap_local(kaddr); + return true; + } + } else { + /* + * Another path for early exit once the number + * of non-zero filled pages exceeds threshold. + */ + num_filled_pages++; + if (num_filled_pages >= HPAGE_PMD_NR - khugepaged_max_ptes_none) { + kunmap_local(kaddr); + return false; + } + } + kunmap_local(kaddr); + } + return false; +} + static unsigned long deferred_split_scan(struct shrinker *shrink, struct shrink_control *sc) { @@ -3555,17 +3589,45 @@ static unsigned long deferred_split_scan(struct shrinker *shrink, spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); list_for_each_entry_safe(folio, next, &list, _deferred_list) { + bool did_split = false; + bool underutilized = false; + + if (folio_test_partially_mapped(folio)) + goto split; + underutilized = thp_underutilized(folio); + if (underutilized) + goto split; + continue; +split: if (!folio_trylock(folio)) - goto next; - /* split_huge_page() removes page from list on success */ - if (!split_folio(folio)) - split++; + continue; + did_split = !split_folio(folio); folio_unlock(folio); -next: - folio_put(folio); + if (did_split) { + /* Splitting removed folio from the list, drop reference here */ + folio_put(folio); + if (underutilized) + count_vm_event(THP_UNDERUTILIZED_SPLIT_PAGE); + split++; + } } + spin_lock_irqsave(&ds_queue->split_queue_lock, flags); - list_splice_tail(&list, &ds_queue->split_queue); + /* + * Only add back to the queue if folio is partially mapped. + * If thp_underutilized returns false, or if split_folio fails in + * the case it was underutilized, then consider it used and don't + * add it back to split_queue. + */ + list_for_each_entry_safe(folio, next, &list, _deferred_list) { + if (folio_test_partially_mapped(folio)) + list_move(&folio->_deferred_list, &ds_queue->split_queue); + else { + list_del_init(&folio->_deferred_list); + ds_queue->split_queue_len--; + } + folio_put(folio); + } spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); /* diff --git a/mm/khugepaged.c b/mm/khugepaged.c index cdd1d8655a76..02e1463e1a79 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -85,7 +85,7 @@ static DECLARE_WAIT_QUEUE_HEAD(khugepaged_wait); * * Note that these are only respected if collapse was initiated by khugepaged. */ -static unsigned int khugepaged_max_ptes_none __read_mostly; +unsigned int khugepaged_max_ptes_none __read_mostly; static unsigned int khugepaged_max_ptes_swap __read_mostly; static unsigned int khugepaged_max_ptes_shared __read_mostly; @@ -1235,6 +1235,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, pgtable_trans_huge_deposit(mm, pmd, pgtable); set_pmd_at(mm, address, pmd, _pmd); update_mmu_cache_pmd(vma, address, pmd); + deferred_split_folio(folio, false); spin_unlock(pmd_ptl); folio = NULL; diff --git a/mm/vmstat.c b/mm/vmstat.c index c3a402ea91f0..91cd7d4d482b 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1384,6 +1384,7 @@ const char * const vmstat_text[] = { "thp_split_page", "thp_split_page_failed", "thp_deferred_split_page", + "thp_underutilized_split_page", "thp_split_pmd", "thp_scan_exceed_none_pte", "thp_scan_exceed_swap_pte", From patchwork Tue Aug 13 12:02:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 13761890 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D5CBC52D7C for ; Tue, 13 Aug 2024 12:04:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EAFA46B00A5; Tue, 13 Aug 2024 08:03:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E38916B00A6; Tue, 13 Aug 2024 08:03:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C3C7E6B00A7; Tue, 13 Aug 2024 08:03:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A04F36B00A5 for ; Tue, 13 Aug 2024 08:03:54 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 55BFC14080B for ; Tue, 13 Aug 2024 12:03:54 +0000 (UTC) X-FDA: 82447088388.05.8769B00 Received: from mail-qv1-f41.google.com (mail-qv1-f41.google.com [209.85.219.41]) by imf12.hostedemail.com (Postfix) with ESMTP id B766C4000F for ; Tue, 13 Aug 2024 12:03:51 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="b1/Jcwr2"; spf=pass (imf12.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.219.41 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723550619; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fpDDmqOnJijcRyZ/zRVr0dfh/EUV8mM0nwD7hiV/Er4=; b=A5McyKCGGr458qX1AyUV8qHNuLjmQNZR43h2UalFHBrLJGUcnajK1h+lXOfiA4plEZ5May attq2C39bX2A4d5vjyeGh5i8H3DUsi7kru3WrKY0N55Kf0jN/b+beJyelS422FsFj0Y8xn ZqJ7AuUMOgmUUxr2eS1+nkeJ/j4CZoE= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="b1/Jcwr2"; spf=pass (imf12.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.219.41 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723550620; a=rsa-sha256; cv=none; b=Sn5IWPDlLjVbabFckDvvJ3D5h+KpAah4q7eN4mhtQushbb+XSbkadMSvnivETuCGKykHgH pi3xiA2kUtnhaUZ4z4oPJc/8WM3lnMPVtgBPhxC3wzq2msIEcy+LERsaEgM2w2asGpsnxG DjqhjYqQWvnGa4nQVPUH315qiCzx7HE= Received: by mail-qv1-f41.google.com with SMTP id 6a1803df08f44-6bbc1cee9b7so32087476d6.3 for ; Tue, 13 Aug 2024 05:03:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1723550630; x=1724155430; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fpDDmqOnJijcRyZ/zRVr0dfh/EUV8mM0nwD7hiV/Er4=; b=b1/Jcwr2P0cWdUlmVaFUCQbtjRmwuveBlMdTHFe4xFKVLdBetufTmdZYD4Udx5lXH/ 7q7yZHVB6usGwCSDl3iImGuRV4S68k3x6K63GNOqHxbejIXIKgkWZNUNGX5rWLpI37Oe SBE53pgOVA54uryL+hr3qrFVS1Ac64V+jBwDQfwkkPwLzZitKaoZTZqXtUg795+HjPmv BqnordLZ6HrTZ5JO7bwyEF/hiBXioliKWsJkxNiUsQk2xITHO1xqCijfY6E9m1WXFJFs pBbJopVAXVVWuEGzV3qK0zNzjvl1QXYMWDScw0nOT0eTWKiYKYAbib1lCti+NRIQaHJw uHjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723550630; x=1724155430; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fpDDmqOnJijcRyZ/zRVr0dfh/EUV8mM0nwD7hiV/Er4=; b=VPs4oQjnqiTI0UakBXmBjswi9jOBwPy9Npih2rd9K7M2en3V2ksuGJVAhSh/crXqfW HlxWMYbspAbON/A70TnbRXIV6hyb3WxYFlfGdUrki1ahiurIoEou72JA+U0jcH7TGJff oG4+1Fci+Q8/PQnuwchCx2EV7oESNYgcFK59IJnuJ6X14oM+23saxs35YrxL5A94yw05 hmWcca0l8vG5Xfztres0oY+NWKhvtOAdw2WwtLzuXk329qJNfNK8GFLimyw/5u+44erV jBcGzJfu0UIPJGrhfyjwV13fgk79V9xu8aD99e/AldJN9IzmEnoD8LIfeBYxqHGgInKG rq9Q== X-Forwarded-Encrypted: i=1; AJvYcCWnylAQl4R6v0KQGQwOUDQXFIevluh8nV7bLFHBPDpi2RbVbeK7PebGYDfygffv2Xqnf7yI9kZgX/81x5eu9mbeyPk= X-Gm-Message-State: AOJu0YxAF9dYVKsN379jglOaP5ye702GK5gcqF/Fi7E8t//1eWhJwpS6 OUiEFjI5Kyx3k7sKNcUdCwQXf9TnxnoUNBVKKvYGauLbTvtzBxiLKxp5fCRr X-Google-Smtp-Source: AGHT+IG/p4KHdjtbRHl8jrRDtZpiTzA00oY1agex2KO6TAfMpoEbk5fNk2V++Y8FeIZ5s4O/KzF07g== X-Received: by 2002:a05:6214:5f0b:b0:6b7:980b:e0ac with SMTP id 6a1803df08f44-6bf4f7e2315mr29329476d6.32.1723550630417; Tue, 13 Aug 2024 05:03:50 -0700 (PDT) Received: from localhost (fwdproxy-ash-011.fbsv.net. [2a03:2880:20ff:b::face:b00c]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6bd82c81cd0sm33464656d6.53.2024.08.13.05.03.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Aug 2024 05:03:50 -0700 (PDT) From: Usama Arif To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, roman.gushchin@linux.dev, yuzhao@google.com, david@redhat.com, baohua@kernel.org, ryan.roberts@arm.com, rppt@kernel.org, willy@infradead.org, cerasuolodomenico@gmail.com, corbet@lwn.net, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kernel-team@meta.com, Usama Arif Subject: [PATCH v3 6/6] mm: add sysfs entry to disable splitting underutilized THPs Date: Tue, 13 Aug 2024 13:02:49 +0100 Message-ID: <20240813120328.1275952-7-usamaarif642@gmail.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20240813120328.1275952-1-usamaarif642@gmail.com> References: <20240813120328.1275952-1-usamaarif642@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: r8njj9ypqrpza7xw35dr9g7nwjxepjnd X-Rspamd-Queue-Id: B766C4000F X-Rspamd-Server: rspam11 X-HE-Tag: 1723550631-588671 X-HE-Meta: U2FsdGVkX1/RzsgS6idMtji/5yoPf85sChspLzhZQZND78Tww9EFuDwAOxV+It58hdPJb+iwglK1h66R2BRTAa/xNzCtgUaX3QA+SoFATN7J77uFqtW6PUD7SsxWp4bo6L8WrSLdrTtD5PQJ4OxIU5312Vf7A3cRkW1+ZkYK9EYiI7T5ZXCF8yODY9O/xuPoiNDU51j3qUqLTEfzY6SiydCyvtkXvK5vt2e7gy50sSjKJmUwkX/0nv3JFJVOUEcjudNRvVYCX6Yd8UbN4RKworxHVnLFyRQ3Kf7l4OGlUlconojaoZkNwV8saxhEUL1WYSILxUWh26uU3/xDpkhr469eqYqG9HEwBOSJsdFHK6DvKLPIffnWYKCr736YP/0hycJVKm/B0zvIe3+ZKn8Ikp7/fjtwcx5G41Xw5rHRKTecxIEfuXXX7jt7ocFiYhCSZf3C/M7KmAs7WxkjD+63C/f4Inp5Fk2o2oNm+u6MucV7eNu4YQcOY16Z8HSAIMiT7KMM40+5I2UR0MuBI+642mYjjGFw+Q6qqIglQxjmfk8jT5dc8dGDaVkVc+gWi23UOsrxugYyWxm2jVb4LUCVMwXVCfhqugy9uxHKkLOmytzdQGB6mOZP3+EXmz0D9hHXi+Yle/m3XJrO95wgXPP3V0YEz8xsb4AI1QZcqhPEPYYhwRrSZ3MC5cGPlWqx6lUAU9whX1U/nmzp/ouodFJyQJ7c94tQSW1469gBEcYaIEqgaKuu2BeZ3uhZLRp4FDKEL2RL+b43Mv5Wf00/+N14OBdyyLuffcpRpOiLxZKcCVgxxSSdcqEpG9jh17g5fWqvNfZa57IPMIqIK4o8Ukdnl9kbg164AjO3RW8dQlBir6VVcvTGU01pHfFtRKuN40jIeI4kXF3uacFqDlRCDqvzjMWicf6Opcg+o0Xs9b/DUGuBuNYxU68qTADYeMcTJh1sr5RWjDke5MAHL1HHVA1 NG3vAWIv oP0hl0nPGND6y7a8yAOuSPBwUMEu8oy/DndN9oq/g6UmLaA7wSsYrgYL7mG+KBxi4r5dz3cap3I5DyEe9dn8CU3NVHlBPxXkL01WDmtL25wVco8w63nGIiGfu996/Je0f2mMHCWkfCKCBy0ODyZ+pFxGpIY5K1eaEQadjIUGKf3KADLVKVaFfldS3t6YtY8/Tyla/rNH8JYBuY65oGvewjUH/H1vQat2K20q0ki5PIwLSO4EOsP1B9pOWd1UxRhC1H9/YciYpf8AMO/oOihJBhh2xc7uts6sqaBZ4dMLsnNhL4iVmI/JaG5VvjKTV1s+zOTlJXKjm77b1hT7zPj/ePi4HsQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: If disabled, THPs faulted in or collapsed will not be added to _deferred_list, and therefore won't be considered for splitting under memory pressure if underutilized. Signed-off-by: Usama Arif --- mm/huge_memory.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 6b32b2d4ab1e..b4d72479330d 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -74,6 +74,7 @@ static unsigned long deferred_split_count(struct shrinker *shrink, struct shrink_control *sc); static unsigned long deferred_split_scan(struct shrinker *shrink, struct shrink_control *sc); +static bool split_underutilized_thp = true; static atomic_t huge_zero_refcount; struct folio *huge_zero_folio __read_mostly; @@ -439,6 +440,27 @@ static ssize_t hpage_pmd_size_show(struct kobject *kobj, static struct kobj_attribute hpage_pmd_size_attr = __ATTR_RO(hpage_pmd_size); +static ssize_t split_underutilized_thp_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + return sysfs_emit(buf, "%d\n", split_underutilized_thp); +} + +static ssize_t split_underutilized_thp_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + int err = kstrtobool(buf, &split_underutilized_thp); + + if (err < 0) + return err; + + return count; +} + +static struct kobj_attribute split_underutilized_thp_attr = __ATTR( + thp_low_util_shrinker, 0644, split_underutilized_thp_show, split_underutilized_thp_store); + static struct attribute *hugepage_attr[] = { &enabled_attr.attr, &defrag_attr.attr, @@ -447,6 +469,7 @@ static struct attribute *hugepage_attr[] = { #ifdef CONFIG_SHMEM &shmem_enabled_attr.attr, #endif + &split_underutilized_thp_attr.attr, NULL, }; @@ -3475,6 +3498,9 @@ void deferred_split_folio(struct folio *folio, bool partially_mapped) if (folio_order(folio) <= 1) return; + if (!partially_mapped && !split_underutilized_thp) + return; + /* * The try_to_unmap() in page reclaim path might reach here too, * this may cause a race condition to corrupt deferred split queue.