From patchwork Tue Jan 7 00:03:41 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gregory Price X-Patchwork-Id: 13927908 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52D47E77199 for ; Tue, 7 Jan 2025 00:03:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CADC16B00C5; Mon, 6 Jan 2025 19:03:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C323B6B00C6; Mon, 6 Jan 2025 19:03:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AAC7D6B00C7; Mon, 6 Jan 2025 19:03:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 7E8706B00C5 for ; Mon, 6 Jan 2025 19:03:55 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 2BE221A0512 for ; Tue, 7 Jan 2025 00:03:55 +0000 (UTC) X-FDA: 82978707630.18.B822434 Received: from mail-qv1-f52.google.com (mail-qv1-f52.google.com [209.85.219.52]) by imf20.hostedemail.com (Postfix) with ESMTP id 58EAE1C0007 for ; Tue, 7 Jan 2025 00:03:53 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=gourry.net header.s=google header.b=shHBcH37; dmarc=none; spf=pass (imf20.hostedemail.com: domain of gourry@gourry.net designates 209.85.219.52 as permitted sender) smtp.mailfrom=gourry@gourry.net ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736208233; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=h+k7/fZDgwShiWAnDxVUmiJM02Oc5NicKXchGCm3s6Y=; b=xrAc/b4a+coahNuL044tNDvIp5RkXRRhbQc4ZJFw2eTItXtfaAbW4sYIAjEVoFWOCu8IRQ uPp51r/gFpHxp/z0d9hcKCETWn7Q7u7tNe4KcVybY8HQ1j+KbxwI4O43auO+gtCNXg1pgM NFTQwuVLkcjbH7tqFS3AOEnL6ubv8Kw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736208233; a=rsa-sha256; cv=none; b=m8ygUhKgqXDzwoClQVKN3vwrP7aPCqEi9G9Yjxnz/Aw2ScOeZ0ncsdQ2ZD4OtSyl1oLKS+ etVeXuSiOQsC/CZCfrzGr4VLLTn3pm3TKPoiStyr1RjzIo4LkpQ5Vby2w5QKiQzLznaLdW yJvpbJoSdP8nDcLtkX/YRXx1hGLMq9M= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=gourry.net header.s=google header.b=shHBcH37; dmarc=none; spf=pass (imf20.hostedemail.com: domain of gourry@gourry.net designates 209.85.219.52 as permitted sender) smtp.mailfrom=gourry@gourry.net Received: by mail-qv1-f52.google.com with SMTP id 6a1803df08f44-6d91653e9d7so131817206d6.1 for ; Mon, 06 Jan 2025 16:03:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1736208232; x=1736813032; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=h+k7/fZDgwShiWAnDxVUmiJM02Oc5NicKXchGCm3s6Y=; b=shHBcH37yjy8ogoZsMHILb8cZn9OaRZw52f/OvSzGvznxhhFRTYcYxtXqcHCVRZYZD u316Off3JDHxoJTctHxFaalFSMIPtQzEYRDhGd8CSA5zLD/Blsn4wbNp/70bNj5LjJj6 FKFK2LxrEFbHB27RtTACI13gW36dMnyiVbH9iaqul6+m5J4IkEUnIXCkfbOXmUwFZwZl v29krnmZZMb3lhBBFRepIZai4lsBqgtvTDWBoS2JCNi5v2jzKQtZrrzGZTrT+sMSjSYe BCkzUe7vo9ifiMHLGnN5WrqWcczFnhb0jzyV/1Ey3M/wYtiqoS0ygAUVv2qJK82X0zP8 po1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736208232; x=1736813032; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=h+k7/fZDgwShiWAnDxVUmiJM02Oc5NicKXchGCm3s6Y=; b=BWd5syPw0Pi+RIOMDwcGezynFfGLejAANBROEwAoOhvWy9NIX5Trmd7B1+DEkrL3n/ WVRHEGqXzA3DI2rYaq7gJQQ5JiAhg5YJEo482xFAn/dMHfNBY6cIGLigxfpAfsK/hYTI Owbi5RNcY/lbh/NRxRCcFQP6Hm7MISsgtZ30cpqV6FGgQcVJvbramOrChD3145XMcf44 XRi8XCfWGmsk+1ZcgonsCOw+Yi7OckxS19OVSUsE53hXIFlyhjJLt/NtOu20jjD6YDwr iflP9V6LoSPg4r/+7OmgC0nRbOyLx5TyOxMrePFrHjdmG1rZMnBt+hA3rhYpq8GHwxtw NAqg== X-Gm-Message-State: AOJu0YwzoF1Y9o7FH6eEDXJxA2DpdOX4G7gZ05XCPtJAUtcvQx7hzTUi hhO58wAorwnyglNF8lx3IUHGCpRFIEpbGuNUVQPRH0QKIzWwV0EcysN8CpllFDG1QyTmprAA7rF J X-Gm-Gg: ASbGnctYXMcxtBLd23GWf/PQ31RMJmfN3hSi+GMUW0jDsfQxLbdp0MhKQXrWWANdmy2 5VX0ih7O01YTuqDBt8G4+jMXZaKBHFRXEkkDpQll7Lz1SLXCiXpsc5jBCrDdJ9TMbx+Ccdl2Sit 5ODVqWNmwQ041OYIcRrB3VTVecmE+EShdOOcEImat++Ix72Inx2AI3kzaUNgHqHMQ0S9WgjS9pB rd/7dqOnc0TXVtOAE4X170h9hyeczBTKHgg0zgzIlmh5JRkCQTAThN8ReWdbtGNMoNnhh2cIFoX 3drEIBIcq6tA37cnxn+oKfmOizZMN0YGq2DhnBk/elEY X-Google-Smtp-Source: AGHT+IHOqKIeQ7ampwkkS1JPMTLmaW/686eOaPbp3AeW5D0x9/M0zZRBfxMo3FbicJmEkXL/0K0/aw== X-Received: by 2002:a05:6214:31a0:b0:6d8:81cd:a0d2 with SMTP id 6a1803df08f44-6dd233ad8e3mr1025601246d6.41.1736208232286; Mon, 06 Jan 2025 16:03:52 -0800 (PST) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6dd18137218sm174104476d6.57.2025.01.06.16.03.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Jan 2025 16:03:51 -0800 (PST) From: Gregory Price To: linux-mm@kvack.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, nehagholkar@meta.com, abhishekd@meta.com, david@redhat.com, nphamcs@gmail.com, gourry@gourry.net, akpm@linux-foundation.org, hannes@cmpxchg.org, kbusch@meta.com, ying.huang@linux.alibaba.com, feng.tang@intel.com, donettom@linux.ibm.com Subject: [PATCH v3 1/6] migrate: Allow migrate_misplaced_folio_prepare() to accept a NULL VMA. Date: Mon, 6 Jan 2025 19:03:41 -0500 Message-ID: <20250107000346.1338481-2-gourry@gourry.net> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250107000346.1338481-1-gourry@gourry.net> References: <20250107000346.1338481-1-gourry@gourry.net> MIME-Version: 1.0 X-Stat-Signature: zdtpnrqzkgntbucr8y39wmcu99ut4fis X-Rspamd-Queue-Id: 58EAE1C0007 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1736208233-95547 X-HE-Meta: U2FsdGVkX1/NWf1V2vnF29Kw0+9gRwFmkbB/LR9K/2ZycQBvL2mkJ62qicZIz41XJjAL3tjR8buFgAu8f9DFN0jaD0eoCEFQvcRrRomhCnbUsww4fVEttYwukj6/1cTwPQdiCRDMCaiqcRHPuDqI7nyiccjh8P+xwmM1gq95j/KebHIYazecTmwzOqqnDPvF2mDPMia9Q548QPxs3ovBL5150xZGtsmueP+qbbYevVN6WRrnnywhhf8y+W7WvVmDhn1yZOvUZ8sdJi5iiKEd/dj+Z/Y+JtifnYQhdzIC5ON2UsPUYMAVNnj7AazSbt5EL33x1PEi4BN4ecpfccpdu/L+JM4+ymcH4R31sWEC6ei7dLyeIrFP5wnmxG8h4ZOklzxydpHtrwCzUBloxlRs+cDH2HmLLTKv3BKrhgcNjK3skMlfeTie7YcosXimXFinuxYynC+tFSi01I2jxwJuKaEpDitpFiCTBZyJ3aNjN7WK2EgeGBxo2+Wwsq3ICx9QntWMAV82sqjb0aVH4P/Xo2Oiknttab931Z/Ta7wW/7CcQsodlL/449Ndgena+xcNEtaeND52JRM1thU7dyGUF4B0QukIA0xPLJ6eFtBMOWjeWlzD9RsjTGrBiFn8t0WWvOrQpb/XqkY5ZlzXl+1qyP3ntMN7uS1TNPdWk9HBj8Fvo1uRsfGCDcOslx6zfR0WFEe6o3c4nikwIzM5LEuNPIW2unzABUWg+35sZ5ow+VVR8bARSW3002pkRuyBa9FAHSULjx6LQaD3p5Ydi0fdKBGUK6bmVTPrSaWbwWOzek0XenWBd1J25a2spHt1D7DbeOZfttQqavhUG2xAxI2AsE+jDAYQ1gngAg0KwZnzEyQ5ybFfC9nhorrZ0e5A1ZtGBfXk6W2nObTsOxRAiIyjZ6IceMN6dB3y16szwdhVKA02nAO9beAHjaLPbW1IfNSVouL92MR+qyqsJQCWA7/ vE0bTIuw gEmKdr+4NHBtOxNL8SzRr/k27H+AeXceKK01Y+V/HVYDmotHmNMIqtEMBAC3CunHrPe8uFWpMA2g7lpq56eib8Bkw4J6o8M3nT5E+KFCjA5r5vzEzJew6CUe0BRjesrPfqCygvdTVyYi0wA6YKI7DOEKvTwwwAJjpzUcJBJf2dpYc2JbBWgJPm6hPFJh54O6NeQiPoeuzq0rrUDQR/fiHqTpp9TE3M/uWOMBz0eK4RxK9jtRT2QQJacHgOI92wwtF+uKwIfLQmmdmOytKwH8j+kvLcrj65ncNtKpBFWcma+Fj9/NIKV6KjjLG6z7qezEtsGNr1NxnuTyOw4DQwylWZP7bjWs6NSgQno3kGt5f2hEKvRY9m8z9uZTsOAjUUMDSEPuxOXyKCWBsoFNsEvvzU4UQBtLKOLsbvsjj X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: migrate_misplaced_folio_prepare() may be called on a folio without a VMA, and so it must be made to accept a NULL VMA. Suggested-by: Johannes Weiner Signed-off-by: Gregory Price --- mm/migrate.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/migrate.c b/mm/migrate.c index caadbe393aa2..ea20d9bc4f40 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -2627,7 +2627,7 @@ int migrate_misplaced_folio_prepare(struct folio *folio, * See folio_likely_mapped_shared() on possible imprecision * when we cannot easily detect if a folio is shared. */ - if ((vma->vm_flags & VM_EXEC) && + if (vma && (vma->vm_flags & VM_EXEC) && folio_likely_mapped_shared(folio)) return -EACCES; From patchwork Tue Jan 7 00:03:42 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gregory Price X-Patchwork-Id: 13927909 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF5ADE77198 for ; Tue, 7 Jan 2025 00:03:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 753A76B0083; Mon, 6 Jan 2025 19:03:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 702B86B00A3; Mon, 6 Jan 2025 19:03:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 52DDA6B00C7; Mon, 6 Jan 2025 19:03:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 2EFCE6B0083 for ; Mon, 6 Jan 2025 19:03:58 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id DEFB4A04F0 for ; Tue, 7 Jan 2025 00:03:57 +0000 (UTC) X-FDA: 82978707714.14.3C04ECF Received: from mail-qv1-f42.google.com (mail-qv1-f42.google.com [209.85.219.42]) by imf01.hostedemail.com (Postfix) with ESMTP id 12ACB4001A for ; Tue, 7 Jan 2025 00:03:55 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gourry.net header.s=google header.b=A0aOXGEN; dmarc=none; spf=pass (imf01.hostedemail.com: domain of gourry@gourry.net designates 209.85.219.42 as permitted sender) smtp.mailfrom=gourry@gourry.net ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736208236; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZWCvp/kko+WPP0qjUKzUrB7Xv2NearCGyN4SJ89IJRc=; b=EmdtpT/1PpEKK/zaKAxQuJ8uKF53TIRaQvN0lTD2ZUdisMEg8T2jjT2rT3ecmtv5ndGIPK kGbAuJqX7KjSPp0PjAZSA/Sd2DeWOwVg9w0oUo+YNCXPEcfCDwPvIIcN/9r0jFqGG7vmeG OaBvSubkBnmsFr+F9b8sYvX0qgI4nJA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736208236; a=rsa-sha256; cv=none; b=xk2KdGWNpdRan8w81zraZugG5eIiy7R4ytLl3s7kTHriAtlwyWoM1xrol9Gvp+W6SAzRD6 imyK85gDLeZFaMs4qw2ISB8bSqA8MYKFlrH1oEdzXTmGCszr/5OYMjOUegBwz3n8tX+ou0 Y0s+C35oUThxkNbsl2wkM1Rj4uLmQ5c= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=gourry.net header.s=google header.b=A0aOXGEN; dmarc=none; spf=pass (imf01.hostedemail.com: domain of gourry@gourry.net designates 209.85.219.42 as permitted sender) smtp.mailfrom=gourry@gourry.net Received: by mail-qv1-f42.google.com with SMTP id 6a1803df08f44-6dd43aa1558so108407926d6.0 for ; Mon, 06 Jan 2025 16:03:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1736208235; x=1736813035; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZWCvp/kko+WPP0qjUKzUrB7Xv2NearCGyN4SJ89IJRc=; b=A0aOXGENZutUyLwIyrwKedAx45/02UBAhxeE3UhSkLgrcyhzAkxQ+Ok3Dpl1h3wuzM 4HmhB2M5YRowfcKaMasaAVJ/wgl9tFfJ2Grf7VhzH9y9VayW40eTcz6mRrXyy8/RROBw avC5+XZt/Ojrnef8xan9nnJ0f13AVIkllfa6S5PI4EhHwyv6Sf7pvPt33FBgOdXXEOET AClGikt9nzzufATWD1zrEgrKV6YEmjkPviKWZFXz5aHKe+x+ApwcTAiMQkdt2GmDchK5 vCzhRu3vOMb14UZdGCDW7+RzkxZJhI5lJJY2oqQxaRmmmN2G19Sfsliw/Lvx4vv2oy80 fj+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736208235; x=1736813035; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZWCvp/kko+WPP0qjUKzUrB7Xv2NearCGyN4SJ89IJRc=; b=ujlLqnT6osRZtmf7NKvrl3A2dw9Zsw3z5zJgNbk4MzkT2ZNIl0WuPhL+94RQU2zG1E IgtM9qpoUlk8OZPSdHyYaNSsHWLd1vIBpSF+D1LP9kiDPalLrtPaMxJb2cz40JpEx+CG cmhHzmk4xad0Tye7riW+KdkMu/R90/aZyHrUOQF/puq0QCP1JLAWQUh8vg4uCVGO0PhC LCoV7KwJYHuquEQjZMab/4G0xFm70pdadjS2PWWE7uE09rzgbaFuq0LtJM9kRpsPrgbT 6IlfkybRhmR39DRpGkwyUSpzCkImWYtCLtscqngw9Rrw2M6DxnY+1oe0MX0aEJAdFz8v y1QA== X-Gm-Message-State: AOJu0YxfuTzeMidJyJfUMs+93cj7+p/4MN/hjvLILS7H4ZOYfVwJJ6tl BF6jTw0CyhVl8YumpNB9QY7wX4kPJ31m9tCiJciKG39HDbUNxYvL6KenWDHr3h1AYGg/dost2Y8 k X-Gm-Gg: ASbGncvii9/WhREDaLcxCRf9143vLTvdux1qlNZ7XEIY4bu3Qt2DZy8bNouYJiA/IKY sCV+igJekF5fr1dpoObCfej03NjlhvsmZB+2ttiF/ImFqhPx0TvSafhn+g3i7WHo/LvZ7Gji37M kblVVXO6ejxNcc6s/FR5HBsXeYkrTT5YnYG2oSB2kkjas/mPnf4jqp1sVqdr3HOlF06tbs0pAQY aB8JqlQ6bRZx5CpmZiJ9pNmZb/QU6NwdXqIA198RKr8Jnqs16C9NoIf3UzrQYlPtDbBTIHed+0S VQ4deapWQFBDm8kQ+PLWvhOyhHCvdovVXXsJY3CgeadA X-Google-Smtp-Source: AGHT+IHrk99R5d/+390QkKolkyUXQWqv1/fjT8A0hpydPLDQYb+viHAXemLnF0B4ZPb3yXMDp4MjqQ== X-Received: by 2002:a05:6214:c4d:b0:6d4:3c10:5065 with SMTP id 6a1803df08f44-6dd233a0527mr872530576d6.32.1736208234983; Mon, 06 Jan 2025 16:03:54 -0800 (PST) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6dd18137218sm174104476d6.57.2025.01.06.16.03.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Jan 2025 16:03:54 -0800 (PST) From: Gregory Price To: linux-mm@kvack.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, nehagholkar@meta.com, abhishekd@meta.com, david@redhat.com, nphamcs@gmail.com, gourry@gourry.net, akpm@linux-foundation.org, hannes@cmpxchg.org, kbusch@meta.com, ying.huang@linux.alibaba.com, feng.tang@intel.com, donettom@linux.ibm.com Subject: [PATCH v3 2/6] memory: move conditionally defined enums use inside ifdef tags Date: Mon, 6 Jan 2025 19:03:42 -0500 Message-ID: <20250107000346.1338481-3-gourry@gourry.net> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250107000346.1338481-1-gourry@gourry.net> References: <20250107000346.1338481-1-gourry@gourry.net> MIME-Version: 1.0 X-Stat-Signature: yk4kuirwtnd51izw1yb9b3ws8pidw3yp X-Rspamd-Queue-Id: 12ACB4001A X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1736208235-685367 X-HE-Meta: U2FsdGVkX18baMuYoEYrr1/HLbg/8J/DezcaNE4SG2/f1RZqM1qpqzPibe7EPfbDD3lgn9seKh6faa13ty4j91c/eNb5QvTLdZtCs9ttOkus+cEwU/ITwRfjixPHKzL0xC8CVBmoDeJT3HORqBFEsVid2MJMOKNjMYxxwUkYTC9X/bDZzN09yBCgLEbJm6iPefRwknioLxykCbtCoXfrrHxhvAwQAvGr63SuoMmFx27AkBNw1ivoavlatMhK6fweTdTVF6nnFbkY8fb348mJwE3ZnVWNYChD9pYRceZlODuhlE5xZ/JigT8O49XVXz4aTeflRXcv8nJtyw6Q+fwJnE9YmbA3p8qbRFctnddkjKILPh1YupksuhsaMnhcbmlsEeOmko7d4WH7CPbD+EbZmBlC6HunABjkVKJU6cu58dmYae2d/xy08M+o6toIbyi9BauCLTxZ1bm/T8ZqQcpgpljEOBTtWgbS8uvPGK3taoM4PbVlZGhFZ6ijHRVJJTiGm3PCws9xxS3LwAYmYEzUK+O9MhsffAPTZfAM4qsj1VO+v1hMNEtluZMX8vTi6MMjMCNA7vZP8X7SWK3YCBPgrsdd0OYV0ViG7z4jFVi+e1N4ro0CFeH5XAO6ccSieBLHl7PMaYmWv7VqIsxYdBKWQfG1GzSOioFilqV9gcFZOyQQQ/F/BNjqA+z+3wCYk5CzuLc+xH5uxcRmKc7RrjtDMsysSayQ3BEcAVj/noEDlW+lMhmuJwXVUkEvcOA2fnYJdC8bSuxMucgDp82j3X1w9wg9RbkLLWOq5vF9m6FsUnxtvMfJMvAEOs2rt8ILEtHYOy9ggSzxwb1XkK6jCSoKwfqcTUhcON8ufOx5cOZ9Fj4jvP7ViicdLvAJyY2AC5Ye/iU9LJ+aT3i2x/5ngNuJ1z56is+T/hUK2jWJ2q8liJkSnxcVs1jqpRWDinZ8jFM45KO7+K5B4SEKwfNAmjv QAFHaOZD rzS4IX+JHJTx8PQ3sjX5+STEbc3rLE10QZ4KzNsq7ROkdBYMsLv/hLp7kT+BhoTLoy1XaDRLMG6mr0R5KsF3WXYjFwXsONryogX0J/SwyTra857CteWFCv/S0HQOX/vsCwcYSRSx5ymkHWKUR7aeUjntk/jOb3tRhBZeSiwhfSIpVP7VSOlw04advn31NBu/lUiipeDPZPMkSlzAaXhqndsvRvt7IcGe83+jo5BaO40wZ4AqfR8uEPngwKsfvVDCf3/BwanbwVbvsupoDDDV20hew3M2vPwMN2d0mFR1PprdskZB6/RM8t6yiSA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000622, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: NUMA_HINT_FAULTS and NUMA_HINT_FAULTS_LOCAL are only defined if CONFIG_NUMA_BALANCING is defined, but are used outside the tags in numa_migrate_check(). Fix this. TNF_SHARED is only used if CONFIG_NUMA_BALANCING is enabled, so moving this line inside the ifdef is also safe - despite use of TNF_* elsewhere in the function. TNF_* are not conditionally defined. Signed-off-by: Gregory Price --- mm/memory.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 9cc93c2f79f3..8d254e97840d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5571,14 +5571,14 @@ int numa_migrate_check(struct folio *folio, struct vm_fault *vmf, /* Record the current PID acceesing VMA */ vma_set_access_pid_bit(vma); - count_vm_numa_event(NUMA_HINT_FAULTS); #ifdef CONFIG_NUMA_BALANCING + count_vm_numa_event(NUMA_HINT_FAULTS); count_memcg_folio_events(folio, NUMA_HINT_FAULTS, 1); -#endif if (folio_nid(folio) == numa_node_id()) { count_vm_numa_event(NUMA_HINT_FAULTS_LOCAL); *flags |= TNF_FAULT_LOCAL; } +#endif return mpol_misplaced(folio, vmf, addr); } From patchwork Tue Jan 7 00:03:43 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gregory Price X-Patchwork-Id: 13927910 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65F26E77198 for ; Tue, 7 Jan 2025 00:04:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D6CE16B00C8; Mon, 6 Jan 2025 19:04:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D1F836B00CA; Mon, 6 Jan 2025 19:04:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AAC416B00CB; Mon, 6 Jan 2025 19:04:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 497086B00C8 for ; Mon, 6 Jan 2025 19:04:00 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 18C901A0531 for ; Tue, 7 Jan 2025 00:04:00 +0000 (UTC) X-FDA: 82978707840.07.E85808A Received: from mail-qk1-f176.google.com (mail-qk1-f176.google.com [209.85.222.176]) by imf18.hostedemail.com (Postfix) with ESMTP id 48BB21C0015 for ; Tue, 7 Jan 2025 00:03:58 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=gourry.net header.s=google header.b=Mi7AFiPm; spf=pass (imf18.hostedemail.com: domain of gourry@gourry.net designates 209.85.222.176 as permitted sender) smtp.mailfrom=gourry@gourry.net; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736208238; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EhQxXMUURDFkAZ5scirPLfGBPx39x7Cq7gDAj234rcY=; b=bz/IGJY2YnU8aXdxTQR7kRHFZd5LaWlOlHh8nCZ0FjOAl9X+r1jGYZSzVHQxvlCWOAu++V LD59YSaG+c7X977bCUJlrRnxoyAE1U89jtWyOfdIUY6Lvk63WOW6aOakt24o53S27rYCMZ 5p2FD9M4dj8zTfM2JxEp+rkkm+IUSmg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736208238; a=rsa-sha256; cv=none; b=mtBzZkQesbx34Yf2+jMHlPsmrTdivgSkYyAveQ3tdM2INmlQA9AqM4lD91Opslj2f/NPE+ 48xXCQLLj9pbj4apUobYgFTUMD5V4jl0J6MKV5ZnFsI7d5c6RLcKisNC3BvuB5wVbfBUp2 dd5/HRlos6ZZRQ4uD+fEYeZiHhalKJM= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=gourry.net header.s=google header.b=Mi7AFiPm; spf=pass (imf18.hostedemail.com: domain of gourry@gourry.net designates 209.85.222.176 as permitted sender) smtp.mailfrom=gourry@gourry.net; dmarc=none Received: by mail-qk1-f176.google.com with SMTP id af79cd13be357-7b6c3629816so769967785a.1 for ; Mon, 06 Jan 2025 16:03:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1736208237; x=1736813037; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=EhQxXMUURDFkAZ5scirPLfGBPx39x7Cq7gDAj234rcY=; b=Mi7AFiPm04TJlOGt+VFge9IemM7hAz3XskZhtBJ2BqZQoKjs02tvACrGfM/M6BA2mb H62JCPP6klTgy2Po5kvVDiYRpD4EYkTrGOM9uUSKPKpx91j1YN3YZCR5B/5Pk5VP+gYQ IU95ZkNh8rPDckZnvStp5n8SNrf8VCaWJrbnDncGD9cRFF5BZ+NIrCfLGsEb9P0UJ4Hz ZkWtjEqLIQpW4sKT7LnjkyHz+WQv/pAdMWhCxD+LDplLsXdHYrVjSvVX2zmfH9h8BSzE xp1x2ozeFjiYGpfvp0yWNXkRncgkq5c9MNvbAYAIy4fct3P5+3Q/QPmFddz56iM3C7nv wzDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736208237; x=1736813037; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EhQxXMUURDFkAZ5scirPLfGBPx39x7Cq7gDAj234rcY=; b=R2xFb1kiFzKO5y+T5oQZRdU75k8gyyghAW7nghyFhtA2PrqxVdP7Xh8qXr9mD8GcYE 1A6b3VCkTr2xN45tVyQKq1m/jcGnjJK01GBlEjJY3S3YbRF5YsLUwWYwtmpnkmhlkude UGO5Mgy5oBxU4xzgNU0aYTbsQWKr7sJf6aJ62ipi+9FEw2+urNFjc9JxNsNAd0Q4KoV5 QEl0oJddA/uLkG6vTVErx9Q/qKXcCRG7Ixh4xy04Ziv/gKSBOgj/ckycn1ZVCSEgwwkt peNhBxcNImcKE8suIhaD6DdghJ4FneuXRoS6mybnwcs4JdyPqGQCLG11Ms5wT4D6Yoo+ oSug== X-Gm-Message-State: AOJu0YwTVnjpyXVBHADUQKebUE7wHYu1GvHIwimDEFOZefdfo9gafm9q /IbdLobKomOBZALWK3uMQsu04eeLnnVwWabvzenh8WHHS+XuvqdEAONRppEMEx0iz+GMcBHjmaB J X-Gm-Gg: ASbGnct+HdOBbyeeHdKrDIoHJ0A07VOY+O2cKxg9FtknqvbCSp9eFSXYjktZFIAbtoM ALB4YcfhTzBmFnPSMzcHOAKVgNFRfToRdudVz+ySFWWdWOjY6ummuaAtFQ2tynlJ7GoyhgH5KGy dIB4ZBhKQQOgFHlCpkwNsyN0Wy2JFxkR3wF/6PtUTKwR/JjfJjWbEGatyEl9tXjTBCOc0fqP1In nOk3KKh+xy1edNVyih1tn7y2CYzaER8feqDOMc/oI1Q7GTEsLSDMklIWWBSJvXUENAPp+4CVxOd MsU4LfTXBYj21DJKsNnjSylES5GPXjlv2gOjGUoB+qJ7 X-Google-Smtp-Source: AGHT+IGxVrqjx+Di9dmI2xKMOrXvW3nylvanO8oTfhpUycZZo5Se/q08YTn9jxqPITzbOyS3wtn8GQ== X-Received: by 2002:ad4:4eac:0:b0:6da:dc79:a3c9 with SMTP id 6a1803df08f44-6dd23306794mr895686556d6.9.1736208237050; Mon, 06 Jan 2025 16:03:57 -0800 (PST) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6dd18137218sm174104476d6.57.2025.01.06.16.03.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Jan 2025 16:03:56 -0800 (PST) From: Gregory Price To: linux-mm@kvack.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, nehagholkar@meta.com, abhishekd@meta.com, david@redhat.com, nphamcs@gmail.com, gourry@gourry.net, akpm@linux-foundation.org, hannes@cmpxchg.org, kbusch@meta.com, ying.huang@linux.alibaba.com, feng.tang@intel.com, donettom@linux.ibm.com Subject: [PATCH v3 3/6] memory: allow non-fault migration in numa_migrate_check path Date: Mon, 6 Jan 2025 19:03:43 -0500 Message-ID: <20250107000346.1338481-4-gourry@gourry.net> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250107000346.1338481-1-gourry@gourry.net> References: <20250107000346.1338481-1-gourry@gourry.net> MIME-Version: 1.0 X-Rspamd-Queue-Id: 48BB21C0015 X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: 6hinjaywwfybmnmqsdrjbm7rdt4mmjnu X-HE-Tag: 1736208238-454439 X-HE-Meta: U2FsdGVkX18a4lhULjlR8bcB5az2XK0hzuIGtL9MsLC3UmLw6n7sbmJwc9jxazILQb7qiwXoZKLqY82zmlgVe1HsdkdYTLZql2hbEHkBSo9M9UE8i48aw0V3Jz0562PNRGInCjU/7CAI+yP7eVgHgdG1n2E6YhO+TpDc0ikHaG+NjU2le+1WH/dwdJy4/eq8VoyqIcNmsp1tpDFwj9uNGoziqXQarRWQayrg6rkNS0Lr2kPqywMk9qp77o+a0Lmm2svnpApPvh4fFRV8mnGfSdrakR/Qx2A3lrpogfj8HfeK/FXUOD8M2PvE7FY2Eyju1E4nOASppwIL8OO50Gmv6Tb37cEM3iFWDKiJzhWTlZ6j0CbFm7lcFBwJGrIIUhjeC8Igd6OSF0pvtrYON+BLijQh5nR3yARjkCbKCF7XMG+nwmqfUVHKuaV1sH9qxIc4SJpPLkgqx0+k+BMVEZ2GT/7Mgoy82RxIH+GwwB6DjIvaDFAc57/oU9IR+fKooW7NYzWG+dGQansbHPI4YPYLeAdpc8zLZ9rPm7pgUPdRHa9nE3LTH5dn5YZAjwfnCJ6YKLbr+IvNCCQY6hs73OAMeMbMWOkps2rnO9n0hmQCKCUtcynUQX+VwCOQMY9hkbmOixL/UERwGsVRjOjws13acdUfq4nrF0jcSK0fZeYaklrle7kvKyTBfFlcBHKLN8Zh1zBGoMSyDiT4yNLkWywBf3/pFsfVImz18hCwd+APeYj423o8zMOWc90nxkOOhjrGc2mYoIIrSXSXAcVTX7dELziRiwcS4yFb3ICN6dEbvwRsW+7ftG1TQNUQKvu/y+ePR9yN6GALI2yParpPdZNtG5718SlHEDnxr9zEFZve0ieaEQayToeoGYjiLiOu05QUJVXOkuDCVGiaPwQLxOFoEQojvyw3ap9oUs2KvJTUMmINNxdfoYmmGUJAx7un8JJLHM3DCd67CUw/CsgrAgO hXt2aqJZ Mrom3iYpIbBCJN0bXU2xT9H53/ra2UA7eKnyKtSIW0Zj0NcpcsNyg+biGwdKmZXH3wzhrlvH7AZrR4KxzU7Nen9V1BmcXAGc3XcBWIskK4B+iSDQa+JOs5yaKYhab+ciMbB+8mKSdh/6oh12OsLLedr11ERQbEsjAu0NWnQW36OiekqRlYmkPHsl1z9R4XJLywNlrcmGKia4Y182w4PswjBefNt1ICEHW8l0U8OfGLeyLd8BtqbsyvTSP1raJ0pyWXEI2pg0n+Rdz8uXE5UKA3Bru0Uag1ZaYDXsa+jG+wn1ZkMAJ5cue7s8zIgHRTDCdswQVzk4zpP2HfExApq2s2WxcURG+w6HUBmLK X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: numa_migrate_check and mpol_misplaced presume callers are in the fault path with accessed to a VMA. To enable migrations from page cache, re-using the same logic to handle migration prep is preferable. Mildly refactor numa_migrate_check and mpol_misplaced so that they may be called with (vmf = NULL) from non-faulting paths. Signed-off-by: Gregory Price --- mm/memory.c | 24 ++++++++++++++---------- mm/mempolicy.c | 25 +++++++++++++++++-------- 2 files changed, 31 insertions(+), 18 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 8d254e97840d..24acac94399c 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5540,7 +5540,20 @@ int numa_migrate_check(struct folio *folio, struct vm_fault *vmf, unsigned long addr, int *flags, bool writable, int *last_cpupid) { - struct vm_area_struct *vma = vmf->vma; + if (vmf) { + struct vm_area_struct *vma = vmf->vma; + const vm_flags_t vmflags = vma->vm_flags; + + /* + * Flag if the folio is shared between multiple address spaces. + * This used later when determining whether to group tasks. + */ + if (folio_likely_mapped_shared(folio)) + *flags |= vmflags & VM_SHARED ? TNF_SHARED : 0; + + /* Record the current PID acceesing VMA */ + vma_set_access_pid_bit(vma); + } /* * Avoid grouping on RO pages in general. RO pages shouldn't hurt as @@ -5553,12 +5566,6 @@ int numa_migrate_check(struct folio *folio, struct vm_fault *vmf, if (!writable) *flags |= TNF_NO_GROUP; - /* - * Flag if the folio is shared between multiple address spaces. This - * is later used when determining whether to group tasks together - */ - if (folio_likely_mapped_shared(folio) && (vma->vm_flags & VM_SHARED)) - *flags |= TNF_SHARED; /* * For memory tiering mode, cpupid of slow memory page is used * to record page access time. So use default value. @@ -5568,9 +5575,6 @@ int numa_migrate_check(struct folio *folio, struct vm_fault *vmf, else *last_cpupid = folio_last_cpupid(folio); - /* Record the current PID acceesing VMA */ - vma_set_access_pid_bit(vma); - #ifdef CONFIG_NUMA_BALANCING count_vm_numa_event(NUMA_HINT_FAULTS); count_memcg_folio_events(folio, NUMA_HINT_FAULTS, 1); diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 305aa3012173..9a7804f65782 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -2747,12 +2747,16 @@ static void sp_free(struct sp_node *n) * mpol_misplaced - check whether current folio node is valid in policy * * @folio: folio to be checked - * @vmf: structure describing the fault + * @vmf: structure describing the fault (NULL if called outside fault path) * @addr: virtual address in @vma for shared policy lookup and interleave policy + * Ignored if vmf is NULL. * * Lookup current policy node id for vma,addr and "compare to" folio's - * node id. Policy determination "mimics" alloc_page_vma(). - * Called from fault path where we know the vma and faulting address. + * node id - or task's policy node id if vmf is NULL. Policy determination + * "mimics" alloc_page_vma(). + * + * vmf must be non-NULL if called from fault path where we know the vma and + * faulting address. The PTL must be held by caller if vmf is not NULL. * * Return: NUMA_NO_NODE if the page is in a node that is valid for this * policy, or a suitable node ID to allocate a replacement folio from. @@ -2764,7 +2768,6 @@ int mpol_misplaced(struct folio *folio, struct vm_fault *vmf, pgoff_t ilx; struct zoneref *z; int curnid = folio_nid(folio); - struct vm_area_struct *vma = vmf->vma; int thiscpu = raw_smp_processor_id(); int thisnid = numa_node_id(); int polnid = NUMA_NO_NODE; @@ -2774,18 +2777,24 @@ int mpol_misplaced(struct folio *folio, struct vm_fault *vmf, * Make sure ptl is held so that we don't preempt and we * have a stable smp processor id */ - lockdep_assert_held(vmf->ptl); - pol = get_vma_policy(vma, addr, folio_order(folio), &ilx); + if (vmf) { + lockdep_assert_held(vmf->ptl); + pol = get_vma_policy(vmf->vma, addr, folio_order(folio), &ilx); + } else { + pol = get_task_policy(current); + } if (!(pol->flags & MPOL_F_MOF)) goto out; switch (pol->mode) { case MPOL_INTERLEAVE: - polnid = interleave_nid(pol, ilx); + polnid = vmf ? interleave_nid(pol, ilx) : + interleave_nodes(pol); break; case MPOL_WEIGHTED_INTERLEAVE: - polnid = weighted_interleave_nid(pol, ilx); + polnid = vmf ? weighted_interleave_nid(pol, ilx) : + weighted_interleave_nodes(pol); break; case MPOL_PREFERRED: From patchwork Tue Jan 7 00:03:44 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gregory Price X-Patchwork-Id: 13927911 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68656E77188 for ; Tue, 7 Jan 2025 00:04:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 421CF6B00CA; Mon, 6 Jan 2025 19:04:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3834D6B00CB; Mon, 6 Jan 2025 19:04:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 09A8D6B00CC; Mon, 6 Jan 2025 19:04:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D976E6B00CA for ; Mon, 6 Jan 2025 19:04:01 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 91C0E14053A for ; Tue, 7 Jan 2025 00:04:01 +0000 (UTC) X-FDA: 82978707882.05.F902484 Received: from mail-qv1-f47.google.com (mail-qv1-f47.google.com [209.85.219.47]) by imf27.hostedemail.com (Postfix) with ESMTP id C6C234000B for ; Tue, 7 Jan 2025 00:03:59 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=gourry.net header.s=google header.b=ss92SRMJ; dmarc=none; spf=pass (imf27.hostedemail.com: domain of gourry@gourry.net designates 209.85.219.47 as permitted sender) smtp.mailfrom=gourry@gourry.net ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736208239; a=rsa-sha256; cv=none; b=6m/a/pz3yOxBVOJ4OZ1xnA8eHRjB9XbLFpQjOutdgKEIT18IbPjj8zqoToAHI5V+0gMppR KLFxcwmjPg39DHlQkwfU+OwDgMzHU2Pdtyn9P73LwknDd4wtQ+cQA8JUBSPQMgsV5arl1I R0P8duZCTbVvnvgJYSTetZFuuVrJgbQ= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=gourry.net header.s=google header.b=ss92SRMJ; dmarc=none; spf=pass (imf27.hostedemail.com: domain of gourry@gourry.net designates 209.85.219.47 as permitted sender) smtp.mailfrom=gourry@gourry.net ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736208239; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EsdX3K0L6RUajk4aeowkzYla6EH2ycdI9VpsUm/ueEE=; b=vm7UQyk7PUiQpW4PTrM6xYwioCtacmO3SMm69+7T+RmdyKGkm4rgCv1mxSMkY146kVFLW1 +Cg5frzc/VwFf/QF+IeXK38dwVpZjaQ8shpJaDfoIKSZzr6zhvqVfwkWMlkzCmjfJ926Os PjETAjP5oZ/75kdvHY9MwJSN/zFvzmg= Received: by mail-qv1-f47.google.com with SMTP id 6a1803df08f44-6d8f99cb0d9so121178896d6.0 for ; Mon, 06 Jan 2025 16:03:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1736208239; x=1736813039; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=EsdX3K0L6RUajk4aeowkzYla6EH2ycdI9VpsUm/ueEE=; b=ss92SRMJt+mUm/NGq17BpbixU08ONkGOp/Nwv/AZctjz9UkPwzgaEhCxf4JGele3x1 iVIfgnEAZmCNjyLneEtbWuovwmjxobKXA6j5yLy/xyrCrF5FiZDnoD5rlvw2GWHXuzoW 4h72oCVBVJv/R/yUiIKV81oEBDP45f+3+X/hskrqK79GQJxs52wef/2pgzql+gbCvfAZ CP8sJm7+cacZewAxtMQJ9zbAtHmIDJz8gGFJqW5Q+nzQMqNpVslWVEnZPqz7NTK6gWVZ iISjOh5qyBG8oQ+Y6RglJ05vgK6eCAoqor+JTJn1FGixkavd8bMQ1D6NY4qik+PaXgnS KPaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736208239; x=1736813039; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EsdX3K0L6RUajk4aeowkzYla6EH2ycdI9VpsUm/ueEE=; b=ujujQ9xWXhuiNocQE9xK8cpgQl+eGvFvATrANdgUByniWowaCflEAOSpAUjmoasbyI iDMbbfIwdZY1N+LhGo+1UsJx3pUyUeswdxyN34/QkGpsUMQMt/hfNyuESape4e9IpTNC IZ5Yn+dlILi5Q2prkEQ7YqGrRMBuzNVvYbKRkTqTb+DXyj0YWOoIxRes9Ct/U8tzLTxi YDgtw3ckYY0aGvUBGzz86rGbuG/08TpaI7YzIahWc8+YOsNZ7eRMTPzocwsO/QkDI986 Tbgf5zmy7BMV+schnF9lpm/YFovVbYlotlv53JIJ+hSO9MhFebmPZGPaMMUyq1QUfS9C rKHg== X-Gm-Message-State: AOJu0YwzAi7iiGJuU1AyXMvl4fZK1CZpJWh1GZOScTFN/8wecKQHU1X9 HAuh1TOlHb9TZBlrmSa/YDcuzUGpkRinI7puycrgsYiNyFQ0IGy03liY8NVjthbuq46Xzy1GGpM u X-Gm-Gg: ASbGncucBhCSc3e5cOlIFm5ZxCksQWB93MhLRE29UstEs6lS32sZh7eY6vYO6J5QXB+ uPLpMNM54GTbrdKjxpq2wHfrjprWg4NKo6ThN/PdLi+BuSl0Pr87TZErPOAwI2V9pR7mREuGwfy 37anfc3I8R62tiBcU8hHGYO9jYN5L9YzbMxavdFDuke0ByuukdBOvoHiKzPbo+N7CNXYC9aAg0F ChGnWmHp2GutiTMu5424jlsqhXo6QyhepNoZhOn5Jczl00HThUcihQI6k4IIg9kpFBz+8lOhb9a gLZMMqAlcHNfj7eYRMw3M47sZfcknbHnZUBiwSNYRclM X-Google-Smtp-Source: AGHT+IFjcy08ueMQthdN8WT7xWa+MvL9mh5Nnz9fWO29b2RIBwq+TvZ+nNviuLcDL7R+9H1KK37ATw== X-Received: by 2002:a05:6214:495:b0:6d4:25c4:e77d with SMTP id 6a1803df08f44-6dd2339fe2amr1069592226d6.34.1736208238692; Mon, 06 Jan 2025 16:03:58 -0800 (PST) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6dd18137218sm174104476d6.57.2025.01.06.16.03.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Jan 2025 16:03:58 -0800 (PST) From: Gregory Price To: linux-mm@kvack.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, nehagholkar@meta.com, abhishekd@meta.com, david@redhat.com, nphamcs@gmail.com, gourry@gourry.net, akpm@linux-foundation.org, hannes@cmpxchg.org, kbusch@meta.com, ying.huang@linux.alibaba.com, feng.tang@intel.com, donettom@linux.ibm.com Subject: [PATCH v3 4/6] vmstat: add page-cache numa hints Date: Mon, 6 Jan 2025 19:03:44 -0500 Message-ID: <20250107000346.1338481-5-gourry@gourry.net> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250107000346.1338481-1-gourry@gourry.net> References: <20250107000346.1338481-1-gourry@gourry.net> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: C6C234000B X-Stat-Signature: muqn38c3g8hqoe5ty8r45hwre74cz1i7 X-HE-Tag: 1736208239-27876 X-HE-Meta: U2FsdGVkX1+eo368+lUdmAubeLwcVod491G9Fjv2JUb11/SCAtNt6eYYq8980RtOoua2c6x7tO5Pr/wGqkIh8iyE+cbEi4IVTawC7m472I9wql2SuvqSr1+hDh9ESTLXxxKJHrNfFJQ4FBRjmOYdR45YO1GbLJXGbns4w63kQA2orSLiSwac3e6GVkdvnJyVBORqqnNBTFXFww7M6jsDh0qezJQ5maITmoCDoScg0i0NmNLHpQZWsLz0KzoDSNPC6dCqbnidJcTpulaVOY/KefEcFlwwLL+BXjC/5qeTW7uj/ojPs83ZOg9VWrGg/wQd6/mcvnd1jkIPbRI3+Te3wML3e8yQXEftRcw1SMi86rHxRIYD+eFIc8dClverFNHjr/ICbban+3SRDKaOy5Bk0LrkwA4NGGIJ3FKDyzTzOlEL5aIA7uHetA56dmtPTJpEFq3U2WCOVKiV+9KjC3S41UmcQHbdo07NFxbwjsIYp123nIYVHkfKKhyXcz5ioaMYu+OXnNkCsLXDAldqdbfe3+hrAtUSjMt8G5liuUG1XB8yTEAiwy3bN+fDbCTU5MbshyTxxemYHJa7zgMfZaMpQXwQwSqcKv1pobfEd6fcBLfG+6s0dviH4pXC6kHa7AkG1q4PqvR6IsE8wJukD5vWVTLwML7bnQJrdEz12anW3TQnkgvS3uudBWgBnLEnDg+WBb2+14XU8vzCj8DoAonATAmMxcaQXJ9SCt5T5C35UxzpuJM9zPrOIfd7Gn2st3CsQMUGBV8yCAoSE4SlMTfDzrFTfSVllZAzfhzZYBWaGH28koaJ9/K38xqaan4zmaGkNpWo/zHfZKTEYuIqeHmVGY8FM05wptds0b5VC6gl4qIjM/lM51E/L4y8/AJGrrVp14UQGfuH6SiNeslIXsJpjyCN99KTKXBGNGB0ljW5dfyzVIw9fxW8sxhENdo20pcK2JREo1xXJpLOe3Ysos0 1wEAEi+i RgX+Ggw+LBEiuEoytSPci1OAq4Z7gey7yA7/4kDv695zoLwZtK67reVf14rMG9WxNRTAGZ5bx5Q4xzeGWN0No+Oki+LWY7W2SPg52EJaNyDxu57dAguU2fLhQPzqrE5Mz/YlDDubst5oeFx6vibkWTpIP19UstYsnkmJjXZhafllqAACHROUgMkXD3qvUtrJBpq9Yr3TCeal18tyaL0WvsXXzjiLSRFD66z80zdE2MVoCAwIJbyTl+pPtlsLPBVw7fNRhUCYlA7oXrg7oSmRhKWWWAFACvFcP+2SzuRVSDE9Dz6ps3J3x3bqMjA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Count non-page-fault events as page-cache numa hints instead of fault hints in vmstat. Add a define to select the hint type to keep the code clean. Signed-off-by: Gregory Price --- include/linux/vm_event_item.h | 8 ++++++++ mm/memcontrol.c | 1 + mm/memory.c | 6 +++--- mm/vmstat.c | 2 ++ 4 files changed, 14 insertions(+), 3 deletions(-) diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index f70d0958095c..c5abb0f7cca7 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -63,6 +63,8 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, NUMA_HUGE_PTE_UPDATES, NUMA_HINT_FAULTS, NUMA_HINT_FAULTS_LOCAL, + NUMA_HINT_PAGE_CACHE, + NUMA_HINT_PAGE_CACHE_LOCAL, NUMA_PAGE_MIGRATE, #endif #ifdef CONFIG_MIGRATION @@ -185,6 +187,12 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, NR_VM_EVENT_ITEMS }; +#ifdef CONFIG_NUMA_BALANCING +#define NUMA_HINT_TYPE(vmf) (vmf ? NUMA_HINT_FAULTS : NUMA_HINT_PAGE_CACHE) +#define NUMA_HINT_TYPE_LOCAL(vmf) (vmf ? NUMA_HINT_FAULTS_LOCAL : \ + NUMA_HINT_PAGE_CACHE_LOCAL) +#endif + #ifndef CONFIG_TRANSPARENT_HUGEPAGE #define THP_FILE_ALLOC ({ BUILD_BUG(); 0; }) #define THP_FILE_FALLBACK ({ BUILD_BUG(); 0; }) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 46f8b372d212..865c9c64068e 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -460,6 +460,7 @@ static const unsigned int memcg_vm_event_stat[] = { NUMA_PAGE_MIGRATE, NUMA_PTE_UPDATES, NUMA_HINT_FAULTS, + NUMA_HINT_PAGE_CACHE, #endif }; diff --git a/mm/memory.c b/mm/memory.c index 24acac94399c..3f63cfd24296 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5576,10 +5576,10 @@ int numa_migrate_check(struct folio *folio, struct vm_fault *vmf, *last_cpupid = folio_last_cpupid(folio); #ifdef CONFIG_NUMA_BALANCING - count_vm_numa_event(NUMA_HINT_FAULTS); - count_memcg_folio_events(folio, NUMA_HINT_FAULTS, 1); + count_vm_numa_event(NUMA_HINT_TYPE(vmf)); + count_memcg_folio_events(folio, NUMA_HINT_TYPE(vmf), 1); if (folio_nid(folio) == numa_node_id()) { - count_vm_numa_event(NUMA_HINT_FAULTS_LOCAL); + count_vm_numa_event(NUMA_HINT_TYPE_LOCAL(vmf)); *flags |= TNF_FAULT_LOCAL; } #endif diff --git a/mm/vmstat.c b/mm/vmstat.c index 0889b75cef14..1b74d1faf089 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1338,6 +1338,8 @@ const char * const vmstat_text[] = { "numa_huge_pte_updates", "numa_hint_faults", "numa_hint_faults_local", + "numa_hint_page_cache", + "numa_hint_page_cache_local", "numa_pages_migrated", #endif #ifdef CONFIG_MIGRATION From patchwork Tue Jan 7 00:03:45 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gregory Price X-Patchwork-Id: 13927912 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85260E77188 for ; Tue, 7 Jan 2025 00:04:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D233D6B00CB; Mon, 6 Jan 2025 19:04:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C5A536B00CC; Mon, 6 Jan 2025 19:04:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AAC9F6B00CD; Mon, 6 Jan 2025 19:04:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 888E26B00CB for ; Mon, 6 Jan 2025 19:04:03 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 46A9F43802 for ; Tue, 7 Jan 2025 00:04:03 +0000 (UTC) X-FDA: 82978707966.20.B2F3C51 Received: from mail-qk1-f170.google.com (mail-qk1-f170.google.com [209.85.222.170]) by imf01.hostedemail.com (Postfix) with ESMTP id 6B0DD40016 for ; Tue, 7 Jan 2025 00:04:01 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gourry.net header.s=google header.b=uA7u9b2V; spf=pass (imf01.hostedemail.com: domain of gourry@gourry.net designates 209.85.222.170 as permitted sender) smtp.mailfrom=gourry@gourry.net; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736208241; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Tds7HvhDScwv0+DAQP1llxCN/3UO28LajkOSfhGyREo=; b=IrPP60dHkIzvWB8QB/99iNwA+2ewWm3pm90ZBCwqOt+2oEeSyYMXK1duYgG1k65yNQLZtt 15FuPcMUfSYV1j6pfxtieb4/Y1eyEb7pCiZIboRbxBhDbZ660S/+6H8gP3teYKa+9m+2o2 SJRwlKFcBqs0iQqC0quTYSAF/nSk+RA= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=gourry.net header.s=google header.b=uA7u9b2V; spf=pass (imf01.hostedemail.com: domain of gourry@gourry.net designates 209.85.222.170 as permitted sender) smtp.mailfrom=gourry@gourry.net; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736208241; a=rsa-sha256; cv=none; b=A2Mp6VWNJ2Ahbzr5An7q1fb6GV8bEnWHfB6G1UMC0FKkUK3zcAdgC4coPUCuB+8xJtnr5e H+jGw1oBJxUgdak3gZDoirNccHdZ1vBK7A4ndqW2jR+dKNAPHNtPfZQCGABxg2Abs+FSNf itYKKf6t6gjsPmnw0EekABPqrlCno1c= Received: by mail-qk1-f170.google.com with SMTP id af79cd13be357-7b6f0afda3fso1436093385a.2 for ; Mon, 06 Jan 2025 16:04:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1736208240; x=1736813040; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Tds7HvhDScwv0+DAQP1llxCN/3UO28LajkOSfhGyREo=; b=uA7u9b2VpII6wM43sn2662gy/VeERXWqgMzgZPyQcp/u3sm9s9pZJ5nc6bcR83HkfC xcwthly5HdcJjQVoYWQB1cjv3YNZwqD2NKj/FIrdKS0y6zRswyMNr4d1T5akS6OIIVRR Peg0N3qyeT1EGPE5R9jTLu4rgpXNWfBTqDmiAZ5ISPkr45zX5J7XdPEnGLb3y6dgsSmz Hb4xdK1SshO2xqPZgzhBIr3QZQUpJoE5sLVp0YnuWHDBv07tGNpY7wolx6LfgnVJit9v TYY2RJavIdS+8uZQDKIEIvefNwGFy4KlJe5eUVpYO3KPhQKbHHmgl17XMb/haYQies+l vapg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736208240; x=1736813040; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Tds7HvhDScwv0+DAQP1llxCN/3UO28LajkOSfhGyREo=; b=f7jW3j3KhvwaMMBlhBED77lfJZdDiypROAxKXxk9cgvOKOvhE0IVmILEJKhQyqkin7 cbcDz5j7RbrSbRqi3Lm6meekFmrcV9OFR5OGKzS97w3AXvokyTxiFLmrbQvcVHTpSHeW Gd9dRW88DydT5BkT3UJ7lLkSgJ6xM+5ycIwiEwkeGGfYYvzxLGpFyRZ+NsyKKW0v7cXG RJrx3o6LKq8jg5kdl9gjP6etthwFfu0LT2WYVZJagLzBJ1C4jAML404le9onW5IO3g9i ou0h1HJYsVJXElco48Tc0A2OlGdntGoi4t3wAHhNmW+50Y9hM8RlqY0twoEM0eq7IBoY tvRA== X-Gm-Message-State: AOJu0YxF+ramultSs/es6py5HK9iuOsx4zJC2mItHQLUbsenHuXsmTTY +EmpoLzbHJ3Y+M+dOa91cF8YZ0inoJodL6gXzdVX6f2WfwXkWG8NdNNRiAbEu1nMcPUxEolWVvw c X-Gm-Gg: ASbGncuwHNGKqqKWEw5+Mlp0le/zzEbYV6RaYQh8ZtjRU2UrJWamE3Ea5/a5r/EXQgD FOweINwgYwPppTnD688RLRuABa9NnK1eFMMA9uFDPEqCQ9HXui4TtQk9Db/QiEfOQKoqSfALBGI lti/uP5P7ui8iilAGaqJUrG2Nlk1BQHnmA0ZY/Mclzt4WI+kZbhlqVqB+IlRHIDq07l+ULBz+AN ArthXbRH1xVYY6Dhz1TOsh/809id09hkCnSef/BZpM2+PCz0uDwDCNBDb54B7rKoCfjEo9tKZRl sSF9De/bizrUOqoT9Vj1J9eBYNXG+3dr8fSQKFU8oRNf X-Google-Smtp-Source: AGHT+IEehELWmcBbdecL9es36IQOdasnFILqW/uOVVAwd8+bi/x2W25liHS2wXN9vDABgW4tYHvLfg== X-Received: by 2002:a05:6214:766:b0:6d4:1b2b:c836 with SMTP id 6a1803df08f44-6dd233b7f4cmr1061964726d6.47.1736208240369; Mon, 06 Jan 2025 16:04:00 -0800 (PST) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6dd18137218sm174104476d6.57.2025.01.06.16.03.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Jan 2025 16:04:00 -0800 (PST) From: Gregory Price To: linux-mm@kvack.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, nehagholkar@meta.com, abhishekd@meta.com, david@redhat.com, nphamcs@gmail.com, gourry@gourry.net, akpm@linux-foundation.org, hannes@cmpxchg.org, kbusch@meta.com, ying.huang@linux.alibaba.com, feng.tang@intel.com, donettom@linux.ibm.com Subject: [PATCH v3 5/6] migrate: implement migrate_misplaced_folio_batch Date: Mon, 6 Jan 2025 19:03:45 -0500 Message-ID: <20250107000346.1338481-6-gourry@gourry.net> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250107000346.1338481-1-gourry@gourry.net> References: <20250107000346.1338481-1-gourry@gourry.net> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Stat-Signature: 8n5anpz71qbuqk7yzmpcrbwd3fbhqsb3 X-Rspamd-Queue-Id: 6B0DD40016 X-Rspam-User: X-HE-Tag: 1736208241-444651 X-HE-Meta: U2FsdGVkX182Ah2gTnkJz6q6O44+TVo98VjiV5QW9vrUvBb3mJwMk+FEqC4aKZDOYyMGKR97GZhZ2BLagXyKVmv8ZW+zirErSF90r6MtQfnYSZ/XpTO/RuuMu2zljyHKiNAhmiGA7l7b4D8LbUYXfRpVcTv+k/q/Oug0SQ43bl8d6dZWewCgplK4f43TgwulEv1CT5j7Xlm0tY0i3ixSgHU4YlBj1cCVcI7o2Rjx/2MEhOn5/FT7FYUCJR9IscAkKbXpcX/O2JvXOHRPzh4gods2EovrDb9Pmc6zUlRD+XcmuIScjtsBeckwkAOVNaMt95O76oYxnuCIGKAISZVOI3+vC6IWGrhxVXsHRDogHmwwvzyV25uFU9sgMX7HaHDVvDwnQ77c+WnqHPGMZoZ9t78/+cVE44AH6QnooEQVQcswTdeGqA1QJ685yqGIgf4DrIu30DXklzR2Cb6lD/rPc95vT6KU5QsBX2V78sHg9XxA2B3+kqLTFO1aP2ddlSMtpHMPW6HxdU1C3rjp/yNOqnKTcivH6kH53kc9rY0kUZDejEC+IWAQCvKHFHiuEUdRrbWsPtfqj8hFCzC7YD7YvD/jmJlUp/ywWcmS/FOvysmtQARMpQ4WLrIu9WI71NoJ7bqzCfp4HqE4K6f2kzevfJmEnagGg7jfYfDXb+EmJmPT2lL/vP5vcTs51hEcaIqyVuzZzlK6mc4/+KR1dyfy4zGPXGv8eL80JOS8fD9D0UtVygG+Smg0c3s0Om8xMhFBJ4dgdHb5ar4CUDs/lVfZFqZR3kdXT619CZUb3ZLazT6tRQTaCYWQmIqRqlGvqHNALum6cAyedXbQdts2Yjp5QGzgopMuMickezMZOOOIWKH7pcVt7oUpRc5i/1P7CHkq2XOzkc054AudoJvEkhmpnS9brI4+58jI45j4/Np2HqTZ96DvhnEjPnO8lI64U4TG6P1CSR8CAQQYL/gxHGm BDkqMVrl DCFB+6Vm0sPLACkKcMiLK47sIhVFSBfi5C7OIM2dl9MhifpA8Ajc8qjoiQovtYdLD3jtmLodst8e5A9qr66IlBKqlrzRNfA8LA4n25aCcQ6ky0Fmi8qCYCwcTIMRzmt0qX4//Cjgij6v6/LcIiRfTifyPsHVJFd11GGKM8qAsPkbN5LKh6Hvj9JXXkGY0B1OMzC2BJw9odTsPiuDQVxGwIg3Pdq1sxVipum0bTWY4//jpLmWKOQwbBvFbc2d2MqLs8OlzUSRN8cWtUORUp+wSdp+H7N8yeFJyMsrjR/DVPxEMbPvrD+/AxwDCt0yZsfhntSuAa1GEIQxvCshkWHXr/Z627nfllSRnFc2L X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: A common operation in tiering is to migrate multiple pages at once. The migrate_misplaced_folio function requires one call for each individual folio. Expose a batch-variant of the same call for use when doing batch migrations. Signed-off-by: Gregory Price --- include/linux/migrate.h | 5 +++++ mm/migrate.c | 31 +++++++++++++++++++++++++++++++ 2 files changed, 36 insertions(+) diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 29919faea2f1..3dfbe7c1cc83 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -145,6 +145,7 @@ const struct movable_operations *page_movable_ops(struct page *page) int migrate_misplaced_folio_prepare(struct folio *folio, struct vm_area_struct *vma, int node); int migrate_misplaced_folio(struct folio *folio, int node); +int migrate_misplaced_folio_batch(struct list_head *foliolist, int node); #else static inline int migrate_misplaced_folio_prepare(struct folio *folio, struct vm_area_struct *vma, int node) @@ -155,6 +156,10 @@ static inline int migrate_misplaced_folio(struct folio *folio, int node) { return -EAGAIN; /* can't migrate now */ } +int migrate_misplaced_folio_batch(struct list_head *foliolist, int node) +{ + return -EAGAIN; /* can't migrate now */ +} #endif /* CONFIG_NUMA_BALANCING */ #ifdef CONFIG_MIGRATION diff --git a/mm/migrate.c b/mm/migrate.c index ea20d9bc4f40..a751a995f2d9 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -2705,5 +2705,36 @@ int migrate_misplaced_folio(struct folio *folio, int node) BUG_ON(!list_empty(&migratepages)); return nr_remaining ? -EAGAIN : 0; } + +/* + * Batch variant of migrate_misplaced_folio. Attempts to migrate + * a folio list to the specified destination. + * + * Caller is expected to have isolated the folios by calling + * migrate_misplaced_folio_prepare(), which will result in an + * elevated reference count on the folio. + * + * This function will un-isolate the folios, dereference them, and + * remove them from the list before returning. + */ +int migrate_misplaced_folio_batch(struct list_head *folio_list, int node) +{ + pg_data_t *pgdat = NODE_DATA(node); + unsigned int nr_succeeded; + int nr_remaining; + + nr_remaining = migrate_pages(folio_list, alloc_misplaced_dst_folio, + NULL, node, MIGRATE_ASYNC, + MR_NUMA_MISPLACED, &nr_succeeded); + if (nr_remaining) + putback_movable_pages(folio_list); + + if (nr_succeeded) { + count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded); + mod_node_page_state(pgdat, PGPROMOTE_SUCCESS, nr_succeeded); + } + BUG_ON(!list_empty(folio_list)); + return nr_remaining ? -EAGAIN : 0; +} #endif /* CONFIG_NUMA_BALANCING */ #endif /* CONFIG_NUMA */ From patchwork Tue Jan 7 00:03:46 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gregory Price X-Patchwork-Id: 13927914 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD6E0E77188 for ; Tue, 7 Jan 2025 00:04:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E58256B00CD; Mon, 6 Jan 2025 19:04:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E097B6B00CF; Mon, 6 Jan 2025 19:04:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BBF046B00D0; Mon, 6 Jan 2025 19:04:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 962626B00CD for ; Mon, 6 Jan 2025 19:04:05 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 3EE5B1404FF for ; Tue, 7 Jan 2025 00:04:05 +0000 (UTC) X-FDA: 82978708050.28.319B9B4 Received: from mail-qv1-f47.google.com (mail-qv1-f47.google.com [209.85.219.47]) by imf29.hostedemail.com (Postfix) with ESMTP id 68C9D12000C for ; Tue, 7 Jan 2025 00:04:03 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gourry.net header.s=google header.b=hLk1WsX3; dmarc=none; spf=pass (imf29.hostedemail.com: domain of gourry@gourry.net designates 209.85.219.47 as permitted sender) smtp.mailfrom=gourry@gourry.net ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736208243; a=rsa-sha256; cv=none; b=Js2pX+ECAZkGDh/IPXWfhBWLyZWoWT80SMggF3b+i/hVrDi7mWkLT0p9DLoFGp5PTNy3KR 1x9VfQqkQTQKRgGMDZSXtRp0pokR1AnPuNo+cqPVhjpE99ikt/VovHZQJMJBtiasqeXfo9 bnk7eW0xqhek7hmFwkz4wQC0g5JLohM= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=gourry.net header.s=google header.b=hLk1WsX3; dmarc=none; spf=pass (imf29.hostedemail.com: domain of gourry@gourry.net designates 209.85.219.47 as permitted sender) smtp.mailfrom=gourry@gourry.net ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736208243; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=V7/RT8+H7mI8JxtxXU0vRyxqoG6aLJf762JcOXh6uPk=; b=enBQN/lXO3HzRa38WEgv4QVxIQWHqnR9Jsm8BMSPeVuJVP/OSZiFCzXeRF3iQeiYlak9j/ WLym9Y3A/iFRCvJ2IEQakP4nRwzSkT+Aicj0KQ22Ee+U2LqWYRqD8GAgayVnvQ7geU1Bwo m1D5YNUbtuCOyh2F5P3uHaJo5Pu42eA= Received: by mail-qv1-f47.google.com with SMTP id 6a1803df08f44-6d8fd060e27so111505986d6.1 for ; Mon, 06 Jan 2025 16:04:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1736208242; x=1736813042; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=V7/RT8+H7mI8JxtxXU0vRyxqoG6aLJf762JcOXh6uPk=; b=hLk1WsX39dtzGhr5OL95AZBG0PU6z21xShZHp5Hm2DMSYapZoAisqTEWzFcxS1d0Qg qX41v8aekXAXvIJxLugX4x2+DJbOkOFeYoOTj0D374zx9VCo0PcGLKu+OyzSCb1AMiux ZXl99fofkdg/EkoF/Ey1qhpxBKiKbRe7kINoAyUDpXZuL9rNZezaAxAbLU9ISRav376V 9AURACGz0RGnr14+m0YfNxxY8b54YQ/2T2NJ92rVQXdqugcy4C6rZaFKvh/NcM5VB0dx HDEa8qgil8itWNwmwyt0Ey6zuXd0oexkqWojy6HCl9goVJfNjwHfUH7WQH5Dn4QqZ6Rx T86g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736208242; x=1736813042; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=V7/RT8+H7mI8JxtxXU0vRyxqoG6aLJf762JcOXh6uPk=; b=jH86XMXtlUnvuQZ7f2N178AgxL8rZAXwXvCig9+/hRrBXF2gUGsIBMsiNdiBqreTzU UH+YG48NEx2A/EYnT0tNOhGFun7BtoBTMvghdAgNf31VwZBYIzCPbSARQZMU0acA4Zkl 55o0tyVFHT+82yqP2jDpnWn5jdL7quxyNk3iGxUoZpmSBLeFoYQVHFIO+ul2jN6VOnrY XNE5QMTyONBcEJTFJffFjwaoXW60YGdD6sJnPUHSOsi6bnanrqvOmv5Vk9sBxjcl8Rvd a0nW9+Gar4wiiRzMQb8fAdgBBYQq8zWBKNPk3Y1p4CAT35C672ptwNlf9YfSTldB8vo5 axSQ== X-Gm-Message-State: AOJu0Yx4FU6dJxB/TpxtLKiSqO6wHWQLq8CAJtRKKJJvSQ0Sm8AA8l9u Dn7cEzpNmhMBaSH1uV2wrr2ua8CxL2JavGlR5oqMwU/L7C7hWWXX8m8csA0qx22S8Ti8Rt+bVo0 O X-Gm-Gg: ASbGncvUmEjc9FZvA74sOSwEWmv4n3227GS8DGuxiM2afzofYs+LqpObI6YCB5sig1m h2+0+Y4VqRpZ8Sz9+sQ+VqAA6lfD4lsjhxmpVgvU7oRf9cYg1qQeopsXVSKmKSKCFxE3zQuUbQo 5LgZXCPmb/ISiX65bS/M6Lex55HPRK4JWvu4L6Bgcmohlay+i1oe/oVoVK2MTD4qFOb0Lgf7fpd CcqLJqOxQWdXG0DEOzKcQInO+4CGcGw596EXYeczWuQK+toLuSgL4mX3m7jy7Dhzkz1zRDE6WbS mrYVHNzQVuhVaVt1h3qX0pIZ83TMDMxAweESi/MCX3SC X-Google-Smtp-Source: AGHT+IH2SA3s6BT+Bp1QFg246/HWcHqeI2BOW5TZn2KK9a/w4rSSBzDE/HpgoQ9zK1LVGRBVtpinZg== X-Received: by 2002:ad4:5aa8:0:b0:6d8:883b:142a with SMTP id 6a1803df08f44-6dd2330898emr1077157996d6.2.1736208242230; Mon, 06 Jan 2025 16:04:02 -0800 (PST) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6dd18137218sm174104476d6.57.2025.01.06.16.04.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Jan 2025 16:04:01 -0800 (PST) From: Gregory Price To: linux-mm@kvack.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, nehagholkar@meta.com, abhishekd@meta.com, david@redhat.com, nphamcs@gmail.com, gourry@gourry.net, akpm@linux-foundation.org, hannes@cmpxchg.org, kbusch@meta.com, ying.huang@linux.alibaba.com, feng.tang@intel.com, donettom@linux.ibm.com Subject: [PATCH v3 6/6] migrate,sysfs: add pagecache promotion Date: Mon, 6 Jan 2025 19:03:46 -0500 Message-ID: <20250107000346.1338481-7-gourry@gourry.net> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250107000346.1338481-1-gourry@gourry.net> References: <20250107000346.1338481-1-gourry@gourry.net> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 68C9D12000C X-Stat-Signature: uxgthwar7pffbodk1eseomra614ie3t7 X-HE-Tag: 1736208243-684489 X-HE-Meta: U2FsdGVkX19C+qBiga148kJbE+G7+uWn/iUVCne663JuBVcaSqdSwaPAXHRXRbM+qoBmEWvkz9hPQDIa+thm4OwEefLKEOErf+gxAKMZnAO5K/yBwOURQuxutZpxyKUVCWPikJEd6le5yHYzMnqn8wsvu5OSpKjMqteu9o5bUq0Gx6esUO1mCtVxNhkrn5l5PZEEEHSJcPdp1nVMMCElRGN7Z3f3oFwew/9bdRaRhFRqtdeyNeABD5uTcTk+U08BkNZpiFPOacXt79VCbcfTxH8QkFqRlvXeSHWm0K523avrBbZ+M1IX50K/wXJEXce83PgeIT9mcBNx7jhUCEYDYrPR4IHmkSEuqsOaY2TpHI3MgK0Za0HVAaqIt0/hW6fvIGyyhyGaoh0Ta1XQ/hqhNf3AS3n3bwizTB6VQRBlM4S7DVRFKLfEMwjPf7/HXrGUxxkBIl0SpXvGbasV5g5NpgFjesXLh779rjsDNW/bvVTAbIt1Tbc80NwOXLFpyRsaOFnP8kWxPqvstDCWhjIYhhmewRHr4pM6d/VTn12GZAugerBLF0Xk5hW/4/u411BHLvnpGzokcNeSKlpBMWRd3egB2zJMd45VIoYaQkvOxjLAA1uVksUZ8jOABouFqfYVpQ0kM+4YCaSVof8c/x/dP7heHPpQxSG8mVrNA3/28UEnA+TfZqfW/wi+zSTn4TVuFRWG7GmYEcDn3NsnH8A8xl72ccVbOAK757R/6bN0w4D7d8Ah2rfLrFWHaxgEAwCa+hOEZIc9MPrgRY16ieWVn6jzXGebtk3JMMmI/+lEY6OAzD3XJzyq+PDZKCDYa9gPrhKNprONEjdORS9Y9C+VLceCjKSqedIUJmQme+BjY0JE6nfqpBHWBdWcBHZwQ6JPndrNKmZGGmkv1r9ew0lPPKnV4291ZlMLXw1FXZmiQjL0cGQeYlCSDptchV7v4c4pAXxkufg2NbUVrp9V+kk Ht57Hxiz DC+EtbddaB03hicRbHCS4500zP+8CdtnLa7eWlneghPdG8S+51muh8pSqxMYXn4hykHKLiXBqFWCgFy7HGu6D9WZt1xAXZ1qoCk+9W1tm7zaEnD/qIMSnR758ODhxIMdPd7M3Fsk0PUls9MU2wmOi0Jv6ijx1wkDK3guLiMHQo6EsAkR7FKCxGAmKb55pkAA7o3a4hBaR9k/SwweUX4+z1ZbYRAYJqRaIAhsnr3LBzqTUr4mrKBiMh7yDXg6gVnGUYVHlERXL2rD/+6Vc6b+afdiVFnYLNb38wz5CRJALqG/rAeESnAV2I5uTkHkyHU/+aukqxP33+f1X2KwVzhtEw6NwXS3YRXUm4v78rPJAck5MxWV6aca+E3OPsJWH4x7ppuDrG9m+OKEOLCQFrGJ7iXj5Li9L0mkY3k4of0JMgcknqkE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: adds /sys/kernel/mm/numa/pagecache_promotion_enabled When page cache lands on lower tiers, there is no way for promotion to occur unless it becomes memory-mapped and exposed to NUMA hint faults. Just adding a mechanism to promote pages unconditionally, however, opens up significant possibility of performance regressions. Similar to the `demotion_enabled` sysfs entry, provide a sysfs toggle to enable and disable page cache promotion. This option will enable opportunistic promotion of unmapped page cache during syscall access. This option is intended for operational conditions where demoted page cache will eventually contain memory which becomes hot - and where said memory likely to cause performance issues due to being trapped on the lower tier of memory. A Page Cache folio is considered a promotion candidates when: 0) tiering and pagecache-promotion are enabled 1) the folio resides on a node not in the top tier 2) the folio is already marked referenced and active. 3) Multiple accesses in (referenced & active) state occur quickly. Since promotion is not safe to execute unconditionally from within folio_mark_accessed, we defer promotion to a new task_work captured in the task_struct. This ensures that the task doing the access has some hand in promoting pages - even among deduplicated read only files. We limit the total number of folios on the promotion list to the promotion rate limit to limit the amount of inline work done during large reads - avoiding significant overhead. We do not use the existing rate-limit check function this checked during the migration anyway. The promotion node is always the local node of the promoting cpu. Suggested-by: Johannes Weiner Signed-off-by: Gregory Price --- .../ABI/testing/sysfs-kernel-mm-numa | 20 +++++++ include/linux/memory-tiers.h | 2 + include/linux/migrate.h | 5 ++ include/linux/sched.h | 4 ++ include/linux/sched/sysctl.h | 1 + init/init_task.c | 2 + kernel/sched/fair.c | 24 +++++++- mm/memory-tiers.c | 27 +++++++++ mm/migrate.c | 55 +++++++++++++++++++ mm/swap.c | 8 +++ 10 files changed, 147 insertions(+), 1 deletion(-) diff --git a/Documentation/ABI/testing/sysfs-kernel-mm-numa b/Documentation/ABI/testing/sysfs-kernel-mm-numa index 77e559d4ed80..ebb041891db2 100644 --- a/Documentation/ABI/testing/sysfs-kernel-mm-numa +++ b/Documentation/ABI/testing/sysfs-kernel-mm-numa @@ -22,3 +22,23 @@ Description: Enable/disable demoting pages during reclaim the guarantees of cpusets. This should not be enabled on systems which need strict cpuset location guarantees. + +What: /sys/kernel/mm/numa/pagecache_promotion_enabled +Date: January 2025 +Contact: Linux memory management mailing list +Description: Enable/disable promoting pages during file access + + Page migration during file access is intended for systems + with tiered memory configurations that have significant + unmapped file cache usage. By default, file cache memory + on slower tiers will not be opportunistically promoted by + normal NUMA hint faults, because the system has no way to + track them. This option enables opportunistic promotion + of pages that are accessed via syscall (e.g. read/write) + if multiple accesses occur in quick succession. + + It may move data to a NUMA node that does not fall into + the cpuset of the allocating process which might be + construed to violate the guarantees of cpusets. This + should not be enabled on systems which need strict cpuset + location guarantees. diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h index 0dc0cf2863e2..fa96a67b8996 100644 --- a/include/linux/memory-tiers.h +++ b/include/linux/memory-tiers.h @@ -37,6 +37,7 @@ struct access_coordinate; #ifdef CONFIG_NUMA extern bool numa_demotion_enabled; +extern bool numa_pagecache_promotion_enabled; extern struct memory_dev_type *default_dram_type; extern nodemask_t default_dram_nodes; struct memory_dev_type *alloc_memory_type(int adistance); @@ -76,6 +77,7 @@ static inline bool node_is_toptier(int node) #else #define numa_demotion_enabled false +#define numa_pagecache_promotion_enabled false #define default_dram_type NULL #define default_dram_nodes NODE_MASK_NONE /* diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 3dfbe7c1cc83..80438ddd76c7 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -146,6 +146,7 @@ int migrate_misplaced_folio_prepare(struct folio *folio, struct vm_area_struct *vma, int node); int migrate_misplaced_folio(struct folio *folio, int node); int migrate_misplaced_folio_batch(struct list_head *foliolist, int node); +void promotion_candidate(struct folio *folio); #else static inline int migrate_misplaced_folio_prepare(struct folio *folio, struct vm_area_struct *vma, int node) @@ -160,6 +161,10 @@ int migrate_misplaced_folio_batch(struct list_head *foliolist, int node) { return -EAGAIN; /* can't migrate now */ } +static inline void promotion_candidate(struct folio *folio) +{ + return; +} #endif /* CONFIG_NUMA_BALANCING */ #ifdef CONFIG_MIGRATION diff --git a/include/linux/sched.h b/include/linux/sched.h index 66b311fbd5d6..84b9bcfaa01d 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1363,6 +1363,10 @@ struct task_struct { unsigned long numa_faults_locality[3]; unsigned long numa_pages_migrated; + + struct callback_head numa_promo_work; + struct list_head promo_list; + unsigned long promo_count; #endif /* CONFIG_NUMA_BALANCING */ #ifdef CONFIG_RSEQ diff --git a/include/linux/sched/sysctl.h b/include/linux/sched/sysctl.h index 5a64582b086b..50b1d1dc27e2 100644 --- a/include/linux/sched/sysctl.h +++ b/include/linux/sched/sysctl.h @@ -25,6 +25,7 @@ enum sched_tunable_scaling { #ifdef CONFIG_NUMA_BALANCING extern int sysctl_numa_balancing_mode; +extern unsigned int sysctl_numa_balancing_promote_rate_limit; #else #define sysctl_numa_balancing_mode 0 #endif diff --git a/init/init_task.c b/init/init_task.c index e557f622bd90..47162ed14106 100644 --- a/init/init_task.c +++ b/init/init_task.c @@ -187,6 +187,8 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = { .numa_preferred_nid = NUMA_NO_NODE, .numa_group = NULL, .numa_faults = NULL, + .promo_list = LIST_HEAD_INIT(init_task.promo_list), + .promo_count = 0, #endif #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) .kasan_depth = 1, diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 3e9ca38512de..7612d5c2d75d 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -42,6 +42,7 @@ #include #include #include +#include #include #include #include @@ -126,7 +127,7 @@ static unsigned int sysctl_sched_cfs_bandwidth_slice = 5000UL; #ifdef CONFIG_NUMA_BALANCING /* Restrict the NUMA promotion throughput (MB/s) for each target node. */ -static unsigned int sysctl_numa_balancing_promote_rate_limit = 65536; +unsigned int sysctl_numa_balancing_promote_rate_limit = 65536; #endif #ifdef CONFIG_SYSCTL @@ -3537,6 +3538,25 @@ static void task_numa_work(struct callback_head *work) } } +static void task_numa_promotion_work(struct callback_head *work) +{ + struct task_struct *p = current; + struct list_head *promo_list= &p->promo_list; + int nid = numa_node_id(); + + SCHED_WARN_ON(p != container_of(work, struct task_struct, numa_promo_work)); + + work->next = work; + + if (list_empty(promo_list)) + return; + + migrate_misplaced_folio_batch(promo_list, nid); + current->promo_count = 0; + return; +} + + void init_numa_balancing(unsigned long clone_flags, struct task_struct *p) { int mm_users = 0; @@ -3561,8 +3581,10 @@ void init_numa_balancing(unsigned long clone_flags, struct task_struct *p) RCU_INIT_POINTER(p->numa_group, NULL); p->last_task_numa_placement = 0; p->last_sum_exec_runtime = 0; + INIT_LIST_HEAD(&p->promo_list); init_task_work(&p->numa_work, task_numa_work); + init_task_work(&p->numa_promo_work, task_numa_promotion_work); /* New address space, reset the preferred nid */ if (!(clone_flags & CLONE_VM)) { diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c index fc14fe53e9b7..e8acb54aa8df 100644 --- a/mm/memory-tiers.c +++ b/mm/memory-tiers.c @@ -935,6 +935,7 @@ static int __init memory_tier_init(void) subsys_initcall(memory_tier_init); bool numa_demotion_enabled = false; +bool numa_pagecache_promotion_enabled = false; #ifdef CONFIG_MIGRATION #ifdef CONFIG_SYSFS @@ -957,11 +958,37 @@ static ssize_t demotion_enabled_store(struct kobject *kobj, return count; } +static ssize_t pagecache_promotion_enabled_show(struct kobject *kobj, + struct kobj_attribute *attr, + char *buf) +{ + return sysfs_emit(buf, "%s\n", + numa_pagecache_promotion_enabled ? "true" : "false"); +} + +static ssize_t pagecache_promotion_enabled_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + ssize_t ret; + + ret = kstrtobool(buf, &numa_pagecache_promotion_enabled); + if (ret) + return ret; + + return count; +} + + static struct kobj_attribute numa_demotion_enabled_attr = __ATTR_RW(demotion_enabled); +static struct kobj_attribute numa_pagecache_promotion_enabled_attr = + __ATTR_RW(pagecache_promotion_enabled); + static struct attribute *numa_attrs[] = { &numa_demotion_enabled_attr.attr, + &numa_pagecache_promotion_enabled_attr.attr, NULL, }; diff --git a/mm/migrate.c b/mm/migrate.c index a751a995f2d9..97f86ee6fd0d 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -44,6 +44,8 @@ #include #include #include +#include +#include #include @@ -2736,5 +2738,58 @@ int migrate_misplaced_folio_batch(struct list_head *folio_list, int node) BUG_ON(!list_empty(folio_list)); return nr_remaining ? -EAGAIN : 0; } + +/** + * promotion_candidate: report a promotion candidate folio + * + * The folio will be isolated from LRU if selected, and task_work will + * putback the folio on promotion failure. + * + * Candidates may not be promoted and may be returned to the LRU. + * + * Takes a folio reference that will be released in task work. + */ +void promotion_candidate(struct folio *folio) +{ + struct task_struct *task = current; + struct list_head *promo_list= &task->promo_list; + struct callback_head *work = &task->numa_promo_work; + int nid = folio_nid(folio); + int flags, last_cpupid; + + /* do not migrate toptier folios or in kernel context */ + if (node_is_toptier(nid) || task->flags & PF_KTHREAD) + return; + + /* + * Limit per-syscall migration rate to balancing rate limit. This avoids + * excessive work during large reads knowing that task work is likely to + * hit the rate limit and put excess folios back on the LRU anyway. + */ + if (task->promo_count >= sysctl_numa_balancing_promote_rate_limit) + return; + + /* Isolate the folio to prepare for migration */ + nid = numa_migrate_check(folio, NULL, 0, &flags, folio_test_dirty(folio), + &last_cpupid); + if (nid == NUMA_NO_NODE) + return; + + if (migrate_misplaced_folio_prepare(folio, NULL, nid)) + return; + + /* + * If work is pending, add this folio to the list. Otherwise, ensure + * the task will execute the work, otherwise we can leak folios. + */ + if (list_empty(promo_list) && task_work_add(task, work, TWA_RESUME)) { + folio_putback_lru(folio); + return; + } + list_add_tail(&folio->lru, promo_list); + task->promo_count += folio_nr_pages(folio); + return; +} +EXPORT_SYMBOL(promotion_candidate); #endif /* CONFIG_NUMA_BALANCING */ #endif /* CONFIG_NUMA */ diff --git a/mm/swap.c b/mm/swap.c index 746a5ceba42c..c006be6048f4 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -37,6 +37,10 @@ #include #include #include +#include +#include +#include +#include #include "internal.h" @@ -474,6 +478,10 @@ void folio_mark_accessed(struct folio *folio) __lru_cache_activate_folio(folio); folio_clear_referenced(folio); workingset_activation(folio); + } else if (!folio_test_isolated(folio) && + (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) && + numa_pagecache_promotion_enabled) { + promotion_candidate(folio); } if (folio_test_idle(folio)) folio_clear_idle(folio);