From patchwork Tue Aug 13 16:37:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yuanchu Xie X-Patchwork-Id: 13762282 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6B5AC52D7C for ; Tue, 13 Aug 2024 16:38:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 17B186B008A; Tue, 13 Aug 2024 12:38:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 12CB16B0092; Tue, 13 Aug 2024 12:38:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F35696B0095; Tue, 13 Aug 2024 12:38:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D2C876B008A for ; Tue, 13 Aug 2024 12:38:17 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 2BD2EA098B for ; Tue, 13 Aug 2024 16:38:17 +0000 (UTC) X-FDA: 82447779834.03.7EBFA28 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf17.hostedemail.com (Postfix) with ESMTP id 7CC4540021 for ; Tue, 13 Aug 2024 16:38:14 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=GKoJr2dZ; spf=pass (imf17.hostedemail.com: domain of 39Yu7ZgcKCDgsoUhWboaiiafY.Wigfchor-ggepUWe.ila@flex--yuanchu.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=39Yu7ZgcKCDgsoUhWboaiiafY.Wigfchor-ggepUWe.ila@flex--yuanchu.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723567038; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=R6SDifdhjqO17yC0HFRwUIHNeNat6a47KY3Ccl7msPA=; b=mpAPIL7NjXj9mioW+ds7Cr7l2BHftO9VIPAjQnCptNZd0xYxZHBrK5lA/9QkWb2mcLoNr/ 3UxytzKGqKpmO6kht7B34QAXdvDYYDsNJ+CCxOVM/SNZJEvgiNQaX7T5sdQ1KG3zdJvzQT yF/+3aE8XJd4L0RKZ+c5Tgz2c7xZ7Mg= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=GKoJr2dZ; spf=pass (imf17.hostedemail.com: domain of 39Yu7ZgcKCDgsoUhWboaiiafY.Wigfchor-ggepUWe.ila@flex--yuanchu.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=39Yu7ZgcKCDgsoUhWboaiiafY.Wigfchor-ggepUWe.ila@flex--yuanchu.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723567038; a=rsa-sha256; cv=none; b=DzeGR/l6zqLrCNgUaEtArLzEb2DxjAaH1G7/o850jLEKKFu4Iw1+gOFouv2WaJooAhTcF1 kWZVTudJHCyH6ngPjPjwK3/S8BnyxofuhJOxmeA0xKzH+xDlFkb4xxJVCMX+nMi9FwwsdY dyGp61SOJFAdjvIKTR7L+BZAyZQqgn8= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-666010fb35cso775397b3.0 for ; Tue, 13 Aug 2024 09:38:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723567093; x=1724171893; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=R6SDifdhjqO17yC0HFRwUIHNeNat6a47KY3Ccl7msPA=; b=GKoJr2dZkL9TtIQzxKRKDV++lL2BGxAXYiCP1OeWSnivzzmGQ7pY0q0xz44ak/iDVd RTfBYa6d3GIYMzxitEVL3Yh/VYokumE0kjXao7Iw0oMwOkLIyej/bWy8q10Po+5Aq9S0 DjRuPgif6YiYaXXv1I9UdVgStsxyK10ZOTy5+uLi+PZztp4w1WGjJ4eo8cdtPkxQUFlL rZg9K0LngC7e2yA257/5tmdk0x3mzDwahJv1LummWAolcRJ3JO+0QPAOV3sfMEkPdU6v SArOzUOH/L180vrdyZVgE8m+0cMKcmx+gTzGi1/DBraYaH+tRheuUil2gysKi5YQnP4k akUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723567093; x=1724171893; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=R6SDifdhjqO17yC0HFRwUIHNeNat6a47KY3Ccl7msPA=; b=GUehW0XzaQKUdg106/R49XSTNln5GkOVFWZd7zirMvv8DCCLCTRVwGSTr9GzcQmzfD +FvvRoVlbK1uzjXgKn3Fo6Nuk8T1NenAxwx5gkeUpAq4YVOPgjpNwStXc65y1d6ZRcLI vK95pvA4S8Y/LRO2OVSUsQ5s5uZHKOXhc1qMRhz1/N3owPP8TYUaLs8dBWblsjZbpIg3 M9YW2on4jk1Y793DG/9xlLJ55Ce9KgWAFohK2nnMrH8PqqsFZswzsoNNoB6i5pM9VSw2 oSP/x8RBuD22ik/TnrO47dqN20SjssQPggFcJLbhik0NbOweuzy6cKgL8f3YFS9UtGbY oBoA== X-Forwarded-Encrypted: i=1; AJvYcCW5YXZ5D3R/oUnpiew1PoIiWt6G8mBr2OoCaQcCwbVrIGTKB08rAFocaOWV3Vu1lZkZqYvUnESCVw==@kvack.org X-Gm-Message-State: AOJu0Yx0xCNsOaNo5EML1kdT/l886IeWsIjorEeCogQBp2u91IJAK+ak uu7vbpVeaALgyyj8IzHa5rScGYNruSc//iou1bu9zZ36vMAye01pCkXbjy96xYIpS9PIOHHiIzH HQ2kdPw== X-Google-Smtp-Source: AGHT+IGcnXWZqgJ+ZR2B4X2Y8tCJ7S7CD9pQ8Z0L73phBV4+zSqxFBKdePQrELcLsgKcQMYnrvHZtZWB2mEG X-Received: from yuanchu-desktop.svl.corp.google.com ([2620:15c:2a3:200:b50c:66e8:6532:a371]) (user=yuanchu job=sendgmr) by 2002:a81:77c1:0:b0:648:2f1d:1329 with SMTP id 00721157ae682-6ac707e5fc2mr93647b3.1.1723567093393; Tue, 13 Aug 2024 09:38:13 -0700 (PDT) Date: Tue, 13 Aug 2024 09:37:59 -0700 Mime-Version: 1.0 X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240813163759.742675-1-yuanchu@google.com> Subject: [PATCH] mm: multi-gen LRU: ignore non-leaf pmd_young for force_scan=true From: Yuanchu Xie To: Andrew Morton , Lance Yang , Huang Ying , Yu Zhao Cc: Yuanchu Xie , linux-kernel@vger.kernel.org, linux-mm@kvack.org X-Stat-Signature: prat8d1cbz9wbx34yi5pif4tum6yqx7o X-Rspam-User: X-Rspamd-Queue-Id: 7CC4540021 X-Rspamd-Server: rspam02 X-HE-Tag: 1723567094-136318 X-HE-Meta: U2FsdGVkX1+TBpKHTbNa0YuYNB9HgBAIY7rA4qV/YV+RdM3ZRFJ7hbpCzJqpWmw5+9OcrK4TWgBSLDH7jLEOrbwb4tmUH5IvvwhUsF8CsfC+O3Q5xselXDTOq1lQ/FbTJxrwHOLFGmLgsdVFJY1bzWezMj9ur8NjbXKLBwMnesLJrB2AjgqLLSo6dFccMuY7BVuWDH6cgqnp4n//cg2ZGZ33l2nlp3YMKVn+zqFN6UGM3GjS+SEgk/Amacx/BV4QCFVbLgV0j2ShDx6L804IdWnip0KCR5ehpnnnpKvUHsFynO5AV6EWyjx6QYNpOHLr+Y5GLw9IpkIiHQZLmBEztm9Ub1vVEQXIQMuMyTP6qXHikv0kMKgnTLnZ3FmFs0lo4hlm5i0WmCLcCckIwAyRDTFj+eAKw6p1MU5Ag4vwfvU+g5fobkE4bYO3MmS/49cE7XUJZqzIoRIP/6r9JG68AtSqYRV8gMU/7TFtEj+Nt3hkvoojVI98zD7Zn6p5xfEwUcpXfxO/bMR/bN+LT7vX08teaSEI2V2UOW7yV0tdAhglqoIMkzhtEYxZMmL2zZjElY6efo6JLaYubjAdEHWmw/S1L1NzPwor5HaOKLM7pdzWq0Sx0+6OzqTbSmz2q4ILijXVn9ivjV0Rf89FLpLLX4+2ps+vLPMMDKfjUin9oQ74Ri8VkPKJ61Z3GcE1yQMWUGngbQWuor5+sYFfY9GG3o4Bqo6cRbrZ4X+KFxxHm65FOt6ngwSfQk8OABTCu4rh7h9jbhP4eOXQZDroL0+iCdwiztLSfiWpTBdIx5h8Ag0ZwF/NbgN5wyF1iRWIUvbZW6p4QRSJ+MhlVdnSlDGgzRtFG/Qh9c2Z6inK62dKOQXm3natXrNavpRJflcJkj8BYE2F2pd1PZ5pJd/jWmgfLBGAzN0rZAOX1a0FveMVpHUT5QPl975s6Ugsw3b0z/w8tLmlVO44KdX59BMM7Rh 5dt3Q2xR 964u29nk6IjOPKoySTBqaZAqMYFqIboB+Qyra69Iu1SAufCXrSQfW1EfZBZ5qvTgB1Yh54FA6Tpef/ZYL6VlOmjpKoPfUZh+OO2HDf8fkc14S1cj7eDyKwD2pQiGTaKd/Ufk7dn5yoEyLkVxo6zz40kR1+GP3wDpVemCyTQ3xXS4UAm/ljrP0tDhNTMBfSFIMdjSDBoCEoGCoUusxUrB1f6i8nhNTu3akAo0DVVq4tfdGszvJ/JKHikTy/YNijSl2HbGQXYfUYdmgLoR0PbJuUTitWPyGAkiZN/FxTCgr629zBsKQWGHyU2GhHpbht1PQ0IKnIIPtFXOJ4n2mMLJUjd/w/MlrI8IhPTH88hYH1TiVRZHv6a9am6Vbck3MPFmI0M/aJ+GF6J6XMWJT3G3AkYPbKWfLYRJutXp+WYotZgkSanINk4z64Qts7nPhhe5G5oL5VmBdIQa4p6yMFHBW4Wo5JbVmKNeO4Mb/m5azPHquQkjn8wS9K7uFjoTCzc1qm4gYFQ9EfyiCBNFz9/oYebPbCOGPcGqXCJvEcHEbEt4edpXP/qitELAKhC7d0ArFS9fT X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When non-leaf pmd accessed bits are available, MGLRU page table walks can clear the non-leaf pmd accessed bit and ignore the accessed bit on the pte if it's on a different node, skipping a generation update as well. If another scan occurs on the same node as said skipped pte. the non-leaf pmd accessed bit might remain cleared and the pte accessed bits won't be checked. While this is sufficient for reclaim-driven aging, where the goal is to select a reasonably cold page, the access can be missed when aging proactively for workingset estimation of a node/memcg. In more detail, get_pfn_folio returns NULL if the folio's nid != node under scanning, so the page table walk skips processing of said pte. Now the pmd_young flag on this pmd is cleared, and if none of the pte's are accessed before another scan occurs on the folio's node, the pmd_young check fails and the pte accessed bit is skipped. Since force_scan disables various other optimizations, we check force_scan to ignore the non-leaf pmd accessed bit. Signed-off-by: Yuanchu Xie Acked-by: Yu Zhao --- mm/vmscan.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index cfa839284b92..4a112c2d1a64 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3476,7 +3476,7 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area goto next; if (!pmd_trans_huge(pmd[i])) { - if (should_clear_pmd_young()) + if (!walk->force_scan && should_clear_pmd_young()) pmdp_test_and_clear_young(vma, addr, pmd + i); goto next; } @@ -3563,7 +3563,7 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end, walk->mm_stats[MM_NONLEAF_TOTAL]++; - if (should_clear_pmd_young()) { + if (!walk->force_scan && should_clear_pmd_young()) { if (!pmd_young(val)) continue;