From patchwork Tue Dec 31 04:35:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13923614 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3016CE7718B for ; Tue, 31 Dec 2024 04:35:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 800D06B0085; Mon, 30 Dec 2024 23:35:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 78ACE6B0088; Mon, 30 Dec 2024 23:35:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 567806B0089; Mon, 30 Dec 2024 23:35:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 2A1556B0085 for ; Mon, 30 Dec 2024 23:35:53 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id E9C21160D15 for ; Tue, 31 Dec 2024 04:35:52 +0000 (UTC) X-FDA: 82953988992.01.A338488 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf05.hostedemail.com (Postfix) with ESMTP id 28632100008 for ; Tue, 31 Dec 2024 04:34:19 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=rz5x2o9M; spf=pass (imf05.hostedemail.com: domain of 3pXRzZwYKCJ4WSXF8MEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--yuzhao.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3pXRzZwYKCJ4WSXF8MEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735619715; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=n2jVLy+uRLHye3kRN9bJLrDt1hriY6TngJrphV7c4w0=; b=DaaWIiTZ0A8aHs6VoNoGhopSYz2JXYPiNb9e2KtwqI3inJRVUvczJe5f/fRR5wnCNvewXo x91GWOCTc3qcvH+NwcOJPM9C9yDLhDuVE9EeHniIbSall0ojsDIycjJi3ceSzWEaBauO4h nJhVTJ2Ho7+r7bPn6Yo1rkZ298vjg2A= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=rz5x2o9M; spf=pass (imf05.hostedemail.com: domain of 3pXRzZwYKCJ4WSXF8MEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--yuzhao.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3pXRzZwYKCJ4WSXF8MEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735619715; a=rsa-sha256; cv=none; b=BQnVZrOCDcvmgeFolF0/+ELJKgYEmWD1nA1J/FdzAFKhmKQfle+omHTbqRSTjRg4rezRWW k0RuhVInATHgiw92v/fZZUWMKyY7580KI/+n1q6kQz6GQwPZln+qMxuRvKOwFnJNprgWR1 +3IfHHKuC4bRilt/AhdBD73SOjf6l/w= Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-216405eea1fso122044725ad.0 for ; Mon, 30 Dec 2024 20:35:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1735619750; x=1736224550; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=n2jVLy+uRLHye3kRN9bJLrDt1hriY6TngJrphV7c4w0=; b=rz5x2o9Mv8SsgYlHC/5vbaD2FFhAGSqftqf2JHSOtm+mfAcOQu+c1ecIXKZqsj3ZSG hCNKZO5CCJFlvFjqKuTWCYgbJUFPZhg2uUKXIK0HBZTfRD3nFIJMSgZWrXx61V/sG/5L giGfWChKzB6JZUGv9Cbc4sYiKoRotAMkZW2K7vrteNw4YQ3Sk4SSPyw502VK6pk+1aiu 2zHW76N//eZUDwqcf33LcVWu3n2ZZNLmgziWBxbW+mkqW7EnVlIETSO1gb3FYya3Nh3Y SxTjUDTAKQYYkQoKSJdfP7H2I54Yl8cKCXzABiSSufa4iPpcr/sHijRBAhNxlRQmRJF8 wzxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735619750; x=1736224550; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=n2jVLy+uRLHye3kRN9bJLrDt1hriY6TngJrphV7c4w0=; b=aSGZcZIK0MYhHOs+2rKKgCdr8S0Vr9JI6lK3mabVV9BAQ6YPI3zu1s4lEz9lejwG1F kR9EbIVakCvD1osEOUcXOynlkKht/3Adn26xEeIXihCvOo93fyDIcbmnFHFQ5u86QhqY 3tK+2qbp8yylQ8mlF1Y6erhS+6BHEkz+5jFS5dUcbhqVU5D+qP8Be02QtQaSoZYmPFb3 zbJ467/VDSHsq1aue7hUsN41Z/uyG1hnAXZBfKqBkkB0SLp3/Uo89SK8ovbRn0MXCltN UZjS5j+Zgd+pYwCWLJ6XoKOzCrZpQZcLOzD75arrJEE5fTdzmWXnvpn7vWbc48zUewdI eAVA== X-Gm-Message-State: AOJu0Yxrbi4/6Tqv9xGJVwUckI9JY2HSh3jpb4cmrQhPKUFf3s3VDkL+ qTQHwcQ8X0KOEl1C0OV4VddJiekrDpkQUBvvU0n2Iap/fNqHo7aYBxIOj+OJ4AlVkbOFx1FO7lh kvw== X-Google-Smtp-Source: AGHT+IEyFV9cXf/CY6ZxXUeDDPF7xVQGYuDgmn8dUSrnoiU1eyKeiE9Z7M5a9yhOQj+r6LPmuJijye0+/+0= X-Received: from pgtq9.prod.google.com ([2002:a65:6849:0:b0:845:b983:9e34]) (user=yuzhao job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:ebc6:b0:216:501e:e321 with SMTP id d9443c01a7336-219e6f2700cmr576287415ad.56.1735619749657; Mon, 30 Dec 2024 20:35:49 -0800 (PST) Date: Mon, 30 Dec 2024 21:35:33 -0700 In-Reply-To: <20241231043538.4075764-1-yuzhao@google.com> Mime-Version: 1.0 References: <20241231043538.4075764-1-yuzhao@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241231043538.4075764-3-yuzhao@google.com> Subject: [PATCH mm-unstable v4 2/7] mm/mglru: optimize deactivation From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao , Bharata B Rao , Kalesh Singh X-Rspamd-Queue-Id: 28632100008 X-Rspamd-Server: rspam12 X-Stat-Signature: rsj11znwikngrf94gpogos16wsffm66c X-Rspam-User: X-HE-Tag: 1735619659-986033 X-HE-Meta: U2FsdGVkX19q2w9yUcG/9TaBCAXg6qUcwUVo6oDlqawq3RPAwse/aEF6Yv4Oj3LuMJcFFoKbMGt3KGmuppEA0yIR2nJeU7QY6HeXgjxGq1YPGNc+JuoJsDcTDHLK8AfKU1O6cJs1mJxBJm233DbHeX5RzLvzpMRxJuSNW4/aCe+MrUSbwQXC0KeQskiK1mHfjueDnfLNZjGT91cPnB3dVFCI9P7rWFoQC58Nxfd9iR01JMUhWAPu9P5j4fKTkfgZYT9XabQYKfyilNFrMeGYq3OGiLfkaIFDV6z32mMKEgx7qfj6qhg6kuaC32YtWUy+yKUoLOYXfSg5jp7o8g7GgZufC+HGlZdtv8+BTNlgNztmru0HRlzNfiRm+wdwSD0ETWoes7VskApuNTZKwinjBtEBB9dSzdNnRaXJbErP7XyFsm/Ivh+erM6HUx4skQEcHGAlYT8ISwN7/IcGmRwi5e033HuGtt1GDMjZdigRCT/XJ4r24XKu1A7OIBp91YuITndth8CgmCHlPcMmhHmKtzZeld07GYtDmqCvMTyJSopALCv40lLcsaV6OEhwJZmzgGR06RNDmvbk/NrDk2EXf4oXrht8T4yqcbR3aXTK8DbXTTaEIWa800XRY6YRUkhEiQEswS/dgPDjGy55y4miLw6cN+wZj/KcTEznVNqy8sOat430V996/f2gT8S5Zt9LIMWF4RZWgJkV38XEyVsRV8ZK04VfmC0YxF7YYTPQEyv6kLeO6WNfhP4WbQpl7pNvy/3RZ71DG4qhqqYDZw+631TD7A2D8JDbUeiZKXUBwENsL4f6vc0m7LTwqA336B32Y16dQ3hbBZlTaZnx04qAFG3c4dRH3MS9WaBydgQsY85KmMttuDGikzNf5RRKiGRN75YsDdB59GdkPhtn1DRT1sLEoAvf8TxUYCaPjIRXH6hYIY0KpNTuisLS8dSPbaSbBZDFEFfJmV3QF3gQmSL EpyOpPtF 3KPnM8FbI1JjJOJ3nbu+laH0godHQpTgWJFKXOg6N0WANbiVhSn6SUARGYia5NvR1zmlz2vbab5OJedT8qb3vNaJcFgKrdaQE1c5jzqLFLFoVZ2n+/ExQOHa7c2sjykaWAYxprYg9fC7MuXnResmmLoPwE1AjoNf6fTJR41VtnytE47LbjLBPcG3dXm9L62g0/E+aAQFRMLM6GIhL5gRkCnr0npoZHvlRmgiXgFgJsqqng05f+IO87w7+dbpt7tS3N/49VbX1Xpt4GKKWwWAxd0qWI/bk6Tq9WEzSSWKCiJdnICUOz8kKxZw2+1a5N6Ct1WfSoJrP/Oq7rMIdhkUBh5tSVG2rMb9OO3hr+5lpSiMb7aCdpu4Mm3AX0hmtYnJQ166FNAWbfeMnD3SsMTOGtXlFEFrzYirBRleLZpPc9vYzqNo1Pc/qtdOD8xIyhdVsuez6qwg2c5IpYsrvtu7Cu7F5F4zunRTxCYT7YluhzYvo3VswR4Bl7/XkjPNYgI7xS7u7nbTqhdbjVtbhZkhHG6zoV6tSpAcMELWjOVxU3iM8BGvcVLkWswVKMzqgZAjpdaU63yds6Ei7FzMN363UkuNPfas82rR0c/8qulhTEOvVkgg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Do not shuffle a folio in the deactivation paths if it is already in the oldest generation. This reduces the LRU lock contention. Before this patch, the contention is reproducible by FIO, e.g., fio -filename=/dev/nvme1n1p2 -direct=0 -thread -size=1024G \ -rwmixwrite=30 --norandommap --randrepeat=0 -ioengine=sync \ -bs=4k -numjobs=400 -runtime=25000 --time_based \ -group_reporting -name=mglru 98.96%--_raw_spin_lock_irqsave folio_lruvec_lock_irqsave | --98.78%--folio_batch_move_lru | --98.63%--deactivate_file_folio mapping_try_invalidate invalidate_mapping_pages invalidate_bdev blkdev_common_ioctl blkdev_ioctl After this patch, deactivate_file_folio() bails out early without taking the LRU lock. A side effect is that a folio can be left at the head of the oldest generation, rather than the tail. If reclaim happens at the same time, it cannot reclaim this folio immediately. Since there is no known correlation between truncation and reclaim, this side effect is considered insignificant. Reported-by: Bharata B Rao Closes: https://lore.kernel.org/CAOUHufawNerxqLm7L9Yywp3HJFiYVrYO26ePUb1jH-qxNGWzyA@mail.gmail.com/ Signed-off-by: Yu Zhao Tested-by: Kalesh Singh --- mm/swap.c | 48 +++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 41 insertions(+), 7 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index 3a01acfd5a89..649ef7f2b74b 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -379,7 +379,8 @@ static void __lru_cache_activate_folio(struct folio *folio) } #ifdef CONFIG_LRU_GEN -static void folio_inc_refs(struct folio *folio) + +static void lru_gen_inc_refs(struct folio *folio) { unsigned long new_flags, old_flags = READ_ONCE(folio->flags); @@ -406,10 +407,34 @@ static void folio_inc_refs(struct folio *folio) new_flags |= old_flags & ~LRU_REFS_MASK; } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags)); } -#else -static void folio_inc_refs(struct folio *folio) + +static bool lru_gen_clear_refs(struct folio *folio) { + struct lru_gen_folio *lrugen; + int gen = folio_lru_gen(folio); + int type = folio_is_file_lru(folio); + + if (gen < 0) + return true; + + set_mask_bits(&folio->flags, LRU_REFS_MASK | LRU_REFS_FLAGS, 0); + + lrugen = &folio_lruvec(folio)->lrugen; + /* whether can do without shuffling under the LRU lock */ + return gen == lru_gen_from_seq(READ_ONCE(lrugen->min_seq[type])); } + +#else /* !CONFIG_LRU_GEN */ + +static void lru_gen_inc_refs(struct folio *folio) +{ +} + +static bool lru_gen_clear_refs(struct folio *folio) +{ + return false; +} + #endif /* CONFIG_LRU_GEN */ /** @@ -428,7 +453,7 @@ static void folio_inc_refs(struct folio *folio) void folio_mark_accessed(struct folio *folio) { if (lru_gen_enabled()) { - folio_inc_refs(folio); + lru_gen_inc_refs(folio); return; } @@ -524,7 +549,7 @@ void folio_add_lru_vma(struct folio *folio, struct vm_area_struct *vma) */ static void lru_deactivate_file(struct lruvec *lruvec, struct folio *folio) { - bool active = folio_test_active(folio); + bool active = folio_test_active(folio) || lru_gen_enabled(); long nr_pages = folio_nr_pages(folio); if (folio_test_unevictable(folio)) @@ -589,7 +614,10 @@ static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio) lruvec_del_folio(lruvec, folio); folio_clear_active(folio); - folio_clear_referenced(folio); + if (lru_gen_enabled()) + lru_gen_clear_refs(folio); + else + folio_clear_referenced(folio); /* * Lazyfree folios are clean anonymous folios. They have * the swapbacked flag cleared, to distinguish them from normal @@ -657,6 +685,9 @@ void deactivate_file_folio(struct folio *folio) if (folio_test_unevictable(folio)) return; + if (lru_gen_enabled() && lru_gen_clear_refs(folio)) + return; + folio_batch_add_and_move(folio, lru_deactivate_file, true); } @@ -670,7 +701,10 @@ void deactivate_file_folio(struct folio *folio) */ void folio_deactivate(struct folio *folio) { - if (folio_test_unevictable(folio) || !(folio_test_active(folio) || lru_gen_enabled())) + if (folio_test_unevictable(folio)) + return; + + if (lru_gen_enabled() ? lru_gen_clear_refs(folio) : !folio_test_active(folio)) return; folio_batch_add_and_move(folio, lru_deactivate, true);