From patchwork Wed Jan 18 00:18:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "T.J. Alumbaugh" X-Patchwork-Id: 13105289 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 404F9C00A5A for ; Wed, 18 Jan 2023 00:18:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E77B56B007E; Tue, 17 Jan 2023 19:18:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DD8EF6B0080; Tue, 17 Jan 2023 19:18:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BB92C6B0082; Tue, 17 Jan 2023 19:18:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 950256B007E for ; Tue, 17 Jan 2023 19:18:51 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 6FBA4804EC for ; Wed, 18 Jan 2023 00:18:51 +0000 (UTC) X-FDA: 80366009262.29.FE683F2 Received: from mail-il1-f201.google.com (mail-il1-f201.google.com [209.85.166.201]) by imf20.hostedemail.com (Postfix) with ESMTP id DA9231C0016 for ; Wed, 18 Jan 2023 00:18:49 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ehaqOT56; spf=pass (imf20.hostedemail.com: domain of 36TrHYwgKCKATALUMBAUGOOGLE.COMLINUX-MMKVACK.ORG@flex--talumbau.bounces.google.com designates 209.85.166.201 as permitted sender) smtp.mailfrom=36TrHYwgKCKATALUMBAUGOOGLE.COMLINUX-MMKVACK.ORG@flex--talumbau.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674001129; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6H89U5IeidH5uN+ug8sUPQpQtuhKWJCjw7/0EqS2ku0=; b=F2mGSXLHUGqriaVt8znHnIK0DxhmU/y1r+S75gwCgNRaLsTJDFwAmLqCqHxdNmAac2p8XP dDf2m7WNmyQIU32A7THRPMYy3CdP4bSRm33g+OO9CXxBqHVMk+puKQaKMorR6hjIW/QMI0 pprF0GwSwv/HZzj13Jh7RGPxRmKH/NM= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ehaqOT56; spf=pass (imf20.hostedemail.com: domain of 36TrHYwgKCKATALUMBAUGOOGLE.COMLINUX-MMKVACK.ORG@flex--talumbau.bounces.google.com designates 209.85.166.201 as permitted sender) smtp.mailfrom=36TrHYwgKCKATALUMBAUGOOGLE.COMLINUX-MMKVACK.ORG@flex--talumbau.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674001129; a=rsa-sha256; cv=none; b=6lU7yfIrp3fxrq6DLnPleoouBpVtn6rIiaRqOBr6SThlD1L8mGkKRvBuPYTU2OfdsOB00Y F1XfLtVVFdZ7uadTzjVVNn8Wf++S/nrOgCVHiPWLrg//2cxrpCUCMJ5wjjnc/ZDhiEM28x 4MaoPtWyIjmPJcqNBWQAPxwxLeFxNK4= Received: by mail-il1-f201.google.com with SMTP id n18-20020a056e02101200b0030f2b79c2ffso860852ilj.20 for ; Tue, 17 Jan 2023 16:18:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6H89U5IeidH5uN+ug8sUPQpQtuhKWJCjw7/0EqS2ku0=; b=ehaqOT566l9lQNTopDpFadeCKqnbnQXDRjA4DPY/fYefKLjsNzYv9N4HuPSWfNkL/T 4Tg/kA81AyBkg4BImcq3oqRmF/8TJybpfdFgXPSLolkWJ5mw6QHUgFCZBkBVQDibpWXw f/lXMaLAbJt0/JF5UEDKskx+8SxTkQBmDQJgygYwaPhtsoZOQXj+1AcNg97s38dmmyVz p2SYvvXrygWzdMYec36TtEcLVug0YvASHX2223Ft7SAt6HO5CKdMQRQazFhMMGSgulaE rAipKhxVwfCNyD+CnFMmGB/6pVkrS4W1S5qqf6Glc6OZDlHbiS2ijfudvQJ3gTUyA8mo KJmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6H89U5IeidH5uN+ug8sUPQpQtuhKWJCjw7/0EqS2ku0=; b=u4/4eKeSMpRqiJk8VfYzyjppiLBgslloobfmpdmoG7MihdLs76VK8agXBxnY8N4ngD iMEZDNfioigf2IX1EWDNvwgGpjsgWgJ/psQioxOhRSLNMEO0eN5cId/GERFjaJN0kSwH Hlw9ihoctUvDGcsbvNd5JcGcQgYZi4L9xER1IzEYC+Q9vNACU9ectaK6tmzhLx3JWUoO +Np+hLZtp/XMCBCZ56kLFL0+QIpwcw/oe33hKakVBhnWa/u1tjh7vhZq3EiShPgmJfER T+4qLktVC0BChtFofuTSwIhyHX3bsRnyYd6N5cv1nx2SK16HRPtS0SlduKhNzGUL4rUU 0QHQ== X-Gm-Message-State: AFqh2kpnuDwWpGyYgI5gR1b3HK1xXV0RQllhw0T32sHq5SmPeu2sIlpD wx4nCR80/yIt/uWeUhDRS7+67bmKoB5JPg== X-Google-Smtp-Source: AMrXdXszXwn79l9Gfb+KYtiDbMPXtgqCNArs+vjslWDauUohP23J2vUQSyakr6tDwA+c7CfabNXqBrhVmzxocQ== X-Received: from talumbau.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:90d]) (user=talumbau job=sendgmr) by 2002:a05:6e02:4a4:b0:30b:e92a:56b with SMTP id e4-20020a056e0204a400b0030be92a056bmr547629ils.53.1674001129175; Tue, 17 Jan 2023 16:18:49 -0800 (PST) Date: Wed, 18 Jan 2023 00:18:27 +0000 In-Reply-To: <20230118001827.1040870-1-talumbau@google.com> Mime-Version: 1.0 References: <20230118001827.1040870-1-talumbau@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230118001827.1040870-8-talumbau@google.com> Subject: [PATCH mm-unstable v1 7/7] mm: multi-gen LRU: simplify lru_gen_look_around() From: "T.J. Alumbaugh" To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-mm@google.com, "T.J. Alumbaugh" X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: DA9231C0016 X-Rspam-User: X-Stat-Signature: ypoi5s365kgth1zojyphqqmim9gamctk X-HE-Tag: 1674001129-535080 X-HE-Meta: U2FsdGVkX18VsIeCfPC2PKaePlmU56dJrRWpXaQFFcS+yY5Is4DxyaGBNPLGeSBhHsAzROGxzb55eRr2e4mDbE+3wx96U/Q7Y1FLgwrjTS0jDFIjMvTnWJPCYkjlnSkbbPzW+rQ3Nne6L0GnNZIJzk3Cf96Ic4ZxuOYGut5Ki0YyMa6GtoSI58XxPZJ4nBH0fRgi51Ax9rZuuycMz3NGEDkZZScQCBcjBao0JD7ktudw/xNzd/OPyaTCuUCzFzTaTF8uEu2Zpokr3IVrM/ysJxr4RKv8s+iGcqkqDL4SaC6Og04VU396ddu/9SXXO/5Rtqw2/UxKhmPkZox800gXsMkexRgVjgbuwkksvhmKPUDmmHh3bWDNctpBc/ELt0QzZwHD2Mdzj/KRFVJSz8S7HXy69L4tAAkUYIK9GQEM0dqqa6uq8O3uyBfFDYR8VT/IAC/lhXei5TM1y80Nz/zjYSjFJwZ2Wyniq3PCpX9N/EJMSnn5ve2vWo2lxJDcTJZ2AGW2ECHq7OedoNxf2dABX2KKWDVxWrsVI4ioj/z97nO4xNrSsx+1PvxpEwLz/GxF00+jJirkFVnPxzS36MdS/ZRsoDMTfMLjowJqWspunEY4fbzVX7CceBwG8uTjvUg0yeofd4SW1OhKnrVEVbeKdy1E2e4drPyk3sKlvLFPkvh+8i8VLZ+TqxzjK32Ai9oz8n2SkSM2Ad/S+qqv+OInuFpBIoWdBGl1WsRyPdGaM3MhMUNQJRyqVYtaac7hBnUlV8RWEe0He9Zua5unQEH02sXahgjEAwhviTE8lJaRtGMX34IgrBdQmsgpn5cfzNYU0X7QHdwiA6M3XfC5PjXIGwEmLC99DdILghAWVjxMqenttsbGvCKgO3B1U4RDeURFesOvZeifFRBRcE0YmrHBNmFKYrnyNE++Ly3jbawgvGv6Pq5Y2UZ+IqdnffLO6Jhy0vxU65LPHhMW4ffizG7 MP0A1Bxd 6mj1CPvBBO5gOgxvqFMiWPMQBYxo055H/Btp6PqSsBF6iCsIOxxNz+NgAlMxIZVZdgzJE/xyzVKhB49u709Apw0ySRDGxjchdrI91BQcBZPytQdhG4s/rtPkY8m/STXeeaMy58dzQEgARsduhgj5+bhh8xscwI6u6612FUDsdD0lVgwyDb1Y+68zT5RXVDubkzdhbQOiVbsb8CG0lBmNVdy5wLJI4sm7md+Wc2B8AngTaaQXNpzwt2HwibA2YulRe35PktOFIIUj7d7AfxsQ7rSdkbEAle+jq1Q3Pq6V+jRN4fcOT+B//wVcbSZK+XfAVgS7KUlFmCkNqO4abSVxudSbutFZFExGHacrIc7DQ9xmpCaRjupYjEpX5q9vh1T+FdHu5a4GbPtVirAqj561Ikp7DK+MVvCEYFmklq4qeZnQYN30= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Update the folio generation in place with or without current->reclaim_state->mm_walk. The LRU lock is held for longer, if mm_walk is NULL and the number of folios to update is more than PAGEVEC_SIZE. This causes a measurable regression from the LRU lock contention during a microbencmark. But a tiny regression is not worth the complexity. Signed-off-by: T.J. Alumbaugh --- mm/vmscan.c | 73 +++++++++++++++++------------------------------------ 1 file changed, 23 insertions(+), 50 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index ff3b4aa3c31f..ac51150d2d36 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4587,13 +4587,12 @@ static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_control *sc) void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) { int i; - pte_t *pte; unsigned long start; unsigned long end; - unsigned long addr; struct lru_gen_mm_walk *walk; int young = 0; - unsigned long bitmap[BITS_TO_LONGS(MIN_LRU_BATCH)] = {}; + pte_t *pte = pvmw->pte; + unsigned long addr = pvmw->address; struct folio *folio = pfn_folio(pvmw->pfn); struct mem_cgroup *memcg = folio_memcg(folio); struct pglist_data *pgdat = folio_pgdat(folio); @@ -4610,25 +4609,28 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) /* avoid taking the LRU lock under the PTL when possible */ walk = current->reclaim_state ? current->reclaim_state->mm_walk : NULL; - start = max(pvmw->address & PMD_MASK, pvmw->vma->vm_start); - end = min(pvmw->address | ~PMD_MASK, pvmw->vma->vm_end - 1) + 1; + start = max(addr & PMD_MASK, pvmw->vma->vm_start); + end = min(addr | ~PMD_MASK, pvmw->vma->vm_end - 1) + 1; if (end - start > MIN_LRU_BATCH * PAGE_SIZE) { - if (pvmw->address - start < MIN_LRU_BATCH * PAGE_SIZE / 2) + if (addr - start < MIN_LRU_BATCH * PAGE_SIZE / 2) end = start + MIN_LRU_BATCH * PAGE_SIZE; - else if (end - pvmw->address < MIN_LRU_BATCH * PAGE_SIZE / 2) + else if (end - addr < MIN_LRU_BATCH * PAGE_SIZE / 2) start = end - MIN_LRU_BATCH * PAGE_SIZE; else { - start = pvmw->address - MIN_LRU_BATCH * PAGE_SIZE / 2; - end = pvmw->address + MIN_LRU_BATCH * PAGE_SIZE / 2; + start = addr - MIN_LRU_BATCH * PAGE_SIZE / 2; + end = addr + MIN_LRU_BATCH * PAGE_SIZE / 2; } } - pte = pvmw->pte - (pvmw->address - start) / PAGE_SIZE; + /* folio_update_gen() requires stable folio_memcg() */ + if (!mem_cgroup_trylock_pages(memcg)) + return; - rcu_read_lock(); arch_enter_lazy_mmu_mode(); + pte -= (addr - start) / PAGE_SIZE; + for (i = 0, addr = start; addr != end; i++, addr += PAGE_SIZE) { unsigned long pfn; @@ -4653,56 +4655,27 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) !folio_test_swapcache(folio))) folio_mark_dirty(folio); + if (walk) { + old_gen = folio_update_gen(folio, new_gen); + if (old_gen >= 0 && old_gen != new_gen) + update_batch_size(walk, folio, old_gen, new_gen); + + continue; + } + old_gen = folio_lru_gen(folio); if (old_gen < 0) folio_set_referenced(folio); else if (old_gen != new_gen) - __set_bit(i, bitmap); + folio_activate(folio); } arch_leave_lazy_mmu_mode(); - rcu_read_unlock(); + mem_cgroup_unlock_pages(); /* feedback from rmap walkers to page table walkers */ if (suitable_to_scan(i, young)) update_bloom_filter(lruvec, max_seq, pvmw->pmd); - - if (!walk && bitmap_weight(bitmap, MIN_LRU_BATCH) < PAGEVEC_SIZE) { - for_each_set_bit(i, bitmap, MIN_LRU_BATCH) { - folio = pfn_folio(pte_pfn(pte[i])); - folio_activate(folio); - } - return; - } - - /* folio_update_gen() requires stable folio_memcg() */ - if (!mem_cgroup_trylock_pages(memcg)) - return; - - if (!walk) { - spin_lock_irq(&lruvec->lru_lock); - new_gen = lru_gen_from_seq(lruvec->lrugen.max_seq); - } - - for_each_set_bit(i, bitmap, MIN_LRU_BATCH) { - folio = pfn_folio(pte_pfn(pte[i])); - if (folio_memcg_rcu(folio) != memcg) - continue; - - old_gen = folio_update_gen(folio, new_gen); - if (old_gen < 0 || old_gen == new_gen) - continue; - - if (walk) - update_batch_size(walk, folio, old_gen, new_gen); - else - lru_gen_update_size(lruvec, folio, old_gen, new_gen); - } - - if (!walk) - spin_unlock_irq(&lruvec->lru_lock); - - mem_cgroup_unlock_pages(); } /******************************************************************************