From patchwork Wed Apr 5 18:54:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13202375 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3107DC7619A for ; Wed, 5 Apr 2023 18:54:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ABBF76B0074; Wed, 5 Apr 2023 14:54:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A6C5C6B0075; Wed, 5 Apr 2023 14:54:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8E6A06B0078; Wed, 5 Apr 2023 14:54:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 7D27B6B0074 for ; Wed, 5 Apr 2023 14:54:34 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 59BE1404A2 for ; Wed, 5 Apr 2023 18:54:34 +0000 (UTC) X-FDA: 80648238468.05.9DB906B Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf22.hostedemail.com (Postfix) with ESMTP id 9A0AAC0003 for ; Wed, 5 Apr 2023 18:54:32 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Ros0vevj; spf=pass (imf22.hostedemail.com: domain of 358MtZAoKCE4E487Eqx2utw44w1u.s421y3AD-220Bqs0.47w@flex--yosryahmed.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=358MtZAoKCE4E487Eqx2utw44w1u.s421y3AD-220Bqs0.47w@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680720872; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oB2sMXwt0v6x8lwXtmYlxh/hoioeozIJIVJcuuQJKWA=; b=yX1nmrVIC3v5yxSmH73o4RhIzmR7OcTc6ra5mc+HIFa2NoqlpsLP155a4fkpCEe8ySqOYw PSrvxkpYukJcMQqnVj5IeJCTDLbl+Aq9UvXyQhnPFXH3dMNk4Fqlm2eZiBk6OPpmaL3Jfm gCZFSE2PFG4ql3A57JTStRDBXaQrQms= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Ros0vevj; spf=pass (imf22.hostedemail.com: domain of 358MtZAoKCE4E487Eqx2utw44w1u.s421y3AD-220Bqs0.47w@flex--yosryahmed.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=358MtZAoKCE4E487Eqx2utw44w1u.s421y3AD-220Bqs0.47w@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680720872; a=rsa-sha256; cv=none; b=XU8BDqykTFjQ1PlU+bCuNx7Olvpcna65cUD6CnROzeMd53uqn9CMj3in4aQvYJcI27cFGU SrL0MdA3W7BuYrzMBcZe+V1nzeRzBh+B0b7hRrsx8iq5yKdnezGtwuPrXo/jJei4xatKfo TWAWV9EcVDCDwfKoeR2yqKmwO8jHCic= Received: by mail-yb1-f202.google.com with SMTP id 192-20020a250dc9000000b00b8bf38f0643so176301ybn.11 for ; Wed, 05 Apr 2023 11:54:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680720871; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=oB2sMXwt0v6x8lwXtmYlxh/hoioeozIJIVJcuuQJKWA=; b=Ros0vevjUiwI3/TjUKg+KCeA5SHnfksHcC4OK3M0Fn4fXpXuUrynI6b8MaR9/c3jSk kYtK4Ske7kxweFphiQTIVIVAbS/Fuv7jZA8Wq1GWNo0oxzQ5Nh5pojz9jDp+do2ph3Jj IztRjzkxwy2vWa3oGo525fcM6H4ZEmqQU6p4rYHtfJsYsBqy2LUHEZvuMUuNIKTo/uqK Xwb0LLWXZaQYVo8lUIEKsrrCdSMMCxNVxpuQ+1xdxcKxOZL6ys2NO41rWyag1l5oLhzt LVX/O/QEV66s0ajoxgyOizIIfKMnIekWa445aOdc74s+Q8ItuYHAsLFXOFgFRUjGTZ12 4y1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680720871; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=oB2sMXwt0v6x8lwXtmYlxh/hoioeozIJIVJcuuQJKWA=; b=qLhdUavNHECLzFd4vh8Fe9QkdsyWAloWgoTAjkOfdDf9pPcy9lRBOUo7/bDZR3scXp teVciQfE7Mdd3BGAB998+vNBM2yHJ73+RotIemPquxkkSq3P0Qke08doglZyc+/bUbMc QL4tMUE9YGV2kqBmtOLV24c7CmlEtrkv4I57Rfep5pDhmFCA+KQa/Ill36DnJrqexnm0 btllN7akL5Xb76Koh42fQfOHNaH4oXpx17T5DgH+PQzlWCTPs2oG5+UOBpiqiYI4HDRh HfBvqh79kBJDCuQBCfd5OvcPi0uP0koVs4d3DeVYbofyuNAbVIKQozRdykXl5onuJsCo Cb9Q== X-Gm-Message-State: AAQBX9fTEM8E21FvVZDqlrwIh5yvz1mMBCZZY+ExRuGJurm+aOTdDXwF FKzwa0I/XbmUm4e00A+5vWHTAegyc0ehzaR+ X-Google-Smtp-Source: AKy350ZCAIete9IDtcJDwB8xbNQZBALRdtx0fEvZ0QLo3PyT+UesbYmBb8Uc0roxQG5ktSucSqhPRJ3viu7kH1ie X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a25:d986:0:b0:b8b:eea7:525c with SMTP id q128-20020a25d986000000b00b8beea7525cmr163875ybg.9.1680720871655; Wed, 05 Apr 2023 11:54:31 -0700 (PDT) Date: Wed, 5 Apr 2023 18:54:26 +0000 In-Reply-To: <20230405185427.1246289-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230405185427.1246289-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230405185427.1246289-2-yosryahmed@google.com> Subject: [PATCH v5 1/2] mm: vmscan: ignore non-LRU-based reclaim in memcg reclaim From: Yosry Ahmed To: Andrew Morton , Alexander Viro , "Darrick J. Wong" , Christoph Lameter , David Rientjes , Joonsoo Kim , Vlastimil Babka , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, "Matthew Wilcox (Oracle)" , Miaohe Lin , David Hildenbrand , Johannes Weiner , Peter Xu , NeilBrown , Shakeel Butt , Michal Hocko , Yu Zhao , Dave Chinner Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-xfs@vger.kernel.org, linux-mm@kvack.org, Yosry Ahmed , stable@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: jfr98epqyxsn34b6fdydhmaf45xqcg4u X-Rspamd-Queue-Id: 9A0AAC0003 X-HE-Tag: 1680720872-194964 X-HE-Meta: U2FsdGVkX1/Q3hU2sdgWufqB070q6O+8dT1yFW1E7/6iGGRJJp9bu7exNv3p9fCDm6LIoAzDJxFsesDSgWH4s3F7b2dAzInUzvXb5okoxJufnVwV/CRog5FWthYVJ1f2J6KeFSCzYcpPOwnDufZrhKcH85GTlTfqDwgkI4Sqov8imT5yEPXUMH2RuKfmHoU+SC2OBwoX08Y2GOOFbFlh+kwtCDb73R5ticwAu9STnO8FlMFoKljqGtwSDohONDyj6pdqOxJVt3lo8mQavu6sB3t5CLJ/1s8rySPr9cThMxZy8DhoyYaAz3H4qbtYob3l7A7jaL+4t8oL78aHBhzEaabCfn5A+cDTr7GTRm5stCA/TCcNq/u3BTQ8AnNTDuWreg1l5oorYEPl8J+eE+15dFpOigqJF9sR38A14Nk47dGiEI8rtjybfLu3SC9pBZZwfDPId6/TsSj7L3gUusPG2vlVus5u5PW/j8K4XeCEYr56nsxZnDBFOF1lVcWg4L3nfGDIsM6UlDLsaFsVMSc16o7SMJsaWfiMEugqVB1zW/dcjIZC1DoBXhx8vmTWn9LGvockxRlomTqkO1Cvp6eW3KV3ekc1rpe6z+xpHMvWEYTfIE/9pIdN/bHoZF9rx/Y2LLnhzOFZsqn3KSvFvppvsTrgnZdyDUUfi8ewLlV2gcxZ8r3/lgKMXAbNsq9RWJFFHUr5XsZHcuGtv0gO68vmGj6AHO9Igr2sAAxLLTNwUlM3tcSsjtqw+b1qvpWkhGx9uJFKCFHhnC0w5J+qWDgQ8sh59xxxMaDhDucUYZ7hPhADwqIYzHO4gzxEZ4uymIbPhEKwyxW3NwUzQnbg4ewzS5+nCCojoixX+jcSf7EP47VOgdiFXG9h8AosJycHtQ4e5biYTZCZTEaahCnhvyeM7uDWjExe35x1FHiMsr/NlEzpFBd9rzARo0uxXTXvR0HzSm1uongqITtkgP2IRW+ aoC5d5Kh AfKJucLQmFMtWEjl+4nxZR1zAoSxoxMtgZCoMbbKfG6PiyJWFELpqtZT0GldRLvBnRgXGLkS1+h6d6d/DZ8wojPyPHTpJIDCgBpoV6mo2Xu3kx3JCyOD52TAEkXINFQwGgCuqk4EBMTKNUEIYrrRVrJav4OH71qMdQLC/iOynnK28HfKQW/cSYfsXvyznI6VFLk4Uufa6CE6XXtqRNAUXk5KibC5VkW47o46xlYF58xVCE+X8EMAWlpRDud0AMcAOHF3jm/fPmfoylJtYWPyX9wHfmMkOp90GE3rY+LUTShvrv1AJZE2teQsGJjGgrjjD5D3dRXRMr4iKW609fogeZTReeJUFqUtzh56kGW81rf8oYLqBGUx3Qkl3/axajxO2f2ONqQOZh9RX+XthNEi1P1RYLfK0OPEL9tM+zSaz/LY0zSpM5ixTNJB5YL0pdkysgctyFnIJ3Im11Bcp1th4qW9A2L3JOvfwYIPXCnhQvqJ36SoVg6AXKUpxZO5i/6HRQ2572XUF8VCucx010qwhdeOw7aGK3+BgmYz+yDFvRtM0d+dRUVP5ZRU5DS4QzMgALGXmC/C2RkwErPW2bJB7335Fi4Uwxisge94e1DX0dIKMEd7IpFxSBy8/yHctF1MPxJS3B17qmFaQ/tIAReGCPlZ7LkUabn3eknLbAriD6zMg3pqg+vkKK5EF39Xrl/bwB5EVXAgsL11N08MOnE3Zbn08N7MUI+rlOZ9jQ6D0SIcmvFok/+mnIh701Y81UzrC432aWWZAwrjF2axUcvtJTaW4uQs8nDMdpovY X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We keep track of different types of reclaimed pages through reclaim_state->reclaimed_slab, and we add them to the reported number of reclaimed pages. For non-memcg reclaim, this makes sense. For memcg reclaim, we have no clue if those pages are charged to the memcg under reclaim. Slab pages are shared by different memcgs, so a freed slab page may have only been partially charged to the memcg under reclaim. The same goes for clean file pages from pruned inodes (on highmem systems) or xfs buffer pages, there is no simple way to currently link them to the memcg under reclaim. Stop reporting those freed pages as reclaimed pages during memcg reclaim. This should make the return value of writing to memory.reclaim, and may help reduce unnecessary reclaim retries during memcg charging. Writing to memory.reclaim on the root memcg is considered as cgroup_reclaim(), but for this case we want to include any freed pages, so use the global_reclaim() check instead of !cgroup_reclaim(). Generally, this should make the return value of try_to_free_mem_cgroup_pages() more accurate. In some limited cases (e.g. freed a slab page that was mostly charged to the memcg under reclaim), the return value of try_to_free_mem_cgroup_pages() can be underestimated, but this should be fine. The freed pages will be uncharged anyway, and we can charge the memcg the next time around as we usually do memcg reclaim in a retry loop. The next patch performs some cleanups around reclaim_state and adds an elaborate comment explaining this to the code. This patch is kept minimal for easy backporting. Signed-off-by: Yosry Ahmed Cc: stable@vger.kernel.org --- global_reclaim(sc) does not exist in kernels before 6.3. It can be replaced with: !cgroup_reclaim(sc) || mem_cgroup_is_root(sc->target_mem_cgroup) --- mm/vmscan.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 9c1c5e8b24b8f..c82bd89f90364 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -5346,8 +5346,10 @@ static int shrink_one(struct lruvec *lruvec, struct scan_control *sc) vmpressure(sc->gfp_mask, memcg, false, sc->nr_scanned - scanned, sc->nr_reclaimed - reclaimed); - sc->nr_reclaimed += current->reclaim_state->reclaimed_slab; - current->reclaim_state->reclaimed_slab = 0; + if (global_reclaim(sc)) { + sc->nr_reclaimed += current->reclaim_state->reclaimed_slab; + current->reclaim_state->reclaimed_slab = 0; + } return success ? MEMCG_LRU_YOUNG : 0; } @@ -6472,7 +6474,7 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc) shrink_node_memcgs(pgdat, sc); - if (reclaim_state) { + if (reclaim_state && global_reclaim(sc)) { sc->nr_reclaimed += reclaim_state->reclaimed_slab; reclaim_state->reclaimed_slab = 0; } From patchwork Wed Apr 5 18:54:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13202376 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E3FDC77B6C for ; Wed, 5 Apr 2023 18:54:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BA70E6B0075; Wed, 5 Apr 2023 14:54:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B58AB6B0078; Wed, 5 Apr 2023 14:54:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9AA276B007B; Wed, 5 Apr 2023 14:54:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 8A0196B0075 for ; Wed, 5 Apr 2023 14:54:36 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 248821C6B04 for ; Wed, 5 Apr 2023 18:54:36 +0000 (UTC) X-FDA: 80648238552.21.4339ED1 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) by imf05.hostedemail.com (Postfix) with ESMTP id 42482100011 for ; Wed, 5 Apr 2023 18:54:34 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=TF78184W; spf=pass (imf05.hostedemail.com: domain of 36cMtZAoKCFAG6A9Gsz4wvy66y3w.u64305CF-442Dsu2.69y@flex--yosryahmed.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=36cMtZAoKCFAG6A9Gsz4wvy66y3w.u64305CF-442Dsu2.69y@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680720874; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4yIEsCxGxp+Goi3FcKFGmOxdWE8VaKQI4GXE/OXZVyU=; b=vSukebkeevmBjQQQAtfjGdv3QqroXkCWc01CfsD9Y0UT+WoHUzVE+0rkm/sDheRNNCHsxy Z6kOnQzGG1erTOfR/XY7io7P6BPybARZTGH98zmqbFe/oOp1sxKnos+a9r7kQpvhwfFibA NlVyoTOUo5gsD5VUgHCnHq165XYqIrw= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=TF78184W; spf=pass (imf05.hostedemail.com: domain of 36cMtZAoKCFAG6A9Gsz4wvy66y3w.u64305CF-442Dsu2.69y@flex--yosryahmed.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=36cMtZAoKCFAG6A9Gsz4wvy66y3w.u64305CF-442Dsu2.69y@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680720874; a=rsa-sha256; cv=none; b=1k62J0fRzMgL4UYPGUPdU+ZrV5FvQ9VB4bPWc3cKeTBDsGgnkFB0sbgmhCbXbJpjgq9AwP hc24E6+Yi4qot4NZyo0YI3vRyOWf5be+rffNSbj1ZzG498RMuMAGgzQRWMS8KWJHr9mEXz zQ2HYWlFqctrn3fFGDJ83HP/8tjHkPI= Received: by mail-pf1-f202.google.com with SMTP id c2-20020a62f842000000b0062d93664ad5so12555220pfm.19 for ; Wed, 05 Apr 2023 11:54:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680720873; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=4yIEsCxGxp+Goi3FcKFGmOxdWE8VaKQI4GXE/OXZVyU=; b=TF78184WybJytfdYm4GCjpWzSx4mUisBexI8bJfnyd6K4JeOX369L2ClHVn4Ys8I7V JV+y1m7UAB4SCcuHTJGFIgQF6+zPgKdblexEZW4hF8aKIUP946k53euPlWdSC0diJGII +4BQZqOlikwwOc1ffTqbrRHhEVpr3wXhWZ0WF/Lu4nUjgmKclpwmru3gluXbSweQuK7W C3JM5wV7WTynZX48f/Lp4lcdapj5Vg97KMACQe2OSnlckxh/DO5uCHLUHH1nrwiXFUnK JKlKgk5qWcBAj572MmVExUzKlzT1AeS/W+Ao4ZB3tIG21TaLoaVVgW0UNpmmTPc769jm okiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680720873; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4yIEsCxGxp+Goi3FcKFGmOxdWE8VaKQI4GXE/OXZVyU=; b=l8QpUSK/T2pssQHIxDPZhJ+vbmZCZVGTvEQTCPmzB9SgU8t6CrKEeiPzWBACYTUFZf ph079o6rjX7+e0t2uVCiIHClzArXNpUcs0AahA0CSXq7x41MMcvS5PFuyBuRhMsPtiAG fDexvz2gGT1V0cAknEAOdZ/1XWyKrGb+Di1OjAahvkrFckRzbfEHiB9Mpx3W4wpuc9tL usVZBE/gYuTNiyqpaYUPa3jzG5CeLZfd8OcP5hhdQGF/AWZSeCWnsxwwWtp2MjVQOK+G Ndb9HsKXxQvGYIFsKKecJzBsE/22CcOv/Kz3Is/FV49huEJUUmjc9W+18juq39d76xkQ dg6Q== X-Gm-Message-State: AAQBX9cSdcNpXPDcU0o63RVJZavrc0k/6hnEmaHGQYOf9Jua0XkamchK Jaa6BBbTCyKeYhq22Z8FW5r0OfmaAhH4F6Hw X-Google-Smtp-Source: AKy350acXJNtGWroYsn6aFMZG+1teLm2kBpgOPT0A7iZ6zD9rHb+p4s9om4KF91yo2bhBQ0j07DSjkYPZ7RoOPdC X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a05:6a00:2da6:b0:625:ccea:1627 with SMTP id fb38-20020a056a002da600b00625ccea1627mr3896876pfb.5.1680720873145; Wed, 05 Apr 2023 11:54:33 -0700 (PDT) Date: Wed, 5 Apr 2023 18:54:27 +0000 In-Reply-To: <20230405185427.1246289-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230405185427.1246289-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230405185427.1246289-3-yosryahmed@google.com> Subject: [PATCH v5 2/2] mm: vmscan: refactor reclaim_state helpers From: Yosry Ahmed To: Andrew Morton , Alexander Viro , "Darrick J. Wong" , Christoph Lameter , David Rientjes , Joonsoo Kim , Vlastimil Babka , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, "Matthew Wilcox (Oracle)" , Miaohe Lin , David Hildenbrand , Johannes Weiner , Peter Xu , NeilBrown , Shakeel Butt , Michal Hocko , Yu Zhao , Dave Chinner Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-xfs@vger.kernel.org, linux-mm@kvack.org, Yosry Ahmed X-Rspamd-Queue-Id: 42482100011 X-Stat-Signature: tdn5erwuowehfwmc3aduxme7u5bqb9ab X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1680720874-820682 X-HE-Meta: U2FsdGVkX1/4hmuKkO1BFxr1lCT+ieDDIzQfrpTBGDqHVoY4jikSDjG5cX5WIVGwnBVZ1kDhOXK6/rL6Xjet8zCkyM2YkR4fVsXuDiwPfB/72Z3A2g1+JXWeuRq5fP6LHgvjC6UPzQej3iHtrxn2MJV4mppQ8w8xDyQQ+89A2pSu4xYzPiRsywyWlnbx3u8XpaWf63gPbjke/xrLv36swBQrLuoUtbiOqdbIHItsD/7ezxweWzKH5iRazsZVsehf88pLTXG2ersNDwrU0CEzT4fimWUODRxf6krk7c60hSKe1RMy6KQ2p7hjNBwQoA1NAGRVtAuwTprT7j1j0uGJhVKgk5lmp9iw/0FAyOBdwqYbh5j+W8aWTZWMyK7ptEHCf7DM+Ti2B2eRemrr2yfdWc3gWJ/RaRIa09tMygHIvnlDCrtYPwqGqIZTeahn/eREZgGk7li2yGdtkycZWv04eGcvvrjZm089aOF0pNE9WIU5r5K0VKFCeYz/CZFRP3as2LwaE7a79aOo6IGahWw/C2X9FZjjeZohaj4uMeoU9b1UtDMQTdJXxh6mD4WP5O6glSVFRd62ZpUtHsFEm/rfgdtjutvUtERfJPONH7IeB81sH3baEr0pmZOQFTZI8SZVlADS04yKS4gTB8w3mrcwunBLlQvcygFBMpBh8cNT8Dn0xy3yxUny8Zyp5nXah5BRvOgHIw3sdcp26YZwk98WU5YR98Y35oYHY+UaKnnVcf/UOWT8l/nZzfKL8S8HKYBfz1oZofr60TiiE3OD6L6K7iFXgceRtXUhS8uyOI+CpnpgHjk5urKeXISQtV6x39YKbqgmyBkYDR9baI7qV9mAbRQoloR9nTKUCRsfkZCntf1pQQeXVUb6pXLBbMR0q2UL5duLWGQ/5N8sEZs/tPTANa9nJWnK8QClhwvA9nFozgA4ljkmaj+HZSoKDryXHvH1oPzgfEx+eBZ6gapxSrt rELNAt2G L4KNekijuJdvexOrGPSdKWKlcKhJoRnOgCjw9+Xqvnpsl0H7ECJodW9kWVJALWoVjIuVwvZHXvIqaARh1Rc11FHsXkqrxGVmOYLvhnmGILtrE/3sWBPRVUAoIL7RDm4g9o9S2aK7Vtdeo3d7oA11M2P6Pl3e8AhjCyLdQBIxQmMpB6v64VuqMqMly3HIXKwDo9tSx1KACZ3MzhMuFgjVzowwTs1EXjT08xC7oPZMnApp08ZwTHqZ02oWlwuRa9X0InIBngzSg2Na7OBtTkBGtL0RGbJb34Lp5hb405f3/nLjWzvKnS49HL7UG7QJOeCFWHjXZ9q2Aq74mIHZA/pC3OChORPYjpwB0QyIH6ZvuZpUT173Nt0KqmO6qzoLFJNMww8jy51UKmkBmN3ahjTRL08F60A9aEfUJL2CcV2n2YY20Za3xuYbDJUuE2ACCIv4oNxKxqJZ7CE7MNVyIvWhsXgXJPVEdJB1HYmnVpaKW3K6MTE2fDz/fOyPvztmW6R72mtjjWCMbPDxdtjHYAO/1P5/Sx8+ysblJs5gDLBAfVnRMsMG/mExa6Xk5kTWmCvRbDk/30oEplEVaS2Sjyypc3KieMvamXuhXEz8qDWOYW+bx6DyLXuMQRtCo4moGbw+qiOOrHUOSgLDnqI9bTesHZYMZVQ64YAz2EJ8HWmWSZ1QyfzyUyiFJi+uAPHbrYdC1w0xWAb2Wet5ZvQnBIcOfovP11SkjcNSGEZpxvIqCsBZrL2wzzHFMgYgJqkc3SxVs0jqsKCTyRqHELoEUZT5y2cJ26Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: During reclaim, we keep track of pages reclaimed from other means than LRU-based reclaim through scan_control->reclaim_state->reclaimed_slab, which we stash a pointer to in current task_struct. However, we keep track of more than just reclaimed slab pages through this. We also use it for clean file pages dropped through pruned inodes, and xfs buffer pages freed. Rename reclaimed_slab to reclaimed, and add a helper function that wraps updating it through current, so that future changes to this logic are contained within mm/vmscan.c. Additionally, add a flush_reclaim_state() helper to wrap using reclaim_state->reclaimed to updated sc->nr_reclaimed, and use that helper to add an elaborate comment about why we only do the update for global reclaim. Finally, move set_task_reclaim_state() next to flush_reclaim_state() so that all reclaim_state helpers are in close proximity for easier readability. Signed-off-by: Yosry Ahmed --- fs/inode.c | 3 +- fs/xfs/xfs_buf.c | 3 +- include/linux/swap.h | 17 +++++++++- mm/slab.c | 3 +- mm/slob.c | 6 ++-- mm/slub.c | 5 ++- mm/vmscan.c | 75 ++++++++++++++++++++++++++++++++------------ 7 files changed, 78 insertions(+), 34 deletions(-) diff --git a/fs/inode.c b/fs/inode.c index 4558dc2f13557..e60fcc41faf17 100644 --- a/fs/inode.c +++ b/fs/inode.c @@ -864,8 +864,7 @@ static enum lru_status inode_lru_isolate(struct list_head *item, __count_vm_events(KSWAPD_INODESTEAL, reap); else __count_vm_events(PGINODESTEAL, reap); - if (current->reclaim_state) - current->reclaim_state->reclaimed_slab += reap; + mm_account_reclaimed_pages(reap); } iput(inode); spin_lock(lru_lock); diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c index 54c774af6e1c6..15d1e5a7c2d34 100644 --- a/fs/xfs/xfs_buf.c +++ b/fs/xfs/xfs_buf.c @@ -286,8 +286,7 @@ xfs_buf_free_pages( if (bp->b_pages[i]) __free_page(bp->b_pages[i]); } - if (current->reclaim_state) - current->reclaim_state->reclaimed_slab += bp->b_page_count; + mm_account_reclaimed_pages(bp->b_page_count); if (bp->b_pages != bp->b_page_array) kmem_free(bp->b_pages); diff --git a/include/linux/swap.h b/include/linux/swap.h index 209a425739a9f..e131ac155fb95 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -153,13 +153,28 @@ union swap_header { * memory reclaim */ struct reclaim_state { - unsigned long reclaimed_slab; + /* pages reclaimed outside of LRU-based reclaim */ + unsigned long reclaimed; #ifdef CONFIG_LRU_GEN /* per-thread mm walk data */ struct lru_gen_mm_walk *mm_walk; #endif }; +/* + * mm_account_reclaimed_pages(): account reclaimed pages outside of LRU-based + * reclaim + * @pages: number of pages reclaimed + * + * If the current process is undergoing a reclaim operation, increment the + * number of reclaimed pages by @pages. + */ +static inline void mm_account_reclaimed_pages(unsigned long pages) +{ + if (current->reclaim_state) + current->reclaim_state->reclaimed += pages; +} + #ifdef __KERNEL__ struct address_space; diff --git a/mm/slab.c b/mm/slab.c index dabc2a671fc6f..64bf1de817b24 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1392,8 +1392,7 @@ static void kmem_freepages(struct kmem_cache *cachep, struct slab *slab) smp_wmb(); __folio_clear_slab(folio); - if (current->reclaim_state) - current->reclaim_state->reclaimed_slab += 1 << order; + mm_account_reclaimed_pages(1 << order); unaccount_slab(slab, order, cachep); __free_pages(&folio->page, order); } diff --git a/mm/slob.c b/mm/slob.c index fe567fcfa3a39..79cc8680c973c 100644 --- a/mm/slob.c +++ b/mm/slob.c @@ -61,7 +61,7 @@ #include #include -#include /* struct reclaim_state */ +#include /* mm_account_reclaimed_pages() */ #include #include #include @@ -211,9 +211,7 @@ static void slob_free_pages(void *b, int order) { struct page *sp = virt_to_page(b); - if (current->reclaim_state) - current->reclaim_state->reclaimed_slab += 1 << order; - + mm_account_reclaimed_pages(1 << order); mod_node_page_state(page_pgdat(sp), NR_SLAB_UNRECLAIMABLE_B, -(PAGE_SIZE << order)); __free_pages(sp, order); diff --git a/mm/slub.c b/mm/slub.c index 39327e98fce34..7aa30eef82350 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -11,7 +11,7 @@ */ #include -#include /* struct reclaim_state */ +#include /* mm_account_reclaimed_pages() */ #include #include #include @@ -2063,8 +2063,7 @@ static void __free_slab(struct kmem_cache *s, struct slab *slab) /* Make the mapping reset visible before clearing the flag */ smp_wmb(); __folio_clear_slab(folio); - if (current->reclaim_state) - current->reclaim_state->reclaimed_slab += pages; + mm_account_reclaimed_pages(pages); unaccount_slab(slab, order, s); __free_pages(&folio->page, order); } diff --git a/mm/vmscan.c b/mm/vmscan.c index c82bd89f90364..049e39202e6ce 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -188,18 +188,6 @@ struct scan_control { */ int vm_swappiness = 60; -static void set_task_reclaim_state(struct task_struct *task, - struct reclaim_state *rs) -{ - /* Check for an overwrite */ - WARN_ON_ONCE(rs && task->reclaim_state); - - /* Check for the nulling of an already-nulled member */ - WARN_ON_ONCE(!rs && !task->reclaim_state); - - task->reclaim_state = rs; -} - LIST_HEAD(shrinker_list); DECLARE_RWSEM(shrinker_rwsem); @@ -511,6 +499,59 @@ static bool writeback_throttling_sane(struct scan_control *sc) } #endif +static void set_task_reclaim_state(struct task_struct *task, + struct reclaim_state *rs) +{ + /* Check for an overwrite */ + WARN_ON_ONCE(rs && task->reclaim_state); + + /* Check for the nulling of an already-nulled member */ + WARN_ON_ONCE(!rs && !task->reclaim_state); + + task->reclaim_state = rs; +} + +/* + * flush_reclaim_state(): add pages reclaimed outside of LRU-based reclaim to + * scan_control->nr_reclaimed. + */ +static void flush_reclaim_state(struct scan_control *sc, + struct reclaim_state *rs) +{ + /* + * Currently, reclaim_state->reclaimed includes three types of pages + * freed outside of vmscan: + * (1) Slab pages. + * (2) Clean file pages from pruned inodes. + * (3) XFS freed buffer pages. + * + * For all of these cases, we have no way of finding out whether these + * pages were related to the memcg under reclaim. For example, a freed + * slab page could have had only a single object charged to the memcg + * under reclaim. Also, populated inodes are not on shrinker LRUs + * anymore except on highmem systems. + * + * Instead of over-reporting the reclaimed pages in a memcg reclaim, + * only count such pages in global reclaim. This prevents unnecessary + * retries during memcg charging and false positive from proactive + * reclaim (memory.reclaim). + * + * For uncommon cases were the freed pages were actually significantly + * charged to the memcg under reclaim, and we end up under-reporting, it + * should be fine. The freed pages will be uncharged anyway, even if + * they are not reported properly, and we will be able to make forward + * progress in charging (which is usually in a retry loop). + * + * We can go one step further, and report the uncharged objcg pages in + * memcg reclaim, to make reporting more accurate and reduce + * under-reporting, but it's probably not worth the complexity for now. + */ + if (rs && global_reclaim(sc)) { + sc->nr_reclaimed += rs->reclaimed; + rs->reclaimed = 0; + } +} + static long xchg_nr_deferred(struct shrinker *shrinker, struct shrink_control *sc) { @@ -5346,10 +5387,7 @@ static int shrink_one(struct lruvec *lruvec, struct scan_control *sc) vmpressure(sc->gfp_mask, memcg, false, sc->nr_scanned - scanned, sc->nr_reclaimed - reclaimed); - if (global_reclaim(sc)) { - sc->nr_reclaimed += current->reclaim_state->reclaimed_slab; - current->reclaim_state->reclaimed_slab = 0; - } + flush_reclaim_state(sc, current->reclaim_state); return success ? MEMCG_LRU_YOUNG : 0; } @@ -6474,10 +6512,7 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc) shrink_node_memcgs(pgdat, sc); - if (reclaim_state && global_reclaim(sc)) { - sc->nr_reclaimed += reclaim_state->reclaimed_slab; - reclaim_state->reclaimed_slab = 0; - } + flush_reclaim_state(sc, reclaim_state); /* Record the subtree's reclaim efficiency */ if (!sc->proactive)