From patchwork Wed Oct 5 17:37:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 12999511 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49C5EC433FE for ; Wed, 5 Oct 2022 17:37:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CA41C6B0072; Wed, 5 Oct 2022 13:37:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C538D6B0073; Wed, 5 Oct 2022 13:37:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AF4838E0001; Wed, 5 Oct 2022 13:37:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 90EFB6B0072 for ; Wed, 5 Oct 2022 13:37:17 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 4C362C0551 for ; Wed, 5 Oct 2022 17:37:17 +0000 (UTC) X-FDA: 79987602114.15.AD7E5A3 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf10.hostedemail.com (Postfix) with ESMTP id E0513C000B for ; Wed, 5 Oct 2022 17:37:16 +0000 (UTC) Received: by mail-pj1-f73.google.com with SMTP id o15-20020a17090a0a0f00b0020ae208febfso1227166pjo.5 for ; Wed, 05 Oct 2022 10:37:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date; bh=rlW8penUCTiIbLOsfrslJ/J/g+TenjXX9VkYuhroyUE=; b=cre+57nEzmahI/Jg40WpCpvVTDra/Np6hwgfHPiHb9kvUAEW6v66vCHfRJHy69FvkT kSZx/SqkG8Qpv8Hq2If8J6iuXxv/mc6j5C8Yoglkw1Ei7M4l2S0YQIltO+5M1vw3CkOc 6nlZ2fg7N4ebt+lgDXLPSBDhdUmChoewoKY66ezVqioZk/qfXiRg5Y2WgDulfrBlU7/2 27z2NsHcZ6oyTLUaEXcRfL5qfW8C7g0sATkpk9bURP0kN49QvJMqwD9Ym3G/rL+iUGpW Le2W9Wf5i7EgN5mTB26OTqxUSYbfvjf6BpxPtRd14y0xdfXzkV3xQ1GOiOYTmEPwy0In 6Qaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date; bh=rlW8penUCTiIbLOsfrslJ/J/g+TenjXX9VkYuhroyUE=; b=gv5SHbvgiy3WgIw8XHn5f4YHs80Lc4xB0ZdrkgzK+i6F79pRwbgMeTOWfoqxhWqKxU CX0RVfNTdqQwT5XaYXOG+PlJR++9qSnsk13RKRfPdIOhUsUq4T+Ru01aMWyvB/MRroIi rlzuFzNDSHFqKxfdmdNzG5X6pezEpigboeSE7jC7HdJMT22WEP4X6osm8e2eYsInZxr7 l/lJDj6i+q9if4UgBCpbuoAlozzTxgSpuN2IVFPVTWy/+7/C6vBPVRTrX4qtxPevYe35 dNvuY7JZLm62wVFm8i7fxm8SIulLwFP8+m/LDbugmDv/ktHcgRe+fbARB4owyCIVVwcy LNzw== X-Gm-Message-State: ACrzQf1RgVHDhVBTVKrhr37FpPNSpfYuAhbMW/C1NU/DTQiLnxspqfZB 6Gk4z/h40nrSDvblg09Vihl2RkyOXYKVa2fm X-Google-Smtp-Source: AMsMyM4O1bI7xHUu+DyS7ikNiTdgZOK+8YGSqLRq/aLCcEuTBve3ylHNTvqqAAIgShF3oCOy/2xW86pG+uk7n7nQ X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a17:902:f707:b0:17f:8541:c04b with SMTP id h7-20020a170902f70700b0017f8541c04bmr484894plo.98.1664991435828; Wed, 05 Oct 2022 10:37:15 -0700 (PDT) Date: Wed, 5 Oct 2022 17:37:13 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.38.0.rc1.362.ged0d419d3c-goog Message-ID: <20221005173713.1308832-1-yosryahmed@google.com> Subject: [PATCH v2] mm/vmscan: check references from all memcgs for swapbacked memory From: Yosry Ahmed To: Andrew Morton , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song Cc: Greg Thelen , David Rientjes , cgroups@vger.kernel.org, linux-mm@kvack.org, Yosry Ahmed ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1664991437; a=rsa-sha256; cv=none; b=mL1Jf9cTz6z9+ye9lk29rod0U7r66aPSDuUH/IhNbIpTXT3faViLhSalqZxMnJ48ZfSmT8 40rZYxSsVbDgLAfjtHe6hiiYPaFU0dZTPi63cuLamR4mX7akQRGJ5dYacL2opw044cWomA u9ozjJYhRXiQOQoUzcIrRPlEbJ7v/Wk= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=cre+57nE; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of 3y8A9YwoKCGQaQUTaCJOGFIQQING.EQONKPWZ-OOMXCEM.QTI@flex--yosryahmed.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3y8A9YwoKCGQaQUTaCJOGFIQQING.EQONKPWZ-OOMXCEM.QTI@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1664991437; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=rlW8penUCTiIbLOsfrslJ/J/g+TenjXX9VkYuhroyUE=; b=7OkCSYYqXm8bU58C9bmebqu+k8+NF3j4wjJk6SYojnyGCA4xi3Hoy4MYPx2xKY6zkJsJqq NbEsUixtg8n3LA66AbNoNAfgEn2USps0j6tV758e/vCkkyshM3Otc5X1e25A8bXnXePiBV lIbpRoRfl3/VmXg913M+W6t5cA9yGIg= X-Rspam-User: Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=cre+57nE; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of 3y8A9YwoKCGQaQUTaCJOGFIQQING.EQONKPWZ-OOMXCEM.QTI@flex--yosryahmed.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3y8A9YwoKCGQaQUTaCJOGFIQQING.EQONKPWZ-OOMXCEM.QTI@flex--yosryahmed.bounces.google.com X-Stat-Signature: 7gi3a34jw6nea1ps7fy4sj35nso7br7n X-Rspamd-Queue-Id: E0513C000B X-Rspamd-Server: rspam09 X-HE-Tag: 1664991436-538958 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: During page/folio reclaim, we check if a folio is referenced using folio_referenced() to avoid reclaiming folios that have been recently accessed (hot memory). The rationale is that this memory is likely to be accessed soon, and hence reclaiming it will cause a refault. For memcg reclaim, we currently only check accesses to the folio from processes in the subtree of the target memcg. This behavior was originally introduced by commit bed7161a519a ("Memory controller: make page_referenced() cgroup aware") a long time ago. Back then, refaulted pages would get charged to the memcg of the process that was faulting them in. It made sense to only consider accesses coming from processes in the subtree of target_mem_cgroup. If a page was charged to memcg A but only being accessed by a sibling memcg B, we would reclaim it if memcg A is is the reclaim target. memcg B can then fault it back in and get charged for it appropriately. Today, this behavior still makes sense for file pages. However, unlike file pages, when swapbacked pages are refaulted they are charged to the memcg that was originally charged for them during swapping out. Which means that if a swapbacked page is charged to memcg A but only used by memcg B, and we reclaim it from memcg A, it would simply be faulted back in and charged again to memcg A once memcg B accesses it. In that sense, accesses from all memcgs matter equally when considering if a swapbacked page/folio is a viable reclaim target. Modify folio_referenced() to always consider accesses from all memcgs if the folio is swapbacked. Signed-off-by: Yosry Ahmed Acked-by: Johannes Weiner --- v1 -> v2: - Move the folio_test_swapbacked() check inside folio_referenced() (Johannes). - Slight rephrasing of the commit log and comment to make them clearer. - Renamed memcg argument to folio_referenced() to target_memcg. --- include/linux/rmap.h | 2 +- mm/rmap.c | 35 +++++++++++++++++++++++++++-------- 2 files changed, 28 insertions(+), 9 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index ca3e4ba6c58c..8130586eb637 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -352,7 +352,7 @@ static inline int page_try_share_anon_rmap(struct page *page) * Called from mm/vmscan.c to handle paging out */ int folio_referenced(struct folio *, int is_locked, - struct mem_cgroup *memcg, unsigned long *vm_flags); + struct mem_cgroup *target_memcg, unsigned long *vm_flags); void try_to_migrate(struct folio *folio, enum ttu_flags flags); void try_to_unmap(struct folio *, enum ttu_flags flags); diff --git a/mm/rmap.c b/mm/rmap.c index b6743c2b8b5f..7c98d0ca7cc6 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -882,7 +882,7 @@ static bool invalid_folio_referenced_vma(struct vm_area_struct *vma, void *arg) * folio_referenced() - Test if the folio was referenced. * @folio: The folio to test. * @is_locked: Caller holds lock on the folio. - * @memcg: target memory cgroup + * @target_memcg: target memory cgroup of reclaim. * @vm_flags: A combination of all the vma->vm_flags which referenced the folio. * * Quick test_and_clear_referenced for all mappings of a folio, @@ -891,12 +891,12 @@ static bool invalid_folio_referenced_vma(struct vm_area_struct *vma, void *arg) * the function bailed out due to rmap lock contention. */ int folio_referenced(struct folio *folio, int is_locked, - struct mem_cgroup *memcg, unsigned long *vm_flags) + struct mem_cgroup *target_memcg, unsigned long *vm_flags) { int we_locked = 0; struct folio_referenced_arg pra = { .mapcount = folio_mapcount(folio), - .memcg = memcg, + .memcg = target_memcg, }; struct rmap_walk_control rwc = { .rmap_one = folio_referenced_one, @@ -919,13 +919,32 @@ int folio_referenced(struct folio *folio, int is_locked, } /* - * If we are reclaiming on behalf of a cgroup, skip - * counting on behalf of references from different - * cgroups + * We check references to folios to make sure we don't reclaim hot + * folios that are likely to be refaulted soon. If we are reclaiming + * memory on behalf of a memcg, we may want to skip references from + * processes outside the target memcg's subtree. + * + * For file folios, we only consider references from processes in the + * subtree of the target memcg. If memcg A is under reclaim, and a + * folio is charged to memcg A but only referenced by processes in + * memcg B, we ignore references from memcg B and try to reclaim it. + * If it is later accessed by memcg B it will be faulted back in and + * charged appropriately to memcg B. For memcg A, this is cold memory + * that should be reclaimed. + * + * On the other hand, when swapbacked folios are faulted in, they get + * charged to the memcg that was originally charged for them at the time + * of swapping out. This means that if a folio that is charged to + * memcg A gets swapped out, it will get charged back to A when any + * process in any memcg accesses it. In that sense, we need to consider + * references from all memcgs when considering whether to reclaim a + * swapbacked folio. + * + * Hence, only skip references from outside the target memcg (if any) if + * the folio is not swapbacked. */ - if (memcg) { + if (target_memcg && !folio_test_swapbacked(folio)) rwc.invalid_vma = invalid_folio_referenced_vma; - } rmap_walk(folio, &rwc); *vm_flags = pra.vm_flags;