From patchwork Thu Apr 14 18:06:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zach O'Keefe X-Patchwork-Id: 12813849 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C759C433EF for ; Thu, 14 Apr 2022 18:06:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2457C6B0082; Thu, 14 Apr 2022 14:06:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1E6716B0085; Thu, 14 Apr 2022 14:06:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F052C6B0082; Thu, 14 Apr 2022 14:06:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.26]) by kanga.kvack.org (Postfix) with ESMTP id E1BB06B0080 for ; Thu, 14 Apr 2022 14:06:50 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id 4FC41834A9 for ; Thu, 14 Apr 2022 18:06:49 +0000 (UTC) X-FDA: 79356265338.18.867038E Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf25.hostedemail.com (Postfix) with ESMTP id A7D4AA000D for ; Thu, 14 Apr 2022 18:06:48 +0000 (UTC) Received: by mail-pj1-f74.google.com with SMTP id v14-20020a17090a0c8e00b001cb778cc439so3215469pja.3 for ; Thu, 14 Apr 2022 11:06:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=TSyQ00cmc2Sqw2+m7G2pC8yk3etzd3mGCxmpa7uLA/c=; b=S21sub5WAGAm1ijMOnsvbppyFOUubRLWWqL1lFVynKqX4tjwrDuya1HZElexqTL7p9 gosoZx/Xdi/fwkCGGkifaaVbiK3MhxMJvm0rNGQouHA2xeGlOq1X3F1TDQx92DvXDq+V 3qBeOmLWKFJd58tpnEd3WYFTFt5oMrneog4eerlY4yBNUG59+23ZfHxz9YiLnekcDhmS T31moboexSxgXf8ZGrnYgl0LGD+J14JImArUzUCaEfmC0DxzCt0TLTGcQ4EA9Ns8/5df 1JR6bGd4AXUmfGide8+exM7IBS0pUFrWhBNT4p01yERaAJyU4GC/RSc+qZq/lsfdb2lW ihTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=TSyQ00cmc2Sqw2+m7G2pC8yk3etzd3mGCxmpa7uLA/c=; b=GZe0sBk9M0n1VhN+Clmh6qGzRqJIqH5bF5uv/07vn0FJpWVccs2ZuHnbDC7R1r0fh5 MrsoWMpwVzcqMlzXRBbk43LiQm8DKYFCCnCgpHwBOmZP/09xPL3PB17ftubq5oQT/zZL qtFW5DZRrKXogw/eosaKGWJcwF4nCLVlxTO3SnLh29Eh5TaT++cPHl8aD63XGYJpiks5 3HpxHIw+BpiQxGkzZMFmKew/vja5XZ64o/+fqUteEa/C3Gw+F8GDsJnPlP4evZTCOz/N dyyfAsuWXi9EBM4ImcHffAmYhRu73zr4kxzDjEZxM6uWkv7w5eY8vk4eqjHFJLSXGKHx St6w== X-Gm-Message-State: AOAM531pxGMDAMG/SjXO+IdAQ6m69ELpbRLRLdkyEHS5x4ZyI9zkSPak ObennPBUapzfLOoN55/BwxgLf/WBocKN X-Google-Smtp-Source: ABdhPJycg04QNpoRfyq2QO82oPFG7IIyjQoP4/R0tNE2T6GG0vSniPRTxR+YRAWXH8pxz/FIbmzAi1iRC61C X-Received: from zokeefe3.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:1b6]) (user=zokeefe job=sendgmr) by 2002:a05:6a00:894:b0:4fe:25d7:f59e with SMTP id q20-20020a056a00089400b004fe25d7f59emr5174895pfj.58.1649959607680; Thu, 14 Apr 2022 11:06:47 -0700 (PDT) Date: Thu, 14 Apr 2022 11:06:08 -0700 In-Reply-To: <20220414180612.3844426-1-zokeefe@google.com> Message-Id: <20220414180612.3844426-9-zokeefe@google.com> Mime-Version: 1.0 References: <20220414180612.3844426-1-zokeefe@google.com> X-Mailer: git-send-email 2.36.0.rc0.470.gd361397f0d-goog Subject: [PATCH v2 08/12] mm/khugepaged: add flag to ignore page young/referenced requirement From: "Zach O'Keefe" To: Alex Shi , David Hildenbrand , David Rientjes , Matthew Wilcox , Michal Hocko , Pasha Tatashin , SeongJae Park , Song Liu , Vlastimil Babka , Yang Shi , Zi Yan , linux-mm@kvack.org Cc: Andrea Arcangeli , Andrew Morton , Arnd Bergmann , Axel Rasmussen , Chris Kennelly , Chris Zankel , Helge Deller , Hugh Dickins , Ivan Kokshaysky , "James E.J. Bottomley" , Jens Axboe , "Kirill A. Shutemov" , Matt Turner , Max Filippov , Miaohe Lin , Minchan Kim , Patrick Xia , Pavel Begunkov , Peter Xu , Thomas Bogendoerfer , "Zach O'Keefe" Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=S21sub5W; spf=pass (imf25.hostedemail.com: domain of 3t2JYYgcKCPYxmiccdcemmejc.amkjglsv-kkitYai.mpe@flex--zokeefe.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3t2JYYgcKCPYxmiccdcemmejc.amkjglsv-kkitYai.mpe@flex--zokeefe.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Stat-Signature: ugcsmr81uo7i46doccjg55jnidreagxy X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: A7D4AA000D X-HE-Tag: 1649959608-798642 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add enforce_young flag to struct collapse_control that allows context to ignore requirement that some pages in region being collapsed be young or referenced. Set this flag in khugepaged collapse context to preserve existing khugepaged behavior and unset the flag in madvise collapse context since the user presumably has reason to believe the collapse will be beneficial. Signed-off-by: Zach O'Keefe --- mm/khugepaged.c | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 2f95f60431aa..b9bf15faba26 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -90,6 +90,9 @@ struct collapse_control { /* Respect khugepaged_max_ptes_[none|swap|shared] */ bool enforce_pte_scan_limits; + /* Require memory to be young */ + bool enforce_young; + /* Num pages scanned per node */ int node_load[MAX_NUMNODES]; @@ -737,9 +740,10 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, list_add_tail(&page->lru, compound_pagelist); next: /* There should be enough young pte to collapse the page */ - if (pte_young(pteval) || - page_is_young(page) || PageReferenced(page) || - mmu_notifier_test_young(vma->vm_mm, address)) + if (cc->enforce_young && + (pte_young(pteval) || page_is_young(page) || + PageReferenced(page) || mmu_notifier_test_young(vma->vm_mm, + address))) referenced++; if (pte_write(pteval)) @@ -748,7 +752,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, if (unlikely(!writable)) { result = SCAN_PAGE_RO; - } else if (unlikely(!referenced)) { + } else if (unlikely(cc->enforce_young && !referenced)) { result = SCAN_LACK_REFERENCED_PAGE; } else { result = SCAN_SUCCEED; @@ -1408,14 +1412,16 @@ static void scan_pmd(struct mm_struct *mm, struct vm_area_struct *vma, cr->result = SCAN_PAGE_COUNT; goto out_unmap; } - if (pte_young(pteval) || - page_is_young(page) || PageReferenced(page) || - mmu_notifier_test_young(vma->vm_mm, address)) + if (cc->enforce_young && + (pte_young(pteval) || page_is_young(page) || + PageReferenced(page) || mmu_notifier_test_young(vma->vm_mm, + address))) referenced++; } if (!writable) { cr->result = SCAN_PAGE_RO; - } else if (!referenced || (unmapped && referenced < HPAGE_PMD_NR/2)) { + } else if (cc->enforce_young && (!referenced || (unmapped && referenced + < HPAGE_PMD_NR / 2))) { cr->result = SCAN_LACK_REFERENCED_PAGE; } else { cr->result = SCAN_SUCCEED; @@ -2375,6 +2381,7 @@ static int khugepaged(void *none) struct mm_slot *mm_slot; struct collapse_control cc = { .enforce_pte_scan_limits = true, + .enforce_young = true, .last_target_node = NUMA_NO_NODE, .alloc_hpage = &khugepaged_alloc_page, }; @@ -2517,6 +2524,7 @@ int madvise_collapse(struct vm_area_struct *vma, struct vm_area_struct **prev, { struct collapse_control cc = { .enforce_pte_scan_limits = false, + .enforce_young = false, .last_target_node = NUMA_NO_NODE, .hpage = NULL, .alloc_hpage = &alloc_hpage,