From patchwork Fri Jul 19 04:25:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13736853 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75965C3DA59 for ; Fri, 19 Jul 2024 04:25:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BE9EE6B0082; Fri, 19 Jul 2024 00:25:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B9B5A6B0083; Fri, 19 Jul 2024 00:25:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A61DA6B0088; Fri, 19 Jul 2024 00:25:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 87DBA6B0082 for ; Fri, 19 Jul 2024 00:25:10 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 1AE341203C0 for ; Fri, 19 Jul 2024 04:25:10 +0000 (UTC) X-FDA: 82355212380.06.E41A721 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf15.hostedemail.com (Postfix) with ESMTP id 69AFAA0011 for ; Fri, 19 Jul 2024 04:25:08 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=zkweEQyn; spf=pass (imf15.hostedemail.com: domain of 3o-qZZgYKCBwQMR92G8GG8D6.4GEDAFMP-EECN24C.GJ8@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3o-qZZgYKCBwQMR92G8GG8D6.4GEDAFMP-EECN24C.GJ8@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721363076; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=+NNarvcCPGmC6xaUjCG54gZuwTNt3t3JLl9nhr1UDkw=; b=oPoyZ56JjtCUW1YVKHX8XPMKqcL0z+O1vDmaR7kXtfZdo6EJgHi5KNAU/RqNLvryKZrz8C yycl7hSZTYe1XoOlxNIdTCeBIq7B0Z5bBhZUi9+PSA8DBc9INnRu8a0ORolB6b2isgfWHA fhZ0He7yvekBT/GLTPoB/0dmA1YoJTo= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=zkweEQyn; spf=pass (imf15.hostedemail.com: domain of 3o-qZZgYKCBwQMR92G8GG8D6.4GEDAFMP-EECN24C.GJ8@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3o-qZZgYKCBwQMR92G8GG8D6.4GEDAFMP-EECN24C.GJ8@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721363076; a=rsa-sha256; cv=none; b=PGE1X/Bc35jc5/MoeIkK+41WUrKzTR70X8GNAxxliNUKB9Cu8Zl0IFsf0EVqU0iY/9Om5N 4cjBgPK7uNUfYRuRSD6eDhD4PzE8a6ZY9bNNXT41ypmrlLDxIoONv/F6OHL/uOlr7RWRNC 26UdJK8oGqzBDBz7ENn6Dgou9tYKqVw= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e035949cc4eso3630353276.1 for ; Thu, 18 Jul 2024 21:25:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1721363107; x=1721967907; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=+NNarvcCPGmC6xaUjCG54gZuwTNt3t3JLl9nhr1UDkw=; b=zkweEQynn+hCSKjB7wQz/KgVFoUoM7sphVZKsHzw5mV2hcGLyOS7eb2OWGlPPPtJoN 2z7BZcwEz04bHHE8cQ3w7BHp03x/ghPt+vEL4ISbZwQg0USPpNVIDWg1dRC3QRy20uSy mUa5gAK7zhAghYmQ4HKVnVaVihQbH8LRrHk2V7JFVCYU44jUb/RLKN2hSKd55Xd5EetW cTBfH6dk+4XVLoEFnuqu5zJwzKCmL/D0xGiBDALFd/fYUX7VAM7ZdHdxWG5EOHrM4nlY X0+A1r+gV6Z1EmJFcjiSUdYk4R569nwLaPIUxQfNDcoZmLGTcZ9+sKjVUQ79mLNsemJ4 kCzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721363107; x=1721967907; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=+NNarvcCPGmC6xaUjCG54gZuwTNt3t3JLl9nhr1UDkw=; b=ioBOtvvSE2nzvd7jLbzlJiSNpeAFxnkatITxaDl468rkVjseh/1A6KUfc/kfgxw9Wk G+Ks347tKqVFSvqTwsMdrNgmUPgMHmZzSnooQpWcX6qqujQWvib9CTXvig0OX5zWnYwA pNpnwU7ltmM48NUDEJqUl6zQ64gXpULCBTqqKXYaRZJA1QJF4zjUW1F4QLJq0dDIuYUu O0taf9Y8cEg0M/0fGfuZJ1xoJO7+KcqiayAzMmO0uEKp3DfctvFYOdubLoAF2n44KuII vVuvU6GS7mbrDopS0s6tQtiH37+ENG9/t4rjhwgrIaU2X+7YaXx0wXoYt2IDUnGtaiUW 43uw== X-Gm-Message-State: AOJu0YxwLdtAHH6Lo5jEqdsnkFbUlalPYhKuS1XkfdWjyYXsRvQ/NuXo IEmXc6dvtJNlkHDmqM0uCF51FkEXx2q9bC7dOu9wgtR6rk5XisbWwTbYI2P3s0PRvXkmvXebWwp VkA== X-Google-Smtp-Source: AGHT+IE0YSx9HNsiQ/WXGG3ku7Od6DV4r5MXGVHv4m+T1aFzqES6PloIb+AJgOXSlV8hWcPpO1Vu5jk9/MA= X-Received: from yuzhao2.bld.corp.google.com ([2a00:79e0:2e28:6:9f7f:3650:b9e0:5d4b]) (user=yuzhao job=sendgmr) by 2002:a05:6902:c03:b0:e05:fc91:8935 with SMTP id 3f1490d57ef6-e05feb62eebmr84080276.3.1721363107223; Thu, 18 Jul 2024 21:25:07 -0700 (PDT) Date: Thu, 18 Jul 2024 22:25:03 -0600 Mime-Version: 1.0 X-Mailer: git-send-email 2.45.2.1089.g2a221341d9-goog Message-ID: <20240719042503.2752316-1-yuzhao@google.com> Subject: [PATCH mm-unstable v1] mm/hugetlb_vmemmap: don't synchronize_rcu() without HVO From: Yu Zhao To: Andrew Morton , Muchun Song Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao , kernel test robot , Janosch Frank , Marc Hartmayer X-Stat-Signature: 9qgzu1s56pr3iw4awqxtpuqfwohor1yj X-Rspam-User: X-Rspamd-Queue-Id: 69AFAA0011 X-Rspamd-Server: rspam02 X-HE-Tag: 1721363108-927148 X-HE-Meta: U2FsdGVkX18DczolPgzcUUGMAoMNv5utxbVzVokrZkXhKbQ1Pk+XLQWlC4QbJSvBXelUqT85mRR6SfYQEBbkPX1m1GPOZaFyfs2CNofhocU6ar2THd6JcSDuIYdSa3Zh/u9k7hRJOXBzMAFWXBBsgNlnlbyo9dDwr7FRDLwPWJPK6rmSKaj2Ilr8lpyLUhKn5Iw4PIypuoccsvNdql3ntonyqZL692hG43tuul9ouIJyQVdksUasbyDuOZ8RZeXqQFiSMZrVJcrxPBsk0mLurwBLzzMoifX0XwhywBKy3Qkx056elx+ywcPq/8VWaIqwNrNFovN0cpMm3Kuy4SXhb4D2hzfvIkouhr2b3Deo7qGw0uCG7Qt7UDpCmg4fM1G3NHNiC7PwbtlBDtoYzF1PSBpSIk5w1rbrdElr5bhn6bRsoNeMwrJ8GbcR4Qjkd1c+itYjtR4KVricpCThEZp3HhsBggGC4pfP9/OkLxmIJ+O3Rq97ZlJ9RPtrJ9z6LxAqLFAcKqiNCxTPHidY4eO9tSXg5S7cM6/oBRByFqxJ+EHFJFJsFs5b5gDv6tGTLr2XYSXZLfRjlTLGQ0sNDLDikpjO8pT7IQlJ3MZyGLfU22kux70WAyqVOAxYockzk06lsRrGVAfnzOXIEn1NY6ObFLe+dQQhsoO+6okpDhc0bBfSpyR+grk/FJNRVH3RLLVOBu3X1Q6EGudvUPzAZCfw1lPEmSd3LFOM7EkPDAeitd6aNqlh0bBKCmE6fL7vhb0oxT9U1RJ/BBOjWlkkK0a/5sqldEp5yBYAXdxhZuS1X4rj2rZNK65W2TpKimO5lrB38Dx3NwFIwHgILJL9NRADmW/HukRcpWWRcJYBN0bhtVzirBLV1svqfoiXhM0/sLzglk6Q7ZVLq08KuGHS6d9GW+fuwIEB/YJUMyJv8RYguBuyYso6ySPLuU64ttlEL7nV0fSzVg6uwd6T+FaElrx oX+987vb ic3MfzQWpz6jXp9e5aGMUWiewrbVSQ8R72z3ivhrJFd+gdeWrXJ4eduVxptOPIOLYZ+qgahyIPLAaCZfuHm/0c+k9quLu+G1inL7ozhdAb78hrPZejyePqT/epfuLWliEVoaSfWhJjcoAOBwbUrwHzLX2r2SIogFuBYBua+Z+6Ko5zv0cPFmEAfEkWPuCaFYAeYhVO3hNazuz4ZGPDpMuiJNQr1wfpo/XCipxkPbN38cqNSqYsaTf/it81vKTZQb6gKFzTmLhKcaPzLQvFvTLgpt39GKkbrv4r6t9Z0WiZDlIZyQF9x7WVNrDsaTQeFFpndNVHmldQBGrTMkHa01aJE6ca1VoS7+g0C0MFm8qAmhpYWTzcsxcQ/3uFhPXA6IH6LjGvgspnYgnLMpLRW5RSDz5Z3jQ/+ouyDZ3GCGlJYkkqfdXi5R88dHzVE+20KoNrDnXl13vhnJf4M31YFx9h0zIvicJIqaQnkDlCnF0oAPMrSuvv9R8UixB26RpUj/vS9FqiJYnZsqsQz2Z3Krq4m49USAXsStaBvw+vKvGSR2TA4gcvdgp1HrXETN/1YU0NMHD2TWQeqD8ED45c2tzDWXt6axHWxU4UMu51Zp6BkOLkEnzl1rk0qxtglk39w/BcI/4hCyRYnQyQrgXRfBsfLsS2w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: hugetlb_vmemmap_optimize_folio() and hugetlb_vmemmap_restore_folio() are wrappers meant to be called regardless of whether HVO is enabled. Therefore, they should not call synchronize_rcu(). Otherwise, it regresses use cases not enabling HVO. So move synchronize_rcu() to __hugetlb_vmemmap_optimize_folio() and __hugetlb_vmemmap_restore_folio(), and call it once for each batch of folios when HVO is enabled. Fixes: bd225530a4c7 ("mm/hugetlb_vmemmap: fix race with speculative PFN walkers") Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-lkp/202407091001.1250ad4a-oliver.sang@intel.com Reported-by: Janosch Frank Tested-by: Marc Hartmayer Signed-off-by: Yu Zhao Acked-by: Muchun Song --- mm/hugetlb_vmemmap.c | 40 ++++++++++++++++++++-------------------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 829112b0a914..809ce74bbdfd 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -43,6 +43,8 @@ struct vmemmap_remap_walk { #define VMEMMAP_SPLIT_NO_TLB_FLUSH BIT(0) /* Skip the TLB flush when we remap the PTE */ #define VMEMMAP_REMAP_NO_TLB_FLUSH BIT(1) +/* synchronize_rcu() to avoid writes from page_ref_add_unless() */ +#define VMEMMAP_SYNCHRONIZE_RCU BIT(2) unsigned long flags; }; @@ -460,6 +462,9 @@ static int __hugetlb_vmemmap_restore_folio(const struct hstate *h, if (!folio_test_hugetlb_vmemmap_optimized(folio)) return 0; + if (flags & VMEMMAP_SYNCHRONIZE_RCU) + synchronize_rcu(); + vmemmap_end = vmemmap_start + hugetlb_vmemmap_size(h); vmemmap_reuse = vmemmap_start; vmemmap_start += HUGETLB_VMEMMAP_RESERVE_SIZE; @@ -492,10 +497,7 @@ static int __hugetlb_vmemmap_restore_folio(const struct hstate *h, */ int hugetlb_vmemmap_restore_folio(const struct hstate *h, struct folio *folio) { - /* avoid writes from page_ref_add_unless() while unfolding vmemmap */ - synchronize_rcu(); - - return __hugetlb_vmemmap_restore_folio(h, folio, 0); + return __hugetlb_vmemmap_restore_folio(h, folio, VMEMMAP_SYNCHRONIZE_RCU); } /** @@ -518,14 +520,14 @@ long hugetlb_vmemmap_restore_folios(const struct hstate *h, struct folio *folio, *t_folio; long restored = 0; long ret = 0; - - /* avoid writes from page_ref_add_unless() while unfolding vmemmap */ - synchronize_rcu(); + unsigned long flags = VMEMMAP_REMAP_NO_TLB_FLUSH | VMEMMAP_SYNCHRONIZE_RCU; list_for_each_entry_safe(folio, t_folio, folio_list, lru) { if (folio_test_hugetlb_vmemmap_optimized(folio)) { - ret = __hugetlb_vmemmap_restore_folio(h, folio, - VMEMMAP_REMAP_NO_TLB_FLUSH); + ret = __hugetlb_vmemmap_restore_folio(h, folio, flags); + /* only need to synchronize_rcu() once for each batch */ + flags &= ~VMEMMAP_SYNCHRONIZE_RCU; + if (ret) break; restored++; @@ -573,6 +575,9 @@ static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h, return ret; static_branch_inc(&hugetlb_optimize_vmemmap_key); + + if (flags & VMEMMAP_SYNCHRONIZE_RCU) + synchronize_rcu(); /* * Very Subtle * If VMEMMAP_REMAP_NO_TLB_FLUSH is set, TLB flushing is not performed @@ -620,10 +625,7 @@ void hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *folio) { LIST_HEAD(vmemmap_pages); - /* avoid writes from page_ref_add_unless() while folding vmemmap */ - synchronize_rcu(); - - __hugetlb_vmemmap_optimize_folio(h, folio, &vmemmap_pages, 0); + __hugetlb_vmemmap_optimize_folio(h, folio, &vmemmap_pages, VMEMMAP_SYNCHRONIZE_RCU); free_vmemmap_page_list(&vmemmap_pages); } @@ -650,6 +652,7 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l { struct folio *folio; LIST_HEAD(vmemmap_pages); + unsigned long flags = VMEMMAP_REMAP_NO_TLB_FLUSH | VMEMMAP_SYNCHRONIZE_RCU; list_for_each_entry(folio, folio_list, lru) { int ret = hugetlb_vmemmap_split_folio(h, folio); @@ -666,14 +669,12 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l flush_tlb_all(); - /* avoid writes from page_ref_add_unless() while folding vmemmap */ - synchronize_rcu(); - list_for_each_entry(folio, folio_list, lru) { int ret; - ret = __hugetlb_vmemmap_optimize_folio(h, folio, &vmemmap_pages, - VMEMMAP_REMAP_NO_TLB_FLUSH); + ret = __hugetlb_vmemmap_optimize_folio(h, folio, &vmemmap_pages, flags); + /* only need to synchronize_rcu() once for each batch */ + flags &= ~VMEMMAP_SYNCHRONIZE_RCU; /* * Pages to be freed may have been accumulated. If we @@ -687,8 +688,7 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l flush_tlb_all(); free_vmemmap_page_list(&vmemmap_pages); INIT_LIST_HEAD(&vmemmap_pages); - __hugetlb_vmemmap_optimize_folio(h, folio, &vmemmap_pages, - VMEMMAP_REMAP_NO_TLB_FLUSH); + __hugetlb_vmemmap_optimize_folio(h, folio, &vmemmap_pages, flags); } }