From patchwork Mon Aug 19 02:30:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 13767730 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D93C3C52D7C for ; Mon, 19 Aug 2024 02:32:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7B0776B008C; Sun, 18 Aug 2024 22:31:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 714246B008A; Sun, 18 Aug 2024 22:31:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4EED86B008C; Sun, 18 Aug 2024 22:31:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 2BB446B0089 for ; Sun, 18 Aug 2024 22:31:59 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id A88B040EAF for ; Mon, 19 Aug 2024 02:31:58 +0000 (UTC) X-FDA: 82467419916.13.885B022 Received: from mail-qt1-f173.google.com (mail-qt1-f173.google.com [209.85.160.173]) by imf06.hostedemail.com (Postfix) with ESMTP id D755218000E for ; Mon, 19 Aug 2024 02:31:56 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=kJy1gz69; spf=pass (imf06.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.160.173 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724034614; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7qMuNDyBvGp6REiF+VtGrTnesU92EL9IKEWMO4HzFdQ=; b=NBXCSqS4dUS919SlwvV8KwjE1cRs5NWHkW+WQlBzjrIpkTMkPROsuhREko4s/il7ONqmkj IcR0I9svDofRd9po8eBls+l7bK13sfYGO2r+3bDm3HK4lM7Ik45OWvHXxEDKPUi7Tkfsxi r0QlYS2odaPkgGbJYQgYJBcuWfC0MxI= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=kJy1gz69; spf=pass (imf06.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.160.173 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724034614; a=rsa-sha256; cv=none; b=X9k9zR7tgBOq2M6UeZC9gCX553GjDIV0KoAPVcu8GEJsarRgSPDXhgfbi9ztqnAIk1qQjy 99wIyeBZf5UqgWN3UAWOHAovUHy2BTfbT0RBFtv5rTnpTlpC2RrxbMBxPkSWmQgTmFuCcm EBuaQMbs0YYWa5LNEiQeek4ENW1ly14= Received: by mail-qt1-f173.google.com with SMTP id d75a77b69052e-44fea44f725so31123751cf.1 for ; Sun, 18 Aug 2024 19:31:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1724034716; x=1724639516; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7qMuNDyBvGp6REiF+VtGrTnesU92EL9IKEWMO4HzFdQ=; b=kJy1gz69rZL/2Hw+RrceeyTiHby/1vFJaKP8cBZGme6xqlvzmHprc15Jer+B+7a4HU ODQ3JeUgQUUYmmDeBuBb9dU9pt/E3mK+mjTsRl3ukVFUXIrF5EpK1LYA+n1VjMi7f9Up 3n4EfwSn1Cy5NZZiFVsUiA1utHIhJvVErY789mk9IY5H06OZ2MutYzCB3YojdDOqv7Wm BDNNgbOevwP4YON0bHJcu7GKceaU9kbredziuB6w9GeozFndAnj0AS5p4yw8F9KBpkhR sFI/tZ1ZfpjPfEDXw4kZd/5l1lbx2jZUHSWSZjyuQylVPJw/Gaq6SObzdh1DHk4rq9Gv c44g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724034716; x=1724639516; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7qMuNDyBvGp6REiF+VtGrTnesU92EL9IKEWMO4HzFdQ=; b=vkndlEoI+vxoWeSOwBEdu/B49mPwySSMkpa/KXn4JhYGdjiGroJm8NWIcgq3jtHwoL P/y3TdKHIUwhRJNKCYO0/ovPTYsWHkOsKr8SO6BP64J3rsVq9smRO7Lj5RuuBJqUruUI XNdyzwJDiOsgWy6eQtkVMlw6zIhQnjGjEUmtrBX5tepKB1e4jYA/e+RMD+1MgCtftnDy xXIGINd/7DOj7aQ4HdeB1cUVgTLxySciu4DqGm7XigZLzFmV7tavi9bkSGfTZvUTUvW0 gZcLSRDH9IPHpFd4g6BHLDqZPw2zZJjajVzHOyNWCsQwb9RMDmtyTGQ1KUU8q+ngTd70 ArgA== X-Forwarded-Encrypted: i=1; AJvYcCW4/o/LwTFuNHK6Z/NBciIguSvNwwhNQPTeW2oDiAkbQQZDXtZbzAvn+yDEDSNywljccmNPuUJ/CoKdXXZVvOBGiHk= X-Gm-Message-State: AOJu0YziVROdNYlvcSGlKyGM7XlFbr7I5O39Zf+VpZ4h2tBR6J4OZIa9 ADpbjDQIwOhsYl8RSYSaGNspeUhkRT/dwrAZ/Ca5HcaLOX0EfLEn X-Google-Smtp-Source: AGHT+IFmtPd7hU7za122IjXT3WTFdEVF9e2slXlOQoczJAWqePZoKzw0RZWmiFcdFLHebyhRGR/CwA== X-Received: by 2002:a05:622a:4d0c:b0:453:1334:9725 with SMTP id d75a77b69052e-45374f5468cmr173712171cf.3.1724034715718; Sun, 18 Aug 2024 19:31:55 -0700 (PDT) Received: from localhost (fwdproxy-ash-000.fbsv.net. [2a03:2880:20ff::face:b00c]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45369fee649sm36828591cf.29.2024.08.18.19.31.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Aug 2024 19:31:55 -0700 (PDT) From: Usama Arif To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, roman.gushchin@linux.dev, yuzhao@google.com, david@redhat.com, baohua@kernel.org, ryan.roberts@arm.com, rppt@kernel.org, willy@infradead.org, cerasuolodomenico@gmail.com, ryncsn@gmail.com, corbet@lwn.net, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kernel-team@meta.com, Usama Arif Subject: [PATCH v4 4/6] mm: Introduce a pageflag for partially mapped folios Date: Mon, 19 Aug 2024 03:30:57 +0100 Message-ID: <20240819023145.2415299-5-usamaarif642@gmail.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20240819023145.2415299-1-usamaarif642@gmail.com> References: <20240819023145.2415299-1-usamaarif642@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: D755218000E X-Stat-Signature: agxgrxs964jt6f7qon68bxzhwsqswo8j X-Rspam-User: X-HE-Tag: 1724034716-993560 X-HE-Meta: U2FsdGVkX19YEDC3u4burDNEqVMvDa3U90zKqkGOLX8emUvnFqSLWmZtj4n7sJGBNCzmKrxV6h5/z8jcU6bhpvhKJg9SG2HitO5BJPi5hdZ8jnicZv/4HRqTqo5A7Tgoafy3YxW9kp1X511MHaCqdS9GVZEIqZnxWKDYRWCRsYxzUMfwfO6WJ0VG6IH/LTke5xLxF3hdQnKxdFzXozw93+YR/ol4ZobeHl4NqvpZgJp/WgksbsFrmJl7V5mRHZ+N6BoQxZwO7gFNowQdUEsTh7CmAIf1os1vlnkdAMBexN1i/Dqdx9aX+AscdZIQm0Qf7rY9jeziTvPIPTv3mLwv4l0BtSlkRSXr6I2Iop6n5SksHrsaSJhh37djWaRpyQWy17u+OYJpWw8ZCiOZ5ieOzBnSmcxgG5AbEhJNLpYMG1qz3et0+X9MFXVrslz1Zgc0iza0z3LrjL/Q4nQLMe37HhJuK/Q5yWIPYgQsIdKV5q9W5phMxJ8CyeWEny8sTmM4wXSfGDQW0OCLPmmhLXG8nRCwc5isyvmAjJ+19RzR7W0wGeETxkn+lYHU2Ew4RjC68j4zcEsguVy+PRptzrWEOQEoOtYBLaX7d7SZFqnmcXUlyxLhe/rc5u3GRGTwbDaiPHu7NniGEUDQwZ2h0CHtJ9nrx4J9XCmblPn4S2hv+JKVIuxZBnVxzD8PkNziuDONkFePXChpXvndHjWSUl/FidMGLw5VBolCEHmyzoqBLCz100tv3JA1mAGHTXqKtGorAvXBoQ0x8s8V3lJ74yJwqeMp+ec8+max+F8WM4ziAqXRhfSCuhUTB8kTbo48YYOsu3lBBj9x+I4vBvFzXUk9Wz0IprJlTU1t3iv/D9MC9sbcaaQRdolUHjO9/uqNVM28Yk8aXyb7NIQb3xY0pKxbLfKYkfvnaNwgaS0LI2ynf+SVM6z7eOfb2vM2JQ8cttmvnS6+AU236hKsjq7GCqG ECkaWuk6 WiRPgQfR5381TVZzvARXg/M+kfugTsBXYZ1z1eLnjS5ORWQRj7Vck2EDNlBu8mHLWbTFjYusVQhVvjG6oFAaQ5C0B013h0DtHPhYmIBDH+yyqNqlcoA02zFIK+3JmhQksimBys85VSdd44sn8ndC/i63CO5zfi72n4n1PHheFksx01gu+iWErGNHWFZildM5XAstddtafz5gzY+NiqUB/BWozN5DAcISb5a1EqO1ewPTzXnKnVExF+Cj2u1hr9XTUpR9Juhbb5OIETu+jqPLq9jkHxwicLw7H9sKB4hYNv5Z5FlyFKff8fZXVGHorljphWJ7BqHK1h7MrYkaE7Td/X3xWjg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently folio->_deferred_list is used to keep track of partially_mapped folios that are going to be split under memory pressure. In the next patch, all THPs that are faulted in and collapsed by khugepaged are also going to be tracked using _deferred_list. This patch introduces a pageflag to be able to distinguish between partially mapped folios and others in the deferred_list at split time in deferred_split_scan. Its needed as __folio_remove_rmap decrements _mapcount, _large_mapcount and _entire_mapcount, hence it won't be possible to distinguish between partially mapped folios and others in deferred_split_scan. Eventhough it introduces an extra flag to track if the folio is partially mapped, there is no functional change intended with this patch and the flag is not useful in this patch itself, it will become useful in the next patch when _deferred_list has non partially mapped folios. Signed-off-by: Usama Arif Signed-off-by: Usama Arif --- include/linux/huge_mm.h | 4 ++-- include/linux/page-flags.h | 11 +++++++++++ mm/huge_memory.c | 23 ++++++++++++++++------- mm/internal.h | 4 +++- mm/memcontrol.c | 3 ++- mm/migrate.c | 3 ++- mm/page_alloc.c | 5 +++-- mm/rmap.c | 5 +++-- mm/vmscan.c | 3 ++- 9 files changed, 44 insertions(+), 17 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 4c32058cacfe..969f11f360d2 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -321,7 +321,7 @@ static inline int split_huge_page(struct page *page) { return split_huge_page_to_list_to_order(page, NULL, 0); } -void deferred_split_folio(struct folio *folio); +void deferred_split_folio(struct folio *folio, bool partially_mapped); void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, unsigned long address, bool freeze, struct folio *folio); @@ -495,7 +495,7 @@ static inline int split_huge_page(struct page *page) { return 0; } -static inline void deferred_split_folio(struct folio *folio) {} +static inline void deferred_split_folio(struct folio *folio, bool partially_mapped) {} #define split_huge_pmd(__vma, __pmd, __address) \ do { } while (0) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index a0a29bd092f8..c3bb0e0da581 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -182,6 +182,7 @@ enum pageflags { /* At least one page in this folio has the hwpoison flag set */ PG_has_hwpoisoned = PG_active, PG_large_rmappable = PG_workingset, /* anon or file-backed */ + PG_partially_mapped = PG_reclaim, /* was identified to be partially mapped */ }; #define PAGEFLAGS_MASK ((1UL << NR_PAGEFLAGS) - 1) @@ -861,8 +862,18 @@ static inline void ClearPageCompound(struct page *page) ClearPageHead(page); } FOLIO_FLAG(large_rmappable, FOLIO_SECOND_PAGE) +FOLIO_TEST_FLAG(partially_mapped, FOLIO_SECOND_PAGE) +/* + * PG_partially_mapped is protected by deferred_split split_queue_lock, + * so its safe to use non-atomic set/clear. + */ +__FOLIO_SET_FLAG(partially_mapped, FOLIO_SECOND_PAGE) +__FOLIO_CLEAR_FLAG(partially_mapped, FOLIO_SECOND_PAGE) #else FOLIO_FLAG_FALSE(large_rmappable) +FOLIO_TEST_FLAG_FALSE(partially_mapped) +__FOLIO_SET_FLAG_NOOP(partially_mapped) +__FOLIO_CLEAR_FLAG_NOOP(partially_mapped) #endif #define PG_head_mask ((1UL << PG_head)) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 2d77b5d2291e..70ee49dfeaad 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3398,6 +3398,7 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, * page_deferred_list. */ list_del_init(&folio->_deferred_list); + __folio_clear_partially_mapped(folio); } spin_unlock(&ds_queue->split_queue_lock); if (mapping) { @@ -3454,11 +3455,13 @@ void __folio_undo_large_rmappable(struct folio *folio) if (!list_empty(&folio->_deferred_list)) { ds_queue->split_queue_len--; list_del_init(&folio->_deferred_list); + __folio_clear_partially_mapped(folio); } spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); } -void deferred_split_folio(struct folio *folio) +/* partially_mapped=false won't clear PG_partially_mapped folio flag */ +void deferred_split_folio(struct folio *folio, bool partially_mapped) { struct deferred_split *ds_queue = get_deferred_split_queue(folio); #ifdef CONFIG_MEMCG @@ -3486,14 +3489,19 @@ void deferred_split_folio(struct folio *folio) if (folio_test_swapcache(folio)) return; - if (!list_empty(&folio->_deferred_list)) - return; - spin_lock_irqsave(&ds_queue->split_queue_lock, flags); + if (partially_mapped) { + if (!folio_test_partially_mapped(folio)) { + __folio_set_partially_mapped(folio); + if (folio_test_pmd_mappable(folio)) + count_vm_event(THP_DEFERRED_SPLIT_PAGE); + count_mthp_stat(folio_order(folio), MTHP_STAT_SPLIT_DEFERRED); + } + } else { + /* partially mapped folios cannot become non-partially mapped */ + VM_WARN_ON_FOLIO(folio_test_partially_mapped(folio), folio); + } if (list_empty(&folio->_deferred_list)) { - if (folio_test_pmd_mappable(folio)) - count_vm_event(THP_DEFERRED_SPLIT_PAGE); - count_mthp_stat(folio_order(folio), MTHP_STAT_SPLIT_DEFERRED); list_add_tail(&folio->_deferred_list, &ds_queue->split_queue); ds_queue->split_queue_len++; #ifdef CONFIG_MEMCG @@ -3542,6 +3550,7 @@ static unsigned long deferred_split_scan(struct shrinker *shrink, } else { /* We lost race with folio_put() */ list_del_init(&folio->_deferred_list); + __folio_clear_partially_mapped(folio); ds_queue->split_queue_len--; } if (!--sc->nr_to_scan) diff --git a/mm/internal.h b/mm/internal.h index 52f7fc4e8ac3..27cbb5365841 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -662,8 +662,10 @@ static inline void prep_compound_head(struct page *page, unsigned int order) atomic_set(&folio->_entire_mapcount, -1); atomic_set(&folio->_nr_pages_mapped, 0); atomic_set(&folio->_pincount, 0); - if (order > 1) + if (order > 1) { INIT_LIST_HEAD(&folio->_deferred_list); + __folio_clear_partially_mapped(folio); + } } static inline void prep_compound_tail(struct page *head, int tail_idx) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index e1ffd2950393..0fd95daecf9a 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4669,7 +4669,8 @@ static void uncharge_folio(struct folio *folio, struct uncharge_gather *ug) VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); VM_BUG_ON_FOLIO(folio_order(folio) > 1 && !folio_test_hugetlb(folio) && - !list_empty(&folio->_deferred_list), folio); + !list_empty(&folio->_deferred_list) && + folio_test_partially_mapped(folio), folio); /* * Nobody should be changing or seriously looking at diff --git a/mm/migrate.c b/mm/migrate.c index 2d2e65d69427..ef4a732f22b1 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1735,7 +1735,8 @@ static int migrate_pages_batch(struct list_head *from, * use _deferred_list. */ if (nr_pages > 2 && - !list_empty(&folio->_deferred_list)) { + !list_empty(&folio->_deferred_list) && + folio_test_partially_mapped(folio)) { if (!try_split_folio(folio, split_folios, mode)) { nr_failed++; stats->nr_thp_failed += is_thp; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 408ef3d25cf5..a145c550dd2a 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -957,8 +957,9 @@ static int free_tail_page_prepare(struct page *head_page, struct page *page) break; case 2: /* the second tail page: deferred_list overlaps ->mapping */ - if (unlikely(!list_empty(&folio->_deferred_list))) { - bad_page(page, "on deferred list"); + if (unlikely(!list_empty(&folio->_deferred_list) && + folio_test_partially_mapped(folio))) { + bad_page(page, "partially mapped folio on deferred list"); goto out; } break; diff --git a/mm/rmap.c b/mm/rmap.c index a6b9cd0b2b18..4c330635aa4e 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1578,8 +1578,9 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, * Check partially_mapped first to ensure it is a large folio. */ if (partially_mapped && folio_test_anon(folio) && - list_empty(&folio->_deferred_list)) - deferred_split_folio(folio); + !folio_test_partially_mapped(folio)) + deferred_split_folio(folio, true); + __folio_mod_stat(folio, -nr, -nr_pmdmapped); /* diff --git a/mm/vmscan.c b/mm/vmscan.c index 25e43bb3b574..25f4e8403f41 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1233,7 +1233,8 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, * Split partially mapped folios right away. * We can free the unmapped pages without IO. */ - if (data_race(!list_empty(&folio->_deferred_list)) && + if (data_race(!list_empty(&folio->_deferred_list) && + folio_test_partially_mapped(folio)) && split_folio_to_list(folio, folio_list)) goto activate_locked; }