From patchwork Tue Jul 30 12:46:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 13747381 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8865CC3DA7E for ; Tue, 30 Jul 2024 12:54:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A8E366B008C; Tue, 30 Jul 2024 08:54:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A1ACA6B0095; Tue, 30 Jul 2024 08:54:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 81E196B0096; Tue, 30 Jul 2024 08:54:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 5E8CB6B008C for ; Tue, 30 Jul 2024 08:54:16 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 0D8A4402D9 for ; Tue, 30 Jul 2024 12:54:16 +0000 (UTC) X-FDA: 82396412112.23.158199E Received: from mail-qk1-f174.google.com (mail-qk1-f174.google.com [209.85.222.174]) by imf01.hostedemail.com (Postfix) with ESMTP id 440EB4000D for ; Tue, 30 Jul 2024 12:54:14 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="g/59N/X/"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf01.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.222.174 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1722343993; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Ktt9C/EJSJHB77GbE72Wr4Bnix41Ywh1hlDOhut1uLE=; b=JEbkl8Y8rEbMXqKAeYaTWMNnQ2YFK5fMLgqveKEQ5BgAAKKB5Yfjux9kfDcX3cAyOQk+9w g+lTC5m9poO3kY7JVxoBCpXbWNwSEqEZWy+IXMmIy8fncFQqNdEPAmd9Me3fCGF3x/cyYg N4u2duV1r7ZGR4bGtJl92B1Ml/HSy1o= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1722343993; a=rsa-sha256; cv=none; b=mjasP1vS5Y+EFLMxB9rmAxMfyCpLwN7TALynfutf7o5XpQ+AOZ43W5slgQddY2vu8b5nQb 2JWHCRu70jbQE8eW4yduodMimqfeZYmuyeugIgnxSOAND/pcrjQf/DyoR/PvQw9jt38nun +KvKy3VapH1LuA5FgOqoHpi+k80CL7k= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="g/59N/X/"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf01.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.222.174 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com Received: by mail-qk1-f174.google.com with SMTP id af79cd13be357-7a1d067d5bbso281716385a.3 for ; Tue, 30 Jul 2024 05:54:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1722344053; x=1722948853; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Ktt9C/EJSJHB77GbE72Wr4Bnix41Ywh1hlDOhut1uLE=; b=g/59N/X/FDWgCaVygq9rKW0eXUV3VQtQWFGDL7Ndgf2X3UZLDjukUCq8GYOz2f3adh J4tfz+uDLAczzukNQPbawV0mSkxopFv6hARob9vr8zZBmMFoeVhlYTkGxI1ZkY8b65yu 3nmi7pDMqkXAqHAgYx5hdSJnYNH+jfCEjlP9/O0uAerNYVmg9hCYToEFWNor8yhbQLe9 Pcw1xsLvf4Bj7GFhauqBiBCTySEyCSdjEIlLzrnY4Wy/eVBArnU86kMS8b7j1HvYF/ZT ATAMpuMxpojJKSlxVnhaZIBx+MDiylWv6ZpdOjU+wSBbxwU7PjiYWlA2qi2EgO5DykXk Zt5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722344053; x=1722948853; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ktt9C/EJSJHB77GbE72Wr4Bnix41Ywh1hlDOhut1uLE=; b=XBLwVmcAPEdE1W+HczaS5xaHyY0yuFR4ES9/oyCKJdOHTlCmElKHrvPFSlsFSQON1L McMn67QSReOZHPXahie/GZh+ZOnYdL2WJoji/mfKc7OdznYBRE3jnji6CD7f3kxphIu1 F/NZxCMm0SLHX6JCcrD+Ebz70oWzVzhDEmRpylf1RNQrlnDok2jnvcpP7RxpOQe7BfuB MSjLhV2glAo7ttPbZOh2W+s3vVmy9REMOl1Gn4UpziA15ww3W/G+fL5MXM0kmkFYTpTX dptWg06zm/rEfN18Ye4FFpp1jTzM+4YKuIR++5CmtLiZ7NRl3vcYu96EAAkwHWLB1vrY 7QwQ== X-Forwarded-Encrypted: i=1; AJvYcCU7NAw19c5TCXHPNUePJJfOSLMh8d+1yZ6HnvoCWLd5MYcdK0Qgn1gJgn2SQPUL2SD2wsMo+X78wwKP0YcefO0YEEc= X-Gm-Message-State: AOJu0YwYgIkVPE53QHvjKExr4Aeda96HxhzJhxyvB4r/Etqc35AgquvG I70raNFxICf4zXBT96jkTCrj83MOgIf+1e1Yj6W7Gld46UNGeUXf X-Google-Smtp-Source: AGHT+IGs+7p/NLEKs6R6CF0jeACef2MQxZrB+Zf+vdpZds2BGZMJaXfelQzjq7hVci+EokgCJHDthQ== X-Received: by 2002:a05:620a:4496:b0:79f:dce:76c7 with SMTP id af79cd13be357-7a1e5302964mr1165797885a.68.1722344053350; Tue, 30 Jul 2024 05:54:13 -0700 (PDT) Received: from localhost (fwdproxy-ash-113.fbsv.net. [2a03:2880:20ff:71::face:b00c]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-44fe8171e52sm49968221cf.42.2024.07.30.05.54.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Jul 2024 05:54:13 -0700 (PDT) From: Usama Arif To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, roman.gushchin@linux.dev, yuzhao@google.com, david@redhat.com, baohua@kernel.org, ryan.roberts@arm.com, rppt@kernel.org, willy@infradead.org, cerasuolodomenico@gmail.com, corbet@lwn.net, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kernel-team@meta.com, Shuang Zhai , Usama Arif Subject: [PATCH 3/6] mm: free zapped tail pages when splitting isolated thp Date: Tue, 30 Jul 2024 13:46:00 +0100 Message-ID: <20240730125346.1580150-4-usamaarif642@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240730125346.1580150-1-usamaarif642@gmail.com> References: <20240730125346.1580150-1-usamaarif642@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 440EB4000D X-Stat-Signature: ekr6z84dc8n8dkhjgffbm9s9usdbfw4z X-Rspam-User: X-HE-Tag: 1722344054-143602 X-HE-Meta: U2FsdGVkX18AZ2foRfFjR2HyFkNagwEy4VYVXDfVn1tCgX83gPPOn9RumwWiiAimuvP1iFxu+WI/8kJNL3K8G7LTOaOgMY92u3r0EekYjqk5jvdjnzMdA1uV8If3gyOGMrHbJUlYYlolVMIkV3E0yx040vGBRzXYTFEjaH2hwUPhUaAK2USItMmAPOCCrxq33pCdFE3hnF0DWjvxi/pHun/AFoMjhRGbg9S1AgAFT8rOCHUayV+M74fmWsfWlHyW1YUChwi3R9nAOhPEWiGdSN/L/v6SC6wCcMXl6cr7ifdUrKBjO0NvBsEPGiICyYB1VDLkDC/zhbtZHFlvcFEhQglmnPUb8sx4qU7Bl17sWmbXagVy3qw/wPj6jdWHbMfJxP/mrqwX/849OJj1gu+21hDr/xSd0SsYX1BxIpsm0AoFkUB7aP+FCGb5RSnStDLEcGBMkqHX9OQOBGl/JCmg0nVJl0F6zr2Slo60DzGZTQCjTwCOJJVM/Oed9VtkIMmgHGBCwyMg3BwKrYEbP8MWH4WRrDsq9r049wlSJevSsYFNyFzGC6pvwbCzsyz+CXQcOeX3I5yw9VtJmMgg+t4AydmAl2c2VqacKlTo+Hm06e+c4u1vv38CQtUOmwl2zqyypkj6dCPtdjvqChN2TyF+nP3rUMoNUelBDE56CUZn6ou62ldPzmu/c4xGfbPuhPbqBMKCnyMrAsFItytXX1BLZ8rLuWY2dbKGjnrk8gnEGIlrhKEo7uGQThacTiTXivPjgkoRHNssTI7fGXsykcPpLhjCplSCyZMST2turuxxEIT27g42LGReYV6323jNZPS8bm1XhFDnpvZZci5Jm/8t25M8OXK8qfvNBIsPDmWZpSTIvhBINDhf7Dslhhqvuzbg63UCHihV5gKg5Bb7UhE4g9zRSqtt3bAf1VY1o8RiN7lhjltf3OPODv+ufM1EGAT51YKeQr31oltqaNfjsk6 pljNAOXs qcbymWOOOGcTvQZoFpZcIdPxXDb91p7EMUvwCusMRk/n4Y/uwFWsboyjoafp36CdW3yzZ3sL9VvvaHQcC/AJDMScSLVZaxDOA31EA74tVS3VYKRVu4BN3a0ZGvg7o8R1OjZtGVwJa+jbEffYXoP+t6pZ+25G3N2zwLHTLeKL8zQcvNdRlZSOyPKcV/pRLjWWO7Bcc8oc7G19sU8qC+wJulr5odzciqO85cXsmJackQ4InPjmtyKtf+E0IOXCJpc4gzp0mHJgBaokoei+NzzU6SALBUhtC6aSOiNtvPlJwHd5iPtlDH+Ab4uMZJUxmT8jMn/ZjjDKYfJHjQYwTEsThvY73Z0GQr+tTc02TJ9XhMQ3tfMk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Yu Zhao If a tail page has only two references left, one inherited from the isolation of its head and the other from lru_add_page_tail() which we are about to drop, it means this tail page was concurrently zapped. Then we can safely free it and save page reclaim or migration the trouble of trying it. Signed-off-by: Yu Zhao Tested-by: Shuang Zhai Signed-off-by: Usama Arif --- mm/huge_memory.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 0167dc27e365..76a3b6a2b796 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2923,6 +2923,8 @@ static void __split_huge_page(struct page *page, struct list_head *list, unsigned int new_nr = 1 << new_order; int order = folio_order(folio); unsigned int nr = 1 << order; + LIST_HEAD(pages_to_free); + int nr_pages_to_free = 0; /* complete memcg works before add pages to LRU */ split_page_memcg(head, order, new_order); @@ -3007,6 +3009,24 @@ static void __split_huge_page(struct page *page, struct list_head *list, if (subpage == page) continue; folio_unlock(new_folio); + /* + * If a tail page has only two references left, one inherited + * from the isolation of its head and the other from + * lru_add_page_tail() which we are about to drop, it means this + * tail page was concurrently zapped. Then we can safely free it + * and save page reclaim or migration the trouble of trying it. + */ + if (list && page_ref_freeze(subpage, 2)) { + VM_BUG_ON_PAGE(PageLRU(subpage), subpage); + VM_BUG_ON_PAGE(PageCompound(subpage), subpage); + VM_BUG_ON_PAGE(page_mapped(subpage), subpage); + + ClearPageActive(subpage); + ClearPageUnevictable(subpage); + list_move(&subpage->lru, &pages_to_free); + nr_pages_to_free++; + continue; + } /* * Subpages may be freed if there wasn't any mapping @@ -3017,6 +3037,12 @@ static void __split_huge_page(struct page *page, struct list_head *list, */ free_page_and_swap_cache(subpage); } + + if (!nr_pages_to_free) + return; + + mem_cgroup_uncharge_list(&pages_to_free); + free_unref_page_list(&pages_to_free); } /* Racy check whether the huge page can be split */