From patchwork Tue Sep 3 21:36:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13789330 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C80BCD3440 for ; Tue, 3 Sep 2024 21:37:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B29368D01E5; Tue, 3 Sep 2024 17:37:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AD9F08D01E4; Tue, 3 Sep 2024 17:37:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 92AC28D01E5; Tue, 3 Sep 2024 17:37:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 6EE498D01E4 for ; Tue, 3 Sep 2024 17:37:03 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 2756C80719 for ; Tue, 3 Sep 2024 21:37:03 +0000 (UTC) X-FDA: 82524737526.03.7B8077A Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf08.hostedemail.com (Postfix) with ESMTP id 57DBB16000D for ; Tue, 3 Sep 2024 21:37:01 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=bF6jsTcK; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf08.hostedemail.com: domain of 3fIHXZgYKCBsPLQ81F7FF7C5.3FDC9ELO-DDBM13B.FI7@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3fIHXZgYKCBsPLQ81F7FF7C5.3FDC9ELO-DDBM13B.FI7@flex--yuzhao.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1725399315; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7lRbWVNLaQ1IZX12itRVMEELUogajjWtfSLYwZb6BSc=; b=BWfyl+XiVZB5MDsGfn8ZqL42Cclr6tFdPX76mj3X1odm6aj+zkiSd3AvZ30+Z1BlUCz73H YVHVYJtST8CmUCw+iISlDlSaDMlAzLG8IAky7FE35FLjayPMNlhiyUKuNUxTIYFQFPkJXz IAzJp5aVXZIOhJiFN9QCP4kFq+Gdkss= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1725399315; a=rsa-sha256; cv=none; b=nc7flOMC2PudIMIrO79A1LjZfulDTIIkhx8mhq64e5ugb1UUWy5nnZsO0RZSeySKYLlrjf N4zap5MqlA/CQQgGRmb57MREcRKscfC0+oxv++oLTtYY785G3SZJ/zwyc/TNv//tX8oq/y h8iLSnBnTL4Wxv3hYKnf41ltaB5AiOA= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=bF6jsTcK; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf08.hostedemail.com: domain of 3fIHXZgYKCBsPLQ81F7FF7C5.3FDC9ELO-DDBM13B.FI7@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3fIHXZgYKCBsPLQ81F7FF7C5.3FDC9ELO-DDBM13B.FI7@flex--yuzhao.bounces.google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e116c697904so11715490276.1 for ; Tue, 03 Sep 2024 14:37:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1725399420; x=1726004220; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7lRbWVNLaQ1IZX12itRVMEELUogajjWtfSLYwZb6BSc=; b=bF6jsTcKdBKjA+fNgi4VPdFAVateGaQCdpkUzVC2eiLzyjgRPVLjlzXwbQAZgww79Q lDLagmpSGcCvJR+m4PG7vHIHxerPTekWSbP9x9TvCpSdjZGte1SSoPEAOJ+QGcMUQ3kA 1TusYN5/l7+yTavHh1Tmcp53+jg8/Fh5kNcq4loYtQl8FekBWLjI/Ugv62+KElO7sepo BawCvCJwQvJQOEDgGftONH7Fdc6X/B0VE5CJjcwkytQgurlCWAdoLDsowWR5RWPjATka ZWDd3CSO7BUjgLi1dwemrSPNEfoVnbIgDboMCBMTxrVAsMNqqNlEsIgJAh522z0uDcY3 1/NQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725399420; x=1726004220; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7lRbWVNLaQ1IZX12itRVMEELUogajjWtfSLYwZb6BSc=; b=kF+yJUP7FZUTtP6L7s0fLYmr01PDNW+B8/jsGrUZNrThjdPxY+ldu6Db//4QzsiR8y H2ZT/guukY19vVbUZ8IvCNG+sezAmakYD0Wk9+MeLj8Zixa57w2CYGJmc64NX0de/QkB 7O3oLAFbq5i5pCZYnfM83wUbrUEaCMhgtkQeBvUqqFZoEPYmzLBxOjF8/EWKdC409wtN oMgQvba847INpcJLRNw4NIM73kNpJyOjvVNoQcEr13hnqk4g/SHAHBaRF63WbfSqvEbU qmww1FfU3U4ATUmgT/ynlwtVvXP55fWWNhEkw8x1a7qzJdaPNQDHxdnLA6k3K+EwYnOe GGoQ== X-Forwarded-Encrypted: i=1; AJvYcCVeQy9mFFabPNND1fARvCHSXBojnP+hrOnNGZRoZdiD7RoJW2I8GwX5QILRKVv2AJBCVFDUtpgQCA==@kvack.org X-Gm-Message-State: AOJu0YwGgLXTu8bpftlYue9IgsvonsSnajAit8FzAZoLdm1BdS+5Rqjc pxdFX0oNXsQsuhAV3QJouWs3p+9CJZ+GxjUR+yiKD3NEJUimd7EJDXREPgMZt+oWNB6vE4yGKLt NAA== X-Google-Smtp-Source: AGHT+IFKMuOj0a8gCYr8Reqa7ik1LwLwagBRCBwi0mhu448Z4IqlPF9etRzCqLqKmBASiXsj6Jc8sLxDbM4= X-Received: from yuzhao2.bld.corp.google.com ([2a00:79e0:2e28:6:bb58:3d85:3f9c:c6c6]) (user=yuzhao job=sendgmr) by 2002:a25:264d:0:b0:e11:7246:963b with SMTP id 3f1490d57ef6-e1a79fb492dmr639799276.1.1725399420440; Tue, 03 Sep 2024 14:37:00 -0700 (PDT) Date: Tue, 3 Sep 2024 15:36:49 -0600 In-Reply-To: <20240903213649.3566695-1-yuzhao@google.com> Mime-Version: 1.0 References: <20240903213649.3566695-1-yuzhao@google.com> X-Mailer: git-send-email 2.46.0.469.g59c65b2a67-goog Message-ID: <20240903213649.3566695-3-yuzhao@google.com> Subject: [PATCH mm-unstable v1 3/3] mm/codetag: add pgalloc_tag_copy() From: Yu Zhao To: Andrew Morton , Kent Overstreet , Suren Baghdasaryan Cc: Muchun Song , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 57DBB16000D X-Stat-Signature: rmr5hqokpawpsdfe5twz3i6kgizk9tsk X-Rspam-User: X-HE-Tag: 1725399421-144028 X-HE-Meta: U2FsdGVkX18oH4liRwZn4m/tAdKhIM1tWHCBKAhA/8FOIdQhz4+77aMSJ/eDHd+GeiVqzAQjbjbmaE+sgvQU+ZgSkfIj8DtlqEceNjbMsk7nA5AiELLjd7wUuoySZaOp1OBwbAqeVIR+NMqEPUgPDv/mxdz7mjBVUG9RlCivTXrWC3tpBdTYKrqU9Pb2rUFX729Sb9nT0BeRjB/PUAOkGu+ePhpYac8iG7xmz/pkEa+LI3JhSsazAypbJ7iamHib2tlh1NdmakKoQPrfjx6ppiMDFdi+NKY/l++3jhYLP14GTzm6W4j2HgG5Hj/2Qnb1hBQqWgHTA9OhHRNsxEEAxJiwWrVMhovNlnd/mlcaKIp5gN3HVmWbyEfWNN/kiFNIiPwoaEDvY2ownfEJy8HopXgAhXz2r/F4N8+KtkDUB+LfREco/yH0hH2WjtEgwYPYHIAMR+U/gu0utwkmJBJG9dI14Z2Xe4QB8KM/fXpBpewscrmB7oWNlaEixwz050L9nHLa/D6WhfK9ftV6ug3eSBtZPC43d9A5SLidi9c5Yd9EKtPN7aRSXPA/1YAMwpe57SbRNGs19cpr5X5AJvGWCfaYUqJ/ygvEYO/8qzyNDD9z/GUJI0i/iF0XbrK4o2UfFIjcUrnMYiEUrhY/cpJfobSpMSJNyuU1B9nZRDq+lCuB3eXI0mfwnU2rddqdzVJ5AWT6QvDaxTdcRGxjqFq0wNNGm9qcdx5kRjxCtoNcJG+2zbG2GKih3axQZvmrAzQMT9/OSLIwKhnX7RUtNMuj8Ma7PWYCTUmy51I9GM0Sgx9/ny1XoKNdQWo6dPq1HZJZLlQVrriq+Lxjdicu3n7sbeIim9XzEOsZ1zAP5w5ybo68VDOLD8uBMIsqtey+YH7f/ruRTWBfBb5nSkSAoL6dk6wFnHqAe/8TNwaPAWzqknI5yyrnAnt66enMK4xAo7viZ089TYfGNkgC0Uwb/lo Clo4cCm7 8oKfUv4x4PKPdh+2pJrEXiaSvJxtpf5CfnNvqlVtMRb70g99zZIRSfA4ADlJu7RK7hAwr8GbYwrBGigJR8clRFiDtC8XSnu/VEZA4Z6dtmYDZT6Stmm2/6pulsckzPccIok9GYfjCe7lAeENAAY36Oj3y5K1BFT3wwXbnCu888I7rLTaGOAHkFztGCzri+O6UFmp6HLeB3QSi9WqOah1mXpD2AfRr2oF4KsGj74trhutpK4v57CmRP733bJbJRoNb1RDztg3R4tMLdMVZOYbTW29IJzanPhbCdMRD6CS7TOdCIHfwmiYASQG3arsF00+1xxDGdlham0Cmwl0rMdLnvk2+G0H3ARaOxArRutBy8u4+TomIhT+ZwhpfsJbdURCYWuSF3phC+eSEFOdlnAwUU9IHevwtryjxI2rWZgPNBFlFFCs7ECO85JjYBDpwH+ITCf0UlR7bClzxnP2JzmJ+P0f7q2zaBKQPGwoP0/U/3CiK7HLYhOV67P2KHSqsrQOTsROovATYBA+SRIyfYGOytCpPBEPxUHx9wzzPaSCOSzgsE+o= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add pgalloc_tag_copy() to transfer the codetag from the old folio to the new one during migration. This makes original allocation sites persist cross migration rather than lump into compaction_alloc, e.g., # echo 1 >/proc/sys/vm/compact_memory # grep compaction_alloc /proc/allocinfo Before this patch: 132968448 32463 mm/compaction.c:1880 func:compaction_alloc After this patch: 0 0 mm/compaction.c:1880 func:compaction_alloc Signed-off-by: Yu Zhao Acked-by: Suren Baghdasaryan --- include/linux/alloc_tag.h | 24 ++++++++++-------------- include/linux/mm.h | 25 +++++++++++++++++++++++++ mm/migrate.c | 1 + 3 files changed, 36 insertions(+), 14 deletions(-) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index 896491d9ebe8..1f0a9ff23a2c 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -137,7 +137,16 @@ static inline void alloc_tag_sub_check(union codetag_ref *ref) {} /* Caller should verify both ref and tag to be valid */ static inline void __alloc_tag_ref_set(union codetag_ref *ref, struct alloc_tag *tag) { + alloc_tag_add_check(ref, tag); + if (!ref || !tag) + return; + ref->ct = &tag->ct; +} + +static inline void alloc_tag_ref_set(union codetag_ref *ref, struct alloc_tag *tag) +{ + __alloc_tag_ref_set(ref, tag); /* * We need in increment the call counter every time we have a new * allocation or when we split a large allocation into smaller ones. @@ -147,22 +156,9 @@ static inline void __alloc_tag_ref_set(union codetag_ref *ref, struct alloc_tag this_cpu_inc(tag->counters->calls); } -static inline void alloc_tag_ref_set(union codetag_ref *ref, struct alloc_tag *tag) -{ - alloc_tag_add_check(ref, tag); - if (!ref || !tag) - return; - - __alloc_tag_ref_set(ref, tag); -} - static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, size_t bytes) { - alloc_tag_add_check(ref, tag); - if (!ref || !tag) - return; - - __alloc_tag_ref_set(ref, tag); + alloc_tag_ref_set(ref, tag); this_cpu_add(tag->counters->bytes, bytes); } diff --git a/include/linux/mm.h b/include/linux/mm.h index a07e93adb8ad..1b98d843a5e9 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -4161,10 +4161,35 @@ static inline void pgalloc_tag_split(struct folio *folio, int old_order, int new } } } + +static inline void pgalloc_tag_copy(struct folio *new, struct folio *old) +{ + struct alloc_tag *tag; + union codetag_ref *ref; + + tag = pgalloc_tag_get(&old->page); + if (!tag) + return; + + ref = get_page_tag_ref(&new->page); + if (!ref) + return; + + /* Clear the old ref to the original allocation site. */ + clear_page_tag_ref(&old->page); + /* Decrement the counters of the tag on get_new_folio. */ + alloc_tag_sub(ref, folio_nr_pages(new)); + __alloc_tag_ref_set(ref, tag); + put_page_tag_ref(ref); +} #else /* !CONFIG_MEM_ALLOC_PROFILING */ static inline void pgalloc_tag_split(struct folio *folio, int old_order, int new_order) { } + +static inline void pgalloc_tag_copy(struct folio *new, struct folio *old) +{ +} #endif /* CONFIG_MEM_ALLOC_PROFILING */ #endif /* _LINUX_MM_H */ diff --git a/mm/migrate.c b/mm/migrate.c index 35cc9d35064b..0b24021d5fee 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -750,6 +750,7 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio) folio_set_readahead(newfolio); folio_copy_owner(newfolio, folio); + pgalloc_tag_copy(newfolio, folio); mem_cgroup_migrate(folio, newfolio); }