From patchwork Sat Feb 1 23:18:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13956397 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9F9F5C0218A for ; Sat, 1 Feb 2025 23:18:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CE76D6B007B; Sat, 1 Feb 2025 18:18:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C97566B0083; Sat, 1 Feb 2025 18:18:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B5EDB6B0085; Sat, 1 Feb 2025 18:18:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 976006B007B for ; Sat, 1 Feb 2025 18:18:09 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 39C7B121C85 for ; Sat, 1 Feb 2025 23:18:09 +0000 (UTC) X-FDA: 83072941098.25.9468E97 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf25.hostedemail.com (Postfix) with ESMTP id 82316A0007 for ; Sat, 1 Feb 2025 23:18:07 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=fY4sGA2e; spf=pass (imf25.hostedemail.com: domain of 3rqueZwYKCMM130nwkpxxpun.lxvurw36-vvt4jlt.x0p@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3rqueZwYKCMM130nwkpxxpun.lxvurw36-vvt4jlt.x0p@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738451887; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=/Kvraf4w4weerjiWyvf6hTFiVHSzsydM/sRRmUt4qn4=; b=u79Lb9Q8OdkccZwPmNQG0eOXq2bA22J+wEO1z71RDd3nT9COFXP/xE82W4sejfL500Uq8F WvhTS098Im86NS8CFp/Xyl4R8nIsX+orq5Qmhuj9eKp+5L8N6mI1KLKo7GhcluZVj9rqZv jVs4dKKYg3rT5VgpI5WRZ4qKTE14cZU= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=fY4sGA2e; spf=pass (imf25.hostedemail.com: domain of 3rqueZwYKCMM130nwkpxxpun.lxvurw36-vvt4jlt.x0p@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3rqueZwYKCMM130nwkpxxpun.lxvurw36-vvt4jlt.x0p@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738451887; a=rsa-sha256; cv=none; b=Qa62M0q7+tW8DYlO61W2QIsfbQJdmHE0IU5Fd8bueZuVEnCnGQQxexyKL8BbG1VlmZOGds ulGxT+BEYx81ORANbNrYtFu+y3U5WmUhhK4XTWECoLKjYD5P3j7dyFzCYvO4OgzeZUbB9Y IzZqSeC+J2HrbTn+elFimfKGRBqViGY= Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-21661949f23so97945125ad.3 for ; Sat, 01 Feb 2025 15:18:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738451886; x=1739056686; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=/Kvraf4w4weerjiWyvf6hTFiVHSzsydM/sRRmUt4qn4=; b=fY4sGA2e+z5xsInyLnD1ZWLzpe72yxxPlqJwO649YGukm7jma8/I4IBaYUFkqkv/gk jZyQo7N1zMPy4EdHiI+ols26TLjUjZCNupn/XLvgZm9cWohnZi5MnAVABtDhNePAtL2p zBhO4GDgI2k7GINQtk/l6ZV/QeMjCCiUzhHXVe8DX0BXHPkfwN5EC8F/QDKscos+2OQK thIp6M9PNVfVIMzqxosGkA8NuSe5VVWEzwqqD/SqoKIg2CNoZOgVJil5/03mahnoV28j AoX516M5Dxpzh2MP6HOmc72YZHDq0AXC32F36ejDzd26ZW64cBGojiqCmSv5MF1WyRlt i66Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738451886; x=1739056686; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=/Kvraf4w4weerjiWyvf6hTFiVHSzsydM/sRRmUt4qn4=; b=IKoJKyNM3uDubh0qbPExQT4xODHUfMlpbBlXYC9kD9Jh4x2k9V6UO6Rmh6uqiFmczx QpxBYnGiw0qthge9/U+oOQUtGhUuwLeD25cdKtZeBN98W0yOQLEOAcTj8QIml3XD9XT/ fRZIJC09J9C9r7MhsYuLLe/SKfcuMTxXF4/VTGiOXNrTzlXPYcdimE4SEfDrP+hA+Jqb xWqB3bivyHOnct/ychg5KEuxgjP9SBMzhBoK+vGCq25gsHP9eAYOvBLeOY+kK5H4x+5g cwF96KtM2TOgcZEErSjT6VE2YLRohIaVGtqp/HQigT8d91Y5b0UmdHQDL+56oKa34tIv JIRQ== X-Forwarded-Encrypted: i=1; AJvYcCXDdMGleOaaliMVxYUMsS3fhpYziJa/OHusFmHI/0VpyyrVas8AbdNvYUTMqA6jWXKsVGBZz2t44w==@kvack.org X-Gm-Message-State: AOJu0YxIvWI9oGyCEINsqI4SQLxzsX3/GKuQGI2ZxFxI/2uNmfXGVS95 R6BCTLYLWSQOwQwQ9CfhoIAQOVLmA12+1b2mcV7MeEiUlKW9awZkFVsie8WG3+iPZf0Rd5eXxxc fNA== X-Google-Smtp-Source: AGHT+IF7rl+0kYYhaXTekrdVVwheczJIWDrZLww791nC4RwD+CIsfaA9QQuUOCf1vuXCQHD3GZRrfqGvVg0= X-Received: from plse12.prod.google.com ([2002:a17:902:b78c:b0:216:44e0:4ac0]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:244d:b0:215:6816:6345 with SMTP id d9443c01a7336-21dd7c65450mr249942315ad.16.1738451886129; Sat, 01 Feb 2025 15:18:06 -0800 (PST) Date: Sat, 1 Feb 2025 15:18:00 -0800 Mime-Version: 1.0 X-Mailer: git-send-email 2.48.1.362.g079036d154-goog Message-ID: <20250201231803.2661189-1-surenb@google.com> Subject: [PATCH v2 1/3] mm: avoid extra mem_alloc_profiling_enabled() checks From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, vbabka@suse.cz, rostedt@goodmis.org, peterz@infradead.org, yuzhao@google.com, minchan@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, 00107082@163.com, quic_zhenhuah@quicinc.com, surenb@google.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 82316A0007 X-Stat-Signature: zr83xtzu3prj58pzrxs6urgc7hkjmgbs X-Rspam-User: X-HE-Tag: 1738451887-407693 X-HE-Meta: U2FsdGVkX19MVuz1hwzvoR4pnN3TDwHvSWZsBhFIb4EN9KV1nLf6qoQnQoEiyTY5nkRoYL+IqDFH1TAbhjI+J9HSbkKBrMxFSO+/ohw4OG41erVWLGZuxulImmm/QVvLXpzekifka7AalzeVwtQ3Y2wDQ3DPvCB1P1NFmvY8spTqTn4p9T7IxcBf8VYcNHo/GugxDk2YI4JE+hJqs4PHLtCUN5wcxKLdBcJqrrZBItUT+gUrMHLJDH9e4EkdAUZc1fFTHDMzH9I8Xmh8aNNzryl+IeJW//C/8nHQheBT7ShOq/lYspO+P+QNX4oDCAm5yu4cO3IEdIRipTFDEkMKYW1sOmOKSnAtfXOqVJPcagB4WyXK6iUCKa1X3nPDkOfwzmaUXPfwkOavVgo0dfsdgzEnfyuCflnCNH2IrCNOSb0fJggnwvhJnkn/hknHWwBOQEqu8tKgf7+AOdQSvRnfbHXElhIhVY2MHzL8HaSv3wQLyuM4yP/bBPbYU5bDjWIhCbMNVR/d+5VGbhzHYdgow5bggVR3qhpKqV/TEkVvy8aSZYlvNR+2rWNvuYtDaSWMmbHf3ohHCbcagYo7VbYdIj2NdTxiruTpwKCAdOo34QtBUJjZ+bKknsT7GVNvJQpqGIcvh9v41+MtUhz2zfoe3iDQHnwAeWbc2ufL0eE+ncXNJE9ZInFRMvSxAW4WlC2h/omdDaOul6T+m4Ie/oDHwHf2vuR9HpIM9EORkTw66nPHKDlbuAyyYamOmdMFDlBQ4sFCr6Zyi7udlCAs24pUTEVwrC/NMYeMIri/BJWhdScuw9U+Sw0vzN8wgf0VuKQ2Iul8nSmQJIXHJ5hkSpK9tV78CLTLmxPyEJDkkVROG7ivhNlvBviXdGkT/9Tfb3+vANw7BYBbmM0h36fW7YzJNg1VogH799ZMF8d4UxVTWNmxlNMT9VOj2vhQtA3aEdp3LFSlAMy9ZgKfogm+pjB 3l9B458/ 4xwKMeUfmKV8Eu+IVWJ30ZUsqPbuYb2DZ/nd+qGGY8QIe3pV8Hq47IqXhpdrIID0K9cfyRfOoT/x5ssOU734QirZP27hRs8AjzeltRGnjLA4sE8Z6huXUk1PSoHo2NZvL326DGaQgPKYKW0wi9RHM73ep+Kx0BaWgM0F+bOuE7yBpbCGG8xb/2r+7+uEK8r56tKcexscgA8PSLbmoZ1cWBumjQaI8Ifq1Pkb0KyDzOdfRfoScC96qH5IVW2oIX2HduNb2OxT16v52u4tjHpXe19uM7O2OBL3GS8p3qagsTfPtgda2kBhHbe+wjc7xfhDJPTv57XArEkyrVEid3VjHrKP7euVuTEG+uH1fwJHVGV+YnyVcxS4slqlUkbbO//m9dyfjeKkxA9VOsx3Tn3Cj7HOkS0KqeidMPObtVw9Kf+5NUzwh0bQ7nW41LehnyYI5xZgXH5jG2ej4iQ3Tv2E/60pdgFA654Z84MgpY71rTRO/3sAcNenFARMOGRy8mky4UCIr9ybYjpYuYP8bKjXNIpzy7jjWps+r8PGkiGfuamruLWk1lz5KDLfpwS9pEpYSsO37e2VNCn0EWLsZJNsNNM/10w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.005519, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Refactor code to avoid extra mem_alloc_profiling_enabled() checks inside pgalloc_tag_get() function which is often called after that check was already done. Signed-off-by: Suren Baghdasaryan Reviewed-by: Shakeel Butt --- include/linux/pgalloc_tag.h | 35 +++++++++++++++++++---------------- lib/alloc_tag.c | 6 +++--- mm/page_alloc.c | 3 +-- 3 files changed, 23 insertions(+), 21 deletions(-) base-commit: 7de6fd8ab65003f050aa58e705592745717ed318 diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index 3469c4b20105..4a82b6b4820e 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -205,28 +205,32 @@ static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) } } -static inline struct alloc_tag *pgalloc_tag_get(struct page *page) +/* Should be called only if mem_alloc_profiling_enabled() */ +static inline struct alloc_tag *__pgalloc_tag_get(struct page *page) { struct alloc_tag *tag = NULL; - - if (mem_alloc_profiling_enabled()) { - union pgtag_ref_handle handle; - union codetag_ref ref; - - if (get_page_tag_ref(page, &ref, &handle)) { - alloc_tag_sub_check(&ref); - if (ref.ct) - tag = ct_to_alloc_tag(ref.ct); - put_page_tag_ref(handle); - } + union pgtag_ref_handle handle; + union codetag_ref ref; + + if (get_page_tag_ref(page, &ref, &handle)) { + alloc_tag_sub_check(&ref); + if (ref.ct) + tag = ct_to_alloc_tag(ref.ct); + put_page_tag_ref(handle); } return tag; } -static inline void pgalloc_tag_sub_pages(struct alloc_tag *tag, unsigned int nr) +static inline void pgalloc_tag_sub_pages(struct page *page, unsigned int nr) { - if (mem_alloc_profiling_enabled() && tag) + struct alloc_tag *tag; + + if (!mem_alloc_profiling_enabled()) + return; + + tag = __pgalloc_tag_get(page); + if (tag) this_cpu_sub(tag->counters->bytes, PAGE_SIZE * nr); } @@ -241,8 +245,7 @@ static inline void clear_page_tag_ref(struct page *page) {} static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, unsigned int nr) {} static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) {} -static inline struct alloc_tag *pgalloc_tag_get(struct page *page) { return NULL; } -static inline void pgalloc_tag_sub_pages(struct alloc_tag *tag, unsigned int nr) {} +static inline void pgalloc_tag_sub_pages(struct page *page, unsigned int nr) {} static inline void alloc_tag_sec_init(void) {} static inline void pgalloc_tag_split(struct folio *folio, int old_order, int new_order) {} static inline void pgalloc_tag_swap(struct folio *new, struct folio *old) {} diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index 19b45617bdcf..1d893e313614 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -174,7 +174,7 @@ void pgalloc_tag_split(struct folio *folio, int old_order, int new_order) if (!mem_alloc_profiling_enabled()) return; - tag = pgalloc_tag_get(&folio->page); + tag = __pgalloc_tag_get(&folio->page); if (!tag) return; @@ -200,10 +200,10 @@ void pgalloc_tag_swap(struct folio *new, struct folio *old) if (!mem_alloc_profiling_enabled()) return; - tag_old = pgalloc_tag_get(&old->page); + tag_old = __pgalloc_tag_get(&old->page); if (!tag_old) return; - tag_new = pgalloc_tag_get(&new->page); + tag_new = __pgalloc_tag_get(&new->page); if (!tag_new) return; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 579789600a3c..b7e3b45183ed 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4832,12 +4832,11 @@ void __free_pages(struct page *page, unsigned int order) { /* get PageHead before we drop reference */ int head = PageHead(page); - struct alloc_tag *tag = pgalloc_tag_get(page); if (put_page_testzero(page)) free_frozen_pages(page, order); else if (!head) { - pgalloc_tag_sub_pages(tag, (1 << order) - 1); + pgalloc_tag_sub_pages(page, (1 << order) - 1); while (order-- > 0) free_frozen_pages(page + (1 << order), order); } From patchwork Sat Feb 1 23:18:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13956398 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14734C02192 for ; Sat, 1 Feb 2025 23:18:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A02BD6B0083; Sat, 1 Feb 2025 18:18:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9B1A86B0085; Sat, 1 Feb 2025 18:18:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 805056B0088; Sat, 1 Feb 2025 18:18:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 60CD46B0083 for ; Sat, 1 Feb 2025 18:18:11 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 0C1AB140F44 for ; Sat, 1 Feb 2025 23:18:11 +0000 (UTC) X-FDA: 83072941182.24.8EA88AD Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf02.hostedemail.com (Postfix) with ESMTP id 3F09180009 for ; Sat, 1 Feb 2025 23:18:09 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=FfKAqiPc; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of 3sKueZwYKCMU352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3sKueZwYKCMU352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738451889; a=rsa-sha256; cv=none; b=SQg4nOJcRbgncvOYF4g2+gBVZzNxtyfNb5qbn4g8EHLsLqH0Twaboz0pZn6savDv8bNLtx 3hiT+GeH8EnnU1lZWqwFYL8zeFGsSgtvKOnuX6dYZ2RvCRAWO92X613Yxc0ru1HUXo7bpY RHvCjQnkzTdKWzo00wfvEoAl//pmZdY= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=FfKAqiPc; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of 3sKueZwYKCMU352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3sKueZwYKCMU352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738451889; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+LHGz0Xil5zeJ4evV+ex8j8EcdekSjxZZZh0JGQMV40=; b=B8lKTncyD6XPqpTayZ+jJkMrR8rPPZypcnHC8L2sqGbl4pvnVlvr396uSPKVG7QuZ5efn8 LGCI6J6UxOqUwyF4V9mM8Km1KzwGHsoBpN3gEaiTayxCEjFC7ofgu9REc/WyKvFywdexjz CxDA3u6iCaPklcuUtFLS4dpFJH1cTgw= Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-2166f9f52fbso99676015ad.2 for ; Sat, 01 Feb 2025 15:18:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738451888; x=1739056688; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=+LHGz0Xil5zeJ4evV+ex8j8EcdekSjxZZZh0JGQMV40=; b=FfKAqiPcCMCM2hI4YsFSV9cAyLzKAYL9DkzrPk5nFSDcYsuTN/Teu8lZlEoYc9d96W b4LbyshEyHXYn6agVTZs8EeVQf2vpdViQqkMuBZIgoPiaD6B80C8ZyL1Bwcp12RZvzuJ b7kx+2FG6RokxaUagVyLx+mvSklob6kljnKByzpmUUGkz7lF7I5pbjBzQm8+7uA9zORu bb/XIvb9eQ54pAtIKRd/NvdBwIYo4nYaotQl8Zi/biYxJVaK1fxOGo2PIcFVE7h1d6p5 mHT9udKtZE8gumT4mznbDnR+D16iiZxhxBSyg3wC2qZrDy+LY6n2NKYMBnCwI1Fk7koQ zqnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738451888; x=1739056688; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+LHGz0Xil5zeJ4evV+ex8j8EcdekSjxZZZh0JGQMV40=; b=a2dTwYTn3Uf79IiXYK/GEpI0CuJHN31hNToAnXPXfen3vHNUt1R3QL9A/Fz35JVWk9 +RS6EaAg2Cl05nNl4YVSDJT1Gwj5Vj3betW//lVCBwtjKh7diB9cBztkSFqbJ3qWu6TP BM3xOLudr85TVl63mtudNT4ebEb1HJOxbJsnHu+Wx33vgPfSyiz4MuL0qeehb6GjI49V U6PqByO3ZhmCAiKUbyFIVyZUrKB3Xn9uBxdnqtg8vRW/AgZPfb2DudkVHfmi8k1O2dg4 rf2TF5RnCwgPJ9sUBTaQEArbfrOp8qIGeR3gaVYCr49YiMGRk2OUd448jUtICgrujQOn E3zw== X-Forwarded-Encrypted: i=1; AJvYcCUeNanfD1d7n3ld3rKnVh0xesp2OV4c21hTeCQeYOIV1809e2d8CXGBbBPG4+eFUe/70A8yO/x6NQ==@kvack.org X-Gm-Message-State: AOJu0YzXI6meECvK581sjqtw4B3LU/yjoo9lst/rvjpLyA72r8WvjpVX wQeSdHpKb/6jDow4cNj9jnIR9HO57mj/4+6UyYpSrWPqL9iFxzQOVStKTTnqctWqF60M1J9X8w0 oGw== X-Google-Smtp-Source: AGHT+IE2vxYeAR9419oUgZ2ihycmI1jsvSuPTWn2lYlIeN6Pbw/ZzCEAbsoRXT2NV98TB1fIQxgW3UhypMQ= X-Received: from pjbpa13.prod.google.com ([2002:a17:90b:264d:b0:2ef:8055:93d9]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:c941:b0:215:5ea2:654b with SMTP id d9443c01a7336-21dd7c49718mr257314855ad.1.1738451888074; Sat, 01 Feb 2025 15:18:08 -0800 (PST) Date: Sat, 1 Feb 2025 15:18:01 -0800 In-Reply-To: <20250201231803.2661189-1-surenb@google.com> Mime-Version: 1.0 References: <20250201231803.2661189-1-surenb@google.com> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog Message-ID: <20250201231803.2661189-2-surenb@google.com> Subject: [PATCH v2 2/3] alloc_tag: uninline code gated by mem_alloc_profiling_key in slab allocator From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, vbabka@suse.cz, rostedt@goodmis.org, peterz@infradead.org, yuzhao@google.com, minchan@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, 00107082@163.com, quic_zhenhuah@quicinc.com, surenb@google.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 3F09180009 X-Stat-Signature: 9t9ah4j5jd6t1wnphdfxngtghjzyu67x X-Rspam-User: X-HE-Tag: 1738451889-449091 X-HE-Meta: U2FsdGVkX184ApUNld7+UoXY64PBNkEJiHCQt6MXzdvike+dZNM1aoDROi2LsVMKLF8mldG4GoqFAMjR7lcj71BkoNrP8GVNPp4JX1tQyAE2dXu1yYtTpAbIXc66N6jc48j/A+GmwqHgHcHqOo/Pi5MhwGibxFTQpADHG8mf7s+3soow4OOCeS6CabY7lm38R8+YVqrReAn6EBtoZoz//gyShIdbX6UYPA4+0ueXzUvHF42C6PHFIggSr5a5U2jTgjRRZlTJuF+nubWo9/2i6QtvSZfWoNxLHi/8SZ3kWUMU0ftbwFWA/R3TiNnSvCabYOSFhoX3XCg1sR515q92daPtPRue11GkcP1ntId7LPSmoKVTVqG6syVkcmAB+dr+bMcjUUwhAf/qlGoGQmtXyXQRnbatHCeuMKZ2d8Bk8hvhSRzD0HWG9b7pRcpzd1VNdedsqM8K9KYMJ4kuMHDntPqxZ8xgyKVG1B0DK+hRKB5yGMoRLP+CDvXSUzIQ4o+oevFSpabaKzvn1stE/cafiFWrgLJiygKqDcyJIX/MGVE7EDWiYTvHuax3xTXFahG1inR4JyeZBT1Sct/jUAIChkxPhUJ4TIcQiRHXcDd1yDJNdMe5X6BH19lXDno3y3bq/W9uEWr+Tn6dw/V11XDoHj8HDTNnzb9gv8CA+ta4jo+4RZ6SQTLx+EsS0R1ue5rGxrTfm0NFHmJOIY9zFI/WpXXvO8XKkYbS3+n4k2DIuceR6caDyXjZzcORIoKBthcq2HibV70qwqpEWB6Hc77PVSauGwIVm7BVeA8ndRJYfmKF6AAnarZmZARy4LtIlFs1Q2/iTRW0IgFoPSXF5SnHSQAHNB7Z7cbsHBdABedxJLDuphSm222oTEss9bn8nktI+11nWHT0yx/Bro58SwRv4aWk0NLApupeP+KnJH0GAb/KoGW+V1nqzHcsxtJLjUFWn1MUFhATy/nl3hNvOfo X6m2Yf8p FKtDJT35/hgmhfpK82EmaMdCswf4i8yp826zZeqiv4RiiCJG6/gTog5Mrp9Bm0rZUW1iSxFZcvlxDdZX3WVqVmIGB7tail9Ztj/d8/UqF01lsqepVqZRAcOtZ52UF4t4m/kBIT2CXCLbDtU9CI3MjMvTa7Z5LFOrDDB+EhBHVBfAdfNauATZuhY3xMdxQox3IB/gGEdL/DcI62qRBNfvP8EQ0d+DQFnzfqqvjRRbF5MCi5agAI8/BJDHVS5mGMSuG3kEfBgEu7evL8fQPBhh8emYO8ALjLB+3mQ70Dl+Vi1nBDuhZoyvyQH0wKQJvghZ1xfH99kXsOTPGEMz/HRu9aIxQRrEb/WxOpRkYlU8n4jKAttxtM852xAq1qoYcKcFIMl/uX17VDqR35x1SiJxnC2y5p4UdSvWcU9K6Dx3lI9Yp43ni4wi94jGe6K7eFpGQ9ZM6NT8vYbQCp1/c31YZHh2zG0Cfh3QX1dmyEmV0zloj3y/go9GyEn3ySFWOgrs6BmnAcN2MQQH0SiEAMGqkQ7/QmXNsE3pka2qtUkfvf1y9MoKvLMF2Yjb7lG1lW3r0lbDyxSRJl6xgzFMyPI/P7FUdWApw+6VdlbTMsBKeIzloHM8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000047, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When a sizable code section is protected by a disabled static key, that code gets into the instruction cache even though it's not executed and consumes the cache, increasing cache misses. This can be remedied by moving such code into a separate uninlined function. On a Pixel6 phone, slab allocation profiling overhead measured with CONFIG_MEM_ALLOC_PROFILING=y and profiling disabled is: baseline modified Big core 3.31% 0.17% Medium core 3.79% 0.57% Little core 6.68% 1.28% This improvement comes at the expense of the configuration when profiling gets enabled, since there is now an additional function call. The overhead from this additional call on Pixel6 is: Big core 0.66% Middle core 1.23% Little core 2.42% However this is negligible when compared with the overall overhead of the memory allocation profiling when it is enabled. On x86 this patch does not make noticeable difference because the overhead with mem_alloc_profiling_key disabled is much lower (under 1%) to start with, so any improvement is less visible and hard to distinguish from the noise. The overhead from additional call when profiling is enabled is also within noise levels. Signed-off-by: Suren Baghdasaryan Acked-by: Vlastimil Babka Reviewed-by: Shakeel Butt --- Changes since v1 [1]: - Removed inline_if_mem_alloc_prof and uninlined the gated code unconditionally, per Steven Rostedt and Vlastimil Babka - Updated changelog to include overhead when profiling is enabled [1] https://lore.kernel.org/all/20250126070206.381302-2-surenb@google.com/ mm/slub.c | 51 ++++++++++++++++++++++++++++++++------------------- 1 file changed, 32 insertions(+), 19 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 1f50129dcfb3..184fd2b14758 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2000,7 +2000,8 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, return 0; } -static inline void free_slab_obj_exts(struct slab *slab) +/* Should be called only if mem_alloc_profiling_enabled() */ +static noinline void free_slab_obj_exts(struct slab *slab) { struct slabobj_ext *obj_exts; @@ -2077,33 +2078,37 @@ prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p) return slab_obj_exts(slab) + obj_to_index(s, slab, p); } -static inline void -alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags) +/* Should be called only if mem_alloc_profiling_enabled() */ +static noinline void +__alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags) { - if (need_slab_obj_ext()) { - struct slabobj_ext *obj_exts; + struct slabobj_ext *obj_exts; - obj_exts = prepare_slab_obj_exts_hook(s, flags, object); - /* - * Currently obj_exts is used only for allocation profiling. - * If other users appear then mem_alloc_profiling_enabled() - * check should be added before alloc_tag_add(). - */ - if (likely(obj_exts)) - alloc_tag_add(&obj_exts->ref, current->alloc_tag, s->size); - } + obj_exts = prepare_slab_obj_exts_hook(s, flags, object); + /* + * Currently obj_exts is used only for allocation profiling. + * If other users appear then mem_alloc_profiling_enabled() + * check should be added before alloc_tag_add(). + */ + if (likely(obj_exts)) + alloc_tag_add(&obj_exts->ref, current->alloc_tag, s->size); } static inline void -alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, - int objects) +alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags) +{ + if (need_slab_obj_ext()) + __alloc_tagging_slab_alloc_hook(s, object, flags); +} + +/* Should be called only if mem_alloc_profiling_enabled() */ +static noinline void +__alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, + int objects) { struct slabobj_ext *obj_exts; int i; - if (!mem_alloc_profiling_enabled()) - return; - /* slab->obj_exts might not be NULL if it was created for MEMCG accounting. */ if (s->flags & (SLAB_NO_OBJ_EXT | SLAB_NOLEAKTRACE)) return; @@ -2119,6 +2124,14 @@ alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, } } +static inline void +alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, + int objects) +{ + if (mem_alloc_profiling_enabled()) + __alloc_tagging_slab_free_hook(s, slab, p, objects); +} + #else /* CONFIG_MEM_ALLOC_PROFILING */ static inline void From patchwork Sat Feb 1 23:18:02 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13956399 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14D19C0218A for ; Sat, 1 Feb 2025 23:18:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 98DCE6B0088; Sat, 1 Feb 2025 18:18:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8F1E46B0089; Sat, 1 Feb 2025 18:18:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7426E6B008A; Sat, 1 Feb 2025 18:18:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 4C1F86B0088 for ; Sat, 1 Feb 2025 18:18:13 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 07C8CB21F5 for ; Sat, 1 Feb 2025 23:18:13 +0000 (UTC) X-FDA: 83072941266.08.FC178A4 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf09.hostedemail.com (Postfix) with ESMTP id 3621B140008 for ; Sat, 1 Feb 2025 23:18:10 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=AcXptLa7; spf=pass (imf09.hostedemail.com: domain of 3saueZwYKCMY463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3saueZwYKCMY463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738451891; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=REEyzR4VwcXBG/MiulZy61h2cqis6B2iwbKUK+/Z7ro=; b=16TGjUDJhGdCc3Xdbc5tkqXG0cik/ScMtLvLtC60hZAXnsn3jeJ9IIappTzIgMbmDxNx6a wn2uvf88G6frUlzj2+6bV0fNeVcd3d6IyhmoG+zuRFuMJGJgLCjyrc3atCambcnQeDdCr7 rXonVeNAlHkwqE51h7sTiT4xIpuJyTA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738451891; a=rsa-sha256; cv=none; b=RR/9e1Wz6YcdViYzheTjfjsin7hDxB0JA2LN/ZYmGkR7YvDelmM8q6YTOCe8c+TN95AIn0 v0DlK+dJ1RutP9ivbbpW/080zmZFRpJZg5mbLjpCtZgb3U/vbXhm42O18R8rWqktwX9EEU uDnclTp+TWj+hSyWH7fO6ZnNWZzPdkk= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=AcXptLa7; spf=pass (imf09.hostedemail.com: domain of 3saueZwYKCMY463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3saueZwYKCMY463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2eebfd6d065so8996801a91.3 for ; Sat, 01 Feb 2025 15:18:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738451890; x=1739056690; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=REEyzR4VwcXBG/MiulZy61h2cqis6B2iwbKUK+/Z7ro=; b=AcXptLa72a2pLuG37EDmKfBj/wdHxjLDgeH7x17Txb3y0/lJ473iy/WzbhEwsoQk08 HbQXCEsIb7kY3P43TVOlp0bQUAGoA1lIiKkrtb6kImugMhy3gVbF7rd9E7yHJFJLlop9 t46fj9cj5JiULwbSHtdVyVpeEoZJ1UiMuS+tVGLQVT5rIx4vr2Rk0YrRzRKqfAYwKlCL h0U7qy2Tk0OZgdtz5Oook999+dGXBvTWWslBDeslSVo5ylDjifarHeInHo/ZPhJRPFFP 01IrD2KTLFtG3IhZTPeubUZYunx0ethlbqeNsSJGO+zE5c8cuReUHtOQ2+zuJchHRjGe pp6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738451890; x=1739056690; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=REEyzR4VwcXBG/MiulZy61h2cqis6B2iwbKUK+/Z7ro=; b=IP5VqOUWSUM7UvusOwydnsB5r6nP/MMCW5TgA9g50HA+n3VZFpBR0r2XUCxrP2+ogm DzGkaQ+C1KMjFCoLGNN9aZY5AkJ2QOCF1lrC+9qGv2koFjzpp9DxUz3vqjKbIEbjTXBi dkAVQ+l7DDEEXESU0Yd8iL7tb1im5iuUuDflhwFrWeAXJT0vg/t/euDmrMsEXRF8y5DB P3/zMY1McDBuk1XLC6/LC7EuLSZRtlWYmmrtSYuhjYbecLHWHRLMIp+qF2dUGktoSaOJ NkSjw41CAn/cDHEFr7ULdqtqHpXxdj+ZNnn6eNMUyGVUag2giE5eJYAEpT6SarT6PgiE EYNw== X-Forwarded-Encrypted: i=1; AJvYcCVf3VcEBGpp/uhkqzlFmJkUtc6SIWcMBbwBsIjMCkrbfBmrGPre2fwA++oegYdn2b+9JR+Pv9bQ3A==@kvack.org X-Gm-Message-State: AOJu0YyZCuYnrK1CIjS+BqChyLNtvayRCUFqS1JADOmfIE1MLqBCOJHe iAZg1LkkOy8LQgFUW/bXukGnnHnx1ZLt4qMdk0QLDzSVTeiYKSy8Z6cLsktf2OT7iZ97dj6rp+q rvQ== X-Google-Smtp-Source: AGHT+IH3XLGWaDaT1t9/Cp7o6c+RF7ry2yIsdfJHITI/fssp3IRkvyZrbgiqnj5YWj1ktwSp36mfOaC0fRY= X-Received: from pfwo11.prod.google.com ([2002:a05:6a00:1bcb:b0:72a:bc54:8507]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:aa7:930b:0:b0:726:8366:40ca with SMTP id d2e1a72fcca58-72fd0bbd6dcmr22976159b3a.1.1738451889825; Sat, 01 Feb 2025 15:18:09 -0800 (PST) Date: Sat, 1 Feb 2025 15:18:02 -0800 In-Reply-To: <20250201231803.2661189-1-surenb@google.com> Mime-Version: 1.0 References: <20250201231803.2661189-1-surenb@google.com> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog Message-ID: <20250201231803.2661189-3-surenb@google.com> Subject: [PATCH v2 3/3] alloc_tag: uninline code gated by mem_alloc_profiling_key in page allocator From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, vbabka@suse.cz, rostedt@goodmis.org, peterz@infradead.org, yuzhao@google.com, minchan@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, 00107082@163.com, quic_zhenhuah@quicinc.com, surenb@google.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org X-Stat-Signature: dio44dqu4yjwdb5mjhj93k94fjiij58b X-Rspam-User: X-Rspamd-Queue-Id: 3621B140008 X-Rspamd-Server: rspam03 X-HE-Tag: 1738451890-555478 X-HE-Meta: U2FsdGVkX18iM23ipA9mrvUxZ0Xq+//bmZeXa8OLXrr41we4PTdcyQkkUm+xDH3kYn9p0U4GdHBvWpJEz3XAByKqxfJ/DYc69RNogelk66+iA0i9pcAqDC7+zNdUKN8PQ1qwfX27GEL2AZOYvCYYa3FYgi0rAurb9b1pgL5BQjmEZaiZiaYTv0bc0XbX2zu3HNL8FztMCW8n4+uAHhAJ4rTKjqYTM/WxFPBzSl9ypvUd/lDbAGLXgcSH3P04j6BooLaAaO5Zuk9tEinay7cSnkKmAIf5eKR+4Zb7XhYLvjSOplvHQ3dZC1h/cJBUHMa9Z+88f0aXclksSwL3/dv+1yJ9AWUz9lNnFyxcvm/l+g6TSSFYEXtBLMhOU9bFpjSgbBECO+F1KChh+WyR1t82u/MVunDtDyNBEoddikN5gTx2eP9DI/tsI4Q+r3/e9VHCFI4IOBHo8L7B+EJiE6IwiqOum7P2TfX6ZyBv5GQmUKCt1m/dhTf3qmrp29g7hy4TjQdkoI8O+2sGaT7Phk0kUMhQOTNROnG+2jULsuo1i+jReaVh0Nbik82cwkCcRvvQHVc7FFfZ0O8dksht5rG6FaW4pEdi4Tupnxb3UXGeZMT/qLVVpu434DxK+rzi2o0AghhX1IkgwpYZBYjlqxnbqrrbsMZl5K4lEV861BrlwldxUNy23BrHjLuYUtWQOXrRIiSwK3Gbm6PPUaFvqCLcnLYU8BoonENOZAjzkKV6/RKTn6fIMBYRsBgWopIFC6pOZ2rHGKa94O5VwsyvF4bhaVOF7C7FnWYSFv2if2Gla8/w958fD39IFLSxTC/Mk2z1zTLVdGZcz8nq2zjS/qHWrdos8Nw6RRpKYa8NjPH+d8hUkTtupyJ30eKKU3sjYAMRf1RYWatvNwth6Rjjv7Jn9MXRGZI+Ue1Lnok+Z9ldLTDy7NmwZ/kkR5zkrUOXGUc2AV5cQ3I65YGNOYB6/eU E6OW0xMC fy0FBDjY9SZGDmjsfVKo5pWq0ECVTGis0e+nzNqeV/EEzPyMSaHDceafI/P0ARO2KzPFLcrbjJJEj0rGUGx9QfdpC26A6z9EtxGBHAogrB/UcovATNHdUtS5tKAUeFw9XJaghAMiJ6HFDsBeLuBbeOWr8MU3qITn9/ft6KVbf5V01CNvOf9vlPRzPH8TxHEu4BKP+QiKvjilyvxGAo7LKg3hMXbwwAVSFx9G2f31tpPhukMDA5j85PSIDKW2tqZcazYmTx12w+Wa3hJsAwc82j5mrIypgNkrOQRisICu+SFzc7hYQT8boQmrBXtbVyZgLhozJC8R4Xv3SNGzarUc0OnjOZ6000359T6SK3X2XZhkemfCAXTOoLj9aMzFnbszAnselDk2RX43OXF52tjJ8CTTe4H82fT+HqiV5LEV8ol638iNrTEV3977YwOOkiNlyZTBcQ7qh/H5aG5UjhZF3mGCSKHkkSwfwOGpmblv5F4T6uCc7XcLKu0v6Z9GQXe20ry2nbcwbJJ278JS1r/T5BgZnYAwIXOmAGZqq3Y+Qa0QGBZSxQFEPtu7W/YpLUYiiGgfZc2SrjDoE3uA07tN6eIAvgqO0N5T0tJF08elN5aQ6oDk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000013, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When a sizable code section is protected by a disabled static key, that code gets into the instruction cache even though it's not executed and consumes the cache, increasing cache misses. This can be remedied by moving such code into a separate uninlined function. On a Pixel6 phone, page allocation profiling overhead measured with CONFIG_MEM_ALLOC_PROFILING=y and profiling disabled is: baseline modified Big core 4.93% 1.53% Medium core 4.39% 1.41% Little core 1.02% 0.36% This improvement comes at the expense of the configuration when profiling gets enabled, since there is now an additional function call. The overhead from this additional call on Pixel6 is: Big core 0.24% Middle core 0.63% Little core 1.1% However this is negligible when compared with the overall overhead of the memory allocation profiling when it is enabled. On x86 this patch does not make noticeable difference because the overhead with mem_alloc_profiling_key disabled is much lower (under 1%) to start with, so any improvement is less visible and hard to distinguish from the noise. The overhead from additional call when profiling is enabled is also within noise levels. Signed-off-by: Suren Baghdasaryan Reviewed-by: Shakeel Butt --- Changes since v1 [1]: - Removed inline_if_mem_alloc_prof and uninlined the gated code unconditionally, per Steven Rostedt and Vlastimil Babka - Updated changelog to include overhead when profiling is enabled [1] https://lore.kernel.org/all/20250126070206.381302-3-surenb@google.com/ include/linux/pgalloc_tag.h | 60 +++------------------------- mm/page_alloc.c | 78 +++++++++++++++++++++++++++++++++++++ 2 files changed, 83 insertions(+), 55 deletions(-) diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index 4a82b6b4820e..c74077977830 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -162,47 +162,13 @@ static inline void update_page_tag_ref(union pgtag_ref_handle handle, union code } } -static inline void clear_page_tag_ref(struct page *page) -{ - if (mem_alloc_profiling_enabled()) { - union pgtag_ref_handle handle; - union codetag_ref ref; - - if (get_page_tag_ref(page, &ref, &handle)) { - set_codetag_empty(&ref); - update_page_tag_ref(handle, &ref); - put_page_tag_ref(handle); - } - } -} - -static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, - unsigned int nr) -{ - if (mem_alloc_profiling_enabled()) { - union pgtag_ref_handle handle; - union codetag_ref ref; - - if (get_page_tag_ref(page, &ref, &handle)) { - alloc_tag_add(&ref, task->alloc_tag, PAGE_SIZE * nr); - update_page_tag_ref(handle, &ref); - put_page_tag_ref(handle); - } - } -} +/* Should be called only if mem_alloc_profiling_enabled() */ +void __clear_page_tag_ref(struct page *page); -static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) +static inline void clear_page_tag_ref(struct page *page) { - if (mem_alloc_profiling_enabled()) { - union pgtag_ref_handle handle; - union codetag_ref ref; - - if (get_page_tag_ref(page, &ref, &handle)) { - alloc_tag_sub(&ref, PAGE_SIZE * nr); - update_page_tag_ref(handle, &ref); - put_page_tag_ref(handle); - } - } + if (mem_alloc_profiling_enabled()) + __clear_page_tag_ref(page); } /* Should be called only if mem_alloc_profiling_enabled() */ @@ -222,18 +188,6 @@ static inline struct alloc_tag *__pgalloc_tag_get(struct page *page) return tag; } -static inline void pgalloc_tag_sub_pages(struct page *page, unsigned int nr) -{ - struct alloc_tag *tag; - - if (!mem_alloc_profiling_enabled()) - return; - - tag = __pgalloc_tag_get(page); - if (tag) - this_cpu_sub(tag->counters->bytes, PAGE_SIZE * nr); -} - void pgalloc_tag_split(struct folio *folio, int old_order, int new_order); void pgalloc_tag_swap(struct folio *new, struct folio *old); @@ -242,10 +196,6 @@ void __init alloc_tag_sec_init(void); #else /* CONFIG_MEM_ALLOC_PROFILING */ static inline void clear_page_tag_ref(struct page *page) {} -static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, - unsigned int nr) {} -static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) {} -static inline void pgalloc_tag_sub_pages(struct page *page, unsigned int nr) {} static inline void alloc_tag_sec_init(void) {} static inline void pgalloc_tag_split(struct folio *folio, int old_order, int new_order) {} static inline void pgalloc_tag_swap(struct folio *new, struct folio *old) {} diff --git a/mm/page_alloc.c b/mm/page_alloc.c index b7e3b45183ed..16dfcf7ade74 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1041,6 +1041,84 @@ static void kernel_init_pages(struct page *page, int numpages) kasan_enable_current(); } +#ifdef CONFIG_MEM_ALLOC_PROFILING + +/* Should be called only if mem_alloc_profiling_enabled() */ +void __clear_page_tag_ref(struct page *page) +{ + union pgtag_ref_handle handle; + union codetag_ref ref; + + if (get_page_tag_ref(page, &ref, &handle)) { + set_codetag_empty(&ref); + update_page_tag_ref(handle, &ref); + put_page_tag_ref(handle); + } +} + +/* Should be called only if mem_alloc_profiling_enabled() */ +static noinline +void __pgalloc_tag_add(struct page *page, struct task_struct *task, + unsigned int nr) +{ + union pgtag_ref_handle handle; + union codetag_ref ref; + + if (get_page_tag_ref(page, &ref, &handle)) { + alloc_tag_add(&ref, task->alloc_tag, PAGE_SIZE * nr); + update_page_tag_ref(handle, &ref); + put_page_tag_ref(handle); + } +} + +static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, + unsigned int nr) +{ + if (mem_alloc_profiling_enabled()) + __pgalloc_tag_add(page, task, nr); +} + +/* Should be called only if mem_alloc_profiling_enabled() */ +static noinline +void __pgalloc_tag_sub(struct page *page, unsigned int nr) +{ + union pgtag_ref_handle handle; + union codetag_ref ref; + + if (get_page_tag_ref(page, &ref, &handle)) { + alloc_tag_sub(&ref, PAGE_SIZE * nr); + update_page_tag_ref(handle, &ref); + put_page_tag_ref(handle); + } +} + +static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) +{ + if (mem_alloc_profiling_enabled()) + __pgalloc_tag_sub(page, nr); +} + +static inline void pgalloc_tag_sub_pages(struct page *page, unsigned int nr) +{ + struct alloc_tag *tag; + + if (!mem_alloc_profiling_enabled()) + return; + + tag = __pgalloc_tag_get(page); + if (tag) + this_cpu_sub(tag->counters->bytes, PAGE_SIZE * nr); +} + +#else /* CONFIG_MEM_ALLOC_PROFILING */ + +static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, + unsigned int nr) {} +static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) {} +static inline void pgalloc_tag_sub_pages(struct page *page, unsigned int nr) {} + +#endif /* CONFIG_MEM_ALLOC_PROFILING */ + __always_inline bool free_pages_prepare(struct page *page, unsigned int order) {