From patchwork Sun Jan 26 07:02:04 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13950574 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A37D2C02181 for ; Sun, 26 Jan 2025 07:02:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B53322800A7; Sun, 26 Jan 2025 02:02:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ADC782800A6; Sun, 26 Jan 2025 02:02:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A38A2800A7; Sun, 26 Jan 2025 02:02:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 795D02800A6 for ; Sun, 26 Jan 2025 02:02:13 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id D9D5FC1312 for ; Sun, 26 Jan 2025 07:02:12 +0000 (UTC) X-FDA: 83048708904.29.8D32258 Received: from mail-oo1-f74.google.com (mail-oo1-f74.google.com [209.85.161.74]) by imf12.hostedemail.com (Postfix) with ESMTP id 2D7384000D for ; Sun, 26 Jan 2025 07:02:10 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="0Rv/kO1w"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 38d2VZwYKCEg241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com designates 209.85.161.74 as permitted sender) smtp.mailfrom=38d2VZwYKCEg241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737874931; a=rsa-sha256; cv=none; b=P0/lqnGkP/5eDiN61/3I5zWaKHW7hkkjvxRh1ODKAD3fR9q+BxJ/fOaqkIQFiE727gyFO+ bcWY1rCrdNYQO9+uLytlo/5b5gyOMlNxxeDXFVa9s0VS3ZXN7tvP2FK2eFzQo9r9NbDyb/ LFlMne5EM1SuvkX1ENLxVf6JTWP8dxo= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="0Rv/kO1w"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 38d2VZwYKCEg241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com designates 209.85.161.74 as permitted sender) smtp.mailfrom=38d2VZwYKCEg241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737874931; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=Gjibp4HHI09y13Cb+jMTPU7WntuA24yRPhNo2MxmLn0=; b=H6XtuVDUbAoRbhVrV5x0++B9EDwsr0Gn8TOoo2FQw0oQb3RzuabC4AaC4obu6zrUv3hapm gchQlizlyf0C+yFodWgCiYB89qt6w5xsIr/DTvEExd2tI/LJHaoIKdersyXONmcOeTUjuM j/XheSZrdQltomfKBHtdXaTlSfRZYpA= Received: by mail-oo1-f74.google.com with SMTP id 006d021491bc7-5f333a71d91so1252074eaf.0 for ; Sat, 25 Jan 2025 23:02:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1737874930; x=1738479730; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=Gjibp4HHI09y13Cb+jMTPU7WntuA24yRPhNo2MxmLn0=; b=0Rv/kO1wPvD2mP+5mWCSRjWISxtUWdnd4nn6KrEZmavJvHL24lC2jsdp+oMDuNZfpS Nthqvzojl7UXDl1/PLRk1DTHZMv2gykaIM9iD6c/FHPgQb0k3Us6fhKRZkWkp0GJP1K+ esMKW3QcBETDrZy0OIhEdh+hazCXccScQBoWVf8NIDNblmoB6hnyxkKzUL6Cjx81yXKP fKa5xLxT06EVa7wHg30lO8kxbBnTnyl02a4N4rlsh9dkLLM6XYBs7m1XnLr5yEfSREMM hvuV62h1lUyfcY4riaC+yk0QwmMTD6z3qwPNdcFQGd71KrK1M2YzKppr0m+yPd1Jt4HT GqHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737874930; x=1738479730; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=Gjibp4HHI09y13Cb+jMTPU7WntuA24yRPhNo2MxmLn0=; b=oQrAYESSdeqds4u5akb9bDghTdCfOm8n/XgPGW3rj5pgR2O8kXnwfyk9f9reDoV0R4 PmHujft2+29JwFadwUpKQluPZHHN3lzrW/WOtsYOdsTuGEgrLN31a+LooImQ5gZ6OmrG 0JzZt8cEl3X3B57VqBFNhY0bZq74uLUHBSaatSYhaj8HTu4WbwntMi6XxpjLQIh/2nSO j+0uA1MHa/+HFIQc/cGB9tjT6rvf+2FU96GnTb0sqjnlwzIEhbZPmBR5S6Cgy9+wDTSp HB8j65QXQ62At9hcrj9yiVB0rSkbyfXusv9x3eWaZHNBdw4WXwa+lMLLUJKwxOziBzrg 7sNA== X-Forwarded-Encrypted: i=1; AJvYcCXcleCtGxPQgqx0IDnbJVsWql6EGhpHugqO6S4QKZy3cl+zfht95gUkNRBCFcqOezVi9W6r5msbEQ==@kvack.org X-Gm-Message-State: AOJu0YwpiuVC1lQv25Vw24j9eJsK09sBTNdNIEWVqjJZfmNA3uPUuGA+ spUEOfl3RGL8raRznNcusY7CvFopkd+6adiZY/AJ+xVUKJoL6TZiJtQBFOQKtLB5xex83gVxusv sCQ== X-Google-Smtp-Source: AGHT+IFx729Tm/RHbxsPZVc8EhlfE/xBJUd+ZMgbbEChjn50lBiPKyouSgWjqxGyD7UokRCh6tCkDMiH8jU= X-Received: from oacpz5.prod.google.com ([2002:a05:6871:e485:b0:29e:147:e258]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6870:4e0d:b0:29e:2f3a:761a with SMTP id 586e51a60fabf-2b1c0a349f5mr20640727fac.20.1737874929840; Sat, 25 Jan 2025 23:02:09 -0800 (PST) Date: Sat, 25 Jan 2025 23:02:04 -0800 Mime-Version: 1.0 X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog Message-ID: <20250126070206.381302-1-surenb@google.com> Subject: [PATCH 1/3] mm: avoid extra mem_alloc_profiling_enabled() checks From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, vbabka@suse.cz, yuzhao@google.com, minchan@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, 00107082@163.com, quic_zhenhuah@quicinc.com, surenb@google.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 2D7384000D X-Stat-Signature: rudppqqr6oh8ijmt5tapgpxps4n4ym98 X-Rspam-User: X-HE-Tag: 1737874930-37603 X-HE-Meta: U2FsdGVkX19a3opnVCmGRGqKHyPVjD2T8nxUfVxi9kGebGME5jOUh+nyS1KnM7U9MhUHplrMgAz9unS7IUusM3lOqu7o/XZ5V83iQ5IOAwG0JCbEt8HjxyApQhdIyEbEs7hyPbyTDun8HLIvuiCY+wX0VS/R40ydR2njOzxqt81cUNqFCfanLt4QCw4+fTiVPFUOxHdNuVRByjXETA0foYrObVY+Gb3bep+GQ409xqVlT0Ef4LzQShC0PIDR83qLB0vNCv13+OKzTbd+JkemgwOUoHJqUXNhxxeexWXdxJ7f8UTrRA1LDuKgZvjaoGUe7YaRr4tf/IMYIVQcEfvmy1GOBbrtv0fqo/V327QyuK1isgqtkuqaj7XzSf0kGLFD2aKJocE/aeHvYa8L6BEy+UgNPk6vnjFtUyJ0hjPIq4j5ron/4ORbQlihCtFEp8jtE/nLRGccHixJSNpzXX2DpAjqkYk2Hnl2pBsK90f1sFdhUPd1KRSUlTAhhK/2nfKITHlJJY/GvJLhP8EeIFpDH4tbKJzUuLBmPk52XT4FwgqCPQejsGRYYdTGVB7cv48Uu4MEU4JrZrnIasyz51Kc7oj7iy0RTCrZ/Y35WTE+p2bbwiOKT7QGAq17bT6vyfyWdTs/6WoSgkoivxEEjfQJh+c25pVu2dSjChO+FJxP9pUnAJspR/Wpnw3QzuTquSNKXAFnCoaBqonR3RYjgVNIYmc2Ij2QT2ifRv4TXdhIpuERt3TDLnzPCNSp1Z5OWYKxdbZ6TNome6KwhfhlSEM9yB62F+AJV72PwOsWYqHwETz50NniFMToO2XkFHStRzh1OVTBySf6f6Kdb9cIXtjjdJx177YFzkeseZsjlWX2bRh1vu1X0RbaQ3OcUV0httIJOUmzb5n7IvDBRAk3prmSSQR7raUy6YS7WO9/8qCzZu7DjbKsnzjpcYdIPsMq8f7Jv+kOdGyS8TpTas5wpnj e53QfDem pyRl4fFgsKfH95qOR9ZTElN5b2UAyJ+9lTsP0vR6OJM1mzEIhSJWmB8+J+G5mA5r2rqwhBNgz8IoKozdOGA3U2bi2Bsdc7Mp7StPlYDJTNwmCPqvgPz2PEI7iYRb1ZwCu0A9kBxDO1zESIDwj71+1EwZCHnBAnK4l2XaaWkv2fgHHlfpn+cSgU64Wk3W3vIpSeekb2Nr0L3NwBvDt25L43IrWF815xwArLOMm0DbEOmuK82jnoSj3MtoAqKOna+0SbhCZjUTKisTRtSlHX0A3AZHqnrq95dIAY5dEWt0gdjtjArrq4BQUzFn+3erPtjofWwUG28gY2yo0O7dGV67l6xLoJ4Gk+j/jIgposFIjw9kLLJ8DHcVnrGl8guZivMAR8lV2QY73/PhBxo31ASQ+w4UYoWxQSo5xMJogDVmIOdTFE43Xe4D0fsyK4C00ePGktvaiFw9qYzj+1NWcFwAf5cK769w7tNT0CQvUwvovsNxMPGhH1irRiqrMubDELjP3UaL0Ss0YK77MTqBvfmBuHhCeapTJCIST31KggFZWpUBtZFf9KGRGTe18YWxOTVLLV2Wed7x1lKjgIeM0++9vjkWkJQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000083, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Refactor code to avoid extra mem_alloc_profiling_enabled() checks inside pgalloc_tag_get() function which is often called after that check was already done. Signed-off-by: Suren Baghdasaryan --- include/linux/pgalloc_tag.h | 35 +++++++++++++++++++---------------- lib/alloc_tag.c | 6 +++--- mm/page_alloc.c | 3 +-- 3 files changed, 23 insertions(+), 21 deletions(-) base-commit: a227a25fe3fdd08f743a20a6bf6367a47b20c125 diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index 3469c4b20105..4a82b6b4820e 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -205,28 +205,32 @@ static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) } } -static inline struct alloc_tag *pgalloc_tag_get(struct page *page) +/* Should be called only if mem_alloc_profiling_enabled() */ +static inline struct alloc_tag *__pgalloc_tag_get(struct page *page) { struct alloc_tag *tag = NULL; - - if (mem_alloc_profiling_enabled()) { - union pgtag_ref_handle handle; - union codetag_ref ref; - - if (get_page_tag_ref(page, &ref, &handle)) { - alloc_tag_sub_check(&ref); - if (ref.ct) - tag = ct_to_alloc_tag(ref.ct); - put_page_tag_ref(handle); - } + union pgtag_ref_handle handle; + union codetag_ref ref; + + if (get_page_tag_ref(page, &ref, &handle)) { + alloc_tag_sub_check(&ref); + if (ref.ct) + tag = ct_to_alloc_tag(ref.ct); + put_page_tag_ref(handle); } return tag; } -static inline void pgalloc_tag_sub_pages(struct alloc_tag *tag, unsigned int nr) +static inline void pgalloc_tag_sub_pages(struct page *page, unsigned int nr) { - if (mem_alloc_profiling_enabled() && tag) + struct alloc_tag *tag; + + if (!mem_alloc_profiling_enabled()) + return; + + tag = __pgalloc_tag_get(page); + if (tag) this_cpu_sub(tag->counters->bytes, PAGE_SIZE * nr); } @@ -241,8 +245,7 @@ static inline void clear_page_tag_ref(struct page *page) {} static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, unsigned int nr) {} static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) {} -static inline struct alloc_tag *pgalloc_tag_get(struct page *page) { return NULL; } -static inline void pgalloc_tag_sub_pages(struct alloc_tag *tag, unsigned int nr) {} +static inline void pgalloc_tag_sub_pages(struct page *page, unsigned int nr) {} static inline void alloc_tag_sec_init(void) {} static inline void pgalloc_tag_split(struct folio *folio, int old_order, int new_order) {} static inline void pgalloc_tag_swap(struct folio *new, struct folio *old) {} diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index 19b45617bdcf..1d893e313614 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -174,7 +174,7 @@ void pgalloc_tag_split(struct folio *folio, int old_order, int new_order) if (!mem_alloc_profiling_enabled()) return; - tag = pgalloc_tag_get(&folio->page); + tag = __pgalloc_tag_get(&folio->page); if (!tag) return; @@ -200,10 +200,10 @@ void pgalloc_tag_swap(struct folio *new, struct folio *old) if (!mem_alloc_profiling_enabled()) return; - tag_old = pgalloc_tag_get(&old->page); + tag_old = __pgalloc_tag_get(&old->page); if (!tag_old) return; - tag_new = pgalloc_tag_get(&new->page); + tag_new = __pgalloc_tag_get(&new->page); if (!tag_new) return; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 6e469c7ef9a4..55ed2f245f80 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4832,12 +4832,11 @@ void __free_pages(struct page *page, unsigned int order) { /* get PageHead before we drop reference */ int head = PageHead(page); - struct alloc_tag *tag = pgalloc_tag_get(page); if (put_page_testzero(page)) free_frozen_pages(page, order); else if (!head) { - pgalloc_tag_sub_pages(tag, (1 << order) - 1); + pgalloc_tag_sub_pages(page, (1 << order) - 1); while (order-- > 0) free_frozen_pages(page + (1 << order), order); } From patchwork Sun Jan 26 07:02:05 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13950575 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35B3AC0218E for ; Sun, 26 Jan 2025 07:02:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8E4592800B0; Sun, 26 Jan 2025 02:02:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 87C822800A6; Sun, 26 Jan 2025 02:02:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6BE9F2800B0; Sun, 26 Jan 2025 02:02:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 43AD42800A6 for ; Sun, 26 Jan 2025 02:02:15 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id E12231A12E0 for ; Sun, 26 Jan 2025 07:02:14 +0000 (UTC) X-FDA: 83048708988.16.E8A00CD Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf01.hostedemail.com (Postfix) with ESMTP id 1193B4000A for ; Sun, 26 Jan 2025 07:02:12 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ayeYCH6H; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of 3892VZwYKCEo463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3892VZwYKCEo463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737874933; a=rsa-sha256; cv=none; b=vu4bBsnH9pw2viZigi4UxU14GZSdY4WXYT/8E6bM7R6ZAnJVlot97UbQ/88/0ajPSOiszn /SukVQNGSs/uq4wWlLkT6e3GoHUUVBTUuE6454Y1u6VEXTPBAIT+jrJc4AQj5DKVv7iBnr ZSqwfanKfpOZ7VX7PNjNtolwQUMfftk= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ayeYCH6H; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of 3892VZwYKCEo463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3892VZwYKCEo463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737874933; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=u+wqmnI/8fK6TKpLWm3RbIHTrNperIZqHqFOsh7k2n8=; b=2hsSFkTI0dYdwUqjHIv1wq5bRY1glPuNK0QKzBz8m03BlkAnM2mRR/99KtVffBl38/XS0I cFOtX4Q92IMYxfD1d2rMFa0IYa/MG8rrk678h+tLmMJMry36OytVljO7JgdKn7Giu59PMW ziu0iCezD3vafDGuyqw7gy0/wGHvdEk= Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2f816a85facso587083a91.3 for ; Sat, 25 Jan 2025 23:02:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1737874932; x=1738479732; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=u+wqmnI/8fK6TKpLWm3RbIHTrNperIZqHqFOsh7k2n8=; b=ayeYCH6HVHlVeLpbT6g+liEVR+CyLTsFG6WFnD29P5M69+14YKaLrT8eciRtGLh00M bpwyN1QU1Gr6WCVtqXZnkN2u1SLR0PHZtll1EI7n0h/SLvaccRzL7rUMwlsU3I85vtrT 3L4aWKbXViER7B+SzWDaylyd8dhpneJg8FYZEgfdiZzHcwb7GFZ9EZVehbYslFnnz12M Vh0xVTTkXuzKcCHv6vzXwGLGMg+9EvpG9KPvKkMCzQaFE0mt+sKn1jPt31Pt+A5/PfCu BPfX9s2gMBkgVPV+6kYTzj3pmhlFirXrdGAEZFbeYs5L0iSPuZ38e+s8iMKoSVcPM6wi uMQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737874932; x=1738479732; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=u+wqmnI/8fK6TKpLWm3RbIHTrNperIZqHqFOsh7k2n8=; b=Hh+bg6GqUvVUhbQjt1tlc8CIu2ZEzRQWpCPJFXmN9zTldf+iQUMK5HULWpi5a8mh4v F+NTeai+xdi17KyoBQ20/0G1fSyh58Zcc7xfg7ssV7ydd8r9HuOOtqGGRQdYrc8ojLBi nmC7+7kVC5dCMduJe4FlenZijwBtZSeYPf47jhuQmw6uafpuloUU+iqRA4gukuDqoTde c23V0y/rA9R3nk7vMeUl+dwUGJ/eT77QBOrw4hgOKHDhYkSMgg2jINdeXnyXauBsqR/8 q3PSEB8KhtPVaTWdzGz8OQfwpkJV/VaRaX3lYwmsCJ69HRraJ5zpOuR0B357c8fUMFF0 rRzA== X-Forwarded-Encrypted: i=1; AJvYcCV/7e40CeZ0OLJYqo3iafh5nYlKZTrQacQouZ5sdmuTTWbrPgYiGXx6KDsASslaZFYfUnJufmowyw==@kvack.org X-Gm-Message-State: AOJu0YxQUnygcNLa5UmrcRt20rsBNMnk0YoBYONbqofFt4KZm/7m+8fY fblUK+aEKfYGK5ClbElBoGeEcJKgTMsnpfQsqEP+Coa9oBd4TypwiEpi2l2Ic5A3sCtX4SykgTu RnA== X-Google-Smtp-Source: AGHT+IHTGE6/Vf5/+TRu1bIQ3qHUzvpK90Z3vXbQx7gL95QZe8ucUUi3LcdMyv8Ds5MGltY+8AwhAlKYMaY= X-Received: from pjbsw13.prod.google.com ([2002:a17:90b:2c8d:b0:2ea:6b84:3849]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:3a0e:b0:2ee:b4bf:2d06 with SMTP id 98e67ed59e1d1-2f782cb9fe6mr49139163a91.19.1737874931632; Sat, 25 Jan 2025 23:02:11 -0800 (PST) Date: Sat, 25 Jan 2025 23:02:05 -0800 In-Reply-To: <20250126070206.381302-1-surenb@google.com> Mime-Version: 1.0 References: <20250126070206.381302-1-surenb@google.com> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog Message-ID: <20250126070206.381302-2-surenb@google.com> Subject: [PATCH 2/3] alloc_tag: uninline code gated by mem_alloc_profiling_key in slab allocator From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, vbabka@suse.cz, yuzhao@google.com, minchan@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, 00107082@163.com, quic_zhenhuah@quicinc.com, surenb@google.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org X-Rspam-User: X-Rspamd-Queue-Id: 1193B4000A X-Rspamd-Server: rspam10 X-Stat-Signature: jfkxzhc5ubu7y4e9fiqckx74exrw9iwd X-HE-Tag: 1737874932-51260 X-HE-Meta: U2FsdGVkX1+yy/UGVH2wjkeWipdkqYAqzyF3fDNbrKUyc/gV8HzK8uBX91uYe2j9usw42RjfeLXla94+a6dc2oea2c7JNENa5ErJwrhbKL+4waZJFjdhw3S5+5SGU1WaeIo/uPwYj7kXdlznQYYJP2qieDJEGM1LK9LbofroqwHXTY4IWVusG+Bsmwx6MYdrJ7Mgxft/R5v2fZo0LxPpGqxEGFcLKHHvCUcgVZ5ne2WvGRtpQZhilPcWpF8M9n2gP1SYXcs9QOH6YAORI5PlSPPrTYWh85Lsdm6XwrlQnk9phF+uR3FfUUprbVr4EZ54c8AQZEFgi2TSYfNPvehgJkhMVxl88+o0loCkCv2iUBl5+yOuZggqXzt/ZRuxLCWP4RaX4sgD8rOSsjIvl7yu+4B8WYvozpMRwhzhKOPI3xP4Adkg4JBF2sMbly7W9cTklPcIsuY/A7m24T9HYrJhtS1RlgIA2pr3RPth5JCsKhSQKDxqxyqVM4M6cAgHGwB8v/cHgIyIglewnMaiYGODvyLMSyTWOGQc1W8Z9i6PbQBNrZ9WdVPzE6ii8rx4B7r1d9ys9myDzgTSkFm1wqjVNGyHD6IPZketahPyQcBdTNpoUUm1UHpAC3k2gaUSvZjdSuyjgAnbUrtu7SoYcXE00bLIy2PpxaOd1ENFdk/QOmrrS/zUcnlLbwHs5TYrVSWF+69jBx0ijqjYAXSOd+QxIrrFLiH1ag45xeuMp+H2vmr6zBSGur98GrHg/LgEZtjktuPbia2OUlWAnIzvyXlY2KFlKQ+E27pCcH+P/8DfvDaS0pwqIfegQZ8cyBnFYVRzEwb8NlAhfYB6ndWDq0DACmvPNKFijQbpdqB7uYRaah6bs3uvAIrLirdwMm3GZwhAA+2/Fk9OLzE4dOscQgG/sK00rBje0PU+m9TjJiAe1P+u7oWxJvb4nk2xXk8FuoUqrxK+Sg6G49XbAtUrnHH 13M04lWC HU8x/aFT+9zeg4QRfFpwOeKBhq3t+7teeotmAC4XZ16R6TOkJJzGh09sMcacK0jFlxZbc+3aXkB9p6i+wJQ60ppvEigEvjWtjXmN8iJF1fxLC6gkC5PzT5SYlE7fIyV2EvT0g5Xw/epAFPqYJ0lT8vE4eRRqKcLrUT4E4fuEpYjDoObpFWzaHT9a+tGInBjyNUKyO7Ejeo7IOqfsqacibkNbPpTsBMi+9ASpYUW2k27yt5GEKyR8RKbCL9a0lXHhGDOi7m6ok3pEOzc/+TlvHCBXDKdcdB6w2CwVnaHIkNEs6Vch7S4CO9W+JpktzsvAQlVHVNUcnivTH2L8viKFKqSOvYhvFtMcxZVyYdJh7wDShx/kACq0LB/p8XFTstn5ecHbMSMUnIN0eUP2XfMYNwVWAlc/fOHVsGBKsTHqWFEAhYiHXVATpWki1TL2sYr3pMFd+4GZON5KTu70HSyojlDqycChpNnHuxx9p0jqp62Bd+NjA7IPiEEPQRaGdrLEkJweORZGYywXpPkrrHN5HA1A60ZqYwY+LQhCRfJ4TaqefBL2M0Yp2Ss4mgAcfDnDuXuXb1KbZazIVx3BwaXduIPA+lw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.175081, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When a sizable code section is protected by a disabled static key, that code gets into the instruction cache even though it's not executed and consumes the cache, increasing cache misses. This can be remedied by moving such code into a separate uninlined function. The improvement however comes at the expense of the configuration when this static key gets enabled since there is now an additional function call. The default state of the mem_alloc_profiling_key is controlled by CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT. Apply this optimization only if CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=n, improving the performance of the default configuration. When CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=y the functions are inlined and performance does not change. On a Pixel6 phone, slab allocation profiling overhead measured with CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=n and profiling disabled: baseline modified Big 3.31% 0.17% Medium 3.79% 0.57% Little 6.68% 1.28% When CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=n and memory allocation profiling gets enabled, the difference in performance before and after this change stays within noise levels. On x86 this patch does not make noticeable difference because the overhead with mem_alloc_profiling_key disabled is much lower (under 1%) to start with, so any improvement is less visible and hard to distinguish from the noise. Signed-off-by: Suren Baghdasaryan --- include/linux/alloc_tag.h | 6 +++++ mm/slub.c | 46 ++++++++++++++++++++++++--------------- 2 files changed, 34 insertions(+), 18 deletions(-) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index a946e0203e6d..c5de2a0c1780 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -116,6 +116,12 @@ DECLARE_PER_CPU(struct alloc_tag_counters, _shared_alloc_tag); DECLARE_STATIC_KEY_MAYBE(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT, mem_alloc_profiling_key); +#ifdef CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT +#define inline_if_mem_alloc_prof inline +#else +#define inline_if_mem_alloc_prof noinline +#endif + static inline bool mem_alloc_profiling_enabled(void) { return static_branch_maybe(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT, diff --git a/mm/slub.c b/mm/slub.c index 996691c137eb..3107d43dfddc 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2000,7 +2000,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, return 0; } -static inline void free_slab_obj_exts(struct slab *slab) +static inline_if_mem_alloc_prof void free_slab_obj_exts(struct slab *slab) { struct slabobj_ext *obj_exts; @@ -2077,33 +2077,35 @@ prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p) return slab_obj_exts(slab) + obj_to_index(s, slab, p); } -static inline void -alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags) +static inline_if_mem_alloc_prof void +__alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags) { - if (need_slab_obj_ext()) { - struct slabobj_ext *obj_exts; + struct slabobj_ext *obj_exts; - obj_exts = prepare_slab_obj_exts_hook(s, flags, object); - /* - * Currently obj_exts is used only for allocation profiling. - * If other users appear then mem_alloc_profiling_enabled() - * check should be added before alloc_tag_add(). - */ - if (likely(obj_exts)) - alloc_tag_add(&obj_exts->ref, current->alloc_tag, s->size); - } + obj_exts = prepare_slab_obj_exts_hook(s, flags, object); + /* + * Currently obj_exts is used only for allocation profiling. + * If other users appear then mem_alloc_profiling_enabled() + * check should be added before alloc_tag_add(). + */ + if (likely(obj_exts)) + alloc_tag_add(&obj_exts->ref, current->alloc_tag, s->size); } static inline void -alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, +alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags) +{ + if (need_slab_obj_ext()) + __alloc_tagging_slab_alloc_hook(s, object, flags); +} + +static inline_if_mem_alloc_prof void +__alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, int objects) { struct slabobj_ext *obj_exts; int i; - if (!mem_alloc_profiling_enabled()) - return; - /* slab->obj_exts might not be NULL if it was created for MEMCG accounting. */ if (s->flags & (SLAB_NO_OBJ_EXT | SLAB_NOLEAKTRACE)) return; @@ -2119,6 +2121,14 @@ alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, } } +static inline void +alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, + int objects) +{ + if (mem_alloc_profiling_enabled()) + __alloc_tagging_slab_free_hook(s, slab, p, objects); +} + #else /* CONFIG_MEM_ALLOC_PROFILING */ static inline void From patchwork Sun Jan 26 07:02:06 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13950576 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B709EC0218D for ; Sun, 26 Jan 2025 07:02:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ADF9B2800C7; Sun, 26 Jan 2025 02:02:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A414E2800A6; Sun, 26 Jan 2025 02:02:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 86BC72800C7; Sun, 26 Jan 2025 02:02:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 63AFF2800A6 for ; Sun, 26 Jan 2025 02:02:17 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 141751612B4 for ; Sun, 26 Jan 2025 07:02:17 +0000 (UTC) X-FDA: 83048709114.19.AF3AD50 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf01.hostedemail.com (Postfix) with ESMTP id 3D98540004 for ; Sun, 26 Jan 2025 07:02:15 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=UxIA+8SJ; spf=pass (imf01.hostedemail.com: domain of 39d2VZwYKCEw685s1pu22uzs.q20zw18B-00y9oqy.25u@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=39d2VZwYKCEw685s1pu22uzs.q20zw18B-00y9oqy.25u@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737874935; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VuMsPHTSMtWsNhCdVIn1M3R177GHzQc3VDuucKTMIUU=; b=DS9/zqXmyKNxrLX/Vu4tCA1yaRlU+khbyknwUxSMPXEttcmqtfeFWyjDzdz5pxpWzv4vC7 G4XUg8UeL32lGVgdfPnodffmIs/Z/93QCTAahPMsh3WRLq2Nerb8QWaUHe0g6NL8Lf6Er7 ALXzJBWajM0RdLuZ0iIgl9y4NP/Ds3g= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=UxIA+8SJ; spf=pass (imf01.hostedemail.com: domain of 39d2VZwYKCEw685s1pu22uzs.q20zw18B-00y9oqy.25u@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=39d2VZwYKCEw685s1pu22uzs.q20zw18B-00y9oqy.25u@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737874935; a=rsa-sha256; cv=none; b=m5aMA3Bg+vzoUT5jfc9+CR1UvfLj/KUlnM/DkXcKgjWDwxT5w3HKa357ALDHaLrm0X4OKP CcVGlsaMolXI3uSeLXVl8uIyvZpauX91HLAA8/YzLbQe4J0KBA4GgxuNOovn+gsK0YHfSr YG7NRYYQEc+3/AhCBOTtnP224zrDpik= Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-21631cbf87dso67394375ad.3 for ; Sat, 25 Jan 2025 23:02:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1737874934; x=1738479734; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=VuMsPHTSMtWsNhCdVIn1M3R177GHzQc3VDuucKTMIUU=; b=UxIA+8SJD5K5FY6JPpSVmHkHAcQ+oEQWRNRfSQhggudlMLaqrswJYZHsO7bz3LrSYo Q1LA6NFrcro13+nbzpdo2dnwFz/201iRpTRhjSHUjIVfuFRKQ1VPejUakEnQr2r2ljYq er2X2U2lQhqmQImah44mP9Ny/6jM4BoPtfQ3z7zu6uTLxdy2dl+pywJSF+pwzcMKX31v fHiV56OHUBjEXdouaLcAYX3hW/7NUng/JUtPyhW2L2O4M4YMvCR9U+N9qcXADpFqbOfW 0p20vFzk1gFz4k1S68tVfoyUMNEu7qoYuKr4GzBLgvX52wVb8hKwZSz99thNuUZiuO8C QK4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737874934; x=1738479734; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=VuMsPHTSMtWsNhCdVIn1M3R177GHzQc3VDuucKTMIUU=; b=JA6XtWXdrilOC3NVmGBZjCA16humGJrdBeZ5zfcTVwy2yxbhZOSX3AKpi9iAYPwscZ FEcNCnfaNDYDGYxkBREeUPGD/fg9AzzFALrEsVwgWMQCfEb/U+wuPeB7ckrIGGu+yby9 AnitOKt8G2PfaKfBvr7sHd0rgzXqIJxlxa+VURg8a6OjhRRiIkupTJuo253xOwbDJzbn fifMhVjpD5hK92K5ye+fB5fMZCGDkbM5w9p21yBmc73diJDyvrB5wN75l3vIROoqIF5n /W8oBiyI+i38oMF/kTVyr82CJroT50I/RMDuzIB4zDXgRF+64bC7ysEzl0J1QNdUuXqh D8og== X-Forwarded-Encrypted: i=1; AJvYcCVBPj3QbTRt1EPdghh1kZ2hJbR3bth3ghMOkCeKD0WnEycAjc3wjQPcvkP7wO+MthIxuQLmSjJ6HA==@kvack.org X-Gm-Message-State: AOJu0YyuZxMbKmgKpjaxxGP4bQRU2txUXEUaiqBxQEWQLK/V6nKVrIq7 CnG7z5nRHHT0dRl6UBZbJIID35JUdJQWZJszALM9aUY08b4W0y9RIGt1tiJk8neOYf/NqwnkdfE ONg== X-Google-Smtp-Source: AGHT+IHy+6ALp0tq0HhC4NNxguhtqx7IJQwGFlemwAnliRSj9IqTbeoXX2L4PWoV+tmh3gWwLoTTF0wsrrU= X-Received: from pgbdh19.prod.google.com ([2002:a05:6a02:b93:b0:7fd:50ab:dc45]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:81a3:b0:1e8:bff6:8356 with SMTP id adf61e73a8af0-1eb214cb158mr41901012637.20.1737874933839; Sat, 25 Jan 2025 23:02:13 -0800 (PST) Date: Sat, 25 Jan 2025 23:02:06 -0800 In-Reply-To: <20250126070206.381302-1-surenb@google.com> Mime-Version: 1.0 References: <20250126070206.381302-1-surenb@google.com> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog Message-ID: <20250126070206.381302-3-surenb@google.com> Subject: [PATCH 3/3] alloc_tag: uninline code gated by mem_alloc_profiling_key in page allocator From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, vbabka@suse.cz, yuzhao@google.com, minchan@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, 00107082@163.com, quic_zhenhuah@quicinc.com, surenb@google.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 3D98540004 X-Stat-Signature: ek1kjs3gw5p5osd58d47h43w3grw4q3e X-Rspam-User: X-HE-Tag: 1737874935-997208 X-HE-Meta: U2FsdGVkX19mQu8PSKg/IvU5L1IRjf/SS50ahnu0B43QIsIwdRwlV2zu7nUu7JVI8pLgQQ2C65uY97gtchMakQKjDu9PbRt0DiJOZOg1q5JDEsMHnPOKCQhHY+i1CynWrYVN17azalLJmAAbsSzhZJBL0ykf1OVCNOMdUqhL6JFIXko6lv9btSLI9g5Szwj1VHgo/etO/mdiUyepmHZmMRktNTpeiFby5ZtMBgm8jW8e7h7RuFybWZvtJhAkPoVALuupIaonJ7aw6SoWWVIR6iZD/ke3wROU1wXJjbWnEE3a1yaed9cXqB/EuZUP0CRE/UfQtFWfSZ1Z7sy+uD5Oz+ZGwmtk4JF9eWrrSKTKpvAz4qIC+kleHqXgIxHm5R5c+nnnDSqIHxvPHT//SPW/4zwKgB+4q56EE5oQAni8FjejIjgAvmKAsMO3bApt3CvFTBOc7Vjl2S1qVxLo7q3nJPLiluCmEg6hPYW5NjCLpF+wlz2LuwoODmH/UYTD4nXkAzPNgrj/rIooZH31GXfoXFrQiaaxVwQl48bsKLMVvS6Wx0iuSGWHxiKT923i4MutlhxJIkdGgv5Pbf5ItWdnzVpN6GC5F7ECShqnR/EerAeNN3SNwsmD225HqrqN1qNNzV6bp7AHomDD4sT3tNmPKUfNrywkhgfkW1YRr2Spa5+JRUcyBYQ8+c9TmVaiAJ2bfwqVdF50QiNeiBTqqSWDauJt+8wq6wksO4n5K+xrLdlbRgeLm2EewB195zvLB5CuZShGwDogwCFW/XAbxIoJtgOGWHm4Zc2846jVlURpS/25rS4mDh5z1LPlz+tMDBc8Ip/JkHTS5uMBjE0j40HKVrOuOq0UstSEwxsJ2TRZZ3/6cpngeYXbE3jsCMVO7J6KA3VoDUo7HG9FVsrc+6bdoYaIQ74accQTovOLES/DVocEEE+i5UeiPwVRCEjrMN3kAln/zxU8HUlmMeU6nWp FOEG8bSS P5yWVX1pdw+uZTPIszTP0WBLrh8OMooDb0CBr9iyLd7RiphdUvDRajukSIfCS624PONnMfVPUjIOz97P59Vc+n83qA3Hxk3CvGPPo+yzHhLfm4av/JG3irqxh+76VqUMG9iOQwI11RnMwlwnIPX9Hk2NKUXxNlBHb7Oto9DPOuTVj6zZUaPi/9JHlHqNxASOus0xMUxN3hZ4lkdfUnFgwuaUgPPWUZde3AIkU6kvK2zlE5GYG74pRw+Wi089CPt+e0au1M54cMrXqM95BX/AoFIgECIWTFnyiKk7G/dSi3I/Xi1Gf7nt4CmiKqQITHVeTcQkpW9B5LjTRyxiVdANZko/ca3J0BZ9St0IhHcnwABJmQJz+8+Rty5F0sXFYGC/Y1pf5awMtw8VaWqqJcDKCsz9g5vDT/Ii2t/cMihZmgQ5V4Doy8Ce+Eqng7sUIer45Pf4e/b/hhlJgs3fUxj7XGi5dbAbM3aiDwoAE8ESfAMGaESMQuoNT9ZInpzGxmwtYxQ66ovkjDb5QwclyLUfNuenOjlf8kSRmi4nhSvVqD7fHaXbWZ4ZxFOXb+bNpGp2WZx2CQhaD0+bTnXeGzxqkhNJlWQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.016717, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When a sizable code section is protected by a disabled static key, that code gets into the instruction cache even though it's not executed and consumes the cache, increasing cache misses. This can be remedied by moving such code into a separate uninlined function. The improvement however comes at the expense of the configuration when this static key gets enabled since there is now an additional function call. The default state of the mem_alloc_profiling_key is controlled by CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT. Apply this optimization only if CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=n, improving the performance of the default configuration. When CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=y the functions are inlined and performance does not change. On a Pixel6 phone, page allocation profiling overhead measured with CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=n and profiling disabled: baseline modified Big 4.93% 1.53% Medium 4.39% 1.41% Little 1.02% 0.36% When CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=n and memory allocation profiling gets enabled, the difference in performance before and after this change stays within noise levels. On x86 this patch does not make noticeable difference because the overhead with mem_alloc_profiling_key disabled is much lower (under 1%) to start with, so any improvement is less visible and hard to distinguish from the noise. Signed-off-by: Suren Baghdasaryan --- include/linux/pgalloc_tag.h | 60 +++------------------------- mm/page_alloc.c | 78 +++++++++++++++++++++++++++++++++++++ 2 files changed, 83 insertions(+), 55 deletions(-) diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index 4a82b6b4820e..c74077977830 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -162,47 +162,13 @@ static inline void update_page_tag_ref(union pgtag_ref_handle handle, union code } } -static inline void clear_page_tag_ref(struct page *page) -{ - if (mem_alloc_profiling_enabled()) { - union pgtag_ref_handle handle; - union codetag_ref ref; - - if (get_page_tag_ref(page, &ref, &handle)) { - set_codetag_empty(&ref); - update_page_tag_ref(handle, &ref); - put_page_tag_ref(handle); - } - } -} - -static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, - unsigned int nr) -{ - if (mem_alloc_profiling_enabled()) { - union pgtag_ref_handle handle; - union codetag_ref ref; - - if (get_page_tag_ref(page, &ref, &handle)) { - alloc_tag_add(&ref, task->alloc_tag, PAGE_SIZE * nr); - update_page_tag_ref(handle, &ref); - put_page_tag_ref(handle); - } - } -} +/* Should be called only if mem_alloc_profiling_enabled() */ +void __clear_page_tag_ref(struct page *page); -static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) +static inline void clear_page_tag_ref(struct page *page) { - if (mem_alloc_profiling_enabled()) { - union pgtag_ref_handle handle; - union codetag_ref ref; - - if (get_page_tag_ref(page, &ref, &handle)) { - alloc_tag_sub(&ref, PAGE_SIZE * nr); - update_page_tag_ref(handle, &ref); - put_page_tag_ref(handle); - } - } + if (mem_alloc_profiling_enabled()) + __clear_page_tag_ref(page); } /* Should be called only if mem_alloc_profiling_enabled() */ @@ -222,18 +188,6 @@ static inline struct alloc_tag *__pgalloc_tag_get(struct page *page) return tag; } -static inline void pgalloc_tag_sub_pages(struct page *page, unsigned int nr) -{ - struct alloc_tag *tag; - - if (!mem_alloc_profiling_enabled()) - return; - - tag = __pgalloc_tag_get(page); - if (tag) - this_cpu_sub(tag->counters->bytes, PAGE_SIZE * nr); -} - void pgalloc_tag_split(struct folio *folio, int old_order, int new_order); void pgalloc_tag_swap(struct folio *new, struct folio *old); @@ -242,10 +196,6 @@ void __init alloc_tag_sec_init(void); #else /* CONFIG_MEM_ALLOC_PROFILING */ static inline void clear_page_tag_ref(struct page *page) {} -static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, - unsigned int nr) {} -static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) {} -static inline void pgalloc_tag_sub_pages(struct page *page, unsigned int nr) {} static inline void alloc_tag_sec_init(void) {} static inline void pgalloc_tag_split(struct folio *folio, int old_order, int new_order) {} static inline void pgalloc_tag_swap(struct folio *new, struct folio *old) {} diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 55ed2f245f80..67e205286dbf 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1041,6 +1041,84 @@ static void kernel_init_pages(struct page *page, int numpages) kasan_enable_current(); } +#ifdef CONFIG_MEM_ALLOC_PROFILING + +/* Should be called only if mem_alloc_profiling_enabled() */ +void __clear_page_tag_ref(struct page *page) +{ + union pgtag_ref_handle handle; + union codetag_ref ref; + + if (get_page_tag_ref(page, &ref, &handle)) { + set_codetag_empty(&ref); + update_page_tag_ref(handle, &ref); + put_page_tag_ref(handle); + } +} + +/* Should be called only if mem_alloc_profiling_enabled() */ +static inline_if_mem_alloc_prof +void __pgalloc_tag_add(struct page *page, struct task_struct *task, + unsigned int nr) +{ + union pgtag_ref_handle handle; + union codetag_ref ref; + + if (get_page_tag_ref(page, &ref, &handle)) { + alloc_tag_add(&ref, task->alloc_tag, PAGE_SIZE * nr); + update_page_tag_ref(handle, &ref); + put_page_tag_ref(handle); + } +} + +static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, + unsigned int nr) +{ + if (mem_alloc_profiling_enabled()) + __pgalloc_tag_add(page, task, nr); +} + +/* Should be called only if mem_alloc_profiling_enabled() */ +static inline_if_mem_alloc_prof +void __pgalloc_tag_sub(struct page *page, unsigned int nr) +{ + union pgtag_ref_handle handle; + union codetag_ref ref; + + if (get_page_tag_ref(page, &ref, &handle)) { + alloc_tag_sub(&ref, PAGE_SIZE * nr); + update_page_tag_ref(handle, &ref); + put_page_tag_ref(handle); + } +} + +static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) +{ + if (mem_alloc_profiling_enabled()) + __pgalloc_tag_sub(page, nr); +} + +static inline void pgalloc_tag_sub_pages(struct page *page, unsigned int nr) +{ + struct alloc_tag *tag; + + if (!mem_alloc_profiling_enabled()) + return; + + tag = __pgalloc_tag_get(page); + if (tag) + this_cpu_sub(tag->counters->bytes, PAGE_SIZE * nr); +} + +#else /* CONFIG_MEM_ALLOC_PROFILING */ + +static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, + unsigned int nr) {} +static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) {} +static inline void pgalloc_tag_sub_pages(struct page *page, unsigned int nr) {} + +#endif /* CONFIG_MEM_ALLOC_PROFILING */ + __always_inline bool free_pages_prepare(struct page *page, unsigned int order) {