From patchwork Fri Jan 24 03:56:50 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13948872 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A3B5D1D5CD7 for ; Fri, 24 Jan 2025 03:57:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737691024; cv=none; b=AATIZ1MkIUwvchalzXtqIrpZXT4bka1TF4mufAW4f/avc4NEGwWqIDg0bS7L0C9XDXjOLDt4/0DMwCwsjSeY+ht3aiPhdShiTW3Fi4u+HOHHK/pZvV1tR+ti3lcllK4cjCuhV7tcTyeHkByscGB2rc4VNAa9yYgPCOtKTSvfB+s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737691024; c=relaxed/simple; bh=glP/o009xF95ZbKehzr+bPHtJrR4LnHnNUp7ZPZxW3w=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=b4jS+BTMbAi8GGtTIkWQxcLKh6ohoCnLotKFFVp/XKvL5nz4d6BQPtsTGezDuYcP8cUMTeHjgmuYSJqfGCFfan86Mqj+1GVa/DCywgHIqw9ZY03H48MBhSfXy4v0SGoO84wE43swJeGIkPO4NNxhpqur1kJuGAZWnUttR+dLu48= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=U4GwAezg; arc=none smtp.client-ip=209.85.214.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="U4GwAezg" Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-2165cb60719so28482965ad.0 for ; Thu, 23 Jan 2025 19:57:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1737691021; x=1738295821; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=G89tmHrRovh0M5Rk+1LopEs8NOXI2BpuMblkeiiZfDc=; b=U4GwAezgXP3hSTZc3T7Nor9W5aiGcPYrMs4Y8dVF4OULSaWG31up8rwuI8oPDXqTdf DVt9QUMbcOpagETJlOTAWL4U6FO+wVTsKH40/u0rj/AFbCza1efENnNkSrXytNh81XTj qoFYwv/qIsMNEq+/5I+EV+mH+VMAEcKiUBWnml/EJ44l2sfEstE1qbbLBYMcMaT7CnvS rIx2DlL23iK5XCaIOGjYm0ujYoiKD/Rx7E2u6HfYArUldi7AgU7Mv4jAY3ySkfybWkR2 m33gBm2CE6cO8qraG88+onatuCPlhaYWIgnTFbUUc3bkbnCHiwQIVqR/LK0xfkxPHJzy GCXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737691021; x=1738295821; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=G89tmHrRovh0M5Rk+1LopEs8NOXI2BpuMblkeiiZfDc=; b=UTSQbKtXnHxJGwZ9fPyy6RNROeoaNZ1Wcx4p/WZ2sN9BkEEel51EW6QgDd9fgheJm8 6FOFjJ+6G92aNZ5UeQ+llCcS9lG69RG0ozGDsiU3JW7GDq4wNnOHeQXp7Pb65GW1xO8j OIOpHosHYBnVdJyb1BCAJs0yBO6ysvfuTJCgrGeiwHnM26GHCh16L6V0hpK1Ea4VIvWs 5cV6uil1/mwC7OoTSXq/tPWpJmjIyKLOrEFB2GyXTgWZ/mIfbZQQkqOXrk9upbJB+xFs 0dsOedn1Kb/1PPG74g34vpltwmauiqM3Oei1gDALPvPyXygIZhqXbc/mipENLx2W3VLv q/oQ== X-Gm-Message-State: AOJu0YzsCThTJTxEHlDOkJh0PqFXK6wpQxk34qBKH6UteZG0UrSoi2ix rx3T3OZyPwyCdbvYiargBkFS0CccUD23gpzgLhk5TamVO0vsIhDwpzNnZg== X-Gm-Gg: ASbGncs7ETGvzUUQw/Xu0vPFS+1cyXcelAGCqgPAKdMIJErVQWmerUIumxEobih87od yv8NrL1E/4JEf3Dnjmod/ntXlu9tYPmDNYq8xqIIql6FxSSkk7NWf9bI2FXfoY31IbZl4e0YSGA njcrvV7gtJKki4oKuG+sIl6JKhLuWYT2Tg+zXv/Plp7PqtXGZ52TMNxBJnstuFbpSNWLvpPDMnt x9VJ9UQpANneLtbPU/tj1ZiyqBNofUXVmDWn4G/Kv1VBYs+UH2KPQqtqtJ9jywAClbY+ycD+fWp JSgCIlxpNaKm/gf7McMV+buTAcyOFf8IpSS7cs4= X-Google-Smtp-Source: AGHT+IEvNDPkJRiZfkwwi/aPaSWxbkYozYuQgk6SDROUjP76ZWWgmUKEf41JFPWQtip/8VwELQfVqQ== X-Received: by 2002:a17:902:ce8a:b0:216:6855:15f with SMTP id d9443c01a7336-21c3562198emr408647685ad.51.1737691021406; Thu, 23 Jan 2025 19:57:01 -0800 (PST) Received: from macbookpro.lan ([2603:3023:16e:5000:8af:ecd2:44cd:8027]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-21da424c145sm6717335ad.240.2025.01.23.19.57.00 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 23 Jan 2025 19:57:01 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v6 1/6] mm, bpf: Introduce try_alloc_pages() for opportunistic page allocation Date: Thu, 23 Jan 2025 19:56:50 -0800 Message-Id: <20250124035655.78899-2-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250124035655.78899-1-alexei.starovoitov@gmail.com> References: <20250124035655.78899-1-alexei.starovoitov@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net From: Alexei Starovoitov Tracing BPF programs execute from tracepoints and kprobes where running context is unknown, but they need to request additional memory. The prior workarounds were using pre-allocated memory and BPF specific freelists to satisfy such allocation requests. Instead, introduce gfpflags_allow_spinning() condition that signals to the allocator that running context is unknown. Then rely on percpu free list of pages to allocate a page. try_alloc_pages() -> get_page_from_freelist() -> rmqueue() -> rmqueue_pcplist() will spin_trylock to grab the page from percpu free list. If it fails (due to re-entrancy or list being empty) then rmqueue_bulk()/rmqueue_buddy() will attempt to spin_trylock zone->lock and grab the page from there. spin_trylock() is not safe in PREEMPT_RT when in NMI or in hard IRQ. Bailout early in such case. The support for gfpflags_allow_spinning() mode for free_page and memcg comes in the next patches. This is a first step towards supporting BPF requirements in SLUB and getting rid of bpf_mem_alloc. That goal was discussed at LSFMM: https://lwn.net/Articles/974138/ Acked-by: Michal Hocko Acked-by: Vlastimil Babka Acked-by: Sebastian Andrzej Siewior Signed-off-by: Alexei Starovoitov Reviewed-by: Shakeel Butt --- include/linux/gfp.h | 22 ++++++++++ lib/stackdepot.c | 5 ++- mm/internal.h | 1 + mm/page_alloc.c | 104 ++++++++++++++++++++++++++++++++++++++++++-- 4 files changed, 127 insertions(+), 5 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index b0fe9f62d15b..82bfb65b8d15 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -39,6 +39,25 @@ static inline bool gfpflags_allow_blocking(const gfp_t gfp_flags) return !!(gfp_flags & __GFP_DIRECT_RECLAIM); } +static inline bool gfpflags_allow_spinning(const gfp_t gfp_flags) +{ + /* + * !__GFP_DIRECT_RECLAIM -> direct claim is not allowed. + * !__GFP_KSWAPD_RECLAIM -> it's not safe to wake up kswapd. + * All GFP_* flags including GFP_NOWAIT use one or both flags. + * try_alloc_pages() is the only API that doesn't specify either flag. + * + * This is stronger than GFP_NOWAIT or GFP_ATOMIC because + * those are guaranteed to never block on a sleeping lock. + * Here we are enforcing that the allocation doesn't ever spin + * on any locks (i.e. only trylocks). There is no high level + * GFP_$FOO flag for this use in try_alloc_pages() as the + * regular page allocator doesn't fully support this + * allocation mode. + */ + return !(gfp_flags & __GFP_RECLAIM); +} + #ifdef CONFIG_HIGHMEM #define OPT_ZONE_HIGHMEM ZONE_HIGHMEM #else @@ -347,6 +366,9 @@ static inline struct page *alloc_page_vma_noprof(gfp_t gfp, } #define alloc_page_vma(...) alloc_hooks(alloc_page_vma_noprof(__VA_ARGS__)) +struct page *try_alloc_pages_noprof(int nid, unsigned int order); +#define try_alloc_pages(...) alloc_hooks(try_alloc_pages_noprof(__VA_ARGS__)) + extern unsigned long get_free_pages_noprof(gfp_t gfp_mask, unsigned int order); #define __get_free_pages(...) alloc_hooks(get_free_pages_noprof(__VA_ARGS__)) diff --git a/lib/stackdepot.c b/lib/stackdepot.c index 245d5b416699..377194969e61 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -591,7 +591,8 @@ depot_stack_handle_t stack_depot_save_flags(unsigned long *entries, depot_stack_handle_t handle = 0; struct page *page = NULL; void *prealloc = NULL; - bool can_alloc = depot_flags & STACK_DEPOT_FLAG_CAN_ALLOC; + bool allow_spin = gfpflags_allow_spinning(alloc_flags); + bool can_alloc = (depot_flags & STACK_DEPOT_FLAG_CAN_ALLOC) && allow_spin; unsigned long flags; u32 hash; @@ -630,7 +631,7 @@ depot_stack_handle_t stack_depot_save_flags(unsigned long *entries, prealloc = page_address(page); } - if (in_nmi()) { + if (in_nmi() || !allow_spin) { /* We can never allocate in NMI context. */ WARN_ON_ONCE(can_alloc); /* Best effort; bail if we fail to take the lock. */ diff --git a/mm/internal.h b/mm/internal.h index 9826f7dce607..6c3c664aa346 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1174,6 +1174,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone, #define ALLOC_NOFRAGMENT 0x0 #endif #define ALLOC_HIGHATOMIC 0x200 /* Allows access to MIGRATE_HIGHATOMIC */ +#define ALLOC_TRYLOCK 0x400 /* Only use spin_trylock in allocation path */ #define ALLOC_KSWAPD 0x800 /* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ /* Flags that allow allocations below the min watermark. */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 01eab25edf89..a82bc67abbdb 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2306,7 +2306,11 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order, unsigned long flags; int i; - spin_lock_irqsave(&zone->lock, flags); + if (!spin_trylock_irqsave(&zone->lock, flags)) { + if (unlikely(alloc_flags & ALLOC_TRYLOCK)) + return 0; + spin_lock_irqsave(&zone->lock, flags); + } for (i = 0; i < count; ++i) { struct page *page = __rmqueue(zone, order, migratetype, alloc_flags); @@ -2906,7 +2910,11 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, do { page = NULL; - spin_lock_irqsave(&zone->lock, flags); + if (!spin_trylock_irqsave(&zone->lock, flags)) { + if (unlikely(alloc_flags & ALLOC_TRYLOCK)) + return NULL; + spin_lock_irqsave(&zone->lock, flags); + } if (alloc_flags & ALLOC_HIGHATOMIC) page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC); if (!page) { @@ -4511,7 +4519,12 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, might_alloc(gfp_mask); - if (should_fail_alloc_page(gfp_mask, order)) + /* + * Don't invoke should_fail logic, since it may call + * get_random_u32() and printk() which need to spin_lock. + */ + if (!(*alloc_flags & ALLOC_TRYLOCK) && + should_fail_alloc_page(gfp_mask, order)) return false; *alloc_flags = gfp_to_alloc_flags_cma(gfp_mask, *alloc_flags); @@ -7028,3 +7041,88 @@ static bool __free_unaccepted(struct page *page) } #endif /* CONFIG_UNACCEPTED_MEMORY */ + +/** + * try_alloc_pages_noprof - opportunistic reentrant allocation from any context + * @nid - node to allocate from + * @order - allocation order size + * + * Allocates pages of a given order from the given node. This is safe to + * call from any context (from atomic, NMI, and also reentrant + * allocator -> tracepoint -> try_alloc_pages_noprof). + * Allocation is best effort and to be expected to fail easily so nobody should + * rely on the success. Failures are not reported via warn_alloc(). + * See always fail conditions below. + * + * Return: allocated page or NULL on failure. + */ +struct page *try_alloc_pages_noprof(int nid, unsigned int order) +{ + /* + * Do not specify __GFP_DIRECT_RECLAIM, since direct claim is not allowed. + * Do not specify __GFP_KSWAPD_RECLAIM either, since wake up of kswapd + * is not safe in arbitrary context. + * + * These two are the conditions for gfpflags_allow_spinning() being true. + * + * Specify __GFP_NOWARN since failing try_alloc_pages() is not a reason + * to warn. Also warn would trigger printk() which is unsafe from + * various contexts. We cannot use printk_deferred_enter() to mitigate, + * since the running context is unknown. + * + * Specify __GFP_ZERO to make sure that call to kmsan_alloc_page() below + * is safe in any context. Also zeroing the page is mandatory for + * BPF use cases. + * + * Though __GFP_NOMEMALLOC is not checked in the code path below, + * specify it here to highlight that try_alloc_pages() + * doesn't want to deplete reserves. + */ + gfp_t alloc_gfp = __GFP_NOWARN | __GFP_ZERO | __GFP_NOMEMALLOC; + unsigned int alloc_flags = ALLOC_TRYLOCK; + struct alloc_context ac = { }; + struct page *page; + + /* + * In PREEMPT_RT spin_trylock() will call raw_spin_lock() which is + * unsafe in NMI. If spin_trylock() is called from hard IRQ the current + * task may be waiting for one rt_spin_lock, but rt_spin_trylock() will + * mark the task as the owner of another rt_spin_lock which will + * confuse PI logic, so return immediately if called form hard IRQ or + * NMI. + * + * Note, irqs_disabled() case is ok. This function can be called + * from raw_spin_lock_irqsave region. + */ + if (IS_ENABLED(CONFIG_PREEMPT_RT) && (in_nmi() || in_hardirq())) + return NULL; + if (!pcp_allowed_order(order)) + return NULL; + +#ifdef CONFIG_UNACCEPTED_MEMORY + /* Bailout, since try_to_accept_memory_one() needs to take a lock */ + if (has_unaccepted_memory()) + return NULL; +#endif + /* Bailout, since _deferred_grow_zone() needs to take a lock */ + if (deferred_pages_enabled()) + return NULL; + + if (nid == NUMA_NO_NODE) + nid = numa_node_id(); + + prepare_alloc_pages(alloc_gfp, order, nid, NULL, &ac, + &alloc_gfp, &alloc_flags); + + /* + * Best effort allocation from percpu free list. + * If it's empty attempt to spin_trylock zone->lock. + */ + page = get_page_from_freelist(alloc_gfp, order, alloc_flags, &ac); + + /* Unlike regular alloc_pages() there is no __alloc_pages_slowpath(). */ + + trace_mm_page_alloc(page, order, alloc_gfp, ac.migratetype); + kmsan_alloc_page(page, order, alloc_gfp); + return page; +} From patchwork Fri Jan 24 03:56:51 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13948873 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4A2BC1D5CD7 for ; Fri, 24 Jan 2025 03:57:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737691028; cv=none; b=MV3BxOA1rDhGfK2ljWcxWYxLO4NwIbwWo2Bl9w/n47WaRYDDjqwfJ6mtL/jLwYpBgeICBQiP2DrW2uVKIa5vV58AE6INgMN2sW13R81ev3ziscIoSW1cJ1Gi0CQuxmPy0ShX1uK8tXXwvW0BbJQyNXxibpm8Wlb3dUSOWsHHLIM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737691028; c=relaxed/simple; bh=ycjmyv+0eFg+yrY+J9OUW+woZ90LXfMcS86TmoA5qD8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=JHNMn5JV10eErOtdlOyUm5MuLNZ5332ckeCOaKwbR7ePfiEcGKTSqmW+8OuSO0Yqn5UnJ1wWwOVnx5cJDpPUjv4lxqgPR6yvvTz+2p3P5aE9nfzw4wyh7xxG8779uicnHzUK070pBLE/tGgvkap+t1DEUMAC7Z6vjh85H4alH54= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=YVM/IjmB; arc=none smtp.client-ip=209.85.214.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="YVM/IjmB" Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-21619108a6bso27395185ad.3 for ; Thu, 23 Jan 2025 19:57:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1737691025; x=1738295825; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Fz2IAlBDxhiT58/U161LSQMwkAoY7PfGXhcEsriHk/0=; b=YVM/IjmBQ0iIbB0stoMZTdT7oAc3hhQon/l1qg0K+Yqwo6HH27KrH+edkLlOMLJTQY tp+KlxW810tyUaxuftsIsVwumsHP88IxW/i9hWcmXTErabjfsaxw8ES2horihETlw39C oaOMrjjxc/XT9b4uqaqokyNQm4E5SgXNe0Rvn4SiNm47NVYr1rkPBi+lSimwMr0xH+i8 fGbnYnsiNxQJgkwsEMSx8b1GRE01j6pNRwp/gDR/pX7BlAUY5Hqdw9yZXDhaoXMRsr7v b4uL+Q7g98fMB/ZIlrxMBWzyCYWmX0iFCrGzERH3bFc0UqARmj52TdJqapnyS7xdMuLv JVVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737691025; x=1738295825; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Fz2IAlBDxhiT58/U161LSQMwkAoY7PfGXhcEsriHk/0=; b=qMAfBhLAMdD8LCk7NQ7bFc9eoB8FHIH/uflUTYHyDWqucKPofd4kLWQlLlaIcjNqNZ aXzGcVP3T9m1aT6opjUOVhZsUhiaC9H5TASul2sgVL11X1wakCfoDqs3B/j1TxX78GsI M4G8p9NKuE3aJnLIGr5YhFtvSdjRi+bcCpWvFuhR+Yii8h1sbvYiUYmGz4QB/orPKxbi vmamDTPx270xEsqjiOHm+AjTXl5meCD9IR8HRnyG9wZ6CalErlE3RqlEIm9B9g9i3pgt eymB+pnNEmAQufNBGwMEwAQlSobRwW4Bbk/XuoRCRfu7/gPGEee+alpGxu1fNX1lLJBl YVLg== X-Gm-Message-State: AOJu0Yzho+UiXMlN9wuXjmebbU4XxE4/a5ZiWfgmd0irLI2cAxYKiAF6 0+o09OQebTg5w9VJNqT4zNxDmLlQZ13wgQQAJsu3QeRcDXCZfRW5RNEuUg== X-Gm-Gg: ASbGncu20afofyX1bQkF5hV9GYgrJ06PT6ld7RTqw5bzal8vbeU6HPzPUG7bAxow/ge PBBY+fqOPFNNfOBQj4AQc4XTarhlEVyUHXbc580Dqfyu5/xQVjPGjlRu33Cs9HrmAInBII47tEQ Zx9lQaSRK+Ka6LMd5SyBFOFyyBdUTf5frdxS/yRFZWGNTlM+50Zs71Dqh1pRmXmDUC6kT18h4aI YZtASO6kl63uYsMV7EvsdcxFbkxqWljUKtHlHx3mhFry+ZpBC3d5BA6cUXtNhk1dRnZuAoZfXwu w88hHz9jV5edqoMGZkhvc3/kPBdDZhRBB1Rh80k= X-Google-Smtp-Source: AGHT+IHZeimjkRMY4H4j9FWOv2afkQfL53TrxMWXGw2hDPca6jWcxwmgWQSstHxk/bosCoPa9wzktQ== X-Received: by 2002:a17:902:e84c:b0:216:6769:9eca with SMTP id d9443c01a7336-21c355b9442mr374872965ad.37.1737691024611; Thu, 23 Jan 2025 19:57:04 -0800 (PST) Received: from macbookpro.lan ([2603:3023:16e:5000:8af:ecd2:44cd:8027]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-21da414151fsm6640485ad.121.2025.01.23.19.57.03 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 23 Jan 2025 19:57:04 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v6 2/6] mm, bpf: Introduce free_pages_nolock() Date: Thu, 23 Jan 2025 19:56:51 -0800 Message-Id: <20250124035655.78899-3-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250124035655.78899-1-alexei.starovoitov@gmail.com> References: <20250124035655.78899-1-alexei.starovoitov@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net From: Alexei Starovoitov Introduce free_pages_nolock() that can free pages without taking locks. It relies on trylock and can be called from any context. Since spin_trylock() cannot be used in PREEMPT_RT from hard IRQ or NMI it uses lockless link list to stash the pages which will be freed by subsequent free_pages() from good context. Do not use llist unconditionally. BPF maps continuously allocate/free, so we cannot unconditionally delay the freeing to llist. When the memory becomes free make it available to the kernel and BPF users right away if possible, and fallback to llist as the last resort. Acked-by: Vlastimil Babka Acked-by: Sebastian Andrzej Siewior Signed-off-by: Alexei Starovoitov Reviewed-by: Shakeel Butt --- include/linux/gfp.h | 1 + include/linux/mm_types.h | 4 ++ include/linux/mmzone.h | 3 ++ lib/stackdepot.c | 5 ++- mm/page_alloc.c | 90 +++++++++++++++++++++++++++++++++++----- mm/page_owner.c | 8 +++- 6 files changed, 98 insertions(+), 13 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index 82bfb65b8d15..a8233d09acfa 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -391,6 +391,7 @@ __meminit void *alloc_pages_exact_nid_noprof(int nid, size_t size, gfp_t gfp_mas __get_free_pages((gfp_mask) | GFP_DMA, (order)) extern void __free_pages(struct page *page, unsigned int order); +extern void free_pages_nolock(struct page *page, unsigned int order); extern void free_pages(unsigned long addr, unsigned int order); #define __free_page(page) __free_pages((page), 0) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 825c04b56403..583bf59e2627 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -99,6 +99,10 @@ struct page { /* Or, free page */ struct list_head buddy_list; struct list_head pcp_list; + struct { + struct llist_node pcp_llist; + unsigned int order; + }; }; /* See page-flags.h for PAGE_MAPPING_FLAGS */ struct address_space *mapping; diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index b36124145a16..1a854e0a9e3b 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -953,6 +953,9 @@ struct zone { /* Primarily protects free_area */ spinlock_t lock; + /* Pages to be freed when next trylock succeeds */ + struct llist_head trylock_free_pages; + /* Write-intensive fields used by compaction and vmstats. */ CACHELINE_PADDING(_pad2_); diff --git a/lib/stackdepot.c b/lib/stackdepot.c index 377194969e61..73d7b50924ef 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -672,7 +672,10 @@ depot_stack_handle_t stack_depot_save_flags(unsigned long *entries, exit: if (prealloc) { /* Stack depot didn't use this memory, free it. */ - free_pages((unsigned long)prealloc, DEPOT_POOL_ORDER); + if (!allow_spin) + free_pages_nolock(virt_to_page(prealloc), DEPOT_POOL_ORDER); + else + free_pages((unsigned long)prealloc, DEPOT_POOL_ORDER); } if (found) handle = found->handle.handle; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index a82bc67abbdb..fa750c46e0fc 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -88,6 +88,9 @@ typedef int __bitwise fpi_t; */ #define FPI_TO_TAIL ((__force fpi_t)BIT(1)) +/* Free the page without taking locks. Rely on trylock only. */ +#define FPI_TRYLOCK ((__force fpi_t)BIT(2)) + /* prevent >1 _updater_ of zone percpu pageset ->high and ->batch fields */ static DEFINE_MUTEX(pcp_batch_high_lock); #define MIN_PERCPU_PAGELIST_HIGH_FRACTION (8) @@ -1249,13 +1252,44 @@ static void split_large_buddy(struct zone *zone, struct page *page, } while (1); } +static void add_page_to_zone_llist(struct zone *zone, struct page *page, + unsigned int order) +{ + /* Remember the order */ + page->order = order; + /* Add the page to the free list */ + llist_add(&page->pcp_llist, &zone->trylock_free_pages); +} + static void free_one_page(struct zone *zone, struct page *page, unsigned long pfn, unsigned int order, fpi_t fpi_flags) { + struct llist_head *llhead; unsigned long flags; - spin_lock_irqsave(&zone->lock, flags); + if (!spin_trylock_irqsave(&zone->lock, flags)) { + if (unlikely(fpi_flags & FPI_TRYLOCK)) { + add_page_to_zone_llist(zone, page, order); + return; + } + spin_lock_irqsave(&zone->lock, flags); + } + + /* The lock succeeded. Process deferred pages. */ + llhead = &zone->trylock_free_pages; + if (unlikely(!llist_empty(llhead) && !(fpi_flags & FPI_TRYLOCK))) { + struct llist_node *llnode; + struct page *p, *tmp; + + llnode = llist_del_all(llhead); + llist_for_each_entry_safe(p, tmp, llnode, pcp_llist) { + unsigned int p_order = p->order; + + split_large_buddy(zone, p, page_to_pfn(p), p_order, fpi_flags); + __count_vm_events(PGFREE, 1 << p_order); + } + } split_large_buddy(zone, page, pfn, order, fpi_flags); spin_unlock_irqrestore(&zone->lock, flags); @@ -2598,7 +2632,7 @@ static int nr_pcp_high(struct per_cpu_pages *pcp, struct zone *zone, static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, struct page *page, int migratetype, - unsigned int order) + unsigned int order, fpi_t fpi_flags) { int high, batch; int pindex; @@ -2633,6 +2667,14 @@ static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, } if (pcp->free_count < (batch << CONFIG_PCP_BATCH_SCALE_MAX)) pcp->free_count += (1 << order); + + if (unlikely(fpi_flags & FPI_TRYLOCK)) { + /* + * Do not attempt to take a zone lock. Let pcp->count get + * over high mark temporarily. + */ + return; + } high = nr_pcp_high(pcp, zone, batch, free_high); if (pcp->count >= high) { free_pcppages_bulk(zone, nr_pcp_free(pcp, batch, high, free_high), @@ -2647,7 +2689,8 @@ static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, /* * Free a pcp page */ -void free_unref_page(struct page *page, unsigned int order) +static void __free_unref_page(struct page *page, unsigned int order, + fpi_t fpi_flags) { unsigned long __maybe_unused UP_flags; struct per_cpu_pages *pcp; @@ -2656,7 +2699,7 @@ void free_unref_page(struct page *page, unsigned int order) int migratetype; if (!pcp_allowed_order(order)) { - __free_pages_ok(page, order, FPI_NONE); + __free_pages_ok(page, order, fpi_flags); return; } @@ -2673,24 +2716,34 @@ void free_unref_page(struct page *page, unsigned int order) migratetype = get_pfnblock_migratetype(page, pfn); if (unlikely(migratetype >= MIGRATE_PCPTYPES)) { if (unlikely(is_migrate_isolate(migratetype))) { - free_one_page(page_zone(page), page, pfn, order, FPI_NONE); + free_one_page(page_zone(page), page, pfn, order, fpi_flags); return; } migratetype = MIGRATE_MOVABLE; } zone = page_zone(page); + if (unlikely((fpi_flags & FPI_TRYLOCK) && IS_ENABLED(CONFIG_PREEMPT_RT) + && (in_nmi() || in_hardirq()))) { + add_page_to_zone_llist(zone, page, order); + return; + } pcp_trylock_prepare(UP_flags); pcp = pcp_spin_trylock(zone->per_cpu_pageset); if (pcp) { - free_unref_page_commit(zone, pcp, page, migratetype, order); + free_unref_page_commit(zone, pcp, page, migratetype, order, fpi_flags); pcp_spin_unlock(pcp); } else { - free_one_page(zone, page, pfn, order, FPI_NONE); + free_one_page(zone, page, pfn, order, fpi_flags); } pcp_trylock_finish(UP_flags); } +void free_unref_page(struct page *page, unsigned int order) +{ + __free_unref_page(page, order, FPI_NONE); +} + /* * Free a batch of folios */ @@ -2779,7 +2832,7 @@ void free_unref_folios(struct folio_batch *folios) trace_mm_page_free_batched(&folio->page); free_unref_page_commit(zone, pcp, &folio->page, migratetype, - order); + order, FPI_NONE); } if (pcp) { @@ -4843,22 +4896,37 @@ EXPORT_SYMBOL(get_zeroed_page_noprof); * Context: May be called in interrupt context or while holding a normal * spinlock, but not in NMI context or while holding a raw spinlock. */ -void __free_pages(struct page *page, unsigned int order) +static void ___free_pages(struct page *page, unsigned int order, + fpi_t fpi_flags) { /* get PageHead before we drop reference */ int head = PageHead(page); struct alloc_tag *tag = pgalloc_tag_get(page); if (put_page_testzero(page)) - free_unref_page(page, order); + __free_unref_page(page, order, fpi_flags); else if (!head) { pgalloc_tag_sub_pages(tag, (1 << order) - 1); while (order-- > 0) - free_unref_page(page + (1 << order), order); + __free_unref_page(page + (1 << order), order, + fpi_flags); } } +void __free_pages(struct page *page, unsigned int order) +{ + ___free_pages(page, order, FPI_NONE); +} EXPORT_SYMBOL(__free_pages); +/* + * Can be called while holding raw_spin_lock or from IRQ and NMI for any + * page type (not only those that came from try_alloc_pages) + */ +void free_pages_nolock(struct page *page, unsigned int order) +{ + ___free_pages(page, order, FPI_TRYLOCK); +} + void free_pages(unsigned long addr, unsigned int order) { if (addr != 0) { diff --git a/mm/page_owner.c b/mm/page_owner.c index 2d6360eaccbb..90e31d0e3ed7 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -294,7 +294,13 @@ void __reset_page_owner(struct page *page, unsigned short order) page_owner = get_page_owner(page_ext); alloc_handle = page_owner->handle; - handle = save_stack(GFP_NOWAIT | __GFP_NOWARN); + /* + * Do not specify GFP_NOWAIT to make gfpflags_allow_spinning() == false + * to prevent issues in stack_depot_save(). + * This is similar to try_alloc_pages() gfp flags, but only used + * to signal stack_depot to avoid spin_locks. + */ + handle = save_stack(__GFP_NOWARN); __update_page_owner_free_handle(page_ext, handle, order, current->pid, current->tgid, free_ts_nsec); page_ext_put(page_ext); From patchwork Fri Jan 24 03:56:52 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13948874 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CE40F1D7998 for ; Fri, 24 Jan 2025 03:57:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737691030; cv=none; b=OrdM+jQA42N04/e87eHaPW6WQNOmgvNjYDkL6FcA//pN4tbiRS+v+VG9I1QYdUN+ygeXF6w+ijE44T9H06gQLwqy6dNk7aI6n8cegt6mZ6OLHtQT3v+4i0LbE1pkvYGdQkpZm2ZM41SIJ1u2wiUG/HNnh1MidPa11EA50+KOiyg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737691030; c=relaxed/simple; bh=6Qk1+Qx2cCbWbbuR9eij2Zo5YTD9MoCgs+QDBDGb+LQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=dm1zNseJQJr9py49qbbQ7wQcYOoi6WYq4rS7HRGMzOtWf9O5thLiYDiAb0ZgPZQ8JnjsV2YCqZCWSmeLnUZ3ZTrbc3mJjgpiz7DBuWHcC0D2D+wmEqQoWoL0/wwdYrVPflrpXwj5SHdKCYTMn4bNf6HQ9hN1G5tlGOs9VttrNFo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=KVXttmkK; arc=none smtp.client-ip=209.85.216.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="KVXttmkK" Received: by mail-pj1-f54.google.com with SMTP id 98e67ed59e1d1-2ee51f8c47dso2510929a91.1 for ; Thu, 23 Jan 2025 19:57:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1737691028; x=1738295828; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AhWoq2FKuOuB0wGZug1HgiVNgCDD6ysBALAFLwpbX6U=; b=KVXttmkKyPW4PtSV2KwZtKz3Zv1DdLfcZ7y4Uon9jOrXNrZ8HcQMXkMiArYc4k7FAh NCZCBHec+2uwD4YVmgSmmfNWh3sCpRbn/1rl8uO2bfyNkuiAC9acZlgBEPprT7c9ILKH PTXr2E2zt1GbaSES+GYL5uxwM1o0s8p7Lvj9YEVwSOgl7p/v130MYC9qhX58Uh5XGTKY F6CTQRRFt94xXKhcX1fcE/pbPvdqkMTluOcXE53jt9TihzNuJvEE5pgaupxYeTTfipXj w4+uGqUiIUonglXeBc0q1sHvS+GsjTid/tGcXLSD1dn7WLpxVLRe+KcBJVZJPmwVuOej C6aA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737691028; x=1738295828; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AhWoq2FKuOuB0wGZug1HgiVNgCDD6ysBALAFLwpbX6U=; b=kKDeTSeJ5bp8Q2E5FOpwARh0MIh/tuPSHoW8KgwzOiJFPk97kyw7mgrd2v9MlLupDZ Jz7udMjHy9kd/uqVTqhv8PlL9eZZD9yacUaw6Kz1nDpr6sKw3gqOUA6tadotEP+Q00Jq TSK7ovc4X80S43PsVVK9Ls7MgKCCM/cdpHqHEv75LNWxSRXizgwVvgyP0ILriImV6ncg 8GBFOzUd5Aq01rdP2HDjAyBMJJw6CK4kjtkytA87YPg0hIaB3EbGiNQBURBYbO8Ml7sU 8exmvlqZhlTJ3Y9ROoO2E6IyIbzhNXIs9mB4I0/JFgW2cQNLpxfE1G6ig2qMiZHSwTlE 9lgA== X-Gm-Message-State: AOJu0Ywt8dYpNoIdgehVCOn0XrDtRozUYgT/ppnR//RpVQL9DtWl/19s YEa9o6sQ6+AY4U9e3wvyN9Z/eU0iCgUyF53xNcFGOfpC216gqDz36r6XOA== X-Gm-Gg: ASbGnctup1vZ50RdsC7p3wxYUGkNxNd8e4+S8RDYeermp7Bn0pmrkxj5vQzWHlhz9iv 1+4c6xGLlKJCc0xkVZagkt6t1A4VNeCKMy5+44PXoSKtdchdLusuFkKHjQJJXb0q6duKAWfOD2u Xo8zMZDYosE4tc1iF89LUw0iNLpU8FA95YKJGmcdEe9NpezPjm5oD7cRDdqLbp4tG5lqHbq97As YXAiPHhkc1z46Gs9Nlfx8dWl7vZeAP+2Lx+dUPTVB+ob4Md+zOoJ2HPRKTf+Qpx65EXHAnVW4SW Dbn8Q6wOGQXLefjjqy0xqjZiYzr1SnH9x5Ok8qA= X-Google-Smtp-Source: AGHT+IHjf895nN6HJ8zSuYUNSFGgzi7wNzn53xXi9fqFCiweThxJGpq1eNj/CM9sNN+YbDe1QrSLvw== X-Received: by 2002:a17:90b:3bc3:b0:2ee:f80c:6889 with SMTP id 98e67ed59e1d1-2f782d9a9eamr46529484a91.33.1737691027837; Thu, 23 Jan 2025 19:57:07 -0800 (PST) Received: from macbookpro.lan ([2603:3023:16e:5000:8af:ecd2:44cd:8027]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2f7ffaf8b34sm541383a91.30.2025.01.23.19.57.06 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 23 Jan 2025 19:57:07 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v6 3/6] locking/local_lock: Introduce local_trylock_t and local_trylock_irqsave() Date: Thu, 23 Jan 2025 19:56:52 -0800 Message-Id: <20250124035655.78899-4-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250124035655.78899-1-alexei.starovoitov@gmail.com> References: <20250124035655.78899-1-alexei.starovoitov@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net From: Alexei Starovoitov In !PREEMPT_RT local_lock_irqsave() disables interrupts to protect critical section, but it doesn't prevent NMI, so the fully reentrant code cannot use local_lock_irqsave() for exclusive access. Introduce local_trylock_t and local_trylock_irqsave() that disables interrupts and sets active=1, so local_trylock_irqsave() from NMI of the same lock will return false. In PREEMPT_RT local_lock_irqsave() maps to preemptible spin_lock(). Map local_trylock_irqsave() to preemptible spin_trylock(). When in hard IRQ or NMI return false right away, since spin_trylock() is not safe due to PI issues. Note there is no need to use local_inc for active variable, since it's a percpu variable with strict nesting scopes. Usage: local_lock_t lock; // sizeof(lock) == 0 in !RT local_lock_irqsave(&lock, ...); // irqsave as before if (local_trylock_irqsave(&lock, ...)) // compilation error local_trylock_t lock; // sizeof(lock) == 4 in !RT local_lock_irqsave(&lock, ...); // irqsave and active = 1 if (local_trylock_irqsave(&lock, ...)) // if (!active) irqsave Signed-off-by: Alexei Starovoitov --- include/linux/local_lock.h | 9 ++++ include/linux/local_lock_internal.h | 79 ++++++++++++++++++++++++++++- 2 files changed, 86 insertions(+), 2 deletions(-) diff --git a/include/linux/local_lock.h b/include/linux/local_lock.h index 091dc0b6bdfb..f4bc3e9b2b20 100644 --- a/include/linux/local_lock.h +++ b/include/linux/local_lock.h @@ -30,6 +30,15 @@ #define local_lock_irqsave(lock, flags) \ __local_lock_irqsave(lock, flags) +/** + * local_trylock_irqsave - Try to acquire a per CPU local lock, save and disable + * interrupts. Fails in PREEMPT_RT when in hard IRQ or NMI. + * @lock: The lock variable + * @flags: Storage for interrupt flags + */ +#define local_trylock_irqsave(lock, flags) \ + __local_trylock_irqsave(lock, flags) + /** * local_unlock - Release a per CPU local lock * @lock: The lock variable diff --git a/include/linux/local_lock_internal.h b/include/linux/local_lock_internal.h index 8dd71fbbb6d2..14757b7aea99 100644 --- a/include/linux/local_lock_internal.h +++ b/include/linux/local_lock_internal.h @@ -15,6 +15,19 @@ typedef struct { #endif } local_lock_t; +typedef struct { +#ifdef CONFIG_DEBUG_LOCK_ALLOC + struct lockdep_map dep_map; + struct task_struct *owner; +#endif + /* + * Same layout as local_lock_t with 'active' field + * at the end, since (local_trylock_t *) will be + * casted to (local_lock_t *). + */ + int active; +} local_trylock_t; + #ifdef CONFIG_DEBUG_LOCK_ALLOC # define LOCAL_LOCK_DEBUG_INIT(lockname) \ .dep_map = { \ @@ -31,6 +44,13 @@ static inline void local_lock_acquire(local_lock_t *l) l->owner = current; } +static inline void local_trylock_acquire(local_lock_t *l) +{ + lock_map_acquire_try(&l->dep_map); + DEBUG_LOCKS_WARN_ON(l->owner); + l->owner = current; +} + static inline void local_lock_release(local_lock_t *l) { DEBUG_LOCKS_WARN_ON(l->owner != current); @@ -45,6 +65,7 @@ static inline void local_lock_debug_init(local_lock_t *l) #else /* CONFIG_DEBUG_LOCK_ALLOC */ # define LOCAL_LOCK_DEBUG_INIT(lockname) static inline void local_lock_acquire(local_lock_t *l) { } +static inline void local_trylock_acquire(local_lock_t *l) { } static inline void local_lock_release(local_lock_t *l) { } static inline void local_lock_debug_init(local_lock_t *l) { } #endif /* !CONFIG_DEBUG_LOCK_ALLOC */ @@ -87,10 +108,37 @@ do { \ #define __local_lock_irqsave(lock, flags) \ do { \ + local_trylock_t *tl; \ + local_lock_t *l; \ local_irq_save(flags); \ - local_lock_acquire(this_cpu_ptr(lock)); \ + l = (local_lock_t *)this_cpu_ptr(lock); \ + tl = (local_trylock_t *)l; \ + _Generic((lock), \ + local_trylock_t *: ({ \ + lockdep_assert(tl->active == 0);\ + WRITE_ONCE(tl->active, 1); \ + }), \ + default:(void)0); \ + local_lock_acquire(l); \ } while (0) + +#define __local_trylock_irqsave(lock, flags) \ + ({ \ + local_trylock_t *tl; \ + local_irq_save(flags); \ + tl = this_cpu_ptr(lock); \ + if (READ_ONCE(tl->active) == 1) { \ + local_irq_restore(flags); \ + tl = NULL; \ + } else { \ + WRITE_ONCE(tl->active, 1); \ + local_trylock_acquire( \ + (local_lock_t *)tl); \ + } \ + !!tl; \ + }) + #define __local_unlock(lock) \ do { \ local_lock_release(this_cpu_ptr(lock)); \ @@ -105,7 +153,17 @@ do { \ #define __local_unlock_irqrestore(lock, flags) \ do { \ - local_lock_release(this_cpu_ptr(lock)); \ + local_trylock_t *tl; \ + local_lock_t *l; \ + l = (local_lock_t *)this_cpu_ptr(lock); \ + tl = (local_trylock_t *)l; \ + _Generic((lock), \ + local_trylock_t *: ({ \ + lockdep_assert(tl->active == 1);\ + WRITE_ONCE(tl->active, 0); \ + }), \ + default:(void)0); \ + local_lock_release(l); \ local_irq_restore(flags); \ } while (0) @@ -125,6 +183,7 @@ do { \ * critical section while staying preemptible. */ typedef spinlock_t local_lock_t; +typedef spinlock_t local_trylock_t; #define INIT_LOCAL_LOCK(lockname) __LOCAL_SPIN_LOCK_UNLOCKED((lockname)) @@ -148,6 +207,22 @@ typedef spinlock_t local_lock_t; __local_lock(lock); \ } while (0) +#define __local_trylock_irqsave(lock, flags) \ + ({ \ + __label__ out; \ + int ret = 0; \ + typecheck(unsigned long, flags); \ + flags = 0; \ + if (in_nmi() || in_hardirq()) \ + goto out; \ + migrate_disable(); \ + ret = spin_trylock(this_cpu_ptr((lock))); \ + if (!ret) \ + migrate_enable(); \ + out: \ + ret; \ + }) + #define __local_unlock(__lock) \ do { \ spin_unlock(this_cpu_ptr((__lock))); \ From patchwork Fri Jan 24 03:56:53 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13948875 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 21F381D7982 for ; Fri, 24 Jan 2025 03:57:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737691033; cv=none; b=mRlvNRnqJfdBNUWMCkTEhlM6cTuQCjCZIbd2ZM3G87I2pvUUm6dmqquiRNXrX+QTO/u/urn9FAA4mPYrCnHirts5XTENAqUnhcOpDeRvFkYGQP1H9BWwoFf67hcn1trY01ylixch11rFH4mmlJa3lNEdVFOhE2yp+Sq0O+HAUko= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737691033; c=relaxed/simple; bh=wOI4QvmQ5PKB479PGhfquAy80qR9vCS2TKGkgdqfjOI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=PrV+FZRihC+JECo3FaFOkTt0NOfW7cGqoNro95pD99g44T93LXOmIwBNeiRavHQl9n9dU8u5ql2pTSz3v+NRwrbZIWV9LT39Q0tjdp95NHLHWuS+yko8NcjKfpQVOPb76ik+U0hDSJ5CJg8M/8in/SKQAMRQKFw0fXUj3VtRTjI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=DL6ZBxd8; arc=none smtp.client-ip=209.85.216.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="DL6ZBxd8" Received: by mail-pj1-f49.google.com with SMTP id 98e67ed59e1d1-2f4448bf96fso2415766a91.0 for ; Thu, 23 Jan 2025 19:57:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1737691031; x=1738295831; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=c4la5/cXBbk2HatOeHKkn8ryi3z3XRRBiHexjMZ10FA=; b=DL6ZBxd8h7GfSiDO1QHldhe3kqE9Du45oIcQGDdqUHg7mM3WqfBmF4QaNyggf8VkHs V8To5nmyJIgZaU3TK9jupmKPz41k+LNaSfiZrgsiOmrFVs5J744qe5l8k55Qj+yVS+u8 SPaOYWBdGx2OfNSq5fFvGpRefqTg6KE5PClws3QYdz9kytlGouxsb7p5ZOIanw2Q8C1G Hnqi7mt3hf5FqqSKdgfzucBzsaCehIU/1qUq1ssUpU0pkjDhJdQ55iH9MiJH+o+MWwUs lStMLFw2X6ufId97PukQh11yYmHVWFlbljDieAiULGLVlfHBVq+c3k1NmBOYwbXlPwR+ GnvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737691031; x=1738295831; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=c4la5/cXBbk2HatOeHKkn8ryi3z3XRRBiHexjMZ10FA=; b=aFV/fy4w9Nmfe/MYI8iINwW7IjEVcnsJ1K5mEvuPJoAq0IWSjDGfwweO1gVAOPq7Uv NQ6CnIrEzFy8qmaR9tut3dkzpBW8PTg2nZ7Srl0IxGxgRBGnu4mf2juxqE6fAfYGC05L Yqhmz7iQ27dy7ZYCANs1r+MUlniA6s2oa5Cxf5mZ+yjhnyklSMuSLzEac73POmE94PHA NWTb4tu/KinO6R5ppFEF6GCE4vIKVKuIlzXqSPysh+IEBJ19VFGE9vv2PheGxH3+VYwc KZ5uWCPtZ3Dg2apcNgy8hlww6VRfpUGzl4YvY1YIvebcuOgGAeiqfAwPNj743A1OOsp8 LvtQ== X-Gm-Message-State: AOJu0Yw1qJ3CucXFa7e+M5SaNWWfxgj+ya2vuW7oLoVWYEzV96qMuzeU di2GZaXGKfILiFFCNN7oSTktNGG2vRTSbt3i9lHcCfJcPZLUzugRBXXzxQ== X-Gm-Gg: ASbGncsRgyj6I3Gcwc4Yd0FGA/9EpiAY1Gl3VBU0h4kQ6Lrdm+EtossfVqwqicP4j0C XfTEiN1yaGCErKLZzi8GYsYpwwUxt0Ziva0j62tjyIwx8UJx7jOhMtFypWfMSFpUUzRzPIBZUka P5UDdwRxYmy+dYyrlLBhC1tIczFMxPzP6LRRaDbABNxlXpZkkydiYZ1DYJnr2S9+fPv8WNDaeDh etPlQOSfkb+CJASmVL7T2IzW+ta52QxPLfkcK+ONI46SdwbD94/wWf0h/wsXwa7c2ReuUbMLZiZ NOufjRGMMLNIngyKsuuhiz8lVP8qaRrKO9rlHzE= X-Google-Smtp-Source: AGHT+IEwvNwHQhd2JUrRjyZznwdi2ypsyJojHQDiGm/2CjTDfPwIknvaSpVusJ/6NdDxjUU11w5G/A== X-Received: by 2002:a17:90b:4c45:b0:2ee:bc1d:f98b with SMTP id 98e67ed59e1d1-2f782d5d822mr38678528a91.31.1737691031062; Thu, 23 Jan 2025 19:57:11 -0800 (PST) Received: from macbookpro.lan ([2603:3023:16e:5000:8af:ecd2:44cd:8027]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2f7ffa6b2bdsm549950a91.25.2025.01.23.19.57.09 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 23 Jan 2025 19:57:10 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v6 4/6] memcg: Use trylock to access memcg stock_lock. Date: Thu, 23 Jan 2025 19:56:53 -0800 Message-Id: <20250124035655.78899-5-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250124035655.78899-1-alexei.starovoitov@gmail.com> References: <20250124035655.78899-1-alexei.starovoitov@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net From: Alexei Starovoitov Teach memcg to operate under trylock conditions when spinning locks cannot be used. local_trylock might fail and this would lead to charge cache bypass if the calling context doesn't allow spinning (gfpflags_allow_spinning). In those cases charge the memcg counter directly and fail early if that is not possible. This might cause a pre-mature charge failing but it will allow an opportunistic charging that is safe from try_alloc_pages path. Acked-by: Michal Hocko Acked-by: Vlastimil Babka Acked-by: Shakeel Butt Signed-off-by: Alexei Starovoitov --- mm/memcontrol.c | 30 +++++++++++++++++++++++++----- 1 file changed, 25 insertions(+), 5 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 7b3503d12aaf..9caca00cb7de 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1722,7 +1722,7 @@ void mem_cgroup_print_oom_group(struct mem_cgroup *memcg) } struct memcg_stock_pcp { - local_lock_t stock_lock; + local_trylock_t stock_lock; struct mem_cgroup *cached; /* this never be root cgroup */ unsigned int nr_pages; @@ -1756,7 +1756,8 @@ static bool obj_stock_flush_required(struct memcg_stock_pcp *stock, * * returns true if successful, false otherwise. */ -static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) +static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages, + gfp_t gfp_mask) { struct memcg_stock_pcp *stock; unsigned int stock_pages; @@ -1766,7 +1767,11 @@ static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) if (nr_pages > MEMCG_CHARGE_BATCH) return ret; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + if (!local_trylock_irqsave(&memcg_stock.stock_lock, flags)) { + if (!gfpflags_allow_spinning(gfp_mask)) + return ret; + local_lock_irqsave(&memcg_stock.stock_lock, flags); + } stock = this_cpu_ptr(&memcg_stock); stock_pages = READ_ONCE(stock->nr_pages); @@ -1851,7 +1856,18 @@ static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) { unsigned long flags; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + if (!local_trylock_irqsave(&memcg_stock.stock_lock, flags)) { + /* + * In case of unlikely failure to lock percpu stock_lock + * uncharge memcg directly. + */ + if (mem_cgroup_is_root(memcg)) + return; + page_counter_uncharge(&memcg->memory, nr_pages); + if (do_memsw_account()) + page_counter_uncharge(&memcg->memsw, nr_pages); + return; + } __refill_stock(memcg, nr_pages); local_unlock_irqrestore(&memcg_stock.stock_lock, flags); } @@ -2196,9 +2212,13 @@ int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, unsigned long pflags; retry: - if (consume_stock(memcg, nr_pages)) + if (consume_stock(memcg, nr_pages, gfp_mask)) return 0; + if (!gfpflags_allow_spinning(gfp_mask)) + /* Avoid the refill and flush of the older stock */ + batch = nr_pages; + if (!do_memsw_account() || page_counter_try_charge(&memcg->memsw, batch, &counter)) { if (page_counter_try_charge(&memcg->memory, batch, &counter)) From patchwork Fri Jan 24 03:56:54 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13948876 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4AC3D1D7982 for ; Fri, 24 Jan 2025 03:57:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737691036; cv=none; b=Esyldn/YSPvWUnedESLeVfz+zjaUZBpnIaHTh1KhL24yO5rnlQx+/4W+ir+aZp8yTIrxCw3sJge7BYS93vOuJMI0ZfKnJYQzl7vz22qhlTzwafr2ao1t4CjRgvQB6kqTwSx9nGGRRH+1w0yhiS1YCpT2D0mfVD2F7622leRR3Ps= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737691036; c=relaxed/simple; bh=PkbsL7XC8j5LBWMTJb1ZtearOWIvOtlSfq++Sl18MI8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=uWzPkZcnYhhFuPAG7JSSgl+uT0bpzW++qkEPCD6AGLs0rZ6mW83rULjxP6nc4LmyuHDsU9ZyDW9iiib8iIVxv82VIPHS09GQm1uQEAi3ItsJyPvjOHepXLQG1ayG8HhYCGs8xgI1zS6EJd3Te/szZXCKOMScqzmJPJ8wOVFXkFA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=KxagcC2w; arc=none smtp.client-ip=209.85.214.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="KxagcC2w" Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-21631789fcdso37739505ad.1 for ; Thu, 23 Jan 2025 19:57:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1737691034; x=1738295834; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=iXCJ0gdG5EF9yWrbtRQyNOOIbBLtByDZRqzMTlbBC2I=; b=KxagcC2w9euR1H3y5MEnb+TAXiJT+j/PLy7Ra5vCemvy2DbRrzcCFwmNTxaMsZ26G5 hrmdK3GNKW8vVm64XFsZS4AkCGrepzSRlsHfPw7iKACPb331PWTlpQR5QGuWyetmWpaq Gmyz+iXJ3yw5PBO8uhRoShNbeLI9vn6kCv/dy+eDjtDlKLTifWXVeK3ZG+l3/nPhqSnS egWNXzVrJyIpn8fcBWsJ82TK1B04yVnUjrkjhAy5Q1UFUm0nb1AiOpBMf5I8XkgGQ9IE uVqUlihRt+RpwNUq6gmaJsbT8xjZpDEVLPi0p3FQ38DDDoBX05yzgkq8vwMoM/PvZY5P cFiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737691034; x=1738295834; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iXCJ0gdG5EF9yWrbtRQyNOOIbBLtByDZRqzMTlbBC2I=; b=Iaj1K1twdjIBsjaDk+FZCL7XrGVsNlg9zSyHkXybeGGm6sOS9oQTRSsvBDOpsokjwS ZkJMCdyw0ga6GVasHR78xEMklX5OZO0dQ9yW7D/T7rcQsZ6DJmFNSVJ15oyzkzHl00g8 IAfIvfzTwW3SxqleTIRRJb2uLVeFbYDRS7hqcD7CB3rPVtzepVTfVwzsVMtySUto+5op UTT1vP9z/WltC8J3BDU6JciLRzn7STGofrRYhJQ12rb+RqzQ9Vh4bp4jhk3nLng55tBd srkl8PZ4u5ZcC1W48LRyYURFQTrjSwj3TP5VcAxeP/Y3Q3F20WRq6yZCZ3ZaklvaUWUU lLIg== X-Gm-Message-State: AOJu0YwnaVb1JFTPb0hZTQ1FP5fYnk1HkSly1i3zDwFdx4lWnn3K3Air j9bPnAcM1kaRufGYAOhD+OW28/EhTfLTz/I0iCvM/Y/a3wfofwFfd3lZ4w== X-Gm-Gg: ASbGncskredptSn+t1cgh2JkrCv2eCpUbQsKGOlcNowwXbkHQB+fV93jWiGszWqinE1 4X3GAAwZ5fr2WXFcmIbSUdQaNkayV0uW1YB5UjK7nSKiOjp5qvB88z+gcQwkm8LGP3/10muLpuh R6iljzHxjYGDMyJsTCyju0kLQ676uzGvXgtURwMABWjG5/6taM4vdNQTkwhi7r8ZEmERJGmJgs2 ZFSTyUaP5gcLajcEzbw+AXHO2ZYAqw9IYusqEvlsBcPNY24ZD0Cc8JPQhqGMDb+piwVo1nYj4SD M6C7tyocb3oKuvaPSunWsGfLlvQeP1LfyzBCCLA= X-Google-Smtp-Source: AGHT+IGfeaoENw63hXIHNaiPfBmvJhfA/W3E60RsZYSAmDLMCHiy5FG9LHtcRMUd/Av34DsSkZ4Ksg== X-Received: by 2002:a05:6a21:9210:b0:1e1:a7a1:22a9 with SMTP id adf61e73a8af0-1eb76cf0566mr2444457637.16.1737691034346; Thu, 23 Jan 2025 19:57:14 -0800 (PST) Received: from macbookpro.lan ([2603:3023:16e:5000:8af:ecd2:44cd:8027]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-ac495d5531bsm652054a12.47.2025.01.23.19.57.13 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 23 Jan 2025 19:57:14 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v6 5/6] mm, bpf: Use memcg in try_alloc_pages(). Date: Thu, 23 Jan 2025 19:56:54 -0800 Message-Id: <20250124035655.78899-6-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250124035655.78899-1-alexei.starovoitov@gmail.com> References: <20250124035655.78899-1-alexei.starovoitov@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net From: Alexei Starovoitov Unconditionally use __GFP_ACCOUNT in try_alloc_pages(). The caller is responsible to setup memcg correctly. All BPF memory accounting is memcg based. Acked-by: Vlastimil Babka Acked-by: Shakeel Butt Signed-off-by: Alexei Starovoitov --- mm/page_alloc.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index fa750c46e0fc..931cedcda788 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -7146,7 +7146,8 @@ struct page *try_alloc_pages_noprof(int nid, unsigned int order) * specify it here to highlight that try_alloc_pages() * doesn't want to deplete reserves. */ - gfp_t alloc_gfp = __GFP_NOWARN | __GFP_ZERO | __GFP_NOMEMALLOC; + gfp_t alloc_gfp = __GFP_NOWARN | __GFP_ZERO | __GFP_NOMEMALLOC + | __GFP_ACCOUNT; unsigned int alloc_flags = ALLOC_TRYLOCK; struct alloc_context ac = { }; struct page *page; @@ -7190,6 +7191,11 @@ struct page *try_alloc_pages_noprof(int nid, unsigned int order) /* Unlike regular alloc_pages() there is no __alloc_pages_slowpath(). */ + if (memcg_kmem_online() && page && + unlikely(__memcg_kmem_charge_page(page, alloc_gfp, order) != 0)) { + free_pages_nolock(page, order); + page = NULL; + } trace_mm_page_alloc(page, order, alloc_gfp, ac.migratetype); kmsan_alloc_page(page, order, alloc_gfp); return page; From patchwork Fri Jan 24 03:56:55 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13948877 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A1E1E1D6DC8 for ; Fri, 24 Jan 2025 03:57:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737691040; cv=none; b=hvY3rq4NdlrZN7rX40j2nUY9IKdY0XuJ/XjyZ/AHO5oTBphcOjTpJwSMNwENWcMVN10QO3GjX3CQLOGk77km+yI8nD7IZ5VN/Vd8Q28Tx1UwywkOVupj0fePtbHlwZMsCHQs4lFChNVUSLHhT7dpnTjsYF2mGFSEkdRY1BRQ7kQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737691040; c=relaxed/simple; bh=vfzkPtuMn7THzMDejWzHSa5NwJvatkmrRDhM3Ln6UxU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=uKZEl9/A1KGenw+n7+xVxEjMJ99OCz5e5OBRX614649WmFlskidy/88DRVCtOHKJ036BwcePs+NfvzVXmLeSsICMNg2XpWl5isdE1M6zN0COyyPNjBEsaiX6WX5S/62LHvncG5AtWMkW1+xm0MblieGcayWBmJB1ONyWBg9zU4U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ktn0zRGd; arc=none smtp.client-ip=209.85.216.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ktn0zRGd" Received: by mail-pj1-f45.google.com with SMTP id 98e67ed59e1d1-2ef28f07dbaso2498765a91.2 for ; Thu, 23 Jan 2025 19:57:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1737691037; x=1738295837; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bDuPZRtqsVh7bFH3yxNIFanE739bKkzWZ/UfDyq/wdw=; b=ktn0zRGdtASZN6mTP1vyErFidX5N867PWQOBxH78BelUihvPU4W1xWThYpXjhZSJ7s 2CQETjU0X+KXOclyim2zkJGADrUfiG9Q+9MmzjooAEZHHk12knlCZFoDrXDxrFP3LnLw uwnpgmBkJA/sqA5SzbNuE25q3v093S1yTzTxUQRBcnXO+6+RlH3EjHg2HyIRcDLhgJ8w TqSBKx7FljL9Ko9mYF3gyGyoFNSRb3ASdDEGe+YCHXXqFJy2v44pqV4CfjxHqHXxDJZx cff5iMlIvY8mKp3XJ/MaBVoSD9w3VGiYL8NpKgtL+YCreyWmVJkrjoHBCkswr/tVfLe+ c85Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737691037; x=1738295837; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bDuPZRtqsVh7bFH3yxNIFanE739bKkzWZ/UfDyq/wdw=; b=Wn3qjdNerb6kKZeDikgE84w3uE9LoIftncvqAD+y1tOJvrzO38lGRTCTxr2N7mm7l+ zDJmpSP0vwiZKoTddWIQXIrXkE3TzFtE42NzaMuvlBHv0d91B9DVsEjvfx0pt8UC22MQ xxO5SQgdeXB4Xshvic/cXcUCYNdmAtAbzzYJcbZQj2fn8IS/V+fQ1aY7sz75s21AIxli SQr0wwBshtsA/n6Z03F73xJpMkRipHEde27s1nOSosfGcAz+vqpe6RMTxTsLMXDHTG42 hCeZQjO7RScM2eg42znjhWWznSXM3L3fVnMjIBJTNYGouk/wn0N6ea1QdzmGWpqMdLdh 39HA== X-Gm-Message-State: AOJu0YzLCkIvJyD8o91loGEFRAU3iizB677SAyBjlYdChdxNF0Rylj6Q rwcXMwJ+Rz6HEDfFrBFa8bxphSfdF5TcSxGim5Da3plSVW+8MrQWvEyJCQ== X-Gm-Gg: ASbGncvXrcifPcKcmEmGG78L36IGNj7gGsT0JPrGSTQasi0DhOyNTu/gbrtwDuUMaI9 GeB6UOCsn6Zw7+DfeBqI9wvY89Cm4c1Wv2dpazRjlyf1Tm5ZuTEIhldajtN1QW/9uoWx9O1ocdg Bzd50nijt5sPdK+/RUONLcR+3Fsl0c96L0XjEMM0pGiGorgQ7MdADV9a7xmUwheGgYjS1KvjtH5 45F2pNbq6EJj0K/blxpOI2WQYG2INQSucJmImwH4qWe0D2y9LbodiG0sOkVquyMR6cXXYpnKJRO QW9cX/ZsikVYYXM4CH8R6mlCkoVFpnBJIKf0GvI= X-Google-Smtp-Source: AGHT+IE+LCbRskWbsR6jz9xA0dh2MfdRvUJzJjh376jPaUjhK25k9uihZ7etnl2RsWLAETsaV50r6A== X-Received: by 2002:a17:90b:3547:b0:2f7:ef57:c7df with SMTP id 98e67ed59e1d1-2f7ef57c8fbmr11319328a91.7.1737691037538; Thu, 23 Jan 2025 19:57:17 -0800 (PST) Received: from macbookpro.lan ([2603:3023:16e:5000:8af:ecd2:44cd:8027]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2f7ffaf896csm542025a91.34.2025.01.23.19.57.16 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 23 Jan 2025 19:57:17 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v6 6/6] bpf: Use try_alloc_pages() to allocate pages for bpf needs. Date: Thu, 23 Jan 2025 19:56:55 -0800 Message-Id: <20250124035655.78899-7-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250124035655.78899-1-alexei.starovoitov@gmail.com> References: <20250124035655.78899-1-alexei.starovoitov@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net From: Alexei Starovoitov Use try_alloc_pages() and free_pages_nolock() for BPF needs when context doesn't allow using normal alloc_pages. This is a prerequisite for further work. Signed-off-by: Alexei Starovoitov --- include/linux/bpf.h | 2 +- kernel/bpf/arena.c | 5 ++--- kernel/bpf/syscall.c | 23 ++++++++++++++++++++--- 3 files changed, 23 insertions(+), 7 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index f3f50e29d639..e1838a341817 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -2348,7 +2348,7 @@ int generic_map_delete_batch(struct bpf_map *map, struct bpf_map *bpf_map_get_curr_or_next(u32 *id); struct bpf_prog *bpf_prog_get_curr_or_next(u32 *id); -int bpf_map_alloc_pages(const struct bpf_map *map, gfp_t gfp, int nid, +int bpf_map_alloc_pages(const struct bpf_map *map, int nid, unsigned long nr_pages, struct page **page_array); #ifdef CONFIG_MEMCG void *bpf_map_kmalloc_node(const struct bpf_map *map, size_t size, gfp_t flags, diff --git a/kernel/bpf/arena.c b/kernel/bpf/arena.c index 4b22a651b5d5..642399a5fd9f 100644 --- a/kernel/bpf/arena.c +++ b/kernel/bpf/arena.c @@ -287,7 +287,7 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf) return VM_FAULT_SIGSEGV; /* Account into memcg of the process that created bpf_arena */ - ret = bpf_map_alloc_pages(map, GFP_KERNEL | __GFP_ZERO, NUMA_NO_NODE, 1, &page); + ret = bpf_map_alloc_pages(map, NUMA_NO_NODE, 1, &page); if (ret) { range_tree_set(&arena->rt, vmf->pgoff, 1); return VM_FAULT_SIGSEGV; @@ -465,8 +465,7 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt if (ret) goto out_free_pages; - ret = bpf_map_alloc_pages(&arena->map, GFP_KERNEL | __GFP_ZERO, - node_id, page_cnt, pages); + ret = bpf_map_alloc_pages(&arena->map, node_id, page_cnt, pages); if (ret) goto out; diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 0daf098e3207..55588dbd2fce 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -569,7 +569,24 @@ static void bpf_map_release_memcg(struct bpf_map *map) } #endif -int bpf_map_alloc_pages(const struct bpf_map *map, gfp_t gfp, int nid, +static bool can_alloc_pages(void) +{ + return preempt_count() == 0 && !irqs_disabled() && + !IS_ENABLED(CONFIG_PREEMPT_RT); +} + +static struct page *__bpf_alloc_page(int nid) +{ + if (!can_alloc_pages()) + return try_alloc_pages(nid, 0); + + return alloc_pages_node(nid, + GFP_KERNEL | __GFP_ZERO | __GFP_ACCOUNT + | __GFP_NOWARN, + 0); +} + +int bpf_map_alloc_pages(const struct bpf_map *map, int nid, unsigned long nr_pages, struct page **pages) { unsigned long i, j; @@ -582,14 +599,14 @@ int bpf_map_alloc_pages(const struct bpf_map *map, gfp_t gfp, int nid, old_memcg = set_active_memcg(memcg); #endif for (i = 0; i < nr_pages; i++) { - pg = alloc_pages_node(nid, gfp | __GFP_ACCOUNT, 0); + pg = __bpf_alloc_page(nid); if (pg) { pages[i] = pg; continue; } for (j = 0; j < i; j++) - __free_page(pages[j]); + free_pages_nolock(pages[j], 0); ret = -ENOMEM; break; }