From patchwork Tue Jan 14 02:19:17 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13938319 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85727C02180 for ; Tue, 14 Jan 2025 02:19:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 13B29280001; Mon, 13 Jan 2025 21:19:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0EB8C6B0095; Mon, 13 Jan 2025 21:19:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ECE80280001; Mon, 13 Jan 2025 21:19:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D2F1D6B0093 for ; Mon, 13 Jan 2025 21:19:39 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 7C565C04F9 for ; Tue, 14 Jan 2025 02:19:39 +0000 (UTC) X-FDA: 83004451278.26.8652B0E Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) by imf01.hostedemail.com (Postfix) with ESMTP id 8FEB34000F for ; Tue, 14 Jan 2025 02:19:37 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=SUTLrB+I; spf=pass (imf01.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.216.41 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736821177; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=akgdVcj5xEeNq6DmKIL978uvRiSwDxIZz8GY/zQxfoo=; b=jN5TxVrU50iXG/HBXOWt92dnCuFlK3k/Q6h3GkL3ESnkap2NXFfqZFeaWw34M0zcf1JCUz XfQQgVt6vloMWUYv5kmYnn/pixWr+IYgveujtqnyTKuc9hiK1cgmgWMwFzXCC5DY/a8Bne nPJA+S88G9rFqVTp+87yDK6AazYciHQ= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=SUTLrB+I; spf=pass (imf01.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.216.41 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736821177; a=rsa-sha256; cv=none; b=0Cykhzp3jUvT9xnef3A9VsMpXO1uBtlsYuEF1trS4EnznBMM2bI8R0Y6/alpgC/dsUWtRT saRUPaZaoQZSoKx2id8d2W+lBQA7brtDvTW+PovDeCXnYi2n3pkW+FxcAf+FSq1MsbxcOO IEVAvYg7TWMTtv0/EUfZjHqnOxRllqs= Received: by mail-pj1-f41.google.com with SMTP id 98e67ed59e1d1-2ef72924e53so8326294a91.3 for ; Mon, 13 Jan 2025 18:19:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1736821176; x=1737425976; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=akgdVcj5xEeNq6DmKIL978uvRiSwDxIZz8GY/zQxfoo=; b=SUTLrB+IjJJjye91I/NLwb7pAglp8GBJR/f41rM2yoYEa1any2p1Z9V93CjERbTZzq U9fIF6wX0nuxqSrXK4+ZlF0QijsbJsBweASClqpU7k8xU+9J88CkBXLx7UbJ9kq7s8Er dpRi5LmtezUG2/F5tw7XphYQLpTL/Jhg3K22vrOC+2jaDaq8kSNbdqUy8kQvqhVpg3aE 1eMJz1tsAki6Hk0QigdeofZIHPUjzaT82/gvN+4BP+Ib+jg3tV8JR4yfZYXuhg6N7D6h //oDU/47cRiGXIWJnZLx+c5Qsabby2lxhf2RrjoITJKRjz6xuFj4jLiCOxeLiKeYcZ5Z J37Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736821176; x=1737425976; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=akgdVcj5xEeNq6DmKIL978uvRiSwDxIZz8GY/zQxfoo=; b=Zc8XPQV0LheejRineauLjWA+IGNz2RCVR1ZL5QHF6dvNl4b9qFJAhwdhbbFcsdORSo Y017RyGbkbhaTPmKJiBt/WpM4hc/ay/s0ed8fhYXfiLP9oBbPD+T62uo37Tts/7rwaNt KtJXx6386F3eXsI1p869Bkk7tSda8MnJ3Fw+2gmBxbrkypOlxQaPmxp19ygG7N1mLYzr LzkZ7YTRHBPi2jaI2eSgxTtDTdwwE7+nZJxpmXcQXIA4RdUGG3pT8mh9vEod9cOub9P1 T+QGcTjoKRzm4uMG+SpeoQ3d1Z7hnC/JTZ2u+baZZNR9unycikW2bpgahirF+ZNsrPu6 9Xfw== X-Forwarded-Encrypted: i=1; AJvYcCU201WzChZ7a8Q0hTLANkK2zACdOHsuUNxOevJMQa2JV2iftQ2udXjcApyo5YE1+P90EA/yWt9Baw==@kvack.org X-Gm-Message-State: AOJu0YzuI3807kkdoZH6Bs1SEO2/T2jNnx9c03x4MaVT2Xf4u6NgArQX Wgnsw+9T4xQxgbQ4BLrguvbM2zAA4TNVTznPQgd/6k6xNl2aNC4I X-Gm-Gg: ASbGncsWLvERJzBHM5XVScIt8sHwQVBRN+YPlsV87G8ITFxTmZAvErWWvCO2cu5IH0c gyd1AxCkBYJA9p91eTvrqR7gk/NPjvofcfgyDj/z0FLQD+GJ+7RBqt70WCnKwebKS2D2EeyvXav n7dnjDGT9+n33eTwz4wbrtXD41jbm/7IOt9E84jPNPwaMOCS8BY+niZ9TVZM6lYTgVQKboDD6zB pzgv4fYLzZWnUfQCqtvnANsekCEb7kfnPD13J5h2iUnCNkQ7t8/18s6yzxMnt8lJSZlFK3TZnxg 8LnJQpnu X-Google-Smtp-Source: AGHT+IHnBshZ4mFPAZZGQFlNI39VdZrv1X1j+B7cThKNElaLFNGw9ViuDJNNiPM+Xr5/Kv9IgpitWg== X-Received: by 2002:a17:90b:2712:b0:2ee:48bf:7dc9 with SMTP id 98e67ed59e1d1-2f548eb3a99mr35445590a91.15.1736821176273; Mon, 13 Jan 2025 18:19:36 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:4043]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2f55942eed5sm8658076a91.27.2025.01.13.18.19.34 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 13 Jan 2025 18:19:35 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v4 1/6] mm, bpf: Introduce try_alloc_pages() for opportunistic page allocation Date: Mon, 13 Jan 2025 18:19:17 -0800 Message-Id: <20250114021922.92609-2-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250114021922.92609-1-alexei.starovoitov@gmail.com> References: <20250114021922.92609-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 8FEB34000F X-Rspamd-Server: rspam12 X-Stat-Signature: cf6cs38c611yf356zdhxu3cuendghogw X-Rspam-User: X-HE-Tag: 1736821177-4403 X-HE-Meta: U2FsdGVkX1/6Q4ZjmmNOAUiRTdimDtKAxH/7VtZ1jARHvNlArlRyLL0m0MuBUtgINpWUB7cUnIhRLDv/9JyPgOPKe9hjlbtHExIlghxBJcs41wakDdgWORg0fY/221j4BWDhxaPouJ6E26HMXrF5HRHbic45hG7qxkNxjwlPKKIlNb2O2iM+QODauFFW5awZL7Fn1na3hFMcf5a7GsqunXBhi/qFCKVl4/PJdRc4uz0Mu2eEQmP/jgEfCcI0al01jU4F3e3Rwl7M1fBTdJosBDdPvtgY2n9340BMNWvMvnsfkO4iqZ0GQb8h+9+HNFWYGj4yF4Y1VE7BtrlZeCRq10VZRDEVz9XQnHALUnB49MkB003wtW7MkEPURRu7YQtKKQv6iETkFOD3FUW7U0xasdTbQKbiqj8P+vW/pEiCW863ckENGL27iHBnefP+mLE0YGQoVNiNnPHCArYNQaHv/AvMMzmbDpKWpb4ZPAASzbFOAONDlM6vRCQj2TWXptx4KZLakct9khFX0xYM8BJeu5ZBN5SK/XyQPjZEEZnc+s+F8cXtWjCsnbHTjBgQLEAxp1zwT/86y6yZOMzZg9h3is67CLMm8daqivIwZvAk742V4msl4GNwr124zdKibVDG5quLyCRvFjAWVhJDUprKHVR27rAvQJRjAjwvq9+6QxfKees61zSdHSWiQSaIuMiBd5qwiTqq+gA/YrgVwAFlLor1W9pAmXEi+QYDT7ezHN6JS8UFJbxJRegUCkIMw5M2Iq39tsrtX4NV7cMfsTuryiy3oX+/glDVZDahzT02z6IbpzszTt3aoUJWN0C+CIF4N78UqYp+Hma7MQD01tOhigTtYYIrP+oqWkbVCaTcMY4JL7nR8buMIuR8N7dKnwxBlY00gI9sDJ5fDkX22cyqXj5D6fFDyg+JlrcGCq5VtMfWqYzONvafxPUNQ7zFapdXxGhKPNHYVo+AEBPAPgy x/UAPdPs iTWiF7bpC7aK1qFZ7ZBstMWeK5BSk0e5PVGAd2wmKJaDSlBg/Uj5cYmTGFWvOsv9J7b4c6nE7Ta3uxLAflMBT45kprDh66TV/jlxj4z8EWrjzxGqkiUktnsw1ygl9YV2PnfErwjJz/3KOQeAONGL47rjKAMNETu5rw+Qr9q+MS/k48y4gbJmn/uj472oC1d4il4bsjuDnriJ58vyb4/2RkIwWUM+JunkQKXEe5wA+QZUfTPKRCRkftZv1xfIza67M+8ep+t1H9fnVn2LRpe2WdhIXlf0qq59IyF8b5zAFcGJ9GFLmUCfBb2yXq3VG4moGHn6fSwxkj7/5uyHJIlCVjDwNoyXGQFS3f658h/dyvQOwUx//WC1J5t8VRwkFQNTHyqtrfvt/tE3vUaBqzBWonu2pGJKC5H0TEnF1xzQj3tz0l2PF3knKTnAXHxQjPqhyV+PKoBihwgOZ4mIoE/vI77E/ebD5XKShsBNq0Mr5txUeESPKUn/15KDgd1hCRRZuwPwUXjOpZO6lU1eqDpZPmTDXLVbREZQWMxCWvNSu1TkfoCNgvnPuPIL1Ww== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Tracing BPF programs execute from tracepoints and kprobes where running context is unknown, but they need to request additional memory. The prior workarounds were using pre-allocated memory and BPF specific freelists to satisfy such allocation requests. Instead, introduce gfpflags_allow_spinning() condition that signals to the allocator that running context is unknown. Then rely on percpu free list of pages to allocate a page. The rmqueue_pcplist() should be able to pop the page from. If it fails (due to IRQ re-entrancy or list being empty) then try_alloc_pages() attempts to spin_trylock zone->lock and refill percpu freelist as normal. BPF program may execute with IRQs disabled and zone->lock is sleeping in RT, so trylock is the only option. In theory we can introduce percpu reentrance counter and increment it every time spin_lock_irqsave(&zone->lock, flags) is used, but we cannot rely on it. Even if this cpu is not in page_alloc path the spin_lock_irqsave() is not safe, since BPF prog might be called from tracepoint where preemption is disabled. So trylock only. Note, free_page and memcg are not taught about gfpflags_allow_spinning() condition. The support comes in the next patches. This is a first step towards supporting BPF requirements in SLUB and getting rid of bpf_mem_alloc. That goal was discussed at LSFMM: https://lwn.net/Articles/974138/ Signed-off-by: Alexei Starovoitov Acked-by: Michal Hocko --- include/linux/gfp.h | 22 ++++++++++++ mm/internal.h | 1 + mm/page_alloc.c | 85 +++++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 105 insertions(+), 3 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index b0fe9f62d15b..b41bb6e01781 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -39,6 +39,25 @@ static inline bool gfpflags_allow_blocking(const gfp_t gfp_flags) return !!(gfp_flags & __GFP_DIRECT_RECLAIM); } +static inline bool gfpflags_allow_spinning(const gfp_t gfp_flags) +{ + /* + * !__GFP_DIRECT_RECLAIM -> direct claim is not allowed. + * !__GFP_KSWAPD_RECLAIM -> it's not safe to wake up kswapd. + * All GFP_* flags including GFP_NOWAIT use one or both flags. + * try_alloc_pages() is the only API that doesn't specify either flag. + * + * This is stronger than GFP_NOWAIT or GFP_ATOMIC because + * those are guaranteed to never block on a sleeping lock. + * Here we are enforcing that the allaaction doesn't ever spin + * on any locks (i.e. only trylocks). There is no highlevel + * GFP_$FOO flag for this use in try_alloc_pages() as the + * regular page allocator doesn't fully support this + * allocation mode. + */ + return !(gfp_flags & __GFP_RECLAIM); +} + #ifdef CONFIG_HIGHMEM #define OPT_ZONE_HIGHMEM ZONE_HIGHMEM #else @@ -347,6 +366,9 @@ static inline struct page *alloc_page_vma_noprof(gfp_t gfp, } #define alloc_page_vma(...) alloc_hooks(alloc_page_vma_noprof(__VA_ARGS__)) +struct page *try_alloc_pages_noprof(int nid, unsigned int order); +#define try_alloc_pages(...) alloc_hooks(try_alloc_pages_noprof(__VA_ARGS__)) + extern unsigned long get_free_pages_noprof(gfp_t gfp_mask, unsigned int order); #define __get_free_pages(...) alloc_hooks(get_free_pages_noprof(__VA_ARGS__)) diff --git a/mm/internal.h b/mm/internal.h index cb8d8e8e3ffa..5454fa610aac 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1174,6 +1174,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone, #define ALLOC_NOFRAGMENT 0x0 #endif #define ALLOC_HIGHATOMIC 0x200 /* Allows access to MIGRATE_HIGHATOMIC */ +#define ALLOC_TRYLOCK 0x400 /* Only use spin_trylock in allocation path */ #define ALLOC_KSWAPD 0x800 /* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ /* Flags that allow allocations below the min watermark. */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 1cb4b8c8886d..0f4be88ff131 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2304,7 +2304,11 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order, unsigned long flags; int i; - spin_lock_irqsave(&zone->lock, flags); + if (!spin_trylock_irqsave(&zone->lock, flags)) { + if (unlikely(alloc_flags & ALLOC_TRYLOCK)) + return 0; + spin_lock_irqsave(&zone->lock, flags); + } for (i = 0; i < count; ++i) { struct page *page = __rmqueue(zone, order, migratetype, alloc_flags); @@ -2904,7 +2908,11 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, do { page = NULL; - spin_lock_irqsave(&zone->lock, flags); + if (!spin_trylock_irqsave(&zone->lock, flags)) { + if (unlikely(alloc_flags & ALLOC_TRYLOCK)) + return NULL; + spin_lock_irqsave(&zone->lock, flags); + } if (alloc_flags & ALLOC_HIGHATOMIC) page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC); if (!page) { @@ -4509,7 +4517,8 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, might_alloc(gfp_mask); - if (should_fail_alloc_page(gfp_mask, order)) + if (!(*alloc_flags & ALLOC_TRYLOCK) && + should_fail_alloc_page(gfp_mask, order)) return false; *alloc_flags = gfp_to_alloc_flags_cma(gfp_mask, *alloc_flags); @@ -7023,3 +7032,73 @@ static bool __free_unaccepted(struct page *page) } #endif /* CONFIG_UNACCEPTED_MEMORY */ + +struct page *try_alloc_pages_noprof(int nid, unsigned int order) +{ + /* + * Do not specify __GFP_DIRECT_RECLAIM, since direct claim is not allowed. + * Do not specify __GFP_KSWAPD_RECLAIM either, since wake up of kswapd + * is not safe in arbitrary context. + * + * These two are the conditions for gfpflags_allow_spinning() being true. + * + * Specify __GFP_NOWARN since failing try_alloc_pages() is not a reason + * to warn. Also warn would trigger printk() which is unsafe from + * various contexts. We cannot use printk_deferred_enter() to mitigate, + * since the running context is unknown. + * + * Specify __GFP_ZERO to make sure that call to kmsan_alloc_page() below + * is safe in any context. Also zeroing the page is mandatory for + * BPF use cases. + * + * Though __GFP_NOMEMALLOC is not checked in the code path below, + * specify it here to highlight that try_alloc_pages() + * doesn't want to deplete reserves. + */ + gfp_t alloc_gfp = __GFP_NOWARN | __GFP_ZERO | __GFP_NOMEMALLOC; + unsigned int alloc_flags = ALLOC_TRYLOCK; + struct alloc_context ac = { }; + struct page *page; + + /* + * In RT spin_trylock() may call raw_spin_lock() which is unsafe in NMI. + * If spin_trylock() is called from hard IRQ the current task may be + * waiting for one rt_spin_lock, but rt_spin_trylock() will mark the + * task as the owner of another rt_spin_lock which will confuse PI + * logic, so return immediately if called form hard IRQ or NMI. + * + * Note, irqs_disabled() case is ok. This function can be called + * from raw_spin_lock_irqsave region. + */ + if (IS_ENABLED(CONFIG_PREEMPT_RT) && (in_nmi() || in_hardirq())) + return NULL; + if (!pcp_allowed_order(order)) + return NULL; + +#ifdef CONFIG_UNACCEPTED_MEMORY + /* Bailout, since try_to_accept_memory_one() needs to take a lock */ + if (has_unaccepted_memory()) + return NULL; +#endif + /* Bailout, since _deferred_grow_zone() needs to take a lock */ + if (deferred_pages_enabled()) + return NULL; + + if (nid == NUMA_NO_NODE) + nid = numa_node_id(); + + prepare_alloc_pages(alloc_gfp, order, nid, NULL, &ac, + &alloc_gfp, &alloc_flags); + + /* + * Best effort allocation from percpu free list. + * If it's empty attempt to spin_trylock zone->lock. + */ + page = get_page_from_freelist(alloc_gfp, order, alloc_flags, &ac); + + /* Unlike regular alloc_pages() there is no __alloc_pages_slowpath(). */ + + trace_mm_page_alloc(page, order, alloc_gfp, ac.migratetype); + kmsan_alloc_page(page, order, alloc_gfp); + return page; +} From patchwork Tue Jan 14 02:19:18 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13938320 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D515C02180 for ; Tue, 14 Jan 2025 02:19:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C5E56280003; Mon, 13 Jan 2025 21:19:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C0F7E280002; Mon, 13 Jan 2025 21:19:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A89BE280003; Mon, 13 Jan 2025 21:19:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 85E3E280002 for ; Mon, 13 Jan 2025 21:19:44 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 3D618C04B3 for ; Tue, 14 Jan 2025 02:19:44 +0000 (UTC) X-FDA: 83004451488.18.D672BCE Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf28.hostedemail.com (Postfix) with ESMTP id 44B5CC0012 for ; Tue, 14 Jan 2025 02:19:42 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=JCyamgJi; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf28.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736821182; a=rsa-sha256; cv=none; b=aRl0UaZkVrDAXp8Fx+90QeBFpJTO7s9Cce3+YAIa6XS60izJgZ9CW/obCakjQgsHt9VYoc AJf+F6emneyAqfIObzbCeYv4fTfDKn45vYP1lQ7sbj2NtiLAmYA6D8xypq5aTdcbanJ0fe 90StZAZDfNsjkdfcMi2DBj3vAlNQXkI= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=JCyamgJi; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf28.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736821182; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SQPsX2DHIButJsUpdhJcEvvwgfeMxBo/GFMw0XDovgw=; b=m+VK8ksrpOYd60UJs4l36kNznwEsBnuTMhZIPgW7RTrhiu8j94+cBtU5t6GR1zLrHs/XmC 3aBfXWbM8aJwIxI0teVeQFLO1onbxUFVNzGprd4hfhdS5h6JMa/wr74qx7SYc6MUTTStH5 +o6DceAMAYfbIZrtKKlyJ15tHFhkyzY= Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-2166651f752so110785145ad.3 for ; Mon, 13 Jan 2025 18:19:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1736821181; x=1737425981; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SQPsX2DHIButJsUpdhJcEvvwgfeMxBo/GFMw0XDovgw=; b=JCyamgJi+HcLSmxYsr36Yn/+dxqNGiqfqfeutX+jykH1NlnATve35UaDfE6vQ3BN+K ZJ2UhlA8TmgKqOMF7Ke55dBnOMZC3yDs8WU5BNgHg6FMDXTjOP+cHN1TvNjz9GoavKR1 Nc2t0R8wrPLqkJvTOAXOOMsbgVW2c5JLnPKbZnB8Z3NrznNtWKlbodkaKxhjDrnDQHLs wblzbTCGXXjzLFQWe+dkGxfgK8hjMsFzbxgFwNrMCYg558A+TaTnXoRpKx5a70liPU01 k3YsAsFKSqu2t7LVzcnvjqcMezJ7Lot9fPmn6e4QdkRx7LUJ9qLFGArplEYzMv5xgiKi +fbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736821181; x=1737425981; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SQPsX2DHIButJsUpdhJcEvvwgfeMxBo/GFMw0XDovgw=; b=fXRjyJ7m8LVk5F7ePwhvE3Jwe55lEBtfxS6AQ7e0cSY7tHaP+ATOKK1l5lFQMC7vIv LMzxotmgm7b5mYlCL4Kp88s+COsYwkhHavXL66rN+FjpyXnD4NMrZ4SK0uUdUsBRQzcP RTW8GMLmwr+CYATtcEDYe2Fy/Ch8lf7cIHD9DNkflO/qU+p6B+he+ytD6PpUSnYUWz/T U6aYvwJcCqcMU6OMiL33LLoXvrXNlzti7k7/w5yrm03Ti6y28zX72MV3K2lREM9buQMZ 3tbKCphKlFhigGvAFI3tZ+yGMdtOWbCTJAih7dirxdaK4wk76V2PTUNvRbVxo+G7x/wP UQTA== X-Forwarded-Encrypted: i=1; AJvYcCVhs3s2wG0lo/n9bzA2PlaaeR4x8uFLx1w3vMImxEdfo8+R2qfcwEViGjuHpNnJ6iTkToHZGSVC7w==@kvack.org X-Gm-Message-State: AOJu0Yx173gbFNsos48aEUi6tECIOZTv12dkkk8HZD8Syog7epNiSKKV YzR4XViM9EIGOQ/sKJ22tqWxVdPSmMQKLv5h73Nq0HKo/5/9Kkj1 X-Gm-Gg: ASbGncsUYPfYu8nIwFlL+aKdwQNdIINDcvCL8LoslJ8N1sxKhGtcw9QWVAoL7HQanpr uwA0+Dg/pHF81WudfsnMiDA8A4i1i0ZmcZX0CTQta1eARxxrGt2cIxn6VjPo7MZoj3wNMH4PqWS IRaXZliis0EQVXyfF9xDh5Av6WM64G4wdduaRmMaCnKSfftswib31DkhF+UdwKbQybJDdMmHarf rHmZF5/DquE9O+gaWyKQLg5JkvCSKm85sGACGqM5ppgCNXml8Hzr3G7ai14dW4h4ZcZCVX/dpBt YbRb7IZQ X-Google-Smtp-Source: AGHT+IH4C8oQxYrxEc0hMPntr3TFj0/HQ3tRiTFBDtPEqYrgkCC5XatENgLtg7g5dRzWvlp6uSkr4w== X-Received: by 2002:a17:903:1245:b0:216:48f4:4f20 with SMTP id d9443c01a7336-21a83f56f58mr324043955ad.16.1736821180973; Mon, 13 Jan 2025 18:19:40 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:4043]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-21a9f10e011sm59713495ad.3.2025.01.13.18.19.38 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 13 Jan 2025 18:19:40 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v4 2/6] mm, bpf: Introduce free_pages_nolock() Date: Mon, 13 Jan 2025 18:19:18 -0800 Message-Id: <20250114021922.92609-3-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250114021922.92609-1-alexei.starovoitov@gmail.com> References: <20250114021922.92609-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 44B5CC0012 X-Stat-Signature: ocbpik8tz4fyf81yjsa3o861prss4j7y X-HE-Tag: 1736821182-334403 X-HE-Meta: U2FsdGVkX18uFAoSWaZCesVqc6mnIpH2UWhmCIYmPZcMmmIfZyQy4SoOAV6JcTB7cm59zQpCZ63XmsRwDpzqYksWl0X2GE6oNruPXjhj2KNA62i3mnI1ZwS2koHf6TxxPxrA5uOrzMnRZ4ymZCGtm2qyWTIN6xf7+/Ujf+ZmQt0ic/4PsdXvx16iaAwiP71UM76Lbaug5MeqpZo9eft0wRfOK5SgFKpebBezfyDiQeJOi8/y4xc8W062KOnlm0Ff9T43L4zWhEmqHA0+KYiv6JNwwwvpZaPzMx3o/L4ReOTqhqqA1La6wDse4hEls0SAWx/l1pMVs6SA7bkt3cos697FnPGRrmN6Sqc1TR35GFe0vXUs6nzzUhbETZxzu2wr/bcJOwl160jNtEmYOwMaHsbuFHXb2e1gQILap+OK48G+1zeBksQ3j5I6VM19MfbbdtnlWjxnaUr6IaYtzJ2a8PXFBOuYEeGQPVTUvF10XnEb28A3KswPrG/G9pyDDGl+GIeVUhlx1dUGnsZb/yA2Cth+2GbhSd0GtS5KITMuPcf2NIfM5lNTY4cT2DvgjtVZ+NTaHB+NnAKHy/GmjMM9Qj9G6KesSIf7MLogN1dCtNGJYPyDfxsY6FhGNddIHC0fKq2uBtWyfKGv5NcLNi33rearFYijCk9EsZN6jXIuADPKubzE6trIcL+CSFjMmNpDiKXiSZJiz/SMYFbjp+JWCRBxeMT9XEeP4kiZaRUwkRIaLpxj/kVoiaSqHrAbqkuvHaoGtkONXZl7fvK5rWYSB5fkBtfSb+Oyo1B+tsnAum/2ELlwoYwSnBT7G6H0wMhe+CjhiTIIGvhCidc0vEdliR1GK6HnEz/aNyye9rONcvZ0dbgWRIpLLFw2rFDlBJPQaiAsi3mF1MrqYbZLwe5xZUmjhDCOq9BHZ+5sPqINrn+aU5QjhBxic9/fNX2eKkxJv9i4RnRgDYM8TATR446 C3tHXucc P/FRxuH4tOEPbaPSKnEL71MCp/tRW/Ltj5A0PoHDDyiQI1IVthyVIQDAsR8w0ZqXNhKWgJ2tUEvpYRsQ8QwNGBphIu6hhS1FajnumkBOkc78AVyZbf+Qu03ffJV/BkfOzqO7LRKKcCR1+0tOhWJlGCM3gqvfuRkrvc2kbtmamf2pMWAKWrZemRJEzCequmAjGykmPzW5kr+VTNyt8CWnxMCnqK4ckwaI/YGPuBJfV0s40ihXB/isgwyJIaczl9tcQV4nfQzWyCCY0kJDCkBCBRpE57ChaRbu0uLIrYLb7Xpv0QlGYqYv+O+dQIJhHi+VBJMr3XomcvzHHFR4pQ2a+YP2Hd3TYpNChYch/57VB+mWP42rIZLS4q6xciINzlVS/ZxeDrY7HJpIVvLZ4xR2j6vzsQDpuIT+JfaP7Y5WajwEie9Z0oSUT60gJ2vM7k+VLRpdWHii5pdJ5UVaefdvtP/QI5lPmsSWAi42zlSvgGtb6qpc+J2DcEeBsBw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Introduce free_pages_nolock() that can free pages without taking locks. It relies on trylock and can be called from any context. Since spin_trylock() cannot be used in RT from hard IRQ or NMI it uses lockless link list to stash the pages which will be freed by subsequent free_pages() from good context. Do not use llist unconditionally. BPF maps continuously allocate/free, so we cannot unconditionally delay the freeing to llist. When the memory becomes free make it available to the kernel and BPF users right away if possible, and fallback to llist as the last resort. Signed-off-by: Alexei Starovoitov --- include/linux/gfp.h | 1 + include/linux/mm_types.h | 4 ++ include/linux/mmzone.h | 3 ++ mm/page_alloc.c | 79 ++++++++++++++++++++++++++++++++++++---- 4 files changed, 79 insertions(+), 8 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index b41bb6e01781..6eba2d80feb8 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -391,6 +391,7 @@ __meminit void *alloc_pages_exact_nid_noprof(int nid, size_t size, gfp_t gfp_mas __get_free_pages((gfp_mask) | GFP_DMA, (order)) extern void __free_pages(struct page *page, unsigned int order); +extern void free_pages_nolock(struct page *page, unsigned int order); extern void free_pages(unsigned long addr, unsigned int order); #define __free_page(page) __free_pages((page), 0) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 7361a8f3ab68..52547b3e5fd8 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -99,6 +99,10 @@ struct page { /* Or, free page */ struct list_head buddy_list; struct list_head pcp_list; + struct { + struct llist_node pcp_llist; + unsigned int order; + }; }; /* See page-flags.h for PAGE_MAPPING_FLAGS */ struct address_space *mapping; diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index b36124145a16..1a854e0a9e3b 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -953,6 +953,9 @@ struct zone { /* Primarily protects free_area */ spinlock_t lock; + /* Pages to be freed when next trylock succeeds */ + struct llist_head trylock_free_pages; + /* Write-intensive fields used by compaction and vmstats. */ CACHELINE_PADDING(_pad2_); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 0f4be88ff131..f967725898be 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -88,6 +88,9 @@ typedef int __bitwise fpi_t; */ #define FPI_TO_TAIL ((__force fpi_t)BIT(1)) +/* Free the page without taking locks. Rely on trylock only. */ +#define FPI_TRYLOCK ((__force fpi_t)BIT(2)) + /* prevent >1 _updater_ of zone percpu pageset ->high and ->batch fields */ static DEFINE_MUTEX(pcp_batch_high_lock); #define MIN_PERCPU_PAGELIST_HIGH_FRACTION (8) @@ -1247,13 +1250,44 @@ static void split_large_buddy(struct zone *zone, struct page *page, } } +static void add_page_to_zone_llist(struct zone *zone, struct page *page, + unsigned int order) +{ + /* Remember the order */ + page->order = order; + /* Add the page to the free list */ + llist_add(&page->pcp_llist, &zone->trylock_free_pages); +} + static void free_one_page(struct zone *zone, struct page *page, unsigned long pfn, unsigned int order, fpi_t fpi_flags) { + struct llist_head *llhead; unsigned long flags; - spin_lock_irqsave(&zone->lock, flags); + if (!spin_trylock_irqsave(&zone->lock, flags)) { + if (unlikely(fpi_flags & FPI_TRYLOCK)) { + add_page_to_zone_llist(zone, page, order); + return; + } + spin_lock_irqsave(&zone->lock, flags); + } + + /* The lock succeeded. Process deferred pages. */ + llhead = &zone->trylock_free_pages; + if (unlikely(!llist_empty(llhead) && !(fpi_flags & FPI_TRYLOCK))) { + struct llist_node *llnode; + struct page *p, *tmp; + + llnode = llist_del_all(llhead); + llist_for_each_entry_safe(p, tmp, llnode, pcp_llist) { + unsigned int p_order = p->order; + + split_large_buddy(zone, p, page_to_pfn(p), p_order, fpi_flags); + __count_vm_events(PGFREE, 1 << p_order); + } + } split_large_buddy(zone, page, pfn, order, fpi_flags); spin_unlock_irqrestore(&zone->lock, flags); @@ -2596,7 +2630,7 @@ static int nr_pcp_high(struct per_cpu_pages *pcp, struct zone *zone, static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, struct page *page, int migratetype, - unsigned int order) + unsigned int order, fpi_t fpi_flags) { int high, batch; int pindex; @@ -2631,6 +2665,14 @@ static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, } if (pcp->free_count < (batch << CONFIG_PCP_BATCH_SCALE_MAX)) pcp->free_count += (1 << order); + + if (unlikely(fpi_flags & FPI_TRYLOCK)) { + /* + * Do not attempt to take a zone lock. Let pcp->count get + * over high mark temporarily. + */ + return; + } high = nr_pcp_high(pcp, zone, batch, free_high); if (pcp->count >= high) { free_pcppages_bulk(zone, nr_pcp_free(pcp, batch, high, free_high), @@ -2645,7 +2687,8 @@ static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, /* * Free a pcp page */ -void free_unref_page(struct page *page, unsigned int order) +static void __free_unref_page(struct page *page, unsigned int order, + fpi_t fpi_flags) { unsigned long __maybe_unused UP_flags; struct per_cpu_pages *pcp; @@ -2654,7 +2697,7 @@ void free_unref_page(struct page *page, unsigned int order) int migratetype; if (!pcp_allowed_order(order)) { - __free_pages_ok(page, order, FPI_NONE); + __free_pages_ok(page, order, fpi_flags); return; } @@ -2671,24 +2714,33 @@ void free_unref_page(struct page *page, unsigned int order) migratetype = get_pfnblock_migratetype(page, pfn); if (unlikely(migratetype >= MIGRATE_PCPTYPES)) { if (unlikely(is_migrate_isolate(migratetype))) { - free_one_page(page_zone(page), page, pfn, order, FPI_NONE); + free_one_page(page_zone(page), page, pfn, order, fpi_flags); return; } migratetype = MIGRATE_MOVABLE; } zone = page_zone(page); + if (IS_ENABLED(CONFIG_PREEMPT_RT) && (in_nmi() || in_hardirq())) { + add_page_to_zone_llist(zone, page, order); + return; + } pcp_trylock_prepare(UP_flags); pcp = pcp_spin_trylock(zone->per_cpu_pageset); if (pcp) { - free_unref_page_commit(zone, pcp, page, migratetype, order); + free_unref_page_commit(zone, pcp, page, migratetype, order, fpi_flags); pcp_spin_unlock(pcp); } else { - free_one_page(zone, page, pfn, order, FPI_NONE); + free_one_page(zone, page, pfn, order, fpi_flags); } pcp_trylock_finish(UP_flags); } +void free_unref_page(struct page *page, unsigned int order) +{ + __free_unref_page(page, order, FPI_NONE); +} + /* * Free a batch of folios */ @@ -2777,7 +2829,7 @@ void free_unref_folios(struct folio_batch *folios) trace_mm_page_free_batched(&folio->page); free_unref_page_commit(zone, pcp, &folio->page, migratetype, - order); + order, FPI_NONE); } if (pcp) { @@ -4853,6 +4905,17 @@ void __free_pages(struct page *page, unsigned int order) } EXPORT_SYMBOL(__free_pages); +/* + * Can be called while holding raw_spin_lock or from IRQ and NMI, + * but only for pages that came from try_alloc_pages(): + * order <= 3, !folio, etc + */ +void free_pages_nolock(struct page *page, unsigned int order) +{ + if (put_page_testzero(page)) + __free_unref_page(page, order, FPI_TRYLOCK); +} + void free_pages(unsigned long addr, unsigned int order) { if (addr != 0) { From patchwork Tue Jan 14 02:19:19 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13938321 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F180CC02183 for ; Tue, 14 Jan 2025 02:19:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 89106280004; Mon, 13 Jan 2025 21:19:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 83E5C280002; Mon, 13 Jan 2025 21:19:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6B87E280004; Mon, 13 Jan 2025 21:19:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 4D4F8280002 for ; Mon, 13 Jan 2025 21:19:49 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id C1A041603DB for ; Tue, 14 Jan 2025 02:19:48 +0000 (UTC) X-FDA: 83004451656.09.0B3C0CD Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) by imf06.hostedemail.com (Postfix) with ESMTP id CEC5D18000B for ; Tue, 14 Jan 2025 02:19:46 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=WzTK72mL; spf=pass (imf06.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.216.42 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736821186; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=w+Lbt2oOvlSTPMdIT0UeNEpvkZe5nRvEnawf5u/tH+w=; b=5LjbAMTqZBij5F2O3PvKZNagI3TFkWqCxkXD6iGITC3hCOjk/xMkKDTQ/fq6RhYGbo49ik IoQEE4YVfTZ5WQrm5b9WSTO7FbQC/1LvBxpWd0DfEl7Eco/aj8H/QyTmeLv1iA3nHAVB6t s3Pgj0T/K93lSJPHe4mMlyndEpG98eA= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=WzTK72mL; spf=pass (imf06.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.216.42 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736821186; a=rsa-sha256; cv=none; b=oUI8otA7T56hZjPOsR7rlG5L0+KTxpqEkdteKuXjJcOXMo8+3xPPn9hHTMsLI0csm8VON3 WshDGn7lpf1YJ28E2nGkR8lg+imUVATTxzSbBv9GfZFB5kVP+aayPS1Bx1MxPQsByh+N78 7/5x75XLAwQzkmkPcgtfyl/cW0Zb7lM= Received: by mail-pj1-f42.google.com with SMTP id 98e67ed59e1d1-2ef760a1001so8336587a91.0 for ; Mon, 13 Jan 2025 18:19:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1736821185; x=1737425985; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=w+Lbt2oOvlSTPMdIT0UeNEpvkZe5nRvEnawf5u/tH+w=; b=WzTK72mL3sk8DvDUa8caSFyIYmkqSBRyqY7WOdI/i4/+Q5ImzUiy64mdJTQ1fE16tu yeqkeLkFnCGgaznfIAX70O7NdQqhS2SEbrVW2UVbdNFwCWLKDR6ax3TNoQBXHs+WvvSS Pth8tE2q3Wl+bqrG5HHzs6aMnLCZGPoht7K7uBk7OvBzO2OvAEfIMHBHJQTOGk/FuR0d uT4ex9jfAdmH7Rd32vkGUZ6oHEc3u9H+M1NSn3eoNvxYyCIpo0L5HtGreplimHgtOb8W m2hAC5oMn7tT32KQmtzhNt7FUHVt1WNtVGRp1IEDqIihTuFw0YsERZrPnh6T9BEb0fK4 fV1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736821185; x=1737425985; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=w+Lbt2oOvlSTPMdIT0UeNEpvkZe5nRvEnawf5u/tH+w=; b=WJupTQtWDGu5y8CCwduPv4V3k4KKGhE101ePH/oNsCPhxDKXX8SXQYKhZUVtMMBuZ7 M66kWq4gckAaJj98yASzJZXLRI846w5/Fntd2oVnNzttiMzsKMUXwB/CeslpW2AgpswO C54H92F1qSZRyKrdhtkQvrsaBWGecsJNvQsuOoF7elQc/6XUeXUs6ZjF95eBZQqg5H+5 de/6t59eyj2BkgBUZHwFplJuWehYjEXBJgiI5nHFxrHRUCNO1ASviqUBI4PpAvaCIHR6 AoXn5ZMdjm0OttG5KZ3ksvJjmUFh2c11DCi+jE3GZTV6NYn/EgalXtX1MgZsnQGNG7ks Pnsw== X-Forwarded-Encrypted: i=1; AJvYcCUoHBkvwrIDslplDNmTTmz7HqSV7VkfeJntCvoUq3AhNAXr5BEukNDY89L5xceuZ1W0uIbn9YvzMA==@kvack.org X-Gm-Message-State: AOJu0YywYCZDO3C9OC2hJxkzuI7Mr79bO+kVfq9l/c8627npHanvkDC3 BLd/yquDa9x7NcniNqPSKZ2p4u8nL2HeP13fl8Lm6SsiKqWu8Q+5 X-Gm-Gg: ASbGncviJ11vm96eJyVYf5Cf3oVQLej7OhX2Q+ErIiExJSUz7RriZVbIBhKcMIFsj4c Esjw+X+hg8j9lY/XAiuapUFDTD2GRr56Yc21gTE+pZFR5fnkgCFziVNCHGnK8WtGAResKp9a7ai bI8QGwgVIDIXmrt/Egv/cwjPi5nmNk/pdnEEBeqAYhCvqf5UBPglpSwaut0NOeDlluLFb6D8J7n BRScIN6uG68Wp864ooiKDBLPo8pt8VCnxxNZGy6GLO2Fi/N4u4auNBPpZTnzoLzpivEg7gAr5bR 3A2Ue4nV X-Google-Smtp-Source: AGHT+IEFmgpkoNWWnjwaaKDdY0kFEuoA/VYtTI9WUz5vYKzivTLXj4vvKroyeHsRxZ0DCUaCQZ66Tg== X-Received: by 2002:a17:90b:51cb:b0:2ee:863e:9fff with SMTP id 98e67ed59e1d1-2f548f2a4afmr34600926a91.10.1736821185517; Mon, 13 Jan 2025 18:19:45 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:4043]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-a317a07cccbsm7420136a12.11.2025.01.13.18.19.43 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 13 Jan 2025 18:19:45 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v4 3/6] locking/local_lock: Introduce local_trylock_irqsave() Date: Mon, 13 Jan 2025 18:19:19 -0800 Message-Id: <20250114021922.92609-4-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250114021922.92609-1-alexei.starovoitov@gmail.com> References: <20250114021922.92609-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: CEC5D18000B X-Rspamd-Server: rspam12 X-Stat-Signature: 3s6ufr8xfn1y66hoa9ftp1upg6bj3jt8 X-Rspam-User: X-HE-Tag: 1736821186-812587 X-HE-Meta: U2FsdGVkX1/BXfm/HjlKbTZ3hDTDw/gtUC2VqJpiYDB3rJDqVD5F9RO+ekIvernalC+DBlQ2rSsKREe3g3Sje+juZx6C9vFmzjeeROH8ODWk3kd7AVUS9ksVGmnjMsnxf1x6Fbsri8CouVQdHadxmic6juUqmdzYNNB/onUko46d6J/l7dFYEL0KuCMTxH1094kJlWC/g8ec33Skj3tj2ZCL8mivqy9KZHsE5/p83iQgb1BqtUWqaxh+iZlSRgeAm/9+VtHLQPhVzhMLnONC8heicxVEJYzQf42Mw9YtUBP/5wzAn3PAxSc01MyqDocEYjmqwOrLecbQY5E9ZVS4Q/lxtrzOgjigDIdZeB98TJ61yj+D8MiHSRVsjqhLop2Uc7dGtuUy2qG/dG8NwTzJba10LvygWCAtGNpzWQJerXHQzJRLJ4ml7lyBcUk2k4wi2eHGXlbIPzjf045eCsmHSaLLEbe3N0IStzbIQUOQWElhIswNa2DYd5oQb+DvbrLwc/lL7lP81Tm1mbMoytlXc2hMFUhQtxTzIabOtaRBE8QZ/xJOpb/7tC/kdK5Y+Z4QN0o0fyd4sTuqpA5RF2y5ZhiAIS4v5hmbkgUOPnTrrCjdDloObES/g7qA41vPxd04CzpMg5lg7DJxh72m1uh60sbameJKMp/wHH+5zQXWTTM/rdIa+P2aSsNF9kex3+DPoLFWX04P55Rohd0Cy1vh+mOyob+ZCeYsFLRPZaQqBCLYpNZypUNd1gWv/NbfPf92IDCaDZw4X6QpQ76WFpnQvxEEUySIbyviWqINMpqVVimAPRQHqmEvYzuVVwB+Vf/ZAn0Z9WeomRoPF2jHH+L1bOcpE8XF2paPc9HLA78KcPc/i6Hn1qg2sSdf4DHRUjQhBQyyBC9ZLePb6Rs12iTI0GZl1vJpFRLlXdgG//YsQhLTRKVhmPDInlPiNg/23A1gvoyE7RT61g2BLbJxssT HnAuRPQd 3UO8Rhp46KAlTRdR1QVyRScwAF7643rAm0gBxCtFX7yJnAB7ayy7EVsHrW3w2oVnb6SuVh/7UscuUyy+mN6EIHcdCCLqXXLzLCueHxVQEYL9ZIQP+gqay/upb4ahhnLL0vz8F7Jy6F8L3qGu6PKiIuQfWYNrvfTlpfJyQpshzBkwAobKni4fPjZF9zNFKc+3oxAN3Gy9ImsmcFQswvNk1fauevZ48PHZjitM9zkESJaT02hOngkxkdrdsUVtOPSCdvQgNH/h71z3KO9BgWYgKS7IWiMqdUA8f12+bNvSlRkHl7WWLh5NdNT7GeVI/u8S6cG9cX1UZOt8XsbL8EBfFMyZxsnKkRBsnSpAVps6Y8mUUhf59+lRIGBjuE7j6fDibZg8uU9rTzdGjY+qbqL30vqqzBQuCXWImssBTBb3bJG/Lyv0dkxs+FDhuXdeMRZVNwmSr57OrSktCkWgiELAD6XxGG6nHYxu21zM6riqFLIY/om+JxkwmYo56lQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Similar to local_lock_irqsave() introduce local_trylock_irqsave(). This is inspired by 'struct local_tryirq_lock' in: https://lore.kernel.org/all/20241112-slub-percpu-caches-v1-5-ddc0bdc27e05@suse.cz/ Use spin_trylock in PREEMPT_RT when not in hard IRQ and not in NMI and fail instantly otherwise, since spin_trylock is not safe from IRQ due to PI issues. In !PREEMPT_RT use simple active flag to prevent IRQs or NMIs reentering locked region. Note there is no need to use local_inc for active flag. If IRQ handler grabs the same local_lock after READ_ONCE(lock->active) already completed it has to unlock it before returning. Similar with NMI handler. So there is a strict nesting of scopes. It's a per cpu lock. Multiple cpus do not access it in parallel. Signed-off-by: Alexei Starovoitov --- include/linux/local_lock.h | 9 ++++ include/linux/local_lock_internal.h | 76 ++++++++++++++++++++++++++--- 2 files changed, 78 insertions(+), 7 deletions(-) diff --git a/include/linux/local_lock.h b/include/linux/local_lock.h index 091dc0b6bdfb..84ee560c4f51 100644 --- a/include/linux/local_lock.h +++ b/include/linux/local_lock.h @@ -30,6 +30,15 @@ #define local_lock_irqsave(lock, flags) \ __local_lock_irqsave(lock, flags) +/** + * local_trylock_irqsave - Try to acquire a per CPU local lock, save and disable + * interrupts. Always fails in RT when in_hardirq or NMI. + * @lock: The lock variable + * @flags: Storage for interrupt flags + */ +#define local_trylock_irqsave(lock, flags) \ + __local_trylock_irqsave(lock, flags) + /** * local_unlock - Release a per CPU local lock * @lock: The lock variable diff --git a/include/linux/local_lock_internal.h b/include/linux/local_lock_internal.h index 8dd71fbbb6d2..93672127c73d 100644 --- a/include/linux/local_lock_internal.h +++ b/include/linux/local_lock_internal.h @@ -9,6 +9,7 @@ #ifndef CONFIG_PREEMPT_RT typedef struct { + int active; #ifdef CONFIG_DEBUG_LOCK_ALLOC struct lockdep_map dep_map; struct task_struct *owner; @@ -22,7 +23,7 @@ typedef struct { .wait_type_inner = LD_WAIT_CONFIG, \ .lock_type = LD_LOCK_PERCPU, \ }, \ - .owner = NULL, + .owner = NULL, .active = 0 static inline void local_lock_acquire(local_lock_t *l) { @@ -31,6 +32,13 @@ static inline void local_lock_acquire(local_lock_t *l) l->owner = current; } +static inline void local_trylock_acquire(local_lock_t *l) +{ + lock_map_acquire_try(&l->dep_map); + DEBUG_LOCKS_WARN_ON(l->owner); + l->owner = current; +} + static inline void local_lock_release(local_lock_t *l) { DEBUG_LOCKS_WARN_ON(l->owner != current); @@ -45,6 +53,7 @@ static inline void local_lock_debug_init(local_lock_t *l) #else /* CONFIG_DEBUG_LOCK_ALLOC */ # define LOCAL_LOCK_DEBUG_INIT(lockname) static inline void local_lock_acquire(local_lock_t *l) { } +static inline void local_trylock_acquire(local_lock_t *l) { } static inline void local_lock_release(local_lock_t *l) { } static inline void local_lock_debug_init(local_lock_t *l) { } #endif /* !CONFIG_DEBUG_LOCK_ALLOC */ @@ -60,6 +69,7 @@ do { \ 0, LD_WAIT_CONFIG, LD_WAIT_INV, \ LD_LOCK_PERCPU); \ local_lock_debug_init(lock); \ + (lock)->active = 0; \ } while (0) #define __spinlock_nested_bh_init(lock) \ @@ -75,37 +85,73 @@ do { \ #define __local_lock(lock) \ do { \ + local_lock_t *l; \ preempt_disable(); \ - local_lock_acquire(this_cpu_ptr(lock)); \ + l = this_cpu_ptr(lock); \ + lockdep_assert(l->active == 0); \ + WRITE_ONCE(l->active, 1); \ + local_lock_acquire(l); \ } while (0) #define __local_lock_irq(lock) \ do { \ + local_lock_t *l; \ local_irq_disable(); \ - local_lock_acquire(this_cpu_ptr(lock)); \ + l = this_cpu_ptr(lock); \ + lockdep_assert(l->active == 0); \ + WRITE_ONCE(l->active, 1); \ + local_lock_acquire(l); \ } while (0) #define __local_lock_irqsave(lock, flags) \ do { \ + local_lock_t *l; \ local_irq_save(flags); \ - local_lock_acquire(this_cpu_ptr(lock)); \ + l = this_cpu_ptr(lock); \ + lockdep_assert(l->active == 0); \ + WRITE_ONCE(l->active, 1); \ + local_lock_acquire(l); \ } while (0) +#define __local_trylock_irqsave(lock, flags) \ + ({ \ + local_lock_t *l; \ + local_irq_save(flags); \ + l = this_cpu_ptr(lock); \ + if (READ_ONCE(l->active) == 1) { \ + local_irq_restore(flags); \ + l = NULL; \ + } else { \ + WRITE_ONCE(l->active, 1); \ + local_trylock_acquire(l); \ + } \ + !!l; \ + }) + #define __local_unlock(lock) \ do { \ - local_lock_release(this_cpu_ptr(lock)); \ + local_lock_t *l = this_cpu_ptr(lock); \ + lockdep_assert(l->active == 1); \ + WRITE_ONCE(l->active, 0); \ + local_lock_release(l); \ preempt_enable(); \ } while (0) #define __local_unlock_irq(lock) \ do { \ - local_lock_release(this_cpu_ptr(lock)); \ + local_lock_t *l = this_cpu_ptr(lock); \ + lockdep_assert(l->active == 1); \ + WRITE_ONCE(l->active, 0); \ + local_lock_release(l); \ local_irq_enable(); \ } while (0) #define __local_unlock_irqrestore(lock, flags) \ do { \ - local_lock_release(this_cpu_ptr(lock)); \ + local_lock_t *l = this_cpu_ptr(lock); \ + lockdep_assert(l->active == 1); \ + WRITE_ONCE(l->active, 0); \ + local_lock_release(l); \ local_irq_restore(flags); \ } while (0) @@ -148,6 +194,22 @@ typedef spinlock_t local_lock_t; __local_lock(lock); \ } while (0) +#define __local_trylock_irqsave(lock, flags) \ + ({ \ + __label__ out; \ + int ret = 0; \ + typecheck(unsigned long, flags); \ + flags = 0; \ + if (in_nmi() || in_hardirq()) \ + goto out; \ + migrate_disable(); \ + ret = spin_trylock(this_cpu_ptr((lock))); \ + if (!ret) \ + migrate_enable(); \ + out: \ + ret; \ + }) + #define __local_unlock(__lock) \ do { \ spin_unlock(this_cpu_ptr((__lock))); \ From patchwork Tue Jan 14 02:19:20 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13938322 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8FA74C02183 for ; Tue, 14 Jan 2025 02:19:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2C277280005; Mon, 13 Jan 2025 21:19:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 27292280002; Mon, 13 Jan 2025 21:19:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 13C46280005; Mon, 13 Jan 2025 21:19:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id EB0CF280002 for ; Mon, 13 Jan 2025 21:19:53 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 72E28AE531 for ; Tue, 14 Jan 2025 02:19:53 +0000 (UTC) X-FDA: 83004451866.14.580CFCF Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf29.hostedemail.com (Postfix) with ESMTP id 8FAD5120008 for ; Tue, 14 Jan 2025 02:19:51 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=JfeZkClC; spf=pass (imf29.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736821191; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rVaRyHY3N181Ob6HytxKBrw588chz5TJlaChctBoK1M=; b=YGT3e6nWAHM+fzat7HMan8Gjw23+wDddV+TzLDPAVXb+dVorY6Ev3bo7nMiC8UC5Z6VzEo EOeEJzT1dYrRYcgks8y3JuBqDVi1iZW+atzXyU+cIzZ+trckXY1FPyTOMlAd4nKmyOEInN PQpyu/3eWQE1D49an0HDuKgpXzaYX/c= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=JfeZkClC; spf=pass (imf29.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736821191; a=rsa-sha256; cv=none; b=Rk4Kj4S+DbHAZrKLA1QjcijpFLv55Tc4B6gdvNpwSyKG+d9yZGfbwyWa8W90ELQynRG1zG xhidIla9rzqA0BcWVi1ai/Vv0LPm8YMUU/I8CEzlmGLXO2oc/MoYPidYG5ZUv/36UD6L23 6GN50/Attsls6UTO7qXJrSsr/aoqzjg= Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-21634338cfdso73813725ad.2 for ; Mon, 13 Jan 2025 18:19:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1736821190; x=1737425990; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rVaRyHY3N181Ob6HytxKBrw588chz5TJlaChctBoK1M=; b=JfeZkClCCnG5e8XPez4mMPhtFaLi+3YfEGeInD5RRgmPf8OTM+hJpq9UDvmSZDH+TR AM7WcXNHgcnGXNfwYpm4tNiQlpERaQ1qyEMcl3PttWXvQHrnPXAX5OpL1FcnLMBxhynv 0R6E8zY67lVImlHQQwzGmpbPsFL1ZnvTKbsBHPFDQUIZLNF4T33vqbdHPm2GyuUtHDB1 knyrWCvL9txwX+fiZ5d7epFqpqdA8/yPMTFd8yQPIiSrJvY02CCxpx14VCpqiyZpnZWe uzIxSBQanISWETl4tD7eVMmw3Brfh74fyUzaeQJsY9G7+b3oy6TPFIRkg/uSt8GdwFBe nCgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736821190; x=1737425990; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rVaRyHY3N181Ob6HytxKBrw588chz5TJlaChctBoK1M=; b=KeYV2QXnq+8rQly1JD1jYddOR+HE8PhLZHcaH99vkGw7wGqpxryp7uiO7bfTqCLOtg /7XRkvqjAH1OGRqJ7YH7oOZ24nLNBycHrFED1A1pBztKVj3CqGpST5WgVdasRPYhugE9 jt8cX3QdfysDczd09SJvV1HwsbBBEs0bXqz3OIhV+bcPx8X/QaYVP4voR8JwFeIGzmvZ O52KiQ+FLLdFcrru8XdaxwuJ/oPoFv3y/uEqqpm/FhPIzJJmAJXuk80Is3nPxfIKpAZN eahnGzMslKlgeq+UHRlcVmqnnIx0n5vOZiX+DBBiNr0GdKTKJGoym+SRN4r1uAj49RP1 ckaQ== X-Forwarded-Encrypted: i=1; AJvYcCX6667JLm4ZEDYN4bKJ6wSYOeBk5dQjLLGD0SH2/uULPo9K7tOr+OnqDJTBT7nnldoVeBEsy3IeBA==@kvack.org X-Gm-Message-State: AOJu0Yxz+g1SMl7EflzyicP0VS6CWdDNY7EZsqiMAnXYP10PGDxkxts8 ujDWsPkqyAacM9QD01RMNIEk2HEKYJnLJ+rWXFK56itPhTa0d0oD X-Gm-Gg: ASbGnctkNMluHR4yf+fbUcsQVLo3oQ9+bdG9WeC12Hc2rTtoba/R3wQh+qd5kvFX1y4 0hMTzbeq9t6CMK8CLNswgEPfg2lWcnPaJ9IgLel1vjT7VK/+08GnnJRd7DRFzsLlYk+CVTKsFaS ybBjzyp6CzPUEYyVYJ1Qu03t/7Q51vOmuAI7V0kFPoOQFsAkmz/X9wHeHa/Zo8pqkzhpvF964rh fNDfN2fUT6HhjFKjsaun28W8ClxHsJ0/ArPYa5pbs2WfGXeF+/GMyYxLkRLUELY5Ik+TBN3CqUd zinI07g8 X-Google-Smtp-Source: AGHT+IFgd3F1vhcoVYBusGT4YkcS3cyvI9lPQCSRNl7H1SCdyLOMHx5+LSBkcObgthXTbJSLr0qoDQ== X-Received: by 2002:a05:6a00:3cc2:b0:726:f7c9:7b36 with SMTP id d2e1a72fcca58-72d21f325b8mr38139980b3a.8.1736821190203; Mon, 13 Jan 2025 18:19:50 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:4043]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72d406a4dfesm6523242b3a.156.2025.01.13.18.19.48 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 13 Jan 2025 18:19:49 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v4 4/6] memcg: Use trylock to access memcg stock_lock. Date: Mon, 13 Jan 2025 18:19:20 -0800 Message-Id: <20250114021922.92609-5-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250114021922.92609-1-alexei.starovoitov@gmail.com> References: <20250114021922.92609-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 8FAD5120008 X-Stat-Signature: b79tiwnugtckozrjayyfm95r853odte3 X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1736821191-478634 X-HE-Meta: U2FsdGVkX19buNe9rsi8hgy657FRAYbcr+TWkT3tCq8OxUMIDKo3zVTH+bGczFTfXLpNLm3sWzoPcAjCWMqRPg37DTUrMjP5v6DwpNbEOJlvFBjJhV06vAP1NkcQir/glJSSG6KZbTDqluf06nq3E9fVpbKdSr27iz5SqgzFTN7YAknFvI9cWKLVeYN+cAiqs/JzBtNVaQhDuYLXKwRAyiRKNkr1dv8E5huxQt+VVlE4F6qz6YCdV1hqrr4qlPJawovzG3zl8RcNPkMHp2yODvHayBlJ7MNaZmYvZTrKLvWz4EduQ+3nb9GU/d5ow68i+JQySBjFJR7yOyg4qmKEQHmV4lH3B2/A5TBlTLOrFAWmNTVPtDYjx5Mtl7LB4HrlBlPdu3NT8l89N88J7OgLYauCp72ioEKXJdlL8XLG/aUTG5SjnchBpY+Sk3qspS52ApDuJKwiQ3K/PE5SWIUtynJkLncrKNspkQfZCMhW8byRHM9FjEBBnCLShrc8I+INFjToxVZt3OffiZ12Uw+C52CDXsyYW4e2V31/XIfcGcgd0dMuy54ODr18QHKp1dOysYqTZsCu3I3MAMv79n3UFAxKs4pOJBQGROKEwY+PGd4Py5gWL7JluXN1fHpwVPlhU8WbTlgyHLe3zcqTc9gU8RYlQdH7yl7s3eatrgyb24W4ten5vEgULxWObagISo6ZHEKz/AgRmPLkPvq0akioEitt/BkoHK0y274F/JIrKiLljEkmmPzXiGc9/dNSg2gnNIG8oxTeTE1NHmi1XTOFpJq9e2WTqwxYomnDFnydPngHnrnkOnkNQPKN8A1tm3jnyt/qogde8Si0veOS/EJ30Qt3iLnepG99vmtFkzqzKsieyiDwaRBmwlSOI3VvhMGaoG8XEuJmn81DvNqc4Qa5C+aqww9pFHB5dNeDs70Y3K+s6NkMqFa7tUM4V6hivgJOsdTXPSPhsKwIiqDg1mj /zXB6rbE Ut7/Hj79I1I8NStf9hC96NIevklonFtkHyp44au/WcK1PDXfMfHfOjwKfIKO9NC9RmdEzBR8VL88CO9AWPPJKIwruy2TUtRlwQ7BpZEiqibrCpu3Vp+PamLL8yzsZHJ5JqvqSrJGyHkJgnmfzuUCvcfD3xVnLz6QAYiOHQo4yAlG7yTgqWBDEj8Tipzr0BLyGWsds5W+5IS2Y5y4+yy8VRZZ/cq272T+ejadkTU4EHfsMRT1zBsRCQdaU2BinKlc+03D0ia9GFza5R2HK2NGr8XgvOmkHpQHatPHTuJkCPpjzBK2hM13lxSkRSKcZjlUhYRKQ3Ea5NnWRm1vu6OAbaIlxjFK9nJN/dGZElBe9t25C3y8YdeYC5q+85sowKLPaaXQQqGtALfuJwNDVcKyNoKT79x0AmdYgdaE3+ApCY+v0OOsJ2sTQJE/c7m6BB+GqPa7RGXvWJLQySTsqbkExabrZlbpxZf791ywLhyiVcjugbBJFuCvikx0wwA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Teach memcg to operate under trylock conditions when spinning locks cannot be used. The end result is __memcg_kmem_charge_page() and __memcg_kmem_uncharge_page() are safe to use from any context in RT and !RT. In !RT the NMI handler may fail to trylock stock_lock. In RT hard IRQ and NMI handlers will not attempt to trylock. Signed-off-by: Alexei Starovoitov Acked-by: Michal Hocko --- mm/memcontrol.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 7b3503d12aaf..e4c7049465e0 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1756,7 +1756,8 @@ static bool obj_stock_flush_required(struct memcg_stock_pcp *stock, * * returns true if successful, false otherwise. */ -static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) +static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages, + gfp_t gfp_mask) { struct memcg_stock_pcp *stock; unsigned int stock_pages; @@ -1766,7 +1767,11 @@ static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) if (nr_pages > MEMCG_CHARGE_BATCH) return ret; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + if (!local_trylock_irqsave(&memcg_stock.stock_lock, flags)) { + if (!gfpflags_allow_spinning(gfp_mask)) + return ret; + local_lock_irqsave(&memcg_stock.stock_lock, flags); + } stock = this_cpu_ptr(&memcg_stock); stock_pages = READ_ONCE(stock->nr_pages); @@ -1851,7 +1856,14 @@ static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) { unsigned long flags; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + if (!local_trylock_irqsave(&memcg_stock.stock_lock, flags)) { + /* + * In case of unlikely failure to lock percpu stock_lock + * uncharge memcg directly. + */ + mem_cgroup_cancel_charge(memcg, nr_pages); + return; + } __refill_stock(memcg, nr_pages); local_unlock_irqrestore(&memcg_stock.stock_lock, flags); } @@ -2196,9 +2208,13 @@ int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, unsigned long pflags; retry: - if (consume_stock(memcg, nr_pages)) + if (consume_stock(memcg, nr_pages, gfp_mask)) return 0; + if (!gfpflags_allow_spinning(gfp_mask)) + /* Avoid the refill and flush of the older stock */ + batch = nr_pages; + if (!do_memsw_account() || page_counter_try_charge(&memcg->memsw, batch, &counter)) { if (page_counter_try_charge(&memcg->memory, batch, &counter)) From patchwork Tue Jan 14 02:19:21 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13938323 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14B96C02183 for ; Tue, 14 Jan 2025 02:19:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9C7F3280006; Mon, 13 Jan 2025 21:19:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 97751280002; Mon, 13 Jan 2025 21:19:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7F114280006; Mon, 13 Jan 2025 21:19:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 5F124280002 for ; Mon, 13 Jan 2025 21:19:58 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 1A3381A05B9 for ; Tue, 14 Jan 2025 02:19:58 +0000 (UTC) X-FDA: 83004452076.28.3504682 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf02.hostedemail.com (Postfix) with ESMTP id 416C980009 for ; Tue, 14 Jan 2025 02:19:56 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=jFzjlOza; spf=pass (imf02.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736821196; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ygT/gYIDC88MbRDeBH2BI6HaB56L95edS8LxJJawsPI=; b=kkjQMLhKjYUV5tFffm56q8P7COCUIhQMP7ToPpb0e1GJMg188P00FpOk96oFjsgWBSL4Qu MUoeoNXuVGPuY52rQMpScu0BKx4SIKg+eR7CrVh0Nzn2fTMsA2VuMsSNaXtwM1aqD+Zf+D +62/reijbzdpWloOcMdnhI3LQqzP+q0= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=jFzjlOza; spf=pass (imf02.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736821196; a=rsa-sha256; cv=none; b=sNDcl2M+DO2PzIY/8NVHPpUAk1BZws+w4Xr2ofmdMHgUrS3GEbtuCLlR6y0W2J8oXRAX13 Yq1xMJfy9s1Zsw/lPpvi+EOh8P1I8Z7ff9RZBNK6XpYTgTb97i/W/1uzFLouCQXbX0mf0f J5OYV/SUT/g+A8MqZj3VAvOjryRJcoo= Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-2166360285dso86649865ad.1 for ; Mon, 13 Jan 2025 18:19:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1736821195; x=1737425995; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ygT/gYIDC88MbRDeBH2BI6HaB56L95edS8LxJJawsPI=; b=jFzjlOzaU68H+quU0yRm+cx5zMN1F5871DoBbphu7t9JIecrzMlXCj/blI1Kulstug L3GngetX6ertKg7XnSbTotRITxtjMCMVv6gw6NYcBSQcM7gdwynUQEJajdSo6SnWxmiK BGW7xQauBOiGd/3Ab5uBVffYXDRb86boC4Oqm7PXYcbLLr0i7beE8Lh3WUgEXXMWBGIq VkA90vhlJTZu9Sp/XO1rFBbVxpITma065KGDNZ+syJzkYKZvtIUUGYfCvPZCpXNcYyyi Ia2PFIKvW/k/4FTsdneqvzD71ENnI9TGOtRn2TEBaqDRtKBAkJkxEgB6nB7LMAV5QL0w unqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736821195; x=1737425995; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ygT/gYIDC88MbRDeBH2BI6HaB56L95edS8LxJJawsPI=; b=fOiE//GKnXo6ziV04lZxj5TUR30tb1ys3gJoKfnqKw2hOtkoOOZrKAPfI/ZJl3Y5Aj lsMxwfqvujqUg0kxLFjxl0UfolKiAR5GtqArzabe+4ExlniDMC0y+5ewYOlsgMwcDwUN A7s9WJ9zjVCMDiqBtmK/WqWU4+iSJHC4MLVdWbJ76MQ7y6xx+Xcahl9Z+gsDu2Wwx5NZ kOEHwRFJKSiBGSzO99Hcs/zDjDzh3t0QxcrzwPcimAoPKsGy8r8q+V+SuVOIDZJFoodj mthN7PitPMh2eDkCPyFDBdGb2OnnyG5kB2no7i/nUYdvRDHrPn/WxzaARHooe6z3rOTg IJwg== X-Forwarded-Encrypted: i=1; AJvYcCVX/ql/RcCcbc70xiYJpm1+PHUwjrhHYEld4u7v2moEmY5em+GoqyK7S6NLsWtn6x8+gJspU90+xg==@kvack.org X-Gm-Message-State: AOJu0YyYYQV7UNeIb4NYsAvDmaXaZMWlq5EEPToOXI6gqRb7tHR8uE+N 5sCatMAGf1N485a9uSghcDBNyPzpvze5UHiARszILTKodkgPseFn X-Gm-Gg: ASbGncsc6mo7MZopvxP0BRJYihXCXL6I/xqqUb0/3UHlFMZWdw9yQtK/rH0tXIw1BsF wilGnHUISpFCGiYmKTeeO141pxnLsta7VDHbwpCI6iEPO2T/7+7EPKUksvIzo7DIdL5qc4F6bbL B2VLGPXlZ8yJY3bqUevFls8jLbsxj3clKoHbWIjVRXB2QgjuL48jbIMOX9Xsqu+5HuQeN98gX/L Yx1nmO+k5Jt7760pmp1DnsyAyD5+W2U9LfZDY4t4t9DZqpjvJyRwUkMRktLNBGvlxuPBPvBk3GE iqeKRiA9 X-Google-Smtp-Source: AGHT+IFgYqVYLd3bJ1uye+mrUb8xwZqUFfTGtcoiwxRHlc9JoDBAT6ZJfA68H0it17/fmR3KmGQU5g== X-Received: by 2002:a17:902:f54a:b0:216:2426:7668 with SMTP id d9443c01a7336-21a83f4bce1mr257464125ad.13.1736821195029; Mon, 13 Jan 2025 18:19:55 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:4043]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-21a9f22eb8esm59444185ad.166.2025.01.13.18.19.52 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 13 Jan 2025 18:19:54 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v4 5/6] mm, bpf: Use memcg in try_alloc_pages(). Date: Mon, 13 Jan 2025 18:19:21 -0800 Message-Id: <20250114021922.92609-6-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250114021922.92609-1-alexei.starovoitov@gmail.com> References: <20250114021922.92609-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 416C980009 X-Rspam-User: X-Stat-Signature: y7cbjmb6qa6af87moj1g8mxxs7z184xz X-HE-Tag: 1736821196-723965 X-HE-Meta: U2FsdGVkX18ScH30p4NQ+v6mhNUn9It2k/XF/haPf4xS7Q/GqVy5OQs98UcqgVCcGx2N3m1dfwFenjVzJHwj8/qlg2RnDcHAXRd/+EfQyyem83dl+hSruqX+aALT0rkp8XaP7CyAqhUAe2MLkVN2ugX0pa0xixrz/3Wb7nEJ+20+Lle8w9siEvF8NPsxd+bz7OjBtw8V5UrXirWGduVnQAmW/dMoYGC5cGlC6FQIoRDebLbM3V9vORl+OoGfi92e7nLrm5KLYHOKoXrJJt31zAQyo6CpZxN7Hs/galAG62kFyYmN2/4bIT74zNxP94j+hbUXhIXrXBw25SIFU9cJx/T731YJEUSVMUPXqdWpPn/u3AVUBVxrkNvOFfpHAzQ9lgDJv3gbz4kLNDMeBX1Au6111IvNOdSJZwlAdRAyYsCZ3+ne6/jZpW8YvQQkovreJZi36dEF2pCNfI98W3mimUtFoIwTqPUbWWxSDFPmC4c23pAZ/1DUiMuxZNYMe+ILcpICxL/tMja2Z3UG57x0Zm4ItEYfpY7i62uPT0iBT1jay+mX+sC4U80KR9R8z54zdfzJpEQHtwHwZgrNsElf5fkRM09Ey+heo0cl0ZkqkQU8a3RHKmVvGfISyFwTdWl5jx4MypIc4r7hNHdLgCJ8mZPkStXWuMLxh6VTR335yz3HRMwia0TWMKHZRIbrtcxXnhzD5lGA+M8d3n1lv1t29S1DqHf2C0b/P07pPrcVtzpL/c3LWEqvUH57wNUtoZQ5IezDcfAKiokMH+SKWdcNda9klSjQvmEeL7fLpxvZxvyC9r8brnidSacRqnCUoz0jcFDOYAo47Y1rrly9sPHsv5KVYMIAdqWwGTaheLPpGzxZHZgA6rgaXT/j5NZilVOa2M6fGq1Ojy6e7ezeWTJ9Ow4y9Ocr+6IuTW4/nclD+VlpnAybWvzedKH5BjLty+6vLQhb+Gu7ckKBT63J2rE PIaEJehr 1RxZS8139NRbO1YUfYDaoM53Rn54+TwLuZcHbARIk42yjGttVoxixHShRV1HNUyAVP0/kz5K1+/SdLSkb2xtjqSTXvL1skZ33ZBPlmH5gBmwBS/UcMA4ZwzZuyj24E/Pk+a9N8OROJ6r/K10rQ3bsCLrfNVlCo/l0HexuLW1kvz1jR1kadP3m7nwmjtH3R6ONxZ39TdtihdKBwVTDyrHyQ5HzB1CJv1C1tp4RnLG9M00m4MYRCi6D+Hdrnz7RnhosEyw9sYrmg+a7FUZ1mHTeYKF2DH42ewPv8kadK4Hdgk1nDInljK62oz7n4vot1nNEXs97xNQZzvzzV8iZTetCOivBT1JipFfxFjK9yh+oPZaqqJAY6/Nyz6ieNWNZ6lMTNEoUUVKDMLrOiz5EpU1YaX0QaixcFKV+rULBFN56xzB1/PC1O6kEmPhuzVLoadko0dKabjbhGgpRGx6W3QlLSMkpZH+TE0+Nbb1p4OGwbSc57temAs0NdnPC5w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Unconditionally use __GFP_ACCOUNT in try_alloc_pages(). The caller is responsible to setup memcg correctly. All BPF memory accounting is memcg based. Signed-off-by: Alexei Starovoitov --- mm/page_alloc.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index f967725898be..d80d4212c7c6 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -7118,7 +7118,8 @@ struct page *try_alloc_pages_noprof(int nid, unsigned int order) * specify it here to highlight that try_alloc_pages() * doesn't want to deplete reserves. */ - gfp_t alloc_gfp = __GFP_NOWARN | __GFP_ZERO | __GFP_NOMEMALLOC; + gfp_t alloc_gfp = __GFP_NOWARN | __GFP_ZERO | __GFP_NOMEMALLOC + | __GFP_ACCOUNT; unsigned int alloc_flags = ALLOC_TRYLOCK; struct alloc_context ac = { }; struct page *page; @@ -7161,6 +7162,11 @@ struct page *try_alloc_pages_noprof(int nid, unsigned int order) /* Unlike regular alloc_pages() there is no __alloc_pages_slowpath(). */ + if (memcg_kmem_online() && page && + unlikely(__memcg_kmem_charge_page(page, alloc_gfp, order) != 0)) { + free_pages_nolock(page, order); + page = NULL; + } trace_mm_page_alloc(page, order, alloc_gfp, ac.migratetype); kmsan_alloc_page(page, order, alloc_gfp); return page; From patchwork Tue Jan 14 02:19:22 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13938324 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E125DC02180 for ; Tue, 14 Jan 2025 02:20:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6E335280007; Mon, 13 Jan 2025 21:20:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 69155280002; Mon, 13 Jan 2025 21:20:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 50BC0280007; Mon, 13 Jan 2025 21:20:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 30702280002 for ; Mon, 13 Jan 2025 21:20:03 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id E122241799 for ; Tue, 14 Jan 2025 02:20:02 +0000 (UTC) X-FDA: 83004452244.15.758BCAA Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf22.hostedemail.com (Postfix) with ESMTP id F3203C000F for ; Tue, 14 Jan 2025 02:20:00 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=eBXCco45; spf=pass (imf22.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736821201; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6YPnFtJFai+0xFpNHgKwg1vEFPjmUMGue3EQsxvwhos=; b=fZrMRU4jUG9IMI9UkMkER7ofU2b32C/TugIZrYRNRhAqI+jkn07UTHNZSVfIHdR/wlBow/ 6DJvpYYVaeBnl1iqd+UZQKg6Xc75kSC8lf0AT+qvIYnLPw4Aneh1Dax0QHs3TTmxMGsb+x 5SNM0+3qgshrsMHhk7IqmeATznmA32g= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736821201; a=rsa-sha256; cv=none; b=n90cLXQplpW3vqaPRyRSL0HkJ8D0CBE9Epv//VT9cF9MVvViAsc4k351s5P1AAkxgjOpwA XknuTNBLOoHkRUlzDEdsYCpFSFxUTBgsKkaMkxqNAltBrM7BuOFRqjIWqQevydZNbS2zWM Csv3IzE/AnKGdoen+tkF0yTrPe8f4Fc= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=eBXCco45; spf=pass (imf22.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-2165448243fso105616645ad.1 for ; Mon, 13 Jan 2025 18:20:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1736821200; x=1737426000; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6YPnFtJFai+0xFpNHgKwg1vEFPjmUMGue3EQsxvwhos=; b=eBXCco45qDoZtNmr98dh/kY5rPZwMpKeqFD9+wqzjgQUeS/T/HEhGvw4tl9/di9ExM V7q/2k/ztgBlUpAD3e81mWHK/PMKhBgciIFGSo6WebMesz6IxxXFVeHLQfDcR3ZoA+ce pJbJKubX3PJiHL+taQW6WuIxb1yU2ZgC1BuGzHY4ZacWYA7efbTehD/VCoQUnHex+rQn e3mHG/HjQ8TQXpiP6b72HQB2IpQwCJCmxYvHXKu7OcmReR99iVzku7XStnLbnDVjgS7n KsperUXKLXTe0HBZziJTf9skQlvzO/vigYrov0v261slaKSYa5yKCIQh3Pzz3YumpBKY ZBHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736821200; x=1737426000; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6YPnFtJFai+0xFpNHgKwg1vEFPjmUMGue3EQsxvwhos=; b=pNtnSh7NzIE4tP90zDLMGMUph6Du+jrqiG75Krts3KwavA/855Rafne5sOCIjoJ7hJ AueSAt/gRa7T3zki7jcrC/UQmzhrznzlJf2xweUvmZvO3XfoHfwtmREla1nz+EoKYCxO ZszRdKBmKIjyrFObS6yVNx6cziv1Zj00swrD3NmuaeAZDyYthOcPhP7Uscdl0t/2j9Xf ecfUJlJAGK+UyadR3ATFc1XdSlXRgcrvXH3Oejl5fEE20l208FAyb+iECMIyL3ITeIeh UNbW2lbSiVgfE87RNSd+Dm4N5Biff1bJn4VDbNoUm0nwtuXic1rro8OxJxXHiOKwdx2j 8VUw== X-Forwarded-Encrypted: i=1; AJvYcCWV26BcI06wMsadetxFs5JFjuqLJeY1jXjINRmNVxVeXromW15RSH2MhDQi/0W5vyo9wHkCOFBjJQ==@kvack.org X-Gm-Message-State: AOJu0YwqGNyGZUh7zhgcSxP6inUKpuGGjgBm0i2rNEYVvKD2iDtBSVVW lAtv0Am7mfjBzbnFPMmU0x0JEOr3AV+MxaZlJ5pGfGQfytahNZln X-Gm-Gg: ASbGncv17XfINNgnCSLUlVH2ystm0VC7c2LraIH3QW7momzRiwtczBNeE5ymySfm54B fBcW4x7ZsMSJAswAAtExn/czBoVlRz44JtqZUEA+cIR829lDIricjbQkr4GXXqT4ZC9XdZAiwiL oC7xdxnynitygGxVFZSDAgFDM264TavcvEXnuhI/KGS3m3EWFnFLkfWCWPDipg1Z98GxZmjF+0w 8gh6ZOg1C0ry9+rRr23cHyFG4SWt+sIrOSU8dbXWGNLJ5qCXvb4UVm5tAtJdbZNwWR+zBCYiGX+ h5tJLKcd X-Google-Smtp-Source: AGHT+IFK2Q/NzWEMQDd1zkERrN23lZllcwNoRX4XpjEtT4xySg8TTJ5FTM1kqalpNAWOVle2SSYeWQ== X-Received: by 2002:a05:6a20:4394:b0:1e3:e836:8aea with SMTP id adf61e73a8af0-1e88d18eab5mr40855550637.14.1736821199823; Mon, 13 Jan 2025 18:19:59 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:4043]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72d4056a5fesm6475293b3a.48.2025.01.13.18.19.57 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 13 Jan 2025 18:19:59 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v4 6/6] bpf: Use try_alloc_pages() to allocate pages for bpf needs. Date: Mon, 13 Jan 2025 18:19:22 -0800 Message-Id: <20250114021922.92609-7-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250114021922.92609-1-alexei.starovoitov@gmail.com> References: <20250114021922.92609-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: F3203C000F X-Stat-Signature: zqtbrotsaj5z4gg7i7wxy4zry4ep6nib X-Rspam-User: X-HE-Tag: 1736821200-528616 X-HE-Meta: U2FsdGVkX19GqsQA1DK7SZuH289YCkkNA9v3kf3FOYH5s4zYT3ScxAxnIEY9LsEuc62WuVvznE1HwBMmejHQvAZWO9Oz5sHnsmbL12yLow4hRWai4EkiataN2hRcCYQHKk+GvhKsdTR9vbTRgv66YLI0c+mCUUd1GUscN/gvc2Ry9L3XA5z/3sVnTI74GMQcczUVgrvjpH1WbtXo5HbphlBfODzlf4j1GK4cP+GjcWhW7JyGo/LsotUV7WUqtjXY+z71yPx+l+rKvZaRGEFGFXvczQdDEUOlXKpi94QTyWhScooIoipmVBoZJb3Dx7T++LSnooL4fXBbP/LolQ/wTk9/VG7J6IQXpzHfJDKaH9RuGcWEPFkHtYGSMixppwOWpNW3FvujoJcxiB4IWH6lDxZV/RMUjWZ9WKmmtZqCDWOqo9plKdvYORSdlh29CEKc4Mh63IoGOTH9t9VG3KbYsW4YZdzrYCYkwDH9Besu1kiSOnPeMXDblq2nN+lejFCeWEFki6RToaQUN8U2VAYl20gKnHyMMtRhWCjsYezyMnXeAouuyGyE2quGYim1bcvSLkkPPRDg7iZxTYGGcVgKYyMNZoB8OcwORYaePrpkd42o9S0Int+py9XQ2jjsDid+qOubj0VX5aztgYqIL0OGcYXLf2wkn/VWOmcgX2QioftIjp4UYCZ1+aklQhIq7+x4qVQiGgqacxRzrImmMdn6irgvaBKBqiL6xAYVHCLTRephU+qx8wTounTh+umnP2kGFqrxjwLgJ/OwhTJ0fojoIu8JMLPNkofOWIINH/m/5TPxyHR2K6LH8W6xT2dLUvBo8sUAZ7+mCYasuAOM3pSmiDFHIDt0JIHH4t/y0dRL3fzydrDcdShH2JPGv42M4HGKtmRrrJg0rBgGbz3OqkeKOuxVdyVqLZ1tdRm3cEEEQ0MIPCGW/lknhlmbKrEIrTTsco2RTcFp6r3J5aIlHd4 UwV15hD7 hhfIrGQVCACFl5NfeVY0FEGL5Hu/7/J7/Cfo+pGPl40IkK7xOlSYQm9m7yh/2dZNH9OW/8LUItVQ9d0tGMs4FoVz15R8gztgdU2tGt23+/LsR/bfQ1j3XY4JElpQ4O3ckjeBaC6u24UmQB+O48yfUhr6Jm0D2IaTqNt+fDLiGoJpgvAWrIcf0vN0jfUykt/akPQYXUuc9FAGiQgWd5GTicd4C4sEEeSyDSQjRsbFbdy6Oe3UQ5ouWCleaCJYNEoY7hNII9WQBdRMCbfCzSvCf2KWSQZQjqUzDCCDpOyHmuKn7ql4drpAR7abMvSAtQmVc23oE2sd5sbLwg0Ok+agHhS9l7BsVgEs/S7tOj6Z4Vz+T6IcuLhEMM0uDhRyI1gr88dDkmRbM5CtQaCvWrKb5s16GpL0HVk7ICCJvA41x5sHi/tt/eRm+3hpxZt8/5/vZwXaQp/8CBSVnVk2AS8GwM7dhaeeR6osGjDJEkQqnh9rl54gQSlg2TGX55Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Use try_alloc_pages() and free_pages_nolock() Signed-off-by: Alexei Starovoitov --- kernel/bpf/syscall.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 0daf098e3207..8bcf48e31a5a 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -582,14 +582,14 @@ int bpf_map_alloc_pages(const struct bpf_map *map, gfp_t gfp, int nid, old_memcg = set_active_memcg(memcg); #endif for (i = 0; i < nr_pages; i++) { - pg = alloc_pages_node(nid, gfp | __GFP_ACCOUNT, 0); + pg = try_alloc_pages(nid, 0); if (pg) { pages[i] = pg; continue; } for (j = 0; j < i; j++) - __free_page(pages[j]); + free_pages_nolock(pages[j], 0); ret = -ENOMEM; break; }