From patchwork Wed Jan 15 02:17:40 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13939758 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29EE1C02183 for ; Wed, 15 Jan 2025 02:18:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B0E646B0088; Tue, 14 Jan 2025 21:18:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ABFFB6B0089; Tue, 14 Jan 2025 21:18:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 95FAD6B008A; Tue, 14 Jan 2025 21:18:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 75EFB6B0088 for ; Tue, 14 Jan 2025 21:18:01 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 074F61A0F22 for ; Wed, 15 Jan 2025 02:18:01 +0000 (UTC) X-FDA: 83008075962.10.4995DC4 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by imf28.hostedemail.com (Postfix) with ESMTP id 1C1E8C000E for ; Wed, 15 Jan 2025 02:17:58 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=RvjhG6RC; spf=pass (imf28.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736907479; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RRfQnK0Xbq5jILYR704HtD9aPWYF0UoBSTHx/j1KDDk=; b=YKFeL78qubPqLM2UbgOv+ZuSAfbICgj7FkM03pE4h2fCUtPwl/DlbQ8IrRxJD6tdhC6uVx MNwZvTJwoZ9eMLkn9jj+IOVLIJj88QQTpUzk13fbZ73HFHJcdbzgyhUI20c8vqJRnTqR+S rNGV1pnbw01sZqXgtCcyEKdAhNdSVS4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736907479; a=rsa-sha256; cv=none; b=4sNJk5bQWRix0fXM5Xa1bOnw/itJfQLgUpJKEM5GPHhfMV6K0nY/Ss+BPoEbBnoyzxO1xK eCwkk7LPUrnw4akNXmdLSsd/mHjc/DJOV6HpfqcbBAhw9zLZ2sGg8T/6T8rjiqYkc7Iclp 1SzSjD2O1Kn0a6Xb7dC7nH7aNo9wb2Q= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=RvjhG6RC; spf=pass (imf28.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-21675fd60feso136617045ad.2 for ; Tue, 14 Jan 2025 18:17:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1736907478; x=1737512278; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RRfQnK0Xbq5jILYR704HtD9aPWYF0UoBSTHx/j1KDDk=; b=RvjhG6RCnrO0lQxfNUz6HL+Cr8wMIFsWL+h8dno+hKoUQDpp074xbj57mfRIRIkV7e PzjcI5ou1VjELVcQ114VXRnFc6PaJWhK2HjIyDrUHUUeEgQTHWZWIfcotmZbV2S5xVGt ZJctob6G6Owyd05YgmuTreJYUU7U/SpAVtXosQWt59EfqcsyUCb7A7nXw5P9y9TSADEA f2NhLzCzriQIrUR/iXquV5hJarEcEZYq7iMfQgf/dzv/rmL5g2+nicBlnjTLnoeuqOOq DpTe/FSxsd/HilUcBJnzE1yOrc1149nQiIOqmOULuCmdjVWszoMXXW5/Y0hoXiScLkVB iVgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736907478; x=1737512278; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RRfQnK0Xbq5jILYR704HtD9aPWYF0UoBSTHx/j1KDDk=; b=s2w3e82r6WGDqdi2fpw9HJcMh88ySFNI7sA8XLO+biHe982DfDmpGrqxUX1eaRn5p6 Xve1XQ7Czfb6Xkn1OPN/uKf3X7glN9taloKgTY7I9FJiwn7TBCunOjN/m4RMb2VebICZ 0hltf6CIJlUErgN1vi56RExBTeFaPo6lWK8JsLK0iUChTisDfVdWm/PKZ4EN8B/8vfrs 3o/sk6IHRvXbI3H0av9LA3Pa6Asc+nR+r6Dp+Rw81cP9JWXegTBm+gzWMP3zxcu+JtZm 0MbjjP4sRt9QqKadRLqKHGTS6qsCjNbmOKXbo+0QQ//P9fUH0e//yAV0jvpxzR8/RRWU 4hKw== X-Forwarded-Encrypted: i=1; AJvYcCXeGa+d634LsZsXKXrUbfQ2lUKHRWsbWjxd0vItWM/rMRGjAJ6DHJSuoWaxVD11s3HefDZqj9Gdww==@kvack.org X-Gm-Message-State: AOJu0YzrcpuCa2y1l+gPmiCaovem5xxCLE+2OjoVED83G59TUmLLVIm4 e/eFhoKQttUUt/AofyyR8FRko9A8EtgrxdbHhPZHJokjax8A1I4y X-Gm-Gg: ASbGncsy9EGRt6/lCQgEzzblsNu4cQ68WTeWI0YJxL70AEQ3YMNsZW4wswp2jA6Uvrd 0QDfHLeINzSWaK3L0IUvPubxvdBQXyhG504uHIpbjnAJHK7RweuUJFcKjVuvm8Zh3g7bjs8Lw4Z G2roT2r+MmD1RUKTBJLL6RoAByycKQJJnmK+rO6UwRIhGxvwiT1v9XygB9JgCt1IGM8kIiL7++9 vx69yfAyXpVjtNIk2XyW8pNIM9pjk2cCovcMnorZqNXytiJOSV5v4mx0ESRXCVE9BbT2KBP/xs3 QDehlhbj X-Google-Smtp-Source: AGHT+IG6Y9Bj4+sss4AkgBmGlysFV+cBK96Pvs8KQSHpwbaMp6szpurOB66PeQKEY7FIq53RepZsSQ== X-Received: by 2002:a05:6a00:35ce:b0:729:1c0f:b94a with SMTP id d2e1a72fcca58-72d220077aamr40543484b3a.23.1736907477661; Tue, 14 Jan 2025 18:17:57 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:4043]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72d40658dd7sm8357578b3a.102.2025.01.14.18.17.55 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 14 Jan 2025 18:17:57 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v5 1/7] mm, bpf: Introduce try_alloc_pages() for opportunistic page allocation Date: Tue, 14 Jan 2025 18:17:40 -0800 Message-Id: <20250115021746.34691-2-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250115021746.34691-1-alexei.starovoitov@gmail.com> References: <20250115021746.34691-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 1C1E8C000E X-Stat-Signature: ywz8joyb85cnyjhyy6n648zb19spga7x X-Rspam-User: X-HE-Tag: 1736907478-190878 X-HE-Meta: U2FsdGVkX18maOm3gWmKywkSgHJNR+2NAuZ0dY/axeuZAmHHNNWibQFZc0OuSCDgTWPWglKu8/PHLEIInV7ZCMPG0xXyZ2KSkl+EJ8k5JJq9BsGP/4TZG7Ytb2TwIMHTa7ZI0E0DqapAD3k2y10zxnaLC+8MHXDoUDgv57erS4/SHdq2joPcumS3BsYorX93Oz33yWZ/96M6ze4AreHEO9cMe+we9VQzKv8Dxtqd1BuTVzELb9/TaJ+eKNi1n+81ArzYkosSpyugcmZZcAUMl/6zuFltbTiaN+gDb6tn72VvUsnwQycvYVxMtropIJAgFhnpFc45AjS/dxxVtlkTEHvr1f3hfvkhvesDILpdyC2CIQPCZOwJYSVyla3uuarhKZCw19qbthKTleQWQRpABXGX6Aqf9KzEMwSDswewZG8kii1768L5grQbfkTl4TIaAyZ23v/7S7mmf029anQAHy70NFvW6B7WYEv54tlR0e1Pkn9t1nZ+4s9dbzhR+R3v00xVqCijo6DxGPoOcVEGONP9P9lgVO0H1wRQxgMikrkeI7CYHYgxNJs3pxyj2Q0AIVQFD6E9HYT2/F3YRm6V9qP41PMObbZcFTfOJFRHUclq8osqhT8CKC6cnSu3z0+wluGtD1IQYDXrT2MwE6lZwGQ4OeEQuDeOpWVNmuIpd6g/HuuEwYaJjZodT+P2LImyAP54OE8cAM2qZFuVGYMPUK4xCJ6RumEPkWhv+0nL+XdbbW3XxIdAKgsaCisxkSjvd25+ZLsIfVSZalOrM6L6Z5vGgwM4kv3/Upiw+rCF8hW2qrc0mRfAxVt93CvDCkrrBzi1HNbiqligZYEjt4OPLun+s46MOwE1ZGMnARiu4bxxzRwzbi8fbw0qoRzZ/ldWbeUl2O2ZZrijJ6Tt0bZBz2QmDBHJn1isenKMoFi53d3FrtO7nR0hO/xc8reC90R0J99nqh5h73Lhpo17v36 fu7DvQu7 hhnwJAfR363vfj6i5NmAsdmITggmjQMgiFa+typ88l3uBRX6oCLDa8pOYHvifDK0AYeT8UW1vTmsXrsJ++Ti77H2/950ISL7gXif7aoKm37fcSNrDLFllc9fROhMD9tYQ6MuMMBpdd9+ktYIbFxP+KyletYZCh6JEgx+JooUewSKyLDYHWQ0eFvZrNgc68sv2B9uBgUZPyFfJXKeb48d6mgU89dvw+EGyEvKoKJGlrOjHyhnIL/eK3QsWhK4uHa0xZF+2tp6ru/1hbs0BunhzJCbfmNeo4yBO3u3/ZRobsNO85t5PkvyD+NvyUkSTRW9mmY7p5Bzosdx1QSMu3j4as7XXI2eyHlq0W11Uumgm9jmC5HETakVQACFAhQElzq0md5cfklq6AxQHryLvZugJ0mJ2lz8fX6ZHJtgkYYQ4TSCE9H5q2pAtUxA+zNXQJold7aioh+UU90LFzA5BuJCAFeaBesziTLi0I0wYehKFJS8KWhKdmlHMglleiX5tXpwHU0HA1aFTZMOQck+n4ECrCLHLcS57I7DFDZHhou/ZWEbNoTXq9gnPZavg/mQUXCaMpWVVTTstRt7KPc1yMZR2KH2Wa7DgMShhLFGbIAHFk4C320h+TEZziT/mgg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Tracing BPF programs execute from tracepoints and kprobes where running context is unknown, but they need to request additional memory. The prior workarounds were using pre-allocated memory and BPF specific freelists to satisfy such allocation requests. Instead, introduce gfpflags_allow_spinning() condition that signals to the allocator that running context is unknown. Then rely on percpu free list of pages to allocate a page. try_alloc_pages() -> get_page_from_freelist() -> rmqueue() -> rmqueue_pcplist() will spin_trylock to grab the page from percpu free list. If it fails (due to re-entrancy or list being empty) then rmqueue_bulk()/rmqueue_buddy() will attempt to spin_trylock zone->lock and grab the page from there. spin_trylock() is not safe in RT when in NMI or in hard IRQ. Bailout early in such case. The support for gfpflags_allow_spinning() mode for free_page and memcg comes in the next patches. This is a first step towards supporting BPF requirements in SLUB and getting rid of bpf_mem_alloc. That goal was discussed at LSFMM: https://lwn.net/Articles/974138/ Acked-by: Michal Hocko Signed-off-by: Alexei Starovoitov Acked-by: Vlastimil Babka --- include/linux/gfp.h | 22 ++++++++++ mm/internal.h | 1 + mm/page_alloc.c | 98 +++++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 118 insertions(+), 3 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index b0fe9f62d15b..b41bb6e01781 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -39,6 +39,25 @@ static inline bool gfpflags_allow_blocking(const gfp_t gfp_flags) return !!(gfp_flags & __GFP_DIRECT_RECLAIM); } +static inline bool gfpflags_allow_spinning(const gfp_t gfp_flags) +{ + /* + * !__GFP_DIRECT_RECLAIM -> direct claim is not allowed. + * !__GFP_KSWAPD_RECLAIM -> it's not safe to wake up kswapd. + * All GFP_* flags including GFP_NOWAIT use one or both flags. + * try_alloc_pages() is the only API that doesn't specify either flag. + * + * This is stronger than GFP_NOWAIT or GFP_ATOMIC because + * those are guaranteed to never block on a sleeping lock. + * Here we are enforcing that the allaaction doesn't ever spin + * on any locks (i.e. only trylocks). There is no highlevel + * GFP_$FOO flag for this use in try_alloc_pages() as the + * regular page allocator doesn't fully support this + * allocation mode. + */ + return !(gfp_flags & __GFP_RECLAIM); +} + #ifdef CONFIG_HIGHMEM #define OPT_ZONE_HIGHMEM ZONE_HIGHMEM #else @@ -347,6 +366,9 @@ static inline struct page *alloc_page_vma_noprof(gfp_t gfp, } #define alloc_page_vma(...) alloc_hooks(alloc_page_vma_noprof(__VA_ARGS__)) +struct page *try_alloc_pages_noprof(int nid, unsigned int order); +#define try_alloc_pages(...) alloc_hooks(try_alloc_pages_noprof(__VA_ARGS__)) + extern unsigned long get_free_pages_noprof(gfp_t gfp_mask, unsigned int order); #define __get_free_pages(...) alloc_hooks(get_free_pages_noprof(__VA_ARGS__)) diff --git a/mm/internal.h b/mm/internal.h index cb8d8e8e3ffa..5454fa610aac 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1174,6 +1174,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone, #define ALLOC_NOFRAGMENT 0x0 #endif #define ALLOC_HIGHATOMIC 0x200 /* Allows access to MIGRATE_HIGHATOMIC */ +#define ALLOC_TRYLOCK 0x400 /* Only use spin_trylock in allocation path */ #define ALLOC_KSWAPD 0x800 /* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ /* Flags that allow allocations below the min watermark. */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 1cb4b8c8886d..74c2a7af1a77 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2304,7 +2304,11 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order, unsigned long flags; int i; - spin_lock_irqsave(&zone->lock, flags); + if (!spin_trylock_irqsave(&zone->lock, flags)) { + if (unlikely(alloc_flags & ALLOC_TRYLOCK)) + return 0; + spin_lock_irqsave(&zone->lock, flags); + } for (i = 0; i < count; ++i) { struct page *page = __rmqueue(zone, order, migratetype, alloc_flags); @@ -2904,7 +2908,11 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, do { page = NULL; - spin_lock_irqsave(&zone->lock, flags); + if (!spin_trylock_irqsave(&zone->lock, flags)) { + if (unlikely(alloc_flags & ALLOC_TRYLOCK)) + return NULL; + spin_lock_irqsave(&zone->lock, flags); + } if (alloc_flags & ALLOC_HIGHATOMIC) page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC); if (!page) { @@ -4509,7 +4517,8 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, might_alloc(gfp_mask); - if (should_fail_alloc_page(gfp_mask, order)) + if (!(*alloc_flags & ALLOC_TRYLOCK) && + should_fail_alloc_page(gfp_mask, order)) return false; *alloc_flags = gfp_to_alloc_flags_cma(gfp_mask, *alloc_flags); @@ -7023,3 +7032,86 @@ static bool __free_unaccepted(struct page *page) } #endif /* CONFIG_UNACCEPTED_MEMORY */ + +/** + * try_alloc_pages_noprof - opportunistic reentrant allocation from any context + * @nid - node to allocate from + * @order - allocation order size + * + * Allocates pages of a given order from the given node. This is safe to + * call from any context (from atomic, NMI, and also reentrant + * allocator -> tracepoint -> try_alloc_pages_noprof). + * Allocation is best effort and to be expected to fail easily so nobody should + * rely on the success. Failures are not reported via warn_alloc(). + * + * Return: allocated page or NULL on failure. + */ +struct page *try_alloc_pages_noprof(int nid, unsigned int order) +{ + /* + * Do not specify __GFP_DIRECT_RECLAIM, since direct claim is not allowed. + * Do not specify __GFP_KSWAPD_RECLAIM either, since wake up of kswapd + * is not safe in arbitrary context. + * + * These two are the conditions for gfpflags_allow_spinning() being true. + * + * Specify __GFP_NOWARN since failing try_alloc_pages() is not a reason + * to warn. Also warn would trigger printk() which is unsafe from + * various contexts. We cannot use printk_deferred_enter() to mitigate, + * since the running context is unknown. + * + * Specify __GFP_ZERO to make sure that call to kmsan_alloc_page() below + * is safe in any context. Also zeroing the page is mandatory for + * BPF use cases. + * + * Though __GFP_NOMEMALLOC is not checked in the code path below, + * specify it here to highlight that try_alloc_pages() + * doesn't want to deplete reserves. + */ + gfp_t alloc_gfp = __GFP_NOWARN | __GFP_ZERO | __GFP_NOMEMALLOC; + unsigned int alloc_flags = ALLOC_TRYLOCK; + struct alloc_context ac = { }; + struct page *page; + + /* + * In RT spin_trylock() may call raw_spin_lock() which is unsafe in NMI. + * If spin_trylock() is called from hard IRQ the current task may be + * waiting for one rt_spin_lock, but rt_spin_trylock() will mark the + * task as the owner of another rt_spin_lock which will confuse PI + * logic, so return immediately if called form hard IRQ or NMI. + * + * Note, irqs_disabled() case is ok. This function can be called + * from raw_spin_lock_irqsave region. + */ + if (IS_ENABLED(CONFIG_PREEMPT_RT) && (in_nmi() || in_hardirq())) + return NULL; + if (!pcp_allowed_order(order)) + return NULL; + +#ifdef CONFIG_UNACCEPTED_MEMORY + /* Bailout, since try_to_accept_memory_one() needs to take a lock */ + if (has_unaccepted_memory()) + return NULL; +#endif + /* Bailout, since _deferred_grow_zone() needs to take a lock */ + if (deferred_pages_enabled()) + return NULL; + + if (nid == NUMA_NO_NODE) + nid = numa_node_id(); + + prepare_alloc_pages(alloc_gfp, order, nid, NULL, &ac, + &alloc_gfp, &alloc_flags); + + /* + * Best effort allocation from percpu free list. + * If it's empty attempt to spin_trylock zone->lock. + */ + page = get_page_from_freelist(alloc_gfp, order, alloc_flags, &ac); + + /* Unlike regular alloc_pages() there is no __alloc_pages_slowpath(). */ + + trace_mm_page_alloc(page, order, alloc_gfp, ac.migratetype); + kmsan_alloc_page(page, order, alloc_gfp); + return page; +} From patchwork Wed Jan 15 02:17:41 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13939759 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0FF67C02183 for ; Wed, 15 Jan 2025 02:18:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9FD936B008A; Tue, 14 Jan 2025 21:18:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9ADF66B008C; Tue, 14 Jan 2025 21:18:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 84E9A6B0092; Tue, 14 Jan 2025 21:18:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 65A9C6B008A for ; Tue, 14 Jan 2025 21:18:06 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id D7E4B140EC1 for ; Wed, 15 Jan 2025 02:18:05 +0000 (UTC) X-FDA: 83008076130.06.4E83FFD Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by imf19.hostedemail.com (Postfix) with ESMTP id E78811A000A for ; Wed, 15 Jan 2025 02:18:03 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=LA1cwVzi; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf19.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736907484; a=rsa-sha256; cv=none; b=nCk2ChlYsnPPSpZO+KZvq9hfRrA1o13LHs96vhbBsIGd4EJujFCpLLxBF3z2sBNts7xEht VcdAWoe5nteEZvyEJfYvPx/5mMO2x+XFBeCG66aQfBfehR5fi7wNlq39b16Gr7vSnUNunL m0+3AF/X+GXKKgm29A4/diJeVv1PNqw= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=LA1cwVzi; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf19.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736907484; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=S/4MiM3RUKHfwlZ4XiouzlgmWU/NeE4CfHyc1IF4D50=; b=AcJwPMEFwc66WvHuUSWua/lfncusuOMZNZ0Ng7365WYS/a4Z+BOVtZcujipD/lL3NExDVj cfTKSPa+H5h69LnebhRHooNmGHwMkPpdAFKfql4Ko3ZoYTPmICkuLOCbzKygOViuJKD4Xi EdEl/+peXN3d7fwAGwfdiKAknza3+60= Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-21a7ed0155cso106947375ad.3 for ; Tue, 14 Jan 2025 18:18:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1736907483; x=1737512283; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=S/4MiM3RUKHfwlZ4XiouzlgmWU/NeE4CfHyc1IF4D50=; b=LA1cwVziuhrM5j3LRiTFwJdKlUdyM8kpyfbEOR6f5HoRevPOqXcgosqbNYxXmYibF1 0kk5tMXSfkE1CKuNGc1YrmO6CBQzms0+NsJsTLtfXljxxDqGd/uYXhyfWfLXqRHLoDPE LlOAuHPKuz7k7LUSHc7who4n9cpLIy8tT1LrHm0uy9ASJae/kI10SHnyCC1D2IgEOfKb tSH0VF60H/NFXA+ByQp6wao35UMLCie40GD1Mafk1tNY5CuFukCahY2WMhKzh6aO9Voe +ad1pV5kwr3mvR15N8+Dz1xNs2wM3J9HQ381aoJp0fe5tXI61P3cBzHeDzAHzxbOsIBz NJXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736907483; x=1737512283; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=S/4MiM3RUKHfwlZ4XiouzlgmWU/NeE4CfHyc1IF4D50=; b=WVK5ZKvQ8NeLpb8upTtd2MNVT9XJ0PmbrkAm/niznC1MPMmt5G8lcVDa/G81bDc6Td TOIfjVM/79KSKA3khqUXsiTGppZAYJZq+hHcUoQKFHCCQfD9WCcEkEqbi+WFZVwawP3a x9QMrvrQ2rDQHGnhK026L0pTnuuAGkSUWAYRGf7xcrhOdzj4WDq3kbfl0KiGET1dvpLl 9xaLVvcJqGkNhmQsVQ/S/3JK1IbWLxahHd43L8ZLNh5wZ4a7dSL1h6pXsoYsnI518NJa TmTgX9FBeX88UVT0VhKrBUL8AH9Q+CgEexW7g39AdzC3ARoZiE1jG7wCTzMIqp5WJaag ePSA== X-Forwarded-Encrypted: i=1; AJvYcCWVL+dkFsQov2NXswaec+AnzDzfdYM9BEVgTYgALB6pThoj/eL56sKBes7XBqcWb9ODSwbEZxa4Fw==@kvack.org X-Gm-Message-State: AOJu0YyRsxDQ9QmKyMkhuGZfxicVyRQFtfF83ZBN3a68BrhvHGRmP6d8 PW8TLZb3JiqMlZmCA3rSYnmzwDHuWv0Pqq8qtv+Qiw3k36fP+xjT X-Gm-Gg: ASbGncsKixvWZNQxPx49gUxbcqCwTq1T46tpeT6Wc0GocWFXAK7wb5E4F2dM2DdgYMN WMaKQL06Aa4eYyvda5bnOtIGd0tE/wKbL2xHVZlgZkojZdNR1Ji0apd0LTgclLVl9jLo6ibBpea P7vUEcBetmEHVpq9bf+UzEgdNx1LRap9XUAaM0EYbACegxrzNk01JoqllGTB/5YjTPSKMf/f245 e+BxzuhfUZRRs52gbD/jO75MF+VvDYlx7cDvbdXpZusvygBgH3Lcmoda+PK5IFcpFeZlN+oLrvh 6UGWqwLE X-Google-Smtp-Source: AGHT+IE2HSgtTwGa6fxDzYYIuEsJKl/KfTDggfWr6X8IZDgbB2AylvSlI27CX51K5j5TCNfNyEO1Hw== X-Received: by 2002:a17:902:fb8e:b0:215:6426:30a5 with SMTP id d9443c01a7336-21a83fc150dmr338535215ad.40.1736907482518; Tue, 14 Jan 2025 18:18:02 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:4043]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-21a9f25a583sm74065695ad.244.2025.01.14.18.18.00 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 14 Jan 2025 18:18:02 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v5 2/7] mm, bpf: Introduce free_pages_nolock() Date: Tue, 14 Jan 2025 18:17:41 -0800 Message-Id: <20250115021746.34691-3-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250115021746.34691-1-alexei.starovoitov@gmail.com> References: <20250115021746.34691-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: E78811A000A X-Stat-Signature: wphnm81qnzb4dgsfuobbw3ymsx9mfgkt X-HE-Tag: 1736907483-227103 X-HE-Meta: U2FsdGVkX18yAhBDkMc9OhZhk5oFzOIwH9Awax/ajKti+2NDRXpQXm98oc60baTzWdIfqrq+9JUrkBbiLaXrn4VZErypKNtQXOn/BRFLY15yt10yWBRPDRcM/0alao4lBZZ0nX25s8ZluGFadylZ6K0M6WhDl3YxzIrVDSznWk+o+bMngGyucr1wDZtO9ggbg4cCN7moFyQ2gml2vp9hZiRz+QrlVmjmhYOpxqZLJSKosO8BZxPapOPcNtbnVTxRAIH6g42CXuKBIjwXMU9+onD9MADb3XakPI4Uu3pYYt06VHwgjejjWOlM21NS7wYplXKB+xF0Lp3sydmX0EVwIQRks8LPO7/GdmdAguzrvlkE3M735fiblRGLpex4LNJKgDPg+w6kSaG7iJJvLfHsx5sQCJeCogETmFIZQO8Ndfs4hDtceblEXEYx1y48zdNhcXGGi4LRDKmIyBS8NzA8jfDU926HOo5uSEpwNVUMMemU7K4AWf5totHTB4W0NXxIiM7gQDarj8AlJIJsHR9/WY4zfSxS4nWc2Er+X9JOZ9nIKU35UsShA+W1LfC0JaW8OETDtFYL2MvROVmQPmPwiAwtM1nRZiLIboWSpqnoJru8jl5qS5mGv7jVevNHeZ9ykEHBF81Ae7Bk68vjvLX3ENTovjNymSKy3NgrGQ5xudGN/baQkYjvNcQMqP3FHc0JaKGyNhUrIDtPn5TSn+Jsq1fE9zicV4zbAoAHjqSrfsH3XqSMnng2adRkFqcxDgRYYL2lX3o6a5PTWmF+WlQf6Tz8o1jRGDjH8CGxjvd3tN/FAfIK1rC3YS4gy2k2P9lil+pJZYIrYPCkOIy6C3/cypoCKerybxFnQP4CarGusIwkO43H79FxgfIxFKCGp8jUP7TCnomT1q8WidcBZwNILJ9a08094kZ+kQd3N74A0ZzkeDh3nAj9cqUkUM4+qSmBcYI1b0G7KpqQCKkzspp COyuo8PA +y0l6cRD2o3hE1VgkQbSZ0xcFMHUt3uV0r+hxlxT0526cNOHjptX9EJWU1LbyUtEwHGKFrw3m5mFaOZ8+mlVrrahbDoijTElSPK9EWHmRiLBflFr0uNJUXjgIV2IhwFVfkMIMUFVPeL74TLjfAlcG93KSLYATvDKLmmZmy1dPWM7/dlNdrhgKNl9Q6NAoChzfwIxxxNFvVuatiY9XywjjI5eZnCoHpk52AeZFRkIatZHPWOe41TAAeShk4RnbEic/VwJqodfxUyjiJEfKcYhen78vEI8NdPiL5t6RikOX0wQNzMaFuf1AgjtS6KHMPam4GDOgQhPVSxEApPIBNJSXEKAhztS+zmNcjp/7+OD5MLjWUnZ8bNk1Qh/fjhqSzDmOkAaGcO1S/mCvlnFHIZcHTEHf1vdJp0YHI5U5K04SQSk+wlg09VtZNymcF4kYG216+NTEtgiOSeTOH6XpvzyLnZl56mTaa69wQ6JdXQwFteJBNU4TRzng/Lq5og== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Introduce free_pages_nolock() that can free pages without taking locks. It relies on trylock and can be called from any context. Since spin_trylock() cannot be used in RT from hard IRQ or NMI it uses lockless link list to stash the pages which will be freed by subsequent free_pages() from good context. Do not use llist unconditionally. BPF maps continuously allocate/free, so we cannot unconditionally delay the freeing to llist. When the memory becomes free make it available to the kernel and BPF users right away if possible, and fallback to llist as the last resort. Signed-off-by: Alexei Starovoitov Acked-by: Vlastimil Babka --- include/linux/gfp.h | 1 + include/linux/mm_types.h | 4 ++ include/linux/mmzone.h | 3 ++ mm/page_alloc.c | 79 ++++++++++++++++++++++++++++++++++++---- 4 files changed, 79 insertions(+), 8 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index b41bb6e01781..6eba2d80feb8 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -391,6 +391,7 @@ __meminit void *alloc_pages_exact_nid_noprof(int nid, size_t size, gfp_t gfp_mas __get_free_pages((gfp_mask) | GFP_DMA, (order)) extern void __free_pages(struct page *page, unsigned int order); +extern void free_pages_nolock(struct page *page, unsigned int order); extern void free_pages(unsigned long addr, unsigned int order); #define __free_page(page) __free_pages((page), 0) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 7361a8f3ab68..52547b3e5fd8 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -99,6 +99,10 @@ struct page { /* Or, free page */ struct list_head buddy_list; struct list_head pcp_list; + struct { + struct llist_node pcp_llist; + unsigned int order; + }; }; /* See page-flags.h for PAGE_MAPPING_FLAGS */ struct address_space *mapping; diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index b36124145a16..1a854e0a9e3b 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -953,6 +953,9 @@ struct zone { /* Primarily protects free_area */ spinlock_t lock; + /* Pages to be freed when next trylock succeeds */ + struct llist_head trylock_free_pages; + /* Write-intensive fields used by compaction and vmstats. */ CACHELINE_PADDING(_pad2_); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 74c2a7af1a77..a9c639e3db91 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -88,6 +88,9 @@ typedef int __bitwise fpi_t; */ #define FPI_TO_TAIL ((__force fpi_t)BIT(1)) +/* Free the page without taking locks. Rely on trylock only. */ +#define FPI_TRYLOCK ((__force fpi_t)BIT(2)) + /* prevent >1 _updater_ of zone percpu pageset ->high and ->batch fields */ static DEFINE_MUTEX(pcp_batch_high_lock); #define MIN_PERCPU_PAGELIST_HIGH_FRACTION (8) @@ -1247,13 +1250,44 @@ static void split_large_buddy(struct zone *zone, struct page *page, } } +static void add_page_to_zone_llist(struct zone *zone, struct page *page, + unsigned int order) +{ + /* Remember the order */ + page->order = order; + /* Add the page to the free list */ + llist_add(&page->pcp_llist, &zone->trylock_free_pages); +} + static void free_one_page(struct zone *zone, struct page *page, unsigned long pfn, unsigned int order, fpi_t fpi_flags) { + struct llist_head *llhead; unsigned long flags; - spin_lock_irqsave(&zone->lock, flags); + if (!spin_trylock_irqsave(&zone->lock, flags)) { + if (unlikely(fpi_flags & FPI_TRYLOCK)) { + add_page_to_zone_llist(zone, page, order); + return; + } + spin_lock_irqsave(&zone->lock, flags); + } + + /* The lock succeeded. Process deferred pages. */ + llhead = &zone->trylock_free_pages; + if (unlikely(!llist_empty(llhead) && !(fpi_flags & FPI_TRYLOCK))) { + struct llist_node *llnode; + struct page *p, *tmp; + + llnode = llist_del_all(llhead); + llist_for_each_entry_safe(p, tmp, llnode, pcp_llist) { + unsigned int p_order = p->order; + + split_large_buddy(zone, p, page_to_pfn(p), p_order, fpi_flags); + __count_vm_events(PGFREE, 1 << p_order); + } + } split_large_buddy(zone, page, pfn, order, fpi_flags); spin_unlock_irqrestore(&zone->lock, flags); @@ -2596,7 +2630,7 @@ static int nr_pcp_high(struct per_cpu_pages *pcp, struct zone *zone, static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, struct page *page, int migratetype, - unsigned int order) + unsigned int order, fpi_t fpi_flags) { int high, batch; int pindex; @@ -2631,6 +2665,14 @@ static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, } if (pcp->free_count < (batch << CONFIG_PCP_BATCH_SCALE_MAX)) pcp->free_count += (1 << order); + + if (unlikely(fpi_flags & FPI_TRYLOCK)) { + /* + * Do not attempt to take a zone lock. Let pcp->count get + * over high mark temporarily. + */ + return; + } high = nr_pcp_high(pcp, zone, batch, free_high); if (pcp->count >= high) { free_pcppages_bulk(zone, nr_pcp_free(pcp, batch, high, free_high), @@ -2645,7 +2687,8 @@ static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, /* * Free a pcp page */ -void free_unref_page(struct page *page, unsigned int order) +static void __free_unref_page(struct page *page, unsigned int order, + fpi_t fpi_flags) { unsigned long __maybe_unused UP_flags; struct per_cpu_pages *pcp; @@ -2654,7 +2697,7 @@ void free_unref_page(struct page *page, unsigned int order) int migratetype; if (!pcp_allowed_order(order)) { - __free_pages_ok(page, order, FPI_NONE); + __free_pages_ok(page, order, fpi_flags); return; } @@ -2671,24 +2714,33 @@ void free_unref_page(struct page *page, unsigned int order) migratetype = get_pfnblock_migratetype(page, pfn); if (unlikely(migratetype >= MIGRATE_PCPTYPES)) { if (unlikely(is_migrate_isolate(migratetype))) { - free_one_page(page_zone(page), page, pfn, order, FPI_NONE); + free_one_page(page_zone(page), page, pfn, order, fpi_flags); return; } migratetype = MIGRATE_MOVABLE; } zone = page_zone(page); + if (IS_ENABLED(CONFIG_PREEMPT_RT) && (in_nmi() || in_hardirq())) { + add_page_to_zone_llist(zone, page, order); + return; + } pcp_trylock_prepare(UP_flags); pcp = pcp_spin_trylock(zone->per_cpu_pageset); if (pcp) { - free_unref_page_commit(zone, pcp, page, migratetype, order); + free_unref_page_commit(zone, pcp, page, migratetype, order, fpi_flags); pcp_spin_unlock(pcp); } else { - free_one_page(zone, page, pfn, order, FPI_NONE); + free_one_page(zone, page, pfn, order, fpi_flags); } pcp_trylock_finish(UP_flags); } +void free_unref_page(struct page *page, unsigned int order) +{ + __free_unref_page(page, order, FPI_NONE); +} + /* * Free a batch of folios */ @@ -2777,7 +2829,7 @@ void free_unref_folios(struct folio_batch *folios) trace_mm_page_free_batched(&folio->page); free_unref_page_commit(zone, pcp, &folio->page, migratetype, - order); + order, FPI_NONE); } if (pcp) { @@ -4853,6 +4905,17 @@ void __free_pages(struct page *page, unsigned int order) } EXPORT_SYMBOL(__free_pages); +/* + * Can be called while holding raw_spin_lock or from IRQ and NMI, + * but only for pages that came from try_alloc_pages(): + * order <= 3, !folio, etc + */ +void free_pages_nolock(struct page *page, unsigned int order) +{ + if (put_page_testzero(page)) + __free_unref_page(page, order, FPI_TRYLOCK); +} + void free_pages(unsigned long addr, unsigned int order) { if (addr != 0) { From patchwork Wed Jan 15 02:17:42 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13939760 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A99D7C02183 for ; Wed, 15 Jan 2025 02:18:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 40AF36B0092; Tue, 14 Jan 2025 21:18:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3BA3D6B0093; Tue, 14 Jan 2025 21:18:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 25AED6B0095; Tue, 14 Jan 2025 21:18:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 070046B0092 for ; Tue, 14 Jan 2025 21:18:11 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 7F7D98072D for ; Wed, 15 Jan 2025 02:18:10 +0000 (UTC) X-FDA: 83008076340.11.034C4E0 Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) by imf28.hostedemail.com (Postfix) with ESMTP id 9F42BC0015 for ; Wed, 15 Jan 2025 02:18:08 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=UibYau5G; spf=pass (imf28.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.216.51 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736907488; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=w+Lbt2oOvlSTPMdIT0UeNEpvkZe5nRvEnawf5u/tH+w=; b=ya64AUDwnxg51Ko7qINtXyIraw3La1yxHRcto22WozgCwCRrt1+hvyQL4cDqoB1eo+I7Zc ah3CZfKdO7tDDpo+I87Iv5/vtE646Kziq8NeVLN0wApTf9h4EiOJEpUzBjIdA4N8E8CfBE 9vZsRFk2Pxcwkl8uggre2vavL2WSwvM= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=UibYau5G; spf=pass (imf28.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.216.51 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736907488; a=rsa-sha256; cv=none; b=VRKcFrOxdZc419kIEAbFZ28MA4sHLRrYdpKsJTQtCCa4GeF0PpgJjZUIMRIlp8KRk41DS+ don1gYX9uK8nFRV0CjDpcE/Mk3SL0BnegT0sLSBfCIvbrcxeJrmDFRAvqVP3KdbXUkc5hH 5guAYJKn6bs1+BtSaGAvBEajQ74KFgg= Received: by mail-pj1-f51.google.com with SMTP id 98e67ed59e1d1-2f441904a42so10670909a91.1 for ; Tue, 14 Jan 2025 18:18:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1736907487; x=1737512287; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=w+Lbt2oOvlSTPMdIT0UeNEpvkZe5nRvEnawf5u/tH+w=; b=UibYau5GQHLvXKW8hRyUo9f01jJGrC0nrKJulEGUAJi5MNHCVtocvJ29iA+dZ8HU1c uLjb//Nep62t3a3ALLCR6sQAr5TP7Se59eiXmU+XRS/BUhC2e8Q4lNMl2Z5hFvf4R5hc 7eeVtDhxCO2Enl72HBDn4PeO14CY6mtPotQPX7vyZQyPEWGrzbk7VfXgMIfbgZSqUDMx j7UYdV/vjeBEGq2/B06t5RdJjah7sKbhMFN39BwxDyelMtDqwnwjCL0EqCakNBfTVZN1 zc+gKpiMzlK+wyfol5rcbUvPJUZ65rhXAiVgq9UWFIz2fMImpAeHFFFci2bIz9fw1vhr tq3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736907487; x=1737512287; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=w+Lbt2oOvlSTPMdIT0UeNEpvkZe5nRvEnawf5u/tH+w=; b=DMDg92xDYPYzUl9vuuKjZqgz028KolD9Jmb1N/VAr9Z2dIUR2HgN4DWN0B33oIdWxh YRXXjkd4fOdW3BG0GnVvhCW42qsHvpzU2LVPZzwvsLDnyz9NFYid2TYbvs7B4Qht1O2b 69VlRxMHSdKtfjYl3Dja34HE/p6drkpIgAMoR6PCywp/ajjK38IdiVvAmfT0L+u9WPyW CI+QbMq6BD20Npe5wVs7IfX9YGaaVtsA+Og2g577WXpEHI7fK+FztNqNwfaTKFvWfWuE 1zH4QI/SVTFiLA++SvtDuCq53m71/XiySA0MTQl5Br+bzVFYR5f3q1iKbGfUbjJW3EgK rQDQ== X-Forwarded-Encrypted: i=1; AJvYcCX+BGyEa5l5JX+BJ86BARP9/2oJvlddLh/Tl+pP496aXDFw+iCKg6PMUrOQWE2gXAcYdtYEofpbbA==@kvack.org X-Gm-Message-State: AOJu0YwOoRlAAh3qsyAJU8B4VNoJ6OI5fgbHJBV/oLonpsW/ueXsLBBo f/RGqWoHuX4r8nHJ/yfUVy4kY3jDc47z13Rc0XpXlYTUHkwws6x8 X-Gm-Gg: ASbGnctlA8LXip+3kqQhtFWyoczClX9V+xrHaOHNMzHGX8pxGmAupCdQqkNYu7FFWps Q3YEXi115sXSMFcTiBEJfPywf0WlNaGPDuN92dnzuA6WEIAdU2vw0Z35sPS6L7ZiiUKfVmwOUFq 2Dg76dLSdIk+wdpFJvSXzAIlRNpYUH4dlTrhU4VlZCNE1AMZUwGrCkAnrjOldjcuPdiNiHQYjGk KLqcMr96n5pWkq8FEY/sxyRPBvxFxnkzWq1qaIvEgPdcG9/+10Lhi5H64vv2x4+xiYyBv190fjy LTdv4Bqb X-Google-Smtp-Source: AGHT+IG/cFjjNteRnggOPcJd7ml3IweEri0dZgwymeHUFWzv/5oMKi4cBo8NV/fu9YuNibkf/qV0PQ== X-Received: by 2002:a05:6a00:1942:b0:71e:6b8:2f4a with SMTP id d2e1a72fcca58-72d21f7bcccmr39594907b3a.12.1736907487230; Tue, 14 Jan 2025 18:18:07 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:4043]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72d406a4dddsm8016029b3a.155.2025.01.14.18.18.05 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 14 Jan 2025 18:18:06 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v5 3/7] locking/local_lock: Introduce local_trylock_irqsave() Date: Tue, 14 Jan 2025 18:17:42 -0800 Message-Id: <20250115021746.34691-4-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250115021746.34691-1-alexei.starovoitov@gmail.com> References: <20250115021746.34691-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Stat-Signature: 5s7ekij1brbuq431pef9aj1nnz7n1js8 X-Rspamd-Queue-Id: 9F42BC0015 X-Rspam-User: X-HE-Tag: 1736907488-352266 X-HE-Meta: U2FsdGVkX19j8J1eFvpzD4RgYGCWA4tZQEzH5E/pMaAX+kMNsrgJIFxTASXWmXIc21zhRzM1vPD6DjzHKl0HTpS7YmLuEbFo9Cg979XUDQPW4IdqA18AsO3XHwwPoEqhKon1PL9K+LNIOJdVdCk9ZZntfwjlw9iOrE/3WX8P4qs//a3SBd2EUIZeRqIlbtNy3cbxdm1we/5tWbGmFgS07w9K65ZqvpXDpPnXzfYe7G7Inx0eFi+C6anxjVyGavh688j/yMx7TpLiW/XS7a5dF24SWpqP1VXypakh3GHgfnVvXoiUEz969FB5/ZyJa3XglyDNJYNN6RW51GjOFVp6CBUV8VhStOaKZEY/inxmZVbbS9lm/u+4O/nXn+cB2bj+Ihef7APxqYlxfW5bn8vJRXLKmxvzBX/ZfLQ6l4wxyTWZlAfrzZmJF3J6jcohUU2UeTvofUw4C1YWSevEPwsVvcLuzsEKZnOuB5muZMzOkoaHR3kYn1ifmzqxVMm3q5PvkQZA10NdVGXVJrsHs7xvlT1AjcIBrjyb0ycuiYdUeYko9S2GtnRL98GTsXHWnCs+GAoJpVq3J135pGBzi7kW5G7OzPn5NHzzDCcF67Bw0fQCkQSrlS6VCsY94AKJ536ISQNdHo2KzXRHlUGQLSeIYVtSfAftJNZ2V4sO8BK9iuxJ5F3JIbjbhidnXrUMJXay4cegBSS+FOX+kcHII1D7pnU36rcczsflpzEOF7HWLPQ1/3TrZg+a9mul4ntI9VYUOpjvEMv4uyt27r6juyc2K52sVZlNVYuKQ2iWgZxYtuk7HRKGj/ynygExf1fAormvZfztWV9wThjjFSyD/Sx/neAsmOQ3Z8F/UrlFcliscZztAe+GOSEZeZzkubfKG/UOKcr/4dmAiTZTL1etDQL9K39lVGIGVOEXNcFafDV25SULVcsxRS1EA6eMaKTsY38hbllANulRloztggmiYAv hRzNF9Nr 72jrrWWPrdcEBFXejbPub5wnNwzwxWGUR04Ppw0X2XkDfE8OIJ3L0KHtRkCBHQUY5O8sYm9zESBYq1bc/h1Esozid2F3T9zfUonuNy1+QTlJ6OuTSpSNNgXNmdOcx1fgo7YfNa8U/FtYVyWxQEoYY/BzIBqx965qgySwpHPM8F64iA4Jq4INEpvt5O1W9GTTkH9OHTyKkpsDdLYa4rXqLANPtBpiS7YXFveM9n+k2MLelyP7cO7vRHrHmIOX09bi+E2QKp5ttIQT2CadunsCzN3pgvJpskpI/c2SGVKmDtTZlNoVuk0GurzPXVg7igNS+u14zrZaRVm11cIxVFUCTtqHF8Yex5+SSP6uRQT+E5cYz2gzX3EPGxpK/E88C/UrCZZrXFLFeJfqeLZyR0uKQYU91MUI0W8cpzwDMC6Y2QVs3TZsQC/TGc2TsdDHJoNF573wsYvI2sITOrQxsQ54Cx/N+a9r+dFU8gMiuEhdtRxz8UQ+TFzEoa8YYSA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Similar to local_lock_irqsave() introduce local_trylock_irqsave(). This is inspired by 'struct local_tryirq_lock' in: https://lore.kernel.org/all/20241112-slub-percpu-caches-v1-5-ddc0bdc27e05@suse.cz/ Use spin_trylock in PREEMPT_RT when not in hard IRQ and not in NMI and fail instantly otherwise, since spin_trylock is not safe from IRQ due to PI issues. In !PREEMPT_RT use simple active flag to prevent IRQs or NMIs reentering locked region. Note there is no need to use local_inc for active flag. If IRQ handler grabs the same local_lock after READ_ONCE(lock->active) already completed it has to unlock it before returning. Similar with NMI handler. So there is a strict nesting of scopes. It's a per cpu lock. Multiple cpus do not access it in parallel. Signed-off-by: Alexei Starovoitov --- include/linux/local_lock.h | 9 ++++ include/linux/local_lock_internal.h | 76 ++++++++++++++++++++++++++--- 2 files changed, 78 insertions(+), 7 deletions(-) diff --git a/include/linux/local_lock.h b/include/linux/local_lock.h index 091dc0b6bdfb..84ee560c4f51 100644 --- a/include/linux/local_lock.h +++ b/include/linux/local_lock.h @@ -30,6 +30,15 @@ #define local_lock_irqsave(lock, flags) \ __local_lock_irqsave(lock, flags) +/** + * local_trylock_irqsave - Try to acquire a per CPU local lock, save and disable + * interrupts. Always fails in RT when in_hardirq or NMI. + * @lock: The lock variable + * @flags: Storage for interrupt flags + */ +#define local_trylock_irqsave(lock, flags) \ + __local_trylock_irqsave(lock, flags) + /** * local_unlock - Release a per CPU local lock * @lock: The lock variable diff --git a/include/linux/local_lock_internal.h b/include/linux/local_lock_internal.h index 8dd71fbbb6d2..93672127c73d 100644 --- a/include/linux/local_lock_internal.h +++ b/include/linux/local_lock_internal.h @@ -9,6 +9,7 @@ #ifndef CONFIG_PREEMPT_RT typedef struct { + int active; #ifdef CONFIG_DEBUG_LOCK_ALLOC struct lockdep_map dep_map; struct task_struct *owner; @@ -22,7 +23,7 @@ typedef struct { .wait_type_inner = LD_WAIT_CONFIG, \ .lock_type = LD_LOCK_PERCPU, \ }, \ - .owner = NULL, + .owner = NULL, .active = 0 static inline void local_lock_acquire(local_lock_t *l) { @@ -31,6 +32,13 @@ static inline void local_lock_acquire(local_lock_t *l) l->owner = current; } +static inline void local_trylock_acquire(local_lock_t *l) +{ + lock_map_acquire_try(&l->dep_map); + DEBUG_LOCKS_WARN_ON(l->owner); + l->owner = current; +} + static inline void local_lock_release(local_lock_t *l) { DEBUG_LOCKS_WARN_ON(l->owner != current); @@ -45,6 +53,7 @@ static inline void local_lock_debug_init(local_lock_t *l) #else /* CONFIG_DEBUG_LOCK_ALLOC */ # define LOCAL_LOCK_DEBUG_INIT(lockname) static inline void local_lock_acquire(local_lock_t *l) { } +static inline void local_trylock_acquire(local_lock_t *l) { } static inline void local_lock_release(local_lock_t *l) { } static inline void local_lock_debug_init(local_lock_t *l) { } #endif /* !CONFIG_DEBUG_LOCK_ALLOC */ @@ -60,6 +69,7 @@ do { \ 0, LD_WAIT_CONFIG, LD_WAIT_INV, \ LD_LOCK_PERCPU); \ local_lock_debug_init(lock); \ + (lock)->active = 0; \ } while (0) #define __spinlock_nested_bh_init(lock) \ @@ -75,37 +85,73 @@ do { \ #define __local_lock(lock) \ do { \ + local_lock_t *l; \ preempt_disable(); \ - local_lock_acquire(this_cpu_ptr(lock)); \ + l = this_cpu_ptr(lock); \ + lockdep_assert(l->active == 0); \ + WRITE_ONCE(l->active, 1); \ + local_lock_acquire(l); \ } while (0) #define __local_lock_irq(lock) \ do { \ + local_lock_t *l; \ local_irq_disable(); \ - local_lock_acquire(this_cpu_ptr(lock)); \ + l = this_cpu_ptr(lock); \ + lockdep_assert(l->active == 0); \ + WRITE_ONCE(l->active, 1); \ + local_lock_acquire(l); \ } while (0) #define __local_lock_irqsave(lock, flags) \ do { \ + local_lock_t *l; \ local_irq_save(flags); \ - local_lock_acquire(this_cpu_ptr(lock)); \ + l = this_cpu_ptr(lock); \ + lockdep_assert(l->active == 0); \ + WRITE_ONCE(l->active, 1); \ + local_lock_acquire(l); \ } while (0) +#define __local_trylock_irqsave(lock, flags) \ + ({ \ + local_lock_t *l; \ + local_irq_save(flags); \ + l = this_cpu_ptr(lock); \ + if (READ_ONCE(l->active) == 1) { \ + local_irq_restore(flags); \ + l = NULL; \ + } else { \ + WRITE_ONCE(l->active, 1); \ + local_trylock_acquire(l); \ + } \ + !!l; \ + }) + #define __local_unlock(lock) \ do { \ - local_lock_release(this_cpu_ptr(lock)); \ + local_lock_t *l = this_cpu_ptr(lock); \ + lockdep_assert(l->active == 1); \ + WRITE_ONCE(l->active, 0); \ + local_lock_release(l); \ preempt_enable(); \ } while (0) #define __local_unlock_irq(lock) \ do { \ - local_lock_release(this_cpu_ptr(lock)); \ + local_lock_t *l = this_cpu_ptr(lock); \ + lockdep_assert(l->active == 1); \ + WRITE_ONCE(l->active, 0); \ + local_lock_release(l); \ local_irq_enable(); \ } while (0) #define __local_unlock_irqrestore(lock, flags) \ do { \ - local_lock_release(this_cpu_ptr(lock)); \ + local_lock_t *l = this_cpu_ptr(lock); \ + lockdep_assert(l->active == 1); \ + WRITE_ONCE(l->active, 0); \ + local_lock_release(l); \ local_irq_restore(flags); \ } while (0) @@ -148,6 +194,22 @@ typedef spinlock_t local_lock_t; __local_lock(lock); \ } while (0) +#define __local_trylock_irqsave(lock, flags) \ + ({ \ + __label__ out; \ + int ret = 0; \ + typecheck(unsigned long, flags); \ + flags = 0; \ + if (in_nmi() || in_hardirq()) \ + goto out; \ + migrate_disable(); \ + ret = spin_trylock(this_cpu_ptr((lock))); \ + if (!ret) \ + migrate_enable(); \ + out: \ + ret; \ + }) + #define __local_unlock(__lock) \ do { \ spin_unlock(this_cpu_ptr((__lock))); \ From patchwork Wed Jan 15 02:17:43 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13939761 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3E09BE77188 for ; Wed, 15 Jan 2025 02:18:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CBE03280002; Tue, 14 Jan 2025 21:18:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C6E60280001; Tue, 14 Jan 2025 21:18:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AE7F6280002; Tue, 14 Jan 2025 21:18:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 8C033280001 for ; Tue, 14 Jan 2025 21:18:15 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 1C198160E7F for ; Wed, 15 Jan 2025 02:18:15 +0000 (UTC) X-FDA: 83008076550.19.7285320 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by imf26.hostedemail.com (Postfix) with ESMTP id 333FD140007 for ; Wed, 15 Jan 2025 02:18:12 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=JvweOne2; spf=pass (imf26.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736907493; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mxa1zavVGcrL3mhmNj3zaW3g0F2sJ8bBr01E6sisj+Y=; b=qbL35h7euf/2F+A5qBrZiwHDXTDef0WPeVKry8IoJ0mekpSz0RSXahyjzML9K9GIm+9tdV ERtCQT10DKWnxnzfwxs8TO2DHayOwPav+4vP1O9NVlVZIbnqtqzyDwl50NfBWfIopHzfpH jm+WvdkU31BW3/hAHknzKemMMYxhjo8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736907493; a=rsa-sha256; cv=none; b=zyDdL+4KMt01UKW1kW15W+OYD5BmfCHceJ9mq891usJFux8u6DabjnRsmr5RsEirCXxa8w GeXGcH+7J3zGWI6qPqiDjyO2ZQpQJP15leRwHABnjvgECqK6z7+oKO18OG7kdLOtSYchTH E6lXGR1MCg1Mit3UgWOyhaEzZEuM4aI= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=JvweOne2; spf=pass (imf26.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-215770613dbso75286635ad.2 for ; Tue, 14 Jan 2025 18:18:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1736907492; x=1737512292; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mxa1zavVGcrL3mhmNj3zaW3g0F2sJ8bBr01E6sisj+Y=; b=JvweOne23A2wra/5hAhMKf3hAi9v2Df8eSSxMX/wIFNLbqHG+2VRhvkw2A4tw37t7z a9mYRi6eOfTnNmY/851+EZD3LwX93AXpjWlBihkFHw2NkMCBG3MBlAYJAgs07Rn0D8iw E7ttipiI5oP+M3nQhMTPuFBngiXP4xaRNqT43JfrTbhTXs2OCI9MNB9vckTAbERZL8jk MNDs3Dj9ahwsPF8Kf0MJHpLr8+bHDC/LUlja3uAhsD4GsSyYxzUKFsj9hwpLkuF2iFL4 nxasYWohnbycCR+9FOYRknVxFiF3gXcz5x8djchtP/myWdCJLs6hFA0UjTwOV/QL74rK 8Uow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736907492; x=1737512292; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mxa1zavVGcrL3mhmNj3zaW3g0F2sJ8bBr01E6sisj+Y=; b=rWzAVhkjkINGUnfRFW+TfztukVdBx0wHxT2UWvK5wZ+YFmfdtmv0WXwHYw6gXpWM39 LuB1pKGL33+riT6Q7eN7JEvExPXvlMZoKg/E5Ex6tzmHhPKUCCBnyvo0ESpoKkiWRa4r K+2WuRVufSzsvJsfMzxg5JN2NmFif71HPmDbFPRtqxbYh+OmLqtThpNnjAU6Xp34nwcG QNQJpGKL2xLjDd1QWP+jREAbvVsVgdpsUkDRq7GdOPRFMe2r5L9qWnd2SIPbEV9aWiNL fYHy6VVUBWmwHnNVBJYGbaohy+LTd5xyWnpKzcasGY7e43ufkHRZzUrbU32SPnVk/R1v uklw== X-Forwarded-Encrypted: i=1; AJvYcCUMR4Osmw6/E5x4Ki00kZszFvNluaNbs8NOkAaVn4SDFG3fyckIDRGii1QYT+r4cohXxjFb5DVsZg==@kvack.org X-Gm-Message-State: AOJu0YyDJ3VBi7hugFRn47Dn/ZcB/T2E4bJsB1Pvyg04b7Z20XNrJL9f n0c54evXr8L74R87KGF3MjyTgyYsYzrZOkZ94si8ZkL/EV2LcKy/ X-Gm-Gg: ASbGncvCQAFckO4A2WCLxkloZMgSl+giW2Po+5ey7H9xcWAd+dUCavHphag5vTCq+2C yG3P1cKtV2ODp1RR/RM06lxxOZhlJyfMmeOIBUEP20SeOZEtl3uUfBXYcC+axRJOCFnyHYiS0RR B6ZtvSIpqI2Dgow6Ucts0uS8D8/YvHkSSF8TuHZzuL5PkujBNZuNFAnnMU7ZANZRDfzrGA4lNOO 749rpusglPI7WBmW/dA2UczzGJ1+yp2Z/NRBzy5f8cpOIakYnNjER7Z+o4P5AFxMhL1nK6RY/I/ WypWxVV7 X-Google-Smtp-Source: AGHT+IES28E6/UzBuAXcKHyytweWh1DbAVGKG/FlHuIJMf6aRzxb6IkpsZav+M1wSoIo/J2A0mfOZg== X-Received: by 2002:a17:903:41c5:b0:215:19ae:77bf with SMTP id d9443c01a7336-21a83f4ea67mr419970035ad.19.1736907491906; Tue, 14 Jan 2025 18:18:11 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:4043]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-21a9f12f919sm73361445ad.69.2025.01.14.18.18.09 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 14 Jan 2025 18:18:11 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v5 4/7] memcg: Use trylock to access memcg stock_lock. Date: Tue, 14 Jan 2025 18:17:43 -0800 Message-Id: <20250115021746.34691-5-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250115021746.34691-1-alexei.starovoitov@gmail.com> References: <20250115021746.34691-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 333FD140007 X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: oencnryexobtz7k8p4we19eqfx65nxtd X-HE-Tag: 1736907492-690670 X-HE-Meta: U2FsdGVkX18wwm4rfgenU6BAotmVFKeW6kAPZab9wP5wD1HR/jVX5vffI+/xK7pShxKNBp3sDmjls+ottYAVme0GLez1Dr5KSSigKExbqVGFRvRA8ktnpe9u0ffDPCz38mV6bucZpN1vQ/S6Al6ICwu8BoQ94bffWKeWUmi7+INPe/mmlNTYE3Tb8GJPDIfx3+Zxu1t7SUed8XAoX+eBd0GLV9NhMZjwKLzkbiDWVC8rM76YBw5ZzRJYiY1aQ4KEXl/zJ7s+dyPU8UB2V7nPJAav60aUfqs2AddlNJk4XwdjT5ywQZJso8UEKaJwJTFR0ypKtZe4XOCbOt5hLwNJjW9FGlapke9OydAzFMriCvO9S9vxTbwko5lvoBa8VADX74hF0Ujoh/a5VF2UojS8IeYMSq8wbCxXl6wujMa/ka6UA0NZplVpE0FGlScHQjkmWk3eclg3kiBnVBym9C/q0zvhTGJEQEPviWsBiPHEDDkiS6EnH/RGaanwAW5OzuoT7pQZDSxmsWcN5U4xfIxhk9/2FDwfz8XD/9JOn6TvX6vZV04UglVtX+y9itxGO9AOOBMTBYeo6F6aPj/jfjj/51+4S9W3Ej2r5UcgnxHG05RoplmsVxW2UPbRT4U6N861AGyB2nh904TpLTEpODTEkULOWniA2JQeqpfkaIs5Mxudw9SLLByTgMMkYlQAnYtKl9/P84+HAPF/9d+OeLIpGG/2IdsA7ZfULGFAP1ly7U0gpXXf6/ey9EDP75xOOsJjbLnwERPdN9RdU6T7wOAIT+Gj2JxlopBiaQTNJWeOrm/usI6sPZQPVFQw0WHj+9PUyLc9M/NzZmN5Lfq7AwCroYHRQtl/gkcbm8TUlV/1GgyKHgnifCl2718wU2U24Qsb2h3ULxUUB+JXmjLJp7BGl2K71SY0WKfsxQyS05KTgIyqSCualT/kzFFzU8JDOzl363N3BtK1cit/zKX7/S6 riTa1skj UZ322dgWhH6eod97Lp+8Nqz3rYN8psm1IMGR9Gt7JEF7tXIMVnyE+hgMw9Au6rA4B10wQ9FQSPSHBCyHSHOx3MnfZuhkPp/QwZoGUonvLWTHPNkrosYFI1/QmYXVpr0NeitYY1qoinmyxl3eQ/z2Z69lKFt2daL5lJemyjixHkLIHzrjDv3tC4osZbBknJ7MAwray1MqTfoLDPaxXwF8nYXYg2Gn9Z6kCxcL+gmkDiCGKi0IJG+bulz2xUEV7PqnqW9p7ebwzdqh7FZ6OblslNiRUVgOwD3X3/BC24ra/oHZ6jIWs8ZJ2/zJKnOKt7wIyBx3B63LffNQAewy7epG453grian0WpyljDIKGqpYFRmfTGG7l43NGHeQmUTslWEVmt/Er0Z9t0fEDlMJYdpsw9vunO7+5X/D9Zy8gfoguo4hwBXGuN20uYsNR8XX4aafYoAJ2JLm2I3RMOwe16q9olMTGmuccDpHAqY1OmrSSRp6uZbKMDdy21NUrBmFKJ9efmNqzpF3HbrYNZSAI+RNrUxi8A8Nbst1edRHPbBw/pLnJ1JaxMU0+zacKg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Teach memcg to operate under trylock conditions when spinning locks cannot be used. local_trylock might fail and this would lead to charge cache bypass if the calling context doesn't allow spinning (gfpflags_allow_spinning). In those cases charge the memcg counter directly and fail early if that is not possible. This might cause a pre-mature charge failing but it will allow an opportunistic charging that is safe from try_alloc_pages path. Acked-by: Michal Hocko Signed-off-by: Alexei Starovoitov --- mm/memcontrol.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 7b3503d12aaf..e4c7049465e0 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1756,7 +1756,8 @@ static bool obj_stock_flush_required(struct memcg_stock_pcp *stock, * * returns true if successful, false otherwise. */ -static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) +static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages, + gfp_t gfp_mask) { struct memcg_stock_pcp *stock; unsigned int stock_pages; @@ -1766,7 +1767,11 @@ static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) if (nr_pages > MEMCG_CHARGE_BATCH) return ret; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + if (!local_trylock_irqsave(&memcg_stock.stock_lock, flags)) { + if (!gfpflags_allow_spinning(gfp_mask)) + return ret; + local_lock_irqsave(&memcg_stock.stock_lock, flags); + } stock = this_cpu_ptr(&memcg_stock); stock_pages = READ_ONCE(stock->nr_pages); @@ -1851,7 +1856,14 @@ static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) { unsigned long flags; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + if (!local_trylock_irqsave(&memcg_stock.stock_lock, flags)) { + /* + * In case of unlikely failure to lock percpu stock_lock + * uncharge memcg directly. + */ + mem_cgroup_cancel_charge(memcg, nr_pages); + return; + } __refill_stock(memcg, nr_pages); local_unlock_irqrestore(&memcg_stock.stock_lock, flags); } @@ -2196,9 +2208,13 @@ int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, unsigned long pflags; retry: - if (consume_stock(memcg, nr_pages)) + if (consume_stock(memcg, nr_pages, gfp_mask)) return 0; + if (!gfpflags_allow_spinning(gfp_mask)) + /* Avoid the refill and flush of the older stock */ + batch = nr_pages; + if (!do_memsw_account() || page_counter_try_charge(&memcg->memsw, batch, &counter)) { if (page_counter_try_charge(&memcg->memory, batch, &counter)) From patchwork Wed Jan 15 02:17:44 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13939762 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9EA49E77188 for ; Wed, 15 Jan 2025 02:18:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 28482280003; Tue, 14 Jan 2025 21:18:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 234EA280001; Tue, 14 Jan 2025 21:18:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0AFEE280003; Tue, 14 Jan 2025 21:18:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id DDE28280001 for ; Tue, 14 Jan 2025 21:18:19 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 9784080EF9 for ; Wed, 15 Jan 2025 02:18:19 +0000 (UTC) X-FDA: 83008076718.12.9575D96 Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) by imf06.hostedemail.com (Postfix) with ESMTP id BA2F518000B for ; Wed, 15 Jan 2025 02:18:17 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=JG4VS+qL; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf06.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.216.47 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736907497; a=rsa-sha256; cv=none; b=4AaD+B2Sdwyx1DyKv55k2TQHYsHCzvdW9JPIpW+ZoePc1LdTd1a6xkTuqtPkJCGwxyG5xD ya7htJCgTajIYGMHOU8tDw/RQyHVU+rHhTEfN6Crxw+bXz3GmmMiazMeY66fRp7u4hogvP Z37ulj8GzSRjOigjIBrxjyuSZIcEp7Q= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=JG4VS+qL; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf06.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.216.47 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736907497; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9FG/FEC6x+yVU1JM4G4OAfpXBrpZC0py7nMQcVE3sTc=; b=A75rpzON/zFkmGISdLtvTN1o0NUHnrWEDMlO5TztWbEsxQIpxoD0KH2lDCufj2T1ZynWEm DmvEnN61sacaIiJ4FUBS6HTStc2dqcJ6ze5t/YLEVsqUREq6txrCLf1Zvw0jOOhy26ViE4 gbs8r0c7wKEbh9eTwovdiyg5kc0wbHY= Received: by mail-pj1-f47.google.com with SMTP id 98e67ed59e1d1-2ee67e9287fso10271812a91.0 for ; Tue, 14 Jan 2025 18:18:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1736907496; x=1737512296; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9FG/FEC6x+yVU1JM4G4OAfpXBrpZC0py7nMQcVE3sTc=; b=JG4VS+qLSpJD1DWI84G792pWAkl1gcr3u8vcjWOyGVlHv0yz40tyVRNgUkWuL4iQs5 TS+MjA+Vj7+5D9Hx0t5qYIPsdun0F0fPIFjGnsGo+A0PCbiCwfhJA+SDE2g3B/ST6CyV R4xjuJdNq6AN1NlkNl6bsk8EmoFcO42geMyyapRngllIFVm0RfG88HDQHOsdNuE2/aSG f+G+Q7xL0Z7Q04kL5IPf6g9A+0KKsOA5sM5lSKLF47DI/c8QG5/LcG0w+q+EJmYREEWF cTH6dm6u65tkTXX0o/tpNFVj4yYpbQK7FCpiufVLeUd/ZLIlVZDkBCqUW05N5UTv6LC9 orJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736907496; x=1737512296; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9FG/FEC6x+yVU1JM4G4OAfpXBrpZC0py7nMQcVE3sTc=; b=L3CB3cWWCQaQnQw5DMr6C1D2AvkxvhZ0g0f5MPmVYnK+CKOidq9xp8NhqtGoUZ7133 LJuY4mTQ8LXKEzIHNts7Lev4puwKz8iBaG5wy0aWvSiHuFnIk1k0K10KFiqNoHqbAU0l u17Fp/+qaVBfY1BkhDhwEIB7cPxubh6NG1oGI3eKWvDe/wZEooaThMVZkNDYjI5Kc8Ev dUuCt1An6f52S4aNbUhHkcoxwcznrlKmzGSkrxT86KAdp4ZUL7eQu6YFXs79oEie/Tc7 9CMd6jZx9o5UmNA3jnOFgqtVyhF8nz2s/oihzDcBJFlMpEc5oge52ld7Fd3jaybZyojh 5urQ== X-Forwarded-Encrypted: i=1; AJvYcCVRDzLNPLddX1/gTd1oFTgSe96Ak4rRpxc3MY/UYiJ/3jimoQpPalOhn9saVsi7is4LoWbuVenVbQ==@kvack.org X-Gm-Message-State: AOJu0YynTTrQSPkUP9CsO0+c8YFpe9ntkVGHgyS0QzgBw5kBDFki9QFA zqNuCn0o9VKoJZmf28oiwKbasYCFMGWNUlTAOF/80smTBoJIOfdg X-Gm-Gg: ASbGncuPGf7/a/65YOP+ImpG9sOmhufWDtsPvCx7ky3MTgJB1TXj7nbd4zChjkd6d29 15DMS/7QXoAKzGx+H0x1ijMUrQXsgUm/XXrEkEL26GNiQXbjFnHKHFVpnR9EzVxDsxmO9jQTOgf rdFRK9kwwiZQQzWVP4mBtVyzd1TNv+stK1wGGjvV15cFNiW+K7S4bNpILo+TX1FuBE/rNiXVTZr v5VNEaZjjhT3qXXo/1q82zxCou+JLqUPLEcXiSR8g5AvyMYfE5brD+7JAdb31SLwjmbhkb1kK4l y9ZShw2o X-Google-Smtp-Source: AGHT+IHNs63EfYlrVfU/fJaiMNGL7uVjWhVVtEL4nC7WmO69hxtGaNlrKdUTNVZudJVLwEfNIUiYFQ== X-Received: by 2002:a17:90b:2f45:b0:2ef:2f49:7d7f with SMTP id 98e67ed59e1d1-2f548ece7afmr43528398a91.18.1736907496611; Tue, 14 Jan 2025 18:18:16 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:4043]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2f72c157a09sm231999a91.8.2025.01.14.18.18.14 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 14 Jan 2025 18:18:16 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v5 5/7] mm, bpf: Use memcg in try_alloc_pages(). Date: Tue, 14 Jan 2025 18:17:44 -0800 Message-Id: <20250115021746.34691-6-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250115021746.34691-1-alexei.starovoitov@gmail.com> References: <20250115021746.34691-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Stat-Signature: uweyqg6hfgtj3z4o3gc3p36zeiwgqxfj X-Rspam-User: X-Rspamd-Queue-Id: BA2F518000B X-Rspamd-Server: rspam08 X-HE-Tag: 1736907497-902736 X-HE-Meta: U2FsdGVkX18O4iQgyHLLYm2P2Z92Iwk5JWfEc2tqVHMowwWsVUkNEyO12DGvqKimbris/7qNu7Ftfy1moQpLS08HEp87xg/a8qLaUQltuyv3oviFY1Xrp3NLn5lCGuonLcGw7ja/Yu6415O5yuQsvgz/JXgpP8B1Y2YPEeBIv4iqZ9jr3piTYF6URyruqI4X8k9+dUt/TBgGc61MtfaylyPIM2xddkeuk9o+0tF6HgSvLkOFsGCWMpQTPilN57r+ePKIlCUqcwDNfzQMrOnJOh0qJsszHbQhC3vpgzIMd5ULf9Va8HhDQjX6qbPOrm6p7pthNDVNNQj2cgwQ1AZ4F17BOFX0odpVP3/cLSNGnxb4FN7SVMFNcBNNW9Wo56RSERlOc2fVPAYQ3afNADgUJbS7HEzOfnYoGVKPN0s1PlwXettE5V+JtIJcVhyyGTQMEc79Sxo9yBcOJSfks2KcCiRDoRC96reJnUXgItMA1Wujw80qX/benbx2x+uEjmPkmoUsvxBlqGDtkQH8ZzmW35Bj0uIW2It08gzs3gi+gOCsE8EWn2wyTPdzzRhQQFKVDR79pD/djHK6tfXauaWLYQxp4lD1STdIB493j0L/C/WdPnJeaGnsx2Fg4W257SdUEIH1GD15Cd6GGjbrBqWtNfIaWOEn3P/ZWMUsU6iAbgJNTndIqm3oGlzqu2KrttVJAI8b0gC5jdc+Nn69I83+/QhiF24S9wqaLGPZsKkprTr7SXjSr8+UP4x3634XO13O+zuQEhuiKM5jKr0NAZY5h5bqk2uMz1Cp1KreFpt4Ue69PkUwYkRn7V9Z1+mplFGpfv0ZxFI7fnsd+bb0owIZpCukgh9PYYQrO5Yz2964c8MOKmuc4N+gB8+pugo8NL+mqFe/u+Wq7nFRsLziQ3ObAYuRIoI0Traen/sQ4glFYzMZMg+AtwNLUPne1Th4X3roIClA/F/vKRkegCfqOMK tg8bPNGe nJyJFSTGhRSShAKe1Wrn8LNAyxDwl+TjGkbcO6/nBTyVx+oiNAEz5nK9lBFK9QOVT7MgmyNCB9T0qTBXU39L2WTaH9ANtm82IrB0w0vtZJyownEOYOkU7qCbrf2yCgqNYHJWK4yrrNbFZSlykCqIgFNQa8p+iS+OXyMpZdDdq9Tw3JUEjDQU/VfCzUGee0Rufkrir8xt9zSf9IGUs6/fApVj2oGTUXkgQ3MHOKKFPdvOOqW3a2DsFcbJpxLCzKkd4nhK9Ob90tjhjvMMc3/BaiDdPXfnPWDRd1srstIE3FMqVB9Fb6aVvT2hroN5ygaco83rfnfcHQshH0WRVbIdA7EACKKusCR+Gd1mIPsBL3rYyI4NnhAYNKOLIBwBDsEy7Pf1rL2W+clcun4MqyKvWWqhd1dgBcuTwqUu7B3h/POrFB9rikDYfWy++9RMlKmvzzhXzu4vuSxu8SRdBdm45trxDNzVMuzJNzq+vfem86XqakyxmylK6vxdK7g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000092, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Unconditionally use __GFP_ACCOUNT in try_alloc_pages(). The caller is responsible to setup memcg correctly. All BPF memory accounting is memcg based. Signed-off-by: Alexei Starovoitov --- mm/page_alloc.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index a9c639e3db91..c87fd6cc3909 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -7131,7 +7131,8 @@ struct page *try_alloc_pages_noprof(int nid, unsigned int order) * specify it here to highlight that try_alloc_pages() * doesn't want to deplete reserves. */ - gfp_t alloc_gfp = __GFP_NOWARN | __GFP_ZERO | __GFP_NOMEMALLOC; + gfp_t alloc_gfp = __GFP_NOWARN | __GFP_ZERO | __GFP_NOMEMALLOC + | __GFP_ACCOUNT; unsigned int alloc_flags = ALLOC_TRYLOCK; struct alloc_context ac = { }; struct page *page; @@ -7174,6 +7175,11 @@ struct page *try_alloc_pages_noprof(int nid, unsigned int order) /* Unlike regular alloc_pages() there is no __alloc_pages_slowpath(). */ + if (memcg_kmem_online() && page && + unlikely(__memcg_kmem_charge_page(page, alloc_gfp, order) != 0)) { + free_pages_nolock(page, order); + page = NULL; + } trace_mm_page_alloc(page, order, alloc_gfp, ac.migratetype); kmsan_alloc_page(page, order, alloc_gfp); return page; From patchwork Wed Jan 15 02:17:45 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13939763 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B0EFE77188 for ; Wed, 15 Jan 2025 02:18:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D80A6280004; Tue, 14 Jan 2025 21:18:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D305F280001; Tue, 14 Jan 2025 21:18:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BD113280004; Tue, 14 Jan 2025 21:18:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 9C2FA280001 for ; Tue, 14 Jan 2025 21:18:24 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 4D059C0F3D for ; Wed, 15 Jan 2025 02:18:24 +0000 (UTC) X-FDA: 83008076928.11.529334C Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf20.hostedemail.com (Postfix) with ESMTP id 6C5851C0013 for ; Wed, 15 Jan 2025 02:18:22 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="B8i/k22B"; spf=pass (imf20.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736907502; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Q8dNygR7UFFvbzRNwqjbILq73kIw4oFHXTY0L0penA8=; b=aC1cvuX1qPfTd6JugVslW+ISKeQStWTbzrjctbA/MxXXHNwkYh5XGH6/9gWMTsBBiGFAq6 p2NMW6GlbJot5ngDuFiI8SeP/tXUxop4HJgIW3bLTzMpGEBXl1gO2eYh2lUo+C9rNU7LSB mVKreMjUktllml9aTFSWesa9KLogPWY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736907502; a=rsa-sha256; cv=none; b=uKAKphXWYJUJFT/2CH4cR0I7bCOiIzBNjZ595kajDHSRcV8pnBQMcOsmV2I5jCLG5Cx8cb PMQV+gTjUtt4zW7FEnqtyr4LLNVtxFs061mMSaBW3GqawCZmnhtBn6aPl7lNDgKMFquEwp zsfx/A8WLJ9jF5OKXQk0ixx5/fEg+8c= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="B8i/k22B"; spf=pass (imf20.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-216634dd574so72639865ad.2 for ; Tue, 14 Jan 2025 18:18:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1736907501; x=1737512301; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Q8dNygR7UFFvbzRNwqjbILq73kIw4oFHXTY0L0penA8=; b=B8i/k22BOze87K4uHhju5n5/JfxxLk1KisMvCb3QKw9hO5fK8uF/aRM8xd6z0dHeQh XT0tFYW2EghiG+h7bmnB55xTpPnWSyB37WETNTNWKvUE6KIL+ft4gQfJqCUvIAj4EF9t ZVsqKSV9032pz61aJYva1NF/o532I02iOXy0qLtkm7gEp7WL1GyyxL74E5GjdzxW+Ft6 l48jZi38USZi0IqyBdefS7zLT6S6oh2o0hAh1LCekOHk+Otfc2REgNBAUyMSS+gObdWt XNEELopaCXnA8NVPkv+lQMJ841Hc6z/Jp4G3qGW3fufXJcHYgqGPgTxwdZhtDUXXKkEX cGdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736907501; x=1737512301; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Q8dNygR7UFFvbzRNwqjbILq73kIw4oFHXTY0L0penA8=; b=gLmoXMGECXeoHsz3w52Q3Fgy/fAROTLpylBWmohzGRIk9Nj7HS/l+/Zgc/DtMCy02P 54RJKIkrHnZPoZ8+6aonO1Cj0ywHsoxW3fSpWdZdHCrLACtMxy21QlWs1z4W7RIsTJcz ZeEmXvxNWRynUXJsCihdhMc2XkbcDZpzVhf4cJa0A8oSXiybKw8Wy2I3eR1n/yZCni+j WD4wr6tD+K2rTZW4q3kOKXLXc4kP6wPa4dWbKqbwwIq44ArNythnEnSes6cnAbz0i9FT VnTkDsRgM03yq1m7xvRxIkxh+aslHs24GTnL8dv+D0I6UJz3C6kOfuvOF+T7vRKgdkho 8eAA== X-Forwarded-Encrypted: i=1; AJvYcCXIq25mkq4wM8DEAM+iVWQtiu3SsNaJWSkZIo0z3tMFYTOaXDN4yqQlnVx3oAAr70NBeGowS9XmlA==@kvack.org X-Gm-Message-State: AOJu0YxqmoP/CYGq8Ngnx0jgvyIx5L/JXE4Bbs3oCUceYqmxiEGZhTyk IcOz80jT9e/Jr4cQvdpYWhQPLOOd37Ljh2TKS8fuV/RDGOQQ7nxK X-Gm-Gg: ASbGncuHbas34VOHKMkGc7FEuNLDMMSoAzNVZVKeLbMoLo7cXm3j1rblk9apFs0nqsC MVTwmZ1LCai2GyRmAQt4x81KQwH+ya75J8u3kkagjq/Tnr84Gk4TiFxWM4FtcFt2jAb2xRDMUeu K/nuGwrXCshliUobKNKZLOq6OkY2lELAcbGx1V5ce+d7OQ4RjYObdtqvOb9hF+H49pPmdlx4zzN nF8MZQ1T3FFOUehl5LnErecXp3B+D1AEVPeyUS5tXIaoTgo9HtdlKWrYGBiS2lkcBemQSh93ZRS 5lRLTeig X-Google-Smtp-Source: AGHT+IFwyv6v50tn9W5bSyzXASzJW2tC+n3Yvr284DWZQwO0ZzTjXDMbHxo/pZeBv3MUI9+RJdh/7w== X-Received: by 2002:a17:902:cf0b:b0:215:431f:268a with SMTP id d9443c01a7336-21a83f647e1mr449439745ad.31.1736907501257; Tue, 14 Jan 2025 18:18:21 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:4043]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-21a9f254040sm72719235ad.232.2025.01.14.18.18.19 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 14 Jan 2025 18:18:20 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v5 6/7] mm: Make failslab, kfence, kmemleak aware of trylock mode Date: Tue, 14 Jan 2025 18:17:45 -0800 Message-Id: <20250115021746.34691-7-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250115021746.34691-1-alexei.starovoitov@gmail.com> References: <20250115021746.34691-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 6C5851C0013 X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: hbc8rp48koa76qexjqqqdpxssuaw381p X-HE-Tag: 1736907502-276048 X-HE-Meta: U2FsdGVkX1+jjXX1WhTWqah0vd1WuiRpF9ZeASINKCvhhWMLrAfmN7ebnMqWb1JeedQRQ5PUHAR8mjwF6iXwogcSnt//HAU5sPIRQHkoTAQvS8zrsmMYUBvn5CZHicIiPYxYNosHb3/fR0r/lFIr9VtpO+uhQrMMjLB4IzWCHZw7qBr9eCYs5PZ3zwY+NMojqcqMry2T1oEzQGIRVQo3/LzFQ+6+41Lr4To/hUSUVUK5+He/DoCC5D5khvm1Mhc3TOjlD2GXZIpvD027bHlAXmjNlGdMDKeKwkX3VkQjEm7r2JSCpTRYfY51UP17aD1v5cJu4igFDgOAiEaNcQ97Ix+SaSTf74xvuELkHZArIE7t0WFycK7uB5ifvlWZVOZ3gMYAlmut8lcjxOyLr9kLb3cPFoMknNkyInReHzjSqxIl/yqRzqB9IlGqX0t/7kxaTXBjxk7DezwWfg8HKcgG0Ydm28VhrPGwyhQhwqmLqyphofg9qLBohbyu2JwhWPniDgxxxTYF8j8OmKtBiMGMIUtJtPd/UQQ+CvhLD8JtVYfAZamNs0YMTaQZ6EqrizSW+kDd5S+94uKeiHPsqmnKjX6aUZGjScM+EZk3Y+xiGD9cxIDCZ6xW7Z8X1XSokfHJ0BxLH9ZWVhvaqyHiBoVkODmDgldk9oDUjR8zGGgOHtGXRLAoG4ALbZt0NwBJaZNQRRmz+sKIy9sM4soKrdVPhILmdbWGj8Pk2TDTeMoBsAECKcxcwV//h1fUOzUBaqDcCkiLv3zBJ77NfxD21v0VZTlbuiVrSDl3IXTp5Hm2LWX8zwekuTGyQF0EcFo1ONTue/kgRHw9xnvY1lSXSvNj9ScNmuRSipPFT+X6WUTwKJG0Kut/W/5ywWId1pcePEJ8MnD2w1aMvi7KmC1B31NBvhpnMweL8Q1KrWxoZwzfIWhp+vqLDxZ4OtUOxzPrjf0ABKIXqEzBC3qPZMUfaTU kLc9CSDE AGWMBg7bGkjVnmXoEOAe7s5VG/vWzf7+d6Gjff498ZX9ryJ+c+DcmgkZYsWcwrZn0614/iJ5uXw74LHPzb0yo7eq0l0Y67pLEk9x7J5kuF8AfRTWTpjzqthVREOAz26aAmx0mi0PwEfcXnsZX54HK3vmD/Pbs8xIKjvSibE38EjUW6SSMBgGxvrv0Bmwbx+U6HwpRgrIDof+i7DMcXKaDK8/d7ESgwElD3PwWnbVemZDVsgDs+xP3yxqJtjRcQHxuNs3JxMVLOZKfpIzAk/65ZaQPvaJbX4y5pYyQf/Ik8G0T/pSNUFZDx8SMTtV9SKTgj/W6ZF8x2g0eRETe1D4B6Qq1YI9BN1WprFd2R9V5b1UFRbflYwW5tBKT9cOdCYrJVwEYE3GfvWVAMUz8ZqGs+dcF93/pXFv9WRMpe/fGz3o1MSjL1cqT2Js7CPCY83nJl3ONIIcuhh00iZM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov When gfpflags_allow_spinning() == false spin_locks cannot be taken. Make failslab, kfence, kmemleak compliant. Signed-off-by: Alexei Starovoitov --- mm/failslab.c | 3 +++ mm/kfence/core.c | 4 ++++ mm/kmemleak.c | 3 +++ 3 files changed, 10 insertions(+) diff --git a/mm/failslab.c b/mm/failslab.c index c3901b136498..86c7304ef25a 100644 --- a/mm/failslab.c +++ b/mm/failslab.c @@ -27,6 +27,9 @@ int should_failslab(struct kmem_cache *s, gfp_t gfpflags) if (gfpflags & __GFP_NOFAIL) return 0; + if (!gfpflags_allow_spinning(gfpflags)) + return 0; + if (failslab.ignore_gfp_reclaim && (gfpflags & __GFP_DIRECT_RECLAIM)) return 0; diff --git a/mm/kfence/core.c b/mm/kfence/core.c index 67fc321db79b..e5f2d63f3220 100644 --- a/mm/kfence/core.c +++ b/mm/kfence/core.c @@ -1096,6 +1096,10 @@ void *__kfence_alloc(struct kmem_cache *s, size_t size, gfp_t flags) if (s->flags & SLAB_SKIP_KFENCE) return NULL; + /* Bailout, since kfence_guarded_alloc() needs to take a lock */ + if (!gfpflags_allow_spinning(flags)) + return NULL; + allocation_gate = atomic_inc_return(&kfence_allocation_gate); if (allocation_gate > 1) return NULL; diff --git a/mm/kmemleak.c b/mm/kmemleak.c index 2a945c07ae99..64cb44948e9e 100644 --- a/mm/kmemleak.c +++ b/mm/kmemleak.c @@ -648,6 +648,9 @@ static struct kmemleak_object *__alloc_object(gfp_t gfp) { struct kmemleak_object *object; + if (!gfpflags_allow_spinning(gfp)) + return NULL; + object = mem_pool_alloc(gfp); if (!object) { pr_warn("Cannot allocate a kmemleak_object structure\n"); From patchwork Wed Jan 15 02:17:46 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13939764 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF989C02185 for ; Wed, 15 Jan 2025 02:25:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 54AE7280003; Tue, 14 Jan 2025 21:25:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4F8F1280001; Tue, 14 Jan 2025 21:25:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3C23E280003; Tue, 14 Jan 2025 21:25:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 19971280001 for ; Tue, 14 Jan 2025 21:25:09 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id C2C631C80AE for ; Wed, 15 Jan 2025 02:25:08 +0000 (UTC) X-FDA: 83008093896.15.A475988 Received: from mail-oo1-f47.google.com (mail-oo1-f47.google.com [209.85.161.47]) by imf22.hostedemail.com (Postfix) with ESMTP id D7ACDC0006 for ; Wed, 15 Jan 2025 02:25:06 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=AnNzTgkN; spf=pass (imf22.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.161.47 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736907906; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6YPnFtJFai+0xFpNHgKwg1vEFPjmUMGue3EQsxvwhos=; b=kzB6MzX3olnIGm/H4F6FMgWU7bOYbBqbVnKZedG9JTTdGolppWmpZQ9li/fML8FiVI3Cip jd26UJbTmGXEMZI+g/nOlA2wm3+xtKQ5YCcYXCnffFULoJLQ4JcHpOYRZBfp6saOaYtQRa HXbCqq9SSdJ1PvO1E5SaZ9DJSK8t4SQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736907906; a=rsa-sha256; cv=none; b=3DOByXmTXBXWZtKkrHkSZlGmmxelEyyJCAp3MmGftUr4xoZlFq1jsAu+co391WgssYoNxl Gf4HRoTWC8kUl7W1JQaYa1KUcGPgYfmAfRt+0/t8Rdg3J6lik9xxzE6jt2bw1UwfFWbcgF SqdUVpyys4upIDUrQdHJSjWbJti25zU= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=AnNzTgkN; spf=pass (imf22.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.161.47 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-oo1-f47.google.com with SMTP id 006d021491bc7-5f88a93dceeso1385489eaf.2 for ; Tue, 14 Jan 2025 18:25:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1736907906; x=1737512706; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6YPnFtJFai+0xFpNHgKwg1vEFPjmUMGue3EQsxvwhos=; b=AnNzTgkNIFVhRGBOoxTArBMSs4+mbJBZX4lusi51JQMGe7Dxl+jl//YjFemG9/HuJm kBPXlv7wyfkn7o3ZN15fcZy6Ob9ezWV9nkxDAiEWNBw1C3ZdvXTr+iwZWA5sWPjVR1+x NiG7onoWdchYVyxO0xlxd9QKUMUZGPdhZnHYQ3BuZB4/AzWmtytTAghK06YOKuEGwDdT VDstocV6LCZ6kRXFn1EKRI5ZEKM6QiuHTgRQISNK+VrJKItSRUHich184vlgKVnAQ21i 6njz41QHahjvV033MN0Nm71ZquSwsguiYI+sf0xPe6JSPDNy5tfAApJ9b8t+412gsAhO JnAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736907906; x=1737512706; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6YPnFtJFai+0xFpNHgKwg1vEFPjmUMGue3EQsxvwhos=; b=wGNvVzsSkeWwzG31+UsXxW0R3SLlQrr0QShjFlVtIvlFIC2FlcNAIigmZ4vAsYPjbP 6RjnyWuV7xqb6nDHaNuOpX4+mnzLuVNT2Ozu2E5J0cubWWDDYi9LVUt0Ku/6e4R4LLJY DK+Txy2xUIA2NfmmaGUoHuavIFbIIBFDduVHNFSgxBe5W8Ok4i0ul5waD3SIlumBZjfh O4CB1CPly4MDtW8/MBsFFBu23FRnO/NKBYV5C9FMxxjBMyeGKAJ9pCPXSpoLecjfQ1lK t1Tei/yMyEPPr4ohCLpIkwWZoQVYZtb2XqTB6IM4WjgjGtKJGpjFIjNC/Q2NHGfkrZY7 +xIQ== X-Forwarded-Encrypted: i=1; AJvYcCVjYXiLcIPgnbQ7Y0+3Ab4ePYq6N7NbQeQXq+qeFUrSKV6m3XsE2ADGDFoimImjWfGKj/25NREcfA==@kvack.org X-Gm-Message-State: AOJu0Yy2Uad+aRgZbpcth8opa0RL1ypEc8mfCDvuECWhCDmX/T6EyPOU G4RX2TJC4sfDIS+42jHaXYjIi9iphzNiD28Ndc0r9qOLXDFAYw9OTEEQBQ== X-Gm-Gg: ASbGncsuFg7VLvjtX9u2uf9/KhNINBBYyTnJ1nnD4yR+m1DWBx3bIX1SE2thtHfZ4gN W8AIhfQDPzHnvG5vkivaW9LN/EzH8kXv1YhB7gMfVhS5p74jI/QG3HsZ50yqTP+D0oM5OqmTxk9 3RIo1mrJLvyQ9msPnUmf0XOWynoC71S2uiqJuARwYgU4SA+xYv8kRdWvQzDzInBiFryHL0V8LVi EBYITcG0AJjeKxoncPseCA7J0UvkJZuW/Z/xNjoSGfWEH5Z73gtuQPr9J0gG549js3ViafWApOH ruY7qMY+ X-Google-Smtp-Source: AGHT+IHp8XYH01n8SuhXDCfKih1NoDPtrNmfatMAx4uMkp9I9uvhY2pzgjGZtQiUuw/WiH6P6RobmA== X-Received: by 2002:a05:6a21:789a:b0:1e1:9bea:659e with SMTP id adf61e73a8af0-1e88d3612e6mr50844644637.35.1736907505893; Tue, 14 Jan 2025 18:18:25 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:4043]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72d4056a5c3sm8349229b3a.62.2025.01.14.18.18.23 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 14 Jan 2025 18:18:25 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v5 7/7] bpf: Use try_alloc_pages() to allocate pages for bpf needs. Date: Tue, 14 Jan 2025 18:17:46 -0800 Message-Id: <20250115021746.34691-8-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250115021746.34691-1-alexei.starovoitov@gmail.com> References: <20250115021746.34691-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: D7ACDC0006 X-Stat-Signature: dpwetpo4z5ei961w6c87471fi4tabmqo X-Rspam-User: X-HE-Tag: 1736907906-166386 X-HE-Meta: U2FsdGVkX19q7vMQAkXSr5yFIGxBTp2fdIGi1T1j+9ZvIb4Hxs7qgrVQGZ/nXQVSWIWWloq7qOZ44oaMOI+SpPezLo5qj57x/ohqbh0MJpMhM5EsD9vHKL0K5Z+Pw/GDLgJHiD1CWYQIbF8GCZcyH386myuQ2PNomj2ZNq+VilYgUTWZQRD+EHndzEWGIVO6VVE1rolfKSSyXYhn6jyuaw6PA/2i5W703vVFUq8uEVxWvunV/D0/+92M32wjy8PJX1PdiiX98ASqa8Yjeu1/RbrTVnF7u2jnS3GvnZvm7jw5dU605cOMTLkkoFut7bBE+Gtjuq25wnpVxjVJv5BCw872Sp5YOmD1OIKfFZahTim6l0PPSyWM1uQXhZtsu+KdPIHxbA8zk/GiB6pDrNFro/qWwhkh4kSaswP9WIymmmgKeQCAtJCzurzS0N/NzEkuXwkpKMK0DJt3/uyk5CtR2+V0DlYIrw+jW7HB0zeIZiJ3wOSMiwj2S8XvL23w29QFppreKaTQSlImbi3qITqE3RNPz/a0y9KFXQwRlYwIFMSV6ShYqQhJy6DtlKcCHJBO2Z9Z9EzHxmi0OdDzJENGYits2b9tt6dwbduhID40U3Nfj7XCgkPAwbadFSD/Lo/GIJqrjc7wXtuS3Xne18df5gUYHGy0b4j6HMnUotstegMTtR3cmhDlRTHFTGn6hwrlc1O9HV8CjcIDqYt3npbcqLU1mUrXC4bMMus7wcWN2TdgXs1Vmz9knAyUmw6sn6tE/X1CCMxYDArMVHicrPwmK84hAdhhYBU5P8UPsgGP0RSl1HzlEyfIT4upZBI5tDbFJpqfEOc53Fj5kzAeATdIRQ+LI8l4zjbzwxlei+BVDotRig4Fh+bK1rH+avZEVKPQQwobcOQzAPTV0+Pvj53liMFTXJY+TLCtvglUVfJTI5sdlOq4UGbZ8bSrRiqJkVC3yehcy4qUH/o8t/ZGiRe AWNrwrYB WJiPJAaCNJ8gBI9mWvgyafWrPpeZpIoPIBJSrBFBvs1bVahLO0tlKgdjG0SSAsN+3ia6b4GmR4hl7RBr0yjnv90xqalagM9pkhrgGQgwbfrskzPy2IvUwazFklwFTdNusl2FJ0hq/+iBK7GIV52Gx+3vkBgnd0K6gSqekoqBq8pSmQl6S12nb/28Pcg6pdqcn20F8OKWm/ckEiJa30f/hOMfXPS0UTvIsSUt6ZEsvv8miy77pxmAa0Qs1friRJn0X1NVz2LWrY0miq2i8BVPh1jZdCLcOwK31gVruXrggS2hwecti0FG+7zs7FMCsDdZx/32EizlIDk/yB8hM3PwTfXdHz5wX4Sj6BoObTzjImOb9tGroS447mpZYr2haFyKAz/24L0h2sgNpp7+JDz9Wtqxiy1bPXUj2Gwynl6mTtvnj84tFUZHV5R98rIqKuy1boS5zgQErTaYAvHlFCOJZhZrJ8RQo/c/J3wCg0zQ07gbu0oTzxIbYKXj8OQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Use try_alloc_pages() and free_pages_nolock() Signed-off-by: Alexei Starovoitov --- kernel/bpf/syscall.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 0daf098e3207..8bcf48e31a5a 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -582,14 +582,14 @@ int bpf_map_alloc_pages(const struct bpf_map *map, gfp_t gfp, int nid, old_memcg = set_active_memcg(memcg); #endif for (i = 0; i < nr_pages; i++) { - pg = alloc_pages_node(nid, gfp | __GFP_ACCOUNT, 0); + pg = try_alloc_pages(nid, 0); if (pg) { pages[i] = pg; continue; } for (j = 0; j < i; j++) - __free_page(pages[j]); + free_pages_nolock(pages[j], 0); ret = -ENOMEM; break; }