From patchwork Tue Dec 10 02:39:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13900694 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AADF4E7717D for ; Tue, 10 Dec 2024 02:39:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3751E6B00DA; Mon, 9 Dec 2024 21:39:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 325796B00DC; Mon, 9 Dec 2024 21:39:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 19F996B00DD; Mon, 9 Dec 2024 21:39:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id EAE6C6B00DA for ; Mon, 9 Dec 2024 21:39:48 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 6B670AD764 for ; Tue, 10 Dec 2024 02:39:48 +0000 (UTC) X-FDA: 82877493216.25.FE3A25A Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by imf27.hostedemail.com (Postfix) with ESMTP id F268540002 for ; Tue, 10 Dec 2024 02:39:21 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=EJWrhLjb; spf=pass (imf27.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.216.43 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733798364; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8e35BB82VxfWbGWqr1nA/p1/ofzs8olLXSCBkO0zstU=; b=OMFP/iW3ocdfwDeWJWnbJ21fObIsbnGyRSZ3+xdn8xM4eeMQOIU5/TIOrK4MEbJ/pfOz7t PiEUTSLEo/XzFrFFkzyJ9y6vsf73wmyIWd4MOKGu4KBOxzffVRFuC09LZdr/FGSRePCK4p WRS4KvHQjRTIRrTYSTunlsF1ARqLcc0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733798364; a=rsa-sha256; cv=none; b=70+qOJNGadoR9B6bv3pU+BJELvYVslMh0uMdoCX4/nWvBr0ywxg7ql53zcY1uFuqzyk0jz NMxxqcBoFNGLIGVdfKJMAFXHKkugtKcxyZXXmY3sF7GdlFh+muS7qjVXM3+ub/w7a6YUZb hrzEFJ3njZ6fx2vCxKGXL41fZOilK7U= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=EJWrhLjb; spf=pass (imf27.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.216.43 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-2ef8c012913so1813775a91.3 for ; Mon, 09 Dec 2024 18:39:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1733798385; x=1734403185; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8e35BB82VxfWbGWqr1nA/p1/ofzs8olLXSCBkO0zstU=; b=EJWrhLjbDRaEUT1AGF+BMOAVnvpR82VWNs2il5WLo6pLkMiDxZcKalcw+86iwl+3ke kSmKGTVbL8zXdimp3Y3ukU9H7JaK910Y5EKrHtNG/Pp1vYGWmggC3iaVdkjHS0SfSBTa tu/lQIMML5N8xEHUnn1Y40QbyqiwEB7xqDCpeLE/GvMizqGjMDHikleGM5wovD7xUEFJ LmGH3O4KZkSPwZvDPYxpCebEjVVeYtnhVJKYeFLX++czr2vtokKJSbQlM0yVa7Y3eu6y uBTQHex4wXR0O5cFfTYtF/PzzXI5zaioZlngE4P8ac4kYRoTPgMYHd8UucNcx9QutDPX EeDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733798385; x=1734403185; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8e35BB82VxfWbGWqr1nA/p1/ofzs8olLXSCBkO0zstU=; b=LHwodUWSGCymWYRgHaz4+Q3Q7rpijLqU+vfA/ofwj6OUn+LrRzTv8t7F6NdLI9KbJt rnAbYsdZVvvxxAmwTNUztD+eXoyEa50XxKilPUZ2LZ2udhsptaLy7S56LzU7pO8qOoO6 2Vhm7ltdmxQhpSdFXNl8VMUX//ubl3BdSg25o9Cb/PepBb34d79Bm0aaHQ6jFW0Zx4dc NpXWgO5RrMBPECda4u/DDG3q4FaWB0G+xCSza5jatQMQBwpWVTBvfkqBIb2O9MJyeqOn bIQe3/FPnt+ilavvisge6m20pK9WL/CrETypBU2k78D+mzKqne1aXWYaJmVlxbOhvaT9 b0bw== X-Forwarded-Encrypted: i=1; AJvYcCXofgKcXJfoONuAVAh8uMxpgFW1xwjUQsEd3Qr0KJf3ybd/W/mofnv02lTa7rtG6byKSgN+M99AGg==@kvack.org X-Gm-Message-State: AOJu0YwgrHc2JYO4Rfm2PXV7rO4ERl1OgmcunHUAEvX2hEShttfdJtou VyKzNjvJ/aU93ZbDJ0GscYMIyJUWxP+asLfzD/Swhzn4oWmRMSKt X-Gm-Gg: ASbGnctHgOTmBWd/agK6bJhHRRAbBxqDa0x6dPT9+ystV8H6sI04vjZDrzTjnZedafd iC5hcuhSWvkySLQ2ELDhR7fThkar4zXD6QUmL1B4b+d3Be4jYZ1L5sbLc0poK+oYaDk9eKsawTT IlisWHHQHY3GN5zxoFsCg1YWIurzMgAKVt6N59Jc7dWwYlbF9s1LZInX2wF784AXD1ueosH5CEl BjQxSpXl5RXf8niowdxCtYo0YzvXTkJ4HW9cEITm1QaiRIRUUtFnadWFCoQ5hroVX2zoq3I6qwp DmlWqQ== X-Google-Smtp-Source: AGHT+IETlDQVakNHy9XhiE9Y3JSlfzGxx7wfgdM3m/HG+ucSQQHFTGfnOzVwILgjB0Zho8MlhPqG4w== X-Received: by 2002:a17:90b:2e43:b0:2ee:acb4:fecd with SMTP id 98e67ed59e1d1-2ef69e16bd4mr21878450a91.9.1733798385246; Mon, 09 Dec 2024 18:39:45 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:83b0]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2ef4600d2c4sm9544043a91.47.2024.12.09.18.39.43 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 09 Dec 2024 18:39:44 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v2 1/6] mm, bpf: Introduce __GFP_TRYLOCK for opportunistic page allocation Date: Mon, 9 Dec 2024 18:39:31 -0800 Message-Id: <20241210023936.46871-2-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241210023936.46871-1-alexei.starovoitov@gmail.com> References: <20241210023936.46871-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: F268540002 X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: 47stxg5eo487abj9qtahekirhpw6mbz9 X-HE-Tag: 1733798361-522893 X-HE-Meta: U2FsdGVkX18K8gnpXnUCuAHTct64CH97vCzDbMgfQSaBprHcT1yaPaedTuk7lcATFARqDEyfd6x0Qgp1WEQSqMNtq1zt9K+AdT60SzDQdT01AsZRv+hhB+FJMdXJy0lrW06m2NH0qnpm/EKthrpxM/UkqkTB+6r3FjiOyC6RVFVYYAUDYCFYiApotJPOHyDmtxajdgwPRX5NEDObCSjLWywV1hic1O6+8na57NdeMO3nsZNtfgV9AYdHEBDevtqAEk8S8/lomJ//VpIgNGvMvHF/UMT3rF/rgLz2NNQ6q416kHwZ+L1vpZmLdrWg2ibjj5XLa1n3tahhGM4quGc5Jg1ULEpY3+hLeaYobblWy7kcc4O+pBuvsdqwUEh8uQ6c8OrtoF/HBaLR8JrF/vcPWWS2u3UD066k3yNDWZ4ToqjeSL+RWaFCUEIMUNHmsAZ93Dj9HfNSbi0irv9X8dr3Sz0mbmrWEM9KsOG4nqfuHdJUOIsOTOiYbHXSuAQ1XIWsJRfXSWsLRiEWyI68hzmm6SNijgl3zpEBEi5HxCgyd0BTs5BJtH0SJSqoxMfEXv2MZ3AGz+iuXjpEouK45L1T9jsCUIs5h2N6OI7mqSZfnps6Tg9Tm6MaAnCnDbqWMlVP0oKxNv/xpreQwzOyYvJBDjnNysS9Deyyt3Ri7fFkt7j74o3cQHSkWyg/FaEzYNAQe92ByKDbNHZ6E40j4iGRjGgza3wB8bqR15XT88zJgs7Oz0kE1FAuti1n9XqFdWqnEo2eG52Brcj5tfh2zRJJpmE/Lke0HyTV2z2XfKDOK2SS2GFDK3j34/zKcsaDeYnsjWMCShD3pLgSB9xZBan14vYGM8p11eXbKa5Zr0Rr4QEeo6Su9dYF6BVZKDXyHMuAKwljcgPZ8EU3Y36zZJtwfr0yXqD9NpJKcyYSW9DfdXRsxzuHYvEk5J4nDX7QMhmqZ0ridAuC6bRoHLbXwft laOfb1Fy COIjZ+z9ID3VkdvpcLsgqP0K9iTXiEGDca/FELNfDRpgFiRO9GMMVSMAdmMu3yfqfigJKIt6MtapeEGqnO4cwz9NLHlQQCzO46w/VUCNdmXW478VlqwL6xYihIsE8hEQwB1dq7MyR4BaNu+O+XO6o28wIKotzYrxNQ2S+4CS94kDQhwAmmmCua213/6qDzPKqhoeqvRgpZuAMi2r9glYZ7cYzLqyttSSDq/ZcPB2Uq+KOXnsQE5uBmIC30g7oq9NJb+TMZOBbwNH+ICTz74hrKSV8/AZqSEhGrWb5rUDmRHCmxa6A/jN+mrt3fF5srt+IcWJ7LsxHzEGo/rMGX+CzrqEzjrE5cRtUIrfMzHuUwHvFoKtoltWm4t1Y7Pt159T89KApKnkApmTm0dEUdXrES7whBJxBetA0MGyDIarxmwLxcT3PrhSr/+U86/HFOhdRCb3FGdr+rCUF08qUIhHBY40F47/bMFJFWEwtHwIJrr2ZhURIExmPkj2gFWS7/JZ0JHAGxOkK+cqcJwyZoXkUMLod2fM1TBrs/DN+KMBG+Im16BXRkJssl3q12w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Tracing BPF programs execute from tracepoints and kprobes where running context is unknown, but they need to request additional memory. The prior workarounds were using pre-allocated memory and BPF specific freelists to satisfy such allocation requests. Instead, introduce __GFP_TRYLOCK flag that makes page allocator accessible from any context. It relies on percpu free list of pages that rmqueue_pcplist() should be able to pop the page from. If it fails (due to IRQ re-entrancy or list being empty) then try_alloc_pages() attempts to spin_trylock zone->lock and refill percpu freelist as normal. BPF program may execute with IRQs disabled and zone->lock is sleeping in RT, so trylock is the only option. In theory we can introduce percpu reentrance counter and increment it every time spin_lock_irqsave(&zone->lock, flags) is used, but we cannot rely on it. Even if this cpu is not in page_alloc path the spin_lock_irqsave() is not safe, since BPF prog might be called from tracepoint where preemption is disabled. So trylock only. Note, free_page and memcg are not taught about __GFP_TRYLOCK yet. The support comes in the next patches. This is a first step towards supporting BPF requirements in SLUB and getting rid of bpf_mem_alloc. That goal was discussed at LSFMM: https://lwn.net/Articles/974138/ Signed-off-by: Alexei Starovoitov --- include/linux/gfp.h | 25 +++++++++++++++++++++++++ include/linux/gfp_types.h | 3 +++ include/trace/events/mmflags.h | 1 + mm/fail_page_alloc.c | 6 ++++++ mm/internal.h | 1 + mm/page_alloc.c | 17 ++++++++++++++--- tools/perf/builtin-kmem.c | 1 + 7 files changed, 51 insertions(+), 3 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index b0fe9f62d15b..f68daa9c997b 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -347,6 +347,31 @@ static inline struct page *alloc_page_vma_noprof(gfp_t gfp, } #define alloc_page_vma(...) alloc_hooks(alloc_page_vma_noprof(__VA_ARGS__)) +static inline struct page *try_alloc_pages_noprof(int nid, unsigned int order) +{ + /* + * If spin_locks are not held and interrupts are enabled, use normal + * path. BPF progs run under rcu_read_lock(), so in PREEMPT_RT + * rcu_preempt_depth() will be >= 1 and will use trylock path. + */ + if (preemptible() && !rcu_preempt_depth()) + return alloc_pages_node_noprof(nid, + GFP_NOWAIT | __GFP_ZERO, + order); + /* + * Best effort allocation from percpu free list. + * If it's empty attempt to spin_trylock zone->lock. + * Do not specify __GFP_KSWAPD_RECLAIM to avoid wakeup_kswapd + * that may need to grab a lock. + * Do not specify __GFP_ACCOUNT to avoid local_lock. + * Do not warn either. + */ + return alloc_pages_node_noprof(nid, + __GFP_TRYLOCK | __GFP_NOWARN | __GFP_ZERO, + order); +} +#define try_alloc_pages(...) alloc_hooks(try_alloc_pages_noprof(__VA_ARGS__)) + extern unsigned long get_free_pages_noprof(gfp_t gfp_mask, unsigned int order); #define __get_free_pages(...) alloc_hooks(get_free_pages_noprof(__VA_ARGS__)) diff --git a/include/linux/gfp_types.h b/include/linux/gfp_types.h index 65db9349f905..72b385a7888d 100644 --- a/include/linux/gfp_types.h +++ b/include/linux/gfp_types.h @@ -48,6 +48,7 @@ enum { ___GFP_THISNODE_BIT, ___GFP_ACCOUNT_BIT, ___GFP_ZEROTAGS_BIT, + ___GFP_TRYLOCK_BIT, #ifdef CONFIG_KASAN_HW_TAGS ___GFP_SKIP_ZERO_BIT, ___GFP_SKIP_KASAN_BIT, @@ -86,6 +87,7 @@ enum { #define ___GFP_THISNODE BIT(___GFP_THISNODE_BIT) #define ___GFP_ACCOUNT BIT(___GFP_ACCOUNT_BIT) #define ___GFP_ZEROTAGS BIT(___GFP_ZEROTAGS_BIT) +#define ___GFP_TRYLOCK BIT(___GFP_TRYLOCK_BIT) #ifdef CONFIG_KASAN_HW_TAGS #define ___GFP_SKIP_ZERO BIT(___GFP_SKIP_ZERO_BIT) #define ___GFP_SKIP_KASAN BIT(___GFP_SKIP_KASAN_BIT) @@ -293,6 +295,7 @@ enum { #define __GFP_COMP ((__force gfp_t)___GFP_COMP) #define __GFP_ZERO ((__force gfp_t)___GFP_ZERO) #define __GFP_ZEROTAGS ((__force gfp_t)___GFP_ZEROTAGS) +#define __GFP_TRYLOCK ((__force gfp_t)___GFP_TRYLOCK) #define __GFP_SKIP_ZERO ((__force gfp_t)___GFP_SKIP_ZERO) #define __GFP_SKIP_KASAN ((__force gfp_t)___GFP_SKIP_KASAN) diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h index bb8a59c6caa2..592c93ee5f35 100644 --- a/include/trace/events/mmflags.h +++ b/include/trace/events/mmflags.h @@ -50,6 +50,7 @@ gfpflag_string(__GFP_RECLAIM), \ gfpflag_string(__GFP_DIRECT_RECLAIM), \ gfpflag_string(__GFP_KSWAPD_RECLAIM), \ + gfpflag_string(__GFP_TRYLOCK), \ gfpflag_string(__GFP_ZEROTAGS) #ifdef CONFIG_KASAN_HW_TAGS diff --git a/mm/fail_page_alloc.c b/mm/fail_page_alloc.c index 7647096170e9..b3b297d67909 100644 --- a/mm/fail_page_alloc.c +++ b/mm/fail_page_alloc.c @@ -31,6 +31,12 @@ bool should_fail_alloc_page(gfp_t gfp_mask, unsigned int order) return false; if (gfp_mask & __GFP_NOFAIL) return false; + if (gfp_mask & __GFP_TRYLOCK) + /* + * Internals of should_fail_ex() are not compatible + * with trylock concept. + */ + return false; if (fail_page_alloc.ignore_gfp_highmem && (gfp_mask & __GFP_HIGHMEM)) return false; if (fail_page_alloc.ignore_gfp_reclaim && diff --git a/mm/internal.h b/mm/internal.h index cb8d8e8e3ffa..c082b8fa1d71 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1175,6 +1175,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone, #endif #define ALLOC_HIGHATOMIC 0x200 /* Allows access to MIGRATE_HIGHATOMIC */ #define ALLOC_KSWAPD 0x800 /* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ +#define ALLOC_TRYLOCK 0x1000000 /* Only use spin_trylock in allocation path */ /* Flags that allow allocations below the min watermark. */ #define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 1cb4b8c8886d..d511e68903c6 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2304,7 +2304,11 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order, unsigned long flags; int i; - spin_lock_irqsave(&zone->lock, flags); + if (!spin_trylock_irqsave(&zone->lock, flags)) { + if (unlikely(alloc_flags & ALLOC_TRYLOCK)) + return 0; + spin_lock_irqsave(&zone->lock, flags); + } for (i = 0; i < count; ++i) { struct page *page = __rmqueue(zone, order, migratetype, alloc_flags); @@ -2904,7 +2908,11 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, do { page = NULL; - spin_lock_irqsave(&zone->lock, flags); + if (!spin_trylock_irqsave(&zone->lock, flags)){ + if (unlikely(alloc_flags & ALLOC_TRYLOCK)) + return NULL; + spin_lock_irqsave(&zone->lock, flags); + } if (alloc_flags & ALLOC_HIGHATOMIC) page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC); if (!page) { @@ -4001,6 +4009,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order) */ BUILD_BUG_ON(__GFP_HIGH != (__force gfp_t) ALLOC_MIN_RESERVE); BUILD_BUG_ON(__GFP_KSWAPD_RECLAIM != (__force gfp_t) ALLOC_KSWAPD); + BUILD_BUG_ON(__GFP_TRYLOCK != (__force gfp_t) ALLOC_TRYLOCK); /* * The caller may dip into page reserves a bit more if the caller @@ -4009,7 +4018,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order) * set both ALLOC_NON_BLOCK and ALLOC_MIN_RESERVE(__GFP_HIGH). */ alloc_flags |= (__force int) - (gfp_mask & (__GFP_HIGH | __GFP_KSWAPD_RECLAIM)); + (gfp_mask & (__GFP_HIGH | __GFP_KSWAPD_RECLAIM | __GFP_TRYLOCK)); if (!(gfp_mask & __GFP_DIRECT_RECLAIM)) { /* @@ -4509,6 +4518,8 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, might_alloc(gfp_mask); + *alloc_flags |= (__force int) (gfp_mask & __GFP_TRYLOCK); + if (should_fail_alloc_page(gfp_mask, order)) return false; diff --git a/tools/perf/builtin-kmem.c b/tools/perf/builtin-kmem.c index 4d8d94146f8d..1f7f4269fa10 100644 --- a/tools/perf/builtin-kmem.c +++ b/tools/perf/builtin-kmem.c @@ -682,6 +682,7 @@ static const struct { { "__GFP_RECLAIM", "R" }, { "__GFP_DIRECT_RECLAIM", "DR" }, { "__GFP_KSWAPD_RECLAIM", "KR" }, + { "__GFP_TRYLOCK", "TL" }, }; static size_t max_gfp_len; From patchwork Tue Dec 10 02:39:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13900695 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B39C8E7717D for ; Tue, 10 Dec 2024 02:39:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2B2B66B00DC; Mon, 9 Dec 2024 21:39:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 261706B00DE; Mon, 9 Dec 2024 21:39:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 103846B00DF; Mon, 9 Dec 2024 21:39:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D70BB6B00DC for ; Mon, 9 Dec 2024 21:39:53 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 8E5DFA0261 for ; Tue, 10 Dec 2024 02:39:53 +0000 (UTC) X-FDA: 82877493300.08.1E95C63 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by imf14.hostedemail.com (Postfix) with ESMTP id 205C810000D for ; Tue, 10 Dec 2024 02:39:27 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=F9zzBIpy; spf=pass (imf14.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733798381; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XSOnooig0JqcXQ1Y/KpafF9RbsF0QDm5Q4FOwHWSe4I=; b=wpif9X/YqGHqdvk4Re2+61NCilhSeZWifE5yXpmQcVBdbKc4AiPjG2ZNchsiNupM776AZ5 JqjmHctAnPJM1QlyQm0CA7eSqBxqat38RDf90rVE+bDcoqkEpvCpLHt5dl1NHh+GOsiVjz 21+8H7hQtV4XMSYSCGqEppnflYbDeyQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733798381; a=rsa-sha256; cv=none; b=uKipFw2U9qGBEONhAPrNwUflkCmYRJsVSCTU53A/xxXGBcU3pj8+cJpCAYUY043v4oEDWm 494nS97gO7Sx4t7uisiqWjYgukeerVLlYMjzGpH0ugyCCrV5jcxoboGYg7TRDi+TPBFUdf 3YH289e9Q9Uz431QEwuHGYq3mWiow2c= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=F9zzBIpy; spf=pass (imf14.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-2164b1f05caso16795025ad.3 for ; Mon, 09 Dec 2024 18:39:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1733798390; x=1734403190; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XSOnooig0JqcXQ1Y/KpafF9RbsF0QDm5Q4FOwHWSe4I=; b=F9zzBIpyT6ynBmU9Ez9kAJSHUEUM/ZucVwZwhsHLdBSOk661wQFP+OvPCBaw9g8U36 +CThigM7WfHtinJHhctRggeF4zzrqyuycLMb3GlPZpdZ1s/GMUTGclcJ7qySizoEF8DY DKZLHeDaaWJpNeS+nhGtno7jWK0CAfp1DvXL4NGBu3RSECYn3yde/xFyths7gUUs0zF/ Py6numfhD/9q9jM+i1IFuLF449b+lo4VyRqmNu55zDL0s9peP499Nx6Mzu3xEC/uhGcm ANySgDxvB22SXHDN8v/kXkdRFy+qBylqWj9jQTPpELhU2GdNIbXNrvKyFidMuzWxt8de H6Kw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733798390; x=1734403190; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XSOnooig0JqcXQ1Y/KpafF9RbsF0QDm5Q4FOwHWSe4I=; b=AZKMI9jmODAp7Of99/X+7SJXLIS1OQRa5t1q8sJbAGu9MG4BhBCkv/balpOoe4xGb1 YW/e41hamy90HI5OcjfRuzWzjwMF/qFZFzwnJRGYbte843j0a1CnnDHcRIws6iYvmeHF H5t0SpXQC11AWYKBqod81epofINoWvxWvf4J2eFA9Qze1EDQGcL+oZhL1pkUBX1X9p3l 2jjMv5RRPw3nIQ4h4hL9jrjqMQRsftT6FKTKiDs9aqi1htktD2T3YP06bDx8AA+9DLDP g+cxtDhpna7QXhy+udjL/7FRPNTMEk0ggHUB8meljKqWa9QPP/aAqdAH+t+4p0i4vW7w IGwQ== X-Forwarded-Encrypted: i=1; AJvYcCUZSSeQ0/gG1pjKkqsz1wGQ9jiLCZYxg6kOknSuZnTwmAqq5mSWj3+rFf8Efe1o0pMVWUSnJC+juA==@kvack.org X-Gm-Message-State: AOJu0YykycIthlA6EB5e4FdWOIF4l/V+zL5fQGRdTj/lwP+HISAVeNoY T1KIaEQlclzRHmAZhMQu1jdGRWuioGRYtIq0Hl3vlfHIwalYBSA+ X-Gm-Gg: ASbGncvzM1nlm7DDNCADgobV9fy1XovodquFSmGHGtgIkIxpepTcJu7AiLZ8ElOlHq8 kPpe1C9awWVshQ2QiFG2hhWnh72ZNFpwwOXMUZ2o5TL4b2xvII+xkktBfiAQcfyVnY/oM+Qkn6a IQlaFZjFB2pSnnqX51GeTJ71pGyQJ1nBEH0OfOIHP6f+xvGjbgh+hMUA3cwNx+XR4QE80hC6S4j rhm3VPy9vbc2S0hllmVPF89bTBlA/G7FcnADU6YsRVkL12yHD0OLZs/TXVRS1k+N9D5WrQk2Ab3 Bh/5SA== X-Google-Smtp-Source: AGHT+IFs9AQy6lVySBTzfSVqJTRJXQoQU6RtOKdvZs51As+Ukwzy42oozyaRiTjoeSpSVQBsyUhvdA== X-Received: by 2002:a17:903:110f:b0:212:996:3536 with SMTP id d9443c01a7336-21614d2e719mr238769345ad.10.1733798389764; Mon, 09 Dec 2024 18:39:49 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:83b0]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd45b78c7fsm2837312a12.15.2024.12.09.18.39.47 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 09 Dec 2024 18:39:49 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v2 2/6] mm, bpf: Introduce free_pages_nolock() Date: Mon, 9 Dec 2024 18:39:32 -0800 Message-Id: <20241210023936.46871-3-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241210023936.46871-1-alexei.starovoitov@gmail.com> References: <20241210023936.46871-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 205C810000D X-Stat-Signature: q7ksjrjpg45fsz4tjhhharm7kyjxzr9z X-Rspam-User: X-HE-Tag: 1733798367-546405 X-HE-Meta: U2FsdGVkX19Nhb7E4SGryyK56YiZdF8NDitcAr/5agKR1JsAdxmzf6VZ6qjYVc9NIcnGAsHX+ZpSKYv4TPPyDOxDNPSp23YP2CJol6EKHFfg/OMPPr/1gEDxu19UsYqgdUpnm3EtOrsYuJoluZqyUn0UYDdpATkw6brEUnUZORIpOq9S+aWLTv7qJX1YFhW1633D33aR+YWOA4gO3DoB+xm/KrxgI4qu4vmf6LNBnHWGfAXQE3l9mTthw0YdfNrLgE5xk2CWXZhWnksO0PUxQc2rO1njIRQ/Xz2JvUF8SAasmwoqreXeBVUFbbMYeTDG8QiehMunA1cNuJNWzW5LSW5NBHVa7+YOw6HcId6XR2RPMM4u4LzAHhk++yQzWae/sVVYn1SSD5cqV95EVVzpxPNotxtJ35wyepNnmd6fjObdPw6gMeke2EEyTHOs4hXGDM0PTctZZ+CLfnKuHA3IAqnHcTQm/2FSPNlAJvO0D1Z80nsfbZhkZfqbuTQXSbq851tgXR+7IMOYhg91AXeRoQ1VSKxRMhuXuU+B4R/EPRAXM78aDr9ra7Ig4nbYfHBlqjS+z0anyTkikVh7rDaBy1wu3zSAao/l5XsnrYGSxaXmikp8nisIhAEh98/gymvlwVDxZ/xR+Fiawl4vE8KKcJaRxs0nlf/bObiFcycQhjzrZ2Kspw98+ZuvT5ZzgS++IcS5pEgJOQEmCHfjEsEHUUdSAu9xeG00PmUI/UIj7bx65ruBoqO7gpOf2BQzpir97z0JbU2vzwPuUr2JbiVEqiorzsySxgeTfXtLrldUh+FqKnHTgToT/J5cUkhicmZGOSPY2T154DQTYN2DKwtTOSr6jezLDvFz2KEUrG/MVJQrUrp+o19Zi0gZ342XMZco+W5zvk3jmPN+sKK/vP+9g91QCXqjaMzxNXPtVO8f6nDTSfexVVkJdFQ4DY/9BQTrNP3UPNfFZxycPppb0DM 2nj6h+Vb bfpMEQA0rEF9P15ucDfaQH6aHQxqo1xHjeY7XVR3uSLQ7QlU12sByMa2XAhKp2MIeats3Sq66CFLM3uYd/w3jr7S+SNmkB8EGvRWbz6Zb6O3OhociKxW6EGxWS5wrtlffZYYl3Cb/b1YJURMAUGH7iowVsj0DVLcJnvG98EFxojMxHUaLh1Ytc6t3sWI3BcDtcpcci0F6GMNkjXFF0OjTjREAG//U7h9RvBSr2I+rz4cOrTcWipEcqH6arFyhLaKB6V6yYs3cK8fe8RBzev2vI4FUCzPHt24NbkB/NfX18ZK8Zqv7cvIFMnas27Y6hoRWmLlR2yPaa3hRRfiRX3IP6mTYDo2niDYFr/vT21onPV7lGyk4YbSYHw1OMCJou0G3a74HT2v7PmLaudhZST1Q1cLmnsvbbFOGD1sADQI65Q6BAXKsRg9Gyb4d1/BfEJtJaMvlKxpqEMPsbrm2w/yHyU02UpO6rnceBQ2TXfnDUmiaiAzScSvx4P6juA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000483, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Introduce free_pages_nolock() that can free a page without taking locks. It relies on trylock only and can be called from any context. Signed-off-by: Alexei Starovoitov --- include/linux/gfp.h | 1 + include/linux/mm_types.h | 4 +++ include/linux/mmzone.h | 3 ++ mm/page_alloc.c | 72 +++++++++++++++++++++++++++++++++++----- 4 files changed, 72 insertions(+), 8 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index f68daa9c997b..dcae733ed006 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -394,6 +394,7 @@ __meminit void *alloc_pages_exact_nid_noprof(int nid, size_t size, gfp_t gfp_mas __get_free_pages((gfp_mask) | GFP_DMA, (order)) extern void __free_pages(struct page *page, unsigned int order); +extern void free_pages_nolock(struct page *page, unsigned int order); extern void free_pages(unsigned long addr, unsigned int order); #define __free_page(page) __free_pages((page), 0) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 7361a8f3ab68..52547b3e5fd8 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -99,6 +99,10 @@ struct page { /* Or, free page */ struct list_head buddy_list; struct list_head pcp_list; + struct { + struct llist_node pcp_llist; + unsigned int order; + }; }; /* See page-flags.h for PAGE_MAPPING_FLAGS */ struct address_space *mapping; diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index b36124145a16..1a854e0a9e3b 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -953,6 +953,9 @@ struct zone { /* Primarily protects free_area */ spinlock_t lock; + /* Pages to be freed when next trylock succeeds */ + struct llist_head trylock_free_pages; + /* Write-intensive fields used by compaction and vmstats. */ CACHELINE_PADDING(_pad2_); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index d511e68903c6..a969a62ec0c3 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -88,6 +88,9 @@ typedef int __bitwise fpi_t; */ #define FPI_TO_TAIL ((__force fpi_t)BIT(1)) +/* Free the page without taking locks. Rely on trylock only. */ +#define FPI_TRYLOCK ((__force fpi_t)BIT(2)) + /* prevent >1 _updater_ of zone percpu pageset ->high and ->batch fields */ static DEFINE_MUTEX(pcp_batch_high_lock); #define MIN_PERCPU_PAGELIST_HIGH_FRACTION (8) @@ -1251,9 +1254,33 @@ static void free_one_page(struct zone *zone, struct page *page, unsigned long pfn, unsigned int order, fpi_t fpi_flags) { + struct llist_head *llhead; unsigned long flags; - spin_lock_irqsave(&zone->lock, flags); + if (!spin_trylock_irqsave(&zone->lock, flags)) { + if (unlikely(fpi_flags & FPI_TRYLOCK)) { + /* Remember the order */ + page->order = order; + /* Add the page to the free list */ + llist_add(&page->pcp_llist, &zone->trylock_free_pages); + return; + } + spin_lock_irqsave(&zone->lock, flags); + } + + /* The lock succeeded. Process deferred pages. */ + llhead = &zone->trylock_free_pages; + if (unlikely(!llist_empty(llhead))) { + struct llist_node *llnode; + struct page *p, *tmp; + + llnode = llist_del_all(llhead); + llist_for_each_entry_safe(p, tmp, llnode, pcp_llist) { + unsigned int p_order = p->order; + split_large_buddy(zone, p, page_to_pfn(p), p_order, fpi_flags); + __count_vm_events(PGFREE, 1 << p_order); + } + } split_large_buddy(zone, page, pfn, order, fpi_flags); spin_unlock_irqrestore(&zone->lock, flags); @@ -2596,7 +2623,7 @@ static int nr_pcp_high(struct per_cpu_pages *pcp, struct zone *zone, static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, struct page *page, int migratetype, - unsigned int order) + unsigned int order, fpi_t fpi_flags) { int high, batch; int pindex; @@ -2631,6 +2658,14 @@ static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, } if (pcp->free_count < (batch << CONFIG_PCP_BATCH_SCALE_MAX)) pcp->free_count += (1 << order); + + if (unlikely(fpi_flags & FPI_TRYLOCK)) { + /* + * Do not attempt to take a zone lock. Let pcp->count get + * over high mark temporarily. + */ + return; + } high = nr_pcp_high(pcp, zone, batch, free_high); if (pcp->count >= high) { free_pcppages_bulk(zone, nr_pcp_free(pcp, batch, high, free_high), @@ -2645,7 +2680,8 @@ static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, /* * Free a pcp page */ -void free_unref_page(struct page *page, unsigned int order) +static void __free_unref_page(struct page *page, unsigned int order, + fpi_t fpi_flags) { unsigned long __maybe_unused UP_flags; struct per_cpu_pages *pcp; @@ -2654,7 +2690,7 @@ void free_unref_page(struct page *page, unsigned int order) int migratetype; if (!pcp_allowed_order(order)) { - __free_pages_ok(page, order, FPI_NONE); + __free_pages_ok(page, order, fpi_flags); return; } @@ -2671,7 +2707,7 @@ void free_unref_page(struct page *page, unsigned int order) migratetype = get_pfnblock_migratetype(page, pfn); if (unlikely(migratetype >= MIGRATE_PCPTYPES)) { if (unlikely(is_migrate_isolate(migratetype))) { - free_one_page(page_zone(page), page, pfn, order, FPI_NONE); + free_one_page(page_zone(page), page, pfn, order, fpi_flags); return; } migratetype = MIGRATE_MOVABLE; @@ -2681,14 +2717,19 @@ void free_unref_page(struct page *page, unsigned int order) pcp_trylock_prepare(UP_flags); pcp = pcp_spin_trylock(zone->per_cpu_pageset); if (pcp) { - free_unref_page_commit(zone, pcp, page, migratetype, order); + free_unref_page_commit(zone, pcp, page, migratetype, order, fpi_flags); pcp_spin_unlock(pcp); } else { - free_one_page(zone, page, pfn, order, FPI_NONE); + free_one_page(zone, page, pfn, order, fpi_flags); } pcp_trylock_finish(UP_flags); } +void free_unref_page(struct page *page, unsigned int order) +{ + __free_unref_page(page, order, FPI_NONE); +} + /* * Free a batch of folios */ @@ -2777,7 +2818,7 @@ void free_unref_folios(struct folio_batch *folios) trace_mm_page_free_batched(&folio->page); free_unref_page_commit(zone, pcp, &folio->page, migratetype, - order); + order, FPI_NONE); } if (pcp) { @@ -4855,6 +4896,21 @@ void __free_pages(struct page *page, unsigned int order) } EXPORT_SYMBOL(__free_pages); +/* Can be called while holding raw_spin_lock or from IRQ. RCU must be watching. */ +void free_pages_nolock(struct page *page, unsigned int order) +{ + int head = PageHead(page); + struct alloc_tag *tag = pgalloc_tag_get(page); + + if (put_page_testzero(page)) { + __free_unref_page(page, order, FPI_TRYLOCK); + } else if (!head) { + pgalloc_tag_sub_pages(tag, (1 << order) - 1); + while (order-- > 0) + __free_unref_page(page + (1 << order), order, FPI_TRYLOCK); + } +} + void free_pages(unsigned long addr, unsigned int order) { if (addr != 0) { From patchwork Tue Dec 10 02:39:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13900696 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95F47E7717D for ; Tue, 10 Dec 2024 02:39:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1EFDB6B00DE; Mon, 9 Dec 2024 21:39:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 19F2A6B00E0; Mon, 9 Dec 2024 21:39:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F34216B00E1; Mon, 9 Dec 2024 21:39:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D4BD36B00DE for ; Mon, 9 Dec 2024 21:39:57 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 87775140213 for ; Tue, 10 Dec 2024 02:39:57 +0000 (UTC) X-FDA: 82877493636.13.6D369CF Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf08.hostedemail.com (Postfix) with ESMTP id 7B29E160008 for ; Tue, 10 Dec 2024 02:39:41 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=gnKNNsH2; spf=pass (imf08.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733798386; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MT/DPo97yF9oIvPtZgU314ePao/rByOm5QF7hTFkWYs=; b=ZGHf5fv2i/JBsYoNPjWZCTTbNWzxuCL+OWXt5rowtkG/6I2+h2PKl4mFoyDkRUBYmQnpLT ojYN2yDRiFlKwrUrS2jIC1oxcdJVEjrqXbfIzojw+FeF/aoOcevwGNdBL8z1T/9KC8cH1e T/PbHSDHuw9Wg1Kf8lKHSgn7Q2Nsr7I= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=gnKNNsH2; spf=pass (imf08.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733798386; a=rsa-sha256; cv=none; b=09Uavi999/beXjMJOVDIriXYK4a5xU2IEBVrIzfiVW7spSt6dw/YqZCkGXcSST95ug+1tO POkjHoDM9sbBxvj4NbgGcTcHJl2be5A2peZsaXyk2f2cRBTv2ge4BLJOkDTftwwBP+YbVF hPcERxpQ+9d+CTd3YQgPbRmHXx5p+UE= Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-216426b0865so16316805ad.0 for ; Mon, 09 Dec 2024 18:39:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1733798394; x=1734403194; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MT/DPo97yF9oIvPtZgU314ePao/rByOm5QF7hTFkWYs=; b=gnKNNsH2feIM9aqysUokJ9gPzyzb5q4xl6bom1UaD7obxp4N29nYnRYPs6LNuqnLhY imcfcTFdhev07ffXtNIS6wBf3KfyIb61QgmKEOv3Rd0CH2tzXMETovIAY5pSOrEtGD62 uSIdAKjFQHIh/85ycVr2o+JpYhuDNn5acz7tBej5509uQtk+S4ywWWWtnwdYFho0F+ZY qqRoU1Jn4lqmOqjzmJO7yHoSAK9FNS+go/ZyYDsOTgFvkaSLYAGcDWwDZsnMYPtfdf5Z 6UzUkKwYX/8H8hvchmF2nYkRgTZsXoCLWI7PmJOxG5b07x8lsivVcANn8Fx+J8dBUgwb fLbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733798394; x=1734403194; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MT/DPo97yF9oIvPtZgU314ePao/rByOm5QF7hTFkWYs=; b=JvLI212TtDvkz+fgxLSXNMfFiPjTfnWRd94+LqEJ31zSbk9qo0YSZdQ+QUnhyr8Txx NdVaG+QYr3nMWaHRroHM5tTFc4suEhzSHWiUW/3AavcxB5BHmk2mow9EReZrYhRa29Cu ZmvHhX3GduOfwAhRFEOVr/NRwWZyKRLVWN/VbEmJkQmZT7I1/f8iMHyNyJzf4Wm9YPro JNyiEG0F+LHro5Zwe3WPOctM+GxxOrpiX8umpo7DDyYORmVWnwBQeHlO4SNAY8oyQAhh e4gkuzU2UQ72cmHorlim3d2gRGs6fudUjHG79XO+0wkg32EbXJIDqH2ZOA1S18U5w8As tevQ== X-Forwarded-Encrypted: i=1; AJvYcCUKpumESqJZJHg0Exh/PHG/DMosLTkyKItbISJIzkFKusy4D+FdhkIC0NyNdmyksXrm7YuPpKm4Wg==@kvack.org X-Gm-Message-State: AOJu0YwZOpq+G8c2Qqij8PnoU3vT/5Mq0Q7TlTik5GKFC0/vVFLAmvjB cv9GVUSwOHX4lEsNtYhLKaNnkfVqg4EYEmH3wLdZNLC2gHsbmu8K X-Gm-Gg: ASbGnct8MWesd5uP5W4iIObN4BL6oMyif29BshrEhOhSXr4ElOmnGIT6aO3lfXzpT9N x9XdPc+5QmM9RjKbVaylQ7xSrO6WOcurirFfqrOctDQXj+/tSj4uCZt3NAIj9MfNfsj0DDKMaCZ zxIC5SLUnAq4t+ENFvWoDnksPXIRinNHB6ATF+NnhYhoCxKpnhaTAjGHUuQEwHiDDjbUdMu6OGp SlVbHyfCL/eoW9Cz3u3HrkWyukppSnbsBcz2WJq2d6xMlauiJYiyEf9Q2UiMbicIYPC7WNJh8Pk I0fOvQ== X-Google-Smtp-Source: AGHT+IGKLmePoGcXe+HAKRjlAgf3YSGz9J3cPaKQKdSbicAELPOmItuHpzzqliawMg0yEsYXj3ExxQ== X-Received: by 2002:a17:902:ecc5:b0:215:4fbf:11da with SMTP id d9443c01a7336-21669fc28d6mr43610845ad.21.1733798394490; Mon, 09 Dec 2024 18:39:54 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:83b0]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-215f8e5f031sm79221425ad.82.2024.12.09.18.39.52 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 09 Dec 2024 18:39:54 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v2 3/6] locking/local_lock: Introduce local_trylock_irqsave() Date: Mon, 9 Dec 2024 18:39:33 -0800 Message-Id: <20241210023936.46871-4-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241210023936.46871-1-alexei.starovoitov@gmail.com> References: <20241210023936.46871-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 7B29E160008 X-Stat-Signature: 59d4aup5s7zicbmrua7y81wz6c7sy4uw X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1733798381-999040 X-HE-Meta: U2FsdGVkX18vz2qQeWW00DnBAGpksClVaxq1Bwz7a9wYnQdGyia2PFKg98pq/qriqQz9+xbk8UiSY/p7MEFmHplgX+LknPpVv7i2dlAri6kql/q9koaiPMJ1axY7MB67ZNUaO3783FG/f+xcpXa85CX4YmPJRQyspJQwLQtC9BYM85rCVjAQtdKYAEn8eq1x3BmXcW0wqZ/7fQmRpJpL01WxAEQt3Fyc02CKz+RfxPxfJx28qrFddtN/hdM875Mx8z1zroYNlcTM1yVp47O7DnSLDLPrrEnfmHh6FHgNN/2tZPfRPsGvDxFoYtJS2lQEd0DLD0h6623nYkON4ZgNebEHLkL98Iq555oD6dvQff7nux499bFGa3364BNC2WSkDFD3esfiS5UHW2I0swMgU9f4T9eYoaFBVzWdfzecjoukmwl+3mgC+pyrftid7NxXVS+Dw7DrHZyIeY4Jc92oQbGWnnvrzKgCNsrFMwoeFW3Gk9aYvzMV8d3lJWkl4LEs8u/CL3GthDhvUTFWrZEzsWm26cCQwe4fO9DTyNzyVqB+UqyEE/KL7P2SWNJLTlMI+iNsK9Fl9BGA/16dlT2PeofaSZwJTbH0k/DKPe4qH5LkMfW/rKneXI2TxVJPLjyooxKSriXXhskbntPyNfAiSRHQAjfF8pNg0elKE8CWoxzMMnn5j3/gMUWVhGfap3V2afZoRaaRAg8OteoVIuiw7uyxwlUIccsJdPenvKsX4Y83sElKJU5+BbXa9bdSgAQF4+hZw3HTzrilWPo/FMngv+PS7F2c0Nlq+5Upc61Z2+XrJ71vndGxAxcsdptibiSe2ZmTh7UeLWK7Q2M0iD8s9WmuKmX+HIjJi6CKAWhodhp1zGOyvVeN4HukUd2J4eRiywkjjswhP+CYoFSfPU3WZVQWe/XbwD8D3kIGtfm6awDlpr+FkepOkyjtUlDx+7B1Fe+iTi2NEr4/1rHer24 JCO4id3j /VZyRanRdpy/8rhlAQx94E3M3nPKV0srfJApZLhs3BMPBUcZhLmPKqZ2jvjA/nMGFqQIjTzsZEMMlze/hhpNJ8dqxI1xWW7rH7Su6dVAIt8MsekbtgvqOfCwRzKk0WO5zcxDw+J3uk3Q9FgqoZH4syjg/adS+5lljrDTEiyfO7aBGkH5l07bL/dm0dl0kY/a9M/1fg1AfvXUDf4uY51KEWJfVKXRO6xkwdtFTTDJsHd9d8eF9WiBXNpHZUn+ytKxN0wN9g3oRG7vKB/mJDy7r9YhlFW9Y0hC+5WSuB0Om0HNWR40D9bjVI8S3+oNDx0PywcT7IWkPr9CwRiRRVlSKr+7u2dXpP+2F/IgXq6wz0roohxGLUZ8pxi0+UkF2eGJzrsli66FVGkmighpQV6JEQSaDc9VEc1eTiKTT2vfkjOonrX4MMm2TSLbUqimsS8zLdCjLiAr7lPVK9yzaqcAOMsioKnFe1VI9sTPAnFBNihfi52POl7fkbY+A/A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.064903, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Similar to local_lock_irqsave() introduce local_trylock_irqsave(). It uses spin_trylock in PREEMPT_RT and always succeeds when !RT. Signed-off-by: Alexei Starovoitov --- include/linux/local_lock.h | 9 +++++++++ include/linux/local_lock_internal.h | 23 +++++++++++++++++++++++ 2 files changed, 32 insertions(+) diff --git a/include/linux/local_lock.h b/include/linux/local_lock.h index 091dc0b6bdfb..6880c29b8b98 100644 --- a/include/linux/local_lock.h +++ b/include/linux/local_lock.h @@ -30,6 +30,15 @@ #define local_lock_irqsave(lock, flags) \ __local_lock_irqsave(lock, flags) +/** + * local_trylock_irqsave - Try to acquire a per CPU local lock, save and disable + * interrupts. Always succeeds in !PREEMPT_RT. + * @lock: The lock variable + * @flags: Storage for interrupt flags + */ +#define local_trylock_irqsave(lock, flags) \ + __local_trylock_irqsave(lock, flags) + /** * local_unlock - Release a per CPU local lock * @lock: The lock variable diff --git a/include/linux/local_lock_internal.h b/include/linux/local_lock_internal.h index 8dd71fbbb6d2..2c0f8a49c2d0 100644 --- a/include/linux/local_lock_internal.h +++ b/include/linux/local_lock_internal.h @@ -31,6 +31,13 @@ static inline void local_lock_acquire(local_lock_t *l) l->owner = current; } +static inline void local_trylock_acquire(local_lock_t *l) +{ + lock_map_acquire_try(&l->dep_map); + DEBUG_LOCKS_WARN_ON(l->owner); + l->owner = current; +} + static inline void local_lock_release(local_lock_t *l) { DEBUG_LOCKS_WARN_ON(l->owner != current); @@ -45,6 +52,7 @@ static inline void local_lock_debug_init(local_lock_t *l) #else /* CONFIG_DEBUG_LOCK_ALLOC */ # define LOCAL_LOCK_DEBUG_INIT(lockname) static inline void local_lock_acquire(local_lock_t *l) { } +static inline void local_trylock_acquire(local_lock_t *l) { } static inline void local_lock_release(local_lock_t *l) { } static inline void local_lock_debug_init(local_lock_t *l) { } #endif /* !CONFIG_DEBUG_LOCK_ALLOC */ @@ -91,6 +99,13 @@ do { \ local_lock_acquire(this_cpu_ptr(lock)); \ } while (0) +#define __local_trylock_irqsave(lock, flags) \ + ({ \ + local_irq_save(flags); \ + local_trylock_acquire(this_cpu_ptr(lock)); \ + 1; \ + }) + #define __local_unlock(lock) \ do { \ local_lock_release(this_cpu_ptr(lock)); \ @@ -148,6 +163,14 @@ typedef spinlock_t local_lock_t; __local_lock(lock); \ } while (0) +#define __local_trylock_irqsave(lock, flags) \ + ({ \ + typecheck(unsigned long, flags); \ + flags = 0; \ + migrate_disable(); \ + spin_trylock(this_cpu_ptr((__lock))); \ + }) + #define __local_unlock(__lock) \ do { \ spin_unlock(this_cpu_ptr((__lock))); \ From patchwork Tue Dec 10 02:39:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13900697 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC68CE7717D for ; Tue, 10 Dec 2024 02:40:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 39B9B6B00E0; Mon, 9 Dec 2024 21:40:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 325BD6B00E2; Mon, 9 Dec 2024 21:40:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1024A6B00E3; Mon, 9 Dec 2024 21:40:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id DBD466B00E0 for ; Mon, 9 Dec 2024 21:40:02 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 68D5B80210 for ; Tue, 10 Dec 2024 02:40:02 +0000 (UTC) X-FDA: 82877493804.25.56D5A48 Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) by imf02.hostedemail.com (Postfix) with ESMTP id C999780008 for ; Tue, 10 Dec 2024 02:39:19 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Wu0mFc+6; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf02.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.215.177 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733798380; a=rsa-sha256; cv=none; b=mAI6PyowayzJ/2uOwRblKX65tcdB9A7+i2vp/6kLyXZ+3ZMxiAxmcik7WAzC88QG4ATMeC D2HhXAF+tRGjbskfEz0PHFH7dchlUJRpbfSthwLtYiF22TJthOhas8lyOZQsvOCNPYploA 0pVEwlt1jUPlA2FD+vriTwNuYAixCl4= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Wu0mFc+6; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf02.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.215.177 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733798380; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wo7sUpYgRQrXUd48YB6NzK0xhu80+vJLM/DolMWSp9w=; b=5DOOk4VWhzyCMXZFi3WF7iFw1GjUzHCkr8FSw6a+5zK/tegwgGNxjFDsZeTLOIDYtA9Ctg 9fqYbuqz9Ug4Ptal0HFjwV6+z0XfMKFUNN0iACkLEDut/3xmWF50GdcfQLbhnYYsGxiMGi FPT2qWR0NAUSdnQ08n++PfqCS8dX2ZI= Received: by mail-pg1-f177.google.com with SMTP id 41be03b00d2f7-7fc41b4c78bso2970861a12.3 for ; Mon, 09 Dec 2024 18:40:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1733798399; x=1734403199; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wo7sUpYgRQrXUd48YB6NzK0xhu80+vJLM/DolMWSp9w=; b=Wu0mFc+6WmDd6L0TNTDdMxvcNFoaaOXkb9e+Hrr9jbwuHknKI0Y2bHdtomJIZ85X40 8wu2iC8J+XXFobd1UdPKoY/PmU2Hau6pJ+9K2hdLx6hzitnghEp/D1tRZ0xNyC4/08RR iaxfk1MssqI1xPwZJqlJGaKCQwiD34EFSXhypvNCnbjcIY/wsAP5TbV7892oMev6wxK0 4eLLQ2UrIY/oirGhjN1UaRS2O3mp5IE1hn7IzwNeTEJldnwYRKbV7BsWR1/LhdonMCzA a5e/yW3mwQwqI/XPQ5gwTKuqeiJ97CrdPJrd32PKAWhOf7VgmlfNlIJ1wvjbZ82wWpf3 ZjtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733798399; x=1734403199; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wo7sUpYgRQrXUd48YB6NzK0xhu80+vJLM/DolMWSp9w=; b=k68bm5OflAcJN2cQzj/HTSGiKCy14EOlC4njItm0TanGvJwd58lhLqE92hIA2Z4kwa dmshDIcKhPTUOLMYiEPkY15yCvJJheYfuzUS/FLSGbgPAIFaX4RkDQlguvahPDkxXMba YuXLjojZaYtJdOPS9N+7FP4L4owFu0j2N0lWN2q4VAtU1ZWTDD5VH6u7SDWVk31Uf84O LRPBx6SZStWejEYnYN/UyL/dKXdnYs5jnHNkNpWHVa6Nc+p3UmHMRzZ0qpCpSVgP/ruU bJh3ZA0GdMLuUYH5L2RL4mOXiEgLy5A0mgxSPuoCpRzJdZKSXQCbpVszq+mgl9BHQHQR pWzA== X-Forwarded-Encrypted: i=1; AJvYcCUNjAI2DNkOmn9YDnV0FLUDF9NqlhqA7f/R4jHdNDN1GFHyLZmrOP8VHdA/yfzrylumrkWjsJWOFw==@kvack.org X-Gm-Message-State: AOJu0YxuW/X2kIGmDQApq18jqod1hxt5LGABINOOldcbY6qjAtffB0F7 6hBXExBH+w28ZObDrkCR22FdPKXSyjZKMxs876lT11Nmljg/mDuC X-Gm-Gg: ASbGnctRiF2yTY1/84udLHdER68i5+TRfGpbW34HC1xfNp53bApzDN7f4HHJ6pqTXpl kXGReR5mv+jntd0fwK5UB+uS4fAn8Xvoy7XjZniRJmHLxIPRsu8oONBqS+LaZ/FiMzJsYTrt1Em mGWqTfgKyQc+KWDzNeGV23LWg2K8mJ0yq75yza0WdoiKphpvzpUXpYBnuqgzYM1XrzJ3Llofr9E QYv7QHYAb20S+dEO/dpkRwuSjJJxoXmkgVE316SEoJ8Rc7/IzGw62Ay4p4aVOxQ36tfN+kfE4ns CbSf9w== X-Google-Smtp-Source: AGHT+IFtiAsGhIVFD64KUeuc/HqOnxB2YRGY24DKyXhrRpv8mSEuSX8pUEmuRY3vKJ8avlYCY8VlLQ== X-Received: by 2002:a05:6a21:890d:b0:1e1:9fef:e96a with SMTP id adf61e73a8af0-1e1b1a79c40mr3694721637.6.1733798399161; Mon, 09 Dec 2024 18:39:59 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:83b0]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd560985e0sm1471732a12.79.2024.12.09.18.39.57 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 09 Dec 2024 18:39:58 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v2 4/6] memcg: Add __GFP_TRYLOCK support. Date: Mon, 9 Dec 2024 18:39:34 -0800 Message-Id: <20241210023936.46871-5-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241210023936.46871-1-alexei.starovoitov@gmail.com> References: <20241210023936.46871-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Stat-Signature: b5f77unk4onkmu3ksemjiew3kmm4xd9f X-Rspam-User: X-Rspamd-Queue-Id: C999780008 X-Rspamd-Server: rspam08 X-HE-Tag: 1733798359-13189 X-HE-Meta: U2FsdGVkX19oXhuJuKqV9YNaW0LzGFI8Ru07+5NoFIbl8AYMikAZb9jE623PkZLJ1e/CcxnCtWoVH/LnXV0+0bSuwG+nn2ElSNMmLaPiUzEvl/M/yFY9Pu69fq51g4hC9DbFt6EHExkqbzyDVio8aSD7Kdc4vZXTvAJEeM9prnQg+LFb3vsbr0c/H9+ojNRz3Ur46C7xwurnoxSiLyS+RoBtmeptM5poNiihCgDSEWbItg5l2GylLSarF1R3P3amN8g1j3JI3Ya+Fn9IzC11HkFHjMTRWhC4uRwHkMTIC7pjgrDHWhPHnjMVRyLTShdeamgvZvUIRnxEcu9AvCKTCFprhFb92ghN0CRZPp3c8EwS/ObLJ/Y/TSNzeEKAflGPO5WLog/C7PQnOxAsL7Y8A5pqiT0lddm16Wo0akIDeForsyYoIfC8MkihLzCI/1eObPxYvxdzlUoWW3H6Pab4IEh+721Uhip1ROBnfAb/ZuwCAcrJROypvDtRvSBQxlmuHCa4txjFThnOm38UV8eZlBBY998KouE6H+dnPVYYoolRMA2/wqIuGe+Lw+cLo91bWVVhQ2zrzn3QvBoM844lB8PpBjPVeR4dpHnYM8rHTdu3ZDwQmFzpf4wn0XFfHqCLfQeIomhwJXL5aeoPhpvp/7czpz9smkruS5Va/PGJrg7fT/7dfj9X2WkI6VAmPNSN81CLj/953AzYrjh3Fw68yyuZ8VbN05Y42GQiPuKnaqVYPAKd/P2qQSjZ7JuA+Q1IY7BO1xhU6NYci2TO2m6HnvXsIOBET2vhOgOT1p9mE/qUIdYR0x8BZMOu/GDjH3vCxEMrlKjL2axeM1rNzz1TL+k4Gz6P5t3HPjoy5/tEMvweOtRNQbOLFlFkceS0kd8mxtK764RxVWFfwmrdjmtKxTP7soxhbG4iSO2jRTOhVCh72ekWIXW2HWcoB10DB+kBVF1f6aeTeioqZ2e0VOg C+TJoj92 6xsbLIzihkfKg2qTRDMchkzWtVxaQ1L5aU+nMZ8mwe1YwjuHaEvX/5jgSPAIYMdWjj9cvnIvcctKtlk7/HjV8m1d2HjEhZylbm3DOy+mbr8ZKWmXs+sARbRFFsxEQ9hgK1LmqZW5mkK44xdzCf2dBdGeoGVFal4zE5MhhctsV0m2uPIr5zQ668uOJaLgDmS/+/52yr/u6IkB/LTi36Ls8JJbqV2HNOMMKmve+wCNQfeS+zLmCZ5yE13yrG3ZH+X9EjIJeL/CAaMeestPkAwqlBeqtcSk91jgtwmo/juqwU1ulubfBHivQwIX2xkrAUjwU/yEYf4Arlhn3z/E3UEC+IQsIpg2qjBkE1UThdjmq10xfQoMMcXC110L3X+q7fKXBLIEdiiFfYDwoQHmoMuSzzPrd/PveU5PXd8OPo6KFk5qYiCn+ahKIUOTbOcpQm+0D0Nn3soaaP/ajleWEPDxh8cFNDEJgZHNR1pKa1+olC5+xMcJ8TAAYOSPfcA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.390122, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Teach memcg to operate under __GFP_TRYLOCK conditions when spinning locks cannot be used. The end result is __memcg_kmem_charge_page() and __memcg_kmem_uncharge_page() become lockless. Signed-off-by: Alexei Starovoitov --- mm/memcontrol.c | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 7b3503d12aaf..459f35f15819 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1756,7 +1756,8 @@ static bool obj_stock_flush_required(struct memcg_stock_pcp *stock, * * returns true if successful, false otherwise. */ -static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) +static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages, + gfp_t gfp_mask) { struct memcg_stock_pcp *stock; unsigned int stock_pages; @@ -1766,7 +1767,11 @@ static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) if (nr_pages > MEMCG_CHARGE_BATCH) return ret; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + if (!local_trylock_irqsave(&memcg_stock.stock_lock, flags)) { + if (gfp_mask & __GFP_TRYLOCK) + return ret; + local_lock_irqsave(&memcg_stock.stock_lock, flags); + } stock = this_cpu_ptr(&memcg_stock); stock_pages = READ_ONCE(stock->nr_pages); @@ -1851,7 +1856,15 @@ static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) { unsigned long flags; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + if (!local_trylock_irqsave(&memcg_stock.stock_lock, flags)) { + /* + * In !RT local_trylock_irqsave() always succeeds. + * In case of unlikely failure to lock percpu stock_lock in RT + * uncharge memcg directly. + */ + mem_cgroup_cancel_charge(memcg, nr_pages); + return; + } __refill_stock(memcg, nr_pages); local_unlock_irqrestore(&memcg_stock.stock_lock, flags); } @@ -2196,7 +2209,7 @@ int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, unsigned long pflags; retry: - if (consume_stock(memcg, nr_pages)) + if (consume_stock(memcg, nr_pages, gfp_mask)) return 0; if (!do_memsw_account() || From patchwork Tue Dec 10 02:39:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13900698 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DDC1FE77180 for ; Tue, 10 Dec 2024 02:40:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6CFB36B00E2; Mon, 9 Dec 2024 21:40:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 632DF6B00E4; Mon, 9 Dec 2024 21:40:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 485306B00E5; Mon, 9 Dec 2024 21:40:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 1FEAE6B00E2 for ; Mon, 9 Dec 2024 21:40:07 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id D128014020E for ; Tue, 10 Dec 2024 02:40:06 +0000 (UTC) X-FDA: 82877494140.21.1B5AE91 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf19.hostedemail.com (Postfix) with ESMTP id 275581A0004 for ; Tue, 10 Dec 2024 02:39:41 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=GD09FP5H; spf=pass (imf19.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733798384; a=rsa-sha256; cv=none; b=ZtZEoPpk7X8khyeRrBFSuPrPijWXC01CM9p4v28AVpDDk8AC20Mz1bx6BRydhiaZ+nmYDT 6bz8fMLpd0uPa9VRSpriaE9Xcfs8/Zrl5Mb7wzrOVzHfQdzvhpXcYmh9/1kNkYbKij7XwQ pCq0Moz0xM0rjJu+5Jm8fyPFTw/b2QU= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=GD09FP5H; spf=pass (imf19.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733798384; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dZcki3SirXCmN69CdcGKxt31rfH5khiUYpTKaefczMs=; b=OCvtT8yDAEArIh4/FfNQAc5EhuVzd5L43VMeVEXVE4C7fRzA7RtDwG2KAPaM1Sz2w5ARtX +MMZHv+VwTVetfrXl5chkF3UtSN5C9gjIcSz8xZyu+qrzXQavmuqFp3p6RBE2XcD9fKBNq tkYRWghvquxiNnMHV49gzplGC01K5cg= Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-215ac560292so50052335ad.2 for ; Mon, 09 Dec 2024 18:40:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1733798404; x=1734403204; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dZcki3SirXCmN69CdcGKxt31rfH5khiUYpTKaefczMs=; b=GD09FP5Hnzt3loR23imrGSS3Bw+K19nZCd+NMqfHrWfbYLwG6e2dXcoKcwrCSNhRc9 EZpG9imVI89npL2NJZUbOpoGvSen7nRR0ec39uh31lzfm5iOvkliFPfNSpQ8Ol+2dqn4 EfseRoaG0zBFPZJDTizMK4jlri5BDoVJYUoe+jOT3UssTIg9aKSRnMIYms3Ss3qWHJIY pfh584Xsj77cDyhapE5+n9NZY3M6X8kLRLNRpcINd1X6C0aq4MnrKNm5g8GXvR3icvJ2 lAALIaHcedTDs4kB3tbrrbqUuOV5438DbkjvgNnw6+IfxzW4E6zY62Wz2N6Mbp4ymtPe qigg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733798404; x=1734403204; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dZcki3SirXCmN69CdcGKxt31rfH5khiUYpTKaefczMs=; b=PIcq5sDnedrNCLSKewep4XsTWoAojnht6K4JgcdjG3Qgm14E+c+aa9cBg/XHBzfQI0 8yHcQhiXma40Op5qufk6lBPXP1dw5oGkuzFBcWhHLs7Wp9RBY4NYP3EOD7dLUUS9h7x/ taY9wBKczTh+IJ+ZYEDtGobpZueW03MRvDhC7toDNwcBZpZYiFb+dCNWotU/Xw6wOs2w B3M6a2r6CP3vywAQmyUM3o84G7q1rD2btMlVOyl5dY/MLlttkxvX29I/Vuj1ft5LTE7V bXZhG0yjJCNGimEEofF85Hv6YBjipc+i3eJAIHgHRbQtUOX2Ls6M5E5FkeWw0ToX6lm8 TqzQ== X-Forwarded-Encrypted: i=1; AJvYcCXshdRngG07pOjQDYbE9pTWpb5dbEf6oLTkpzWfSaaeMveXkMPNvYC9Tl7pQfuVVCREFjR4LiJDOw==@kvack.org X-Gm-Message-State: AOJu0YzECmDYB4p/QbyrRikU85w3U9cvESvdI2inzBsjRz5IhcqHsG1c OwNG9O3TOJIFUsswpF7Pk1lPmETCQG2fN08pxmZmbrDq60+17RD3 X-Gm-Gg: ASbGnctbSj0qoRi76vabFF/be45XJXbkK2wMu4yJr+6unh7NmLhjRyLXraTSxxebBqO rEMLie34yjF4hBAgtZ7OtEFx8s/kb8PRBr3U9NUTQf8scbFB8F+Rzb69ucAZvK9zCmuEdluLzJX R0JS9fjpiLQXumT0Hl1EPXEXJ2k3kbCNQrD4RZ1cLBXjpmKqRRSDSJjDi5EBXKcnrPaD1XuXNyi hNWOpR585/znOr5LOCq0cxNzsFzEMfmQnkDAqQnTYF6wGBI3baemYsfaVqQHxYUg0mVjFzrdN0T c8e+yg== X-Google-Smtp-Source: AGHT+IGCrCmcyV05haNAd+ohuQ5mWaS3nvBsc6i7L/ZiIyD+Czd5Of6TvkBrVGS6hN/453APVBLTQg== X-Received: by 2002:a17:902:cec4:b0:216:501e:e314 with SMTP id d9443c01a7336-21669fb7abcmr31154365ad.20.1733798403643; Mon, 09 Dec 2024 18:40:03 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:83b0]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-216150886e7sm61609365ad.282.2024.12.09.18.40.01 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 09 Dec 2024 18:40:03 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v2 5/6] mm, bpf: Use __GFP_ACCOUNT in try_alloc_pages(). Date: Mon, 9 Dec 2024 18:39:35 -0800 Message-Id: <20241210023936.46871-6-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241210023936.46871-1-alexei.starovoitov@gmail.com> References: <20241210023936.46871-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 275581A0004 X-Stat-Signature: yzpofoambufxxfj7buzn5nitqbtb49e9 X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1733798381-603822 X-HE-Meta: U2FsdGVkX1+mdaTnGlayjKkEIb/iPl0N4hwZXp91FDIRwjxEWVSdl+NKDO1g/BYv1V6OwSyOL7vQd7Z3/+ZGARaVVQEZdxy1bGawAlnyu9IVF/pMKEtNb2CWlQs36GdJTJ4Ul2THTnJKU2t4btxa03qTLRgFdNbxi6WooRZqUtAAsHGGhWx3lsUY+t52H+HXWoHHPWg1gQzn64wCWASOocYc94ayWsEB6N4SwruuPtbqyfVdEm/9g4tGJO8xibi2vx8RWmoGJIEdxmZLp8rouTbKXDCTL0wTG8eayCPxMfSMN82QVcjoL7ExFodIZHU7Nv/l82Ud35HN4+nCIgTTAG6IKMHgRG3vzJ6qpbOKJ3VvSbcRyLxBOM5L4VtvFkf8SrOYzAtexJIL9hrCM8hKxmn2K/isMspSgkyj8BZoxyDPx/ji4Fgul/CDAUsrJ4faxm71u/iP6Uk49ncne0yMGJuFPnpgfysXLBz2PGjTuMxDF6w0GmM7XI2fyd/ElvNGtsKvV0PXLUhP2BWOOmYk5qPbRfimRt1hnHaBdTR23jyGi4nZ2L9L/ConycZk7NgWAbqLLITayUR0P4IWrfi2xoWT9OE7uBnaa95ro3gLKq0h014Ol66kpoetitJ2nfcggfUPzKovzF8afctuKwhNzzz7pO0bmFNToir9vKKbg1esf3VBLaHgbb96U6eYnTQCccy+Ci73gSUT+0t6rVQfMfSSnAxLZr7ncMInlOXj2ftRZcZO4sLRDcc9nWjJttVUw2qZY1OrIY1P8VGZ0BQRglCXNPXvu2ag72FijwlkgFt3Yw/soMoNiiee/WqLu7wCtg4pbjRtUqz9sUAizNmQY8hbMAX5FJxI6SeE7EN5oUjC1Po4a1Pl5U/YcpHvl3Rd2a3JhC8votZwv8cMRZH+nAafO+vXjr1y+a5PRyeaQnof90CLP/52hayeGP+AK6bZ0HLrSf/0XjJo5AIGpZ1 /6MNkXbX ZdX1IZA/HE/PYUHBSOxdMXkt90f/xhGIefEbtCd+bggcL+Ia6qRmtl7aLBeVtINWbMi24naeX5wQPQ0lucb8nIv7sDq7fLOap1l1ZqN05TekTYdDtvSw1Qy7fT8YU4p9zW6OURWJQbYPdVnbSyzq8sOK60f1Y59XaKVeqDvcwTho6sJ2y8ErkhXrUdZpbPrw85N87CeR8/aLPh/bm2uXTDmMBSbNbXyjYMZAz6w1fUfgTPrqwfSf8ukfvHKJ8XwWjREBb9QG5EeYHSgqjwO0zX0L75eR2TJmGtv4WjKklvZl+QyXhhgdDiFxUpCExwm9ejSqa5h0BM2J6Kn2IT9nwdmkYQE6eX3axHnI8o7SvYag7GUcgV7q6jnh6OE7FruDGwW752o2s6Iok5DmwQIm1zN//0vCQ3KJ6hkG+e0Gdi/mMwlK0rMyKfKc/1/bVX1uxq1MoTeh8viq1ASa61bHJ0Lk7LcrrvBYveYm1lBw0a6syFV/im1A3uYudmA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.046843, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Unconditionally use __GFP_ACCOUNT in try_alloc_pages(). The caller is responsible to setup memcg correctly. All BPF memory accounting is memcg based. Signed-off-by: Alexei Starovoitov --- include/linux/gfp.h | 5 ++--- mm/page_alloc.c | 5 ++++- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index dcae733ed006..820c4938c9cd 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -356,18 +356,17 @@ static inline struct page *try_alloc_pages_noprof(int nid, unsigned int order) */ if (preemptible() && !rcu_preempt_depth()) return alloc_pages_node_noprof(nid, - GFP_NOWAIT | __GFP_ZERO, + GFP_NOWAIT | __GFP_ZERO | __GFP_ACCOUNT, order); /* * Best effort allocation from percpu free list. * If it's empty attempt to spin_trylock zone->lock. * Do not specify __GFP_KSWAPD_RECLAIM to avoid wakeup_kswapd * that may need to grab a lock. - * Do not specify __GFP_ACCOUNT to avoid local_lock. * Do not warn either. */ return alloc_pages_node_noprof(nid, - __GFP_TRYLOCK | __GFP_NOWARN | __GFP_ZERO, + __GFP_TRYLOCK | __GFP_NOWARN | __GFP_ZERO | __GFP_ACCOUNT, order); } #define try_alloc_pages(...) alloc_hooks(try_alloc_pages_noprof(__VA_ARGS__)) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index a969a62ec0c3..1fada16b8a14 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4818,7 +4818,10 @@ struct page *__alloc_pages_noprof(gfp_t gfp, unsigned int order, out: if (memcg_kmem_online() && (gfp & __GFP_ACCOUNT) && page && unlikely(__memcg_kmem_charge_page(page, gfp, order) != 0)) { - __free_pages(page, order); + if (unlikely(gfp & __GFP_TRYLOCK)) + free_pages_nolock(page, order); + else + __free_pages(page, order); page = NULL; } From patchwork Tue Dec 10 02:39:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13900699 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9B3D3E77180 for ; Tue, 10 Dec 2024 02:40:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1A1676B00E4; Mon, 9 Dec 2024 21:40:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 102F76B00E6; Mon, 9 Dec 2024 21:40:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E715C6B00E7; Mon, 9 Dec 2024 21:40:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id BD2B26B00E4 for ; Mon, 9 Dec 2024 21:40:12 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 724811A01CE for ; Tue, 10 Dec 2024 02:40:12 +0000 (UTC) X-FDA: 82877494434.24.A06F82C Received: from mail-pj1-f46.google.com (mail-pj1-f46.google.com [209.85.216.46]) by imf23.hostedemail.com (Postfix) with ESMTP id 6E25F140009 for ; Tue, 10 Dec 2024 02:39:56 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Wv6xw8JN; spf=pass (imf23.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.216.46 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733798390; a=rsa-sha256; cv=none; b=gQv8MIjuaRMvda+/x2GwXiW8PAaV8oC2qnqnLUTdTtXNNKEK6ZHYTKNCsfyFV4SCxM9R9Y cTothdcsuukc/5/YrpY+pVmHuwnJZPd/Il3dU6sQK/Es4wjQ2GlMJrMRZp+acSdItSm+/g 7Gx6o7sXLdayazf+2OrmWoHViT883OQ= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Wv6xw8JN; spf=pass (imf23.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.216.46 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733798390; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uiTKIgIWEw4pjiXWRJupFp29ffG8tRf5Pk0epbA95Ok=; b=GJ58vZXYK/v6QozT8GQl1RHMe0ql6EaZZ9ctJY9auDCe/wTjMz/ApDReBb5dztfupWrwrx HMYZTaWl0n45jujRVUCKIYoM6QqXB6ov92BZp6Mhx0C1SI1HH2D4IGet/3GOY97hzqCB0j pSgHy1DA8SiKSJVCuIAQxL31Vntlkxs= Received: by mail-pj1-f46.google.com with SMTP id 98e67ed59e1d1-2ef70c7efa5so2529421a91.2 for ; Mon, 09 Dec 2024 18:40:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1733798409; x=1734403209; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=uiTKIgIWEw4pjiXWRJupFp29ffG8tRf5Pk0epbA95Ok=; b=Wv6xw8JNCpBpIVgD7ooev7HESYeCf8fO+rTZVCFn8l8YfKN46zCmQ1C0qJsliRL7vH 6I8z59z3MG1ochQEnHTAmLpBGWHK3ja16JwK6d7XbvRpWy/dmd6C04gapMbM99qNn8VG 3AGWWhAkCW3kCSBW1FsaPUhKvHHUJTz4x2aBFURdP9cke+JSaSd1rCSq6jCdaV43b9Wr OR6S2XrFvThLM5OPFeI9c4LbhhMMzT6abV1OfVuppmKTvRNQUZKA9GyFdqUes+tMEqXv Fj4vniA6ZOCeE9LQghItrNRjVZvGGG8sYaPavwvRwSf0r/4muNj24O9HnJ2GU2S6Js5v sQBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733798409; x=1734403209; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uiTKIgIWEw4pjiXWRJupFp29ffG8tRf5Pk0epbA95Ok=; b=TCo7wix1fJsAls8KDSU/GIiDJEVkbsfq9qqffnBj3pro3Yxu8kN/yfCNUh/oeev1/E bZ1dg3iLs944BU8e590bb5OnkSi3s0xHh01icw944nq/pidAQJMxB+MFi/L8lSaHtW/F 9Yu0EyBVaSf6oIDu8rpjwTdkt4SIDz5kIUME+KW82CCqYU5HunbC7tx6efF/NWYJK/MF rCTrbAe2fBeMolo6xqdQIKEorqRxSy/Izu4aQgNa2KDTTBI/vPFK0jLtLJRd3Zrv4oP/ uGlBETZip3U4UxXBJgcmmgisqbmbyfg/KWADrfsSOSvI5AI2YcV4bGdgjBnQLKh6PFML hG9g== X-Forwarded-Encrypted: i=1; AJvYcCWF/lj/trdqnngEMx07rBly2nEBY/hnYRWNQ03FE8OyJc28cd92Eul/AMN8E+m6XLnu36txxE3p3A==@kvack.org X-Gm-Message-State: AOJu0YzZgTCC4JSEI9Tswe3Un2bDV/ni6e8x8nNS/jjtSJ0bF95bn4bw kbdDtBq7U1siSpVigE1FWsy6Iq3RhzSrzDPxyBXyFlolLjq/Uccf X-Gm-Gg: ASbGncsj3rU50+Rw68rsauDpp7b2TMEebRtmrTRixaFDqSJGXb+7KIAXwfUStbUAIu6 kp0zoGB1QHjCMwYt9F1sJWlASj1Mmreih4xJTnepXgdXNMbk4lGdTDvPrleZpRir+IotrLqh1Np PpcrgkFdtgsupCktqYm3Fj9Wdy66ZKrlnNnctqe+oKNMSUjdn6lqeiV1UcAnDbLXsNPtw20jhi8 i7WlGy4QRmnDPzq8/fFrDgs1++RmvJNCpXZIN6SOpezBRa1JOYtQylTFgqwQbBa/nBpZuDOwktm 3IWlcQ== X-Google-Smtp-Source: AGHT+IEyaw7MLXSrpLiOPJxXq3UM5b+L21nSTK6Jj0nYt9CgSbssrgPTsovcJ8e9A4v907+avThj1Q== X-Received: by 2002:a17:90b:3847:b0:2ee:c4f2:a77d with SMTP id 98e67ed59e1d1-2efcf16f547mr3957084a91.21.1733798409404; Mon, 09 Dec 2024 18:40:09 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:83b0]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2ef1e920587sm8088928a91.1.2024.12.09.18.40.06 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 09 Dec 2024 18:40:09 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v2 6/6] bpf: Use try_alloc_pages() to allocate pages for bpf needs. Date: Mon, 9 Dec 2024 18:39:36 -0800 Message-Id: <20241210023936.46871-7-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241210023936.46871-1-alexei.starovoitov@gmail.com> References: <20241210023936.46871-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 6E25F140009 X-Stat-Signature: 35csusmucw38y1yrj6tsr6j86ddfugn3 X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1733798396-760332 X-HE-Meta: U2FsdGVkX18oPhsFWqxiwDmz3GWyDjt6ulRw0uj+lXkVAgALDLkbDZJlohK04k9xYmHtjgStKeQYMMVf7xnin3IjUVUqv1wz8E//VM2ydNbOiDXJ0SzkKXxlzFs6Dfcf2vagZmWbHDkCNL/3DMPguZEE9OEzHqT3LAiPvqyXYDwtqZC8sN34EiukMiNQqOW+Dxr4GPkwl9zBSP80hBM01kS+4IQR8THn+NUKvW1zBM3gqwZNRR1I1kkEa1iMZfoCxAnXOZXEJA8NsKm1Wp1VJ31+IcAHHjIlqjaQKuBCf7qG+VzoaTAIvkeCwsuO/nKr+/sZT+BfNH1yU6Kku8yPfGYmc7mFjuWJLnatftZ0wnZSnqtvBFPHSFdJ5az21hbH+Qoo0MQ4/wEbZom7UzdrkxZwyVQWl6jZg4oyirCkK30CEmS7u983RQDevL96zkLJAnSZfA8OyB69BeipIIOpPGik42FE5xGCYhvxMPygyV2/m7MpFZPhw4qZy+da8rRA7DoA4tToJRR925M8fNvmZAOt/WLPLtTXW39wftdkE0njKxqSg0s8Tml57Wu9UxRlMpH6oVhNrQOO8JXcMsQIwlLsClp518nrRKEk67b8KVGGbcOuVO2doBBQD0Q434EDX6ReKzbRo0f8BUYZClRfuyQIN0sqLTTZ3Y81zepwGGfFO5fFZqGB7oLhdbvcLf4uQXbqQ5PtGJptACQ4DmZTwXmC+zHcL9rwph9dRf6slc+spi3RuICb7Zc2aGeNew3SN9n+OhXaG5kNHNBB6oak+fGOYRgPm1w8effp+/FpB4rKkiVdvY61QtmwxQNmR6wM8aIUYNI68tuXdO62iO2t7ZMIj4hX+icm9jrnc7rnL5AWdCFFmPMc0BiGxcYmhf4I7YtvaI7YfS4I5WstQWZN3V4tGtWzeD6Ma/ozFkF/XoemiOIkYFow6ik1zuzWJjsLN6LTfLArbwT3ZPnJxqe zNs4OUb/ SjlDVrGP62SjHhZ3IVxdhM4VlHRtcWNv/4qgcqsx4vgjM26Jemuuxp5Yau62EGXroqxg+VVmuRH591nNnPbgkbj0CECtdGbd86i1oVDoA+Kkf9iMLgLPmn/oiqKhnqNqwU/5s3XgFKs6lebqchMeYv/m9W1/IMJ7Infgy30lrDkXyjUmGbT8NpMAKE8EUpXtW21k/+gC6cBnZCge38Q9U2BtqD5OzmgqI6XwIRlz1T5BRnE2YhuuJw7W0eTQzD1rsvpHlmPZ+Cu1Ue5wrMnUVVsbRAlwTBKEswLRIucyNx8WyfssbxUGG38O/bNrj/oTdqN5AGKC9FX+rxWw1wWUjkxHkfXlBnzho7dXL8DqYU8KziWZhZhkeCHGnXtHaQHIF/Y5jUh79cnJymegOtQBsxqLmS3mYB2jMjTqftNk+ZibtRjvWNbHGFgprZ7Y2sdpyh4jJgl7VMZCery3F6ZNKh3TR+IhQxqhrAKKe+YKmibfXSo3C76G/au2JvQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.026570, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Use try_alloc_pages() and free_pages_nolock() Signed-off-by: Alexei Starovoitov --- kernel/bpf/syscall.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 5684e8ce132d..70589208b545 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -582,14 +582,14 @@ int bpf_map_alloc_pages(const struct bpf_map *map, gfp_t gfp, int nid, old_memcg = set_active_memcg(memcg); #endif for (i = 0; i < nr_pages; i++) { - pg = alloc_pages_node(nid, gfp | __GFP_ACCOUNT, 0); + pg = try_alloc_pages(nid, 0); if (pg) { pages[i] = pg; continue; } for (j = 0; j < i; j++) - __free_page(pages[j]); + free_pages_nolock(pages[j], 0); ret = -ENOMEM; break; }