From patchwork Sat Feb 22 02:44:22 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13986506 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pj1-f46.google.com (mail-pj1-f46.google.com [209.85.216.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 03E351FBEB3 for ; Sat, 22 Feb 2025 02:44:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740192279; cv=none; b=mCf93VDHs79oNhU439Q5zZyAhobXN+Z9gvoq0m6ALVkLh+CvMK0f64B4dOZg21JAB0ZE4ke1lWIJJBVGiIU7qUsaJUgSyNbTVN9Amil07sK9kHTgfTdKBwzjHm7VMiq5gmCScWHwtWCsOaLpzEipjdCVheXz0Vpj47YGL4pffJc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740192279; c=relaxed/simple; bh=WL71uV2ZNGnRr+mbHVLa0ks+Vqr1Ig6jqQJVVIKQsBw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=dC87WP8+p8Z2X5T5+9op5hM6nHY+6JI1fPcIEYXwiFdgHZwFupfNJJOywv/BfrOBXiAeoHm05xgNKjluKo+DdTcO1/nhU01oZsXeUqxuHZEaqgN82sFAOx2XhJ/RlvA7moh3Z5rC0g6Gdc0ilEsLSgzwHUtaFlruray8/Nte48Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=UM1ootlN; arc=none smtp.client-ip=209.85.216.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="UM1ootlN" Received: by mail-pj1-f46.google.com with SMTP id 98e67ed59e1d1-2fbf77b2b64so5573301a91.2 for ; Fri, 21 Feb 2025 18:44:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740192276; x=1740797076; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Sk02Orm/sDWbmfQ5EAte0I5oPgAuhzQ+C9dMQxWikoo=; b=UM1ootlN2ZwEm00j3RFU+f/tPn5wSGiSvk1JLNsyBog88J2a03rBdScFABbDFgZVwB lrXlW31XxZXFH4Taddvvj3ZBdmLp8jf06hhll6cd3Bml8LZgiCN5Rsa/5+7r/ZnU6BPt xRyOGHtSUCqH/v6u8nCWNiP4YMDXNGWJKXlja5zUm+Btjl2jDDWCazgMOl2thy2/NI8K i4VQaobh9KtxqDY4KMPt6jXOWB2FtF9R9j+ZLtawh3twlSAEX/S+AsFkGpx+hHDqo4oZ VdId6UtuKUIvGIArM1yqsHgw0jegbopKHaPPN2rZB72JKgiZovrIZrvFTHEDqEWLG2MR atmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740192276; x=1740797076; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Sk02Orm/sDWbmfQ5EAte0I5oPgAuhzQ+C9dMQxWikoo=; b=VQtiGNV+hApehNmesQIO2Jl8KJyoYvFdClHPsBJrb22zyxOBt71gg4TZJ5Hvxxb9HS fYUo74AQOwXckcPSLS/ksXw+h34J3jJ6V5pH5OUOAbeLKh+95Xh60WrlkzpwfnkT8PaV yNv7sdKj5vViexk9ABLatAvhJLhdrU7sT3gSWmxV85N+DePdB7htyVv0QuvwyYJut6Yu tEIYBwFYmMxcy4gHUwOkF0HLF3Jdiv5h8alwpntzv3sz4mKpv7KIShQxYDTHBcIgAT/B mxCKKZHviX0gyJcUz9rBf7vwgHfsN4DDuZKsI4Ezg5HF7Mb+Mk0MEztX6xSurn96JMY9 9kkA== X-Gm-Message-State: AOJu0YyiLNBL7BriqPfSxnhK1tKy4FFrlOWcQcouV6kSUDVLGDpuZXNI 1Bi078mw8m47R9IIOCZ9TL2QQ6RP4nkKIE3cah3stJF6/OqatczN5+rstQ== X-Gm-Gg: ASbGnctM4iiWfsn/yFJq/kT51Psb1u43nVofAEbDzGCyzoI9JFPzU/Y8ViO+Zi+7DLK cqlTR/yhaJqBc1anOKUnG+f0eEpoMXMSI5LdInoocxPt0VMVf2khbF9JV+ie5+BH+TYdfFwAKx7 btOtuIDwYLUz2RscbgA7UHdVvKIyrMVJVgaIGFmkntt4IrQq4Xo7JaafjehnxgKizDw4AWqSf18 qSFKAbRXM1bOnYEgyVtY8839QzOn8bPaLR3GWTtJoOzkzNMmBQ1bRxJki9I2DKu5CYxQM/BXMuz bXP4gu1I9bQfzJ8cbAcUvJ8jTJYosompPX74rHFb8RVDlZDkyGqwDA== X-Google-Smtp-Source: AGHT+IGlSaTzpWPllvJGdoZ8RptnpXSPkTv04MwCkc6eCe1wznNZ2YqQBDcM5ibR5tXrH+w8nZ0TTA== X-Received: by 2002:a05:6a00:4fc4:b0:730:7771:39c6 with SMTP id d2e1a72fcca58-73426cab09emr10492884b3a.8.1740192276435; Fri, 21 Feb 2025 18:44:36 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:fd1b]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-73242546158sm17069167b3a.16.2025.02.21.18.44.34 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 21 Feb 2025 18:44:36 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v9 1/6] locking/local_lock: Introduce localtry_lock_t Date: Fri, 21 Feb 2025 18:44:22 -0800 Message-Id: <20250222024427.30294-2-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250222024427.30294-1-alexei.starovoitov@gmail.com> References: <20250222024427.30294-1-alexei.starovoitov@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net From: Sebastian Andrzej Siewior In !PREEMPT_RT local_lock_irqsave() disables interrupts to protect critical section, but it doesn't prevent NMI, so the fully reentrant code cannot use local_lock_irqsave() for exclusive access. Introduce localtry_lock_t and localtry_lock_irqsave() that disables interrupts and sets acquired=1, so localtry_lock_irqsave() from NMI attempting to acquire the same lock will return false. In PREEMPT_RT local_lock_irqsave() maps to preemptible spin_lock(). Map localtry_lock_irqsave() to preemptible spin_trylock(). When in hard IRQ or NMI return false right away, since spin_trylock() is not safe due to explicit locking in the underneath rt_spin_trylock() implementation. Removing this explicit locking and attempting only "trylock" is undesired due to PI implications. Note there is no need to use local_inc for acquired variable, since it's a percpu variable with strict nesting scopes. Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Vlastimil Babka Signed-off-by: Alexei Starovoitov --- include/linux/local_lock.h | 70 +++++++++++++ include/linux/local_lock_internal.h | 146 ++++++++++++++++++++++++++++ 2 files changed, 216 insertions(+) diff --git a/include/linux/local_lock.h b/include/linux/local_lock.h index 091dc0b6bdfb..1a0bc35839e3 100644 --- a/include/linux/local_lock.h +++ b/include/linux/local_lock.h @@ -51,6 +51,76 @@ #define local_unlock_irqrestore(lock, flags) \ __local_unlock_irqrestore(lock, flags) +/** + * localtry_lock_init - Runtime initialize a lock instance + */ +#define localtry_lock_init(lock) __localtry_lock_init(lock) + +/** + * localtry_lock - Acquire a per CPU local lock + * @lock: The lock variable + */ +#define localtry_lock(lock) __localtry_lock(lock) + +/** + * localtry_lock_irq - Acquire a per CPU local lock and disable interrupts + * @lock: The lock variable + */ +#define localtry_lock_irq(lock) __localtry_lock_irq(lock) + +/** + * localtry_lock_irqsave - Acquire a per CPU local lock, save and disable + * interrupts + * @lock: The lock variable + * @flags: Storage for interrupt flags + */ +#define localtry_lock_irqsave(lock, flags) \ + __localtry_lock_irqsave(lock, flags) + +/** + * localtry_trylock - Try to acquire a per CPU local lock. + * @lock: The lock variable + * + * The function can be used in any context such as NMI or HARDIRQ. Due to + * locking constrains it will _always_ fail to acquire the lock in NMI or + * HARDIRQ context on PREEMPT_RT. + */ +#define localtry_trylock(lock) __localtry_trylock(lock) + +/** + * localtry_trylock_irqsave - Try to acquire a per CPU local lock, save and disable + * interrupts if acquired + * @lock: The lock variable + * @flags: Storage for interrupt flags + * + * The function can be used in any context such as NMI or HARDIRQ. Due to + * locking constrains it will _always_ fail to acquire the lock in NMI or + * HARDIRQ context on PREEMPT_RT. + */ +#define localtry_trylock_irqsave(lock, flags) \ + __localtry_trylock_irqsave(lock, flags) + +/** + * local_unlock - Release a per CPU local lock + * @lock: The lock variable + */ +#define localtry_unlock(lock) __localtry_unlock(lock) + +/** + * local_unlock_irq - Release a per CPU local lock and enable interrupts + * @lock: The lock variable + */ +#define localtry_unlock_irq(lock) __localtry_unlock_irq(lock) + +/** + * localtry_unlock_irqrestore - Release a per CPU local lock and restore + * interrupt flags + * @lock: The lock variable + * @flags: Interrupt flags to restore + */ +#define localtry_unlock_irqrestore(lock, flags) \ + __localtry_unlock_irqrestore(lock, flags) + DEFINE_GUARD(local_lock, local_lock_t __percpu*, local_lock(_T), local_unlock(_T)) diff --git a/include/linux/local_lock_internal.h b/include/linux/local_lock_internal.h index 8dd71fbbb6d2..67bd13d142fa 100644 --- a/include/linux/local_lock_internal.h +++ b/include/linux/local_lock_internal.h @@ -15,6 +15,11 @@ typedef struct { #endif } local_lock_t; +typedef struct { + local_lock_t llock; + unsigned int acquired; +} localtry_lock_t; + #ifdef CONFIG_DEBUG_LOCK_ALLOC # define LOCAL_LOCK_DEBUG_INIT(lockname) \ .dep_map = { \ @@ -31,6 +36,13 @@ static inline void local_lock_acquire(local_lock_t *l) l->owner = current; } +static inline void local_trylock_acquire(local_lock_t *l) +{ + lock_map_acquire_try(&l->dep_map); + DEBUG_LOCKS_WARN_ON(l->owner); + l->owner = current; +} + static inline void local_lock_release(local_lock_t *l) { DEBUG_LOCKS_WARN_ON(l->owner != current); @@ -45,11 +57,13 @@ static inline void local_lock_debug_init(local_lock_t *l) #else /* CONFIG_DEBUG_LOCK_ALLOC */ # define LOCAL_LOCK_DEBUG_INIT(lockname) static inline void local_lock_acquire(local_lock_t *l) { } +static inline void local_trylock_acquire(local_lock_t *l) { } static inline void local_lock_release(local_lock_t *l) { } static inline void local_lock_debug_init(local_lock_t *l) { } #endif /* !CONFIG_DEBUG_LOCK_ALLOC */ #define INIT_LOCAL_LOCK(lockname) { LOCAL_LOCK_DEBUG_INIT(lockname) } +#define INIT_LOCALTRY_LOCK(lockname) { .llock = { LOCAL_LOCK_DEBUG_INIT(lockname.llock) }} #define __local_lock_init(lock) \ do { \ @@ -118,6 +132,104 @@ do { \ #define __local_unlock_nested_bh(lock) \ local_lock_release(this_cpu_ptr(lock)) +/* localtry_lock_t variants */ + +#define __localtry_lock_init(lock) \ +do { \ + __local_lock_init(&(lock)->llock); \ + WRITE_ONCE((lock)->acquired, 0); \ +} while (0) + +#define __localtry_lock(lock) \ + do { \ + localtry_lock_t *lt; \ + preempt_disable(); \ + lt = this_cpu_ptr(lock); \ + local_lock_acquire(<->llock); \ + WRITE_ONCE(lt->acquired, 1); \ + } while (0) + +#define __localtry_lock_irq(lock) \ + do { \ + localtry_lock_t *lt; \ + local_irq_disable(); \ + lt = this_cpu_ptr(lock); \ + local_lock_acquire(<->llock); \ + WRITE_ONCE(lt->acquired, 1); \ + } while (0) + +#define __localtry_lock_irqsave(lock, flags) \ + do { \ + localtry_lock_t *lt; \ + local_irq_save(flags); \ + lt = this_cpu_ptr(lock); \ + local_lock_acquire(<->llock); \ + WRITE_ONCE(lt->acquired, 1); \ + } while (0) + +#define __localtry_trylock(lock) \ + ({ \ + localtry_lock_t *lt; \ + bool _ret; \ + \ + preempt_disable(); \ + lt = this_cpu_ptr(lock); \ + if (!READ_ONCE(lt->acquired)) { \ + WRITE_ONCE(lt->acquired, 1); \ + local_trylock_acquire(<->llock); \ + _ret = true; \ + } else { \ + _ret = false; \ + preempt_enable(); \ + } \ + _ret; \ + }) + +#define __localtry_trylock_irqsave(lock, flags) \ + ({ \ + localtry_lock_t *lt; \ + bool _ret; \ + \ + local_irq_save(flags); \ + lt = this_cpu_ptr(lock); \ + if (!READ_ONCE(lt->acquired)) { \ + WRITE_ONCE(lt->acquired, 1); \ + local_trylock_acquire(<->llock); \ + _ret = true; \ + } else { \ + _ret = false; \ + local_irq_restore(flags); \ + } \ + _ret; \ + }) + +#define __localtry_unlock(lock) \ + do { \ + localtry_lock_t *lt; \ + lt = this_cpu_ptr(lock); \ + WRITE_ONCE(lt->acquired, 0); \ + local_lock_release(<->llock); \ + preempt_enable(); \ + } while (0) + +#define __localtry_unlock_irq(lock) \ + do { \ + localtry_lock_t *lt; \ + lt = this_cpu_ptr(lock); \ + WRITE_ONCE(lt->acquired, 0); \ + local_lock_release(<->llock); \ + local_irq_enable(); \ + } while (0) + +#define __localtry_unlock_irqrestore(lock, flags) \ + do { \ + localtry_lock_t *lt; \ + lt = this_cpu_ptr(lock); \ + WRITE_ONCE(lt->acquired, 0); \ + local_lock_release(<->llock); \ + local_irq_restore(flags); \ + } while (0) + #else /* !CONFIG_PREEMPT_RT */ /* @@ -125,8 +237,10 @@ do { \ * critical section while staying preemptible. */ typedef spinlock_t local_lock_t; +typedef spinlock_t localtry_lock_t; #define INIT_LOCAL_LOCK(lockname) __LOCAL_SPIN_LOCK_UNLOCKED((lockname)) +#define INIT_LOCALTRY_LOCK(lockname) INIT_LOCAL_LOCK(lockname) #define __local_lock_init(l) \ do { \ @@ -169,4 +283,36 @@ do { \ spin_unlock(this_cpu_ptr((lock))); \ } while (0) +/* localtry_lock_t variants */ + +#define __localtry_lock_init(lock) __local_lock_init(lock) +#define __localtry_lock(lock) __local_lock(lock) +#define __localtry_lock_irq(lock) __local_lock(lock) +#define __localtry_lock_irqsave(lock, flags) __local_lock_irqsave(lock, flags) +#define __localtry_unlock(lock) __local_unlock(lock) +#define __localtry_unlock_irq(lock) __local_unlock(lock) +#define __localtry_unlock_irqrestore(lock, flags) __local_unlock_irqrestore(lock, flags) + +#define __localtry_trylock(lock) \ + ({ \ + int __locked; \ + \ + if (in_nmi() | in_hardirq()) { \ + __locked = 0; \ + } else { \ + migrate_disable(); \ + __locked = spin_trylock(this_cpu_ptr((lock))); \ + if (!__locked) \ + migrate_enable(); \ + } \ + __locked; \ + }) + +#define __localtry_trylock_irqsave(lock, flags) \ + ({ \ + typecheck(unsigned long, flags); \ + flags = 0; \ + __localtry_trylock(lock); \ + }) + #endif /* CONFIG_PREEMPT_RT */ From patchwork Sat Feb 22 02:44:23 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13986507 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 70BB91FC7E6 for ; Sat, 22 Feb 2025 02:44:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740192284; cv=none; b=NO8xQmNX0gaPCeYuGHYFp1iN97eKPknni/Si2gpstjdelIjkOr+DUyo5tBUaDVfiUtHHGqPSzyVZ++VQFga6V8y4iP/XF8ndbxVQLGgqzICRoaDYR6Xg1qy02Flem4VxMi2SBjYZ7YkGdDPJk01leC2NBfILEjdQ2BdjabxVa68= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740192284; c=relaxed/simple; bh=8huareBgqZnv1YE/dweCJUp5CiJFrofwZ43IiNfQYhU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=oY2kpmnf9oILr/jUI2VkORNL76m/l0Q5qnL9nyzh6wBuTNgPS31DFk8kYP2OgTVeag+U6d0LxtoJrJE9K0YWvs1ouMbr7Tb40buW+EknUUfkNvnY5ZGcPbIhVP09/D3MWfr1B2Urty10gfKnvhUy80jegWumaKuOjut37MPxfq8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=hEuIb7VD; arc=none smtp.client-ip=209.85.216.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="hEuIb7VD" Received: by mail-pj1-f44.google.com with SMTP id 98e67ed59e1d1-2fbfa8c73a6so5525846a91.2 for ; Fri, 21 Feb 2025 18:44:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740192281; x=1740797081; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=oJx8bfHvcsPdmDmREOhTX7J+3gMm5tOq1v0fsOdzZbU=; b=hEuIb7VD4wqJEiEv8RxnhmytzmO7yfOYGJm9fYUFsW1ss2svIWbt9uzbaE/kngLuBW 91Ai+3uJ8enK2tA05nhy2BEkMPE+Mk1xW9v/XWSj/jvB3Bn8zLsskR7osMAjEM3ts3v+ kG0d+ko9LCIhtW/o+hnnrqQcT4goEAjkuSP2qyqfh6KQkkEfUCrh/VizcocN0Kkg+9Fa T2pOh23xwADBMQpX2viqjJfxbqw3SCxaTp0XpzqkxOngs4FKkI5AWYjE14quavnwjzGO IU5zDsAXeS5W8DOO4/lV0RRdBSA23t2+c54n1ZjrtnSxjuhgBHzRcZy9Vm8ZCk51iR0U mw2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740192281; x=1740797081; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oJx8bfHvcsPdmDmREOhTX7J+3gMm5tOq1v0fsOdzZbU=; b=nRnqLwUvk5Yh7FP20O5eyj3Kh48JgbTMHTLvGs1B00v57g6fW/cjGgELMasRMREoc9 m1jhLYs4v9mGFTgSUAIvGlJCd4Ke0YUbpK0V+u0bHwj3OKs7LyeOI5wA5d/WekFidHC8 eZx0Ug9DjBEmmBRla4F3i+06Bxh65QbOauGdDKcrnzxxrnCXvnnlJkhnAPygb76IaAoW pWz99QFbeghZSn/v1at9vg+cd+fRE5iEvbr7QzFp8E71sfrlCEUT5vZq5kK7R8rUG3OO 5Ku/MiWw0T9rlipKJmO76JalqliOiddzvwRtSJZ0+fuV+Xfirr+LHKBe0Hc5aAc/Cc07 bhPg== X-Gm-Message-State: AOJu0Ywx8Ii49L7BeQ2FnupFKQuUcopCBjMwuh69PmqG6HFOIjjLMmBr Eq6GCLhSAN0yVeW6KOGybF8IijTxnTOzFMyDrDy0m3BQf+khrikKjsSgYg== X-Gm-Gg: ASbGncvEKDVVeTh9hNGnqN5D3vQ9kNLepwQ1rvthqAKqDJeghsmFslIFiQzEJ4ythpj sswc4ex9Lzwe8LfmejouhF0tCLZ/WGyBCebYYi2Q+ersP7X3aQaXOZRphJU/kq4aFr1W7HV3VeC RIJ6dGCM19yQTMZOYbp78SsZsWitFP3gn/lpvstDsQy6nC9+e0DYz4hOUdfnq26EzYwZKN5guLZ W2DI0H+MoR9xjH/DY1FxOK0YB5oyxPdaokhyWl0nncrAqUo9Vs/tidAfe3RzG0tH+lMAFH2sFcj jM9pzfw8WfM6kSjx+9XwiEaxc37Q90phS0Nc53/6jQ0fwV9g741bpQ== X-Google-Smtp-Source: AGHT+IFCLpvNiUlzSLAmScZOHfvBMu91BobDCkKJ+lN7CcdzTuYNWTOHAeEoc/CAjS1q1J3gFUhfzQ== X-Received: by 2002:a05:6a00:98f:b0:732:1eb2:7bf3 with SMTP id d2e1a72fcca58-73426d9f1abmr9857341b3a.21.1740192280978; Fri, 21 Feb 2025 18:44:40 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:fd1b]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-73275615173sm11027845b3a.168.2025.02.21.18.44.38 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 21 Feb 2025 18:44:40 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v9 2/6] mm, bpf: Introduce try_alloc_pages() for opportunistic page allocation Date: Fri, 21 Feb 2025 18:44:23 -0800 Message-Id: <20250222024427.30294-3-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250222024427.30294-1-alexei.starovoitov@gmail.com> References: <20250222024427.30294-1-alexei.starovoitov@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net From: Alexei Starovoitov Tracing BPF programs execute from tracepoints and kprobes where running context is unknown, but they need to request additional memory. The prior workarounds were using pre-allocated memory and BPF specific freelists to satisfy such allocation requests. Instead, introduce gfpflags_allow_spinning() condition that signals to the allocator that running context is unknown. Then rely on percpu free list of pages to allocate a page. try_alloc_pages() -> get_page_from_freelist() -> rmqueue() -> rmqueue_pcplist() will spin_trylock to grab the page from percpu free list. If it fails (due to re-entrancy or list being empty) then rmqueue_bulk()/rmqueue_buddy() will attempt to spin_trylock zone->lock and grab the page from there. spin_trylock() is not safe in PREEMPT_RT when in NMI or in hard IRQ. Bailout early in such case. The support for gfpflags_allow_spinning() mode for free_page and memcg comes in the next patches. This is a first step towards supporting BPF requirements in SLUB and getting rid of bpf_mem_alloc. That goal was discussed at LSFMM: https://lwn.net/Articles/974138/ Acked-by: Michal Hocko Acked-by: Vlastimil Babka Acked-by: Sebastian Andrzej Siewior Reviewed-by: Shakeel Butt Signed-off-by: Alexei Starovoitov --- include/linux/gfp.h | 22 ++++++++++ lib/stackdepot.c | 5 ++- mm/internal.h | 1 + mm/page_alloc.c | 104 ++++++++++++++++++++++++++++++++++++++++++-- 4 files changed, 127 insertions(+), 5 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index 6bb1a5a7a4ae..5d9ee78c74e4 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -39,6 +39,25 @@ static inline bool gfpflags_allow_blocking(const gfp_t gfp_flags) return !!(gfp_flags & __GFP_DIRECT_RECLAIM); } +static inline bool gfpflags_allow_spinning(const gfp_t gfp_flags) +{ + /* + * !__GFP_DIRECT_RECLAIM -> direct claim is not allowed. + * !__GFP_KSWAPD_RECLAIM -> it's not safe to wake up kswapd. + * All GFP_* flags including GFP_NOWAIT use one or both flags. + * try_alloc_pages() is the only API that doesn't specify either flag. + * + * This is stronger than GFP_NOWAIT or GFP_ATOMIC because + * those are guaranteed to never block on a sleeping lock. + * Here we are enforcing that the allocation doesn't ever spin + * on any locks (i.e. only trylocks). There is no high level + * GFP_$FOO flag for this use in try_alloc_pages() as the + * regular page allocator doesn't fully support this + * allocation mode. + */ + return !(gfp_flags & __GFP_RECLAIM); +} + #ifdef CONFIG_HIGHMEM #define OPT_ZONE_HIGHMEM ZONE_HIGHMEM #else @@ -335,6 +354,9 @@ static inline struct page *alloc_page_vma_noprof(gfp_t gfp, } #define alloc_page_vma(...) alloc_hooks(alloc_page_vma_noprof(__VA_ARGS__)) +struct page *try_alloc_pages_noprof(int nid, unsigned int order); +#define try_alloc_pages(...) alloc_hooks(try_alloc_pages_noprof(__VA_ARGS__)) + extern unsigned long get_free_pages_noprof(gfp_t gfp_mask, unsigned int order); #define __get_free_pages(...) alloc_hooks(get_free_pages_noprof(__VA_ARGS__)) diff --git a/lib/stackdepot.c b/lib/stackdepot.c index 245d5b416699..377194969e61 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -591,7 +591,8 @@ depot_stack_handle_t stack_depot_save_flags(unsigned long *entries, depot_stack_handle_t handle = 0; struct page *page = NULL; void *prealloc = NULL; - bool can_alloc = depot_flags & STACK_DEPOT_FLAG_CAN_ALLOC; + bool allow_spin = gfpflags_allow_spinning(alloc_flags); + bool can_alloc = (depot_flags & STACK_DEPOT_FLAG_CAN_ALLOC) && allow_spin; unsigned long flags; u32 hash; @@ -630,7 +631,7 @@ depot_stack_handle_t stack_depot_save_flags(unsigned long *entries, prealloc = page_address(page); } - if (in_nmi()) { + if (in_nmi() || !allow_spin) { /* We can never allocate in NMI context. */ WARN_ON_ONCE(can_alloc); /* Best effort; bail if we fail to take the lock. */ diff --git a/mm/internal.h b/mm/internal.h index 109ef30fee11..10a8b4b3b86e 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1187,6 +1187,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone, #define ALLOC_NOFRAGMENT 0x0 #endif #define ALLOC_HIGHATOMIC 0x200 /* Allows access to MIGRATE_HIGHATOMIC */ +#define ALLOC_TRYLOCK 0x400 /* Only use spin_trylock in allocation path */ #define ALLOC_KSWAPD 0x800 /* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ /* Flags that allow allocations below the min watermark. */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 579789600a3c..1f2a4e1c70ae 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2307,7 +2307,11 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order, unsigned long flags; int i; - spin_lock_irqsave(&zone->lock, flags); + if (!spin_trylock_irqsave(&zone->lock, flags)) { + if (unlikely(alloc_flags & ALLOC_TRYLOCK)) + return 0; + spin_lock_irqsave(&zone->lock, flags); + } for (i = 0; i < count; ++i) { struct page *page = __rmqueue(zone, order, migratetype, alloc_flags); @@ -2907,7 +2911,11 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, do { page = NULL; - spin_lock_irqsave(&zone->lock, flags); + if (!spin_trylock_irqsave(&zone->lock, flags)) { + if (unlikely(alloc_flags & ALLOC_TRYLOCK)) + return NULL; + spin_lock_irqsave(&zone->lock, flags); + } if (alloc_flags & ALLOC_HIGHATOMIC) page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC); if (!page) { @@ -4511,7 +4519,12 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, might_alloc(gfp_mask); - if (should_fail_alloc_page(gfp_mask, order)) + /* + * Don't invoke should_fail logic, since it may call + * get_random_u32() and printk() which need to spin_lock. + */ + if (!(*alloc_flags & ALLOC_TRYLOCK) && + should_fail_alloc_page(gfp_mask, order)) return false; *alloc_flags = gfp_to_alloc_flags_cma(gfp_mask, *alloc_flags); @@ -7071,3 +7084,88 @@ static bool __free_unaccepted(struct page *page) } #endif /* CONFIG_UNACCEPTED_MEMORY */ + +/** + * try_alloc_pages - opportunistic reentrant allocation from any context + * @nid - node to allocate from + * @order - allocation order size + * + * Allocates pages of a given order from the given node. This is safe to + * call from any context (from atomic, NMI, and also reentrant + * allocator -> tracepoint -> try_alloc_pages_noprof). + * Allocation is best effort and to be expected to fail easily so nobody should + * rely on the success. Failures are not reported via warn_alloc(). + * See always fail conditions below. + * + * Return: allocated page or NULL on failure. + */ +struct page *try_alloc_pages_noprof(int nid, unsigned int order) +{ + /* + * Do not specify __GFP_DIRECT_RECLAIM, since direct claim is not allowed. + * Do not specify __GFP_KSWAPD_RECLAIM either, since wake up of kswapd + * is not safe in arbitrary context. + * + * These two are the conditions for gfpflags_allow_spinning() being true. + * + * Specify __GFP_NOWARN since failing try_alloc_pages() is not a reason + * to warn. Also warn would trigger printk() which is unsafe from + * various contexts. We cannot use printk_deferred_enter() to mitigate, + * since the running context is unknown. + * + * Specify __GFP_ZERO to make sure that call to kmsan_alloc_page() below + * is safe in any context. Also zeroing the page is mandatory for + * BPF use cases. + * + * Though __GFP_NOMEMALLOC is not checked in the code path below, + * specify it here to highlight that try_alloc_pages() + * doesn't want to deplete reserves. + */ + gfp_t alloc_gfp = __GFP_NOWARN | __GFP_ZERO | __GFP_NOMEMALLOC; + unsigned int alloc_flags = ALLOC_TRYLOCK; + struct alloc_context ac = { }; + struct page *page; + + /* + * In PREEMPT_RT spin_trylock() will call raw_spin_lock() which is + * unsafe in NMI. If spin_trylock() is called from hard IRQ the current + * task may be waiting for one rt_spin_lock, but rt_spin_trylock() will + * mark the task as the owner of another rt_spin_lock which will + * confuse PI logic, so return immediately if called form hard IRQ or + * NMI. + * + * Note, irqs_disabled() case is ok. This function can be called + * from raw_spin_lock_irqsave region. + */ + if (IS_ENABLED(CONFIG_PREEMPT_RT) && (in_nmi() || in_hardirq())) + return NULL; + if (!pcp_allowed_order(order)) + return NULL; + +#ifdef CONFIG_UNACCEPTED_MEMORY + /* Bailout, since try_to_accept_memory_one() needs to take a lock */ + if (has_unaccepted_memory()) + return NULL; +#endif + /* Bailout, since _deferred_grow_zone() needs to take a lock */ + if (deferred_pages_enabled()) + return NULL; + + if (nid == NUMA_NO_NODE) + nid = numa_node_id(); + + prepare_alloc_pages(alloc_gfp, order, nid, NULL, &ac, + &alloc_gfp, &alloc_flags); + + /* + * Best effort allocation from percpu free list. + * If it's empty attempt to spin_trylock zone->lock. + */ + page = get_page_from_freelist(alloc_gfp, order, alloc_flags, &ac); + + /* Unlike regular alloc_pages() there is no __alloc_pages_slowpath(). */ + + trace_mm_page_alloc(page, order, alloc_gfp, ac.migratetype); + kmsan_alloc_page(page, order, alloc_gfp); + return page; +} From patchwork Sat Feb 22 02:44:24 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13986508 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B6A761FBEB3 for ; Sat, 22 Feb 2025 02:44:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740192288; cv=none; b=rb0UoYpJgLVfRZoBsAlqnyG3ghoMdUXX9i6W/75YKD7qQTZsjDePA9yxN1Brd+CsOEjbRV0LuMFJddqp2TwAaeo40Y2PjYtjFezBCUaq7u+ZyZ20GOtZAL8rQCwvwP1WHIcPZLOgv7n4KGCD1e9y4ElmuxlwWzKQnKAoe654W28= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740192288; c=relaxed/simple; bh=gwRQ71lEugoIVTP5Evm/r7Gna/77psqYrIxADXT07VA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ptO6XuBjfXuC/TUb1TRtAAFjLpTM7WOFQ0f7xGfwvEYB3zvMQTwd/5E8/K+6/gUftNj7IzZHHEe++2SkBW/0HljQhiGW9Bf9NjCmMfwq93Q2FmKE6XzEpgpOE6APSUwlPV6ANaZ7WnjZVKi4ow4Zr/Iu3LdouST7MTGBnpyCtVM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ZgQ8ietc; arc=none smtp.client-ip=209.85.214.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ZgQ8ietc" Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-2211cd4463cso55986555ad.2 for ; Fri, 21 Feb 2025 18:44:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740192286; x=1740797086; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=o7gSAyaMDg2aK78kqVgF49+jZKrcrw3bhnHUdoMZI7o=; b=ZgQ8ietcSmaOabGztSgUq73jxfxkm4qMTfKeZfYtLHtWLALBrv0oL0rnP1vQjgl30I 85fB8uiT++sQlxxkRFhoW1+OjsC7cwaUQGtwJbQm6fNvi8nnRx13S7CsQFvGeF5tkLUy eekESMUgrl5qvx/hXD2oVgvbngAEbX1NumQiP2iwzb5pd/QHMQnppqMrUe2yLB7e8np+ 4242BlJYLzjU0H77NkUSUQXxygMCSjhmr56jd8Eoxeq3XXoBt2piJWQjql4GeHmnZyQT TrCOtAx9RGSDp/WMyTopj5o1QmY/2BdByzgLWm2i5eROrUrRp/xoYWHAqMPt5bHXslXV DS9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740192286; x=1740797086; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=o7gSAyaMDg2aK78kqVgF49+jZKrcrw3bhnHUdoMZI7o=; b=BXt4Xv0Qhn0Av16WNfVVGFCmL5y8VngTMJxbJqvrXIYgWwAMIbQfBZ2O/31Wlomc/e yGKPpdRoRd2fTcdy+bgaTHFXDUYj7G5c2YJzLUDAujlhvsCU5l5x6ODMrwh3f0tdN5Fl EYgsfawP/fojEDtrJTzdKA1e8oIqc7X8vBkOSh5nZHC2uaIH43RSOwK3iqOEJSyU+pqO J13odTxQJbbfvqRmofRXHZc5igo2sUy/DZadMaV0tP9nbTnS4zmWht6rpdJjADiH5Rt2 z1DZ0BBtlUR2+BCyxUfGDzFT5ghtQMVSzz2PJmydEfUhiHEJQrl4eR1P7ny7hEImO5Co vWvQ== X-Gm-Message-State: AOJu0YwBKwyL2JpG0M1P8sU6Ulu6fkUdUWYzE4koe3x18v+Tm/RLneHe vcDgHvZnuSwuUSDQVDo3Bdw3bUKcqUvlYtIj47KAd8VDXalRUQ7Z1BlS3Q== X-Gm-Gg: ASbGnctLVM3EitcemWMOpnsIABJGHAT3Vqut5+f81DYeI4fJTR183I1YPbneXsE/nGV rzhVavsrnUW4cGVUWJxbbub3ThHM8ELf7/OtP+r1kMSlPFodkoueSeaVhGm8Gp9snEra6wcLXgB pZc1JCMi0uSurMZGXnBykozFD6omkNQOnblLOpSOu1qXGzzA3exx5XaDTvKY7ISsluPF5jWmxJp lqcCzE96YB+HczfNLvx3UFJIz3rS4knYuSNhs6zMYKVakNq9/4r5K4VJhvlJzAkAmbq7ihtM4S0 y6fsdIWpq+K2BNd+639LqWYxEL9fNZ7Ovram4gsbMPQQvK/5hJ7Wrg== X-Google-Smtp-Source: AGHT+IH21mr57VC26OTPfaSlSbo/9B6lZeQIfI31TFaxfhjBWjCtyIwbhx9A5VtfOfNAoOWI2FAXPA== X-Received: by 2002:a05:6a20:2444:b0:1ee:d06c:cddc with SMTP id adf61e73a8af0-1eef3da4cf2mr11663031637.30.1740192285626; Fri, 21 Feb 2025 18:44:45 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:fd1b]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-73242568b03sm16979202b3a.54.2025.02.21.18.44.43 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 21 Feb 2025 18:44:45 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v9 3/6] mm, bpf: Introduce free_pages_nolock() Date: Fri, 21 Feb 2025 18:44:24 -0800 Message-Id: <20250222024427.30294-4-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250222024427.30294-1-alexei.starovoitov@gmail.com> References: <20250222024427.30294-1-alexei.starovoitov@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net From: Alexei Starovoitov Introduce free_pages_nolock() that can free pages without taking locks. It relies on trylock and can be called from any context. Since spin_trylock() cannot be used in PREEMPT_RT from hard IRQ or NMI it uses lockless link list to stash the pages which will be freed by subsequent free_pages() from good context. Do not use llist unconditionally. BPF maps continuously allocate/free, so we cannot unconditionally delay the freeing to llist. When the memory becomes free make it available to the kernel and BPF users right away if possible, and fallback to llist as the last resort. Acked-by: Vlastimil Babka Acked-by: Sebastian Andrzej Siewior Reviewed-by: Shakeel Butt Signed-off-by: Alexei Starovoitov --- include/linux/gfp.h | 1 + include/linux/mm_types.h | 4 ++ include/linux/mmzone.h | 3 ++ lib/stackdepot.c | 5 ++- mm/page_alloc.c | 90 +++++++++++++++++++++++++++++++++++----- mm/page_owner.c | 8 +++- 6 files changed, 98 insertions(+), 13 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index 5d9ee78c74e4..ceb226c2e25c 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -379,6 +379,7 @@ __meminit void *alloc_pages_exact_nid_noprof(int nid, size_t size, gfp_t gfp_mas __get_free_pages((gfp_mask) | GFP_DMA, (order)) extern void __free_pages(struct page *page, unsigned int order); +extern void free_pages_nolock(struct page *page, unsigned int order); extern void free_pages(unsigned long addr, unsigned int order); #define __free_page(page) __free_pages((page), 0) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 6b27db7f9496..483aa90242cd 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -99,6 +99,10 @@ struct page { /* Or, free page */ struct list_head buddy_list; struct list_head pcp_list; + struct { + struct llist_node pcp_llist; + unsigned int order; + }; }; /* See page-flags.h for PAGE_MAPPING_FLAGS */ struct address_space *mapping; diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 9540b41894da..e16939553930 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -972,6 +972,9 @@ struct zone { /* Primarily protects free_area */ spinlock_t lock; + /* Pages to be freed when next trylock succeeds */ + struct llist_head trylock_free_pages; + /* Write-intensive fields used by compaction and vmstats. */ CACHELINE_PADDING(_pad2_); diff --git a/lib/stackdepot.c b/lib/stackdepot.c index 377194969e61..73d7b50924ef 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -672,7 +672,10 @@ depot_stack_handle_t stack_depot_save_flags(unsigned long *entries, exit: if (prealloc) { /* Stack depot didn't use this memory, free it. */ - free_pages((unsigned long)prealloc, DEPOT_POOL_ORDER); + if (!allow_spin) + free_pages_nolock(virt_to_page(prealloc), DEPOT_POOL_ORDER); + else + free_pages((unsigned long)prealloc, DEPOT_POOL_ORDER); } if (found) handle = found->handle.handle; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 1f2a4e1c70ae..79b39ad4bb1b 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -88,6 +88,9 @@ typedef int __bitwise fpi_t; */ #define FPI_TO_TAIL ((__force fpi_t)BIT(1)) +/* Free the page without taking locks. Rely on trylock only. */ +#define FPI_TRYLOCK ((__force fpi_t)BIT(2)) + /* prevent >1 _updater_ of zone percpu pageset ->high and ->batch fields */ static DEFINE_MUTEX(pcp_batch_high_lock); #define MIN_PERCPU_PAGELIST_HIGH_FRACTION (8) @@ -1249,13 +1252,44 @@ static void split_large_buddy(struct zone *zone, struct page *page, } while (1); } +static void add_page_to_zone_llist(struct zone *zone, struct page *page, + unsigned int order) +{ + /* Remember the order */ + page->order = order; + /* Add the page to the free list */ + llist_add(&page->pcp_llist, &zone->trylock_free_pages); +} + static void free_one_page(struct zone *zone, struct page *page, unsigned long pfn, unsigned int order, fpi_t fpi_flags) { + struct llist_head *llhead; unsigned long flags; - spin_lock_irqsave(&zone->lock, flags); + if (!spin_trylock_irqsave(&zone->lock, flags)) { + if (unlikely(fpi_flags & FPI_TRYLOCK)) { + add_page_to_zone_llist(zone, page, order); + return; + } + spin_lock_irqsave(&zone->lock, flags); + } + + /* The lock succeeded. Process deferred pages. */ + llhead = &zone->trylock_free_pages; + if (unlikely(!llist_empty(llhead) && !(fpi_flags & FPI_TRYLOCK))) { + struct llist_node *llnode; + struct page *p, *tmp; + + llnode = llist_del_all(llhead); + llist_for_each_entry_safe(p, tmp, llnode, pcp_llist) { + unsigned int p_order = p->order; + + split_large_buddy(zone, p, page_to_pfn(p), p_order, fpi_flags); + __count_vm_events(PGFREE, 1 << p_order); + } + } split_large_buddy(zone, page, pfn, order, fpi_flags); spin_unlock_irqrestore(&zone->lock, flags); @@ -2599,7 +2633,7 @@ static int nr_pcp_high(struct per_cpu_pages *pcp, struct zone *zone, static void free_frozen_page_commit(struct zone *zone, struct per_cpu_pages *pcp, struct page *page, int migratetype, - unsigned int order) + unsigned int order, fpi_t fpi_flags) { int high, batch; int pindex; @@ -2634,6 +2668,14 @@ static void free_frozen_page_commit(struct zone *zone, } if (pcp->free_count < (batch << CONFIG_PCP_BATCH_SCALE_MAX)) pcp->free_count += (1 << order); + + if (unlikely(fpi_flags & FPI_TRYLOCK)) { + /* + * Do not attempt to take a zone lock. Let pcp->count get + * over high mark temporarily. + */ + return; + } high = nr_pcp_high(pcp, zone, batch, free_high); if (pcp->count >= high) { free_pcppages_bulk(zone, nr_pcp_free(pcp, batch, high, free_high), @@ -2648,7 +2690,8 @@ static void free_frozen_page_commit(struct zone *zone, /* * Free a pcp page */ -void free_frozen_pages(struct page *page, unsigned int order) +static void __free_frozen_pages(struct page *page, unsigned int order, + fpi_t fpi_flags) { unsigned long __maybe_unused UP_flags; struct per_cpu_pages *pcp; @@ -2657,7 +2700,7 @@ void free_frozen_pages(struct page *page, unsigned int order) int migratetype; if (!pcp_allowed_order(order)) { - __free_pages_ok(page, order, FPI_NONE); + __free_pages_ok(page, order, fpi_flags); return; } @@ -2675,23 +2718,33 @@ void free_frozen_pages(struct page *page, unsigned int order) migratetype = get_pfnblock_migratetype(page, pfn); if (unlikely(migratetype >= MIGRATE_PCPTYPES)) { if (unlikely(is_migrate_isolate(migratetype))) { - free_one_page(zone, page, pfn, order, FPI_NONE); + free_one_page(zone, page, pfn, order, fpi_flags); return; } migratetype = MIGRATE_MOVABLE; } + if (unlikely((fpi_flags & FPI_TRYLOCK) && IS_ENABLED(CONFIG_PREEMPT_RT) + && (in_nmi() || in_hardirq()))) { + add_page_to_zone_llist(zone, page, order); + return; + } pcp_trylock_prepare(UP_flags); pcp = pcp_spin_trylock(zone->per_cpu_pageset); if (pcp) { - free_frozen_page_commit(zone, pcp, page, migratetype, order); + free_frozen_page_commit(zone, pcp, page, migratetype, order, fpi_flags); pcp_spin_unlock(pcp); } else { - free_one_page(zone, page, pfn, order, FPI_NONE); + free_one_page(zone, page, pfn, order, fpi_flags); } pcp_trylock_finish(UP_flags); } +void free_frozen_pages(struct page *page, unsigned int order) +{ + __free_frozen_pages(page, order, FPI_NONE); +} + /* * Free a batch of folios */ @@ -2780,7 +2833,7 @@ void free_unref_folios(struct folio_batch *folios) trace_mm_page_free_batched(&folio->page); free_frozen_page_commit(zone, pcp, &folio->page, migratetype, - order); + order, FPI_NONE); } if (pcp) { @@ -4841,22 +4894,37 @@ EXPORT_SYMBOL(get_zeroed_page_noprof); * Context: May be called in interrupt context or while holding a normal * spinlock, but not in NMI context or while holding a raw spinlock. */ -void __free_pages(struct page *page, unsigned int order) +static void ___free_pages(struct page *page, unsigned int order, + fpi_t fpi_flags) { /* get PageHead before we drop reference */ int head = PageHead(page); struct alloc_tag *tag = pgalloc_tag_get(page); if (put_page_testzero(page)) - free_frozen_pages(page, order); + __free_frozen_pages(page, order, fpi_flags); else if (!head) { pgalloc_tag_sub_pages(tag, (1 << order) - 1); while (order-- > 0) - free_frozen_pages(page + (1 << order), order); + __free_frozen_pages(page + (1 << order), order, + fpi_flags); } } +void __free_pages(struct page *page, unsigned int order) +{ + ___free_pages(page, order, FPI_NONE); +} EXPORT_SYMBOL(__free_pages); +/* + * Can be called while holding raw_spin_lock or from IRQ and NMI for any + * page type (not only those that came from try_alloc_pages) + */ +void free_pages_nolock(struct page *page, unsigned int order) +{ + ___free_pages(page, order, FPI_TRYLOCK); +} + void free_pages(unsigned long addr, unsigned int order) { if (addr != 0) { diff --git a/mm/page_owner.c b/mm/page_owner.c index 2d6360eaccbb..90e31d0e3ed7 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -294,7 +294,13 @@ void __reset_page_owner(struct page *page, unsigned short order) page_owner = get_page_owner(page_ext); alloc_handle = page_owner->handle; - handle = save_stack(GFP_NOWAIT | __GFP_NOWARN); + /* + * Do not specify GFP_NOWAIT to make gfpflags_allow_spinning() == false + * to prevent issues in stack_depot_save(). + * This is similar to try_alloc_pages() gfp flags, but only used + * to signal stack_depot to avoid spin_locks. + */ + handle = save_stack(__GFP_NOWARN); __update_page_owner_free_handle(page_ext, handle, order, current->pid, current->tgid, free_ts_nsec); page_ext_put(page_ext); From patchwork Sat Feb 22 02:44:25 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13986509 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 872EB1FCFE2 for ; Sat, 22 Feb 2025 02:44:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740192293; cv=none; b=fyBITATWRZPhSbG85VW5saj0bLQIbBx2HAxDB+2DFdKY+920KBaEojjhcgvnViPyasHtg+3DXDjiZkqzeODsuVe/jE6JMTTtDVdYqnKGRwkHQJxaRqe+lGPZ1ABvnS86PZnguM64wcpKYsD9aMbQii6dysEGxAhXWSUPejERUw4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740192293; c=relaxed/simple; bh=Hq/MoataBAFy4xzlMlPj8gRlRWhLycemk1vWsLJqBP8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=tOKkvDcNoTtVdaM1H5VeerRo2fssozQACTehvx0mCIX3VtDoGgzzhtgXhj1rCz4GOjI4WF9K6IEyj4QjtVcYK5v9/dzo27wij7ReGNyV3ylRK1V8/bHQw1weP5vGH5lfTX5519WOGlWHGQIldgghO+0GYJI+ly+hlEVsRotAdP8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Swn6m+8L; arc=none smtp.client-ip=209.85.216.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Swn6m+8L" Received: by mail-pj1-f52.google.com with SMTP id 98e67ed59e1d1-2fc11834404so4414538a91.0 for ; Fri, 21 Feb 2025 18:44:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740192290; x=1740797090; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=v3GSE8xL4BCDC6o3OqMQgPMvNv4GpRTdC8gwhotmjF4=; b=Swn6m+8LVY0EN3QIfQCSQwJqF1gMFj8Iq1y2VRFMJutf2inH4/QHz4HlTe9fhEW4Fo I3g6BxA7Xb+klA5p/Anaa3b7fZHCEFDN2ferlw2ELdKbGYMMv9f6nLf0wQMWZ43es/bf BCRIRXZtOWqaq5UOyW49fExaAtXvFnkg6WxeQ7Bals0hVkbnGyiIfYUqwj0zdoyh7s0A V5o6WuRxBe6c2IsUryp81dqhqkddaGdYKEQbjej7YwqlUfWDMJGLf3FQRSsUf7D+cKj7 R4ImNI9271mVZSzm38TH0JqQxQMAdOVM+4iMfDxFKLv/4A35KxVvSnIEjqh64NwrvtIy 5hIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740192290; x=1740797090; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=v3GSE8xL4BCDC6o3OqMQgPMvNv4GpRTdC8gwhotmjF4=; b=dl6Tq0Qttz8HsCwMVgTxyStOKSpZUNpDnr0+iYxtseZLOsPyFJzO8huFzCrnsTiwDx rpVGQ8XDdnFCYwKmBoXg0vwClldBQvgmPd7lZSzNIQFUiO6TwHF/psXjQnhYUIQMxjmD HTxYgobgEtGDpSZBTk46DPigzNpgqvwpqi3tQakyQyI8o8Ix1eeQdbjw7sQ3+lNK27Re Q4Kv1gNdJiqwSBwBNU0CTDfoSSUXq1K3I6TnaeYPYPd+KvDCOexnJJMiYidPmuAfNzXq 8/1oGlmstQCvJ2LCy85n6l6AztKZgI/rQ21tMmc5R9XGzybkg3NuMSRZIUsOG1UwLPuz T0HA== X-Gm-Message-State: AOJu0YzYJOA432OGbVYyu5Eh6pqtO9cQtyr1SMxWwgt4vEiVDq3M22eb 6Xz6a0mVISxQSVgs4h01yT1k4U3Nv+o4SXUJSa6/xx8cjatyG0ck68YZ3A== X-Gm-Gg: ASbGncsZWX/rKwFZ++L8lmTyfVW+n16T08cQItvEjnh/ih+fxzaB71KIpM9FU3FqC2F edr4xnZ9Rl9jqH1/zsXQiKgPY0KP8U6VFmaO94ibHs/WYymxx8Ve/V/6WpXPgnaCaklAAZ80Lzf EZh97P/8nBRwqKsULCV+qRL/5Lhz0BHyBqKhMuXevG53wPdzhfJ5wTpt1MDZem8nOpuNwJ9tjFk NDJqkFPT0DDHhaSBq+PK8tg71MeMVzdQYK+sIfs63DiRphKR+3K1W/gGLC886CzFNY7kiEx1hH6 JJF5+rckuQc52Raf6CX4pLbG22luZnLEQnfOLYpDG9+RU8yuUk56Dg== X-Google-Smtp-Source: AGHT+IFLGHpgxyBxSlJviOmokWQ79A/ZPTGdiqZO1+sWjAAQ8SXex0TBGIOGdCqxJtQew4NhBBCvqg== X-Received: by 2002:a17:90b:2e8c:b0:2ee:d024:e4e2 with SMTP id 98e67ed59e1d1-2fce868cc3bmr9200563a91.7.1740192290163; Fri, 21 Feb 2025 18:44:50 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:fd1b]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2fceb0a0ab7sm2122738a91.47.2025.02.21.18.44.48 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 21 Feb 2025 18:44:49 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v9 4/6] memcg: Use trylock to access memcg stock_lock. Date: Fri, 21 Feb 2025 18:44:25 -0800 Message-Id: <20250222024427.30294-5-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250222024427.30294-1-alexei.starovoitov@gmail.com> References: <20250222024427.30294-1-alexei.starovoitov@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net From: Alexei Starovoitov Teach memcg to operate under trylock conditions when spinning locks cannot be used. localtry_trylock might fail and this would lead to charge cache bypass if the calling context doesn't allow spinning (gfpflags_allow_spinning). In those cases charge the memcg counter directly and fail early if that is not possible. This might cause a pre-mature charge failing but it will allow an opportunistic charging that is safe from try_alloc_pages path. Acked-by: Michal Hocko Acked-by: Vlastimil Babka Acked-by: Shakeel Butt Signed-off-by: Alexei Starovoitov --- mm/memcontrol.c | 52 ++++++++++++++++++++++++++++++++++--------------- 1 file changed, 36 insertions(+), 16 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 4de6acb9b8ec..97a7307d4932 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1739,7 +1739,7 @@ void mem_cgroup_print_oom_group(struct mem_cgroup *memcg) } struct memcg_stock_pcp { - local_lock_t stock_lock; + localtry_lock_t stock_lock; struct mem_cgroup *cached; /* this never be root cgroup */ unsigned int nr_pages; @@ -1754,7 +1754,7 @@ struct memcg_stock_pcp { #define FLUSHING_CACHED_CHARGE 0 }; static DEFINE_PER_CPU(struct memcg_stock_pcp, memcg_stock) = { - .stock_lock = INIT_LOCAL_LOCK(stock_lock), + .stock_lock = INIT_LOCALTRY_LOCK(stock_lock), }; static DEFINE_MUTEX(percpu_charge_mutex); @@ -1773,7 +1773,8 @@ static bool obj_stock_flush_required(struct memcg_stock_pcp *stock, * * returns true if successful, false otherwise. */ -static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) +static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages, + gfp_t gfp_mask) { struct memcg_stock_pcp *stock; unsigned int stock_pages; @@ -1783,7 +1784,11 @@ static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) if (nr_pages > MEMCG_CHARGE_BATCH) return ret; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + if (!localtry_trylock_irqsave(&memcg_stock.stock_lock, flags)) { + if (!gfpflags_allow_spinning(gfp_mask)) + return ret; + localtry_lock_irqsave(&memcg_stock.stock_lock, flags); + } stock = this_cpu_ptr(&memcg_stock); stock_pages = READ_ONCE(stock->nr_pages); @@ -1792,7 +1797,7 @@ static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) ret = true; } - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + localtry_unlock_irqrestore(&memcg_stock.stock_lock, flags); return ret; } @@ -1831,14 +1836,14 @@ static void drain_local_stock(struct work_struct *dummy) * drain_stock races is that we always operate on local CPU stock * here with IRQ disabled */ - local_lock_irqsave(&memcg_stock.stock_lock, flags); + localtry_lock_irqsave(&memcg_stock.stock_lock, flags); stock = this_cpu_ptr(&memcg_stock); old = drain_obj_stock(stock); drain_stock(stock); clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags); - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + localtry_unlock_irqrestore(&memcg_stock.stock_lock, flags); obj_cgroup_put(old); } @@ -1868,9 +1873,20 @@ static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) { unsigned long flags; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + if (!localtry_trylock_irqsave(&memcg_stock.stock_lock, flags)) { + /* + * In case of unlikely failure to lock percpu stock_lock + * uncharge memcg directly. + */ + if (mem_cgroup_is_root(memcg)) + return; + page_counter_uncharge(&memcg->memory, nr_pages); + if (do_memsw_account()) + page_counter_uncharge(&memcg->memsw, nr_pages); + return; + } __refill_stock(memcg, nr_pages); - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + localtry_unlock_irqrestore(&memcg_stock.stock_lock, flags); } /* @@ -2213,9 +2229,13 @@ int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, unsigned long pflags; retry: - if (consume_stock(memcg, nr_pages)) + if (consume_stock(memcg, nr_pages, gfp_mask)) return 0; + if (!gfpflags_allow_spinning(gfp_mask)) + /* Avoid the refill and flush of the older stock */ + batch = nr_pages; + if (!do_memsw_account() || page_counter_try_charge(&memcg->memsw, batch, &counter)) { if (page_counter_try_charge(&memcg->memory, batch, &counter)) @@ -2699,7 +2719,7 @@ static void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, unsigned long flags; int *bytes; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + localtry_lock_irqsave(&memcg_stock.stock_lock, flags); stock = this_cpu_ptr(&memcg_stock); /* @@ -2752,7 +2772,7 @@ static void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, if (nr) __mod_objcg_mlstate(objcg, pgdat, idx, nr); - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + localtry_unlock_irqrestore(&memcg_stock.stock_lock, flags); obj_cgroup_put(old); } @@ -2762,7 +2782,7 @@ static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) unsigned long flags; bool ret = false; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + localtry_lock_irqsave(&memcg_stock.stock_lock, flags); stock = this_cpu_ptr(&memcg_stock); if (objcg == READ_ONCE(stock->cached_objcg) && stock->nr_bytes >= nr_bytes) { @@ -2770,7 +2790,7 @@ static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) ret = true; } - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + localtry_unlock_irqrestore(&memcg_stock.stock_lock, flags); return ret; } @@ -2862,7 +2882,7 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, unsigned long flags; unsigned int nr_pages = 0; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + localtry_lock_irqsave(&memcg_stock.stock_lock, flags); stock = this_cpu_ptr(&memcg_stock); if (READ_ONCE(stock->cached_objcg) != objcg) { /* reset if necessary */ @@ -2880,7 +2900,7 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, stock->nr_bytes &= (PAGE_SIZE - 1); } - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + localtry_unlock_irqrestore(&memcg_stock.stock_lock, flags); obj_cgroup_put(old); if (nr_pages) From patchwork Sat Feb 22 02:44:26 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13986510 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5E82C1FC0FD for ; Sat, 22 Feb 2025 02:44:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740192297; cv=none; b=Rf/RFccgbEmPhagIjyBBZPOXzFYAvTUSkQJpQhTdTfWvcQXGi1a9076R8l0mge7BPebCtJMfhoiH3/XT//ezXdG/1J+D9eCK3zeKgYijwvfd01b4/Saib1Kh0LDi//4tQ+VEEVIHEINn8dfYeWRSngA39aucEdQLvlhRqyefOj8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740192297; c=relaxed/simple; bh=FOGcEJwCLduPJ0BPyEAlIHEeGXOsaEMppo5JwHbgSgQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=nvaTXFHhuDnj3CF8GMhZouqvaBjYlwXk0amKz3B0qFQ5wR8wpSJqRNmCKWJpNpnLffiZhSXoGUwBx3jdBhK5nRCa7aJq9Vzx1Gid+nBeuIe0BhbG0O9h3ufRefe6kL75g9XJpBEpX4lA5Ii+lz03NvC6iS43bIK/G+WqpZ29kCw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=HriQzGux; arc=none smtp.client-ip=209.85.214.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="HriQzGux" Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-220c4159f87so39129335ad.0 for ; Fri, 21 Feb 2025 18:44:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740192295; x=1740797095; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bolBZqmxiF6GETckqAk9OFKRadV+pg4VUXhxs/9P6QM=; b=HriQzGuxZTxD0B2xpG9x0a6Ce1gMvWf7dxPPEtL9s+283CD2uvXTimNsQSUVGjeBt1 sMkh4wUG4s/Z38zgcWShJehz9C9dvNZLJP78Krp6Y9qQ0bf77ndbWZBhERNyMaP8R0WO 2imP90L2FHvTKY80F24OrGB2ORAj+ZTOOa8UHOf+66E6grup8db9npvywJ8+2p716LFc ZnxIcKZk0VDdw6szVF8YHrC5lPizV7q625mD5U2f1mj+S0EIfgu9nMtr4IXAHVoVNjIw CCRVCl3tjmjGcb57dRfaNuywEtNvsHK9iwkuHyJwKrXGDzgBv0yMfbGt5Uory/ahAA1f zn0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740192295; x=1740797095; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bolBZqmxiF6GETckqAk9OFKRadV+pg4VUXhxs/9P6QM=; b=BMHJ6cSJaLMpeRf0PlJPJATAB1njJHnQCz0DYE8A8eUkuXb6zDaZF7kfe6SaiUWDwF EriYc40Z4QExEqxSecp9q7z/g554rdRD0MZztab/oFpOwiCqrJ8glLeW5yFJy1nKa60O 0JamUPJ8FRnreyl1Wv/Ya5oXXm0tRcfWNR1DxzC0zMxdmh2W1LHK9jWlud4+boM5AOYg sab2aBwI2uWE4WEi6pKTUlvJwiDDZ6X2zex+NKlDNtMbpeMRsv+d5ehCsy2dxEUCmJKM FxefmCrB6YUk7x85PLvqrNP4BH8YgpzjVl9X+yDx48FEkQFRcTteHE9qF7WXpK2oYMnh yUAw== X-Gm-Message-State: AOJu0YyJZngbHZiR7deFWNt+Ahn9e9zx58iCvMYBj4dpb9ghkl3XfrT6 KkDesjUKlXaP7tu9dnZdswfDFQLszguSeO08REEBOMufj7MXnj1JL+UA4A== X-Gm-Gg: ASbGncuheGROQtMIFaW8AQpcyWSuJ8df/sjr3yWADfbD8r+FL/mxBr7oPk+e4a+RCfN GXdbC1x4AIxeL9/pZqoe/zMb+ZdLbp92R4JmBYroqmzFpfsM0ovwYuliUo63vOspSRn/bMuQ7dU KCWVUq1ojWiDhnNcd5isFTbxTy/yz302dyauQrPstQwGBq8VjPs/gsSl1Ovt/da7VAPy6IgN4tS d4mTQQ99oNciJZjVoMAlGao2lcEpXAKJKQKT6unVBGTSYCHHaZJxuNk3/EAxbIOkaI5X0yNKr5X boMS3q6Vma8Xc54utl6tTBJa5EpGbmpL/g3ITOF2jpDFV7bLmHU8cA== X-Google-Smtp-Source: AGHT+IFK4i6e5Q6HRwwkypHWm+mgn/XZd3aa3wu+p1h8KExIL05px6pJ1dJj76k5LTDiVqFHNgy7uw== X-Received: by 2002:a05:6a20:7f81:b0:1ee:c6bf:7c68 with SMTP id adf61e73a8af0-1eef3c8a93emr10233940637.17.1740192294949; Fri, 21 Feb 2025 18:44:54 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:fd1b]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-ae0950ecb95sm7549891a12.41.2025.02.21.18.44.52 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 21 Feb 2025 18:44:54 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v9 5/6] mm, bpf: Use memcg in try_alloc_pages(). Date: Fri, 21 Feb 2025 18:44:26 -0800 Message-Id: <20250222024427.30294-6-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250222024427.30294-1-alexei.starovoitov@gmail.com> References: <20250222024427.30294-1-alexei.starovoitov@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net From: Alexei Starovoitov Unconditionally use __GFP_ACCOUNT in try_alloc_pages(). The caller is responsible to setup memcg correctly. All BPF memory accounting is memcg based. Acked-by: Vlastimil Babka Acked-by: Shakeel Butt Signed-off-by: Alexei Starovoitov --- mm/page_alloc.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 79b39ad4bb1b..4ad0ba7801a8 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -7189,7 +7189,8 @@ struct page *try_alloc_pages_noprof(int nid, unsigned int order) * specify it here to highlight that try_alloc_pages() * doesn't want to deplete reserves. */ - gfp_t alloc_gfp = __GFP_NOWARN | __GFP_ZERO | __GFP_NOMEMALLOC; + gfp_t alloc_gfp = __GFP_NOWARN | __GFP_ZERO | __GFP_NOMEMALLOC + | __GFP_ACCOUNT; unsigned int alloc_flags = ALLOC_TRYLOCK; struct alloc_context ac = { }; struct page *page; @@ -7233,6 +7234,11 @@ struct page *try_alloc_pages_noprof(int nid, unsigned int order) /* Unlike regular alloc_pages() there is no __alloc_pages_slowpath(). */ + if (memcg_kmem_online() && page && + unlikely(__memcg_kmem_charge_page(page, alloc_gfp, order) != 0)) { + free_pages_nolock(page, order); + page = NULL; + } trace_mm_page_alloc(page, order, alloc_gfp, ac.migratetype); kmsan_alloc_page(page, order, alloc_gfp); return page; From patchwork Sat Feb 22 02:44:27 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13986511 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5FDD91FBEB3 for ; Sat, 22 Feb 2025 02:45:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740192302; cv=none; b=OT0EMbzNOuHRf0cisU+AWb4ucDGWrnqLs8+7CQq8te/zCQNsw1yFxjThbUzyj/GspGzrrWjrPXmDuE3vNM02iiZEEDq0LQFra0/+4WMk9y+GTiypnnuQ30Qvnp/4fzJkMAzfDageHVMf46tHhAMImJehJGSU56ENoIVTb0mCMRQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740192302; c=relaxed/simple; bh=C+J4n7BDOuX93zcgob79HbEJgPw+TUEypNT+5BPLQu0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=mc8XlW8J3q+1CxkiE/xZBkzK4ukIrSoM+o9kCXZMmnSzZVpxETPX4o8zUyKzW6+eHwH7STGoVYuSZqpQZ/NMjD05sW57Y+aHIsXRqFnOf2/GuMo9BZq+6yCWCi96usUM9RPnjfvUi3LTgiBnKxUiEKW/1Y0c1R2y31ZDIN7EO4I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=MRST24dw; arc=none smtp.client-ip=209.85.214.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="MRST24dw" Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-2211cd4463cso55988115ad.2 for ; Fri, 21 Feb 2025 18:45:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740192300; x=1740797100; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SPs3yc+PiAaWT0mDird4KgOQtxAcHODpsiThplYmhyc=; b=MRST24dw7q5Jt5dt77oH3rchiBGGqMjkGoCzC3DoZTGH8m1PMawUJ8c1fVLj8WCxV/ 1drPn44FUIrEO+V8kz1dA09pk9lzt2bWcTUUM4inOjqscw/aE4FBPmdTrBObCKaTjKmE p9gjM+kC6invMgCeeMyXFh91KeEEDHxzO6SY9hLzMkaZj0Y2p/LqZfj2+lzRcm0VAbpG D1nhA751BQjsKkWKl471q6RcrRQ8FVlOeuQfIzoEDFFn7ppQhLCR0zj2t9TYRRHy4kaY Ehb6QPw4uMlh5W8012+kO/N524MshLJ7OrfreiCgQFldpw0apOx/8b3blaPi8iZko0BD Emqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740192300; x=1740797100; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SPs3yc+PiAaWT0mDird4KgOQtxAcHODpsiThplYmhyc=; b=W3n526cpzQC3wzsGF5QGxcIsX/10gEnVQlL+tV1yQDwHTCGQJrEw9tEGWPWtRU8/Mk eBAXF4rTmVQTjGy7m7i5pZUOW/qBvK/1pC3Zl9TfZDw18fHmR2ik1nkVgBbIb6rY0jZ2 Deh4yaFxMQI5b7NYBm3WPL87G8pXyU2RVGDohTTESMaJXNyF+3gDuefAosbXyTvmqQSw 6+kC9C4yrMG15CWHqeW7K62RpIAfMAt8KkfLRZSNJuXNekI4ZO++to9gHzmDSSSOfvTp Rf7OSSvHhgr4yA3SjYHUkgvz1U+ZBGguVhT8PHWvSORbXT9qNu660J2ZsKh41XW0xnMP tUnw== X-Gm-Message-State: AOJu0Ywc71/eoG0iuCFYnZWWU1ZyZ1G/dfyR2e4dK1qnDOu29iUKcR0f P1LH2g/TBKjBlG1YRIRn+XC/Ogf/G9C98DOnvmNr+2SnH3hH9uJdDwICYg== X-Gm-Gg: ASbGncsqpjv2MJBnKSgG+HEuG3DLLwZFlZjMssknKlLUqCnmlOyL+IumNE27e1ys7/i 7eIhJ5gwyESwNTpyOa6gS6xwpdZwEUZM4A+ms+yHwHeKkfBBpwZWo+Z2ATwC5b+PIBxs/DUkNDZ X1fXdpCWf2K6Hrye0lVEraFUw0hCeQeUAJ2YqDm2Js3xQcdLa3aeGDcJXw1wXSOiCK8TdhlmX9l vovPv7tR7QgVEys7ZqaU9QusxL+F8WIYmfmVkVtxmCsM9GavzGuyMO3L5atd21cmBwcbcPKNLyI JICLVaOPMmEvH4FBlc+5XL4Z4VagNh3YUE4tWG/HCvsNtzruiM3tUQ== X-Google-Smtp-Source: AGHT+IGe5ruubfI6Kv2JX5V72ftGr8+QNktHSqGRav+UGawI2sLc8KeipECS2WW3ixuo2162gZmKxQ== X-Received: by 2002:a17:903:8c5:b0:21f:4c8b:c514 with SMTP id d9443c01a7336-2219fffa9f3mr82089215ad.45.1740192299832; Fri, 21 Feb 2025 18:44:59 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:fd1b]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-220d545d43dsm144772515ad.158.2025.02.21.18.44.57 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 21 Feb 2025 18:44:59 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v9 6/6] bpf: Use try_alloc_pages() to allocate pages for bpf needs. Date: Fri, 21 Feb 2025 18:44:27 -0800 Message-Id: <20250222024427.30294-7-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250222024427.30294-1-alexei.starovoitov@gmail.com> References: <20250222024427.30294-1-alexei.starovoitov@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net From: Alexei Starovoitov Use try_alloc_pages() and free_pages_nolock() for BPF needs when context doesn't allow using normal alloc_pages. This is a prerequisite for further work. Signed-off-by: Alexei Starovoitov --- include/linux/bpf.h | 2 +- kernel/bpf/arena.c | 5 ++--- kernel/bpf/syscall.c | 23 ++++++++++++++++++++--- 3 files changed, 23 insertions(+), 7 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 15164787ce7f..aec102868b93 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -2354,7 +2354,7 @@ int generic_map_delete_batch(struct bpf_map *map, struct bpf_map *bpf_map_get_curr_or_next(u32 *id); struct bpf_prog *bpf_prog_get_curr_or_next(u32 *id); -int bpf_map_alloc_pages(const struct bpf_map *map, gfp_t gfp, int nid, +int bpf_map_alloc_pages(const struct bpf_map *map, int nid, unsigned long nr_pages, struct page **page_array); #ifdef CONFIG_MEMCG void *bpf_map_kmalloc_node(const struct bpf_map *map, size_t size, gfp_t flags, diff --git a/kernel/bpf/arena.c b/kernel/bpf/arena.c index 647b709d7d77..0d56cea71602 100644 --- a/kernel/bpf/arena.c +++ b/kernel/bpf/arena.c @@ -287,7 +287,7 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf) return VM_FAULT_SIGSEGV; /* Account into memcg of the process that created bpf_arena */ - ret = bpf_map_alloc_pages(map, GFP_KERNEL | __GFP_ZERO, NUMA_NO_NODE, 1, &page); + ret = bpf_map_alloc_pages(map, NUMA_NO_NODE, 1, &page); if (ret) { range_tree_set(&arena->rt, vmf->pgoff, 1); return VM_FAULT_SIGSEGV; @@ -465,8 +465,7 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt if (ret) goto out_free_pages; - ret = bpf_map_alloc_pages(&arena->map, GFP_KERNEL | __GFP_ZERO, - node_id, page_cnt, pages); + ret = bpf_map_alloc_pages(&arena->map, node_id, page_cnt, pages); if (ret) goto out; diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index dbd89c13dd32..28680896c6a4 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -569,7 +569,24 @@ static void bpf_map_release_memcg(struct bpf_map *map) } #endif -int bpf_map_alloc_pages(const struct bpf_map *map, gfp_t gfp, int nid, +static bool can_alloc_pages(void) +{ + return preempt_count() == 0 && !irqs_disabled() && + !IS_ENABLED(CONFIG_PREEMPT_RT); +} + +static struct page *__bpf_alloc_page(int nid) +{ + if (!can_alloc_pages()) + return try_alloc_pages(nid, 0); + + return alloc_pages_node(nid, + GFP_KERNEL | __GFP_ZERO | __GFP_ACCOUNT + | __GFP_NOWARN, + 0); +} + +int bpf_map_alloc_pages(const struct bpf_map *map, int nid, unsigned long nr_pages, struct page **pages) { unsigned long i, j; @@ -582,14 +599,14 @@ int bpf_map_alloc_pages(const struct bpf_map *map, gfp_t gfp, int nid, old_memcg = set_active_memcg(memcg); #endif for (i = 0; i < nr_pages; i++) { - pg = alloc_pages_node(nid, gfp | __GFP_ACCOUNT, 0); + pg = __bpf_alloc_page(nid); if (pg) { pages[i] = pg; continue; } for (j = 0; j < i; j++) - __free_page(pages[j]); + free_pages_nolock(pages[j], 0); ret = -ENOMEM; break; }