From patchwork Fri Jan 24 03:56:52 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13948881 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3BD5EC0218C for ; Fri, 24 Jan 2025 03:57:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BDDEF280032; Thu, 23 Jan 2025 22:57:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B8C6D28002E; Thu, 23 Jan 2025 22:57:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A041C280032; Thu, 23 Jan 2025 22:57:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 830F528002E for ; Thu, 23 Jan 2025 22:57:11 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 04952C0B86 for ; Fri, 24 Jan 2025 03:57:10 +0000 (UTC) X-FDA: 83040985062.10.5743725 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by imf01.hostedemail.com (Postfix) with ESMTP id 18E9C40008 for ; Fri, 24 Jan 2025 03:57:08 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Tsqwbsz1; spf=pass (imf01.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.216.43 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737691029; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AhWoq2FKuOuB0wGZug1HgiVNgCDD6ysBALAFLwpbX6U=; b=TgLupCroZgHqLOxkvr2LUXs96PCSpXo5Jp6OxRuhAWFCznf1Z7almgss1tOkwbqGv/DHjW rqDvB7hYIuPvRGj+cd3oimZvjjXD8VnUO7jdkWamzhM1pqEVB6fGpWblSu2wDwPBDO36JQ XDhGFg46JqDtFtgUkmr9NeJTAqbdOrI= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Tsqwbsz1; spf=pass (imf01.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.216.43 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737691029; a=rsa-sha256; cv=none; b=mC4pxFCspR6xHqupXqxZsPVwxKZtDXief5VCKOuumNiEUldc/NYj9yOajWNVoEWgHfQMVc DNSYRdF6f5FNZzguwyib/pQpuqqG5XSIr0d8fIh3Fp1WUalmazBt3dBv4CdcgfE7sf7oH1 1ASTKQ7sY7lgCjAZ/rRbhCKHwkQpxdc= Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-2eec9b3a1bbso2495829a91.3 for ; Thu, 23 Jan 2025 19:57:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1737691028; x=1738295828; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AhWoq2FKuOuB0wGZug1HgiVNgCDD6ysBALAFLwpbX6U=; b=Tsqwbsz1Xd0HaUBnBYodR6Jm7fEyN0hOtXn2zZaVtL2/EesxXDSMeCe9i4g0MmiiOz 1IxABny9qcXSH29pwmB9lTx3OU4tfaZK0XzWkSSsn2qI+8esI9QnQTFsU4/8/GyrRG/n blmOCL0w0osvjFxjHrEouyZ6dboCDRUVZHSe8RIsXzzROjD/rgpU49kvY1bjR09Mbiec iMHREbqGb2tlTfEnmB0opFQZynfclXAheGWK+10XoOcZs1I4TMgjsPSEPUiayfIEPjK+ TVv+gWPJ5LF1dNDBC9a5LT82DCYLlR5KQJsKiL+DYbYmw6oVFzj7udfVGbyvDCYzuwAi F/sA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737691028; x=1738295828; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AhWoq2FKuOuB0wGZug1HgiVNgCDD6ysBALAFLwpbX6U=; b=H+JHIN3a9G2u011Rfigm5JS002eixkzN4Yf7sFe+7zHrBy9765bRZyvGmeVyOAo20B 5lGjJE/gsW7u6Oy3Eg/GECj5EA2Z8NDNYbdVu8qrtR2qAYyww1FeVQF6J5OpcOLj1dXR cBHm9CWRjWkoEokrvfpU3a6v/X2X3f1UsO1o7dzR9aG+FzB24dJ2gNuYTuxbZ3bkXKOL XmOvx7FUDEwPgKeFua7+BkSyUW9rrV5MiH+FBHEUZ46JpTwwXx6jjjogiWweBMWB4mtN afpI86Q7cE6wl0vsN7eLixWJWDS4VSHj5ZPTI64TMHPtTqUt+NR4SVqgJqvM7hBalKg0 DEeQ== X-Forwarded-Encrypted: i=1; AJvYcCV1p0gmFTvv1m9YHN9YgWFdB8wCJIdpOMURNVwvQYFfUv5fnEdFTsUFLUPfiZS7c6/MtKO0dpKzPA==@kvack.org X-Gm-Message-State: AOJu0YzXiIAQTprZbT0Eh/BQhx81PDT26ZjfmK1ToJ+gT5+xS1cIX2/p z/4XOkN0osnRDKXMFMu/93BjiWbhtQXXP5NGNYoaE2Sc7wbHzQDL X-Gm-Gg: ASbGnctEKMwtzc8BkRNcFFiSbErp9z09sBmUFRJLcehXPQUqfOBbw+4O8PvX59FQFBf U5d9EfZoi/mTvJJiTkTAwKZAk8YmPVYVchg9Lgs5RBS6PJuXN5CmOyRgOemS6UE6L1+0xUMYAH6 ykQT/ErpFlXF3XodJlb5Yfx8JjnyXJlCxFlO6LcRhqCI/a4LuMAH+cY2dy25ivYVEPXotAV0S7Y hLUznLPRQt+gOG4HJfvOW7e5qd6Z0YOYOh8zpr4xxPTD02jgdYfyb56QLmbRY15fCy+lYzW7b4A 7A6u6i+ENQod5NcRZAcahI7sp6B5diT5zf5l6xU= X-Google-Smtp-Source: AGHT+IHjf895nN6HJ8zSuYUNSFGgzi7wNzn53xXi9fqFCiweThxJGpq1eNj/CM9sNN+YbDe1QrSLvw== X-Received: by 2002:a17:90b:3bc3:b0:2ee:f80c:6889 with SMTP id 98e67ed59e1d1-2f782d9a9eamr46529484a91.33.1737691027837; Thu, 23 Jan 2025 19:57:07 -0800 (PST) Received: from macbookpro.lan ([2603:3023:16e:5000:8af:ecd2:44cd:8027]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2f7ffaf8b34sm541383a91.30.2025.01.23.19.57.06 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 23 Jan 2025 19:57:07 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v6 3/6] locking/local_lock: Introduce local_trylock_t and local_trylock_irqsave() Date: Thu, 23 Jan 2025 19:56:52 -0800 Message-Id: <20250124035655.78899-4-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250124035655.78899-1-alexei.starovoitov@gmail.com> References: <20250124035655.78899-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 18E9C40008 X-Stat-Signature: 3mzrsa6oy316e7na38im65bunbe64nxs X-Rspam-User: X-HE-Tag: 1737691028-721034 X-HE-Meta: U2FsdGVkX19FDELhcK2/a9k9F2+3pyZfJpoB1p4yPYgdilbHW3ZaI3OW6FwbygdrYy5sDzhMKJ4pyDB2ASvLK3gxI2rdvjGYSaQRR39Xwn9tzNLI57il/7DtLc6+Jp9gEGNMfXUlTEn2j0RxaEZ5zrVW+8XqDWwmzf1CqqQsX9bLMhKKjWspyyWxvydXDD8ipjoRGgb0/vbiROI+PBkUijr+PUNG0DOlpjKrH8Fer/gxyANz4RznK/YSdiBCdEIQyGlWWH1tIqlZLiLu/OpMxCW1RnCjgiKkU7Nu2w0lfqzZey7F88POvAuS+vTAFrYtbmsCnnAevSQrqa4DnuQWCa1SVkIBFSy2Bx2JhGyGQe6SkaVNe9GaYjk8tXDoWVImVopZ0pssQdjggNmKFhls3w29ficx9+c69+umYqn8IPREZ+lKmmz0IKI/hHA+/XZ+/w6GZptWw2rnNQ4ti2zQ9iSbzfhBc5h2T6/vB2AA4Zk00d5U9v+ujFYP5fJHfddPl6McV77ChEXLCxxX4IHLV1qyyle7G9rK5tRLKkIOcXEwwn9Hnce7vYTxyRvy9NgHNMd3mweBQ5s6gQuFJnpX0JTyn4U/8z9wgQNJEP6TjLKA4aENfsnjob7Q5+KHruHTws5Fjp3cmJwUi5CiwD/1l22GTQtKxuCvea58hpImOXfM8I9CNnbEfaVclLEYrZRYm1XW22Ne9YUcYGF32bovIWIti6QC98mhF0QJ+OsD7YwJF009PhhAx/0d0uUOKmOZPy2dVsL7TP/kbmnOVhEZvL37emqqAB4sJkle4cuKvHLWeBhSUWRFDQvKqH3zvLeJg99w9z5MassKwDUZfK/VOzWSfYUaTPTzXvgbAY/FHFv8iHru9hGx1f8lBrNxs7kkYGo0QKsldTMlxpM1F8ICcP/bEFqAPJyvm9d3ncL925MztCtDPXHRkeQBHyX9LXyiiw8T6bxD0zYq6urNVpc Gt8OnxW4 SfziL4zUGuwMGs1rkU0PUVSF2r+V3uGzfR6MIJpBuwzGmS0dIbBA8BCg5nPnUbK07JvWGF0XHRNdOo4tcKMhW1SQhmrzZ4lHJWT2TQ1xDKgFTRfy/qYn9rFvBBEYw3gH112wtydwJX/sOJu5wSkTTZt7oTm/7IX1mHRqdfhwICtKIlxmErbYabSz/ZASsX48MdldONQvJP0EyClM329dIPHW/gbvGIppzYZVZjZFXaLPw/Lta9By+M/1SLNog0AFashcBUZcXSZ8IlWb9lhostNaE4NqEiYmFOMP3RzgCL9zUutqHGgAOI1CG9QldeU6YxBtdRDUXGi52OLHAKaTh9igH76sZzpSc4fQ0Kh8XcvQaP9TOoXuy6sS4MfNBv6XcoIluSRO3GXncRMWUixKu3ZiOTQwyXltNO9KxJmyDOS+2pI+yzCbGSpaGdAuWN1sVH17uwJBsZsZpIcmrB0DymUzP7ffIfh1eNkoeVsusqojIo2FCjOK8TuHx3A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov In !PREEMPT_RT local_lock_irqsave() disables interrupts to protect critical section, but it doesn't prevent NMI, so the fully reentrant code cannot use local_lock_irqsave() for exclusive access. Introduce local_trylock_t and local_trylock_irqsave() that disables interrupts and sets active=1, so local_trylock_irqsave() from NMI of the same lock will return false. In PREEMPT_RT local_lock_irqsave() maps to preemptible spin_lock(). Map local_trylock_irqsave() to preemptible spin_trylock(). When in hard IRQ or NMI return false right away, since spin_trylock() is not safe due to PI issues. Note there is no need to use local_inc for active variable, since it's a percpu variable with strict nesting scopes. Usage: local_lock_t lock; // sizeof(lock) == 0 in !RT local_lock_irqsave(&lock, ...); // irqsave as before if (local_trylock_irqsave(&lock, ...)) // compilation error local_trylock_t lock; // sizeof(lock) == 4 in !RT local_lock_irqsave(&lock, ...); // irqsave and active = 1 if (local_trylock_irqsave(&lock, ...)) // if (!active) irqsave Signed-off-by: Alexei Starovoitov --- include/linux/local_lock.h | 9 ++++ include/linux/local_lock_internal.h | 79 ++++++++++++++++++++++++++++- 2 files changed, 86 insertions(+), 2 deletions(-) diff --git a/include/linux/local_lock.h b/include/linux/local_lock.h index 091dc0b6bdfb..f4bc3e9b2b20 100644 --- a/include/linux/local_lock.h +++ b/include/linux/local_lock.h @@ -30,6 +30,15 @@ #define local_lock_irqsave(lock, flags) \ __local_lock_irqsave(lock, flags) +/** + * local_trylock_irqsave - Try to acquire a per CPU local lock, save and disable + * interrupts. Fails in PREEMPT_RT when in hard IRQ or NMI. + * @lock: The lock variable + * @flags: Storage for interrupt flags + */ +#define local_trylock_irqsave(lock, flags) \ + __local_trylock_irqsave(lock, flags) + /** * local_unlock - Release a per CPU local lock * @lock: The lock variable diff --git a/include/linux/local_lock_internal.h b/include/linux/local_lock_internal.h index 8dd71fbbb6d2..14757b7aea99 100644 --- a/include/linux/local_lock_internal.h +++ b/include/linux/local_lock_internal.h @@ -15,6 +15,19 @@ typedef struct { #endif } local_lock_t; +typedef struct { +#ifdef CONFIG_DEBUG_LOCK_ALLOC + struct lockdep_map dep_map; + struct task_struct *owner; +#endif + /* + * Same layout as local_lock_t with 'active' field + * at the end, since (local_trylock_t *) will be + * casted to (local_lock_t *). + */ + int active; +} local_trylock_t; + #ifdef CONFIG_DEBUG_LOCK_ALLOC # define LOCAL_LOCK_DEBUG_INIT(lockname) \ .dep_map = { \ @@ -31,6 +44,13 @@ static inline void local_lock_acquire(local_lock_t *l) l->owner = current; } +static inline void local_trylock_acquire(local_lock_t *l) +{ + lock_map_acquire_try(&l->dep_map); + DEBUG_LOCKS_WARN_ON(l->owner); + l->owner = current; +} + static inline void local_lock_release(local_lock_t *l) { DEBUG_LOCKS_WARN_ON(l->owner != current); @@ -45,6 +65,7 @@ static inline void local_lock_debug_init(local_lock_t *l) #else /* CONFIG_DEBUG_LOCK_ALLOC */ # define LOCAL_LOCK_DEBUG_INIT(lockname) static inline void local_lock_acquire(local_lock_t *l) { } +static inline void local_trylock_acquire(local_lock_t *l) { } static inline void local_lock_release(local_lock_t *l) { } static inline void local_lock_debug_init(local_lock_t *l) { } #endif /* !CONFIG_DEBUG_LOCK_ALLOC */ @@ -87,10 +108,37 @@ do { \ #define __local_lock_irqsave(lock, flags) \ do { \ + local_trylock_t *tl; \ + local_lock_t *l; \ local_irq_save(flags); \ - local_lock_acquire(this_cpu_ptr(lock)); \ + l = (local_lock_t *)this_cpu_ptr(lock); \ + tl = (local_trylock_t *)l; \ + _Generic((lock), \ + local_trylock_t *: ({ \ + lockdep_assert(tl->active == 0);\ + WRITE_ONCE(tl->active, 1); \ + }), \ + default:(void)0); \ + local_lock_acquire(l); \ } while (0) + +#define __local_trylock_irqsave(lock, flags) \ + ({ \ + local_trylock_t *tl; \ + local_irq_save(flags); \ + tl = this_cpu_ptr(lock); \ + if (READ_ONCE(tl->active) == 1) { \ + local_irq_restore(flags); \ + tl = NULL; \ + } else { \ + WRITE_ONCE(tl->active, 1); \ + local_trylock_acquire( \ + (local_lock_t *)tl); \ + } \ + !!tl; \ + }) + #define __local_unlock(lock) \ do { \ local_lock_release(this_cpu_ptr(lock)); \ @@ -105,7 +153,17 @@ do { \ #define __local_unlock_irqrestore(lock, flags) \ do { \ - local_lock_release(this_cpu_ptr(lock)); \ + local_trylock_t *tl; \ + local_lock_t *l; \ + l = (local_lock_t *)this_cpu_ptr(lock); \ + tl = (local_trylock_t *)l; \ + _Generic((lock), \ + local_trylock_t *: ({ \ + lockdep_assert(tl->active == 1);\ + WRITE_ONCE(tl->active, 0); \ + }), \ + default:(void)0); \ + local_lock_release(l); \ local_irq_restore(flags); \ } while (0) @@ -125,6 +183,7 @@ do { \ * critical section while staying preemptible. */ typedef spinlock_t local_lock_t; +typedef spinlock_t local_trylock_t; #define INIT_LOCAL_LOCK(lockname) __LOCAL_SPIN_LOCK_UNLOCKED((lockname)) @@ -148,6 +207,22 @@ typedef spinlock_t local_lock_t; __local_lock(lock); \ } while (0) +#define __local_trylock_irqsave(lock, flags) \ + ({ \ + __label__ out; \ + int ret = 0; \ + typecheck(unsigned long, flags); \ + flags = 0; \ + if (in_nmi() || in_hardirq()) \ + goto out; \ + migrate_disable(); \ + ret = spin_trylock(this_cpu_ptr((lock))); \ + if (!ret) \ + migrate_enable(); \ + out: \ + ret; \ + }) + #define __local_unlock(__lock) \ do { \ spin_unlock(this_cpu_ptr((__lock))); \