From patchwork Mon Feb 3 15:05:22 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 13957694 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 65AEA209671; Mon, 3 Feb 2025 15:05:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738595136; cv=none; b=QR8P2Chvy7qM4GEHAOjjei3XoZ+7ZDOgwWP7c2dKvWE20z9CmYGwDd/nN0MeDLgXV6oIEtJQURjP9s/w0AGCwPjMYWlBOh6rR3BFB/8PgxxWW0gVKDqhdoYY/2H7r7Ze70fac+LjcS94hZtq+Cd/eweyK4DMevnqfRxEsvUUyS8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738595136; c=relaxed/simple; bh=m3uESZrSRKPioSjw60MOYzlOJg/046vKL6JeMzoirko=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=AQ0wn0OaOqENGNjn80AruP3tk/ce9i7unr9RjXREK0KK0TTHk9hbg/B4wMQ8dFJ6PxhCoQDQHo2f/DS/xngtG/7gNv+y5tHpRh3Ob/VRWScupUaIrTEaom/2nxQ78UzcU3ucNoLIZ6prYei2K5IVzGmyp5u2ck/dbGpFIRYBKDY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=jC7I1Xiq; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=+O03bFMT; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="jC7I1Xiq"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="+O03bFMT" From: Sebastian Andrzej Siewior DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1738595132; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=V3vC5qYMswUoSmlILzMPlQgwHaBzq1Rws+HGd2K8t+A=; b=jC7I1XiqPX6GgcCNL9KvmCzrW5mo8A24xJ24iIsR9rJdz049U6sTa+xuNH9oDntVfJglL5 2JUwqDsIpwD5Jqi6wUQhZ+/ZL07y8Ygl1+qrONju5i33hEYOtApLsMOHhkftax+UcEAktY pxm5FSa01yqCzIrfqKbWx7L6ZhtTFqFgHfu1mfNCcIfvqYziPblO7PspcPraoW3hnWRBao yHGyYF+3gU/+69jtvvMcAnBbvoNF69urDYfE4do3qbCVnEvq022l4RdrPc3nN6VBd4m/aX mPDcHRUYjSmgJnPd0CMHSSM/ubLqoTuS0gYZMKOzXXfIpYiAzl/63D+2pHVyPQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1738595132; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=V3vC5qYMswUoSmlILzMPlQgwHaBzq1Rws+HGd2K8t+A=; b=+O03bFMTxDKlsMaJIocTGtCulsut/refI1/IpuB794SKR+pSDPZAD9QkEK1jCCPKq3oF6Z 0Je4tF0VCiU5qQCg== To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Andrew Morton , MengEn Sun , Thomas Gleixner , YueHong Wu , "Paul E. McKenney" , Joel Fernandes , Josh Triplett , Boqun Feng , Uladzislau Rezki , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Sebastian Andrzej Siewior Subject: [PATCH 1/4] rcu: Provide a static initializer for hlist_nulls_head. Date: Mon, 3 Feb 2025 16:05:22 +0100 Message-ID: <20250203150525.456525-2-bigeasy@linutronix.de> In-Reply-To: <20250203150525.456525-1-bigeasy@linutronix.de> References: <20250203150525.456525-1-bigeasy@linutronix.de> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Provide a static initializer for hlist_nulls_head so that it can be used in statically defined data structures. Signed-off-by: Sebastian Andrzej Siewior --- include/linux/list_nulls.h | 1 + 1 file changed, 1 insertion(+) diff --git a/include/linux/list_nulls.h b/include/linux/list_nulls.h index fa6e8471bd227..248db9b77ee24 100644 --- a/include/linux/list_nulls.h +++ b/include/linux/list_nulls.h @@ -28,6 +28,7 @@ struct hlist_nulls_node { #define NULLS_MARKER(value) (1UL | (((long)value) << 1)) #define INIT_HLIST_NULLS_HEAD(ptr, nulls) \ ((ptr)->first = (struct hlist_nulls_node *) NULLS_MARKER(nulls)) +#define HLIST_NULLS_HEAD_INIT(nulls) {.first = (struct hlist_nulls_node *)NULLS_MARKER(nulls)} #define hlist_nulls_entry(ptr, type, member) container_of(ptr,type,member) From patchwork Mon Feb 3 15:05:23 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 13957695 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 830D2209677; Mon, 3 Feb 2025 15:05:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738595136; cv=none; b=oHjgyI6imx4tOcNzG2TlG6U0Uhaq7fFNeW+Iae+FZxMrFaCt4xUOdhbi8l2q31DF8+gvFADQH77Vhh2Av9FUpRkKtXrk9j2mlHdFDEp7rEqPrxi4hrf4bUmSyS8e1HlwHO0/J2BPNUiaFdEkEPhHvXxYSLglUi+9jFRmckQVHhA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738595136; c=relaxed/simple; bh=NtVEvi5o+2Y5THcCCOoGtQ1CfQwhYnEubu5wZVXHpt0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=EX0X0VPjVegnXEanhB0md5QoKONBEMNWehG2egOpn1xhXa4Sc9UzzWZUc5ldvL4YHCFq1tmOQQsZIQ2ZlTPy9VsXBHiJzQcsRA3kGrzCSLbi+EThP25G/xtur2b0G99hJUDyjDm9z4u6R9eMYajG/CAUu4y/L5UVTuNx1kYGA4E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=upFoTsQ5; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=ffdWqGo8; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="upFoTsQ5"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="ffdWqGo8" From: Sebastian Andrzej Siewior DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1738595132; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FgDvhm/mNA7fkgOGrealCnDdq680fG6HjgiYD8iSCJw=; b=upFoTsQ5Z0RB/yHDpPEDCeLaIxtvP4eXCtigHDmPRJBNkG4LpCTG8out6dq2A6rWYI5wXq Her978ZQIkhVX4YcH07EtCpRXSpqtY8LsPQli/Avpmbu69YVqm0Y2JHD4MSwghTjde1xy3 dxH72Pqn/z0o5NBLcrhY4VdEYE4F7Wc5rKRDyKtQw34Kw9odl/h7wO6ZIV5eWRG9RjvjH/ AM8woHboRsX9og8nF1c8t+PlSEhm8kY9FdK353DsBu+bqjleM34vdy8qhmLz9nFCNRlSHn isDKrLeOrNRBbmDhW5eStX7v/OAp9Tr63bsNJh6HdA1XSt48bEMICanLN5n+fg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1738595132; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FgDvhm/mNA7fkgOGrealCnDdq680fG6HjgiYD8iSCJw=; b=ffdWqGo8I4UCQin0vD+hOTdKMIHMmKbB0iVsYHTerwXm8KVNMBMvYSSBn1sNS/13zAP/ON ls/hEhruu8yMz8DA== To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Andrew Morton , MengEn Sun , Thomas Gleixner , YueHong Wu , "Paul E. McKenney" , Joel Fernandes , Josh Triplett , Boqun Feng , Uladzislau Rezki , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Sebastian Andrzej Siewior Subject: [PATCH 2/4] ucount: Replace get_ucounts_or_wrap() with atomic_inc_not_zero(). Date: Mon, 3 Feb 2025 16:05:23 +0100 Message-ID: <20250203150525.456525-3-bigeasy@linutronix.de> In-Reply-To: <20250203150525.456525-1-bigeasy@linutronix.de> References: <20250203150525.456525-1-bigeasy@linutronix.de> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 get_ucounts_or_wrap() increments the counter and if the counter is negative then it decrements it again in order to reset the previous increment. This statement can be replaced with atomic_inc_not_zero() to only increment the counter if it is not yet 0. This simplifies the get function because the put (if the get failed) can be removed. atomic_inc_not_zero() is implement as a cmpxchg() loop which can be repeated several times if another get/put is performed in parallel. This will be optimized later. Increment the reference counter only if not yet dropped to zero. Signed-off-by: Sebastian Andrzej Siewior --- kernel/ucount.c | 24 ++++++------------------ 1 file changed, 6 insertions(+), 18 deletions(-) diff --git a/kernel/ucount.c b/kernel/ucount.c index 86c5f1c0bad90..4aa5011538257 100644 --- a/kernel/ucount.c +++ b/kernel/ucount.c @@ -146,25 +146,16 @@ static void hlist_add_ucounts(struct ucounts *ucounts) spin_unlock_irq(&ucounts_lock); } -static inline bool get_ucounts_or_wrap(struct ucounts *ucounts) -{ - /* Returns true on a successful get, false if the count wraps. */ - return !atomic_add_negative(1, &ucounts->count); -} - struct ucounts *get_ucounts(struct ucounts *ucounts) { - if (!get_ucounts_or_wrap(ucounts)) { - put_ucounts(ucounts); - ucounts = NULL; - } - return ucounts; + if (atomic_inc_not_zero(&ucounts->count)) + return ucounts; + return NULL; } struct ucounts *alloc_ucounts(struct user_namespace *ns, kuid_t uid) { struct hlist_head *hashent = ucounts_hashentry(ns, uid); - bool wrapped; struct ucounts *ucounts, *new = NULL; spin_lock_irq(&ucounts_lock); @@ -189,14 +180,11 @@ struct ucounts *alloc_ucounts(struct user_namespace *ns, kuid_t uid) return new; } } - - wrapped = !get_ucounts_or_wrap(ucounts); + if (!atomic_inc_not_zero(&ucounts->count)) + ucounts = NULL; spin_unlock_irq(&ucounts_lock); kfree(new); - if (wrapped) { - put_ucounts(ucounts); - return NULL; - } + return ucounts; } From patchwork Mon Feb 3 15:05:24 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 13957698 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6C1BD2080F9; Mon, 3 Feb 2025 15:05:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738595138; cv=none; b=uLJLP/4POFy2gfy1dhFN1iStKFleIjtnUYNSlp6fj83SmLMTUNmjtjbF/4FDxMxaRzgfu4tjzhIG+3dL9P5/WQqEFIdb4jxP7GRlN++lvupJVdTL1qezbU+MkY4jX+80QZcVhx7Lar93mhfCu/3P379+8jkab+thTQApAp7PL0Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738595138; c=relaxed/simple; bh=tKnh+kKottUjOUdTWNtI3LHAb+KtZ0BffGkQSU7pKrA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Xr+eAx0ef618AYJtLwdkbWU8UHrESY+HBPfQoYD6pLtInnZDyJcYndSIK+cQkvRPY7p0oAs5xShktatZby6PZ0klBcusy5U/zYcfqYtQNFxkqf5+GA5sNOCU+brePoOYX8Br1ycBdpzoeVYRa4WRVXSMKcl3cuFDcmcbgTM/pao= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=0ZbbXXFr; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=Ggj5AKXh; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="0ZbbXXFr"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="Ggj5AKXh" From: Sebastian Andrzej Siewior DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1738595133; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=F/fZu6xb+Szg+d8vtwSZnj4vBZAZohI+wxhZ6Q3N7B8=; b=0ZbbXXFrwenSMc6uxmEYuSRqnbNnDttcLnabAinX0KuKsGrcrgqG9zc11Kj1R5PI4CHn4E 3ivBYxviO60jpMWVu70aXJPaWuWM+n3ZbNwCXCkLKeG6kggkKFhnz3T6OzypRH1f+oNarF 2II3IFvw3EpmNP/ipqWHdNAlCGqB+PaJrzYkIxRWTTsZ0JU1Voa/UYc+TC3dm8iG7iaNbR mKKNVZ9vZzLa/GpHCNJEMd1Ufkma6pa2CPOq0jlWaTZJqqyUVCTmemPZiYPXV5mUAP4UEu DpZJYxMw2WLqmWSJrFzhrK1QIGZEd6d5XX80+o4PDAm/RQrX+Qsn0Gc15oTqEw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1738595133; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=F/fZu6xb+Szg+d8vtwSZnj4vBZAZohI+wxhZ6Q3N7B8=; b=Ggj5AKXhGFY/+u9Ea9vNMSv10xKbJagbXN+6zZwqrWuIOSfKtO7wlgnhTsmyz05IO2D8fR 0VgvTUm99PKBHKBA== To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Andrew Morton , MengEn Sun , Thomas Gleixner , YueHong Wu , "Paul E. McKenney" , Joel Fernandes , Josh Triplett , Boqun Feng , Uladzislau Rezki , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Sebastian Andrzej Siewior Subject: [PATCH 3/4] ucount: Use RCU for ucounts lookups. Date: Mon, 3 Feb 2025 16:05:24 +0100 Message-ID: <20250203150525.456525-4-bigeasy@linutronix.de> In-Reply-To: <20250203150525.456525-1-bigeasy@linutronix.de> References: <20250203150525.456525-1-bigeasy@linutronix.de> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 The ucounts element is looked up under ucounts_lock. This can be optimized by using RCU for a lockless lookup and return and element if the reference can be obtained. Replace hlist_head with hlist_nulls_head which is RCU compatible. Let find_ucounts() search for the required item within a RCU section and return the item if a reference could be obtained. This means alloc_ucounts() will always return an element (unless the memory allocation failed). Let put_ucounts() RCU free the element if the reference counter dropped to zero. Signed-off-by: Sebastian Andrzej Siewior --- include/linux/user_namespace.h | 4 +- kernel/ucount.c | 75 ++++++++++++++++++---------------- 2 files changed, 43 insertions(+), 36 deletions(-) diff --git a/include/linux/user_namespace.h b/include/linux/user_namespace.h index 7183e5aca2829..ad4dbef92597b 100644 --- a/include/linux/user_namespace.h +++ b/include/linux/user_namespace.h @@ -5,6 +5,7 @@ #include #include #include +#include #include #include #include @@ -115,9 +116,10 @@ struct user_namespace { } __randomize_layout; struct ucounts { - struct hlist_node node; + struct hlist_nulls_node node; struct user_namespace *ns; kuid_t uid; + struct rcu_head rcu; atomic_t count; atomic_long_t ucount[UCOUNT_COUNTS]; atomic_long_t rlimit[UCOUNT_RLIMIT_COUNTS]; diff --git a/kernel/ucount.c b/kernel/ucount.c index 4aa5011538257..b6abaf68cdccb 100644 --- a/kernel/ucount.c +++ b/kernel/ucount.c @@ -15,7 +15,10 @@ struct ucounts init_ucounts = { }; #define UCOUNTS_HASHTABLE_BITS 10 -static struct hlist_head ucounts_hashtable[(1 << UCOUNTS_HASHTABLE_BITS)]; +#define UCOUNTS_HASHTABLE_ENTRIES (1 << UCOUNTS_HASHTABLE_BITS) +static struct hlist_nulls_head ucounts_hashtable[UCOUNTS_HASHTABLE_ENTRIES] = { + [0 ... UCOUNTS_HASHTABLE_ENTRIES - 1] = HLIST_NULLS_HEAD_INIT(0) +}; static DEFINE_SPINLOCK(ucounts_lock); #define ucounts_hashfn(ns, uid) \ @@ -24,7 +27,6 @@ static DEFINE_SPINLOCK(ucounts_lock); #define ucounts_hashentry(ns, uid) \ (ucounts_hashtable + ucounts_hashfn(ns, uid)) - #ifdef CONFIG_SYSCTL static struct ctl_table_set * set_lookup(struct ctl_table_root *root) @@ -127,22 +129,28 @@ void retire_userns_sysctls(struct user_namespace *ns) #endif } -static struct ucounts *find_ucounts(struct user_namespace *ns, kuid_t uid, struct hlist_head *hashent) +static struct ucounts *find_ucounts(struct user_namespace *ns, kuid_t uid, + struct hlist_nulls_head *hashent) { struct ucounts *ucounts; + struct hlist_nulls_node *pos; - hlist_for_each_entry(ucounts, hashent, node) { - if (uid_eq(ucounts->uid, uid) && (ucounts->ns == ns)) - return ucounts; + guard(rcu)(); + hlist_nulls_for_each_entry_rcu(ucounts, pos, hashent, node) { + if (uid_eq(ucounts->uid, uid) && (ucounts->ns == ns)) { + if (atomic_inc_not_zero(&ucounts->count)) + return ucounts; + } } return NULL; } static void hlist_add_ucounts(struct ucounts *ucounts) { - struct hlist_head *hashent = ucounts_hashentry(ucounts->ns, ucounts->uid); + struct hlist_nulls_head *hashent = ucounts_hashentry(ucounts->ns, ucounts->uid); + spin_lock_irq(&ucounts_lock); - hlist_add_head(&ucounts->node, hashent); + hlist_nulls_add_head_rcu(&ucounts->node, hashent); spin_unlock_irq(&ucounts_lock); } @@ -155,37 +163,33 @@ struct ucounts *get_ucounts(struct ucounts *ucounts) struct ucounts *alloc_ucounts(struct user_namespace *ns, kuid_t uid) { - struct hlist_head *hashent = ucounts_hashentry(ns, uid); - struct ucounts *ucounts, *new = NULL; + struct hlist_nulls_head *hashent = ucounts_hashentry(ns, uid); + struct ucounts *ucounts, *new; + + ucounts = find_ucounts(ns, uid, hashent); + if (ucounts) + return ucounts; + + new = kzalloc(sizeof(*new), GFP_KERNEL); + if (!new) + return NULL; + + new->ns = ns; + new->uid = uid; + atomic_set(&new->count, 1); spin_lock_irq(&ucounts_lock); ucounts = find_ucounts(ns, uid, hashent); - if (!ucounts) { + if (ucounts) { spin_unlock_irq(&ucounts_lock); - - new = kzalloc(sizeof(*new), GFP_KERNEL); - if (!new) - return NULL; - - new->ns = ns; - new->uid = uid; - atomic_set(&new->count, 1); - - spin_lock_irq(&ucounts_lock); - ucounts = find_ucounts(ns, uid, hashent); - if (!ucounts) { - hlist_add_head(&new->node, hashent); - get_user_ns(new->ns); - spin_unlock_irq(&ucounts_lock); - return new; - } + kfree(new); + return ucounts; } - if (!atomic_inc_not_zero(&ucounts->count)) - ucounts = NULL; - spin_unlock_irq(&ucounts_lock); - kfree(new); - return ucounts; + hlist_nulls_add_head_rcu(&new->node, hashent); + get_user_ns(new->ns); + spin_unlock_irq(&ucounts_lock); + return new; } void put_ucounts(struct ucounts *ucounts) @@ -193,10 +197,11 @@ void put_ucounts(struct ucounts *ucounts) unsigned long flags; if (atomic_dec_and_lock_irqsave(&ucounts->count, &ucounts_lock, flags)) { - hlist_del_init(&ucounts->node); + hlist_nulls_del_rcu(&ucounts->node); spin_unlock_irqrestore(&ucounts_lock, flags); + put_user_ns(ucounts->ns); - kfree(ucounts); + kfree_rcu(ucounts, rcu); } } From patchwork Mon Feb 3 15:05:25 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 13957696 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6C269209669; Mon, 3 Feb 2025 15:05:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738595137; cv=none; b=cLSieN1PS1LudNqVG8Y8BjR5r+0cWdZD7gNdfoI+Gcl4uZOx/5NgGe1XaphEp9KK1GQLMwkZ5ZWTT7eFEAyiVW9dXp9VKFULCXYq4myuwa27wBn/ZKkBsBkNndQrMaEi3FYbiPDaEqwfyinZrFDSKdw/aLue/QgPUOztczjAqzc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738595137; c=relaxed/simple; bh=OK+jjP1BbZyIi0ssNy/G6Co6NiIxx8Mftl1ZXIchREU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kG1QBjap4n97FTtmvyaObxcdAXZ4YyyP7jIoLIIIpywv2frop93zIr7y8z2FvOAtbQnOlRELT3FQjlub4+H0NQnoNOPJSBiBs5bJwcy+4/wPYUlz58W/nySq72N5Pcb2dl9MT8EQJYTggoe1vZsfQwNqKRAtLlPjxtQr70Q4zUI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=Xl3Y7N2g; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=FCsnnDqD; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="Xl3Y7N2g"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="FCsnnDqD" From: Sebastian Andrzej Siewior DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1738595133; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=g/YC+WrCwCy8jU9IgZ0UBmsm9LE5suKhW48TMzGwyHg=; b=Xl3Y7N2gm9AWBuNZWaFwtrshtQeqFuVHdAaTnEzmyCudnVivxTw/C9kNuyfwDIH2Cn/Qts pskpB74IsS1dR4N17fYXX1CH5dUyB1pQ8BkjwDTQipNmPHje498nNipUZo0I04P63m9WFa 21wZZlktX9yOgW3+WyTeERN41fdLs8BhHxWY7OvFoV6F2DRZB409O5MAqo6hTQL8fnqBLE E9VILjj9H2eIAfv6kePJMyOAqn8WPtwjFPGmD9e1qeeZ5VdNEswaHb4LdPtZS4Bj1UiqSq 1oZkq+lBh4XuKBdztjdCfHthpcKqFKfrMcHCOMRP0a+NRH8gozVvdrIrIcACeg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1738595133; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=g/YC+WrCwCy8jU9IgZ0UBmsm9LE5suKhW48TMzGwyHg=; b=FCsnnDqDI7I4cPhfYeSHNQz7gD7v8pFMIbnCHIhJHQLZ1yyLHX+3GZz1BVf8v8v2b7YKJf Lt4GI8bcx8anhZBg== To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Andrew Morton , MengEn Sun , Thomas Gleixner , YueHong Wu , "Paul E. McKenney" , Joel Fernandes , Josh Triplett , Boqun Feng , Uladzislau Rezki , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Sebastian Andrzej Siewior Subject: [PATCH 4/4] ucount: Use rcuref_t for reference counting. Date: Mon, 3 Feb 2025 16:05:25 +0100 Message-ID: <20250203150525.456525-5-bigeasy@linutronix.de> In-Reply-To: <20250203150525.456525-1-bigeasy@linutronix.de> References: <20250203150525.456525-1-bigeasy@linutronix.de> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Use rcuref_t for reference counting. This eliminates the cmpxchg loop in the get and put path. This also eliminates the need to acquire the lock in the put path because once the final user returns the reference, it can no longer be obtained anymore. Use rcuref_t for reference counting. Signed-off-by: Sebastian Andrzej Siewior --- include/linux/user_namespace.h | 11 +++++++++-- kernel/ucount.c | 16 +++++----------- 2 files changed, 14 insertions(+), 13 deletions(-) diff --git a/include/linux/user_namespace.h b/include/linux/user_namespace.h index ad4dbef92597b..a0bb6d0121378 100644 --- a/include/linux/user_namespace.h +++ b/include/linux/user_namespace.h @@ -8,6 +8,7 @@ #include #include #include +#include #include #include #include @@ -120,7 +121,7 @@ struct ucounts { struct user_namespace *ns; kuid_t uid; struct rcu_head rcu; - atomic_t count; + rcuref_t count; atomic_long_t ucount[UCOUNT_COUNTS]; atomic_long_t rlimit[UCOUNT_RLIMIT_COUNTS]; }; @@ -133,9 +134,15 @@ void retire_userns_sysctls(struct user_namespace *ns); struct ucounts *inc_ucount(struct user_namespace *ns, kuid_t uid, enum ucount_type type); void dec_ucount(struct ucounts *ucounts, enum ucount_type type); struct ucounts *alloc_ucounts(struct user_namespace *ns, kuid_t uid); -struct ucounts * __must_check get_ucounts(struct ucounts *ucounts); void put_ucounts(struct ucounts *ucounts); +static inline struct ucounts * __must_check get_ucounts(struct ucounts *ucounts) +{ + if (rcuref_get(&ucounts->count)) + return ucounts; + return NULL; +} + static inline long get_rlimit_value(struct ucounts *ucounts, enum rlimit_type type) { return atomic_long_read(&ucounts->rlimit[type]); diff --git a/kernel/ucount.c b/kernel/ucount.c index b6abaf68cdccb..8686e329b8f2c 100644 --- a/kernel/ucount.c +++ b/kernel/ucount.c @@ -11,7 +11,7 @@ struct ucounts init_ucounts = { .ns = &init_user_ns, .uid = GLOBAL_ROOT_UID, - .count = ATOMIC_INIT(1), + .count = RCUREF_INIT(1), }; #define UCOUNTS_HASHTABLE_BITS 10 @@ -138,7 +138,7 @@ static struct ucounts *find_ucounts(struct user_namespace *ns, kuid_t uid, guard(rcu)(); hlist_nulls_for_each_entry_rcu(ucounts, pos, hashent, node) { if (uid_eq(ucounts->uid, uid) && (ucounts->ns == ns)) { - if (atomic_inc_not_zero(&ucounts->count)) + if (rcuref_get(&ucounts->count)) return ucounts; } } @@ -154,13 +154,6 @@ static void hlist_add_ucounts(struct ucounts *ucounts) spin_unlock_irq(&ucounts_lock); } -struct ucounts *get_ucounts(struct ucounts *ucounts) -{ - if (atomic_inc_not_zero(&ucounts->count)) - return ucounts; - return NULL; -} - struct ucounts *alloc_ucounts(struct user_namespace *ns, kuid_t uid) { struct hlist_nulls_head *hashent = ucounts_hashentry(ns, uid); @@ -176,7 +169,7 @@ struct ucounts *alloc_ucounts(struct user_namespace *ns, kuid_t uid) new->ns = ns; new->uid = uid; - atomic_set(&new->count, 1); + rcuref_init(&new->count, 1); spin_lock_irq(&ucounts_lock); ucounts = find_ucounts(ns, uid, hashent); @@ -196,7 +189,8 @@ void put_ucounts(struct ucounts *ucounts) { unsigned long flags; - if (atomic_dec_and_lock_irqsave(&ucounts->count, &ucounts_lock, flags)) { + if (rcuref_put(&ucounts->count)) { + spin_lock_irqsave(&ucounts_lock, flags); hlist_nulls_del_rcu(&ucounts->node); spin_unlock_irqrestore(&ucounts_lock, flags);