From patchwork Wed Jan 15 16:00:11 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13940577 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE3D3C02180 for ; Wed, 15 Jan 2025 16:00:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C59936B007B; Wed, 15 Jan 2025 11:00:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BE2406B0082; Wed, 15 Jan 2025 11:00:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A83876B0085; Wed, 15 Jan 2025 11:00:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 7E17F6B007B for ; Wed, 15 Jan 2025 11:00:22 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id F3761C14EF for ; Wed, 15 Jan 2025 16:00:21 +0000 (UTC) X-FDA: 83010148284.08.C96FEC3 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) by imf25.hostedemail.com (Postfix) with ESMTP id B282DA001F for ; Wed, 15 Jan 2025 16:00:19 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=qo5KJiNh; spf=none (imf25.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736956820; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nRUfWTA4WQmGLZMi6oqD4UCtpvNH9wADZi4y8ubTS5M=; b=1/vyEZWtF1sbkS1e4+Y/IttXcOsEodBidzEyqo4Bz7aaj992UmPiB3Z05Rnsi73LAwvbUJ sGfeT0NdKF5L5cW4djdxGUWAuXAeaZnlehlLKAOV3WlVz4CxxSqOLv5Nxw/MnSV5niRn51 sXwaVYMuaIdiAy2j0CaoBPssSMROcO8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736956820; a=rsa-sha256; cv=none; b=MnbPJVEUDlNVgMvzpD/buF7fz8zVBnTunTHSPBhEuGp36bC1rWVW/zzYE08o9XmBqJQOUK Da10jjG0I40fOUM3S5Ne0P/8nh4EJNjjQOMAIzm1si6z4vH+jIogoSoxJcq8EoKIp3X+3j 0hN/1fBQVe87rGlCsrAU+1EE5Xpo3FQ= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=qo5KJiNh; spf=none (imf25.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=nRUfWTA4WQmGLZMi6oqD4UCtpvNH9wADZi4y8ubTS5M=; b=qo5KJiNhz4OhxfsvA1aGmmEcaE SeQp7NA8HYn8LtAMqi98+votQ52qw+hfzKUAnElE2GPGRZ+PR/1rsg2be4k/7UwKtdjaLmLHdJBh0 qU/UMvicl7+njT2v4rf64VWTTsNWW5kYK3OTp1Md9UdBPmwOl2rbj9JIFOASihYAuj9xnQ98sLdo4 2aj1QdiAHX0CFdJJjZ5t7s84OmAkvj2NTI4ElRtqWhmnC61HjU/kO8Bwht0fTcTEmaAhJrWHPvN75 9bvlBfbQxb8dFKQKkj6QZcnAU4Z89S9V8/XfR7M+41TiUb13VmhD2gXDIFMatKv0YUB4PcoEE458s L+/iXJAw==; Received: from 77-249-17-89.cable.dynamic.v4.ziggo.nl ([77.249.17.89] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tY5oR-0000000AtLQ-3YnJ; Wed, 15 Jan 2025 16:00:12 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 1000) id 6882E3004DE; Wed, 15 Jan 2025 17:00:11 +0100 (CET) Date: Wed, 15 Jan 2025 17:00:11 +0100 From: Peter Zijlstra To: Suren Baghdasaryan , will@kernel.org, boqun.feng@gmail.com, mark.rutland@arm.com Cc: Mateusz Guzik , akpm@linux-foundation.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, david.laight.linux@gmail.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, richard.weiyang@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com Subject: [PATCH] refcount: Strengthen inc_not_zero() Message-ID: <20250115160011.GG8385@noisy.programming.kicks-ass.net> References: <20250111042604.3230628-1-surenb@google.com> <20250111042604.3230628-12-surenb@google.com> <20250115104841.GX5388@noisy.programming.kicks-ass.net> <20250115111334.GE8385@noisy.programming.kicks-ass.net> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20250115111334.GE8385@noisy.programming.kicks-ass.net> X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: B282DA001F X-Stat-Signature: xjw8ndoj4on6kecsmnx1hp4ctbhbfm9i X-Rspam-User: X-HE-Tag: 1736956819-271555 X-HE-Meta: U2FsdGVkX196fZXVGhMZLfN4h+KPAnfA6wntWTyzqITFHK6TkwoKVPifiNqbogIzxdVR7OQBKBSEu1IkhLygPHfxBJ+4xT30Jw7GknnavpjzmZP8SRldPF+BQIzS5ditKRn2+vjXaeGbIUy4e7BWRFHJYn183f4U4OvUTx1D0A1kT8xpCogAwQzy8kErd0TOWWfwSGgPbI5dEXwh9/yuGcuasCJUI5oeRCN/t81fZ6BM4ZBcBUxLtCLdEAWPV3Hc0lBSOaEwjsq6ySs5HECl97euwzRqlQpssJ0vElWbNrHtyHY+X85UFWp9t5e4mS2XHgRK4S2Mwexu8VF8RP95zcrHjOARSfGGRrVVCDwnzEVqSBgut4T6Nt5v5vjQ65n43DJSiLzKLUGd0Eg+xbbJ2/2tV+M/WITVeBlQLH/QeSaDw92UWxhOy2kG6sKHrkofzow6mAY9dQUr/o/1v/dDNH+I/X1Vluv+fGzglu5L8RO8ai5YWm27xreNwyW0mpHCDj3Tt+FTyTYrgF/hxJMv7xzsAyPAOzg4rguX3D5lcCRhAcxLdLQDV/mPvRO+BLvwBRvkaCY/mE+1GaowyA1D9lgWnWcNI30g17SjYNoesyBP0E3DP4wL+a6/QnP2fQBczo5l1RHOXshazE0KSBGQajLwxkfJZOagrCNYUmYz7uzF2OZtGjxy1DslKV4ZCqKw5ZlEZrp4I7+mI63y7YD98DaNNRGq69+Y5fsT3/zZn3qNfEhNNqvdvX4iaOFWOdzeVvxn/SNmsiwuiyBrqlE5RV+CGjQtJFVU90dnaCJssbxkRHVQInPlTx5jSNlI2iFBvae8se3VqfYDMNLV2YMcwWPi/zeNkZCmVPUs/YUoJ5+8n4+Wt4pzCMCAYgGg6aAboCFbNFrQrDGfKiXlQww076gioqotpbwlm3jqQdyl9EoVmTr4bPQD4KN9v4udlSgTj/MhXbwdUIWOEspItaH 8rqqYMcU tT6DKzEMIr36A1FaaqE/XLo+z5c9eivR+kmzD8OHvmQvO1j2KaCdTk7LE8zyVVk9VtS6MiPm9dT9ionKr4Rn+BSQOCUSEIt32MK8IWIHi7ynnlJMb/Ysck70DJaDbYvfkmYbULveR8e63/6OVEKgIKo7GNQmMJzpJb0eqFSTqxMVWNDjGHq4wrIFFMicQgpOEcZECtprTAt+nWpKoXfe+Mq1yOKH//VRIvgxwB3TKhkfVd4QPoK43NwGk28UzL6aVnDLVsP+4Jt1DNtUXqOjGvkrrGf38/caMsYUmJVhzRbNdZG8xPiYKwJ13jMMx9LBGWJtSOtYGSGJKldpSzlEDBNOCUeQUGtZN7yYcjl9YESEeQGNyhHuqxn/Tzw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Jan 15, 2025 at 12:13:34PM +0100, Peter Zijlstra wrote: > Notably, it means refcount_t is entirely unsuitable for anything > SLAB_TYPESAFE_BY_RCU, since they all will need secondary validation > conditions after the refcount succeeds. > > And this is probably fine, but let me ponder this all a little more. Even though SLAB_TYPESAFE_BY_RCU is relatively rare, I think we'd better fix this, these things are hard enough as they are. Will, others, wdyt? --- Subject: refcount: Strengthen inc_not_zero() For speculative lookups where a successful inc_not_zero() pins the object, but where we still need to double check if the object acquired is indeed the one we set out to aquire, needs this validation to happen *after* the increment. Notably SLAB_TYPESAFE_BY_RCU is one such an example. Signed-off-by: Peter Zijlstra (Intel) --- include/linux/refcount.h | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/include/linux/refcount.h b/include/linux/refcount.h index 35f039ecb272..340e7ffa445e 100644 --- a/include/linux/refcount.h +++ b/include/linux/refcount.h @@ -69,9 +69,10 @@ * its the lock acquire, for RCU/lockless data structures its the dependent * load. * - * Do note that inc_not_zero() provides a control dependency which will order - * future stores against the inc, this ensures we'll never modify the object - * if we did not in fact acquire a reference. + * Do note that inc_not_zero() does provide acquire order, which will order + * future load and stores against the inc, this ensures all subsequent accesses + * are from this object and not anything previously occupying this memory -- + * consider SLAB_TYPESAFE_BY_RCU. * * The decrements will provide release order, such that all the prior loads and * stores will be issued before, it also provides a control dependency, which @@ -144,7 +145,7 @@ bool __refcount_add_not_zero(int i, refcount_t *r, int *oldp) do { if (!old) break; - } while (!atomic_try_cmpxchg_relaxed(&r->refs, &old, old + i)); + } while (!atomic_try_cmpxchg_acquire(&r->refs, &old, old + i)); if (oldp) *oldp = old; @@ -225,9 +226,9 @@ static inline __must_check bool __refcount_inc_not_zero(refcount_t *r, int *oldp * Similar to atomic_inc_not_zero(), but will saturate at REFCOUNT_SATURATED * and WARN. * - * Provides no memory ordering, it is assumed the caller has guaranteed the - * object memory to be stable (RCU, etc.). It does provide a control dependency - * and thereby orders future stores. See the comment on top. + * Provides acquire ordering, such that subsequent accesses are after the + * increment. This is important for the cases where secondary validation is + * required, eg. SLAB_TYPESAFE_BY_RCU. * * Return: true if the increment was successful, false otherwise */