From patchwork Thu Feb 13 22:46:47 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13974138 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C911EC021A4 for ; Thu, 13 Feb 2025 22:47:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B0ACA28000E; Thu, 13 Feb 2025 17:47:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A911E280001; Thu, 13 Feb 2025 17:47:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 86EE228000E; Thu, 13 Feb 2025 17:47:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 5C39E280001 for ; Thu, 13 Feb 2025 17:47:23 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 167381416D8 for ; Thu, 13 Feb 2025 22:47:23 +0000 (UTC) X-FDA: 83116409166.27.C45101A Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf13.hostedemail.com (Postfix) with ESMTP id 3EC062000F for ; Thu, 13 Feb 2025 22:47:21 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=DDnPziv8; spf=pass (imf13.hostedemail.com: domain of 3eHauZwYKCGMTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3eHauZwYKCGMTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739486841; a=rsa-sha256; cv=none; b=2HJDUH/sJky9UYIEuH+THlabhAX3CnGzOtzGu7yQNMWoTXIKA3NNRAoMB5yMntxt67Tk/t QAFbwf6j3/wbNgkFn3+Nh1ue/r8uODDt6MGIvWEo9N/JOxgDxIxi8wmj4XhPLuDKYuKkKt JqKaVw4cPqtwsGcn3h0jj+T3R7mF7jQ= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=DDnPziv8; spf=pass (imf13.hostedemail.com: domain of 3eHauZwYKCGMTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3eHauZwYKCGMTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739486841; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3RwEBR7WcDA6C9/htn1wwdAW8BOhyJsqS310jozD7lM=; b=jtoRJKWkPxQPgx5UZ8CpqXIi0xo/665S0MBcFAUXNEDT8P0v8GGcaHgC1T2Yq4K19eXmmb pWWjCYrEX87zn/Ah3xgajbr2Mia0agA2o4WYmoytPA+9UbphpFWs8mxpTbrZtmNavALCLe N7b3uwGtajdA8UrHucWhRC9iLvyd2m4= Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-220cd43c75aso55463785ad.3 for ; Thu, 13 Feb 2025 14:47:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1739486840; x=1740091640; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=3RwEBR7WcDA6C9/htn1wwdAW8BOhyJsqS310jozD7lM=; b=DDnPziv8fGNxM89rLyWv39XhlG9JoniaFrYt36gEMn27prKe76wD0dda7BlpqwYbId tCYOAac2SbfHPFGNOHWmkUfIBno5Xur3Ya6gGpe4KgFlFuuLypPLVrdyXarIH/r0c8oI K05cvy/iYhxMIwkCFLhVl6bkL7HKDxcxfDHhSSlCkMdODVunyJM8b5e3d4nSlmnX8oFO VekZpPeJ7YGRcqaSwRcM7iofx24CA+tWbSztHNVn834Bzx2j93gVaOoQctRktBJfLOaW 5M8ZpcgTFZxQMXu8wTqy+agamGuOj1edjwtpjAbyQYX2fE9O5S+18IVZIaJvB2uSYvdM TzNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739486840; x=1740091640; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=3RwEBR7WcDA6C9/htn1wwdAW8BOhyJsqS310jozD7lM=; b=wwQWVri7dU6X3rxGXe2m89XFmJ2oXQvNwR+Fq+u/tnT2y3T7GPwmqPI7oDe3nO9fyz 2/V4FYbw86A53XWOUazqZBIUdSzTjvTl+19TTuIPYypy04LqBpaJLQx4XGdeNUSaM2OC JoOERjg2ey9rLQZ6+YMOWlO2XWe2eS+FktF5yWof3kh2SJca9REl9wZmE5whhNKY5g4j ZdzHLhevEYVETgZpLgD+0bb63WIzmcFXKbA10Ra7NN4VXD+KnCBahKhZE4/1p3ShZOng rvf7GlUWAUileEZ6LJCJqUQs5Gr9bLYx4h9dCpAHs/qw+nBMaO3SC915sIEWFRIG1Rws 9J/g== X-Forwarded-Encrypted: i=1; AJvYcCU96e8pvBeHoDQZFO3XziDeqoEinaKQ/yTmku6Me7bYHTIKXgmRgRtRyrKdbDh5NU8S9epI9vxDiQ==@kvack.org X-Gm-Message-State: AOJu0YwNo13RF5WfUJjNsEEU2LSE9Zzk4q7Xh3ZndKnFtHckJXTIxawk yJG2UZLypmYKrY0t4EdGUOvK9qMnDYoUMTosat4rz3C7WcZ3HAxwjp1dpzH0mu/ewRPuC0Pe4d9 /1Q== X-Google-Smtp-Source: AGHT+IG5tsVaIrkzimQ3d73aZezGeIWfqvUQe6a1x7KexmMSk23y9v/qQliOxJeTTumAOAHvlPYYt46pBpQ= X-Received: from pgid14.prod.google.com ([2002:a63:ed0e:0:b0:801:e378:a64a]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:2e81:b0:1ee:5cf2:9c07 with SMTP id adf61e73a8af0-1ee5cf29ed0mr11346671637.3.1739486840113; Thu, 13 Feb 2025 14:47:20 -0800 (PST) Date: Thu, 13 Feb 2025 14:46:47 -0800 In-Reply-To: <20250213224655.1680278-1-surenb@google.com> Mime-Version: 1.0 References: <20250213224655.1680278-1-surenb@google.com> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog Message-ID: <20250213224655.1680278-11-surenb@google.com> Subject: [PATCH v10 10/18] refcount: provide ops for cases when object's memory can be reused From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, david.laight.linux@gmail.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, richard.weiyang@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, Will Deacon X-Rspamd-Queue-Id: 3EC062000F X-Stat-Signature: mpzeg9mx3osj5zrjabgn94x83sr5w1zz X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1739486841-930021 X-HE-Meta: U2FsdGVkX1+ZkSrCtgCO8eW5okTo3USlYgLMUW1Q4aw1lyGX6yCo2XCAMMEGxQ3Q4tf/aj3xM/k+CroVTz8SZwUrBVjNMeF0AinaINYEUWTf8GWnUXaqGIuegdBQqWwX2zaK9pBhhHAsaubGUy6dst35wDmCcTbx4+R6QKi6YdeZPfZds+nqbe/FsNcy6yB6Ds/LdLhi0icPM7Yc+wq0t65tlOSZfmZ7/Q4phILdJNqQr4ZkSwptQxH3Ihnj1kIVe+10kPiMFIElj93+1SZNiPs6L1KNKqPDmF469ytjk+3Yw0WqwcLyJdOMKDv7CxG/sRe9rETKwuS3GlM0uu3vLnOZwZfDIXvGZ4825I0VJhP6E4aQRo+J/8p5Iqinpz0SbUqG4lJRbfwjvcfyU36SOtmd96c2O6Nf7cqlkn+xag50f9L/2TLdEgJx2dqWvKUw7PX3xKjTaEUrhhmkMFBiV35E/plheVMZ37aKcLafuZiZDH69o8jHlTz1uABrD8nvwDM7RUDfS067FL8CmLHsrsVfwDX6lVdjPUTkyH8TMvnIroz2L1K4yO2zXlO3+LVPiid5n3LW/NZxyya4sQOmix0qVVWX43iZ/OPLWKaV/0jg2wYmDAhmUf1RghtVaDmmhweunirWsMpUPBhNAx8lmsq7zb2O2RcroEICY3zJVBTfZr3ID5o9XXJQvFuUv/fn8aZA72G/kz0g7oAkahIT/8yQeuKZTg7Wqx130REyinLtKsowgjEzPujIbdpSkHxZx9F154qHOrHvwya/+iD/cOyP2elyreVbtMlSNhMIgseDzmeneR9YNHUAV1pvVwYAqaKAg8rex4XObWFkgXroz1x67Q4bGNIro7SQNmmbzYJIcF+IDGva5ZQAH4gwTucGQXtwqILUEKYb+kEB6IHIu74vBJsZiwAxTaBAkWZUz5ChpxT83zmWDQvmaHf+oG4OTUNCBJ3nQ8rfjWyHfLA K/BtiNw0 zTfGGI+NOi5H8HnpnYY/3g+FarTEpgdPtXIvEAeqfWLSTxGBk9h65EuKDgC3cz512cCMOEjk5vCgcemxhLYktDj+pCkigTNBcUmjqVrg0oVxQ//SM6+985clZrIsTpEfb/Qzt02HNR3DX2EcUuiTb1ZWjCsv9Hu9iYoD0Hb3X0yjqYi6Uk4+/AUFoPfNuvkVfrhrSxVWy7Mhsig3h01iKAtfaUFQAsGjzzILlY7JznU4c7yxdb42hugB1mxfUTIp8JSkgOqR44QEUoG9b+NK0Ek2X2wOEy6+nD6XdFtNqp0VMmQvuFrmuO/MIKmSlGS7CMYuI0Soqt1nFFHOQj/0C4FBivkbgrHhrnflsj6fqnAKJEVuJHxYFlcjZ+hYSdOBwyLbuJrnxPE0r+8WNolccaLo48BzP+XVbD5ZlvDDqxbjW2Lt9WQXtFq+4SKz89Hah/GS4c0Ru6VIgZUNUTfxxWTAjSSlU8669E0RiROymtHf+rQxoWild5lu97OPOXid8H3A36ME/QkrCmi+kUX4Av8t8T2f+5uOwhQNGiOQgFRXkm/c5Z9ExYBC2XGXRSC7+K+UnTFP2F5hhp2xssN9q3l+0GZpGI3py7bxdmSqJ5tm4W4wyFEfV8U+fu6lVySIDAwLH6VcSrHcxwjuvMiN1k2c8rIHvN074cLjYVH18bJeRAjgd9Y62hDrJbulIFeaRI/gqBKcBU/stw/91SqOlXseX+pNOHFjRfuZF X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For speculative lookups where a successful inc_not_zero() pins the object, but where we still need to double check if the object acquired is indeed the one we set out to acquire (identity check), needs this validation to happen *after* the increment. Similarly, when a new object is initialized and its memory might have been previously occupied by another object, all stores to initialize the object should happen *before* refcount initialization. Notably SLAB_TYPESAFE_BY_RCU is one such an example when this ordering is required for reference counting. Add refcount_{add|inc}_not_zero_acquire() to guarantee the proper ordering between acquiring a reference count on an object and performing the identity check for that object. Add refcount_set_release() to guarantee proper ordering between stores initializing object attributes and the store initializing the refcount. refcount_set_release() should be done after all other object attributes are initialized. Once refcount_set_release() is called, the object should be considered visible to other tasks even if it was not yet added into an object collection normally used to discover it. This is because other tasks might have discovered the object previously occupying the same memory and after memory reuse they can succeed in taking refcount for the new object and start using it. Object reuse example to consider: consumer: obj = lookup(collection, key); if (!refcount_inc_not_zero_acquire(&obj->ref)) return; if (READ_ONCE(obj->key) != key) { /* identity check */ put_ref(obj); return; } use(obj->value); producer: remove(collection, obj->key); if (!refcount_dec_and_test(&obj->ref)) return; obj->key = KEY_INVALID; free(obj); obj = malloc(); /* obj is reused */ obj->key = new_key; obj->value = new_value; refcount_set_release(obj->ref, 1); add(collection, new_key, obj); refcount_{add|inc}_not_zero_acquire() is required to prevent the following reordering when refcount_inc_not_zero() is used instead: consumer: obj = lookup(collection, key); if (READ_ONCE(obj->key) != key) { /* reordered identity check */ put_ref(obj); return; } producer: remove(collection, obj->key); if (!refcount_dec_and_test(&obj->ref)) return; obj->key = KEY_INVALID; free(obj); obj = malloc(); /* obj is reused */ obj->key = new_key; obj->value = new_value; refcount_set_release(obj->ref, 1); add(collection, new_key, obj); if (!refcount_inc_not_zero(&obj->ref)) return; use(obj->value); /* USING WRONG OBJECT */ refcount_set_release() is required to prevent the following reordering when refcount_set() is used instead: consumer: obj = lookup(collection, key); producer: remove(collection, obj->key); if (!refcount_dec_and_test(&obj->ref)) return; obj->key = KEY_INVALID; free(obj); obj = malloc(); /* obj is reused */ obj->key = new_key; /* new_key == old_key */ refcount_set(obj->ref, 1); if (!refcount_inc_not_zero_acquire(&obj->ref)) return; if (READ_ONCE(obj->key) != key) { /* pass since new_key == old_key */ put_ref(obj); return; } use(obj->value); /* USING STALE obj->value */ obj->value = new_value; /* reordered store */ add(collection, key, obj); Signed-off-by: Suren Baghdasaryan Cc: Peter Zijlstra Cc: Will Deacon Cc: Paul E. McKenney Acked-by: Vlastimil Babka #slab --- Documentation/RCU/whatisRCU.rst | 10 ++ Documentation/core-api/refcount-vs-atomic.rst | 37 +++++- include/linux/refcount.h | 106 ++++++++++++++++++ include/linux/slab.h | 9 ++ 4 files changed, 156 insertions(+), 6 deletions(-) diff --git a/Documentation/RCU/whatisRCU.rst b/Documentation/RCU/whatisRCU.rst index 1ef5784c1b84..53faeed7c190 100644 --- a/Documentation/RCU/whatisRCU.rst +++ b/Documentation/RCU/whatisRCU.rst @@ -971,6 +971,16 @@ unfortunately any spinlock in a ``SLAB_TYPESAFE_BY_RCU`` object must be initialized after each and every call to kmem_cache_alloc(), which renders reference-free spinlock acquisition completely unsafe. Therefore, when using ``SLAB_TYPESAFE_BY_RCU``, make proper use of a reference counter. +If using refcount_t, the specialized refcount_{add|inc}_not_zero_acquire() +and refcount_set_release() APIs should be used to ensure correct operation +ordering when verifying object identity and when initializing newly +allocated objects. Acquire fence in refcount_{add|inc}_not_zero_acquire() +ensures that identity checks happen *after* reference count is taken. +refcount_set_release() should be called after a newly allocated object is +fully initialized and release fence ensures that new values are visible +*before* refcount can be successfully taken by other users. Once +refcount_set_release() is called, the object should be considered visible +by other tasks. (Those willing to initialize their locks in a kmem_cache constructor may also use locking, including cache-friendly sequence locking.) diff --git a/Documentation/core-api/refcount-vs-atomic.rst b/Documentation/core-api/refcount-vs-atomic.rst index 79a009ce11df..9551a7bbfd38 100644 --- a/Documentation/core-api/refcount-vs-atomic.rst +++ b/Documentation/core-api/refcount-vs-atomic.rst @@ -86,7 +86,19 @@ Memory ordering guarantee changes: * none (both fully unordered) -case 2) - increment-based ops that return no value +case 2) - non-"Read/Modify/Write" (RMW) ops with release ordering +------------------------------------------- + +Function changes: + + * atomic_set_release() --> refcount_set_release() + +Memory ordering guarantee changes: + + * none (both provide RELEASE ordering) + + +case 3) - increment-based ops that return no value -------------------------------------------------- Function changes: @@ -98,7 +110,7 @@ Memory ordering guarantee changes: * none (both fully unordered) -case 3) - decrement-based RMW ops that return no value +case 4) - decrement-based RMW ops that return no value ------------------------------------------------------ Function changes: @@ -110,7 +122,7 @@ Memory ordering guarantee changes: * fully unordered --> RELEASE ordering -case 4) - increment-based RMW ops that return a value +case 5) - increment-based RMW ops that return a value ----------------------------------------------------- Function changes: @@ -126,7 +138,20 @@ Memory ordering guarantees changes: result of obtaining pointer to the object! -case 5) - generic dec/sub decrement-based RMW ops that return a value +case 6) - increment-based RMW ops with acquire ordering that return a value +----------------------------------------------------- + +Function changes: + + * atomic_inc_not_zero() --> refcount_inc_not_zero_acquire() + * no atomic counterpart --> refcount_add_not_zero_acquire() + +Memory ordering guarantees changes: + + * fully ordered --> ACQUIRE ordering on success + + +case 7) - generic dec/sub decrement-based RMW ops that return a value --------------------------------------------------------------------- Function changes: @@ -139,7 +164,7 @@ Memory ordering guarantees changes: * fully ordered --> RELEASE ordering + ACQUIRE ordering on success -case 6) other decrement-based RMW ops that return a value +case 8) other decrement-based RMW ops that return a value --------------------------------------------------------- Function changes: @@ -154,7 +179,7 @@ Memory ordering guarantees changes: .. note:: atomic_add_unless() only provides full order on success. -case 7) - lock-based RMW +case 9) - lock-based RMW ------------------------ Function changes: diff --git a/include/linux/refcount.h b/include/linux/refcount.h index 35f039ecb272..4589d2e7bfea 100644 --- a/include/linux/refcount.h +++ b/include/linux/refcount.h @@ -87,6 +87,15 @@ * The decrements dec_and_test() and sub_and_test() also provide acquire * ordering on success. * + * refcount_{add|inc}_not_zero_acquire() and refcount_set_release() provide + * acquire and release ordering for cases when the memory occupied by the + * object might be reused to store another object. This is important for the + * cases where secondary validation is required to detect such reuse, e.g. + * SLAB_TYPESAFE_BY_RCU. The secondary validation checks have to happen after + * the refcount is taken, hence acquire order is necessary. Similarly, when the + * object is initialized, all stores to its attributes should be visible before + * the refcount is set, otherwise a stale attribute value might be used by + * another task which succeeds in taking a refcount to the new object. */ #ifndef _LINUX_REFCOUNT_H @@ -125,6 +134,31 @@ static inline void refcount_set(refcount_t *r, int n) atomic_set(&r->refs, n); } +/** + * refcount_set_release - set a refcount's value with release ordering + * @r: the refcount + * @n: value to which the refcount will be set + * + * This function should be used when memory occupied by the object might be + * reused to store another object -- consider SLAB_TYPESAFE_BY_RCU. + * + * Provides release memory ordering which will order previous memory operations + * against this store. This ensures all updates to this object are visible + * once the refcount is set and stale values from the object previously + * occupying this memory are overwritten with new ones. + * + * This function should be called only after new object is fully initialized. + * After this call the object should be considered visible to other tasks even + * if it was not yet added into an object collection normally used to discover + * it. This is because other tasks might have discovered the object previously + * occupying the same memory and after memory reuse they can succeed in taking + * refcount to the new object and start using it. + */ +static inline void refcount_set_release(refcount_t *r, int n) +{ + atomic_set_release(&r->refs, n); +} + /** * refcount_read - get a refcount's value * @r: the refcount @@ -178,6 +212,52 @@ static inline __must_check bool refcount_add_not_zero(int i, refcount_t *r) return __refcount_add_not_zero(i, r, NULL); } +static inline __must_check __signed_wrap +bool __refcount_add_not_zero_acquire(int i, refcount_t *r, int *oldp) +{ + int old = refcount_read(r); + + do { + if (!old) + break; + } while (!atomic_try_cmpxchg_acquire(&r->refs, &old, old + i)); + + if (oldp) + *oldp = old; + + if (unlikely(old < 0 || old + i < 0)) + refcount_warn_saturate(r, REFCOUNT_ADD_NOT_ZERO_OVF); + + return old; +} + +/** + * refcount_add_not_zero_acquire - add a value to a refcount with acquire ordering unless it is 0 + * + * @i: the value to add to the refcount + * @r: the refcount + * + * Will saturate at REFCOUNT_SATURATED and WARN. + * + * This function should be used when memory occupied by the object might be + * reused to store another object -- consider SLAB_TYPESAFE_BY_RCU. + * + * Provides acquire memory ordering on success, it is assumed the caller has + * guaranteed the object memory to be stable (RCU, etc.). It does provide a + * control dependency and thereby orders future stores. See the comment on top. + * + * Use of this function is not recommended for the normal reference counting + * use case in which references are taken and released one at a time. In these + * cases, refcount_inc_not_zero_acquire() should instead be used to increment a + * reference count. + * + * Return: false if the passed refcount is 0, true otherwise + */ +static inline __must_check bool refcount_add_not_zero_acquire(int i, refcount_t *r) +{ + return __refcount_add_not_zero_acquire(i, r, NULL); +} + static inline __signed_wrap void __refcount_add(int i, refcount_t *r, int *oldp) { @@ -236,6 +316,32 @@ static inline __must_check bool refcount_inc_not_zero(refcount_t *r) return __refcount_inc_not_zero(r, NULL); } +static inline __must_check bool __refcount_inc_not_zero_acquire(refcount_t *r, int *oldp) +{ + return __refcount_add_not_zero_acquire(1, r, oldp); +} + +/** + * refcount_inc_not_zero_acquire - increment a refcount with acquire ordering unless it is 0 + * @r: the refcount to increment + * + * Similar to refcount_inc_not_zero(), but provides acquire memory ordering on + * success. + * + * This function should be used when memory occupied by the object might be + * reused to store another object -- consider SLAB_TYPESAFE_BY_RCU. + * + * Provides acquire memory ordering on success, it is assumed the caller has + * guaranteed the object memory to be stable (RCU, etc.). It does provide a + * control dependency and thereby orders future stores. See the comment on top. + * + * Return: true if the increment was successful, false otherwise + */ +static inline __must_check bool refcount_inc_not_zero_acquire(refcount_t *r) +{ + return __refcount_inc_not_zero_acquire(r, NULL); +} + static inline void __refcount_inc(refcount_t *r, int *oldp) { __refcount_add(1, r, oldp); diff --git a/include/linux/slab.h b/include/linux/slab.h index 09eedaecf120..ad902a2d692b 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -136,6 +136,15 @@ enum _slab_flag_bits { * rcu_read_lock before reading the address, then rcu_read_unlock after * taking the spinlock within the structure expected at that address. * + * Note that object identity check has to be done *after* acquiring a + * reference, therefore user has to ensure proper ordering for loads. + * Similarly, when initializing objects allocated with SLAB_TYPESAFE_BY_RCU, + * the newly allocated object has to be fully initialized *before* its + * refcount gets initialized and proper ordering for stores is required. + * refcount_{add|inc}_not_zero_acquire() and refcount_set_release() are + * designed with the proper fences required for reference counting objects + * allocated with SLAB_TYPESAFE_BY_RCU. + * * Note that it is not possible to acquire a lock within a structure * allocated with SLAB_TYPESAFE_BY_RCU without first acquiring a reference * as described above. The reason is that SLAB_TYPESAFE_BY_RCU pages