From patchwork Tue Mar 24 15:36:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 11455799 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6E8B914B4 for ; Tue, 24 Mar 2020 15:37:32 +0000 (UTC) Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.kernel.org (Postfix) with SMTP id B30D62073E for ; Tue, 24 Mar 2020 15:37:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="ooPy8VS1" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B30D62073E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kernel-hardening-return-18160-patchwork-kernel-hardening=patchwork.kernel.org@lists.openwall.com Received: (qmail 24549 invoked by uid 550); 24 Mar 2020 15:37:11 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 24379 invoked from network); 24 Mar 2020 15:37:10 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1585064218; bh=QY0oPyqlXUd/xyVI4B2QSRbEsJgKQpxLLXC0d7PqKfU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ooPy8VS1ZB1zf15FBC0Y9g64wzVn6YQ+dSCb44ZRNwzCo3Y3MdaJG47x25mU8c2rV KA4gW51sECJ1dE9b9lujesOi+Vu9RjU7RfxuHIC5wKlLWU3TWAdKfmiaKyAUOI9Tc9 Ft68clf/VM0E2M7OrXhDYMW2Zxcmaqp5CMT8m1oo= From: Will Deacon To: linux-kernel@vger.kernel.org Cc: Will Deacon , Eric Dumazet , Jann Horn , Kees Cook , Maddie Stone , Marco Elver , "Paul E . McKenney" , Peter Zijlstra , Thomas Gleixner , kernel-team@android.com, kernel-hardening@lists.openwall.com Subject: [RFC PATCH 03/21] list: Annotate lockless list primitives with data_race() Date: Tue, 24 Mar 2020 15:36:25 +0000 Message-Id: <20200324153643.15527-4-will@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200324153643.15527-1-will@kernel.org> References: <20200324153643.15527-1-will@kernel.org> MIME-Version: 1.0 Some list predicates can be used locklessly even with the non-RCU list implementations, since they effectively boil down to a test against NULL. For example, checking whether or not a list is empty is safe even in the presence of a concurrent, tearing write to the list head pointer. Similarly, checking whether or not an hlist node has been hashed is safe as well. Annotate these lockless list predicates with data_race() and READ_ONCE() so that KCSAN and the compiler are aware of what's going on. The writer side can then avoid having to use WRITE_ONCE() in the non-RCU implementation. Cc: Paul E. McKenney Cc: Peter Zijlstra Cc: Marco Elver Signed-off-by: Will Deacon --- include/linux/list.h | 10 +++++----- include/linux/list_bl.h | 5 +++-- include/linux/list_nulls.h | 6 +++--- include/linux/llist.h | 2 +- 4 files changed, 12 insertions(+), 11 deletions(-) diff --git a/include/linux/list.h b/include/linux/list.h index 4fed5a0f9b77..4d9f5f9ed1a8 100644 --- a/include/linux/list.h +++ b/include/linux/list.h @@ -279,7 +279,7 @@ static inline int list_is_last(const struct list_head *list, */ static inline int list_empty(const struct list_head *head) { - return READ_ONCE(head->next) == head; + return data_race(READ_ONCE(head->next) == head); } /** @@ -524,7 +524,7 @@ static inline void list_splice_tail_init(struct list_head *list, */ #define list_first_entry_or_null(ptr, type, member) ({ \ struct list_head *head__ = (ptr); \ - struct list_head *pos__ = READ_ONCE(head__->next); \ + struct list_head *pos__ = data_race(READ_ONCE(head__->next)); \ pos__ != head__ ? list_entry(pos__, type, member) : NULL; \ }) @@ -772,13 +772,13 @@ static inline void INIT_HLIST_NODE(struct hlist_node *h) * hlist_unhashed - Has node been removed from list and reinitialized? * @h: Node to be checked * - * Not that not all removal functions will leave a node in unhashed + * Note that not all removal functions will leave a node in unhashed * state. For example, hlist_nulls_del_init_rcu() does leave the * node in unhashed state, but hlist_nulls_del() does not. */ static inline int hlist_unhashed(const struct hlist_node *h) { - return !READ_ONCE(h->pprev); + return data_race(!READ_ONCE(h->pprev)); } /** @@ -787,7 +787,7 @@ static inline int hlist_unhashed(const struct hlist_node *h) */ static inline int hlist_empty(const struct hlist_head *h) { - return !READ_ONCE(h->first); + return data_race(!READ_ONCE(h->first)); } static inline void __hlist_del(struct hlist_node *n) diff --git a/include/linux/list_bl.h b/include/linux/list_bl.h index ae1b541446c9..1804fdb84dda 100644 --- a/include/linux/list_bl.h +++ b/include/linux/list_bl.h @@ -51,7 +51,7 @@ static inline void INIT_HLIST_BL_NODE(struct hlist_bl_node *h) static inline bool hlist_bl_unhashed(const struct hlist_bl_node *h) { - return !h->pprev; + return data_race(!READ_ONCE(h->pprev)); } static inline struct hlist_bl_node *hlist_bl_first(struct hlist_bl_head *h) @@ -71,7 +71,8 @@ static inline void hlist_bl_set_first(struct hlist_bl_head *h, static inline bool hlist_bl_empty(const struct hlist_bl_head *h) { - return !((unsigned long)READ_ONCE(h->first) & ~LIST_BL_LOCKMASK); + unsigned long first = data_race((unsigned long)READ_ONCE(h->first)); + return !(first & ~LIST_BL_LOCKMASK); } static inline void hlist_bl_add_head(struct hlist_bl_node *n, diff --git a/include/linux/list_nulls.h b/include/linux/list_nulls.h index 3a9ff01e9a11..fa51a801bf32 100644 --- a/include/linux/list_nulls.h +++ b/include/linux/list_nulls.h @@ -60,18 +60,18 @@ static inline unsigned long get_nulls_value(const struct hlist_nulls_node *ptr) * hlist_nulls_unhashed - Has node been removed and reinitialized? * @h: Node to be checked * - * Not that not all removal functions will leave a node in unhashed state. + * Note that not all removal functions will leave a node in unhashed state. * For example, hlist_del_init_rcu() leaves the node in unhashed state, * but hlist_nulls_del() does not. */ static inline int hlist_nulls_unhashed(const struct hlist_nulls_node *h) { - return !READ_ONCE(h->pprev); + return data_race(!READ_ONCE(h->pprev)); } static inline int hlist_nulls_empty(const struct hlist_nulls_head *h) { - return is_a_nulls(READ_ONCE(h->first)); + return data_race(is_a_nulls(READ_ONCE(h->first))); } static inline void hlist_nulls_add_head(struct hlist_nulls_node *n, diff --git a/include/linux/llist.h b/include/linux/llist.h index 2e9c7215882b..c7f042b73899 100644 --- a/include/linux/llist.h +++ b/include/linux/llist.h @@ -186,7 +186,7 @@ static inline void init_llist_head(struct llist_head *list) */ static inline bool llist_empty(const struct llist_head *head) { - return READ_ONCE(head->first) == NULL; + return data_race(READ_ONCE(head->first) == NULL); } static inline struct llist_node *llist_next(struct llist_node *node)