From patchwork Tue Jan 14 20:35:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrew Cooper X-Patchwork-Id: 11332859 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5183C13BD for ; Tue, 14 Jan 2020 20:37:30 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 20B4D24656 for ; Tue, 14 Jan 2020 20:37:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="e4GQ6kj3" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 20B4D24656 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1irSus-0000py-LQ; Tue, 14 Jan 2020 20:35:58 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1irSuq-0000pt-EW for xen-devel@lists.xenproject.org; Tue, 14 Jan 2020 20:35:56 +0000 X-Inumbo-ID: 73af18c6-370d-11ea-8467-12813bfff9fa Received: from esa6.hc3370-68.iphmx.com (unknown [216.71.155.175]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 73af18c6-370d-11ea-8467-12813bfff9fa; Tue, 14 Jan 2020 20:35:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1579034150; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=9wiXdmQbd1rbUfY9H44kInJbaR4A3j4phCEOp47yt1o=; b=e4GQ6kj3YTn71ro8fDD9qCHs23zKEjp+0Boh4AK6yW+ed9lW4VF1mSoz pej8Ja1bfqfrqfmPk4e9frs4rlNGZ4ybyBzQBsP+v38Uf8mEw+ip2plkg 43XQK/PWsHVknZf8dUuDOkzkkn7+lplsq2E7I+VWTUAKq8oNIKDB1fjHb w=; Authentication-Results: esa6.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none; spf=None smtp.pra=andrew.cooper3@citrix.com; spf=Pass smtp.mailfrom=Andrew.Cooper3@citrix.com; spf=None smtp.helo=postmaster@mail.citrix.com Received-SPF: None (esa6.hc3370-68.iphmx.com: no sender authenticity information available from domain of andrew.cooper3@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa6.hc3370-68.iphmx.com; envelope-from="Andrew.Cooper3@citrix.com"; x-sender="andrew.cooper3@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa6.hc3370-68.iphmx.com: domain of Andrew.Cooper3@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa6.hc3370-68.iphmx.com; envelope-from="Andrew.Cooper3@citrix.com"; x-sender="Andrew.Cooper3@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ip4:168.245.78.127 ~all" Received-SPF: None (esa6.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa6.hc3370-68.iphmx.com; envelope-from="Andrew.Cooper3@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: iymLlbW6Wl74yAb08Zj2pqNR98o1Yb7kY163aMvUiVHCqYpZzXThPSKiGIP9rF4MALZ6ztjosr G43leMgpz/mVrxCzSppzmX/pZJEpbxaX4Mr5Xb9z/Xyx1APWe923Bwea05RmBN+nPzDbMBSQyK RRWTbRCCFb8kgrRlC6Qo/vL+kcoNJ57ak6gJ11Bwx28dhQ+8iPhncbmXu4uGgrdEDXkJ9Diljy JEh3ojvIbuZHQCdQ/kZbTaJZO+6eRt1oc2Dcht19HgTtUVxyETZY5GKTIgDwt70x3Ui7//O0KV BFo= X-SBRS: 2.7 X-MesageID: 11343145 X-Ironport-Server: esa6.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.70,320,1574139600"; d="scan'208";a="11343145" From: Andrew Cooper To: Xen-devel Date: Tue, 14 Jan 2020 20:35:45 +0000 Message-ID: <20200114203545.8897-1-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.11.0 MIME-Version: 1.0 Subject: [Xen-devel] [PATCH] xen/list: Remove prefetching X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Stefano Stabellini , Julien Grall , Wei Liu , Andrew Cooper , Jan Beulich , Volodymyr Babchuk , =?utf-8?q?Roger_Pau_Monn?= =?utf-8?q?=C3=A9?= Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Xen inherited its list infrastructure from Linux. One area where has fallen behind is that of prefetching, which as it turns out is a performance penalty in most cases. Prefetch of NULL on x86 is now widely measured to have glacial performance properties, and will unconditionally hit on every hlist use due to the termination condition. Cross-port the following Linux patches: 75d65a425c (2011) "hlist: remove software prefetching in hlist iterators" e66eed651f (2011) "list: remove prefetching from regular list iterators" c0d15cc7ee (2013) "linked-list: Remove __list_for_each" to Xen, which results in the following net diffstat on x86: add/remove: 0/1 grow/shrink: 27/83 up/down: 576/-1648 (-1072) (The code additions comes from a few now-inlined functions, and slightly different basic block padding.) No functional change. Signed-off-by: Andrew Cooper Acked-by: Julien Grall Reviewed-by: Roger Pau Monné --- CC: Jan Beulich CC: Wei Liu CC: Roger Pau Monné CC: Stefano Stabellini CC: Julien Grall CC: Volodymyr Babchuk --- xen/include/xen/list.h | 46 +++++++++++++--------------------------------- 1 file changed, 13 insertions(+), 33 deletions(-) diff --git a/xen/include/xen/list.h b/xen/include/xen/list.h index 1387abb211..dc5a8c461b 100644 --- a/xen/include/xen/list.h +++ b/xen/include/xen/list.h @@ -42,9 +42,6 @@ struct list_head { #define LIST_HEAD_READ_MOSTLY(name) \ struct list_head __read_mostly name = LIST_HEAD_INIT(name) -/* Do not move this ahead of the struct list_head definition! */ -#include - static inline void INIT_LIST_HEAD(struct list_head *list) { list->next = list; @@ -455,20 +452,6 @@ static inline void list_splice_init(struct list_head *list, * @head: the head for your list. */ #define list_for_each(pos, head) \ - for (pos = (head)->next; prefetch(pos->next), pos != (head); \ - pos = pos->next) - -/** - * __list_for_each - iterate over a list - * @pos: the &struct list_head to use as a loop cursor. - * @head: the head for your list. - * - * This variant differs from list_for_each() in that it's the - * simplest possible list iteration code, no prefetching is done. - * Use this for code that knows the list to be very short (empty - * or 1 entry) most of the time. - */ -#define __list_for_each(pos, head) \ for (pos = (head)->next; pos != (head); pos = pos->next) /** @@ -477,8 +460,7 @@ static inline void list_splice_init(struct list_head *list, * @head: the head for your list. */ #define list_for_each_prev(pos, head) \ - for (pos = (head)->prev; prefetch(pos->prev), pos != (head); \ - pos = pos->prev) + for (pos = (head)->prev; pos != (head); pos = pos->prev) /** * list_for_each_safe - iterate over a list safe against removal of list entry @@ -509,7 +491,7 @@ static inline void list_splice_init(struct list_head *list, */ #define list_for_each_entry(pos, head, member) \ for (pos = list_entry((head)->next, typeof(*pos), member); \ - prefetch(pos->member.next), &pos->member != (head); \ + &pos->member != (head); \ pos = list_entry(pos->member.next, typeof(*pos), member)) /** @@ -520,7 +502,7 @@ static inline void list_splice_init(struct list_head *list, */ #define list_for_each_entry_reverse(pos, head, member) \ for (pos = list_entry((head)->prev, typeof(*pos), member); \ - prefetch(pos->member.prev), &pos->member != (head); \ + &pos->member != (head); \ pos = list_entry(pos->member.prev, typeof(*pos), member)) /** @@ -547,7 +529,7 @@ static inline void list_splice_init(struct list_head *list, */ #define list_for_each_entry_continue(pos, head, member) \ for (pos = list_entry(pos->member.next, typeof(*pos), member); \ - prefetch(pos->member.next), &pos->member != (head); \ + &pos->member != (head); \ pos = list_entry(pos->member.next, typeof(*pos), member)) /** @@ -560,7 +542,7 @@ static inline void list_splice_init(struct list_head *list, * Iterate over list of given type, continuing from current position. */ #define list_for_each_entry_from(pos, head, member) \ - for (; prefetch(pos->member.next), &pos->member != (head); \ + for (; &pos->member != (head); \ pos = list_entry(pos->member.next, typeof(*pos), member)) /** @@ -635,7 +617,7 @@ static inline void list_splice_init(struct list_head *list, */ #define list_for_each_rcu(pos, head) \ for (pos = (head)->next; \ - prefetch(rcu_dereference(pos)->next), pos != (head); \ + rcu_dereference(pos) != (head); \ pos = pos->next) #define __list_for_each_rcu(pos, head) \ @@ -672,8 +654,7 @@ static inline void list_splice_init(struct list_head *list, */ #define list_for_each_entry_rcu(pos, head, member) \ for (pos = list_entry((head)->next, typeof(*pos), member); \ - prefetch(rcu_dereference(pos)->member.next), \ - &pos->member != (head); \ + &rcu_dereference(pos)->member != (head); \ pos = list_entry(pos->member.next, typeof(*pos), member)) /** @@ -689,7 +670,7 @@ static inline void list_splice_init(struct list_head *list, */ #define list_for_each_continue_rcu(pos, head) \ for ((pos) = (pos)->next; \ - prefetch(rcu_dereference((pos))->next), (pos) != (head); \ + rcu_dereference(pos) != (head); \ (pos) = (pos)->next) /* @@ -918,8 +899,7 @@ static inline void hlist_add_after_rcu(struct hlist_node *prev, #define hlist_entry(ptr, type, member) container_of(ptr,type,member) #define hlist_for_each(pos, head) \ - for (pos = (head)->first; pos && ({ prefetch(pos->next); 1; }); \ - pos = pos->next) + for (pos = (head)->first; pos; pos = pos->next) #define hlist_for_each_safe(pos, n, head) \ for (pos = (head)->first; pos && ({ n = pos->next; 1; }); \ @@ -934,7 +914,7 @@ static inline void hlist_add_after_rcu(struct hlist_node *prev, */ #define hlist_for_each_entry(tpos, pos, head, member) \ for (pos = (head)->first; \ - pos && ({ prefetch(pos->next); 1;}) && \ + pos && \ ({ tpos = hlist_entry(pos, typeof(*tpos), member); 1;}); \ pos = pos->next) @@ -947,7 +927,7 @@ static inline void hlist_add_after_rcu(struct hlist_node *prev, */ #define hlist_for_each_entry_continue(tpos, pos, member) \ for (pos = (pos)->next; \ - pos && ({ prefetch(pos->next); 1;}) && \ + pos && \ ({ tpos = hlist_entry(pos, typeof(*tpos), member); 1;}); \ pos = pos->next) @@ -959,7 +939,7 @@ static inline void hlist_add_after_rcu(struct hlist_node *prev, * @member: the name of the hlist_node within the struct. */ #define hlist_for_each_entry_from(tpos, pos, member) \ - for (; pos && ({ prefetch(pos->next); 1;}) && \ + for (; pos && \ ({ tpos = hlist_entry(pos, typeof(*tpos), member); 1;}); \ pos = pos->next) @@ -992,7 +972,7 @@ static inline void hlist_add_after_rcu(struct hlist_node *prev, */ #define hlist_for_each_entry_rcu(tpos, pos, head, member) \ for (pos = (head)->first; \ - rcu_dereference(pos) && ({ prefetch(pos->next); 1;}) && \ + rcu_dereference(pos) && \ ({ tpos = hlist_entry(pos, typeof(*tpos), member); 1;}); \ pos = pos->next)