From patchwork Wed Oct 23 13:58:12 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrew Cooper X-Patchwork-Id: 11206885 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 551CA1747 for ; Wed, 23 Oct 2019 13:59:55 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 315C42173B for ; Wed, 23 Oct 2019 13:59:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="NGRw5Emq" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 315C42173B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iNH9o-0004nH-FV; Wed, 23 Oct 2019 13:58:36 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iNH9m-0004mJ-Np for xen-devel@lists.xenproject.org; Wed, 23 Oct 2019 13:58:34 +0000 X-Inumbo-ID: 2ba9895a-f59d-11e9-947f-12813bfff9fa Received: from esa2.hc3370-68.iphmx.com (unknown [216.71.145.153]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 2ba9895a-f59d-11e9-947f-12813bfff9fa; Wed, 23 Oct 2019 13:58:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1571839100; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=HKJkSbXcmX/kEnPedYeC0yfTtOaX516ggrC6KOT5BGE=; b=NGRw5Emq08+0nPDZEZ9FR2me7wvU56xvzN4NBxexyJWBpy1wU4+JA6lG SwQXp1V3j3jyvkkAWkzrC4UW86I5Wrd85jZCZcN+hlpJSD9P8mikT2CEC vCLmdg+P2ci9drLOBtdIK51Tx548bMRmfS+SgWU7wlHM7lf7gcdauFaC4 Q=; Authentication-Results: esa2.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none; spf=None smtp.pra=andrew.cooper3@citrix.com; spf=Pass smtp.mailfrom=Andrew.Cooper3@citrix.com; spf=None smtp.helo=postmaster@mail.citrix.com Received-SPF: None (esa2.hc3370-68.iphmx.com: no sender authenticity information available from domain of andrew.cooper3@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa2.hc3370-68.iphmx.com; envelope-from="Andrew.Cooper3@citrix.com"; x-sender="andrew.cooper3@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa2.hc3370-68.iphmx.com: domain of Andrew.Cooper3@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa2.hc3370-68.iphmx.com; envelope-from="Andrew.Cooper3@citrix.com"; x-sender="Andrew.Cooper3@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ip4:168.245.78.127 ~all" Received-SPF: None (esa2.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa2.hc3370-68.iphmx.com; envelope-from="Andrew.Cooper3@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: CQlkvqemll8CyTNLaBa5YY25hrha+pcVxYWr7FuzBrS4yROwbYWSrtay5l2JX2yUKRK/qGs0CD K4xv2sdWHJasN2a5elDBf8o9HPAbZNE0ut46r8oA+hi9bPv7jiCCWbXzrNvlzPxDzPjVCPKsvm nSCVxTQJXfvVqsIBNewM0vY44EMhg7TQ08BR1nHV97ttspR+E+VAED45ysRGd7w26ntj1R27Rg FP3n+z+A23lTt+Db1QkKQrpYxR+6F+jbuKOTknWRyIDT+RAU2dNE3Atnk0/ZoAnXn87yd8URA8 rvI= X-SBRS: 2.7 X-MesageID: 7322263 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.68,221,1569297600"; d="scan'208";a="7322263" From: Andrew Cooper To: Xen-devel Date: Wed, 23 Oct 2019 14:58:12 +0100 Message-ID: <20191023135812.21348-8-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20191023135812.21348-1-andrew.cooper3@citrix.com> References: <20191023135812.21348-1-andrew.cooper3@citrix.com> MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v3 7/7] x86/nospec: Optimise array_index_mask_nospec() for power-of-2 arrays X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Juergen Gross , Andrew Cooper , Wei Liu , Jan Beulich , =?utf-8?q?Roger_Pau_Monn=C3=A9?= Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" When the compiler can determine that an array bound is a power of two, the array index can be bounded even under speculation with a single and instruction. Respecify array_index_mask_nospec() to allow for masks other than ~0 and 0, and introduce an IS_POWER_OF_2() helper. Signed-off-by: Andrew Cooper --- CC: Jan Beulich CC: Wei Liu CC: Roger Pau Monné CC: Juergen Gross This optimisation is not safe on ARM, because some CPUs do data value speculation, which is why the CSDB barrer was introduced. --- xen/include/asm-x86/nospec.h | 25 +++++++++++++++++++------ xen/include/xen/config.h | 1 + xen/include/xen/nospec.h | 3 ++- 3 files changed, 22 insertions(+), 7 deletions(-) diff --git a/xen/include/asm-x86/nospec.h b/xen/include/asm-x86/nospec.h index 0039cd2713..4f36069eac 100644 --- a/xen/include/asm-x86/nospec.h +++ b/xen/include/asm-x86/nospec.h @@ -7,13 +7,20 @@ #include /** - * array_index_mask_nospec() - generate a mask that is ~0UL when the - * bounds check succeeds and 0 otherwise + * array_index_mask_nospec() - generate a mask to bound an array index + * which is safe even under adverse speculation. * @index: array element index * @size: number of elements in array * - * Returns: + * In general, returns: * 0 - (index < size) + * + * This yeild ~0UL in within-bounds case, and 0 in the out-of-bounds + * case. + * + * When the compiler can determine that the array is a power of two, a + * lower overhead option is to mask the index with a single and + * instruction. */ #define array_index_mask_nospec array_index_mask_nospec static inline unsigned long array_index_mask_nospec(unsigned long index, @@ -21,9 +28,15 @@ static inline unsigned long array_index_mask_nospec(unsigned long index, { unsigned long mask; - asm volatile ( "cmp %[size], %[index]; sbb %[mask], %[mask];" - : [mask] "=r" (mask) - : [size] "g" (size), [index] "r" (index) ); + if ( __builtin_constant_p(size) && IS_POWER_OF_2(size) ) + { + mask = size - 1; + OPTIMIZER_HIDE_VAR(mask); + } + else + asm volatile ( "cmp %[size], %[index]; sbb %[mask], %[mask];" + : [mask] "=r" (mask) + : [size] "g" (size), [index] "r" (index) ); return mask; } diff --git a/xen/include/xen/config.h b/xen/include/xen/config.h index a106380a23..21c763617c 100644 --- a/xen/include/xen/config.h +++ b/xen/include/xen/config.h @@ -75,6 +75,7 @@ #define GB(_gb) (_AC(_gb, ULL) << 30) #define IS_ALIGNED(val, align) (((val) & ((align) - 1)) == 0) +#define IS_POWER_OF_2(val) ((val) && IS_ALIGNED(val, val)) #define __STR(...) #__VA_ARGS__ #define STR(...) __STR(__VA_ARGS__) diff --git a/xen/include/xen/nospec.h b/xen/include/xen/nospec.h index 7578210f16..cfc31f11b7 100644 --- a/xen/include/xen/nospec.h +++ b/xen/include/xen/nospec.h @@ -12,7 +12,8 @@ #include /** - * array_index_mask_nospec() - generate a ~0 mask when index < size, 0 otherwise + * array_index_mask_nospec() - generate a mask to bound an array index + * which is safe even under adverse speculation. * @index: array element index * @size: number of elements in array *