From patchwork Fri Feb 28 00:00:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Garnier X-Patchwork-Id: 11411163 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5A8F914BC for ; Fri, 28 Feb 2020 00:01:36 +0000 (UTC) Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.kernel.org (Postfix) with SMTP id 33E94246B8 for ; Fri, 28 Feb 2020 00:01:35 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="H/bvqaZi" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 33E94246B8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kernel-hardening-return-17992-patchwork-kernel-hardening=patchwork.kernel.org@lists.openwall.com Received: (qmail 20442 invoked by uid 550); 28 Feb 2020 00:01:26 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 20194 invoked from network); 28 Feb 2020 00:01:24 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=wUW0SRos0+A3UgSb4za27Jh5F9h5gOUKnyYh9lyjzV0=; b=H/bvqaZiBMafCK6pgDHsZy/lR9hr6bPq1UUR6mwUKu6sDujgm/vsl55pkO09+XkHxm TOxWM/tWZA2R+YGMSc3uLk69cgFLHEwaTF/6ZoSINnRNlkGo8ZHS9+/dh7zAvAUoDVOs H3DLoGcVzLdv7lbpa+kIkhlYw7QXGlp/J8nok= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=wUW0SRos0+A3UgSb4za27Jh5F9h5gOUKnyYh9lyjzV0=; b=mkhCBNj1I+2khVGjOi4AlVtFzrl0I7LrbwuSKFmj+XE5wP1OkmMccgVEU+3aStsQ+8 anrbY9U/G3iwdBTTif5vwd6iAGc35pT9P6E0eQloCFDuHDFpRbI0mgIoKElKjsBegP5C K0ADOJnfHk8ox5gK/CVF09pImEb3Y3GWDrg8iO3u8F9d5qAomuEAQgR2KznUCrz9biuM N9fOaxBgCWth/bUtuTjIX8NpvPEueY9yCZTH+62PmboqT7zL3qKkE4h48iZxojADYfiM 05hRPO9NNXjvSfo5JJ+tpMUdvfI0IKZT3VXJOcDvkqzAer7hArKoxiCv0Vv/cU0766OX 2fLQ== X-Gm-Message-State: APjAAAV2kPrfu3W9P+xneMbR2o+2bHU6ICgytG4YjLmZ2/sra5srlO32 5FGIR0fQj0jWxgxDaJwXRbPPSu71cPY= X-Google-Smtp-Source: APXvYqy0cebeP/boVv50rQ6COhj6tirOF8ldK9UNbOBgXFGirttPIRnD8x4JFkLfBkZWQMbpBS+Qog== X-Received: by 2002:a17:90a:b009:: with SMTP id x9mr1589285pjq.124.1582848071290; Thu, 27 Feb 2020 16:01:11 -0800 (PST) From: Thomas Garnier To: kernel-hardening@lists.openwall.com Cc: kristen@linux.intel.com, keescook@chromium.org, Thomas Garnier , Herbert Xu , "David S. Miller" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , "H. Peter Anvin" , x86@kernel.org, linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v11 01/11] x86/crypto: Adapt assembly for PIE support Date: Thu, 27 Feb 2020 16:00:46 -0800 Message-Id: <20200228000105.165012-2-thgarnie@chromium.org> X-Mailer: git-send-email 2.25.1.481.gfbce0eb801-goog In-Reply-To: <20200228000105.165012-1-thgarnie@chromium.org> References: <20200228000105.165012-1-thgarnie@chromium.org> MIME-Version: 1.0 Change the assembly code to use only relative references of symbols for the kernel to be PIE compatible. Signed-off-by: Thomas Garnier --- arch/x86/crypto/aegis128-aesni-asm.S | 6 +- arch/x86/crypto/aesni-intel_asm.S | 8 +- arch/x86/crypto/aesni-intel_avx-x86_64.S | 3 +- arch/x86/crypto/camellia-aesni-avx-asm_64.S | 42 ++++----- arch/x86/crypto/camellia-aesni-avx2-asm_64.S | 44 ++++----- arch/x86/crypto/camellia-x86_64-asm_64.S | 8 +- arch/x86/crypto/cast5-avx-x86_64-asm_64.S | 50 +++++----- arch/x86/crypto/cast6-avx-x86_64-asm_64.S | 44 +++++---- arch/x86/crypto/des3_ede-asm_64.S | 96 +++++++++++++------- arch/x86/crypto/ghash-clmulni-intel_asm.S | 4 +- arch/x86/crypto/glue_helper-asm-avx.S | 4 +- arch/x86/crypto/glue_helper-asm-avx2.S | 6 +- arch/x86/crypto/sha256-avx2-asm.S | 18 ++-- 13 files changed, 191 insertions(+), 142 deletions(-) diff --git a/arch/x86/crypto/aegis128-aesni-asm.S b/arch/x86/crypto/aegis128-aesni-asm.S index 51d46d93efbc..c4bd32d8ca43 100644 --- a/arch/x86/crypto/aegis128-aesni-asm.S +++ b/arch/x86/crypto/aegis128-aesni-asm.S @@ -200,8 +200,8 @@ SYM_FUNC_START(crypto_aegis128_aesni_init) movdqa KEY, STATE4 /* load the constants: */ - movdqa .Laegis128_const_0, STATE2 - movdqa .Laegis128_const_1, STATE1 + movdqa .Laegis128_const_0(%rip), STATE2 + movdqa .Laegis128_const_1(%rip), STATE1 pxor STATE2, STATE3 pxor STATE1, STATE4 @@ -681,7 +681,7 @@ SYM_FUNC_START(crypto_aegis128_aesni_dec_tail) punpcklbw T0, T0 punpcklbw T0, T0 punpcklbw T0, T0 - movdqa .Laegis128_counter, T1 + movdqa .Laegis128_counter(%rip), T1 pcmpgtb T1, T0 pand T0, MSG diff --git a/arch/x86/crypto/aesni-intel_asm.S b/arch/x86/crypto/aesni-intel_asm.S index cad6e1bfa7d5..a614e277d15b 100644 --- a/arch/x86/crypto/aesni-intel_asm.S +++ b/arch/x86/crypto/aesni-intel_asm.S @@ -2597,7 +2597,7 @@ SYM_FUNC_END(aesni_cbc_dec) * BSWAP_MASK == endian swapping mask */ SYM_FUNC_START_LOCAL(_aesni_inc_init) - movaps .Lbswap_mask, BSWAP_MASK + movaps .Lbswap_mask(%rip), BSWAP_MASK movaps IV, CTR PSHUFB_XMM BSWAP_MASK CTR mov $1, TCTR_LOW @@ -2724,12 +2724,12 @@ SYM_FUNC_START(aesni_xts_crypt8) cmpb $0, %cl movl $0, %ecx movl $240, %r10d - leaq _aesni_enc4, %r11 - leaq _aesni_dec4, %rax + leaq _aesni_enc4(%rip), %r11 + leaq _aesni_dec4(%rip), %rax cmovel %r10d, %ecx cmoveq %rax, %r11 - movdqa .Lgf128mul_x_ble_mask, GF128MUL_MASK + movdqa .Lgf128mul_x_ble_mask(%rip), GF128MUL_MASK movups (IVP), IV mov 480(KEYP), KLEN diff --git a/arch/x86/crypto/aesni-intel_avx-x86_64.S b/arch/x86/crypto/aesni-intel_avx-x86_64.S index bfa1c0b3e5b4..838221507976 100644 --- a/arch/x86/crypto/aesni-intel_avx-x86_64.S +++ b/arch/x86/crypto/aesni-intel_avx-x86_64.S @@ -660,7 +660,8 @@ _get_AAD_rest0\@: vpshufb and an array of shuffle masks */ movq %r12, %r11 salq $4, %r11 - vmovdqu aad_shift_arr(%r11), \T1 + leaq aad_shift_arr(%rip), %rax + vmovdqu (%rax,%r11,), \T1 vpshufb \T1, \T7, \T7 _get_AAD_rest_final\@: vpshufb SHUF_MASK(%rip), \T7, \T7 diff --git a/arch/x86/crypto/camellia-aesni-avx-asm_64.S b/arch/x86/crypto/camellia-aesni-avx-asm_64.S index d01ddd73de65..8a203c507f81 100644 --- a/arch/x86/crypto/camellia-aesni-avx-asm_64.S +++ b/arch/x86/crypto/camellia-aesni-avx-asm_64.S @@ -53,10 +53,10 @@ /* \ * S-function with AES subbytes \ */ \ - vmovdqa .Linv_shift_row, t4; \ - vbroadcastss .L0f0f0f0f, t7; \ - vmovdqa .Lpre_tf_lo_s1, t0; \ - vmovdqa .Lpre_tf_hi_s1, t1; \ + vmovdqa .Linv_shift_row(%rip), t4; \ + vbroadcastss .L0f0f0f0f(%rip), t7; \ + vmovdqa .Lpre_tf_lo_s1(%rip), t0; \ + vmovdqa .Lpre_tf_hi_s1(%rip), t1; \ \ /* AES inverse shift rows */ \ vpshufb t4, x0, x0; \ @@ -69,8 +69,8 @@ vpshufb t4, x6, x6; \ \ /* prefilter sboxes 1, 2 and 3 */ \ - vmovdqa .Lpre_tf_lo_s4, t2; \ - vmovdqa .Lpre_tf_hi_s4, t3; \ + vmovdqa .Lpre_tf_lo_s4(%rip), t2; \ + vmovdqa .Lpre_tf_hi_s4(%rip), t3; \ filter_8bit(x0, t0, t1, t7, t6); \ filter_8bit(x7, t0, t1, t7, t6); \ filter_8bit(x1, t0, t1, t7, t6); \ @@ -84,8 +84,8 @@ filter_8bit(x6, t2, t3, t7, t6); \ \ /* AES subbytes + AES shift rows */ \ - vmovdqa .Lpost_tf_lo_s1, t0; \ - vmovdqa .Lpost_tf_hi_s1, t1; \ + vmovdqa .Lpost_tf_lo_s1(%rip), t0; \ + vmovdqa .Lpost_tf_hi_s1(%rip), t1; \ vaesenclast t4, x0, x0; \ vaesenclast t4, x7, x7; \ vaesenclast t4, x1, x1; \ @@ -96,16 +96,16 @@ vaesenclast t4, x6, x6; \ \ /* postfilter sboxes 1 and 4 */ \ - vmovdqa .Lpost_tf_lo_s3, t2; \ - vmovdqa .Lpost_tf_hi_s3, t3; \ + vmovdqa .Lpost_tf_lo_s3(%rip), t2; \ + vmovdqa .Lpost_tf_hi_s3(%rip), t3; \ filter_8bit(x0, t0, t1, t7, t6); \ filter_8bit(x7, t0, t1, t7, t6); \ filter_8bit(x3, t0, t1, t7, t6); \ filter_8bit(x6, t0, t1, t7, t6); \ \ /* postfilter sbox 3 */ \ - vmovdqa .Lpost_tf_lo_s2, t4; \ - vmovdqa .Lpost_tf_hi_s2, t5; \ + vmovdqa .Lpost_tf_lo_s2(%rip), t4; \ + vmovdqa .Lpost_tf_hi_s2(%rip), t5; \ filter_8bit(x2, t2, t3, t7, t6); \ filter_8bit(x5, t2, t3, t7, t6); \ \ @@ -444,7 +444,7 @@ SYM_FUNC_END(roundsm16_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab) transpose_4x4(c0, c1, c2, c3, a0, a1); \ transpose_4x4(d0, d1, d2, d3, a0, a1); \ \ - vmovdqu .Lshufb_16x16b, a0; \ + vmovdqu .Lshufb_16x16b(%rip), a0; \ vmovdqu st1, a1; \ vpshufb a0, a2, a2; \ vpshufb a0, a3, a3; \ @@ -483,7 +483,7 @@ SYM_FUNC_END(roundsm16_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab) #define inpack16_pre(x0, x1, x2, x3, x4, x5, x6, x7, y0, y1, y2, y3, y4, y5, \ y6, y7, rio, key) \ vmovq key, x0; \ - vpshufb .Lpack_bswap, x0, x0; \ + vpshufb .Lpack_bswap(%rip), x0, x0; \ \ vpxor 0 * 16(rio), x0, y7; \ vpxor 1 * 16(rio), x0, y6; \ @@ -534,7 +534,7 @@ SYM_FUNC_END(roundsm16_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab) vmovdqu x0, stack_tmp0; \ \ vmovq key, x0; \ - vpshufb .Lpack_bswap, x0, x0; \ + vpshufb .Lpack_bswap(%rip), x0, x0; \ \ vpxor x0, y7, y7; \ vpxor x0, y6, y6; \ @@ -1017,7 +1017,7 @@ SYM_FUNC_START(camellia_ctr_16way) subq $(16 * 16), %rsp; movq %rsp, %rax; - vmovdqa .Lbswap128_mask, %xmm14; + vmovdqa .Lbswap128_mask(%rip), %xmm14; /* load IV and byteswap */ vmovdqu (%rcx), %xmm0; @@ -1066,7 +1066,7 @@ SYM_FUNC_START(camellia_ctr_16way) /* inpack16_pre: */ vmovq (key_table)(CTX), %xmm15; - vpshufb .Lpack_bswap, %xmm15, %xmm15; + vpshufb .Lpack_bswap(%rip), %xmm15, %xmm15; vpxor %xmm0, %xmm15, %xmm0; vpxor %xmm1, %xmm15, %xmm1; vpxor %xmm2, %xmm15, %xmm2; @@ -1134,7 +1134,7 @@ SYM_FUNC_START_LOCAL(camellia_xts_crypt_16way) subq $(16 * 16), %rsp; movq %rsp, %rax; - vmovdqa .Lxts_gf128mul_and_shl1_mask, %xmm14; + vmovdqa .Lxts_gf128mul_and_shl1_mask(%rip), %xmm14; /* load IV */ vmovdqu (%rcx), %xmm0; @@ -1210,7 +1210,7 @@ SYM_FUNC_START_LOCAL(camellia_xts_crypt_16way) /* inpack16_pre: */ vmovq (key_table)(CTX, %r8, 8), %xmm15; - vpshufb .Lpack_bswap, %xmm15, %xmm15; + vpshufb .Lpack_bswap(%rip), %xmm15, %xmm15; vpxor 0 * 16(%rax), %xmm15, %xmm0; vpxor %xmm1, %xmm15, %xmm1; vpxor %xmm2, %xmm15, %xmm2; @@ -1265,7 +1265,7 @@ SYM_FUNC_START(camellia_xts_enc_16way) */ xorl %r8d, %r8d; /* input whitening key, 0 for enc */ - leaq __camellia_enc_blk16, %r9; + leaq __camellia_enc_blk16(%rip), %r9; jmp camellia_xts_crypt_16way; SYM_FUNC_END(camellia_xts_enc_16way) @@ -1283,7 +1283,7 @@ SYM_FUNC_START(camellia_xts_dec_16way) movl $24, %eax; cmovel %eax, %r8d; /* input whitening key, last for dec */ - leaq __camellia_dec_blk16, %r9; + leaq __camellia_dec_blk16(%rip), %r9; jmp camellia_xts_crypt_16way; SYM_FUNC_END(camellia_xts_dec_16way) diff --git a/arch/x86/crypto/camellia-aesni-avx2-asm_64.S b/arch/x86/crypto/camellia-aesni-avx2-asm_64.S index 563ef6e83cdd..d7f4cd4b1702 100644 --- a/arch/x86/crypto/camellia-aesni-avx2-asm_64.S +++ b/arch/x86/crypto/camellia-aesni-avx2-asm_64.S @@ -65,12 +65,12 @@ /* \ * S-function with AES subbytes \ */ \ - vbroadcasti128 .Linv_shift_row, t4; \ - vpbroadcastd .L0f0f0f0f, t7; \ - vbroadcasti128 .Lpre_tf_lo_s1, t5; \ - vbroadcasti128 .Lpre_tf_hi_s1, t6; \ - vbroadcasti128 .Lpre_tf_lo_s4, t2; \ - vbroadcasti128 .Lpre_tf_hi_s4, t3; \ + vbroadcasti128 .Linv_shift_row(%rip), t4; \ + vpbroadcastd .L0f0f0f0f(%rip), t7; \ + vbroadcasti128 .Lpre_tf_lo_s1(%rip), t5; \ + vbroadcasti128 .Lpre_tf_hi_s1(%rip), t6; \ + vbroadcasti128 .Lpre_tf_lo_s4(%rip), t2; \ + vbroadcasti128 .Lpre_tf_hi_s4(%rip), t3; \ \ /* AES inverse shift rows */ \ vpshufb t4, x0, x0; \ @@ -116,8 +116,8 @@ vinserti128 $1, t2##_x, x6, x6; \ vextracti128 $1, x1, t3##_x; \ vextracti128 $1, x4, t2##_x; \ - vbroadcasti128 .Lpost_tf_lo_s1, t0; \ - vbroadcasti128 .Lpost_tf_hi_s1, t1; \ + vbroadcasti128 .Lpost_tf_lo_s1(%rip), t0; \ + vbroadcasti128 .Lpost_tf_hi_s1(%rip), t1; \ vaesenclast t4##_x, x2##_x, x2##_x; \ vaesenclast t4##_x, t6##_x, t6##_x; \ vinserti128 $1, t6##_x, x2, x2; \ @@ -132,16 +132,16 @@ vinserti128 $1, t2##_x, x4, x4; \ \ /* postfilter sboxes 1 and 4 */ \ - vbroadcasti128 .Lpost_tf_lo_s3, t2; \ - vbroadcasti128 .Lpost_tf_hi_s3, t3; \ + vbroadcasti128 .Lpost_tf_lo_s3(%rip), t2; \ + vbroadcasti128 .Lpost_tf_hi_s3(%rip), t3; \ filter_8bit(x0, t0, t1, t7, t6); \ filter_8bit(x7, t0, t1, t7, t6); \ filter_8bit(x3, t0, t1, t7, t6); \ filter_8bit(x6, t0, t1, t7, t6); \ \ /* postfilter sbox 3 */ \ - vbroadcasti128 .Lpost_tf_lo_s2, t4; \ - vbroadcasti128 .Lpost_tf_hi_s2, t5; \ + vbroadcasti128 .Lpost_tf_lo_s2(%rip), t4; \ + vbroadcasti128 .Lpost_tf_hi_s2(%rip), t5; \ filter_8bit(x2, t2, t3, t7, t6); \ filter_8bit(x5, t2, t3, t7, t6); \ \ @@ -478,7 +478,7 @@ SYM_FUNC_END(roundsm32_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab) transpose_4x4(c0, c1, c2, c3, a0, a1); \ transpose_4x4(d0, d1, d2, d3, a0, a1); \ \ - vbroadcasti128 .Lshufb_16x16b, a0; \ + vbroadcasti128 .Lshufb_16x16b(%rip), a0; \ vmovdqu st1, a1; \ vpshufb a0, a2, a2; \ vpshufb a0, a3, a3; \ @@ -517,7 +517,7 @@ SYM_FUNC_END(roundsm32_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab) #define inpack32_pre(x0, x1, x2, x3, x4, x5, x6, x7, y0, y1, y2, y3, y4, y5, \ y6, y7, rio, key) \ vpbroadcastq key, x0; \ - vpshufb .Lpack_bswap, x0, x0; \ + vpshufb .Lpack_bswap(%rip), x0, x0; \ \ vpxor 0 * 32(rio), x0, y7; \ vpxor 1 * 32(rio), x0, y6; \ @@ -568,7 +568,7 @@ SYM_FUNC_END(roundsm32_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab) vmovdqu x0, stack_tmp0; \ \ vpbroadcastq key, x0; \ - vpshufb .Lpack_bswap, x0, x0; \ + vpshufb .Lpack_bswap(%rip), x0, x0; \ \ vpxor x0, y7, y7; \ vpxor x0, y6, y6; \ @@ -1108,7 +1108,7 @@ SYM_FUNC_START(camellia_ctr_32way) vmovdqu (%rcx), %xmm0; vmovdqa %xmm0, %xmm1; inc_le128(%xmm0, %xmm15, %xmm14); - vbroadcasti128 .Lbswap128_mask, %ymm14; + vbroadcasti128 .Lbswap128_mask(%rip), %ymm14; vinserti128 $1, %xmm0, %ymm1, %ymm0; vpshufb %ymm14, %ymm0, %ymm13; vmovdqu %ymm13, 15 * 32(%rax); @@ -1154,7 +1154,7 @@ SYM_FUNC_START(camellia_ctr_32way) /* inpack32_pre: */ vpbroadcastq (key_table)(CTX), %ymm15; - vpshufb .Lpack_bswap, %ymm15, %ymm15; + vpshufb .Lpack_bswap(%rip), %ymm15, %ymm15; vpxor %ymm0, %ymm15, %ymm0; vpxor %ymm1, %ymm15, %ymm1; vpxor %ymm2, %ymm15, %ymm2; @@ -1238,13 +1238,13 @@ SYM_FUNC_START_LOCAL(camellia_xts_crypt_32way) subq $(16 * 32), %rsp; movq %rsp, %rax; - vbroadcasti128 .Lxts_gf128mul_and_shl1_mask_0, %ymm12; + vbroadcasti128 .Lxts_gf128mul_and_shl1_mask_0(%rip), %ymm12; /* load IV and construct second IV */ vmovdqu (%rcx), %xmm0; vmovdqa %xmm0, %xmm15; gf128mul_x_ble(%xmm0, %xmm12, %xmm13); - vbroadcasti128 .Lxts_gf128mul_and_shl1_mask_1, %ymm13; + vbroadcasti128 .Lxts_gf128mul_and_shl1_mask_1(%rip), %ymm13; vinserti128 $1, %xmm0, %ymm15, %ymm0; vpxor 0 * 32(%rdx), %ymm0, %ymm15; vmovdqu %ymm15, 15 * 32(%rax); @@ -1321,7 +1321,7 @@ SYM_FUNC_START_LOCAL(camellia_xts_crypt_32way) /* inpack32_pre: */ vpbroadcastq (key_table)(CTX, %r8, 8), %ymm15; - vpshufb .Lpack_bswap, %ymm15, %ymm15; + vpshufb .Lpack_bswap(%rip), %ymm15, %ymm15; vpxor 0 * 32(%rax), %ymm15, %ymm0; vpxor %ymm1, %ymm15, %ymm1; vpxor %ymm2, %ymm15, %ymm2; @@ -1379,7 +1379,7 @@ SYM_FUNC_START(camellia_xts_enc_32way) xorl %r8d, %r8d; /* input whitening key, 0 for enc */ - leaq __camellia_enc_blk32, %r9; + leaq __camellia_enc_blk32(%rip), %r9; jmp camellia_xts_crypt_32way; SYM_FUNC_END(camellia_xts_enc_32way) @@ -1397,7 +1397,7 @@ SYM_FUNC_START(camellia_xts_dec_32way) movl $24, %eax; cmovel %eax, %r8d; /* input whitening key, last for dec */ - leaq __camellia_dec_blk32, %r9; + leaq __camellia_dec_blk32(%rip), %r9; jmp camellia_xts_crypt_32way; SYM_FUNC_END(camellia_xts_dec_32way) diff --git a/arch/x86/crypto/camellia-x86_64-asm_64.S b/arch/x86/crypto/camellia-x86_64-asm_64.S index 1372e6408850..3a4e5cca583a 100644 --- a/arch/x86/crypto/camellia-x86_64-asm_64.S +++ b/arch/x86/crypto/camellia-x86_64-asm_64.S @@ -77,11 +77,13 @@ #define RXORbl %r9b #define xor2ror16(T0, T1, tmp1, tmp2, ab, dst) \ + leaq T0(%rip), tmp1; \ movzbl ab ## bl, tmp2 ## d; \ + xorq (tmp1, tmp2, 8), dst; \ + leaq T1(%rip), tmp2; \ movzbl ab ## bh, tmp1 ## d; \ - rorq $16, ab; \ - xorq T0(, tmp2, 8), dst; \ - xorq T1(, tmp1, 8), dst; + xorq (tmp2, tmp1, 8), dst; \ + rorq $16, ab; /********************************************************************** 1-way camellia diff --git a/arch/x86/crypto/cast5-avx-x86_64-asm_64.S b/arch/x86/crypto/cast5-avx-x86_64-asm_64.S index 8a6181b08b59..ffa9ed5f096b 100644 --- a/arch/x86/crypto/cast5-avx-x86_64-asm_64.S +++ b/arch/x86/crypto/cast5-avx-x86_64-asm_64.S @@ -83,16 +83,20 @@ #define lookup_32bit(src, dst, op1, op2, op3, interleave_op, il_reg) \ - movzbl src ## bh, RID1d; \ - movzbl src ## bl, RID2d; \ - shrq $16, src; \ - movl s1(, RID1, 4), dst ## d; \ - op1 s2(, RID2, 4), dst ## d; \ - movzbl src ## bh, RID1d; \ - movzbl src ## bl, RID2d; \ - interleave_op(il_reg); \ - op2 s3(, RID1, 4), dst ## d; \ - op3 s4(, RID2, 4), dst ## d; + movzbl src ## bh, RID1d; \ + leaq s1(%rip), RID2; \ + movl (RID2, RID1, 4), dst ## d; \ + movzbl src ## bl, RID2d; \ + leaq s2(%rip), RID1; \ + op1 (RID1, RID2, 4), dst ## d; \ + shrq $16, src; \ + movzbl src ## bh, RID1d; \ + leaq s3(%rip), RID2; \ + op2 (RID2, RID1, 4), dst ## d; \ + movzbl src ## bl, RID2d; \ + leaq s4(%rip), RID1; \ + op3 (RID1, RID2, 4), dst ## d; \ + interleave_op(il_reg); #define dummy(d) /* do nothing */ @@ -151,15 +155,15 @@ subround(l ## 3, r ## 3, l ## 4, r ## 4, f); #define enc_preload_rkr() \ - vbroadcastss .L16_mask, RKR; \ + vbroadcastss .L16_mask(%rip), RKR; \ /* add 16-bit rotation to key rotations (mod 32) */ \ vpxor kr(CTX), RKR, RKR; #define dec_preload_rkr() \ - vbroadcastss .L16_mask, RKR; \ + vbroadcastss .L16_mask(%rip), RKR; \ /* add 16-bit rotation to key rotations (mod 32) */ \ vpxor kr(CTX), RKR, RKR; \ - vpshufb .Lbswap128_mask, RKR, RKR; + vpshufb .Lbswap128_mask(%rip), RKR, RKR; #define transpose_2x4(x0, x1, t0, t1) \ vpunpckldq x1, x0, t0; \ @@ -236,9 +240,9 @@ SYM_FUNC_START_LOCAL(__cast5_enc_blk16) movq %rdi, CTX; - vmovdqa .Lbswap_mask, RKM; - vmovd .Lfirst_mask, R1ST; - vmovd .L32_mask, R32; + vmovdqa .Lbswap_mask(%rip), RKM; + vmovd .Lfirst_mask(%rip), R1ST; + vmovd .L32_mask(%rip), R32; enc_preload_rkr(); inpack_blocks(RL1, RR1, RTMP, RX, RKM); @@ -272,7 +276,7 @@ SYM_FUNC_START_LOCAL(__cast5_enc_blk16) popq %rbx; popq %r15; - vmovdqa .Lbswap_mask, RKM; + vmovdqa .Lbswap_mask(%rip), RKM; outunpack_blocks(RR1, RL1, RTMP, RX, RKM); outunpack_blocks(RR2, RL2, RTMP, RX, RKM); @@ -310,9 +314,9 @@ SYM_FUNC_START_LOCAL(__cast5_dec_blk16) movq %rdi, CTX; - vmovdqa .Lbswap_mask, RKM; - vmovd .Lfirst_mask, R1ST; - vmovd .L32_mask, R32; + vmovdqa .Lbswap_mask(%rip), RKM; + vmovd .Lfirst_mask(%rip), R1ST; + vmovd .L32_mask(%rip), R32; dec_preload_rkr(); inpack_blocks(RL1, RR1, RTMP, RX, RKM); @@ -343,7 +347,7 @@ SYM_FUNC_START_LOCAL(__cast5_dec_blk16) round(RL, RR, 1, 2); round(RR, RL, 0, 1); - vmovdqa .Lbswap_mask, RKM; + vmovdqa .Lbswap_mask(%rip), RKM; popq %rbx; popq %r15; @@ -506,8 +510,8 @@ SYM_FUNC_START(cast5_ctr_16way) vpcmpeqd RKR, RKR, RKR; vpaddq RKR, RKR, RKR; /* low: -2, high: -2 */ - vmovdqa .Lbswap_iv_mask, R1ST; - vmovdqa .Lbswap128_mask, RKM; + vmovdqa .Lbswap_iv_mask(%rip), R1ST; + vmovdqa .Lbswap128_mask(%rip), RKM; /* load IV and byteswap */ vmovq (%rcx), RX; diff --git a/arch/x86/crypto/cast6-avx-x86_64-asm_64.S b/arch/x86/crypto/cast6-avx-x86_64-asm_64.S index 932a3ce32a88..c46a61411f98 100644 --- a/arch/x86/crypto/cast6-avx-x86_64-asm_64.S +++ b/arch/x86/crypto/cast6-avx-x86_64-asm_64.S @@ -83,16 +83,20 @@ #define lookup_32bit(src, dst, op1, op2, op3, interleave_op, il_reg) \ - movzbl src ## bh, RID1d; \ - movzbl src ## bl, RID2d; \ - shrq $16, src; \ - movl s1(, RID1, 4), dst ## d; \ - op1 s2(, RID2, 4), dst ## d; \ - movzbl src ## bh, RID1d; \ - movzbl src ## bl, RID2d; \ - interleave_op(il_reg); \ - op2 s3(, RID1, 4), dst ## d; \ - op3 s4(, RID2, 4), dst ## d; + movzbl src ## bh, RID1d; \ + leaq s1(%rip), RID2; \ + movl (RID2, RID1, 4), dst ## d; \ + movzbl src ## bl, RID2d; \ + leaq s2(%rip), RID1; \ + op1 (RID1, RID2, 4), dst ## d; \ + shrq $16, src; \ + movzbl src ## bh, RID1d; \ + leaq s3(%rip), RID2; \ + op2 (RID2, RID1, 4), dst ## d; \ + movzbl src ## bl, RID2d; \ + leaq s4(%rip), RID1; \ + op3 (RID1, RID2, 4), dst ## d; \ + interleave_op(il_reg); #define dummy(d) /* do nothing */ @@ -175,10 +179,10 @@ qop(RD, RC, 1); #define shuffle(mask) \ - vpshufb mask, RKR, RKR; + vpshufb mask(%rip), RKR, RKR; #define preload_rkr(n, do_mask, mask) \ - vbroadcastss .L16_mask, RKR; \ + vbroadcastss .L16_mask(%rip), RKR; \ /* add 16-bit rotation to key rotations (mod 32) */ \ vpxor (kr+n*16)(CTX), RKR, RKR; \ do_mask(mask); @@ -260,9 +264,9 @@ SYM_FUNC_START_LOCAL(__cast6_enc_blk8) movq %rdi, CTX; - vmovdqa .Lbswap_mask, RKM; - vmovd .Lfirst_mask, R1ST; - vmovd .L32_mask, R32; + vmovdqa .Lbswap_mask(%rip), RKM; + vmovd .Lfirst_mask(%rip), R1ST; + vmovd .L32_mask(%rip), R32; inpack_blocks(RA1, RB1, RC1, RD1, RTMP, RX, RKRF, RKM); inpack_blocks(RA2, RB2, RC2, RD2, RTMP, RX, RKRF, RKM); @@ -286,7 +290,7 @@ SYM_FUNC_START_LOCAL(__cast6_enc_blk8) popq %rbx; popq %r15; - vmovdqa .Lbswap_mask, RKM; + vmovdqa .Lbswap_mask(%rip), RKM; outunpack_blocks(RA1, RB1, RC1, RD1, RTMP, RX, RKRF, RKM); outunpack_blocks(RA2, RB2, RC2, RD2, RTMP, RX, RKRF, RKM); @@ -308,9 +312,9 @@ SYM_FUNC_START_LOCAL(__cast6_dec_blk8) movq %rdi, CTX; - vmovdqa .Lbswap_mask, RKM; - vmovd .Lfirst_mask, R1ST; - vmovd .L32_mask, R32; + vmovdqa .Lbswap_mask(%rip), RKM; + vmovd .Lfirst_mask(%rip), R1ST; + vmovd .L32_mask(%rip), R32; inpack_blocks(RA1, RB1, RC1, RD1, RTMP, RX, RKRF, RKM); inpack_blocks(RA2, RB2, RC2, RD2, RTMP, RX, RKRF, RKM); @@ -334,7 +338,7 @@ SYM_FUNC_START_LOCAL(__cast6_dec_blk8) popq %rbx; popq %r15; - vmovdqa .Lbswap_mask, RKM; + vmovdqa .Lbswap_mask(%rip), RKM; outunpack_blocks(RA1, RB1, RC1, RD1, RTMP, RX, RKRF, RKM); outunpack_blocks(RA2, RB2, RC2, RD2, RTMP, RX, RKRF, RKM); diff --git a/arch/x86/crypto/des3_ede-asm_64.S b/arch/x86/crypto/des3_ede-asm_64.S index fac0fdc3f25d..c841de5fe065 100644 --- a/arch/x86/crypto/des3_ede-asm_64.S +++ b/arch/x86/crypto/des3_ede-asm_64.S @@ -129,21 +129,29 @@ movzbl RW0bl, RT2d; \ movzbl RW0bh, RT3d; \ shrq $16, RW0; \ - movq s8(, RT0, 8), RT0; \ - xorq s6(, RT1, 8), to; \ + leaq s8(%rip), RW1; \ + movq (RW1, RT0, 8), RT0; \ + leaq s6(%rip), RW1; \ + xorq (RW1, RT1, 8), to; \ movzbl RW0bl, RL1d; \ movzbl RW0bh, RT1d; \ shrl $16, RW0d; \ - xorq s4(, RT2, 8), RT0; \ - xorq s2(, RT3, 8), to; \ + leaq s4(%rip), RW1; \ + xorq (RW1, RT2, 8), RT0; \ + leaq s2(%rip), RW1; \ + xorq (RW1, RT3, 8), to; \ movzbl RW0bl, RT2d; \ movzbl RW0bh, RT3d; \ - xorq s7(, RL1, 8), RT0; \ - xorq s5(, RT1, 8), to; \ - xorq s3(, RT2, 8), RT0; \ + leaq s7(%rip), RW1; \ + xorq (RW1, RL1, 8), RT0; \ + leaq s5(%rip), RW1; \ + xorq (RW1, RT1, 8), to; \ + leaq s3(%rip), RW1; \ + xorq (RW1, RT2, 8), RT0; \ load_next_key(n, RW0); \ xorq RT0, to; \ - xorq s1(, RT3, 8), to; \ + leaq s1(%rip), RW1; \ + xorq (RW1, RT3, 8), to; \ #define load_next_key(n, RWx) \ movq (((n) + 1) * 8)(CTX), RWx; @@ -355,65 +363,89 @@ SYM_FUNC_END(des3_ede_x86_64_crypt_blk) movzbl RW0bl, RT3d; \ movzbl RW0bh, RT1d; \ shrq $16, RW0; \ - xorq s8(, RT3, 8), to##0; \ - xorq s6(, RT1, 8), to##0; \ + leaq s8(%rip), RT2; \ + xorq (RT2, RT3, 8), to##0; \ + leaq s6(%rip), RT2; \ + xorq (RT2, RT1, 8), to##0; \ movzbl RW0bl, RT3d; \ movzbl RW0bh, RT1d; \ shrq $16, RW0; \ - xorq s4(, RT3, 8), to##0; \ - xorq s2(, RT1, 8), to##0; \ + leaq s4(%rip), RT2; \ + xorq (RT2, RT3, 8), to##0; \ + leaq s2(%rip), RT2; \ + xorq (RT2, RT1, 8), to##0; \ movzbl RW0bl, RT3d; \ movzbl RW0bh, RT1d; \ shrl $16, RW0d; \ - xorq s7(, RT3, 8), to##0; \ - xorq s5(, RT1, 8), to##0; \ + leaq s7(%rip), RT2; \ + xorq (RT2, RT3, 8), to##0; \ + leaq s5(%rip), RT2; \ + xorq (RT2, RT1, 8), to##0; \ movzbl RW0bl, RT3d; \ movzbl RW0bh, RT1d; \ load_next_key(n, RW0); \ - xorq s3(, RT3, 8), to##0; \ - xorq s1(, RT1, 8), to##0; \ + leaq s3(%rip), RT2; \ + xorq (RT2, RT3, 8), to##0; \ + leaq s1(%rip), RT2; \ + xorq (RT2, RT1, 8), to##0; \ xorq from##1, RW1; \ movzbl RW1bl, RT3d; \ movzbl RW1bh, RT1d; \ shrq $16, RW1; \ - xorq s8(, RT3, 8), to##1; \ - xorq s6(, RT1, 8), to##1; \ + leaq s8(%rip), RT2; \ + xorq (RT2, RT3, 8), to##1; \ + leaq s6(%rip), RT2; \ + xorq (RT2, RT1, 8), to##1; \ movzbl RW1bl, RT3d; \ movzbl RW1bh, RT1d; \ shrq $16, RW1; \ - xorq s4(, RT3, 8), to##1; \ - xorq s2(, RT1, 8), to##1; \ + leaq s4(%rip), RT2; \ + xorq (RT2, RT3, 8), to##1; \ + leaq s2(%rip), RT2; \ + xorq (RT2, RT1, 8), to##1; \ movzbl RW1bl, RT3d; \ movzbl RW1bh, RT1d; \ shrl $16, RW1d; \ - xorq s7(, RT3, 8), to##1; \ - xorq s5(, RT1, 8), to##1; \ + leaq s7(%rip), RT2; \ + xorq (RT2, RT3, 8), to##1; \ + leaq s5(%rip), RT2; \ + xorq (RT2, RT1, 8), to##1; \ movzbl RW1bl, RT3d; \ movzbl RW1bh, RT1d; \ do_movq(RW0, RW1); \ - xorq s3(, RT3, 8), to##1; \ - xorq s1(, RT1, 8), to##1; \ + leaq s3(%rip), RT2; \ + xorq (RT2, RT3, 8), to##1; \ + leaq s1(%rip), RT2; \ + xorq (RT2, RT1, 8), to##1; \ xorq from##2, RW2; \ movzbl RW2bl, RT3d; \ movzbl RW2bh, RT1d; \ shrq $16, RW2; \ - xorq s8(, RT3, 8), to##2; \ - xorq s6(, RT1, 8), to##2; \ + leaq s8(%rip), RT2; \ + xorq (RT2, RT3, 8), to##2; \ + leaq s6(%rip), RT2; \ + xorq (RT2, RT1, 8), to##2; \ movzbl RW2bl, RT3d; \ movzbl RW2bh, RT1d; \ shrq $16, RW2; \ - xorq s4(, RT3, 8), to##2; \ - xorq s2(, RT1, 8), to##2; \ + leaq s4(%rip), RT2; \ + xorq (RT2, RT3, 8), to##2; \ + leaq s2(%rip), RT2; \ + xorq (RT2, RT1, 8), to##2; \ movzbl RW2bl, RT3d; \ movzbl RW2bh, RT1d; \ shrl $16, RW2d; \ - xorq s7(, RT3, 8), to##2; \ - xorq s5(, RT1, 8), to##2; \ + leaq s7(%rip), RT2; \ + xorq (RT2, RT3, 8), to##2; \ + leaq s5(%rip), RT2; \ + xorq (RT2, RT1, 8), to##2; \ movzbl RW2bl, RT3d; \ movzbl RW2bh, RT1d; \ do_movq(RW0, RW2); \ - xorq s3(, RT3, 8), to##2; \ - xorq s1(, RT1, 8), to##2; + leaq s3(%rip), RT2; \ + xorq (RT2, RT3, 8), to##2; \ + leaq s1(%rip), RT2; \ + xorq (RT2, RT1, 8), to##2; #define __movq(src, dst) \ movq src, dst; diff --git a/arch/x86/crypto/ghash-clmulni-intel_asm.S b/arch/x86/crypto/ghash-clmulni-intel_asm.S index bb9735fbb865..59717ade66d7 100644 --- a/arch/x86/crypto/ghash-clmulni-intel_asm.S +++ b/arch/x86/crypto/ghash-clmulni-intel_asm.S @@ -94,7 +94,7 @@ SYM_FUNC_START(clmul_ghash_mul) FRAME_BEGIN movups (%rdi), DATA movups (%rsi), SHASH - movaps .Lbswap_mask, BSWAP + movaps .Lbswap_mask(%rip), BSWAP PSHUFB_XMM BSWAP DATA call __clmul_gf128mul_ble PSHUFB_XMM BSWAP DATA @@ -111,7 +111,7 @@ SYM_FUNC_START(clmul_ghash_update) FRAME_BEGIN cmp $16, %rdx jb .Lupdate_just_ret # check length - movaps .Lbswap_mask, BSWAP + movaps .Lbswap_mask(%rip), BSWAP movups (%rdi), DATA movups (%rcx), SHASH PSHUFB_XMM BSWAP DATA diff --git a/arch/x86/crypto/glue_helper-asm-avx.S b/arch/x86/crypto/glue_helper-asm-avx.S index d08fc575ef7f..a9736f85fef0 100644 --- a/arch/x86/crypto/glue_helper-asm-avx.S +++ b/arch/x86/crypto/glue_helper-asm-avx.S @@ -44,7 +44,7 @@ #define load_ctr_8way(iv, bswap, x0, x1, x2, x3, x4, x5, x6, x7, t0, t1, t2) \ vpcmpeqd t0, t0, t0; \ vpsrldq $8, t0, t0; /* low: -1, high: 0 */ \ - vmovdqa bswap, t1; \ + vmovdqa bswap(%rip), t1; \ \ /* load IV and byteswap */ \ vmovdqu (iv), x7; \ @@ -89,7 +89,7 @@ #define load_xts_8way(iv, src, dst, x0, x1, x2, x3, x4, x5, x6, x7, tiv, t0, \ t1, xts_gf128mul_and_shl1_mask) \ - vmovdqa xts_gf128mul_and_shl1_mask, t0; \ + vmovdqa xts_gf128mul_and_shl1_mask(%rip), t0; \ \ /* load IV */ \ vmovdqu (iv), tiv; \ diff --git a/arch/x86/crypto/glue_helper-asm-avx2.S b/arch/x86/crypto/glue_helper-asm-avx2.S index d84508c85c13..efbf4953707e 100644 --- a/arch/x86/crypto/glue_helper-asm-avx2.S +++ b/arch/x86/crypto/glue_helper-asm-avx2.S @@ -62,7 +62,7 @@ vmovdqu (iv), t2x; \ vmovdqa t2x, t3x; \ inc_le128(t2x, t0x, t1x); \ - vbroadcasti128 bswap, t1; \ + vbroadcasti128 bswap(%rip), t1; \ vinserti128 $1, t2x, t3, t2; /* ab: le0 ; cd: le1 */ \ vpshufb t1, t2, x0; \ \ @@ -119,13 +119,13 @@ tivx, t0, t0x, t1, t1x, t2, t2x, t3, \ xts_gf128mul_and_shl1_mask_0, \ xts_gf128mul_and_shl1_mask_1) \ - vbroadcasti128 xts_gf128mul_and_shl1_mask_0, t1; \ + vbroadcasti128 xts_gf128mul_and_shl1_mask_0(%rip), t1; \ \ /* load IV and construct second IV */ \ vmovdqu (iv), tivx; \ vmovdqa tivx, t0x; \ gf128mul_x_ble(tivx, t1x, t2x); \ - vbroadcasti128 xts_gf128mul_and_shl1_mask_1, t2; \ + vbroadcasti128 xts_gf128mul_and_shl1_mask_1(%rip), t2; \ vinserti128 $1, tivx, t0, tiv; \ vpxor (0*32)(src), tiv, x0; \ vmovdqu tiv, (0*32)(dst); \ diff --git a/arch/x86/crypto/sha256-avx2-asm.S b/arch/x86/crypto/sha256-avx2-asm.S index 499d9ec129de..4b2a08445dc1 100644 --- a/arch/x86/crypto/sha256-avx2-asm.S +++ b/arch/x86/crypto/sha256-avx2-asm.S @@ -592,19 +592,23 @@ last_block_enter: .align 16 loop1: - vpaddd K256+0*32(SRND), X0, XFER + leaq K256(%rip), INP + vpaddd 0*32(INP, SRND), X0, XFER vmovdqa XFER, 0*32+_XFER(%rsp, SRND) FOUR_ROUNDS_AND_SCHED _XFER + 0*32 - vpaddd K256+1*32(SRND), X0, XFER + leaq K256(%rip), INP + vpaddd 1*32(INP, SRND), X0, XFER vmovdqa XFER, 1*32+_XFER(%rsp, SRND) FOUR_ROUNDS_AND_SCHED _XFER + 1*32 - vpaddd K256+2*32(SRND), X0, XFER + leaq K256(%rip), INP + vpaddd 2*32(INP, SRND), X0, XFER vmovdqa XFER, 2*32+_XFER(%rsp, SRND) FOUR_ROUNDS_AND_SCHED _XFER + 2*32 - vpaddd K256+3*32(SRND), X0, XFER + leaq K256(%rip), INP + vpaddd 3*32(INP, SRND), X0, XFER vmovdqa XFER, 3*32+_XFER(%rsp, SRND) FOUR_ROUNDS_AND_SCHED _XFER + 3*32 @@ -614,11 +618,13 @@ loop1: loop2: ## Do last 16 rounds with no scheduling - vpaddd K256+0*32(SRND), X0, XFER + leaq K256(%rip), INP + vpaddd 0*32(INP, SRND), X0, XFER vmovdqa XFER, 0*32+_XFER(%rsp, SRND) DO_4ROUNDS _XFER + 0*32 - vpaddd K256+1*32(SRND), X1, XFER + leaq K256(%rip), INP + vpaddd 1*32(INP, SRND), X1, XFER vmovdqa XFER, 1*32+_XFER(%rsp, SRND) DO_4ROUNDS _XFER + 1*32 add $2*32, SRND From patchwork Fri Feb 28 00:00:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Garnier X-Patchwork-Id: 11411167 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D7B8B138D for ; Fri, 28 Feb 2020 00:01:44 +0000 (UTC) Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.kernel.org (Postfix) with SMTP id 3F121246A5 for ; Fri, 28 Feb 2020 00:01:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="R0fDgyKL" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3F121246A5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kernel-hardening-return-17993-patchwork-kernel-hardening=patchwork.kernel.org@lists.openwall.com Received: (qmail 21529 invoked by uid 550); 28 Feb 2020 00:01:27 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 20371 invoked from network); 28 Feb 2020 00:01:25 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fvhzUd6Qu7cjrcCOJttmYiBqe1hyVQ5v3hyhsFuyKi8=; b=R0fDgyKLTHQkfsWhWRkFBROySWE56w7r6inEPzUviguIQfBELPtivu7yY4Ik9lv7t7 /nOIfMq8t2T13X1qgjHYJJ5cidxN2ymJ2qiAEIYtGpRaD6z6Ni0j6HvemLNDgKpXyqwv tyPUm4U3B3hIYcksrbm95hIv5aO+ZSAMYtXnc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fvhzUd6Qu7cjrcCOJttmYiBqe1hyVQ5v3hyhsFuyKi8=; b=iL9CN488pP72NIBiFXfARVMR8ZI7pT7H3YEHbwGORV0AA27+d/C4oCupelIRpg/U7X 4SEX3ewJoMh0hG5PxO1seHxK9v5qY/dZ2QmIyXL6TgcYzluTAkgBEYZK9/qzCR6PBSMB CvH1YLz++tW1qY+63nIpokEoJ7vAYAA7oSmbizEKd5t7GP858RfEl0IOc9jZ2ujSJL2f n7JIcg+JfV24aiARfEJ8lKS4supujpiUrOMArXA8OGqAVmYxFTpZc/KrIwVPjYtE2bZ0 nEgUGskyHm4QH9qDA938pRyrX7CmtSGrf6qxlbCzdKQJIE/cET7K/wSjk77k4RF/3qCq nQfQ== X-Gm-Message-State: APjAAAWpOhUXl2K2UhELLQ/bCBdkwOKtTCLm9J7zLGrC+rH7Vu2GPa/0 ki0OLALX0lvxA61Jkmw70sEk3djFp/Q= X-Google-Smtp-Source: APXvYqwThB4LJ42cXGHIwD4VP+n+OceP9p72vw3hubV+kmuIORRSU7xn+cXTMOPaKk/qKAWgjHeHnQ== X-Received: by 2002:a63:4e22:: with SMTP id c34mr1833461pgb.263.1582848073452; Thu, 27 Feb 2020 16:01:13 -0800 (PST) From: Thomas Garnier To: kernel-hardening@lists.openwall.com Cc: kristen@linux.intel.com, keescook@chromium.org, Thomas Garnier , Thomas Gleixner , Ingo Molnar , Borislav Petkov , "H. Peter Anvin" , x86@kernel.org, Peter Zijlstra , Masami Hiramatsu , Will Deacon , linux-kernel@vger.kernel.org Subject: [PATCH v11 02/11] x86: Add macro to get symbol address for PIE support Date: Thu, 27 Feb 2020 16:00:47 -0800 Message-Id: <20200228000105.165012-3-thgarnie@chromium.org> X-Mailer: git-send-email 2.25.1.481.gfbce0eb801-goog In-Reply-To: <20200228000105.165012-1-thgarnie@chromium.org> References: <20200228000105.165012-1-thgarnie@chromium.org> MIME-Version: 1.0 Add a new _ASM_MOVABS macro to fetch a symbol address. Replace "_ASM_MOV $, %dst" code construct that are not compatible with PIE. Signed-off-by: Thomas Garnier --- arch/x86/include/asm/asm.h | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/x86/include/asm/asm.h b/arch/x86/include/asm/asm.h index cd339b88d5d4..644bdbf149ee 100644 --- a/arch/x86/include/asm/asm.h +++ b/arch/x86/include/asm/asm.h @@ -32,6 +32,7 @@ #define _ASM_ALIGN __ASM_SEL(.balign 4, .balign 8) #define _ASM_MOV __ASM_SIZE(mov) +#define _ASM_MOVABS __ASM_SEL(movl, movabsq) #define _ASM_INC __ASM_SIZE(inc) #define _ASM_DEC __ASM_SIZE(dec) #define _ASM_ADD __ASM_SIZE(add) From patchwork Fri Feb 28 00:00:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Garnier X-Patchwork-Id: 11411169 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B25DA14BC for ; Fri, 28 Feb 2020 00:01:52 +0000 (UTC) Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.kernel.org (Postfix) with SMTP id 1B2AF2087F for ; Fri, 28 Feb 2020 00:01:51 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="JB82RZGV" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1B2AF2087F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kernel-hardening-return-17994-patchwork-kernel-hardening=patchwork.kernel.org@lists.openwall.com Received: (qmail 21670 invoked by uid 550); 28 Feb 2020 00:01:29 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 21536 invoked from network); 28 Feb 2020 00:01:27 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=t9NQOt5/Iwvu5cqFSUEIZpoioa79suNFyUPWf7TcFVs=; b=JB82RZGVMkxb7ufWndNbPMc9DLJBVuoLI8gqPesLGOqASB0Yq2fjodsUYxz9QdgrWT zM1/pGv08yc4l0mSk65GWRzDC7WMudixLLIWroLROrroJVCpBzp47d8Sil2XIYZfnkfS /bWmEwI9PFbsdfvUzibFyKLw6T0o6x8B66oMQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=t9NQOt5/Iwvu5cqFSUEIZpoioa79suNFyUPWf7TcFVs=; b=Ztqnxg24qil0PtpKnjE+wkhjEmz+m5VRMT3pEb3LwjUDWBO2CE6q3CoFBnsiERE6+i 6K1Fz7KYmZbx5F3cDwdbwoDxX7nkyV3FHWl2Q/n/SKBR3sdBlCuZdEZGX25raWrFo1dd TDUKwoQiI5vG6OLAgtYip1A4LGRjXgcI89Dnluoi9oAO45GG8xS+ZnIb2VJfBUvGEWLB 22En7g/HDysMp8PrtOCvaIBZlM9vRuTO/7sEHbV5I7XYKwAFMVDKB9W/gEUhxwXKka+K Q83n9J2bL/Ntwofx89qVCn4b4x9R31iNbyVQhKQA6xzdfSaRh71o3jiagwqqAHNjRWCS TPIg== X-Gm-Message-State: APjAAAV/WkuOXhG2qedlHlLKBzaiZLyStAb8QByWYjCW/zAYTG5i6FMz WfYY6bokRGnLbxW4jNCii8TCxc4JRrM= X-Google-Smtp-Source: APXvYqzNoR8mlRXvViWtEmB8PnTtW0UYVCLnvzxcFZTeurjSH9rB3lHogTj2OBxirQo1QrneXJkX9w== X-Received: by 2002:a63:cf06:: with SMTP id j6mr1730502pgg.379.1582848075655; Thu, 27 Feb 2020 16:01:15 -0800 (PST) From: Thomas Garnier To: kernel-hardening@lists.openwall.com Cc: kristen@linux.intel.com, keescook@chromium.org, Thomas Garnier , Thomas Gleixner , Ingo Molnar , Borislav Petkov , "H. Peter Anvin" , x86@kernel.org, Allison Randal , Enrico Weigelt , Greg Kroah-Hartman , Jiri Slaby , linux-kernel@vger.kernel.org Subject: [PATCH v11 03/11] x86: relocate_kernel - Adapt assembly for PIE support Date: Thu, 27 Feb 2020 16:00:48 -0800 Message-Id: <20200228000105.165012-4-thgarnie@chromium.org> X-Mailer: git-send-email 2.25.1.481.gfbce0eb801-goog In-Reply-To: <20200228000105.165012-1-thgarnie@chromium.org> References: <20200228000105.165012-1-thgarnie@chromium.org> MIME-Version: 1.0 Change the assembly code to use only absolute references of symbols for the kernel to be PIE compatible. Signed-off-by: Thomas Garnier Reviewed-by: Kees Cook --- arch/x86/kernel/relocate_kernel_64.S | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kernel/relocate_kernel_64.S b/arch/x86/kernel/relocate_kernel_64.S index ef3ba99068d3..c294339df5ef 100644 --- a/arch/x86/kernel/relocate_kernel_64.S +++ b/arch/x86/kernel/relocate_kernel_64.S @@ -206,7 +206,7 @@ SYM_CODE_START_LOCAL_NOALIGN(identity_mapped) movq %rax, %cr3 lea PAGE_SIZE(%r8), %rsp call swap_pages - movq $virtual_mapped, %rax + movabsq $virtual_mapped, %rax pushq %rax ret SYM_CODE_END(identity_mapped) From patchwork Fri Feb 28 00:00:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Garnier X-Patchwork-Id: 11411171 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8D2DA14BC for ; Fri, 28 Feb 2020 00:02:00 +0000 (UTC) Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.kernel.org (Postfix) with SMTP id E9C4D2087F for ; Fri, 28 Feb 2020 00:01:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="KzW3r8Gf" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E9C4D2087F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kernel-hardening-return-17995-patchwork-kernel-hardening=patchwork.kernel.org@lists.openwall.com Received: (qmail 21761 invoked by uid 550); 28 Feb 2020 00:01:29 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 21641 invoked from network); 28 Feb 2020 00:01:28 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/uVx/XBN5lZTmlb4Lyfrp23GlYCSyVwKAYoD5ot60Wo=; b=KzW3r8GfkXZ1bxbOEhc8YIPZ2dejZnQCX6q/qZZVFR10jC5HNMQ2rn8d9jZcCf9BVI 2R1tJWstcf9J0wMi0wHy+fL7hI7CG5ueTsfS0gqeu2FK7W7+cI85U1MQNxuleljfBUMU q4LnILWGp4BeehZnd/ns8zcKBGPdUUms3t65g= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/uVx/XBN5lZTmlb4Lyfrp23GlYCSyVwKAYoD5ot60Wo=; b=kltceMvpBfuppTgCQXXNnBDNxPNOszPLnZn1UWloUz2Sz1u2VwsVPaKaL55vQukDsQ WgKednRs/2o82AhcpjS2acF5thr8QggJA45PYL3FL8dgF/PCA48etiOK6OJmj8UdeCm2 PHeX37a3YM2kd9P7sBiV+p1Q8JyJ3NVn99heph51P49pfAoDLTH5zEq4SUlaFPTEvtn/ 9sPS4xoodDFB0HWRYMOQapPSFsDimuCxI7gI+wNxbxhZ0ynfdmUsGAIcD6gWdw5ZTVHs 5Vk92mWPGku1l9aP1rtlhyc1Yyc29jNABQ7xVgsAbCNB4OCmYN9YS/kOyaw+UEH2bGyW 55fw== X-Gm-Message-State: APjAAAWbJ3dpSGKko3mXPSnjLAIQae/fugrO98gPNtrZtHhTqMAWPhKq K3rVKJL9VSMhfU08em23DRZUITqHgnc= X-Google-Smtp-Source: APXvYqx9fo57gEwEfmT4gKHzsyLQFuNMK/TLuK7SY+45wAhvz5aMLc0Yuj+V+h/0aFrVtq1ep7M1QQ== X-Received: by 2002:a63:c10d:: with SMTP id w13mr1769506pgf.312.1582848076871; Thu, 27 Feb 2020 16:01:16 -0800 (PST) From: Thomas Garnier To: kernel-hardening@lists.openwall.com Cc: kristen@linux.intel.com, keescook@chromium.org, Thomas Garnier , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , "H. Peter Anvin" , x86@kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v11 04/11] x86/entry/64: Adapt assembly for PIE support Date: Thu, 27 Feb 2020 16:00:49 -0800 Message-Id: <20200228000105.165012-5-thgarnie@chromium.org> X-Mailer: git-send-email 2.25.1.481.gfbce0eb801-goog In-Reply-To: <20200228000105.165012-1-thgarnie@chromium.org> References: <20200228000105.165012-1-thgarnie@chromium.org> MIME-Version: 1.0 Change the assembly code to use only relative references of symbols for the kernel to be PIE compatible. Signed-off-by: Thomas Garnier Reviewed-by: Kees Cook --- arch/x86/entry/entry_64.S | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index f2bb91e87877..2c8200d35797 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -1329,7 +1329,8 @@ SYM_CODE_START_LOCAL(error_entry) movl %ecx, %eax /* zero extend */ cmpq %rax, RIP+8(%rsp) je .Lbstep_iret - cmpq $.Lgs_change, RIP+8(%rsp) + leaq .Lgs_change(%rip), %rcx + cmpq %rcx, RIP+8(%rsp) jne .Lerror_entry_done_lfence /* @@ -1529,10 +1530,10 @@ SYM_CODE_START(nmi) * resume the outer NMI. */ - movq $repeat_nmi, %rdx + leaq repeat_nmi(%rip), %rdx cmpq 8(%rsp), %rdx ja 1f - movq $end_repeat_nmi, %rdx + leaq end_repeat_nmi(%rip), %rdx cmpq 8(%rsp), %rdx ja nested_nmi_out 1: @@ -1586,7 +1587,8 @@ nested_nmi: pushq %rdx pushfq pushq $__KERNEL_CS - pushq $repeat_nmi + leaq repeat_nmi(%rip), %rdx + pushq %rdx /* Put stack back */ addq $(6*8), %rsp @@ -1625,7 +1627,11 @@ first_nmi: addq $8, (%rsp) /* Fix up RSP */ pushfq /* RFLAGS */ pushq $__KERNEL_CS /* CS */ - pushq $1f /* RIP */ + pushq $0 /* Space for RIP */ + pushq %rdx /* Save RDX */ + leaq 1f(%rip), %rdx /* Put the address of 1f label into RDX */ + movq %rdx, 8(%rsp) /* Store it in RIP field */ + popq %rdx /* Restore RDX */ iretq /* continues at repeat_nmi below */ UNWIND_HINT_IRET_REGS 1: From patchwork Fri Feb 28 00:00:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Garnier X-Patchwork-Id: 11411173 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A6925138D for ; Fri, 28 Feb 2020 00:02:08 +0000 (UTC) Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.kernel.org (Postfix) with SMTP id 0F0AB2087F for ; Fri, 28 Feb 2020 00:02:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="GXN2TwlT" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0F0AB2087F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kernel-hardening-return-17996-patchwork-kernel-hardening=patchwork.kernel.org@lists.openwall.com Received: (qmail 21973 invoked by uid 550); 28 Feb 2020 00:01:32 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 21810 invoked from network); 28 Feb 2020 00:01:31 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=UhZ7upFOZbJwSu3t6sj7kyMuepSiW+0wthtVyYvfUW8=; b=GXN2TwlTYNwiK+ybctcNCG0N1o4mmEfgJ3tnGih4vm5ywp9y4TrkN7Ufd9T3240JQy E16DACTSaDDNdfolSmZkf2EnIwxAkoDtU4kGsVdvLeQnFGFncVNe6tE644aMszLX9q3B X299mdovMzxD4NedWt/1mwa1YRvXp1azy1t/0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=UhZ7upFOZbJwSu3t6sj7kyMuepSiW+0wthtVyYvfUW8=; b=VH8HohUstlQFYYgxRHgGG3Ggjb6kC9RwChWLRtwmGBRruG5pMVAh0opZ82Qqjofpfl l/XRY4LwIQgjV4MAKdl1Dn2Ug9xb/v//k9SZFuLLG10XaKYX3NuKaLVpQXDuPFjgb+X1 uHMopr3/pWiJ3yX6xHu2Mt8yzDe9rqXozYVwOSnU0BXtPeDNSRiQg9DJxiqiTV2luLFx PLGOyuVThiHuQ7WVxaDftim/3bEAHq8Zq01NFOfM0/Vi2l8HnwczJNbJHYIt/bRl4AM3 QR4+M5c3f5SWh9tjrh2T4+/8f2ZstlG4208A4goc4iTC/zGCRUYCSOylTHDwJgKTXBzq CRyg== X-Gm-Message-State: APjAAAWxWKZLgDYdN+fjWbyRXkNTohtXOTfptxwZA4SyAQoJl/w5gIra O4+RqhAYZ0DsEaW29UU/LiqY0Q0/bPQ= X-Google-Smtp-Source: APXvYqxiJCqNV8OEfXmb1fBc5jiE7n4/3DOvj4v7SG/Rln8ytNDHgILInJ8gkTkQ+TEqyjFoPLN0cw== X-Received: by 2002:a17:90a:d80b:: with SMTP id a11mr1588095pjv.142.1582848079067; Thu, 27 Feb 2020 16:01:19 -0800 (PST) From: Thomas Garnier To: kernel-hardening@lists.openwall.com Cc: kristen@linux.intel.com, keescook@chromium.org, Thomas Garnier , Thomas Gleixner , Ingo Molnar , Borislav Petkov , "H. Peter Anvin" , x86@kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v11 05/11] x86: pm-trace - Adapt assembly for PIE support Date: Thu, 27 Feb 2020 16:00:50 -0800 Message-Id: <20200228000105.165012-6-thgarnie@chromium.org> X-Mailer: git-send-email 2.25.1.481.gfbce0eb801-goog In-Reply-To: <20200228000105.165012-1-thgarnie@chromium.org> References: <20200228000105.165012-1-thgarnie@chromium.org> MIME-Version: 1.0 Change assembly to use the new _ASM_MOVABS macro instead of _ASM_MOV for the assembly to be PIE compatible. Signed-off-by: Thomas Garnier Reviewed-by: Kees Cook --- arch/x86/include/asm/pm-trace.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/include/asm/pm-trace.h b/arch/x86/include/asm/pm-trace.h index bfa32aa428e5..972070806ce9 100644 --- a/arch/x86/include/asm/pm-trace.h +++ b/arch/x86/include/asm/pm-trace.h @@ -8,7 +8,7 @@ do { \ if (pm_trace_enabled) { \ const void *tracedata; \ - asm volatile(_ASM_MOV " $1f,%0\n" \ + asm volatile(_ASM_MOVABS " $1f,%0\n" \ ".section .tracedata,\"a\"\n" \ "1:\t.word %c1\n\t" \ _ASM_PTR " %c2\n" \ From patchwork Fri Feb 28 00:00:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Garnier X-Patchwork-Id: 11411175 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1AD74138D for ; Fri, 28 Feb 2020 00:02:17 +0000 (UTC) Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.kernel.org (Postfix) with SMTP id 7870C2087F for ; Fri, 28 Feb 2020 00:02:16 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="MiPZlyhh" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7870C2087F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kernel-hardening-return-17997-patchwork-kernel-hardening=patchwork.kernel.org@lists.openwall.com Received: (qmail 22131 invoked by uid 550); 28 Feb 2020 00:01:34 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 22030 invoked from network); 28 Feb 2020 00:01:33 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=HVEfaBtq4P7YQEKJlB0rNIhGuSMdX7ZyWHWzM+iNj0k=; b=MiPZlyhhgfM7UPgKZ/frWvRzNlBB5613UuXaClOb+/1qj4My0rxjR2m0/Q68Aqjer2 gZYS9YdvWjfD32foSE/T14ZSJwLXcxaxD5TFllT8NBx0vpNpvH9gXUcwAi9LBcMVYSoD GAqgi2ibQpd1tNuqjY7zAzuM2NwWtDPz6cTyI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=HVEfaBtq4P7YQEKJlB0rNIhGuSMdX7ZyWHWzM+iNj0k=; b=jZFi40MPj5wqGEYHaLQa73LXAGvnnRp4bPxJCmQHPKApMYjy5CIZ/Yb4MXgY1ybXMY 7cxCotpU2ntEpZ2wZ0XtRTr98Z3mf1I02WXpPaZEp0nZQU2vBsF+6xndYcBvEREd0v7E VF7eTOdY9Ye/ZJtQlr6e2dgCO7r65mTQiYk8BhC2TDJM9ioE5yBO1Sgn8U5tc8jt9ZDx ASPJow+nHInybm6aMlYHTn8r0Fwr5SzQQ287skNW/4HltRqZqhKN8MEKtkSFykFcPKa4 WEO0KgAeNhIc7Cy3oHtX6niXkOUfgJJDJ3xFgb1PegmdZiXB+ny2tQ/1YqRKHBGSyn8X Wmsw== X-Gm-Message-State: APjAAAW6W3ciUiNT4mg4kJr30d/bPHk/qEyEVqftaoQFzbgzERxAoU/e 6gn8CdAb1X5CmrM80J6zkWV/ngqnPXI= X-Google-Smtp-Source: APXvYqymY4xDGLBpKhllQYQtdvp8eIxEJSyOvYMBEHU7xpDpUfih++1rH9KsvwXHUVBag4LFr8Qb4Q== X-Received: by 2002:a17:90a:7784:: with SMTP id v4mr1614784pjk.134.1582848081160; Thu, 27 Feb 2020 16:01:21 -0800 (PST) From: Thomas Garnier To: kernel-hardening@lists.openwall.com Cc: kristen@linux.intel.com, keescook@chromium.org, Thomas Garnier , Thomas Gleixner , Ingo Molnar , Borislav Petkov , "H. Peter Anvin" , x86@kernel.org, Andy Lutomirski , "Peter Zijlstra (Intel)" , Len Brown , linux-kernel@vger.kernel.org Subject: [PATCH v11 06/11] x86/CPU: Adapt assembly for PIE support Date: Thu, 27 Feb 2020 16:00:51 -0800 Message-Id: <20200228000105.165012-7-thgarnie@chromium.org> X-Mailer: git-send-email 2.25.1.481.gfbce0eb801-goog In-Reply-To: <20200228000105.165012-1-thgarnie@chromium.org> References: <20200228000105.165012-1-thgarnie@chromium.org> MIME-Version: 1.0 Change the assembly code to use only relative references of symbols for the kernel to be PIE compatible. Signed-off-by: Thomas Garnier Reviewed-by: Kees Cook --- arch/x86/include/asm/processor.h | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h index 09705ccc393c..fdf6366c482d 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -746,11 +746,13 @@ static inline void sync_core(void) "pushfq\n\t" "mov %%cs, %0\n\t" "pushq %q0\n\t" - "pushq $1f\n\t" + "leaq 1f(%%rip), %q0\n\t" + "pushq %q0\n\t" "iretq\n\t" UNWIND_HINT_RESTORE "1:" - : "=&r" (tmp), ASM_CALL_CONSTRAINT : : "cc", "memory"); + : "=&r" (tmp), ASM_CALL_CONSTRAINT + : : "cc", "memory"); #endif } From patchwork Fri Feb 28 00:00:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Garnier X-Patchwork-Id: 11411177 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 659AF1805 for ; Fri, 28 Feb 2020 00:02:26 +0000 (UTC) Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.kernel.org (Postfix) with SMTP id C437A246A6 for ; Fri, 28 Feb 2020 00:02:25 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="K56MuzoH" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C437A246A6 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kernel-hardening-return-17998-patchwork-kernel-hardening=patchwork.kernel.org@lists.openwall.com Received: (qmail 22245 invoked by uid 550); 28 Feb 2020 00:01:35 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 22160 invoked from network); 28 Feb 2020 00:01:34 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=J9YtfCHL5s52dZTQkxs/+RndppOjr/z9RSlKj3vVFK4=; b=K56MuzoHmJUhOwYAvoCivoJ3u+vvHwEXYBXxV/vaNqkwrMWvn6gZRUNGCDOMEWNWxq sZ+TLzbq8VZvxFT/X6aSYuuccKjRlkbPYenzAHHaZxxWuXBdnkU665sEaogmT61k4o7h JOsNBpxmKp7ISBaWRhouukTSaZ5dtB2hELDJg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=J9YtfCHL5s52dZTQkxs/+RndppOjr/z9RSlKj3vVFK4=; b=g9df0bbGdWCiHWUGGkmzbDr6p4FebLjv1qjiyF4G8CE49gGepdyzeagsojPvO+hXS7 nkuH3zIemRCclKLuIHjaFLOLADnYwrZXKMrbf79bE+Xpvymzmt33TIiBmsXuGu8GdOnr alzfH7UAu+hdzVw8+XCfnClFLS7L3/xp4goosDSspZlOWlJwGqoP7lYvL6YZbMqFRxxw 2wJGCw9MR9MHFrTIJr48T1GSQBlYMtO2u73DWigKklT9H6f5Sdif6csNVgx/5YgOTxg6 JQZ/lYNWRYLhJ+efCpqiDdUGhcbGMOARrSR1NbdnGZLzhGaj0McrdtZ6tOp+xP+euD6h ZwCw== X-Gm-Message-State: APjAAAUzOtxJXpp0H7nXh2b9RESt1dQ02rhtGf6MIfG+7ufxiRMtaHdH 0Q746/5T8LImL5zPqDtSbI5BY4IT9PQ= X-Google-Smtp-Source: APXvYqw3AgR5lreXOHV0mcPtons4OK9hEvjDW+aai6cRQfG4L95jHGQf/AsHHClO7ZoWLJCOWZ81ZQ== X-Received: by 2002:aa7:9891:: with SMTP id r17mr1587416pfl.205.1582848082152; Thu, 27 Feb 2020 16:01:22 -0800 (PST) From: Thomas Garnier To: kernel-hardening@lists.openwall.com Cc: kristen@linux.intel.com, keescook@chromium.org, Thomas Garnier , Pavel Machek , "Rafael J . Wysocki" , "Rafael J. Wysocki" , Len Brown , Thomas Gleixner , Ingo Molnar , Borislav Petkov , "H. Peter Anvin" , x86@kernel.org, linux-pm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v11 07/11] x86/acpi: Adapt assembly for PIE support Date: Thu, 27 Feb 2020 16:00:52 -0800 Message-Id: <20200228000105.165012-8-thgarnie@chromium.org> X-Mailer: git-send-email 2.25.1.481.gfbce0eb801-goog In-Reply-To: <20200228000105.165012-1-thgarnie@chromium.org> References: <20200228000105.165012-1-thgarnie@chromium.org> MIME-Version: 1.0 Change the assembly code to use only relative references of symbols for the kernel to be PIE compatible. Signed-off-by: Thomas Garnier Acked-by: Pavel Machek Acked-by: Rafael J. Wysocki Reviewed-by: Kees Cook --- arch/x86/kernel/acpi/wakeup_64.S | 31 ++++++++++++++++--------------- 1 file changed, 16 insertions(+), 15 deletions(-) diff --git a/arch/x86/kernel/acpi/wakeup_64.S b/arch/x86/kernel/acpi/wakeup_64.S index c8daa92f38dc..8e221285d9f1 100644 --- a/arch/x86/kernel/acpi/wakeup_64.S +++ b/arch/x86/kernel/acpi/wakeup_64.S @@ -15,7 +15,7 @@ * Hooray, we are in Long 64-bit mode (but still running in low memory) */ SYM_FUNC_START(wakeup_long64) - movq saved_magic, %rax + movq saved_magic(%rip), %rax movq $0x123456789abcdef0, %rdx cmpq %rdx, %rax je 2f @@ -31,14 +31,14 @@ SYM_FUNC_START(wakeup_long64) movw %ax, %es movw %ax, %fs movw %ax, %gs - movq saved_rsp, %rsp + movq saved_rsp(%rip), %rsp - movq saved_rbx, %rbx - movq saved_rdi, %rdi - movq saved_rsi, %rsi - movq saved_rbp, %rbp + movq saved_rbx(%rip), %rbx + movq saved_rdi(%rip), %rdi + movq saved_rsi(%rip), %rsi + movq saved_rbp(%rip), %rbp - movq saved_rip, %rax + movq saved_rip(%rip), %rax jmp *%rax SYM_FUNC_END(wakeup_long64) @@ -48,7 +48,7 @@ SYM_FUNC_START(do_suspend_lowlevel) xorl %eax, %eax call save_processor_state - movq $saved_context, %rax + leaq saved_context(%rip), %rax movq %rsp, pt_regs_sp(%rax) movq %rbp, pt_regs_bp(%rax) movq %rsi, pt_regs_si(%rax) @@ -67,13 +67,14 @@ SYM_FUNC_START(do_suspend_lowlevel) pushfq popq pt_regs_flags(%rax) - movq $.Lresume_point, saved_rip(%rip) + leaq .Lresume_point(%rip), %rax + movq %rax, saved_rip(%rip) - movq %rsp, saved_rsp - movq %rbp, saved_rbp - movq %rbx, saved_rbx - movq %rdi, saved_rdi - movq %rsi, saved_rsi + movq %rsp, saved_rsp(%rip) + movq %rbp, saved_rbp(%rip) + movq %rbx, saved_rbx(%rip) + movq %rdi, saved_rdi(%rip) + movq %rsi, saved_rsi(%rip) addq $8, %rsp movl $3, %edi @@ -85,7 +86,7 @@ SYM_FUNC_START(do_suspend_lowlevel) .align 4 .Lresume_point: /* We don't restore %rax, it must be 0 anyway */ - movq $saved_context, %rax + leaq saved_context(%rip), %rax movq saved_context_cr4(%rax), %rbx movq %rbx, %cr4 movq saved_context_cr3(%rax), %rbx From patchwork Fri Feb 28 00:00:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Garnier X-Patchwork-Id: 11411179 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0BDC7138D for ; Fri, 28 Feb 2020 00:02:37 +0000 (UTC) Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.kernel.org (Postfix) with SMTP id 67C1F2087F for ; Fri, 28 Feb 2020 00:02:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="cPSt7bG5" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 67C1F2087F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kernel-hardening-return-17999-patchwork-kernel-hardening=patchwork.kernel.org@lists.openwall.com Received: (qmail 22380 invoked by uid 550); 28 Feb 2020 00:01:37 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 22302 invoked from network); 28 Feb 2020 00:01:36 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=FFmbceD7QHP2G9Ca/GnJtf4RpKTt/GJIPcRUSUwSzaM=; b=cPSt7bG5f9am1KrbI/3idQAzXifp6xRIa4U46unWv7iH/cibsg+eZ8l1rB6LPI4A3J 8rCXT984f0GR9+WFpo9bKN6sz8MJaqhBVXKhhhDkItlzI97jFLioOsVRSWGgtH1SyOcq nZKW7rm5ajlWmnAvixODO9wGmSW1zN7cSoBKQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=FFmbceD7QHP2G9Ca/GnJtf4RpKTt/GJIPcRUSUwSzaM=; b=UGyyJHsKrFu5rA1GMm973xmZf1ZRb3y0U5GN8v7GbqfgQZMk6SPHxCQoYdywTuQ7Ys TGkWCT+9wdJFnTLLxXeTgxxd6Ld/+MRdF4HUnndzeMuPjKFLuo5kds38q/ihXySSogCO NU74cdIf9vwq/ga0Jpl2eDliLFTWzYxQl8RO3PsEp75VhOl8D73UN4GPy42EXnpUT9b+ RSOmyRDu/yRXs3WpkdMWLzck/46e1Ty+0kwydD1M+WmKBT01yV+TXa9s1xiJML5QQscc cNUjLEoA+2jQWZ8izjKSU2QkXmFPi6ldMMg4lHUvgyStTsmaZFmV5NYb5JbW4MDJtgDc 5hVg== X-Gm-Message-State: APjAAAVAjMlIVtchSOosRAMWCK6nBBjF3OomZ5l4Jc8UhC5yTV7OQSHy Zn/z1EppEpo4k94urJQmdSSTE5cvZPg= X-Google-Smtp-Source: APXvYqxzxq/RTsPVyBecAqne+dw68kSQHDmNiBFv21TtsNfqzjAmpUojokc1C1FJiTR3aVdLqW7w3Q== X-Received: by 2002:a17:902:708c:: with SMTP id z12mr1371193plk.0.1582848084250; Thu, 27 Feb 2020 16:01:24 -0800 (PST) From: Thomas Garnier To: kernel-hardening@lists.openwall.com Cc: kristen@linux.intel.com, keescook@chromium.org, Thomas Garnier , Thomas Gleixner , Ingo Molnar , Borislav Petkov , "H. Peter Anvin" , x86@kernel.org, Jiri Slaby , Peter Zijlstra , Cao jin , Josh Poimboeuf , linux-kernel@vger.kernel.org Subject: [PATCH v11 08/11] x86/boot/64: Adapt assembly for PIE support Date: Thu, 27 Feb 2020 16:00:53 -0800 Message-Id: <20200228000105.165012-9-thgarnie@chromium.org> X-Mailer: git-send-email 2.25.1.481.gfbce0eb801-goog In-Reply-To: <20200228000105.165012-1-thgarnie@chromium.org> References: <20200228000105.165012-1-thgarnie@chromium.org> MIME-Version: 1.0 Change the assembly code to use absolute reference for transition between address spaces and relative references when referencing global variables in the same address space. Ensure the kernel built with PIE references the correct addresses based on context. Signed-off-by: Thomas Garnier Reviewed-by: Kees Cook --- arch/x86/kernel/head_64.S | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index 4bbc770af632..40a467f8e116 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -87,7 +87,8 @@ SYM_CODE_START_NOALIGN(startup_64) popq %rsi /* Form the CR3 value being sure to include the CR3 modifier */ - addq $(early_top_pgt - __START_KERNEL_map), %rax + movabs $(early_top_pgt - __START_KERNEL_map), %rcx + addq %rcx, %rax jmp 1f SYM_CODE_END(startup_64) @@ -119,7 +120,8 @@ SYM_CODE_START(secondary_startup_64) popq %rsi /* Form the CR3 value being sure to include the CR3 modifier */ - addq $(init_top_pgt - __START_KERNEL_map), %rax + movabs $(init_top_pgt - __START_KERNEL_map), %rcx + addq %rcx, %rax 1: /* Enable PAE mode, PGE and LA57 */ @@ -137,7 +139,7 @@ SYM_CODE_START(secondary_startup_64) movq %rax, %cr3 /* Ensure I am executing from virtual addresses */ - movq $1f, %rax + movabs $1f, %rax ANNOTATE_RETPOLINE_SAFE jmp *%rax 1: @@ -234,11 +236,12 @@ SYM_CODE_START(secondary_startup_64) * REX.W + FF /5 JMP m16:64 Jump far, absolute indirect, * address given in m16:64. */ - pushq $.Lafter_lret # put return address on stack for unwinder + movabs $.Lafter_lret, %rax + pushq %rax # put return address on stack for unwinder xorl %ebp, %ebp # clear frame pointer - movq initial_code(%rip), %rax + leaq initial_code(%rip), %rax pushq $__KERNEL_CS # set correct cs - pushq %rax # target address in negative space + pushq (%rax) # target address in negative space lretq .Lafter_lret: SYM_CODE_END(secondary_startup_64) From patchwork Fri Feb 28 00:00:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Garnier X-Patchwork-Id: 11411181 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AFC1314BC for ; Fri, 28 Feb 2020 00:02:47 +0000 (UTC) Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.kernel.org (Postfix) with SMTP id 183532087F for ; Fri, 28 Feb 2020 00:02:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="ffRKFmzs" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 183532087F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kernel-hardening-return-18000-patchwork-kernel-hardening=patchwork.kernel.org@lists.openwall.com Received: (qmail 23565 invoked by uid 550); 28 Feb 2020 00:01:38 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 22435 invoked from network); 28 Feb 2020 00:01:37 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=kBSUTQnCJbQV5pLuilITF26DLiWa2/2BRxaonElBTNQ=; b=ffRKFmzsputibNvCgTe4Pi3ww1xoPkHdMAQDjclUSxqmZY4+B8q2nf8RNjriw9Jun8 QhjORrkShnXbbpqxKBG/5Su6vB1b54AxLYFiIWsapbzv3qhMKXOF8fcrqkG/eiTa+yHW WFQkVpmaUq0W/I2eBjsIlKnDUM+QUKxZzz2Ck= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kBSUTQnCJbQV5pLuilITF26DLiWa2/2BRxaonElBTNQ=; b=D0p2XeVlEvUE8yQ7FYUkbf5sfWZB0FknXLwUsFcv0keoybVDWhZg1OF3VrwIkiG9H+ PLajkPDH3R6YHavLVLfxDKR+AL46CsGG5ST/8hm1co1+BS3doyQyg9mNzoquC+daSxAO 2HTTOW33Z0iDpMh14xSkFUfAhEiqs2APH6j9xUTGCSqSHUvzuoHqk3LokB8DZpo6fDg2 A4XLncRRrGVc8HWdQZ5dBUzZ7fHS1g6u+nRJwdwQ/cGX38EGXJ52dkSCNqE6YnqYdtgS PPLsux/EOFE9fOCtIKV2XIxEfBz4b7Cv12pIyzCvieP9FdVgqiWrl19AL06icj2QJBBj U9aA== X-Gm-Message-State: APjAAAUz6p780LSM0BLHE9GD+8CwxfWsBSpoZbL4GooH09y7Y/n3/H6o kpGbsSuo7E01Ty0RbzUF6tZBXpOF8PA= X-Google-Smtp-Source: APXvYqzDwLydiJZoarH5wzoZjhL6Dn3ngsWTMAguCr0uuaAVQX0qWk1h8b0wS+NGFKCGYWdFG0VCCg== X-Received: by 2002:a17:90a:da01:: with SMTP id e1mr1626609pjv.100.1582848085161; Thu, 27 Feb 2020 16:01:25 -0800 (PST) From: Thomas Garnier To: kernel-hardening@lists.openwall.com Cc: kristen@linux.intel.com, keescook@chromium.org, Thomas Garnier , Pavel Machek , "Rafael J . Wysocki" , "Rafael J. Wysocki" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , "H. Peter Anvin" , x86@kernel.org, linux-pm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v11 09/11] x86/power/64: Adapt assembly for PIE support Date: Thu, 27 Feb 2020 16:00:54 -0800 Message-Id: <20200228000105.165012-10-thgarnie@chromium.org> X-Mailer: git-send-email 2.25.1.481.gfbce0eb801-goog In-Reply-To: <20200228000105.165012-1-thgarnie@chromium.org> References: <20200228000105.165012-1-thgarnie@chromium.org> MIME-Version: 1.0 Change the assembly code to use only relative references of symbols for the kernel to be PIE compatible. Signed-off-by: Thomas Garnier Acked-by: Pavel Machek Acked-by: Rafael J. Wysocki Reviewed-by: Kees Cook --- arch/x86/power/hibernate_asm_64.S | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/x86/power/hibernate_asm_64.S b/arch/x86/power/hibernate_asm_64.S index 7918b8415f13..977b8ae85045 100644 --- a/arch/x86/power/hibernate_asm_64.S +++ b/arch/x86/power/hibernate_asm_64.S @@ -23,7 +23,7 @@ #include SYM_FUNC_START(swsusp_arch_suspend) - movq $saved_context, %rax + leaq saved_context(%rip), %rax movq %rsp, pt_regs_sp(%rax) movq %rbp, pt_regs_bp(%rax) movq %rsi, pt_regs_si(%rax) @@ -116,7 +116,7 @@ SYM_FUNC_START(restore_registers) movq %rax, %cr4; # turn PGE back on /* We don't restore %rax, it must be 0 anyway */ - movq $saved_context, %rax + leaq saved_context(%rip), %rax movq pt_regs_sp(%rax), %rsp movq pt_regs_bp(%rax), %rbp movq pt_regs_si(%rax), %rsi From patchwork Fri Feb 28 00:00:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Garnier X-Patchwork-Id: 11411183 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9AFB5138D for ; Fri, 28 Feb 2020 00:02:57 +0000 (UTC) Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.kernel.org (Postfix) with SMTP id 023BE2087F for ; Fri, 28 Feb 2020 00:02:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="oEStjVPt" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 023BE2087F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kernel-hardening-return-18001-patchwork-kernel-hardening=patchwork.kernel.org@lists.openwall.com Received: (qmail 23644 invoked by uid 550); 28 Feb 2020 00:01:39 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 22466 invoked from network); 28 Feb 2020 00:01:38 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=H9FWlvjeU2XqZhHoJiWspMH4kFGj3K9GjGYd2WRMYmQ=; b=oEStjVPt0S31nPacabrXXaqaOo12Uk+r85uXc9Xl7Tu1lXk9HmiONOEKMda7M1yCLl DfI6JExzzhDdrzGWYDMCWToyRPzvherlO6BerSNdsLYxiTTSYaVvNUHXPBhmY/g7c40t 6aZqMcdDUuTRkqEUDTm4BWTIy0OXUsHY+h8Gc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=H9FWlvjeU2XqZhHoJiWspMH4kFGj3K9GjGYd2WRMYmQ=; b=T8c4I87pGhrLEEnGvcRwEyBjwajMl/t7WwEZeDDm09zWp+mU57TFnS9onAhKXyZisU fDKkD9r+O/3dVPTipRvjqWTQLkW3b/rRQkYAGRqwNrhpiJzWfGn0DjWC86uKTyJZWt9M opL88BrBtggEDSitVf9+FuyXqg3zxBtT2bsPo8YrUiN8GQn3o9C6etsigSWHcU9Ish3w d2hpTDeJQFI1zlrR/EQVffOFI/KoYLPfN1TcaGP4z6P5r/5VPM7fSSaeUVC2QCZHh4al I2usBD9vB57mFMM37+KAmAunMSm0bqPwfMNlEkAAJ0bE3mbXplGIDMf7UG/JsG8i7SyV GZ2Q== X-Gm-Message-State: APjAAAWmD+DRtgpaoyen3JLYL22uJmQwjKCVNeEJjH1zLJmzc0odFx0Q A6RbOL5uAzYW33uP9kqhAC1OS7eDHXw= X-Google-Smtp-Source: APXvYqzm3a/ZFJ1l5Nvdz79Gm4AfpVbhjeL86Hhy3RQsCyWjAO+rF4ZDCotN8wJqbSnZ9y96RTh5JA== X-Received: by 2002:a63:df02:: with SMTP id u2mr1689153pgg.403.1582848086381; Thu, 27 Feb 2020 16:01:26 -0800 (PST) From: Thomas Garnier To: kernel-hardening@lists.openwall.com Cc: kristen@linux.intel.com, keescook@chromium.org, Thomas Garnier , Juergen Gross , Thomas Hellstrom , "VMware, Inc." , Thomas Gleixner , Ingo Molnar , Borislav Petkov , "H. Peter Anvin" , x86@kernel.org, virtualization@lists.linux-foundation.org, linux-kernel@vger.kernel.org Subject: [PATCH v11 10/11] x86/paravirt: Adapt assembly for PIE support Date: Thu, 27 Feb 2020 16:00:55 -0800 Message-Id: <20200228000105.165012-11-thgarnie@chromium.org> X-Mailer: git-send-email 2.25.1.481.gfbce0eb801-goog In-Reply-To: <20200228000105.165012-1-thgarnie@chromium.org> References: <20200228000105.165012-1-thgarnie@chromium.org> MIME-Version: 1.0 If PIE is enabled, switch the paravirt assembly constraints to be compatible. The %c/i constrains generate smaller code so is kept by default. Signed-off-by: Thomas Garnier Acked-by: Juergen Gross --- arch/x86/include/asm/paravirt_types.h | 32 +++++++++++++++++++++++---- 1 file changed, 28 insertions(+), 4 deletions(-) diff --git a/arch/x86/include/asm/paravirt_types.h b/arch/x86/include/asm/paravirt_types.h index 84812964d3dd..82f7ca22e0ae 100644 --- a/arch/x86/include/asm/paravirt_types.h +++ b/arch/x86/include/asm/paravirt_types.h @@ -336,9 +336,32 @@ extern struct paravirt_patch_template pv_ops; #define PARAVIRT_PATCH(x) \ (offsetof(struct paravirt_patch_template, x) / sizeof(void *)) +#ifdef CONFIG_X86_PIE +#define paravirt_opptr_call "a" +#define paravirt_opptr_type "p" + +/* + * Alternative patching requires a maximum of 7 bytes but the relative call is + * only 6 bytes. If PIE is enabled, add an additional nop to the call + * instruction to ensure patching is possible. + * + * Without PIE, the call is reg/mem64: + * ff 14 25 68 37 02 82 callq *0xffffffff82023768 + * + * With PIE, it is relative to %rip and take 1-less byte: + * ff 15 fa d9 ff 00 callq *0xffd9fa(%rip) # + * + */ +#define PARAVIRT_CALL_POST "nop;" +#else +#define paravirt_opptr_call "c" +#define paravirt_opptr_type "i" +#define PARAVIRT_CALL_POST "" +#endif + #define paravirt_type(op) \ [paravirt_typenum] "i" (PARAVIRT_PATCH(op)), \ - [paravirt_opptr] "i" (&(pv_ops.op)) + [paravirt_opptr] paravirt_opptr_type (&(pv_ops.op)) #define paravirt_clobber(clobber) \ [paravirt_clobber] "i" (clobber) @@ -377,9 +400,10 @@ int paravirt_disable_iospace(void); * offset into the paravirt_patch_template structure, and can therefore be * freely converted back into a structure offset. */ -#define PARAVIRT_CALL \ - ANNOTATE_RETPOLINE_SAFE \ - "call *%c[paravirt_opptr];" +#define PARAVIRT_CALL \ + ANNOTATE_RETPOLINE_SAFE \ + "call *%" paravirt_opptr_call "[paravirt_opptr];" \ + PARAVIRT_CALL_POST /* * These macros are intended to wrap calls through one of the paravirt From patchwork Fri Feb 28 00:00:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Garnier X-Patchwork-Id: 11411185 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 15591138D for ; Fri, 28 Feb 2020 00:03:08 +0000 (UTC) Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.kernel.org (Postfix) with SMTP id 6FBCD2087F for ; Fri, 28 Feb 2020 00:03:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="Ej0qYQRk" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6FBCD2087F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kernel-hardening-return-18002-patchwork-kernel-hardening=patchwork.kernel.org@lists.openwall.com Received: (qmail 23878 invoked by uid 550); 28 Feb 2020 00:01:41 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 23764 invoked from network); 28 Feb 2020 00:01:40 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Oz6De1dIYEgjxhRabTRk4CB78qRzDNYlR7Nzf0r7zm8=; b=Ej0qYQRkax6vt9VbEqKCxSCRfVTpbF6R6yfkIQH8hH+HCk4aampkSCQAJSBx/l3Eul QYdGcYFrpkp7QbTQ9dqQZFDOh3Tk90FzGpRVXkXIXRjJeG0HLpM37JMU0ecBnf624Z4b p9gSB349m5UGPUQTt7tNnLStSo1C536+EubDM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Oz6De1dIYEgjxhRabTRk4CB78qRzDNYlR7Nzf0r7zm8=; b=IRhCAKXHWbZzJ/SnEBZQADMkTosk7fjlyxtsUObXkQORK8jrmRp87kBEo4TBoWy5R9 3nJMPu1sEzmS4W3JY/9zLa9jlVhOnfe0Exa4MlJxeUqyBPiQLFjjBxbCF+3LkDaycQzl LwAHQEFrQmhlA2kOmRDHMDoIFnvWq4iYUdghCGGoHxIFVS9FoDvxhQ3kp+wFStldnhZ/ g5Abps1deIPWY1H7FY3rBS4yGtXkPVKbg2ww9iuRVVosKr9wp6gpqbil7v37cRlyjVAN 40tUMSyKFaeW0Lbnt+VpqVyC03UDrI5XKqSgjzUOZDdsTce7ehzR6B/Gl2ngsVr+Kckm sbMA== X-Gm-Message-State: APjAAAVUVXMKm45L2kpOHf/Jfr6dpOMgNgNIsm/bsRnJ61EATULHSlde NmdFDgPmZDlKdNcMpbAMrKR8MyZ1pyA= X-Google-Smtp-Source: APXvYqyeGsdTWhXYKAUlxHWjZxN7BX3EeYNWjLkfclN9Kp12cCBJnK2qIKd/H/GKqRV9XMIxr86Wng== X-Received: by 2002:a17:90a:868b:: with SMTP id p11mr1609353pjn.60.1582848088646; Thu, 27 Feb 2020 16:01:28 -0800 (PST) From: Thomas Garnier To: kernel-hardening@lists.openwall.com Cc: kristen@linux.intel.com, keescook@chromium.org, Thomas Garnier , Thomas Gleixner , Ingo Molnar , Borislav Petkov , "H. Peter Anvin" , x86@kernel.org, Miguel Ojeda , Rasmus Villemoes , Peter Zijlstra , linux-kernel@vger.kernel.org Subject: [PATCH v11 11/11] x86/alternatives: Adapt assembly for PIE support Date: Thu, 27 Feb 2020 16:00:56 -0800 Message-Id: <20200228000105.165012-12-thgarnie@chromium.org> X-Mailer: git-send-email 2.25.1.481.gfbce0eb801-goog In-Reply-To: <20200228000105.165012-1-thgarnie@chromium.org> References: <20200228000105.165012-1-thgarnie@chromium.org> MIME-Version: 1.0 Change the assembly options to work with pointers instead of integers. The generated code is the same PIE just ensures input is a pointer. Signed-off-by: Thomas Garnier Reviewed-by: Kees Cook --- arch/x86/include/asm/alternative.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/arch/x86/include/asm/alternative.h b/arch/x86/include/asm/alternative.h index 13adca37c99a..43a148042656 100644 --- a/arch/x86/include/asm/alternative.h +++ b/arch/x86/include/asm/alternative.h @@ -243,7 +243,7 @@ static inline int alternatives_text_reserved(void *start, void *end) /* Like alternative_io, but for replacing a direct call with another one. */ #define alternative_call(oldfunc, newfunc, feature, output, input...) \ asm_inline volatile (ALTERNATIVE("call %P[old]", "call %P[new]", feature) \ - : output : [old] "i" (oldfunc), [new] "i" (newfunc), ## input) + : output : [old] "X" (oldfunc), [new] "X" (newfunc), ## input) /* * Like alternative_call, but there are two features and respective functions. @@ -256,8 +256,8 @@ static inline int alternatives_text_reserved(void *start, void *end) asm_inline volatile (ALTERNATIVE_2("call %P[old]", "call %P[new1]", feature1,\ "call %P[new2]", feature2) \ : output, ASM_CALL_CONSTRAINT \ - : [old] "i" (oldfunc), [new1] "i" (newfunc1), \ - [new2] "i" (newfunc2), ## input) + : [old] "X" (oldfunc), [new1] "X" (newfunc1), \ + [new2] "X" (newfunc2), ## input) /* * use this macro(s) if you need more than one output parameter