From patchwork Wed Oct 13 15:22:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12556185 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C58AC433EF for ; Wed, 13 Oct 2021 15:22:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5E08760F21 for ; Wed, 13 Oct 2021 15:22:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229514AbhJMPZA (ORCPT ); Wed, 13 Oct 2021 11:25:00 -0400 Received: from mail.kernel.org ([198.145.29.99]:38312 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229777AbhJMPY7 (ORCPT ); Wed, 13 Oct 2021 11:24:59 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id EA3026113D; Wed, 13 Oct 2021 15:22:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1634138576; bh=flUoePz/ELcAVPcE4t/3y39R7GhsHD4BKh/RcKQ0z/A=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hu+vgOE5Q2S9stUfHBDPDugz9oB89WK6gH/XriFHis7aAX7tSXsemgZIWMoyiQqO2 t1QBMgGqJUBe0iF/3ocgjAEUklT6vBbbcoKKN2+QgEo9qMfLWfYIFscRnZ6XdMMpnI WJfetasZ1V1G/B7Xh+Htus9qWG+UhsARVYXNMlv5oAsxZvUnpUu2yCFMA2yclHzM/r E8HZLcY+mDCwP2LUJYeITP645O+zaIpv8XdTeRf3QC/PSzSRF6G5YSof63yvOc52N5 bYm3q5qT0TewizUH+C0FsgWJjbFAQiQHWhGoTrr9LzYdValf0juXFRraBv+fbdUnLb HJ2iIjvZd09vg== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-hardening@vger.kernel.org, mark.rutland@arm.com, catalin.marinas@arm.com, will@kernel.org, Ard Biesheuvel Subject: [RFC PATCH 1/9] arm64: assembler: enable PAC for non-leaf assembler routines Date: Wed, 13 Oct 2021 17:22:35 +0200 Message-Id: <20211013152243.2216899-2-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211013152243.2216899-1-ardb@kernel.org> References: <20211013152243.2216899-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1085; h=from:subject; bh=flUoePz/ELcAVPcE4t/3y39R7GhsHD4BKh/RcKQ0z/A=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBhZvmyx1T9ZjfUmm1pG7KbdLlq1l1jOmPiOC7NNXoR hU1cfDCJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYWb5sgAKCRDDTyI5ktmPJL70DA DGmzNDA3P176nmDyPRGr1hOmSglwN3TvUuOkL/jZIMFs32EjsnNYmO+iShq/g7MCXzc7Hd0QLHr8wy c2WUEEzxpXDr7aYnNw89G5S1s36fOGm8Ys624CxdenYJvD88FO2nqWu9mX+x+5d+5dn2g8IK4Ft5Kb Wru1SnRVcVJdBapduHdYTelK5hvJFr2soctm2EZZB28YPsmUhVJ1LhftuTf9S3thqhtLsKCefJfx6u FnIGGVdomcqPVqKRjbLXlZ2v+UGt3wppn96v//u4oqlAAqfRpWO116lE0NZXdKrIw2igOqBCbWq8Zn J83jIi2eAZukiT1qwaD6OCMeHZ4ZxyHZ6R/R5iBZkNvtZAE2mVE9vDFrQB1knnLxzpK1Q1yhFgbosY ofniB1oyZ0tp7kwjoQjmhCOW65mvithKKIVxhrbrLEaEU1oHqS6bO6vbVR11rMMDrjip8wP5nskcAZ fR0DpBDYKSL5RIgK7rTT63eC5GqR9ZLeH1OEZyV97yst8= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-hardening@vger.kernel.org Enable pointer signing and authentication when preserving and restoring the linker register to/from the stack for assembler routines that use the frame_push and frame_pop macros to set up their stack frames. This protects the return address from inadvertent modification while stored in memory. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/assembler.h | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 89faca0e740d..ceed84ac4005 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -665,6 +665,9 @@ alternative_endif * for locals. */ .macro frame_push, regcount:req, extra +#ifdef CONFIG_ARM64_PTR_AUTH_KERNEL + paciasp +#endif __frame st, \regcount, \extra .endm @@ -676,6 +679,9 @@ alternative_endif */ .macro frame_pop __frame ld +#ifdef CONFIG_ARM64_PTR_AUTH_KERNEL + autiasp +#endif .endm .macro __frame_regs, reg1, reg2, op, num From patchwork Wed Oct 13 15:22:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12556187 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 50077C433EF for ; Wed, 13 Oct 2021 15:22:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3631360F21 for ; Wed, 13 Oct 2021 15:22:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231938AbhJMPZB (ORCPT ); Wed, 13 Oct 2021 11:25:01 -0400 Received: from mail.kernel.org ([198.145.29.99]:38336 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229851AbhJMPZB (ORCPT ); Wed, 13 Oct 2021 11:25:01 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id D0B8B61056; Wed, 13 Oct 2021 15:22:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1634138578; bh=R2UYEqSTRtChtC4QgeRilpzF9vEGT72cMEAtMxwkrdw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=SD49Vd42WXZLw/PvW6bu69SQXt++KH5NdhwciCvqJ1dpXyvizRJTIkdAuvSQiH020 5E0zj4MOmboXANs3PPglBPm20qZEEUeuMxTw2Ko0onsdZpAcpb0YPCB+Sv/sv5qGLO PocfF7cIBqpkmqMFVOL3o+Ym1h+d+UkjZHOblUqpuiOyRjzLC0XR44CAQysctrD3LX 8j8jvhur0NqYHhu0yFtVn1OuqDTmnGQ9nt+mmu5HqME6YVQccj8dWhVxw4LV5hpiAK HOpcnhoJ0y6pnaeguzs+I9/JXBcaf/ad6jT1aNZ4vpJoWnm3yjszWNaVYtYHlud0eF MYebslUbSc/bg== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-hardening@vger.kernel.org, mark.rutland@arm.com, catalin.marinas@arm.com, will@kernel.org, Ard Biesheuvel Subject: [RFC PATCH 2/9] arm64: cache: use ALIAS version of linkage macros for local aliases Date: Wed, 13 Oct 2021 17:22:36 +0200 Message-Id: <20211013152243.2216899-3-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211013152243.2216899-1-ardb@kernel.org> References: <20211013152243.2216899-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1738; h=from:subject; bh=R2UYEqSTRtChtC4QgeRilpzF9vEGT72cMEAtMxwkrdw=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBhZvm0p29nyIDFC/OgnXq/llxD0h6xSy4cOgfF2mK/ WgyVZWKJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYWb5tAAKCRDDTyI5ktmPJOX1C/ 9Rx/vpSt0ularwZlcpCvbRefn/JQU/xWgTbX5/unHHupZnDI26dncpgWVHQHc94ktpdHJsbTktWvUT 9eDQ/rX+C6YEwwl/+uev6DM4qWjmGdmO0C3/BIL68ntMK4HS8YG/FkBedbwENxjM8dvfEppqHb0dTZ 8RzlNHRY/TwJoBuibcnZIgI7+fYsNVlv1igojMonHn8XJ0p5U4SIbaLDIU8ozTKpBsMHi1rW59afci ULmRmlo+nciO9W4oqVK2BvGoKZSKdt5f533flUd0X+GmIJ6p0BEH/Ro1qrkEGfwVK3UMEAvPktbzQO EMlAzRQaeXeoKXdp+RlLcv6qkClfWBsF22/Lh/MWCp3C7vegrVqi671TgcNLZkUKp3a3oQaSKlu2Ys Ga/egoW2MDQQHt1uco23mAFsr+OYrBtlZ9o6W37pQYU8sDUdvSWH/7+XpoadrZfMK8aYhDWssM62lf Ma3r880sBACKvnaW+nr/fSFXIm1zokM3XYQaZNSIozDFE= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-hardening@vger.kernel.org Upcoming changes to the linkage macros will no longer tolerate duplicate start and end symbols for functions unless they are annotated as aliases. This is needed to avoid emitting mismatched .cfi start/end directives. So update a couple of occurrences in cache.S where a local alias is incorrectly declared as a proper local symbol. Signed-off-by: Ard Biesheuvel --- arch/arm64/mm/cache.S | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/arm64/mm/cache.S b/arch/arm64/mm/cache.S index 5051b3c1a4f1..681a89921992 100644 --- a/arch/arm64/mm/cache.S +++ b/arch/arm64/mm/cache.S @@ -140,7 +140,7 @@ SYM_FUNC_END(dcache_clean_pou) * - start - kernel start address of region * - end - kernel end address of region */ -SYM_FUNC_START_LOCAL(__dma_inv_area) +SYM_FUNC_START_LOCAL_ALIAS(__dma_inv_area) SYM_FUNC_START_PI(dcache_inval_poc) /* FALLTHROUGH */ @@ -167,7 +167,7 @@ SYM_FUNC_START_PI(dcache_inval_poc) dsb sy ret SYM_FUNC_END_PI(dcache_inval_poc) -SYM_FUNC_END(__dma_inv_area) +SYM_FUNC_END_ALIAS(__dma_inv_area) /* * dcache_clean_poc(start, end) @@ -178,7 +178,7 @@ SYM_FUNC_END(__dma_inv_area) * - start - virtual start address of region * - end - virtual end address of region */ -SYM_FUNC_START_LOCAL(__dma_clean_area) +SYM_FUNC_START_LOCAL_ALIAS(__dma_clean_area) SYM_FUNC_START_PI(dcache_clean_poc) /* FALLTHROUGH */ @@ -190,7 +190,7 @@ SYM_FUNC_START_PI(dcache_clean_poc) dcache_by_line_op cvac, sy, x0, x1, x2, x3 ret SYM_FUNC_END_PI(dcache_clean_poc) -SYM_FUNC_END(__dma_clean_area) +SYM_FUNC_END_ALIAS(__dma_clean_area) /* * dcache_clean_pop(start, end) From patchwork Wed Oct 13 15:22:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12556189 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 26AD7C433F5 for ; Wed, 13 Oct 2021 15:23:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 140B260FDA for ; Wed, 13 Oct 2021 15:23:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229851AbhJMPZD (ORCPT ); Wed, 13 Oct 2021 11:25:03 -0400 Received: from mail.kernel.org ([198.145.29.99]:38360 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229777AbhJMPZD (ORCPT ); Wed, 13 Oct 2021 11:25:03 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id B6FDE610E6; Wed, 13 Oct 2021 15:22:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1634138580; bh=5k0BYZS5Q/5ehvuCNbkjkrQtkcOaZ95KgwqdtymNMw8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rSRQFnixLFq34y1EmGz+pYyqzxOdxSI4yOx9AbRTj4cI4zLBmJ8TxtRaPcb7bOmGb Y5O1OgCZjvs9jIARptaA3X3uo6yVxOfFdu2qvgJNF0VvgVH3EcwKBO0GozomRepUjv r+DgAG6o/ZPLRKSNqm9yGQTNyknbNu4cEwZrID6hdpJUtp6CDsfjQn/PzZYFcdy5/9 nAonJzEJ86RxbsnR92LuO2YLPJHLtJeBfoWi4/cOU8A8bkXI0dG82V8KgBkwzOtGtJ l/H53p9iFLKb9YU+TqGuVCH9w2YHLXceaTnmxUJbCUeA1oqkPWBn+8bn9A1IIx4jfz blR5RPYRMZFGA== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-hardening@vger.kernel.org, mark.rutland@arm.com, catalin.marinas@arm.com, will@kernel.org, Ard Biesheuvel Subject: [RFC PATCH 3/9] arm64: crypto: avoid overlapping linkage definitions for AES-CBC Date: Wed, 13 Oct 2021 17:22:37 +0200 Message-Id: <20211013152243.2216899-4-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211013152243.2216899-1-ardb@kernel.org> References: <20211013152243.2216899-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1720; h=from:subject; bh=5k0BYZS5Q/5ehvuCNbkjkrQtkcOaZ95KgwqdtymNMw8=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBhZvm2ZdTKQVU79efbhhoiav8D8jmUODgZpp6seolz 7krY+66JAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYWb5tgAKCRDDTyI5ktmPJPCzC/ 9TsCXIseJHAZqVmbMAnkuIb2BaOu1gpH+jsUsy0F/6R4DS9LOLaM3mxG3DdO5C/FSkAqKWoNNIBeM7 ZnDp01QfO4/qYC4fwqjHTqGQDWyDvJzsdTcWfTpDSRhtY9JLoMsUdsLvT30YITusiEbF5XFbTitXI0 92baTTP8u1kLey0vAQeKf6BII9scGDQouFc417QAo9J7V5G+JOPlm4K9My6wU/y4/uzSIiOo9UPxIh w7OYHP47Lgte9pdS+OpXPQ6PzMUOzO2r39gliZbQZ6CdFzLq5ZuOq49YA99t7I+QFu4SO3EH5SmdUc ATAbhDNNhdiZM6XZJvlZoPM0uT7iupzy6koBXLoK9vJaP3ncdZ1k4mE0NDojqIhVGbiu3Cso+tR/9z FuKzsIvXIxQVet2++w0P18BCxL5oDLfz/1gAvrJg2jySIEfK/pO0qugsBQTU03QM6kcV5OyE4DfKza cPRzKRIOjZ6V6ZFRekTkKs0i54bRYl5uWjA9dYP4dQ6n8= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-hardening@vger.kernel.org The aes_essiv_cbc_[en|de]crypt routines perform a single AES block encryption of the IV before tail calling into the ordinary AES-CBC routines to perform the actual data en/decryption. In the asm code, the symbol definitions currently overlap, which is unnecessary, and becomes problematic once we enable generation of CFI unwind metadata. So instead, move the end marker of the ESSIV versions right after the respective tail calls. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-modes.S | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/arm64/crypto/aes-modes.S b/arch/arm64/crypto/aes-modes.S index b495de22bb38..50427301b4d8 100644 --- a/arch/arm64/crypto/aes-modes.S +++ b/arch/arm64/crypto/aes-modes.S @@ -134,6 +134,7 @@ AES_FUNC_START(aes_essiv_cbc_encrypt) encrypt_block v4, w8, x6, x7, w9 enc_switch_key w3, x2, x6 b .Lcbcencloop4x +AES_FUNC_END(aes_essiv_cbc_encrypt) AES_FUNC_START(aes_cbc_encrypt) ld1 {v4.16b}, [x5] /* get iv */ @@ -168,7 +169,6 @@ AES_FUNC_START(aes_cbc_encrypt) st1 {v4.16b}, [x5] /* return iv */ ret AES_FUNC_END(aes_cbc_encrypt) -AES_FUNC_END(aes_essiv_cbc_encrypt) AES_FUNC_START(aes_essiv_cbc_decrypt) stp x29, x30, [sp, #-16]! @@ -180,6 +180,7 @@ AES_FUNC_START(aes_essiv_cbc_decrypt) enc_prepare w8, x6, x7 encrypt_block cbciv, w8, x6, x7, w9 b .Lessivcbcdecstart +AES_FUNC_END(aes_essiv_cbc_decrypt) AES_FUNC_START(aes_cbc_decrypt) stp x29, x30, [sp, #-16]! @@ -239,7 +240,6 @@ ST5( st1 {v4.16b}, [x0], #16 ) ldp x29, x30, [sp], #16 ret AES_FUNC_END(aes_cbc_decrypt) -AES_FUNC_END(aes_essiv_cbc_decrypt) /* From patchwork Wed Oct 13 15:22:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12556191 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 077CDC433F5 for ; Wed, 13 Oct 2021 15:23:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DC5566112D for ; Wed, 13 Oct 2021 15:23:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232118AbhJMPZF (ORCPT ); Wed, 13 Oct 2021 11:25:05 -0400 Received: from mail.kernel.org ([198.145.29.99]:38386 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229777AbhJMPZF (ORCPT ); Wed, 13 Oct 2021 11:25:05 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 9DD656113D; Wed, 13 Oct 2021 15:23:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1634138582; bh=ev012ZKZHpmXzWbmKkQFRhIEIq9GzO5yF2hyvMrLTl0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=f8UemsnlXse4ampLH7gAriyekzoorTOM1PO9HbMjyEswigGSMiUJI5+z3ikjjV64i DO2yEC5mv0hgA++oRdtJTXQCEmaogDHt5fevUM8Hqo+oP/jLTjo5YQPtORAKShhkmW jyy/LiCfM2U7H+2gs+qzn4KYaHUr+7PQGQ1LnzwW9gXRAiCC1IPHjLH45asZQ5DAkl FdfYEU2c9tKw/Qxw7MAsNrxJPlz2llQZmPrCG+6//rCWUSaRtVPKIP3aZPEMc59jpG DVZv+tcBFbN8X5rrORMXRsoVSMjZSSgMzpmuX0k4ibg1qYQmaCpBCi4l5o2Y/Qc8c8 pPSXeJfWjOG0A== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-hardening@vger.kernel.org, mark.rutland@arm.com, catalin.marinas@arm.com, will@kernel.org, Ard Biesheuvel Subject: [RFC PATCH 4/9] arm64: aes-neonbs: move frame pop to end of function Date: Wed, 13 Oct 2021 17:22:38 +0200 Message-Id: <20211013152243.2216899-5-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211013152243.2216899-1-ardb@kernel.org> References: <20211013152243.2216899-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1127; h=from:subject; bh=ev012ZKZHpmXzWbmKkQFRhIEIq9GzO5yF2hyvMrLTl0=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBhZvm4C4JNezyScCorZ+vZKcwVprM4gy0BSJ+YUPs4 BCPLDBaJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYWb5uAAKCRDDTyI5ktmPJN4NC/ 96ZFbOTa56knGclCnX2tIghJgIz4TvVHbq07xyX4Tf7nuuUylkzWmUn6imw5NTYEZ9GFp8e8GbS7dV k+ajlZtN6j4GxQOVyZ9ZFsvt5IZChpsw3e1K4u6XKJ+3B46Tx//rld5oKZTfTnBFbq8Wfpam69gRaA jtikcQ1wpE1ZCwC2jHtZL00x7f7bSBt2OnObxlOGXa3VqWj4pzuVipdXYegMwksYGMufG8ykkAvccB dbpKR2hH7IYvbKpBk39/Aui0biEF2c7Q81kqEmDx+y/IZdIDUaI4V7OrfQM36MxgehmVda3EwwOUq7 yTcda7zpp75sOsKGZsRYZYT/esccDJZBYJewmlDr1tIu3/9Z8/GR9aIgqb++o+kQTQEjfH1FXKykhq aca7jN7oPLDKGs9xhrLlgvVbQdq7C9KInXLi4dM+ui9TYPPVBNnFTBF10z9r6SsGuoDJ024Hvt0yPl gcOYk8M+nr23X4lM229yjUh8rM3l+ZdM7CA1tSmg+Yn9c= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-hardening@vger.kernel.org In order to decomplicate the generation of CFI unwind metadata for the AES-CTR routine, which would involve preserving/restoring the virtual register set to convey that the state during the handling of inputs less than 8 blocks [which is emitted out of line] equals the state before the frame pop, let's just move it to the end of the function. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-neonbs-core.S | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/arm64/crypto/aes-neonbs-core.S b/arch/arm64/crypto/aes-neonbs-core.S index a3405b8c344b..7104b54448dc 100644 --- a/arch/arm64/crypto/aes-neonbs-core.S +++ b/arch/arm64/crypto/aes-neonbs-core.S @@ -966,10 +966,6 @@ CPU_LE( rev x8, x8 ) b 99b -.Lctr_done: - frame_pop - ret - /* * If we are handling the tail of the input (x6 != NULL), return the * final keystream block back to the caller. @@ -998,4 +994,8 @@ CPU_LE( rev x8, x8 ) 7: cbz x25, 8b st1 {v5.16b}, [x25] b 8b + +.Lctr_done: + frame_pop + ret SYM_FUNC_END(aesbs_ctr_encrypt) From patchwork Wed Oct 13 15:22:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12556193 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4ED44C433F5 for ; Wed, 13 Oct 2021 15:23:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2CEEF61056 for ; Wed, 13 Oct 2021 15:23:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231820AbhJMPZH (ORCPT ); Wed, 13 Oct 2021 11:25:07 -0400 Received: from mail.kernel.org ([198.145.29.99]:38410 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229777AbhJMPZH (ORCPT ); Wed, 13 Oct 2021 11:25:07 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 84D3160F21; Wed, 13 Oct 2021 15:23:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1634138584; bh=NWrNx/WCxNaDBTdOupJGno8eolqNRTIYbYc9Hvg5z4k=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dTEivvIzvfqp65l6/l1C413vgQi3ZAN2b7rqtfkpQw+0EypOQ2zcTN2KgsDZUhEmT fgZinPD7+CjyIUS7kLHF+VTdok/Z6cGhYSlA4dPeuxcGvV9egrWM2Wav6i9we7XMIB /a3cuuR2s3ZhLdtrrvBw43FWqKY2DE/OT9bo2Ys6Rlu+WTbtOnT4KNuD8Gu3xtxGlI 438zLMu8ILpD12JCdEN/YYrjCCC3004biQcyb5FuIj29bCrizwZTpwmyhG/6YRd79S GnYnBAN9QNhkzrKkc25mhbbsdmX12nThyf3pD4ycPX9W7HLcXCpEgmKQWOPcynJsV+ vV6y+Zg1or8BA== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-hardening@vger.kernel.org, mark.rutland@arm.com, catalin.marinas@arm.com, will@kernel.org, Ard Biesheuvel Subject: [RFC PATCH 5/9] arm64: chacha-neon: move frame pop forward Date: Wed, 13 Oct 2021 17:22:39 +0200 Message-Id: <20211013152243.2216899-6-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211013152243.2216899-1-ardb@kernel.org> References: <20211013152243.2216899-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1894; h=from:subject; bh=NWrNx/WCxNaDBTdOupJGno8eolqNRTIYbYc9Hvg5z4k=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBhZvm6pQrN0LW+3wXswe61yUWzNRCeiMYdNIgdzlIW pwNH4v2JAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYWb5ugAKCRDDTyI5ktmPJPaRC/ 0TFaQf1l4oXIYMSl/XixFQjAKPWbE6IaHKS55gaL0B+QSyq8MJEB6ILOWtN1p4IPAg/lVW7LlaTkVz UQu4r+MgOD5tYjB6ObFxmUTKK+N3QuCvFgNmt6G0BQ29ktzJFZBkDtQWoTLKT/lbyVE8WwAULZkbUo DDVbK34xluwBfdvw9pBgdHFqSS72nc5eoyXgN38JgjINryA4HBp7i4vB2Qj/MD274xVfet3sMC82gF f2W31wCccJH/4zypqT191D6WjpjNuG+vdGphKaQRPSPlKEX0Fu/k62r6w8sCqVpQnwrPzswqRitV2D QO5TS6aqz86vHwmIFVLp/Pm3O0O1Qz3/1GmEEtAxREIes15EaScv9Y5JEmDo+i5R0QKtECo2jHaqXo qx043LQnKdjTl0pvJJ7BrP/k7pvPsvcmVrqymvrkMpG6K1W3Uj/KnxkJGJYBOXF1aKrRjkXsWQeHlD m/vwvs4kAVEB19u9g5VbwG7TwnM7oKS8X/hVjXbwQVqoM= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-hardening@vger.kernel.org Instead of branching back to the common exit point of the routine to pop the stack frame and return to the caller, move the frame pop to right after the point where we last use the callee save registers. This simplifies the generation of CFI unwind metadata, and reduces the number of needed branches. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/chacha-neon-core.S | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/arch/arm64/crypto/chacha-neon-core.S b/arch/arm64/crypto/chacha-neon-core.S index b70ac76f2610..918c0beae019 100644 --- a/arch/arm64/crypto/chacha-neon-core.S +++ b/arch/arm64/crypto/chacha-neon-core.S @@ -691,6 +691,8 @@ CPU_BE( rev a15, a15 ) zip2 v15.2d, v29.2d, v31.2d stp a14, a15, [x1, #-8] + frame_pop + tbnz x5, #63, .Lt128 ld1 {v28.16b-v31.16b}, [x2] @@ -726,7 +728,6 @@ CPU_BE( rev a15, a15 ) st1 {v24.16b-v27.16b}, [x1], #64 st1 {v28.16b-v31.16b}, [x1] -.Lout: frame_pop ret // fewer than 192 bytes of in/output @@ -744,7 +745,7 @@ CPU_BE( rev a15, a15 ) eor v23.16b, v23.16b, v31.16b st1 {v20.16b-v23.16b}, [x5] // overlapping stores 1: st1 {v16.16b-v19.16b}, [x1] - b .Lout + ret // fewer than 128 bytes of in/output .Lt128: ld1 {v28.16b-v31.16b}, [x10] @@ -772,7 +773,7 @@ CPU_BE( rev a15, a15 ) eor v31.16b, v31.16b, v3.16b st1 {v28.16b-v31.16b}, [x6] // overlapping stores 2: st1 {v20.16b-v23.16b}, [x1] - b .Lout + ret // fewer than 320 bytes of in/output .Lt320: cbz x7, 3f // exactly 256 bytes? @@ -789,7 +790,7 @@ CPU_BE( rev a15, a15 ) eor v31.16b, v31.16b, v3.16b st1 {v28.16b-v31.16b}, [x7] // overlapping stores 3: st1 {v24.16b-v27.16b}, [x1] - b .Lout + ret SYM_FUNC_END(chacha_4block_xor_neon) .section ".rodata", "a", %progbits From patchwork Wed Oct 13 15:22:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12556195 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 337C5C433EF for ; Wed, 13 Oct 2021 15:23:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0F7C161056 for ; Wed, 13 Oct 2021 15:23:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229903AbhJMPZJ (ORCPT ); Wed, 13 Oct 2021 11:25:09 -0400 Received: from mail.kernel.org ([198.145.29.99]:38434 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232155AbhJMPZJ (ORCPT ); Wed, 13 Oct 2021 11:25:09 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 6B970610E6; Wed, 13 Oct 2021 15:23:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1634138585; bh=r4Go4dh0trlbsGP6xRgj/tkJetmXK5uRKN3kIUWn2dg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GTjBqGENI9J1hRQHCMIOMOmJYdsT+WRhTQOr/eezV4u8ulOYIOQfQNdc54zZnp7vt RGCu0UFxGtG2EH/rF4WXKz7vsdhMtV0rDv+v2FfgkQdwg8cR5oCIuPIN4S1p85FoOd 78pQfL86SbSSvaIgqkxhAefzCdUDHQHy0kFopAm/A1UQ/aAjgIsfrs0wZHflZFnL3k xx6M+GFBJkNx8y91/7A//Fy/bGwXd/Rsjdcqtu3/VBGu4Hmlli9w2rZrr7wOgJtT8Y BnlcsQTPEV2srn7EcQYr+bKE/q5zm3AsCqEJIyCtB4cJBjWRXSnldgQhnTMTczslyO Wp2iJut6UjJnA== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-hardening@vger.kernel.org, mark.rutland@arm.com, catalin.marinas@arm.com, will@kernel.org, Ard Biesheuvel Subject: [RFC PATCH 6/9] arm64: smccc: create proper stack frames for HVC/SMC calls Date: Wed, 13 Oct 2021 17:22:40 +0200 Message-Id: <20211013152243.2216899-7-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211013152243.2216899-1-ardb@kernel.org> References: <20211013152243.2216899-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2910; h=from:subject; bh=r4Go4dh0trlbsGP6xRgj/tkJetmXK5uRKN3kIUWn2dg=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBhZvm82mjLvzjabrwdMggjKzvG7y6a6J+P9fQHWdJX BcIaSamJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYWb5vAAKCRDDTyI5ktmPJMHKDA CT90uHSWGg0U2c5ChDWZ2XV9zpvTw0+gxY0K1gFpCV2fBA83/GD0eHOPhvqIvjWqhjNV0HtIDAc1OS b7OLhQBxkbC7W1gGX3xdGP81E0P6wppREmBWJHMYRvJotMYF3RO9C0MRfQP0LXCY0kXRz9rF/GBPK/ Bx0TnoKmKY95K3390uXkrbVN+y2KkT3NP1+2lS5MEOgOicav8Qd/WW3tjosDk8JTSRqFBVpRf7vlVu IB5ebHxQIl5pCn5Fm22ZI5Y3Zd59MCyr2nXSjeA51w0HcJ5MUiVcTUkzyixnsLzGH5Hxul356bhi77 miEOJ4FsfQrGvw5WPBrzTSbil7UWgN1X7FCRqZjoZcjTDj9tKyZTfj2QVQs+9zAdkeWtEe6GgRAKyU QAM2PkqII8riMZzS+8P6jeYlXu4R6/FdZagRYpFcuswtXihjrTdo56Y7L17QIx/DYb44aaBWVQKjoM EnGL9udMtc/JyYgZwpGL45FdCpc7SUymfVEiorMiNSMhI= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-hardening@vger.kernel.org Create proper stack frames using the provided macros for HVC/SMC calling helpers that use the stack. This adds the PAC return address signing when enabled, and ensures that the unwinder can deal with occurrences of these routines appearing on the call stack. Signed-off-by: Ard Biesheuvel Reviewed-by: Mark Brown --- arch/arm64/kernel/smccc-call.S | 40 +++++++++----------- 1 file changed, 17 insertions(+), 23 deletions(-) diff --git a/arch/arm64/kernel/smccc-call.S b/arch/arm64/kernel/smccc-call.S index 487381164ff6..b1864880159a 100644 --- a/arch/arm64/kernel/smccc-call.S +++ b/arch/arm64/kernel/smccc-call.S @@ -32,8 +32,7 @@ SYM_FUNC_END(__arm_smccc_sve_check) EXPORT_SYMBOL(__arm_smccc_sve_check) .macro SMCCC instr - stp x29, x30, [sp, #-16]! - mov x29, sp + frame_push 0 alternative_if ARM64_SVE bl __arm_smccc_sve_check alternative_else_nop_endif @@ -47,7 +46,7 @@ alternative_else_nop_endif cmp x9, #ARM_SMCCC_QUIRK_QCOM_A6 b.ne 1f str x6, [x4, ARM_SMCCC_QUIRK_STATE_OFFS] -1: ldp x29, x30, [sp], #16 +1: frame_pop ret .endm @@ -74,11 +73,10 @@ SYM_FUNC_END(__arm_smccc_hvc) EXPORT_SYMBOL(__arm_smccc_hvc) .macro SMCCC_1_2 instr - /* Save `res` and free a GPR that won't be clobbered */ - stp x1, x19, [sp, #-16]! + frame_push 2 - /* Ensure `args` won't be clobbered while loading regs in next step */ - mov x19, x0 + mov x19, x0 // preserve args + mov x20, x1 // preserve res /* Load the registers x0 - x17 from the struct arm_smccc_1_2_regs */ ldp x0, x1, [x19, #ARM_SMCCC_1_2_REGS_X0_OFFS] @@ -93,24 +91,20 @@ EXPORT_SYMBOL(__arm_smccc_hvc) \instr #0 - /* Load the `res` from the stack */ - ldr x19, [sp] - /* Store the registers x0 - x17 into the result structure */ - stp x0, x1, [x19, #ARM_SMCCC_1_2_REGS_X0_OFFS] - stp x2, x3, [x19, #ARM_SMCCC_1_2_REGS_X2_OFFS] - stp x4, x5, [x19, #ARM_SMCCC_1_2_REGS_X4_OFFS] - stp x6, x7, [x19, #ARM_SMCCC_1_2_REGS_X6_OFFS] - stp x8, x9, [x19, #ARM_SMCCC_1_2_REGS_X8_OFFS] - stp x10, x11, [x19, #ARM_SMCCC_1_2_REGS_X10_OFFS] - stp x12, x13, [x19, #ARM_SMCCC_1_2_REGS_X12_OFFS] - stp x14, x15, [x19, #ARM_SMCCC_1_2_REGS_X14_OFFS] - stp x16, x17, [x19, #ARM_SMCCC_1_2_REGS_X16_OFFS] - - /* Restore original x19 */ - ldp xzr, x19, [sp], #16 + stp x0, x1, [x20, #ARM_SMCCC_1_2_REGS_X0_OFFS] + stp x2, x3, [x20, #ARM_SMCCC_1_2_REGS_X2_OFFS] + stp x4, x5, [x20, #ARM_SMCCC_1_2_REGS_X4_OFFS] + stp x6, x7, [x20, #ARM_SMCCC_1_2_REGS_X6_OFFS] + stp x8, x9, [x20, #ARM_SMCCC_1_2_REGS_X8_OFFS] + stp x10, x11, [x20, #ARM_SMCCC_1_2_REGS_X10_OFFS] + stp x12, x13, [x20, #ARM_SMCCC_1_2_REGS_X12_OFFS] + stp x14, x15, [x20, #ARM_SMCCC_1_2_REGS_X14_OFFS] + stp x16, x17, [x20, #ARM_SMCCC_1_2_REGS_X16_OFFS] + + frame_pop ret -.endm + .endm /* * void arm_smccc_1_2_hvc(const struct arm_smccc_1_2_regs *args, From patchwork Wed Oct 13 15:22:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12556197 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE83EC433F5 for ; Wed, 13 Oct 2021 15:23:08 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id ABE4160FDA for ; Wed, 13 Oct 2021 15:23:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232376AbhJMPZL (ORCPT ); Wed, 13 Oct 2021 11:25:11 -0400 Received: from mail.kernel.org ([198.145.29.99]:38456 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232155AbhJMPZL (ORCPT ); Wed, 13 Oct 2021 11:25:11 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 51BC261151; Wed, 13 Oct 2021 15:23:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1634138587; bh=WAUPlo+6rkxh20C/Rd3ch6Ny8CbqlBcV9dvmBueVhJc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rydVf2ZGqPUhh8gXLX+BLmUJKyj2OPvFwDS5Xg0Htl5WbXoml3sP3XUgRJpSTQsPj D5tR9TZx5SQaIMp25PCPU5LkOdLbnrMjzaE+8hPT2Up6VvDWVrTl7epVfraOqDhu8d bvPnxIirWIralEu7PksYEN1BANaHw4Tv5W/L3L1/9wYW1/y+rT46III0OW/1PO/OPs YhnAao6Raji9ay6merftk6FkoxwF7xMWlnTWbZ0eZrq1HLicSxP5CP3HQLs/qdLgsv 5XrX14PTKJTIvzSK+1t8AoBI8w66mkfgkCqiCfLuwKSrFCxRp1lQbO+5BLuIW/8rTk nUzVagDxFEbLw== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-hardening@vger.kernel.org, mark.rutland@arm.com, catalin.marinas@arm.com, will@kernel.org, Ard Biesheuvel Subject: [RFC PATCH 7/9] arm64: assembler: add unwind annotations to frame push/pop macros Date: Wed, 13 Oct 2021 17:22:41 +0200 Message-Id: <20211013152243.2216899-8-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211013152243.2216899-1-ardb@kernel.org> References: <20211013152243.2216899-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4153; h=from:subject; bh=WAUPlo+6rkxh20C/Rd3ch6Ny8CbqlBcV9dvmBueVhJc=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBhZvm+f+lPsOaLzrEcFZ+XSixPrrCB/OfJN0rxMhvG 39kq5QqJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYWb5vgAKCRDDTyI5ktmPJHBfC/ 9OkG6Igq017oaBMCAEjx3/AiCKQMrbpzgwdfz9dG43g4m6sv+kOaGkVS7e0mTgzkY6WMprx8oK7Pty 7dexFNDKaeVKwgsfOQDTG4aw9IPmYin6BBoohz0Aen3JnkNUXSmHZI2hftinPr1JHJNciqi04zCCAw gQiXFE47ZTwkqofmsA78KbS1e8i/KQA5PYlBfthuXl0HTp5pvHoU79KG/rdNU+vN57zlfGOZlKCUNb TG6Nn6hFDYjcMIP68RMdHIwXTJchgW/xqNFrlbHMQbrFnysJlDzVj3SsundzCmXCOeSs0fH6ffHGRk hYNKNI46VoadcwjttB4ULrjROO6yR25+lZnBlGFurUgpe0lRiiNf0dH2KpT5gcSiG+lu63xv/F38U9 QaYut8C4ryl+BsMDCAA2s4Mjs/5ZnO7jGBlVcN31/DrIAcm+BRtt0Ed8N9O2QsIGTGW/nW78gMM3tL tIo8K0EQqYUWAmAILNS+PZDeqoc08/XTkmW+yROezSU3E= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-hardening@vger.kernel.org In order to ensure that we can unwind from hand rolled assembly routines, decorate the frame push/pop helper macros that are used by non-leaf assembler routines with the appropriate annotations. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/assembler.h | 26 +++++++++++++++++++- arch/arm64/include/asm/linkage.h | 16 +++++++++++- 2 files changed, 40 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index ceed84ac4005..cebb6c8c489b 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -664,9 +664,10 @@ alternative_endif * the new value of sp. Add @extra bytes of stack space * for locals. */ - .macro frame_push, regcount:req, extra + .macro frame_push, regcount:req, extra=0 #ifdef CONFIG_ARM64_PTR_AUTH_KERNEL paciasp + .cfi_negate_ra_state #endif __frame st, \regcount, \extra .endm @@ -681,14 +682,29 @@ alternative_endif __frame ld #ifdef CONFIG_ARM64_PTR_AUTH_KERNEL autiasp + .cfi_negate_ra_state #endif .endm .macro __frame_regs, reg1, reg2, op, num .if .Lframe_regcount == \num \op\()r \reg1, [sp, #(\num + 1) * 8] + .ifc \op, st + .cfi_offset \reg1, -.Lframe_cfa_offset + .set .Lframe_cfa_offset, .Lframe_cfa_offset - 8 + .else + .cfi_restore \reg1 + .endif .elseif .Lframe_regcount > \num \op\()p \reg1, \reg2, [sp, #(\num + 1) * 8] + .ifc \op, st + .cfi_offset \reg1, -.Lframe_cfa_offset + .cfi_offset \reg2, -.Lframe_cfa_offset + 8 + .set .Lframe_cfa_offset, .Lframe_cfa_offset - 16 + .else + .cfi_restore \reg1 + .cfi_restore \reg2 + .endif .endif .endm @@ -708,7 +724,12 @@ alternative_endif .set .Lframe_regcount, \regcount .set .Lframe_extra, \extra .set .Lframe_local_offset, ((\regcount + 3) / 2) * 16 + .set .Lframe_cfa_offset, .Lframe_local_offset + .Lframe_extra stp x29, x30, [sp, #-.Lframe_local_offset - .Lframe_extra]! + .cfi_def_cfa_offset .Lframe_cfa_offset + .cfi_offset x29, -.Lframe_cfa_offset + .cfi_offset x30, -.Lframe_cfa_offset + 8 + .set .Lframe_cfa_offset, .Lframe_cfa_offset - 16 mov x29, sp .endif @@ -723,6 +744,9 @@ alternative_endif .error "frame_push/frame_pop may not be nested" .endif ldp x29, x30, [sp], #.Lframe_local_offset + .Lframe_extra + .cfi_restore x29 + .cfi_restore x30 + .cfi_def_cfa_offset 0 .set .Lframe_regcount, -1 .endif .endm diff --git a/arch/arm64/include/asm/linkage.h b/arch/arm64/include/asm/linkage.h index 9906541a6861..d984a6750b01 100644 --- a/arch/arm64/include/asm/linkage.h +++ b/arch/arm64/include/asm/linkage.h @@ -4,6 +4,9 @@ #define __ALIGN .align 2 #define __ALIGN_STR ".align 2" +#define SYM_FUNC_CFI_START .cfi_startproc ; +#define SYM_FUNC_CFI_END .cfi_endproc ; + #if defined(CONFIG_ARM64_BTI_KERNEL) && defined(__aarch64__) /* @@ -12,6 +15,9 @@ * instead. */ #define BTI_C hint 34 ; +#else +#define BTI_C +#endif /* * When using in-kernel BTI we need to ensure that PCS-conformant assembly @@ -20,29 +26,37 @@ */ #define SYM_FUNC_START(name) \ SYM_START(name, SYM_L_GLOBAL, SYM_A_ALIGN) \ + SYM_FUNC_CFI_START \ BTI_C #define SYM_FUNC_START_NOALIGN(name) \ SYM_START(name, SYM_L_GLOBAL, SYM_A_NONE) \ + SYM_FUNC_CFI_START \ BTI_C #define SYM_FUNC_START_LOCAL(name) \ SYM_START(name, SYM_L_LOCAL, SYM_A_ALIGN) \ + SYM_FUNC_CFI_START \ BTI_C #define SYM_FUNC_START_LOCAL_NOALIGN(name) \ SYM_START(name, SYM_L_LOCAL, SYM_A_NONE) \ + SYM_FUNC_CFI_START \ BTI_C #define SYM_FUNC_START_WEAK(name) \ SYM_START(name, SYM_L_WEAK, SYM_A_ALIGN) \ + SYM_FUNC_CFI_START \ BTI_C #define SYM_FUNC_START_WEAK_NOALIGN(name) \ SYM_START(name, SYM_L_WEAK, SYM_A_NONE) \ + SYM_FUNC_CFI_START \ BTI_C -#endif +#define SYM_FUNC_END(name) \ + SYM_FUNC_CFI_END \ + SYM_END(name, SYM_T_FUNC) /* * Annotate a function as position independent, i.e., safe to be called before From patchwork Wed Oct 13 15:22:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12556199 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2E5AC433F5 for ; Wed, 13 Oct 2021 15:23:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 897FD60FDA for ; Wed, 13 Oct 2021 15:23:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232459AbhJMPZN (ORCPT ); Wed, 13 Oct 2021 11:25:13 -0400 Received: from mail.kernel.org ([198.145.29.99]:38488 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232155AbhJMPZM (ORCPT ); Wed, 13 Oct 2021 11:25:12 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 3807E60F21; Wed, 13 Oct 2021 15:23:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1634138589; bh=eQD2BO1y1WN78+tF0zBVscKBClg5cy2UDC7R0JabKQ0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TYOWcnI8tY/huxcVvWWtPttfLGRsEXOX4QP3lMR9EVzCGFDWrT3GJtaH5UBwoHROj gABvvuMrJAhXq7a0P9w6wmyigUGjCWKKfk7/jBjVb/TQOPjjnFHA9nrQhDcOs4E3Gz CMFfa+MHdKqeiS5p0eMO+/HyXDAovAgFKsfWbrlgknf2wRbKdNFHPZ2fd02LQs00oS FLL8aqzcFeZulzfyr53TH8sn4kZa2ieWXfgfPtYIRbItcRylaDJfJVOI6TkU8NrvxV KbTN5lw7C4kDh7vikCQKZBiLThO3cwPtqgpEVMbtd7Qh4V1KEEpa/Fa0/qsuJZFxRM e39Z2fKm/04rw== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-hardening@vger.kernel.org, mark.rutland@arm.com, catalin.marinas@arm.com, will@kernel.org, Ard Biesheuvel Subject: [RFC PATCH 8/9] arm64: unwind: add asynchronous unwind tables to the kernel proper Date: Wed, 13 Oct 2021 17:22:42 +0200 Message-Id: <20211013152243.2216899-9-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211013152243.2216899-1-ardb@kernel.org> References: <20211013152243.2216899-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3477; h=from:subject; bh=eQD2BO1y1WN78+tF0zBVscKBClg5cy2UDC7R0JabKQ0=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBhZvnACwv78YQ+ao61GTYFQpPJL8whagcvyk+2z7hF B4a2nc2JAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYWb5wAAKCRDDTyI5ktmPJB4QDA C5IvWhLaJcK5LFXb2wAHEUkFk4eNXft92/H7nBWwPhlEt21U5OXEGDb/vALqI1sdvfqYRcLRDOZO45 Mc5mkTbYez8PmglVQin5Jnghv6wozsg13BpA0YZjNoud8DzOEnwBAjvq5LrLiWZ0+xssZZeDIqPtaa vuxjiDpGE4QUn+iyV8hgGLkaMfdBib/6EkyT9MLu8XPPrVbGV98qpdpmJdT62iyc8BzyOo+Jn1mpBy IRlUODD3t8keapV5jK3u7gH6xlGCsmaWGgZ2NtGBkMk1BIkDMxr3udDsi5YdYusDMWqC8VgUY1U5TR 38kkJLlIp9PMm9lE3k0KDwdG19uvcIy/WUg9V/sfCX5ktV/FRkJ2zz8hO7bdcmpN0zK+54ne2MNPvA vO0NP9PjhLi+7anPOX1BMx+0G4Pnzflb1YzQmyKuOwY8oTE6IvMBuV8K53QWvKU/knLusfrwvZ/KW0 ZJQ8KcynVidoucRcbzpMylbzvlLOZUCNWBj7hCGqD0GQU= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-hardening@vger.kernel.org Signed-off-by: Ard Biesheuvel --- arch/arm64/Kconfig | 3 +++ arch/arm64/Makefile | 7 ++++++- arch/arm64/kernel/vmlinux.lds.S | 20 ++++++++++++++++++++ drivers/firmware/efi/libstub/Makefile | 1 + 4 files changed, 30 insertions(+), 1 deletion(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 077f2ec4eeb2..742baca09343 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -350,6 +350,9 @@ config KASAN_SHADOW_OFFSET default 0xeffffff800000000 if ARM64_VA_BITS_36 && KASAN_SW_TAGS default 0xffffffffffffffff +config UNWIND_TABLES + bool + source "arch/arm64/Kconfig.platforms" menu "Kernel Features" diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile index c744b1e7b356..95ffc4deebb0 100644 --- a/arch/arm64/Makefile +++ b/arch/arm64/Makefile @@ -10,7 +10,7 @@ # # Copyright (C) 1995-2001 by Russell King -LDFLAGS_vmlinux :=--no-undefined -X +LDFLAGS_vmlinux :=--no-undefined -X --eh-frame-hdr ifeq ($(CONFIG_RELOCATABLE), y) # Pass --no-apply-dynamic-relocs to restore pre-binutils-2.27 behaviour @@ -45,8 +45,13 @@ KBUILD_CFLAGS += $(call cc-option,-mabi=lp64) KBUILD_AFLAGS += $(call cc-option,-mabi=lp64) # Avoid generating .eh_frame* sections. +ifneq ($(CONFIG_UNWIND_TABLES),y) KBUILD_CFLAGS += -fno-asynchronous-unwind-tables -fno-unwind-tables KBUILD_AFLAGS += -fno-asynchronous-unwind-tables -fno-unwind-tables +else +KBUILD_CFLAGS += -fasynchronous-unwind-tables +KBUILD_AFLAGS += -fasynchronous-unwind-tables +endif ifeq ($(CONFIG_STACKPROTECTOR_PER_TASK),y) prepare: stack_protector_prepare diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index f6b1a88245db..ed3db80bf696 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -111,6 +111,21 @@ jiffies = jiffies_64; #define TRAMP_TEXT #endif +#ifdef CONFIG_UNWIND_TABLES +#define UNWIND_DATA_SECTIONS \ + .eh_frame_hdr : { \ + __eh_frame_hdr = .; \ + *(.eh_frame_hdr) \ + } \ + .eh_frame : { \ + __eh_frame_start = .; \ + *(.eh_frame) \ + __eh_frame_end = .; \ + } +#else +#define UNWIND_DATA_SECTIONS +#endif + /* * The size of the PE/COFF section that covers the kernel image, which * runs from _stext to _edata, must be a round multiple of the PE/COFF @@ -139,6 +154,9 @@ SECTIONS /DISCARD/ : { *(.interp .dynamic) *(.dynsym .dynstr .hash .gnu.hash) +#ifndef CONFIG_UNWIND_TABLES + *(.eh_frame_hdr .eh_frame) +#endif } . = KIMAGE_VADDR; @@ -217,6 +235,8 @@ SECTIONS __alt_instructions_end = .; } + UNWIND_DATA_SECTIONS + . = ALIGN(SEGMENT_ALIGN); __inittext_end = .; __initdata_begin = .; diff --git a/drivers/firmware/efi/libstub/Makefile b/drivers/firmware/efi/libstub/Makefile index d0537573501e..78c46638707a 100644 --- a/drivers/firmware/efi/libstub/Makefile +++ b/drivers/firmware/efi/libstub/Makefile @@ -20,6 +20,7 @@ cflags-$(CONFIG_X86) += -m$(BITS) -D__KERNEL__ \ # disable the stackleak plugin cflags-$(CONFIG_ARM64) := $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS)) \ -fpie $(DISABLE_STACKLEAK_PLUGIN) \ + -fno-unwind-tables -fno-asynchronous-unwind-tables \ $(call cc-option,-mbranch-protection=none) cflags-$(CONFIG_ARM) := $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS)) \ -fno-builtin -fpic \ From patchwork Wed Oct 13 15:22:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12556201 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E991C433FE for ; Wed, 13 Oct 2021 15:23:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C4BE060F21 for ; Wed, 13 Oct 2021 15:23:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232609AbhJMPZP (ORCPT ); Wed, 13 Oct 2021 11:25:15 -0400 Received: from mail.kernel.org ([198.145.29.99]:38512 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232155AbhJMPZO (ORCPT ); Wed, 13 Oct 2021 11:25:14 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 1EC2E6113E; Wed, 13 Oct 2021 15:23:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1634138591; bh=PXB3zQXwY0Jqsa61K53dco0RurO4Co2tyKwXIIahwnI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qvkjWd9jJn9hTlts0gCzjeMwFGqLvmSiTgqzZQgJYQrrre+w9S8miXZTN7BiPKrdD 7/wsGHO77mhRXWMCAPF0gTKtdgn7jJ28QaY2+7OUCHMn00tm/sta5oGi/6iDVDK8i0 xoGAvqSSjxF8zoxGiU5GphBr5nGgw5+ln8DXxHiUx0xVoLObpcLFKQxZQplAhj2XoO ealTmacuXXqtJkmtrwuOiMZpwrapBT9S0uG5TfpychI5LgmjOxn54lmyFXr3bg+tHx luED3OJUkb7Hg7RWcYp3FpMBRIv5QDtn4E+K1ZrpCW3m0TIO6m4U1e+pClGkHRLAKi UK1PIrfxpOCzw== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-hardening@vger.kernel.org, mark.rutland@arm.com, catalin.marinas@arm.com, will@kernel.org, Ard Biesheuvel Subject: [RFC PATCH 9/9] arm64: implement dynamic shadow call stack for GCC Date: Wed, 13 Oct 2021 17:22:43 +0200 Message-Id: <20211013152243.2216899-10-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211013152243.2216899-1-ardb@kernel.org> References: <20211013152243.2216899-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=9853; h=from:subject; bh=PXB3zQXwY0Jqsa61K53dco0RurO4Co2tyKwXIIahwnI=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBhZvnCc0tGMlh3ZCFYjZM2hyOSuNfLhbLZgsGho1t0 P7mNuiGJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYWb5wgAKCRDDTyI5ktmPJJeeC/ 9lm4HJNbi4cGb6v06k8mFyWNcgjJHCh3uOLd71S/+nknFxy1El071KwD5LlBnfRRdIIucHHYtYupK9 nFSpJ4KwteZH57ktKejg1WvAFkg0oCy4KllJBnAQuQUnVoeGRczXfRLYuffk2n5TJe7Xqok9zQQ2kS o0simTEfnjIjltwa5n4oF7wwuqsFFU9xiEzkpRDMwdzLyaMzdpIRmFKBzvNdtdD09PfjojMiNcovZo M6LBrXwe5Wb1T/vzCxoyUS61YlqLFcpjPZRHDYq/hm0etgKpCv2Lc2KHbsJfPCnX6Y5UTQG8Sjl/kb lm7VpQmFdmxpElNQt9aC2AeDvUE8MDoqxNNUwjZknODCWiIWjCey/87M+rZdilLnDU+gS4oTA7NHuK UKiXp3Th+UH+xHhQcwWilZB/0STKi0indpgYnUUY73g93SlPznpfi8vN/KjYDw011uyUxDl0vNTQo/ yoGxvrhrr9/965QV5fCTP7uiaLyhvoGffecTuBO1XjDhs= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-hardening@vger.kernel.org Implement support for the shadow call stack on GCC, and in a dynamic manner, by parsing the unwind tables at init time to locate all occurrences of PACIASP/AUTIASP, and replacing them with the shadow call stack push and pop instructions, respectively. This is useful because the overhead of the shadow call stack is difficult to justify on hardware that implements pointer authentication (PAC), and given that the PAC instructions are executed as NOPs on hardware that doesn't, we can just replace them. This patch only implements this for the core kernel, but the logic can be reused for modules without much trouble. Signed-off-by: Ard Biesheuvel --- Makefile | 4 +- arch/Kconfig | 4 +- arch/arm64/Kconfig | 8 +- arch/arm64/kernel/Makefile | 2 + arch/arm64/kernel/head.S | 3 + arch/arm64/kernel/patch-scs.c | 223 ++++++++++++++++++++ 6 files changed, 239 insertions(+), 5 deletions(-) diff --git a/Makefile b/Makefile index 7cfe4ff36f44..2d94fed93d9d 100644 --- a/Makefile +++ b/Makefile @@ -933,8 +933,8 @@ LDFLAGS_vmlinux += --gc-sections endif ifdef CONFIG_SHADOW_CALL_STACK -CC_FLAGS_SCS := -fsanitize=shadow-call-stack -KBUILD_CFLAGS += $(CC_FLAGS_SCS) +CC_FLAGS_SCS-$(CONFIG_CC_IS_CLANG) := -fsanitize=shadow-call-stack +KBUILD_CFLAGS += $(CC_FLAGS_SCS-y) export CC_FLAGS_SCS endif diff --git a/arch/Kconfig b/arch/Kconfig index 8df1c7102643..21eeec66bf4c 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -596,8 +596,8 @@ config ARCH_SUPPORTS_SHADOW_CALL_STACK switching. config SHADOW_CALL_STACK - bool "Clang Shadow Call Stack" - depends on CC_IS_CLANG && ARCH_SUPPORTS_SHADOW_CALL_STACK + bool "Shadow Call Stack" + depends on ARCH_SUPPORTS_SHADOW_CALL_STACK depends on DYNAMIC_FTRACE_WITH_REGS || !FUNCTION_GRAPH_TRACER help This option enables Clang's Shadow Call Stack, which uses a diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 742baca09343..6d74822fd386 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -81,7 +81,7 @@ config ARM64 select ARCH_SUPPORTS_DEBUG_PAGEALLOC select ARCH_SUPPORTS_HUGETLBFS select ARCH_SUPPORTS_MEMORY_FAILURE - select ARCH_SUPPORTS_SHADOW_CALL_STACK if CC_HAVE_SHADOW_CALL_STACK + select ARCH_SUPPORTS_SHADOW_CALL_STACK if CC_HAVE_SHADOW_CALL_STACK || CC_IS_GCC select ARCH_SUPPORTS_LTO_CLANG if CPU_LITTLE_ENDIAN select ARCH_SUPPORTS_LTO_CLANG_THIN select ARCH_SUPPORTS_CFI_CLANG @@ -353,6 +353,12 @@ config KASAN_SHADOW_OFFSET config UNWIND_TABLES bool +config UNWIND_PATCH_PAC_INTO_SCS + def_bool y + depends on CC_IS_GCC && SHADOW_CALL_STACK + select UNWIND_TABLES + select ARM64_PTR_AUTH_KERNEL + source "arch/arm64/Kconfig.platforms" menu "Kernel Features" diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index 3f1490bfb938..42b9bd92d51e 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -73,6 +73,8 @@ obj-$(CONFIG_ARM64_PTR_AUTH) += pointer_auth.o obj-$(CONFIG_ARM64_MTE) += mte.o obj-y += vdso-wrap.o obj-$(CONFIG_COMPAT_VDSO) += vdso32-wrap.o +obj-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) += patch-scs.o +CFLAGS_patch-scs.o += -mbranch-protection=none obj-y += probes/ head-y := head.o diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 17962452e31d..5d50d212d3ae 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -447,6 +447,9 @@ SYM_FUNC_START_LOCAL(__primary_switched) bl __pi_memset dsb ishst // Make zero page visible to PTW +#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS + bl scs_patch_vmlinux +#endif #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init #endif diff --git a/arch/arm64/kernel/patch-scs.c b/arch/arm64/kernel/patch-scs.c new file mode 100644 index 000000000000..878a40060550 --- /dev/null +++ b/arch/arm64/kernel/patch-scs.c @@ -0,0 +1,223 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2021 - Google LLC + * Author: Ard Biesheuvel + */ + +#include +#include +#include +#include +#include + +#define DW_CFA_nop 0x00 +#define DW_CFA_set_loc 0x01 +#define DW_CFA_advance_loc1 0x02 +#define DW_CFA_advance_loc2 0x03 +#define DW_CFA_advance_loc4 0x04 +#define DW_CFA_offset_extended 0x05 +#define DW_CFA_restore_extended 0x06 +#define DW_CFA_undefined 0x07 +#define DW_CFA_same_value 0x08 +#define DW_CFA_register 0x09 +#define DW_CFA_remember_state 0x0a +#define DW_CFA_restore_state 0x0b +#define DW_CFA_def_cfa 0x0c +#define DW_CFA_def_cfa_register 0x0d +#define DW_CFA_def_cfa_offset 0x0e +#define DW_CFA_def_cfa_expression 0x0f +#define DW_CFA_expression 0x10 +#define DW_CFA_offset_extended_sf 0x11 +#define DW_CFA_def_cfa_sf 0x12 +#define DW_CFA_def_cfa_offset_sf 0x13 +#define DW_CFA_val_offset 0x14 +#define DW_CFA_val_offset_sf 0x15 +#define DW_CFA_val_expression 0x16 +#define DW_CFA_lo_user 0x1c +#define DW_CFA_negate_ra_state 0x2d +#define DW_CFA_GNU_args_size 0x2e +#define DW_CFA_GNU_negative_offset_extended 0x2f +#define DW_CFA_hi_user 0x3f + +static unsigned long get_uleb128(const u8 **pcur, const u8 *end) +{ + const u8 *cur = *pcur; + unsigned long value; + unsigned int shift; + + for (shift = 0, value = 0; cur < end; shift += 7) { + if (shift + 7 > 8 * sizeof(value) + && (*cur & 0x7fU) >= (1U << (8 * sizeof(value) - shift))) { + cur = end + 1; + break; + } + value |= (unsigned long) (*cur & 0x7f) << shift; + if (!(*cur++ & 0x80)) + break; + } + *pcur = cur; + + return value; +} + +extern const u8 __eh_frame_start[], __eh_frame_end[]; + +struct fde_frame { + s32 initial_loc; + s32 range; +}; + +static int scs_patch_loc(u64 loc) +{ + u32 insn = le32_to_cpup((void *)loc); + + /* + * Sometimes, the unwind data appears to be out of sync, and associates + * the DW_CFA_negate_ra_state directive with the ret instruction + * following the autiasp, rather than the autiasp itself. + */ + if (insn == 0xd65f03c0) { // ret + loc -= 4; + insn = le32_to_cpup((void *)loc); + } + + switch (insn) { + case 0xd503233f: // paciasp + *(u32 *)loc = cpu_to_le32(0xf800865e); + break; + case 0xd50323bf: // autiasp + *(u32 *)loc = cpu_to_le32(0xf85f8e5e); + break; + default: + // ignore + break; + } + return 0; +} + +static int noinstr scs_handle_frame(const u8 eh_frame[], u32 size) +{ + const struct fde_frame *fde; + const u8 *opcode; + u64 loc; + + /* + * For patching PAC opcodes, we only care about the FDE records, and + * not the CIE, which carries the initial CFA directives but they only + * pertain to which register is the stack pointer. + * TODO this is not 100% true - we need the augmentation string and the + * encoding but they are always the same in practice. + */ + if (*(u32 *)eh_frame == 0) + return 0; + + fde = (const struct fde_frame *)(eh_frame + 4); + loc = (u64)offset_to_ptr(&fde->initial_loc); + opcode = (const u8 *)(fde + 1); + + // TODO check augmentation data + WARN_ON(*opcode++); + size -= sizeof(u32) + sizeof(*fde) + 1; + + /* + * Starting from 'loc', apply the CFA opcodes that advance the location + * pointer, and identify the locations of the PAC instructions. + */ + do { + const u8 *end; + + switch (*opcode & 0xC0) { + case 0: + // handle DW_CFA_xxx opcodes + switch (*opcode) { + int ret; + + case DW_CFA_nop: + case DW_CFA_remember_state: + case DW_CFA_restore_state: + break; + + case DW_CFA_advance_loc1: + loc += 4 * *++opcode; + size--; + break; + + case DW_CFA_advance_loc2: + loc += 4 * *++opcode; + loc += 4 * *++opcode << 8; + size -= 2; + break; + + case DW_CFA_def_cfa: + case DW_CFA_def_cfa_offset: + case DW_CFA_def_cfa_register: + opcode++; + size--; + end = opcode + size; + get_uleb128(&opcode, end); + size = end - opcode; + continue; + + case DW_CFA_negate_ra_state: + // patch paciasp/autiasp into shadow stack push/pop + ret = scs_patch_loc(loc - 4); + if (ret) + return ret; + break; + + default: + pr_debug("unhandled opcode: %02x\n", *opcode); + return -ENOEXEC; + } + opcode++; + size--; + break; + + case 0x40: + // advance loc + loc += (*opcode++ & 0x3f) * 4; + size--; + break; + + case 0x80: + opcode++; + size--; + end = opcode + size; + get_uleb128(&opcode, end); + size = end - opcode; + continue; + + default: + // ignore + opcode++; + size--; + break; + } + } while (size > 0); + + return 0; +} + +int noinstr scs_patch(const u8 eh_frame[], int size) +{ + const u8 *p = eh_frame; + + while (size > 4) { + const u32 *frame_size = (const u32 *)p; + int ret; + + if (*frame_size != -1 && *frame_size <= size) { + ret = scs_handle_frame(p + 4, *frame_size); + if (ret) + return ret; + p += 4 + *frame_size; + size -= 4 + *frame_size; + } + } + return 0; +} + +asmlinkage int noinstr scs_patch_vmlinux(void) +{ + return scs_patch(__eh_frame_start, __eh_frame_end - __eh_frame_start); +}