From patchwork Mon Oct 18 14:16:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12566599 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C488C433F5 for ; Mon, 18 Oct 2021 14:20:20 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 45B3360F25 for ; Mon, 18 Oct 2021 14:20:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 45B3360F25 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=/x5q2uFpX0gXQL+LnEYAGt3O5iEHanw2Pob934kb1Fw=; b=XSRHBZgS7oeYep qpaMq2BP7c0rE6X+Zki1dUGsDwErrMgFI78DM7YB07+/WsbEpSkWiAZ8NPgGI3e8MwoRsUtHZNOsc wssDy7ylnjzBxkS8s7mUzQ3qYX4zMGTpQ3k6LaThY474Z4drt8MI2xBDkbguQ4u7w74+M8A8h7OAU 1JD3Owh6CrY68+JrTUWtqLR1/cMkoSSR0qziMmXiAdLTxauuRUBNXnjst8//L92uTFTgRG8cC5HxH lXX/fFsBlQVxJc/cW/v8MrL21azUz70I+mJBi9ZpHTuTd3SWgqq+O+rBlK+tel+125GV+rXmcuJkc fGz4vUZIeErn+qMp/Z7g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mcTT7-00G3Uk-AK; Mon, 18 Oct 2021 14:18:25 +0000 Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mcTRI-00G2jl-Km for linux-arm-kernel@lists.infradead.org; Mon, 18 Oct 2021 14:16:35 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id F2C0960FC2; Mon, 18 Oct 2021 14:16:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1634566592; bh=6I7mRiSzvoFUrl2SdgX9Xfxg6/c5Y9O/sTdnxNSR+SY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GHp6oGEEt1hTJiEFbSgU+n5qGsy6Ajr+XlxK5pZ/FywWEKjWUEpO968qHCMWgD9Xc uzpwTc96EbGEi+sD90BMQnP8WAttrpLcyPxhh3ge0ka79G0x6sJfTfQhAzMXDgLNZT nkWZPavEK8aMvUEd7TsBzJ0r+BOaF7giVyBUWwp0JZc/Fy9+dpBxGOrRNqFxXrNlNQ eS95ocI4iAXRixljkmMbfPPDWfUafQSJfGWR6UEfUtw8rNItvsQER//bvkodLCj9fJ 1//DjZwjVnNCZ5p8ZdzlChlEVLJJ2MDaVo+JCD97re5B3U5ogwW+V2dwmxl+AJl8pX DyGsLt7VSr07Q== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Russell King , Nicolas Pitre , Arnd Bergmann , Kees Cook , Keith Packard , Linus Walleij , Nick Desaulniers Subject: [PATCH v2 1/7] ARM: memcpy: use frame pointer as unwind anchor Date: Mon, 18 Oct 2021 16:16:09 +0200 Message-Id: <20211018141615.682070-2-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211018141615.682070-1-ardb@kernel.org> References: <20211018141615.682070-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=8204; h=from:subject; bh=6I7mRiSzvoFUrl2SdgX9Xfxg6/c5Y9O/sTdnxNSR+SY=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBhbYGit0ac6zh4eQ1jxs/Eem2nkqmIpBFdZ1Wl3uBj BzjTAtWJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYW2BogAKCRDDTyI5ktmPJPbYDA CASPy2+hDnUoB4Lt1PSqqQeF5gCNW4/80Koub+3p93r7gh0Xn4tWCkU+8C7B3ekkwz5+IIEkydfaH1 ePh62ZwdpPHlgLUaygqmZ8NwoU6ypv1FXvoWfvX+JkxG3nrtbvtf7vtd8xL7F7U0yhrzkO5AjsrRU2 NCdkwPxYtLsyAX3QzOx1bJxmweTcRmFGRA18YPGPSvdCkTqWC1VDJdQq7gztumoe5bPbLw7wrYlrJ8 TJAawUmLVd7FqSJ2+pZUv7mm2c1+22xMl/uqREnM98oju0zkbK2tK2cMwgU4UTEHOV2r4xDv8GIi+V L9PFd3DGtpHT7/KLbGm0id7zt6yVKglY1aMiyFeI+2n4A3WfxeKllW+efMFVF5vDinrYu1qhH/AGB5 X/SNhXkgB9+IJYh2EA8X23gHNbFzJXhplRoOUpKaFwfgfMI2zkYCko5DkQpFmGsVILEpgbUExV4aAF /SByZxZzFEuGoqOR0Svw92P4wMZTZqwBXc0k0BlV1fwJw= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211018_071632_781235_3230751C X-CRM114-Status: GOOD ( 17.34 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The memcpy template is a bit unusual in the way it manages the stack pointer: depending on the execution path through the function, the SP assumes different values as different subsets of the register file are preserved and restored again. This is problematic when it comes to EHABI unwind info, as it is not instruction accurate, and does not allow tracking the SP value as it changes. Commit 279f487e0b471 ("ARM: 8225/1: Add unwinding support for memory copy functions") addressed this by carving up the function in different chunks as far as the unwinder is concerned, and keeping a set of unwind directives for each of them, each corresponding with the state of the stack pointer during execution of the chunk in question. This not only duplicates unwind info unnecessarily, but it also complicates unwinding the stack upon overflow. Instead, let's do what the compiler does when the SP is updated halfway through a function, which is to use a frame pointer and emit the appropriate unwind directives to communicate this to the unwinder. Note that Thumb-2 uses R7 for this, while ARM uses R11 aka FP. So let's avoid touching R7 in the body of the template, so that Thumb-2 can use it as the frame pointer. R11 was not modified in the first place. Signed-off-by: Ard Biesheuvel Tested-by: Keith Packard --- arch/arm/lib/copy_from_user.S | 13 ++-- arch/arm/lib/copy_template.S | 67 +++++++------------- arch/arm/lib/copy_to_user.S | 13 ++-- arch/arm/lib/memcpy.S | 13 ++-- 4 files changed, 38 insertions(+), 68 deletions(-) diff --git a/arch/arm/lib/copy_from_user.S b/arch/arm/lib/copy_from_user.S index 480a20766137..270de7debd0f 100644 --- a/arch/arm/lib/copy_from_user.S +++ b/arch/arm/lib/copy_from_user.S @@ -91,18 +91,15 @@ strb\cond \reg, [\ptr], #1 .endm - .macro enter reg1 reg2 + .macro enter regs:vararg mov r3, #0 - stmdb sp!, {r0, r2, r3, \reg1, \reg2} +UNWIND( .save {r0, r2, r3, \regs} ) + stmdb sp!, {r0, r2, r3, \regs} .endm - .macro usave reg1 reg2 - UNWIND( .save {r0, r2, r3, \reg1, \reg2} ) - .endm - - .macro exit reg1 reg2 + .macro exit regs:vararg add sp, sp, #8 - ldmfd sp!, {r0, \reg1, \reg2} + ldmfd sp!, {r0, \regs} .endm .text diff --git a/arch/arm/lib/copy_template.S b/arch/arm/lib/copy_template.S index 810a805d36dc..8fbafb074fe9 100644 --- a/arch/arm/lib/copy_template.S +++ b/arch/arm/lib/copy_template.S @@ -69,13 +69,10 @@ * than one 32bit instruction in Thumb-2) */ - - UNWIND( .fnstart ) - enter r4, lr - UNWIND( .fnend ) - UNWIND( .fnstart ) - usave r4, lr @ in first stmdb block + enter r4, UNWIND(fpreg,) lr + UNWIND( .setfp fpreg, sp ) + UNWIND( mov fpreg, sp ) subs r2, r2, #4 blt 8f @@ -86,12 +83,7 @@ bne 10f 1: subs r2, r2, #(28) - stmfd sp!, {r5 - r8} - UNWIND( .fnend ) - - UNWIND( .fnstart ) - usave r4, lr - UNWIND( .save {r5 - r8} ) @ in second stmfd block + stmfd sp!, {r5, r6, r8, r9} blt 5f CALGN( ands ip, r0, #31 ) @@ -110,9 +102,9 @@ PLD( pld [r1, #92] ) 3: PLD( pld [r1, #124] ) -4: ldr8w r1, r3, r4, r5, r6, r7, r8, ip, lr, abort=20f +4: ldr8w r1, r3, r4, r5, r6, r8, r9, ip, lr, abort=20f subs r2, r2, #32 - str8w r0, r3, r4, r5, r6, r7, r8, ip, lr, abort=20f + str8w r0, r3, r4, r5, r6, r8, r9, ip, lr, abort=20f bge 3b PLD( cmn r2, #96 ) PLD( bge 4b ) @@ -132,8 +124,8 @@ ldr1w r1, r4, abort=20f ldr1w r1, r5, abort=20f ldr1w r1, r6, abort=20f - ldr1w r1, r7, abort=20f ldr1w r1, r8, abort=20f + ldr1w r1, r9, abort=20f ldr1w r1, lr, abort=20f #if LDR1W_SHIFT < STR1W_SHIFT @@ -150,17 +142,14 @@ str1w r0, r4, abort=20f str1w r0, r5, abort=20f str1w r0, r6, abort=20f - str1w r0, r7, abort=20f str1w r0, r8, abort=20f + str1w r0, r9, abort=20f str1w r0, lr, abort=20f CALGN( bcs 2b ) -7: ldmfd sp!, {r5 - r8} - UNWIND( .fnend ) @ end of second stmfd block +7: ldmfd sp!, {r5, r6, r8, r9} - UNWIND( .fnstart ) - usave r4, lr @ still in first stmdb block 8: movs r2, r2, lsl #31 ldr1b r1, r3, ne, abort=21f ldr1b r1, r4, cs, abort=21f @@ -169,7 +158,7 @@ str1b r0, r4, cs, abort=21f str1b r0, ip, cs, abort=21f - exit r4, pc + exit r4, UNWIND(fpreg,) pc 9: rsb ip, ip, #4 cmp ip, #2 @@ -189,13 +178,10 @@ ldr1w r1, lr, abort=21f beq 17f bgt 18f - UNWIND( .fnend ) .macro forward_copy_shift pull push - UNWIND( .fnstart ) - usave r4, lr @ still in first stmdb block subs r2, r2, #28 blt 14f @@ -205,12 +191,8 @@ CALGN( subcc r2, r2, ip ) CALGN( bcc 15f ) -11: stmfd sp!, {r5 - r9} - UNWIND( .fnend ) +11: stmfd sp!, {r5, r6, r8 - r10} - UNWIND( .fnstart ) - usave r4, lr - UNWIND( .save {r5 - r9} ) @ in new second stmfd block PLD( pld [r1, #0] ) PLD( subs r2, r2, #96 ) PLD( pld [r1, #28] ) @@ -219,35 +201,32 @@ PLD( pld [r1, #92] ) 12: PLD( pld [r1, #124] ) -13: ldr4w r1, r4, r5, r6, r7, abort=19f +13: ldr4w r1, r4, r5, r6, r8, abort=19f mov r3, lr, lspull #\pull subs r2, r2, #32 - ldr4w r1, r8, r9, ip, lr, abort=19f + ldr4w r1, r9, r10, ip, lr, abort=19f orr r3, r3, r4, lspush #\push mov r4, r4, lspull #\pull orr r4, r4, r5, lspush #\push mov r5, r5, lspull #\pull orr r5, r5, r6, lspush #\push mov r6, r6, lspull #\pull - orr r6, r6, r7, lspush #\push - mov r7, r7, lspull #\pull - orr r7, r7, r8, lspush #\push + orr r6, r6, r8, lspush #\push mov r8, r8, lspull #\pull orr r8, r8, r9, lspush #\push mov r9, r9, lspull #\pull - orr r9, r9, ip, lspush #\push + orr r9, r9, r10, lspush #\push + mov r10, r10, lspull #\pull + orr r10, r10, ip, lspush #\push mov ip, ip, lspull #\pull orr ip, ip, lr, lspush #\push - str8w r0, r3, r4, r5, r6, r7, r8, r9, ip, abort=19f + str8w r0, r3, r4, r5, r6, r8, r9, r10, ip, abort=19f bge 12b PLD( cmn r2, #96 ) PLD( bge 13b ) - ldmfd sp!, {r5 - r9} - UNWIND( .fnend ) @ end of the second stmfd block + ldmfd sp!, {r5, r6, r8 - r10} - UNWIND( .fnstart ) - usave r4, lr @ still in first stmdb block 14: ands ip, r2, #28 beq 16f @@ -262,7 +241,6 @@ 16: sub r1, r1, #(\push / 8) b 8b - UNWIND( .fnend ) .endm @@ -273,6 +251,7 @@ 18: forward_copy_shift pull=24 push=8 + UNWIND( .fnend ) /* * Abort preamble and completion macros. @@ -282,13 +261,13 @@ */ .macro copy_abort_preamble -19: ldmfd sp!, {r5 - r9} +19: ldmfd sp!, {r5, r6, r8 - r10} b 21f -20: ldmfd sp!, {r5 - r8} +20: ldmfd sp!, {r5, r6, r8, r9} 21: .endm .macro copy_abort_end - ldmfd sp!, {r4, pc} + ldmfd sp!, {r4, UNWIND(fpreg,) pc} .endm diff --git a/arch/arm/lib/copy_to_user.S b/arch/arm/lib/copy_to_user.S index 842ea5ede485..fac49e57cc0b 100644 --- a/arch/arm/lib/copy_to_user.S +++ b/arch/arm/lib/copy_to_user.S @@ -90,18 +90,15 @@ strusr \reg, \ptr, 1, \cond, abort=\abort .endm - .macro enter reg1 reg2 + .macro enter regs:vararg mov r3, #0 - stmdb sp!, {r0, r2, r3, \reg1, \reg2} +UNWIND( .save {r0, r2, r3, \regs} ) + stmdb sp!, {r0, r2, r3, \regs} .endm - .macro usave reg1 reg2 - UNWIND( .save {r0, r2, r3, \reg1, \reg2} ) - .endm - - .macro exit reg1 reg2 + .macro exit regs:vararg add sp, sp, #8 - ldmfd sp!, {r0, \reg1, \reg2} + ldmfd sp!, {r0, \regs} .endm .text diff --git a/arch/arm/lib/memcpy.S b/arch/arm/lib/memcpy.S index e4caf48c089f..90f2b645aa0d 100644 --- a/arch/arm/lib/memcpy.S +++ b/arch/arm/lib/memcpy.S @@ -42,16 +42,13 @@ strb\cond \reg, [\ptr], #1 .endm - .macro enter reg1 reg2 - stmdb sp!, {r0, \reg1, \reg2} + .macro enter regs:vararg +UNWIND( .save {r0, \regs} ) + stmdb sp!, {r0, \regs} .endm - .macro usave reg1 reg2 - UNWIND( .save {r0, \reg1, \reg2} ) - .endm - - .macro exit reg1 reg2 - ldmfd sp!, {r0, \reg1, \reg2} + .macro exit regs:vararg + ldmfd sp!, {r0, \regs} .endm .text From patchwork Mon Oct 18 14:16:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12566601 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 414C1C433EF for ; Mon, 18 Oct 2021 14:21:11 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 13A5960F24 for ; Mon, 18 Oct 2021 14:21:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 13A5960F24 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=V9rnHqcLVOq4DMjwixLazYfP4ajhrf560THC6RaBBx4=; b=qAvclP9Xh/6+y2 UaBROgxuac+deFmrhYUGXGNrTe8KIrUoWK0pOIjpUbUWP1JWP9aomPgBe9IZOi6Nll3jfHGxYXAUw Wa9RvAx8qAouHJdIBlZbsr9Df6ZBGuCr7eKCV7SgpSGzH9Dz+HJt6dC8kMnzIFguyec//DFRObE6U LJhB7PV3OBMJRo7krqTnsY6EQ/9il2ZcU1MTEf27e/WHKATX4HcpO0yKanlPYffcha9sXrz907sT0 C65WJj+m4CQ4RTCRRVi5Q0m3XxTo1t90a3QMB1hLYUjE6Dc3yNP7htS0SeTueK98rQKLFyhiIWbQb zmJ+mFBsELZcwqKEmK/A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mcTU3-00G3oH-BC; Mon, 18 Oct 2021 14:19:25 +0000 Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mcTRL-00G2kq-9Q for linux-arm-kernel@lists.infradead.org; Mon, 18 Oct 2021 14:16:37 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id 9239160F8F; Mon, 18 Oct 2021 14:16:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1634566594; bh=VphAfRFHNmHO/ff4IiJZ9wep4dsw82k4eDCDtPyt8mE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oXoQpFAXHeSZ3b3eZX7WzYZxJdeY0KYreFBg+vYzh6OlAYAcLYCNFkrphmHX0Hgir Nji+Aj8PJ2Bg/MAYtWM8C0vPM3S4RF3ljCxr6Q+9SUxui+5zbjS97zBe+skltN7XnN 6LDk2cxkk86F/C1Q6DumHUMPgonEwRkPq5BCbNPZY1LP/WXduAMWdO+c1ic7QhLsEU TVuyYcVFJFAspCxn+pKhGURpMQyZyKkTdQLcWHB+QQkPgp0rwi/HP/egkFRswtkXoZ A3/1w3+CV+D7b+zQFzKCHczEJZShv4hd6gpF9fyTFvf2Z2tk4Ojq/qPxKrhmwt8cHq RaoyhMmlb50Tg== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Russell King , Nicolas Pitre , Arnd Bergmann , Kees Cook , Keith Packard , Linus Walleij , Nick Desaulniers Subject: [PATCH v2 2/7] ARM: memmove: use frame pointer as unwind anchor Date: Mon, 18 Oct 2021 16:16:10 +0200 Message-Id: <20211018141615.682070-3-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211018141615.682070-1-ardb@kernel.org> References: <20211018141615.682070-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5955; h=from:subject; bh=VphAfRFHNmHO/ff4IiJZ9wep4dsw82k4eDCDtPyt8mE=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBhbYGkc+3GGSP0fabP0TGVxSNt2wnLm9a/dvJ5Pxz0 VMXmSDeJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYW2BpAAKCRDDTyI5ktmPJON3C/ 0XotTC3M+JigkbYJ4yf/4F7mVivu5e+u58xenUrAB3WGLudN31GX1UOIg1CKkJE3yMxJUg2M6zcb12 LflakFo4UcNxIGb1LxQymXF8qZCyHkGV/mHN3C+f7zpNqfGcXTg8HZLJdX+Ux4W/5prpQ5luQL9h80 eQD9NmG5LPZtEFCdfE8pBhPG2DdHjy0UJcVYNKanNhdEJM8iQFF1SObeA4dKiz9zOl8gKs2ieZJqpe tTXZsggDhFG8Xnna8muiY8DsD7rEzQPr7cM4nRXY3fhlhJirfS+ZLjZGrnKN0Xm6mOG0ukKtylQFP1 TyfIMegXPRcgMYHFcWTTOeDf7H4RfjsWN4qFoD4ZXcQYJ3HBFfT6pkdn6AAJrpBSqO9sgl7Jp5vzO6 n52w5qUJVxfXkLZri2jE9IUx/Rom3vyndwJZzRIz7GQAoG4V0Z6YwC1Xr+oKuRi0k2HnPdJ9MTPOZT oukVsYZHmeZQudCVY+EDTxhqjR4gSx7a84tS4JaZReSvE= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211018_071635_403844_06AF8539 X-CRM114-Status: GOOD ( 16.31 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The memmove routine is a bit unusual in the way it manages the stack pointer: depending on the execution path through the function, the SP assumes different values as different subsets of the register file are preserved and restored again. This is problematic when it comes to EHABI unwind info, as it is not instruction accurate, and does not allow tracking the SP value as it changes. Commit 207a6cb06990c ("ARM: 8224/1: Add unwinding support for memmove function") addressed this by carving up the function in different chunks as far as the unwinder is concerned, and keeping a set of unwind directives for each of them, each corresponding with the state of the stack pointer during execution of the chunk in question. This not only duplicates unwind info unnecessarily, but it also complicates unwinding the stack upon overflow. Instead, let's do what the compiler does when the SP is updated halfway through a function, which is to use a frame pointer and emit the appropriate unwind directives to communicate this to the unwinder. Note that Thumb-2 uses R7 for this, while ARM uses R11 aka FP. So let's avoid touching R7 in the body of the function, so that Thumb-2 can use it as the frame pointer. R11 was not modified in the first place. Signed-off-by: Ard Biesheuvel Tested-by: Keith Packard --- arch/arm/lib/memmove.S | 60 +++++++------------- 1 file changed, 20 insertions(+), 40 deletions(-) diff --git a/arch/arm/lib/memmove.S b/arch/arm/lib/memmove.S index 6fecc12a1f51..6410554039fd 100644 --- a/arch/arm/lib/memmove.S +++ b/arch/arm/lib/memmove.S @@ -31,12 +31,13 @@ WEAK(memmove) subs ip, r0, r1 cmphi r2, ip bls __memcpy - - stmfd sp!, {r0, r4, lr} UNWIND( .fnend ) UNWIND( .fnstart ) - UNWIND( .save {r0, r4, lr} ) @ in first stmfd block + UNWIND( .save {r0, r4, fpreg, lr} ) + stmfd sp!, {r0, r4, UNWIND(fpreg,) lr} + UNWIND( .setfp fpreg, sp ) + UNWIND( mov fpreg, sp ) add r1, r1, r2 add r0, r0, r2 subs r2, r2, #4 @@ -48,12 +49,7 @@ WEAK(memmove) bne 10f 1: subs r2, r2, #(28) - stmfd sp!, {r5 - r8} - UNWIND( .fnend ) - - UNWIND( .fnstart ) - UNWIND( .save {r0, r4, lr} ) - UNWIND( .save {r5 - r8} ) @ in second stmfd block + stmfd sp!, {r5, r6, r8, r9} blt 5f CALGN( ands ip, r0, #31 ) @@ -72,9 +68,9 @@ WEAK(memmove) PLD( pld [r1, #-96] ) 3: PLD( pld [r1, #-128] ) -4: ldmdb r1!, {r3, r4, r5, r6, r7, r8, ip, lr} +4: ldmdb r1!, {r3, r4, r5, r6, r8, r9, ip, lr} subs r2, r2, #32 - stmdb r0!, {r3, r4, r5, r6, r7, r8, ip, lr} + stmdb r0!, {r3, r4, r5, r6, r8, r9, ip, lr} bge 3b PLD( cmn r2, #96 ) PLD( bge 4b ) @@ -88,8 +84,8 @@ WEAK(memmove) W(ldr) r4, [r1, #-4]! W(ldr) r5, [r1, #-4]! W(ldr) r6, [r1, #-4]! - W(ldr) r7, [r1, #-4]! W(ldr) r8, [r1, #-4]! + W(ldr) r9, [r1, #-4]! W(ldr) lr, [r1, #-4]! add pc, pc, ip @@ -99,17 +95,13 @@ WEAK(memmove) W(str) r4, [r0, #-4]! W(str) r5, [r0, #-4]! W(str) r6, [r0, #-4]! - W(str) r7, [r0, #-4]! W(str) r8, [r0, #-4]! + W(str) r9, [r0, #-4]! W(str) lr, [r0, #-4]! CALGN( bcs 2b ) -7: ldmfd sp!, {r5 - r8} - UNWIND( .fnend ) @ end of second stmfd block - - UNWIND( .fnstart ) - UNWIND( .save {r0, r4, lr} ) @ still in first stmfd block +7: ldmfd sp!, {r5, r6, r8, r9} 8: movs r2, r2, lsl #31 ldrbne r3, [r1, #-1]! @@ -118,7 +110,7 @@ WEAK(memmove) strbne r3, [r0, #-1]! strbcs r4, [r0, #-1]! strbcs ip, [r0, #-1] - ldmfd sp!, {r0, r4, pc} + ldmfd sp!, {r0, r4, UNWIND(fpreg,) pc} 9: cmp ip, #2 ldrbgt r3, [r1, #-1]! @@ -137,13 +129,10 @@ WEAK(memmove) ldr r3, [r1, #0] beq 17f blt 18f - UNWIND( .fnend ) .macro backward_copy_shift push pull - UNWIND( .fnstart ) - UNWIND( .save {r0, r4, lr} ) @ still in first stmfd block subs r2, r2, #28 blt 14f @@ -152,12 +141,7 @@ WEAK(memmove) CALGN( subcc r2, r2, ip ) CALGN( bcc 15f ) -11: stmfd sp!, {r5 - r9} - UNWIND( .fnend ) - - UNWIND( .fnstart ) - UNWIND( .save {r0, r4, lr} ) - UNWIND( .save {r5 - r9} ) @ in new second stmfd block +11: stmfd sp!, {r5, r6, r8 - r10} PLD( pld [r1, #-4] ) PLD( subs r2, r2, #96 ) @@ -167,35 +151,31 @@ WEAK(memmove) PLD( pld [r1, #-96] ) 12: PLD( pld [r1, #-128] ) -13: ldmdb r1!, {r7, r8, r9, ip} +13: ldmdb r1!, {r8, r9, r10, ip} mov lr, r3, lspush #\push subs r2, r2, #32 ldmdb r1!, {r3, r4, r5, r6} orr lr, lr, ip, lspull #\pull mov ip, ip, lspush #\push - orr ip, ip, r9, lspull #\pull + orr ip, ip, r10, lspull #\pull + mov r10, r10, lspush #\push + orr r10, r10, r9, lspull #\pull mov r9, r9, lspush #\push orr r9, r9, r8, lspull #\pull mov r8, r8, lspush #\push - orr r8, r8, r7, lspull #\pull - mov r7, r7, lspush #\push - orr r7, r7, r6, lspull #\pull + orr r8, r8, r6, lspull #\pull mov r6, r6, lspush #\push orr r6, r6, r5, lspull #\pull mov r5, r5, lspush #\push orr r5, r5, r4, lspull #\pull mov r4, r4, lspush #\push orr r4, r4, r3, lspull #\pull - stmdb r0!, {r4 - r9, ip, lr} + stmdb r0!, {r4 - r6, r8 - r10, ip, lr} bge 12b PLD( cmn r2, #96 ) PLD( bge 13b ) - ldmfd sp!, {r5 - r9} - UNWIND( .fnend ) @ end of the second stmfd block - - UNWIND( .fnstart ) - UNWIND( .save {r0, r4, lr} ) @ still in first stmfd block + ldmfd sp!, {r5, r6, r8 - r10} 14: ands ip, r2, #28 beq 16f @@ -211,7 +191,6 @@ WEAK(memmove) 16: add r1, r1, #(\pull / 8) b 8b - UNWIND( .fnend ) .endm @@ -222,5 +201,6 @@ WEAK(memmove) 18: backward_copy_shift push=24 pull=8 + UNWIND( .fnend ) ENDPROC(memmove) ENDPROC(__memmove) From patchwork Mon Oct 18 14:16:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12566603 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29D4CC433EF for ; Mon, 18 Oct 2021 14:22:02 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E59ED60F25 for ; Mon, 18 Oct 2021 14:22:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org E59ED60F25 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=1HNd+oBeddcrESNjJXmw41pgwFqyNg5qgY/JtViMRMY=; b=adeSn+68P2Tbtt 9MBki4RDVtzf65kFZ8wYKTVXQyDTigjDh7QskEeQ6q02GCx4lFKDyWHrKXocWLIyWajzt3KWjqw00 2rBLL3ATgGGNXiE29/PDaavQgYeboeTVly8tKTfik9bjYeyRezzby+P+tpaa7It8QQkZ97FX9b1zG H/eJqqUPw0QI58ZT0zWqcXrYLoB6EL5bDpvd/8BqJrDZg4bMIBZB5O7NtCqX8mg0xokwV075/2dz7 9isUQa0vm5qHMh7vssQUv0swUnJlTkyYcLRHPf6ITqiMmwKTn11f54acclT2iSU1FmuX8Ze3spqBr iSiHXnk9Gvs6fNwJEzxg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mcTV4-00G4C1-1I; Mon, 18 Oct 2021 14:20:26 +0000 Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mcTRN-00G2mO-TP for linux-arm-kernel@lists.infradead.org; Mon, 18 Oct 2021 14:16:39 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id 318AB60F9F; Mon, 18 Oct 2021 14:16:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1634566597; bh=w5oIXXckLNTqRaq5jRlVsCI6cCEyZWsLVvGoUPHtdmY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GXWhnCu5vYIicKArC8TpjypuiTuE3ebfqzODuFFBsgZEY1oEbBnLEzFs/Tj27fZ5D aPUeU/Rq60gTCAHya5T7XMxhq0J8OwyJvVoq9ecHXw2x5qdlf1G1H4w1UiC8mjVZlz K7Ug/pEazySOc2rz25iaZlbtSWJKoOccMGwpB0o7SzxUeTf06C5xfuMogOXW+NaunP EwK0w9WRaYTdSwl1cT1yeaLBl1b2hQthhENLY8pUTiEDhCd4py/uO3qPLAcJXke/ol qT+bkwIhQHT/PW9Wt3+XkKC85myv53ZLbqGJLqalprKdHhaDC3lu7qx+fBQD1alBCA GhglE4U18aR7Q== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Russell King , Nicolas Pitre , Arnd Bergmann , Kees Cook , Keith Packard , Linus Walleij , Nick Desaulniers Subject: [PATCH v2 3/7] ARM: memset: clean up unwind annotations Date: Mon, 18 Oct 2021 16:16:11 +0200 Message-Id: <20211018141615.682070-4-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211018141615.682070-1-ardb@kernel.org> References: <20211018141615.682070-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1788; h=from:subject; bh=w5oIXXckLNTqRaq5jRlVsCI6cCEyZWsLVvGoUPHtdmY=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBhbYGmAB44pDxM+fIv6AeCNOrvzBdm0b/8CR2NM7cr 3HsPhAGJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYW2BpgAKCRDDTyI5ktmPJEOoDA DD89RQA0n6sDwsH7rl9qjvieNAHgyn9pMPlnxGdF3EULNLW/nSonQ+KfCB1A8IPfj/exvchu7zViZY yhQjS8el0U/xr+6bsTupAQYEFBbE173id+IJUcSPKM+UJ2fDF4L8LDw/0twdYnl5dbgfDPxKQryOud GtWbakD3HW0tw2ThosVBDlrHOQS7dlnRrrlEtKJ4TD1xLHJAV7uKxGefowpAcekBw0vWKaMmvMyG7i zzaYosJRDVDo/anSoY5f3rUQ8zcVFIICYxsOAC7r+ZESgOoc0+XUbATmQxyo12zsadr3hk+LvJ8JFG JGYeWkz3c6SipHqahA6XJnNqhoKRJaOpDCY36b3wBFQPOfoOz3We6/WGABNECJfs9bo+saw521P6Zw f0Zw5OUwTf7/AZgCMV3RGd7lUzePAo6T/CbIgnxQA3YR7pqZ7gJQaOMhk1sNuiA/nAJ0ZLViwlcWwz 0GC7dmRbJXTIcZDINR8vRESL5pqrGMAQl9qtAcOEIfDbI= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211018_071638_050198_3F749710 X-CRM114-Status: GOOD ( 13.67 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The memset implementation carves up the code in different sections, each covered with their own unwind info. In this case, it is done in a way similar to how the compiler might do it, to disambiguate between parts where the return address is in LR and the SP is unmodified, and parts where a stack frame is live, and the unwinder needs to know the size of the stack frame and the location of the return address within it. Only the placement of the unwind directives is slightly odd: the stack pushes are placed in the wrong sections, which may confuse the unwinder when attempting to unwind with PC pointing at the stack push in question. So let's fix this up, by reordering the directives and instructions as appropriate. Signed-off-by: Ard Biesheuvel Tested-by: Keith Packard --- arch/arm/lib/memset.S | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/arch/arm/lib/memset.S b/arch/arm/lib/memset.S index 9817cb258c1a..d71ab61430b2 100644 --- a/arch/arm/lib/memset.S +++ b/arch/arm/lib/memset.S @@ -28,16 +28,16 @@ UNWIND( .fnstart ) mov r3, r1 7: cmp r2, #16 blt 4f +UNWIND( .fnend ) #if ! CALGN(1)+0 /* * We need 2 extra registers for this loop - use r8 and the LR */ - stmfd sp!, {r8, lr} -UNWIND( .fnend ) UNWIND( .fnstart ) UNWIND( .save {r8, lr} ) + stmfd sp!, {r8, lr} mov r8, r1 mov lr, r3 @@ -66,10 +66,9 @@ UNWIND( .fnend ) * whole cache lines at once. */ - stmfd sp!, {r4-r8, lr} -UNWIND( .fnend ) UNWIND( .fnstart ) UNWIND( .save {r4-r8, lr} ) + stmfd sp!, {r4-r8, lr} mov r4, r1 mov r5, r3 mov r6, r1 From patchwork Mon Oct 18 14:16:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12566607 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 412AEC433F5 for ; Mon, 18 Oct 2021 14:23:07 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0C24360F46 for ; Mon, 18 Oct 2021 14:23:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 0C24360F46 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=8PxJMuzVDh2Db7k+kZ1Yg3ulTHqExrf8MPqljA8uqG8=; b=q24PQ3E3HiKlCt hCRmd5rUU1Q9eESsc0C1FUbz5hvuWQNu0IjuvtBa0WVZ9r4NCQCY/+sFG2/KIvy22TzEfYAgfgq4p Sag1AflqdsFWdOT/R+aB+vySxkmfZLWHF/FTjHyxKUfp6RGIgGg92qoCNDgSWQeS2ZaehQ7tQSGBd oa8OB+7oR3+lXZE6qH+HarpgknSRZVhMqEyXr70ebC7xaXtRuFk88XqDftRUlt4K7PbD7ykTirzKD GxAcc+fR1MCNBsD/580dWwzItySd+/coGQL/CRPR3Tq06esuX6NDqmarvBvbwwucHloyd8SZ2a4v3 jZy6uR3rYFK+h13gvk+Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mcTW7-00G4fU-KC; Mon, 18 Oct 2021 14:21:32 +0000 Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mcTRQ-00G2nN-Sn for linux-arm-kernel@lists.infradead.org; Mon, 18 Oct 2021 14:16:42 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id C4B6060FC2; Mon, 18 Oct 2021 14:16:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1634566599; bh=G8bl/jS0UF4ofWVfxX1cIJJ06WAI6ETjTnMacIEZcB4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XaQB3J0Muivcn8tTtUal0wmVFqC31pC2+mJob5CK1u/SJae5qRskK+s4pManjFUc9 euqSE0DSGqpuLIDKfuQWO5o4vICnel6SjVsv2JTfM81F5e5QYOBxEpl5t6WVhW3OXo CyuwUhU+6JSzsJb41YlsVuwxRjRtDPSBg0VbozJZ/YN17DZGBuokpuhFo8xAfKWDzm K5WMTVKAwQ1qUrSsfHwOqlpcbYBhd4HYtXyx/tOhQJDrzwJJeyJ7+a6aM32q2Z2gFS qRb7jHHMK8VX6dXVs21xvjCjf5EcV2fUd4moRhzyDlOu5GjiBzechMGVvU1ba86xaW nf6Zwi37d+eOg== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Russell King , Nicolas Pitre , Arnd Bergmann , Kees Cook , Keith Packard , Linus Walleij , Nick Desaulniers Subject: [PATCH v2 4/7] ARM: unwind: disregard unwind info before stack frame is set up Date: Mon, 18 Oct 2021 16:16:12 +0200 Message-Id: <20211018141615.682070-5-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211018141615.682070-1-ardb@kernel.org> References: <20211018141615.682070-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2389; h=from:subject; bh=G8bl/jS0UF4ofWVfxX1cIJJ06WAI6ETjTnMacIEZcB4=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBhbYGouEi30aIbAnRchjXXGvZ89mMKIIQd149EMa+6 bmr6GyKJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYW2BqAAKCRDDTyI5ktmPJN2pDA CLfK7SOKZkG1fABk1of33RE/+72XRuljrWH9LMMpycdxO40B0SBMbFnmpthb4dmTu/eNriSUBHGOub VRMg/x8yLKvLSaZ4z/6dhWCAJfJ+YBjPBfREJYHwarPtxF76QZjbgKZDRoIRUYlAR459hAXNALHV/0 H31c0oRzZxeYhOlWtydj17vnGitz40PSHejCn70NiPW3/qrfREfaqfIEUIONdy44oqyw58eQUFfdR6 G1Z1qcqzJHi9y0K9jIc56CL6oeWf4pwsvtnCT5M6h2t+RLYOQhvygbWHgNYYna7lHzE1UpE4yr0oV5 wm9JBlNM2n/iRLudjhw4GQoFYO/V1fXwdmFuKLSZ08GjwcTD8MjzXpj5+F//GJgSLk5OkvAmD139ck uK+ap1K2UdNhqfRCVbdMzINLeF1IMW2ZdiZCJRhLUmbqbqlZ8RtScMlINk0WWUMjnDUWbzCqxCp5v0 4EF26fGygnb0QSO7u7wMtORx9hn8Vlc4SN0oK0ZT+0ewU= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211018_071641_071445_C0E208BF X-CRM114-Status: GOOD ( 18.59 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When unwinding the stack from a stack overflow, we are likely to start from a stack push instruction, given that this is the most common way to grow the stack for compiler emitted code. This push instruction rarely appears anywhere else than at offset 0x0 of the function, and if it doesn't, the compiler tends to split up the unwind annotations, given that the stack frame layout is apparently not the same throughout the function. This means that, in the general case, if the frame's PC points at the first instruction covered by a certain unwind entry, there is no way the stack frame that the unwind entry describes could have been created yet, and so we are still on the stack frame of the caller in that case. So treat this as a special case, and return with the new PC taken from the frame's LR, without applying the unwind transformations to the virtual register set. This permits us to unwind the call stack on stack overflow when the overflow was caused by a stack push on function entry. Signed-off-by: Ard Biesheuvel Tested-by: Keith Packard --- arch/arm/kernel/unwind.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/arch/arm/kernel/unwind.c b/arch/arm/kernel/unwind.c index b7a6141c342f..e8d729975f12 100644 --- a/arch/arm/kernel/unwind.c +++ b/arch/arm/kernel/unwind.c @@ -411,7 +411,21 @@ int unwind_frame(struct stackframe *frame) if (idx->insn == 1) /* can't unwind */ return -URC_FAILURE; - else if ((idx->insn & 0x80000000) == 0) + else if (frame->pc == prel31_to_addr(&idx->addr_offset)) { + /* + * Unwinding is tricky when we're halfway through the prologue, + * since the stack frame that the unwinder expects may not be + * fully set up yet. However, one thing we do know for sure is + * that if we are unwinding from the very first instruction of + * a function, we are still effectively in the stack frame of + * the caller, and the unwind info has no relevance yet. + */ + if (frame->pc == frame->lr) + return -URC_FAILURE; + frame->sp_low = frame->sp; + frame->pc = frame->lr; + return URC_OK; + } else if ((idx->insn & 0x80000000) == 0) /* prel31 to the unwind table */ ctrl.insn = (unsigned long *)prel31_to_addr(&idx->insn); else if ((idx->insn & 0xff000000) == 0x80000000) From patchwork Mon Oct 18 14:16:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12566609 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0BB6C433F5 for ; Mon, 18 Oct 2021 14:23:44 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AB8D360F46 for ; Mon, 18 Oct 2021 14:23:44 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org AB8D360F46 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=SFBU3L5qFtKCe70JEKqShC6fezAh8rnCKleOHqT2r5I=; b=mqzj1quVtyDp7q ydweBLG3Jesve+1cdo0jDXrRrY4TcHpLGT3+1/lxmXIcfLFhGuJrqV8K3ZuVrTKpffFsMs/Ll1xd6 hIq8Y2w6if3vmOftlddguTB7d+11HMB5/4EuqmDEwCVV1VKBQqma4XZgwg071FaG1q0DpV6Eyj0pd pUMAsakOZ8tjFpvgx6kkgBAjDPcKVSeMTeqWdadRGLoQe06DdsGIx3/L9QdJi/NlM+LiJortyes3M eHhornGX5KAOtNiEMM2qPWMGStsj5EcFg2A5DXGqnWCNMn4IyQx/h/8h2DMOdDGDOafUFLjWN+vdI 6cMJAs1t+TijACNW8mlA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mcTWe-00G4us-P6; Mon, 18 Oct 2021 14:22:05 +0000 Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mcTRT-00G2oJ-3d for linux-arm-kernel@lists.infradead.org; Mon, 18 Oct 2021 14:16:44 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id 63E7560F8F; Mon, 18 Oct 2021 14:16:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1634566602; bh=vEkn4MAZz7C5At20781VdUYCMQ8lqEM/zhlV+YQbgOE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=uAUe8WGGAEytYzg2RDtzvVCvzuwPy0YGptuGhrlsCGD90u7JxVeBxv/quTt480WoQ 063zccVC3jcNNTVLYLGb9DdCuaR6syr8GfMDWzCTL4utdqcWpIXWSnOIacAQEVMXi/ MYko+Oj1hFzR28fSL2Rd9BL0EA/PC6byrGuFSX69lietGhb6zSzV9g+avRjFouc3HS HBZQRIvxbzwmihT1k75jJfEOmusLqtvMr6lywJ50PUXG43R6ysY6z1wYTrcrLRhLwo IGDTdD1EHAHBpeDS6c24S29fH1c2h98X6ykgYF/Koatx5+mmxZ5UEfmA5/cczOX1ZU qHbE7zboIygHA== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Russell King , Nicolas Pitre , Arnd Bergmann , Kees Cook , Keith Packard , Linus Walleij , Nick Desaulniers Subject: [PATCH v2 5/7] ARM: switch_to: clean up Thumb2 code path Date: Mon, 18 Oct 2021 16:16:13 +0200 Message-Id: <20211018141615.682070-6-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211018141615.682070-1-ardb@kernel.org> References: <20211018141615.682070-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2068; h=from:subject; bh=vEkn4MAZz7C5At20781VdUYCMQ8lqEM/zhlV+YQbgOE=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBhbYGqikGsU9abonAw7uexj8UPiwITP2NBfibJqghY IeeCwpyJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYW2BqgAKCRDDTyI5ktmPJNLmC/ 49MpbTxwiytRUrnnR6bQWpYPlnNumxQsVRGB6LvnY3NwzBp9Du8Y4SLCDd04HoUSXMutq4kLjONqCp 8EHvzoiRgXID2mkE4orCFLKbkVU4LG8ryiq4u0YiALK3RmclqllyMHd9VtgaPN4Cd4hWfa4d9RMYdo ff4gpkk2ATfDG2rUR4wR8VajSrQzDgCfaZLq8ky/bfHy4m1IO9eWMwrH3NOL2WzlFnMg9+i+JZhwFb 7SH2/d69yXndP0Ji5dGSMbu7YwAPzhH7g/Elnrgoi1UPggVLQOErPO7ntgI9SIfu9J3wPt5IVFSti0 s2N+vp5mgf1OzOoZ/l4E87T9S+5XKCtNI9lUzpnjSiJheMpDicdPy7xLXMbib7y+gA1FTluk4ohsto h2nKxKgUWKL/0GJNcPOxMkjosKQPRhFjGyH2msP/Bun+N5ohHAfz0x0xp9aM1VveueeFr9n1T+G828 Ymt/Snbafe4rKIq0Lz7NufYznU72PvaP2ma7GE5OnJGDM= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211018_071643_256139_387FD0C1 X-CRM114-Status: GOOD ( 12.57 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The load-multiple instruction that essentially performs the switch_to operation in ARM mode, by loading all callee save registers as well the stack pointer and the program counter, is split into 3 separate loads for Thumb-2, with the IP register used as a temporary to capture the value of R4 before it gets overwritten. We can clean this up a bit, by sticking with a single LDMIA instruction, but one that pops SP and PC into IP and LR, respectively, and by using ordinary move register and branch instructions to get those values into SP and PC. This also allows us to move the set_current call closer to the assignment of SP, reducing the window where those are mutually out of sync. This is especially relevant for CONFIG_VMAP_STACK, which is being introduced in a subsequent patch, where we need to issue a load that might fault from the new stack while running from the old one, to ensure that stale PMD entries in the VMALLOC space are synced up. Signed-off-by: Ard Biesheuvel Tested-by: Keith Packard --- arch/arm/kernel/entry-armv.S | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/arch/arm/kernel/entry-armv.S b/arch/arm/kernel/entry-armv.S index 2a426bd27a69..18504d7b4d15 100644 --- a/arch/arm/kernel/entry-armv.S +++ b/arch/arm/kernel/entry-armv.S @@ -824,13 +824,17 @@ ENTRY(__switch_to) #if defined(CONFIG_STACKPROTECTOR) && !defined(CONFIG_SMP) str r7, [r8] #endif - THUMB( mov ip, r4 ) mov r0, r5 +#if !defined(CONFIG_THUMB2_KERNEL) set_current r7 - ARM( ldmia r4, {r4 - sl, fp, sp, pc} ) @ Load all regs saved previously - THUMB( ldmia ip!, {r4 - sl, fp} ) @ Load all regs saved previously - THUMB( ldr sp, [ip], #4 ) - THUMB( ldr pc, [ip] ) + ldmia r4, {r4 - sl, fp, sp, pc} @ Load all regs saved previously +#else + mov r1, r7 + ldmia r4, {r4 - sl, fp, ip, lr} @ Load all regs saved previously + set_current r1 + mov sp, ip + ret lr +#endif UNWIND(.fnend ) ENDPROC(__switch_to) From patchwork Mon Oct 18 14:16:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12566611 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 294E3C433F5 for ; Mon, 18 Oct 2021 14:24:34 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E67FF60F59 for ; Mon, 18 Oct 2021 14:24:33 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org E67FF60F59 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=USFguO5UsuEWgDFTf3x4h2XuPnJoDUkWMKgaSMrvPo0=; b=xPsLUyfBU4dp44 PwMSkovigbl5dblftWoqjAEGhWAVgbPSZB2eGxDM3LzJBXkVYAJZTDciZ8xl522dRZp+JWBvDX0fk Bv3fgQkTKJjBCSYnsdIvaVq+48JuxUHSsguEWsVd6Z6x+EoqNF/Jc/uJ41MF072BU5Va3yDdBS8WQ slBhkcdPKHg13s8Wkp5FDkWNKfc6kIZ1n3Ej8i6NcdJOibSWW9JJ8StSa/tOY4E4tgLxasrIop50T rsNmOjttIZUmiQzfScDgxxlQ7neBmmR5qk9KjV3TXNf/I7r+bhXRHNep0WlULDG0pd2qyrR9F0Jv5 6qu6JWpwpa8je4Z2I70w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mcTXO-00G5DP-91; Mon, 18 Oct 2021 14:22:51 +0000 Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mcTRV-00G2pB-Nc for linux-arm-kernel@lists.infradead.org; Mon, 18 Oct 2021 14:16:47 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id 03BBD60F9F; Mon, 18 Oct 2021 14:16:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1634566605; bh=g6fr+qyUJ8Y2ckC9WGQU6bKRli7SF7LlHNWi9L4Pqhs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Auvjhfm2cQ+lFw0uQIHjxSYy8esQ20UYksCUthXyDHdIrHhuem5Nek7vtEkhzn5Hf fmxFMml3bLosfz46nyW0b0XL4kpuemuZwfUtKNJ/tQ/0gh/Qi3adL8YU3OCJjZfPrv lmRgawzZw32RA4GsMThArKnZtqYyv3IfAhSHyMuSIhdUaoR4LIHFH/2hrhPRE+yZVH Gy1KzcC6CI74FTQyLFIIAW80K8btnnV4UhltSFlTgI9ZeJal52Yxc0u2TYavAt6wxL e+OQu7s7xjLEWWxrbg+HXNjZutHjbb3J/pyXtvdb8pOATcWjFwnZy16aJWm9YjyidK EoELnngLeAYpg== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Russell King , Nicolas Pitre , Arnd Bergmann , Kees Cook , Keith Packard , Linus Walleij , Nick Desaulniers Subject: [PATCH v2 6/7] ARM: entry: rework stack realignment code in svc_entry Date: Mon, 18 Oct 2021 16:16:14 +0200 Message-Id: <20211018141615.682070-7-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211018141615.682070-1-ardb@kernel.org> References: <20211018141615.682070-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3193; h=from:subject; bh=g6fr+qyUJ8Y2ckC9WGQU6bKRli7SF7LlHNWi9L4Pqhs=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBhbYGsZvLrI3z/ecDbWUoWayra+wfFaj9UpmG5wamF Wy8p9kSJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYW2BrAAKCRDDTyI5ktmPJOOcDA ChCFtV5g19wKfMAwkljjd+RQKklk5ivD5wmQYHHnRfxl/eawvzfnViNBSBPa7gNK58T/taMdhwdouQ gPMDRXEDLh7urv6LXPyjkuuD+gWThliTw0D18CbLWChSXzUm6X+FZEO0vP56ZXCnIOv0c3nmd7hoF4 oVTSwQR4dY1ZLfvfJjt+5/o+yRX5ebsKexBFLjJQM4k1bkkL2DAq30jnumFbyiALKCNswcK83X2z3i BnFW7t+FQ/Y7u2WPrqXhoXiRBU+XbIKmr9B7VT9Nm5b0vW/n3ZTzZCmXenclfGxacEWoVpb+zIJx+H 1PZI8veX15JeuhktonKe3seznjrAZf45phP9QzHwsTnuCqeKnJ57T19ZZviNgwfHxHJA5wFmNVS7xY JexY9tvOiSn5I9ZysMi4ry6DbL6VnexoNeZWEiFVYObkewxyK4KlEkayRwS76r2U9EnGTbZDsOb4yP oLKlGILSgdYLEIsrCmgvV0IoeOzC/Ntgy7LD0jyf5DPGc= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211018_071645_868441_20FA992C X-CRM114-Status: GOOD ( 16.99 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The original Thumb-2 enablement patches updated the stack realignment code in svc_entry to work around the lack of a STMIB instruction in Thumb-2, by subtracting 4 from the frame size, inverting the sense of the misaligment check, and changing to a STMIA instruction and a final stack push of a 4 byte quantity that results in the stack becoming aligned at the end of the sequence. It also pushes and pops R0 to the stack in order to have a temp register that Thumb-2 allows in general purpose ALU instructions, as TST using SP is not permitted. Both are a bit problematic for vmap'ed stacks, as using the stack is only permitted after we decide that we did not overflow the stack, or have already switched to the overflow stack. As for the alignment check: the current approach creates a corner case where, if the initial SUB of SP ends up right at the start of the stack, we will end up subtracting another 8 bytes and overflowing it. This means we would need to add the overflow check *after* the SUB that deliberately misaligns the stack. However, this would require us to keep local state (i.e., whether we performed the subtract or not) across the overflow check, but without any GPRs or stack available. So let's switch to an approach where we don't use the stack, and where the alignment check of the stack pointer occurs in the usual way, as this is guaranteed not to result in overflow. This means we will be able to do the overflow check first. Signed-off-by: Ard Biesheuvel Acked-by: Nicolas Pitre --- arch/arm/kernel/entry-armv.S | 25 +++++++++++--------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/arch/arm/kernel/entry-armv.S b/arch/arm/kernel/entry-armv.S index 18504d7b4d15..80afab1c344b 100644 --- a/arch/arm/kernel/entry-armv.S +++ b/arch/arm/kernel/entry-armv.S @@ -192,24 +192,27 @@ ENDPROC(__und_invalid) .macro svc_entry, stack_hole=0, trace=1, uaccess=1 UNWIND(.fnstart ) UNWIND(.save {r0 - pc} ) - sub sp, sp, #(SVC_REGS_SIZE + \stack_hole - 4) + sub sp, sp, #(SVC_REGS_SIZE + \stack_hole) #ifdef CONFIG_THUMB2_KERNEL - SPFIX( str r0, [sp] ) @ temporarily saved - SPFIX( mov r0, sp ) - SPFIX( tst r0, #4 ) @ test original stack alignment - SPFIX( ldr r0, [sp] ) @ restored + add sp, r0 @ get SP in a GPR without + sub r0, sp, r0 @ using a temp register + tst r0, #4 @ test stack pointer alignment + sub r0, sp, r0 @ restore original R0 + sub sp, r0 @ restore original SP #else SPFIX( tst sp, #4 ) #endif - SPFIX( subeq sp, sp, #4 ) - stmia sp, {r1 - r12} + SPFIX( subne sp, sp, #4 ) + + ARM( stmib sp, {r1 - r12} ) + THUMB( stmia sp, {r0 - r12} ) @ No STMIB in Thumb-2 ldmia r0, {r3 - r5} - add r7, sp, #S_SP - 4 @ here for interlock avoidance + add r7, sp, #S_SP @ here for interlock avoidance mov r6, #-1 @ "" "" "" "" - add r2, sp, #(SVC_REGS_SIZE + \stack_hole - 4) - SPFIX( addeq r2, r2, #4 ) - str r3, [sp, #-4]! @ save the "real" r0 copied + add r2, sp, #(SVC_REGS_SIZE + \stack_hole) + SPFIX( addne r2, r2, #4 ) + str r3, [sp] @ save the "real" r0 copied @ from the exception stack mov r3, lr From patchwork Mon Oct 18 14:16:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12566613 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CBFE5C433EF for ; Mon, 18 Oct 2021 14:25:27 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 957DD60F46 for ; Mon, 18 Oct 2021 14:25:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 957DD60F46 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Vrz5mhAwo70EaM7PwnSjinKg1NgGpJhDv2z2ybTTph8=; b=crVGY2WRTIna83 XSxY8C97xtnePskYirmAm5SK3J8RAi3VF7rnxRDCxC4ppyz5Jjy8KS0WVKfKOF6Jlnvkxg2gDF+V2 6EcT5EO9IRSBCeKlfIR9j5kO74r9ZxO6bavWhSTIEx/Ye5jqnsPRmhPaCjWNjAu68U/IYLbFQMKSf lTrokUG+Nco1ufi3YZtZqseIQ7OuEqQIwtDc2leJM+SE/PskXlIP6rVDn60spPXWXGJfdOFw4JVpA FSyAfviPyXBgAI30ddS2fdrJyW/XnPzTE2wJtrqgU4CIQUHv9TYu9SdwmEzIT2C93Zv+iKplVd3aA k7pRFRBG7yo6oXX4g32w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mcTY4-00G5Uz-PK; Mon, 18 Oct 2021 14:23:33 +0000 Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mcTRY-00G2qR-D3 for linux-arm-kernel@lists.infradead.org; Mon, 18 Oct 2021 14:16:51 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id 98D1660FD9; Mon, 18 Oct 2021 14:16:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1634566607; bh=w8ONXJ4YgpPBywnMXN/BgNuGK4m4vFEcdaXT3wR0qF0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=sHlK6TDW582IANh2LnGllu1x1CmdmuY3xTQWxMmSY3LVtRuXJX8SP94foJAvz3x6t KZuiX6eLdg+043Tle/pInEipU+uFiH5G42wkbmGjUtchQWCUuicIpoGUfK+RdZ2Lsy VS7aZcK2Sh1xd4EGS9u2sO/aqVtygAjILiJ7tvxzaDMycbytNijTSxMn7ZyWqKncei ij0Kw5DYv7G8s+LYUdavHymuy2iRCXZ7iwBJDtFZxEIiihFYbM8hXlAHdLFnYD+zF6 Ub/qEJ2+HUreqxZKAfjonRrpJIQmpfV8Gum3aT/oJRurjJs8vPPgEuhUyUZuO0SF+I 0vwxK23XzdAbg== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Russell King , Nicolas Pitre , Arnd Bergmann , Kees Cook , Keith Packard , Linus Walleij , Nick Desaulniers Subject: [PATCH v2 7/7] ARM: implement support for vmap'ed stacks Date: Mon, 18 Oct 2021 16:16:15 +0200 Message-Id: <20211018141615.682070-8-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211018141615.682070-1-ardb@kernel.org> References: <20211018141615.682070-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=14301; h=from:subject; bh=w8ONXJ4YgpPBywnMXN/BgNuGK4m4vFEcdaXT3wR0qF0=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBhbYGuzzlcitfXVUztJVPsXmQocR6U9/yUBvRoFUUZ to/+JTWJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYW2BrgAKCRDDTyI5ktmPJFVpDA CQHFMIbn579QyOnO2mhjB8QSsJUVMyzgc6i1U+FCxQZDkVlri71yS7bmbYIe64wuoVKqHcjS/leeUb 10n5iKBhegM2Vb40cdbjD9Kdz0rCXUvHVr9nXmp8WbjZinTfLZsBNSqxmfKke735MlK3yZJFnVLQ9Q M8ebmOKiayn0V89sV41GP+24s2/wOXiYSeiYNpW8Zt0XIkei6GxJRbTzjZn2V8tsCplULpGbkzfefN UAMJswX9zSQ0KOLVKjgVu+i/jjRnFQ3M1TCHGk+UZ52A2S3GABQmSSlZ9gVMRtOzc9SWWJ4NlW+30m 9uUq0D7n94pWSqzt2iSswARIQ//LoLAiOqOuwNZyvWQGELDNcabVoewd9gqtrhzIYddE4C/V8hsR1t 6MXwGLxvVANyH+DTenwzmXizO6GN3vMvs3DbltSlFJ9SqnTYMdBn6v79pFHvfUnnQ0Aq39gOxH/OL9 s+iKJNKijBoFcMr5eYYq3ik2KJplEO9Gi26VFNyEwpJzo= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211018_071648_591470_AAE49C50 X-CRM114-Status: GOOD ( 37.95 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Wire up the generic support for managing task stack allocations via vmalloc, and implement the entry code that detects whether we faulted because of a stack overrun (or future stack overrun caused by pushing the pt_regs array) While this adds a fair amount of tricky entry asm code, it should be noted that it only adds a TST + branch to the svc_entry path. The code implementing the non-trivial handling of the overflow stack is emitted out-of-line into the .text section. Since on ARM, we rely on do_translation_fault() to keep PMD level page table entries that cover the vmalloc region up to date, we need to ensure that we don't hit such a stale PMD entry when accessing the stack. So we do a dummy read from the new stack while still running from the old one on the context switch path, and bump the vmalloc_seq counter when PMD level entries in the vmalloc range are modified, so that the MM switch fetches the latest version of the entries. Signed-off-by: Ard Biesheuvel Tested-by: Keith Packard --- arch/arm/Kconfig | 1 + arch/arm/include/asm/page.h | 4 + arch/arm/include/asm/thread_info.h | 8 ++ arch/arm/kernel/entry-armv.S | 79 ++++++++++++++++++-- arch/arm/kernel/entry-header.S | 57 ++++++++++++++ arch/arm/kernel/irq.c | 9 ++- arch/arm/kernel/traps.c | 65 +++++++++++++++- arch/arm/kernel/unwind.c | 3 +- arch/arm/kernel/vmlinux.lds.S | 4 +- 9 files changed, 219 insertions(+), 11 deletions(-) diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index d46b243e1b26..964cc41a0fed 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -126,6 +126,7 @@ config ARM select RTC_LIB select SYS_SUPPORTS_APM_EMULATION select THREAD_INFO_IN_TASK if CURRENT_POINTER_IN_TPIDRURO + select HAVE_ARCH_VMAP_STACK if THREAD_INFO_IN_TASK select TRACE_IRQFLAGS_SUPPORT if !CPU_V7M # Above selects are sorted alphabetically; please add new ones # according to that. Thanks. diff --git a/arch/arm/include/asm/page.h b/arch/arm/include/asm/page.h index 11b058a72a5b..7b871ed99ccf 100644 --- a/arch/arm/include/asm/page.h +++ b/arch/arm/include/asm/page.h @@ -149,6 +149,10 @@ extern void copy_page(void *to, const void *from); #include #endif +#ifdef CONFIG_VMAP_STACK +#define ARCH_PAGE_TABLE_SYNC_MASK PGTBL_PMD_MODIFIED +#endif + #endif /* CONFIG_MMU */ typedef struct page *pgtable_t; diff --git a/arch/arm/include/asm/thread_info.h b/arch/arm/include/asm/thread_info.h index 787511396f3f..900c8ae484c9 100644 --- a/arch/arm/include/asm/thread_info.h +++ b/arch/arm/include/asm/thread_info.h @@ -25,6 +25,14 @@ #define THREAD_SIZE (PAGE_SIZE << THREAD_SIZE_ORDER) #define THREAD_START_SP (THREAD_SIZE - 8) +#ifdef CONFIG_VMAP_STACK +#define THREAD_ALIGN (2 * THREAD_SIZE) +#else +#define THREAD_ALIGN THREAD_SIZE +#endif + +#define OVERFLOW_STACK_SIZE SZ_4K + #ifndef __ASSEMBLY__ struct task_struct; diff --git a/arch/arm/kernel/entry-armv.S b/arch/arm/kernel/entry-armv.S index 80afab1c344b..6910d6596e08 100644 --- a/arch/arm/kernel/entry-armv.S +++ b/arch/arm/kernel/entry-armv.S @@ -58,6 +58,14 @@ UNWIND( .setfp fpreg, sp ) @ subs r2, sp, r0 @ SP above bottom of IRQ stack? rsbscs r2, r2, #THREAD_SIZE @ ... and below the top? +#ifdef CONFIG_VMAP_STACK + bcs .L\@ + mov_l r2, overflow_stack @ Take base address + add r2, r2, r3 @ Top of this CPU's overflow stack + subs r2, r0, r2 @ Compare with incoming SP + rsbscs r2, r2, #OVERFLOW_STACK_SIZE +.L\@: +#endif movcs sp, r0 @ If so, revert to incoming SP #ifndef CONFIG_UNWINDER_ARM @@ -189,13 +197,18 @@ ENDPROC(__und_invalid) #define SPFIX(code...) #endif - .macro svc_entry, stack_hole=0, trace=1, uaccess=1 + .macro svc_entry, stack_hole=0, trace=1, uaccess=1, overflow_check=1 UNWIND(.fnstart ) - UNWIND(.save {r0 - pc} ) sub sp, sp, #(SVC_REGS_SIZE + \stack_hole) + THUMB( add sp, r0 ) @ get SP in a GPR without + THUMB( sub r0, sp, r0 ) @ using a temp register + + .if \overflow_check + UNWIND(.save {r0 - pc} ) + do_overflow_check (SVC_REGS_SIZE + \stack_hole) + .endif + #ifdef CONFIG_THUMB2_KERNEL - add sp, r0 @ get SP in a GPR without - sub r0, sp, r0 @ using a temp register tst r0, #4 @ test stack pointer alignment sub r0, sp, r0 @ restore original R0 sub sp, r0 @ restore original SP @@ -828,12 +841,20 @@ ENTRY(__switch_to) str r7, [r8] #endif mov r0, r5 -#if !defined(CONFIG_THUMB2_KERNEL) +#if !defined(CONFIG_THUMB2_KERNEL) && !defined(CONFIG_VMAP_STACK) set_current r7 ldmia r4, {r4 - sl, fp, sp, pc} @ Load all regs saved previously #else mov r1, r7 ldmia r4, {r4 - sl, fp, ip, lr} @ Load all regs saved previously +#ifdef CONFIG_VMAP_STACK + @ + @ Do a dummy read from the new stack while running from the old one so + @ that we can rely on do_translation_fault() to fix up any stale PMD + @ entries covering the vmalloc region. + @ + ldr r2, [ip] +#endif set_current r1 mov sp, ip ret lr @@ -841,6 +862,54 @@ ENTRY(__switch_to) UNWIND(.fnend ) ENDPROC(__switch_to) +#ifdef CONFIG_VMAP_STACK + .text +__bad_stack: + @ + @ We detected an overflow in svc_entry, which switched to the + @ overflow stack. Stash the exception regs, and head to our overflow + @ handler. Entered with the orginal value of SP in IP, and the original + @ value of IP in TPIDRURW + @ + +#if defined(CONFIG_UNWINDER_FRAME_POINTER) && defined(CONFIG_CC_IS_GCC) + mov ip, ip @ mov expected by unwinder + push {fp, ip, lr, pc} @ GCC flavor frame record +#else + str ip, [sp, #-8]! @ store original SP + push {fpreg, lr} @ Clang flavor frame record +#endif +UNWIND( ldr ip, [r0, #4] ) @ load exception LR +UNWIND( str ip, [sp, #12] ) @ store in the frame record + mrc p15, 0, ip, c13, c0, 2 @ reload IP + + @ Store the original GPRs to the new stack. + svc_entry uaccess=0, overflow_check=0 + +UNWIND( .save {sp, pc} ) +UNWIND( .save {fpreg, lr} ) +UNWIND( .setfp fpreg, sp ) + + ldr fpreg, [sp, #S_SP] @ Add our frame record + @ to the linked list +#if defined(CONFIG_UNWINDER_FRAME_POINTER) && defined(CONFIG_CC_IS_GCC) + ldr r1, [fp, #4] @ reload SP at entry + add fp, fp, #12 +#else + ldr r1, [fpreg, #8] +#endif + str r1, [sp, #S_SP] @ store in pt_regs + + @ Stash the regs for handle_bad_stack + mov r0, sp + + @ Time to die + bl handle_bad_stack + nop +UNWIND( .fnend ) +ENDPROC(__bad_stack) +#endif + __INIT /* diff --git a/arch/arm/kernel/entry-header.S b/arch/arm/kernel/entry-header.S index ae24dd54e9ef..823dd1aa6e3e 100644 --- a/arch/arm/kernel/entry-header.S +++ b/arch/arm/kernel/entry-header.S @@ -423,3 +423,60 @@ scno .req r7 @ syscall number tbl .req r8 @ syscall table pointer why .req r8 @ Linux syscall (!= 0) tsk .req r9 @ current thread_info + + .macro do_overflow_check, frame_size:req +#ifdef CONFIG_VMAP_STACK + @ + @ Test whether the SP has overflowed. Task and IRQ stacks are aligned + @ so that SP & BIT(THREAD_SIZE_ORDER + PAGE_SHIFT) should always be + @ zero. + @ +ARM( tst sp, #1 << (THREAD_SIZE_ORDER + PAGE_SHIFT) ) +THUMB( tst r0, #1 << (THREAD_SIZE_ORDER + PAGE_SHIFT) ) +THUMB( it ne ) + bne .Lstack_overflow_check\@ + + .pushsection .text +.Lstack_overflow_check\@: + @ + @ Either we've just detected an overflow, or we've taken an exception + @ while on the overflow stack. We cannot use the stack until we have + @ decided which is the case. However, as we won't return to userspace, + @ we can clobber some USR/SYS mode registers to free up GPRs. + @ + + mcr p15, 0, ip, c13, c0, 2 @ Stash IP in TPIDRURW + mrs ip, cpsr + eor ip, ip, #(SVC_MODE ^ SYSTEM_MODE) + msr cpsr_c, ip @ Switch to SYS mode + eor ip, ip, #(SVC_MODE ^ SYSTEM_MODE) + mov sp, ip @ Stash mode in SP_usr + + @ Load the overflow stack into IP using LR_usr as a scratch register + mov_l lr, overflow_stack + OVERFLOW_STACK_SIZE + mrc p15, 0, ip, c13, c0, 4 @ Get CPU offset + add ip, ip, lr @ IP := this CPU's overflow stack + mov lr, sp @ Unstash mode into LR_usr + msr cpsr_c, lr @ Switch back to SVC mode + + @ + @ Check whether we are already on the overflow stack. This may happen, + @ e.g., when performing accesses that may fault when dumping the stack. + @ The overflow stack is not in the vmalloc space so we only need to + @ check whether the incoming SP is below the top of the overflow stack. + @ +ARM( subs ip, sp, ip ) @ Delta with top of overflow stack +THUMB( subs ip, r0, ip ) + mrclo p15, 0, ip, c13, c0, 2 @ Restore IP + blo .Lout\@ @ Carry on + +THUMB( sub r0, sp, r0 ) @ Restore original R0 +THUMB( sub sp, r0 ) @ Restore original SP + sub sp, sp, ip @ Switch to overflow stack + add ip, sp, ip @ Keep incoming SP value in IP + add ip, ip, #\frame_size @ Undo svc_entry's SP change + b __bad_stack + .popsection +.Lout\@: +#endif + .endm diff --git a/arch/arm/kernel/irq.c b/arch/arm/kernel/irq.c index bab3159b9074..25eacfd0f439 100644 --- a/arch/arm/kernel/irq.c +++ b/arch/arm/kernel/irq.c @@ -56,7 +56,14 @@ static void __init init_irq_stacks(void) int cpu; for_each_possible_cpu(cpu) { - stack = (u8 *)__get_free_pages(GFP_KERNEL, THREAD_SIZE_ORDER); + if (!IS_ENABLED(CONFIG_VMAP_STACK)) + stack = (u8 *)__get_free_pages(GFP_KERNEL, + THREAD_SIZE_ORDER); + else + stack = __vmalloc_node(THREAD_SIZE, THREAD_ALIGN, + THREADINFO_GFP, NUMA_NO_NODE, + __builtin_return_address(0)); + if (WARN_ON(!stack)) break; per_cpu(irq_stack_ptr, cpu) = &stack[THREAD_SIZE]; diff --git a/arch/arm/kernel/traps.c b/arch/arm/kernel/traps.c index 1ba9fc2eb564..32fd3fee808d 100644 --- a/arch/arm/kernel/traps.c +++ b/arch/arm/kernel/traps.c @@ -121,7 +121,8 @@ void dump_backtrace_stm(u32 *stack, u32 instruction, const char *loglvl) static int verify_stack(unsigned long sp) { if (sp < PAGE_OFFSET || - (sp > (unsigned long)high_memory && high_memory != NULL)) + (!IS_ENABLED(CONFIG_VMAP_STACK) && + sp > (unsigned long)high_memory && high_memory != NULL)) return -EFAULT; return 0; @@ -291,7 +292,8 @@ static int __die(const char *str, int err, struct pt_regs *regs) if (!user_mode(regs) || in_interrupt()) { dump_mem(KERN_EMERG, "Stack: ", regs->ARM_sp, - ALIGN(regs->ARM_sp, THREAD_SIZE)); + ALIGN(regs->ARM_sp - THREAD_SIZE, THREAD_ALIGN) + + THREAD_SIZE); dump_backtrace(regs, tsk, KERN_EMERG); dump_instr(KERN_EMERG, regs); } @@ -838,3 +840,62 @@ void __init early_trap_init(void *vectors_base) */ #endif } + +#ifdef CONFIG_VMAP_STACK + +DECLARE_PER_CPU(u8 *, irq_stack_ptr); + +asmlinkage DEFINE_PER_CPU_ALIGNED(u8[OVERFLOW_STACK_SIZE], overflow_stack); + +asmlinkage void handle_bad_stack(struct pt_regs *regs) +{ + unsigned long tsk_stk = (unsigned long)current->stack; + unsigned long irq_stk = (unsigned long)this_cpu_read(irq_stack_ptr); + unsigned long ovf_stk = (unsigned long)this_cpu_ptr(overflow_stack); + + console_verbose(); + pr_emerg("Insufficient stack space to handle exception!"); + + pr_emerg("Task stack: [0x%08lx..0x%08lx]\n", + tsk_stk, tsk_stk + THREAD_SIZE); + pr_emerg("IRQ stack: [0x%08lx..0x%08lx]\n", + irq_stk, irq_stk + THREAD_SIZE); + pr_emerg("Overflow stack: [0x%08lx..0x%08lx]\n", + ovf_stk, ovf_stk + OVERFLOW_STACK_SIZE); + + die("kernel stack overflow", regs, 0); +} + +/* + * Normally, we rely on the logic in do_translation_fault() to update stale PMD + * entries covering the vmalloc space in a task's page tables when it first + * accesses the region in question. Unfortunately, this is not sufficient when + * the task stack resides in the vmalloc region, as do_translation_fault() is a + * C function that needs a stack to run. + * + * So we need to ensure that these PMD entries are up to date *before* the MM + * switch. As we already have some logic in the MM switch path that takes care + * of this, let's trigger it by bumping the counter every time the core vmalloc + * code modifies a PMD entry in the vmalloc region. + */ +void arch_sync_kernel_mappings(unsigned long start, unsigned long end) +{ + if (start > VMALLOC_END || end < VMALLOC_START) + return; + + /* + * This hooks into the core vmalloc code to receive notifications of + * any PMD level changes that have been made to the kernel page tables. + * This means it should only be triggered once for every MiB worth of + * vmalloc space, given that we don't support huge vmalloc/vmap on ARM, + * and that kernel PMD level table entries are rarely (if ever) + * updated. + * + * This means that the counter is going to max out at ~250 for the + * typical case. If it overflows, something entirely unexpected has + * occurred so let's throw a warning if that happens. + */ + WARN_ON(++init_mm.context.vmalloc_seq == UINT_MAX); +} + +#endif diff --git a/arch/arm/kernel/unwind.c b/arch/arm/kernel/unwind.c index e8d729975f12..c5ea328c428d 100644 --- a/arch/arm/kernel/unwind.c +++ b/arch/arm/kernel/unwind.c @@ -389,7 +389,8 @@ int unwind_frame(struct stackframe *frame) /* store the highest address on the stack to avoid crossing it*/ ctrl.sp_low = frame->sp; - ctrl.sp_high = ALIGN(ctrl.sp_low, THREAD_SIZE); + ctrl.sp_high = ALIGN(ctrl.sp_low - THREAD_SIZE, THREAD_ALIGN) + + THREAD_SIZE; pr_debug("%s(pc = %08lx lr = %08lx sp = %08lx)\n", __func__, frame->pc, frame->lr, frame->sp); diff --git a/arch/arm/kernel/vmlinux.lds.S b/arch/arm/kernel/vmlinux.lds.S index 20c4f6d20c7a..924cde084293 100644 --- a/arch/arm/kernel/vmlinux.lds.S +++ b/arch/arm/kernel/vmlinux.lds.S @@ -138,12 +138,12 @@ SECTIONS #ifdef CONFIG_STRICT_KERNEL_RWX . = ALIGN(1<