From patchwork Fri Nov 1 22:11:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 11223793 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B3B4F14DB for ; Fri, 1 Nov 2019 22:12:03 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 91EE021897 for ; Fri, 1 Nov 2019 22:12:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="bQly73Dv"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="H/ovEUCU" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 91EE021897 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=u0sYw9VkBZ219IQwYUBUpm4475fuaSSl1zM4nVBroXM=; b=bQly73Dv9Fckg+ PWPsADeBWHJSwKIk4eOCMuJVRVwQVykpdO0LMQcUmrGgrNlkXKZZJXd7CL8AL5L0AtgZm1f9YLy3q ZiaGT/BRs3A4Q6AhMPXTiNBCuOmVcxSb0FQ234TTJhjTadj8ZVN+n4FocrTR/ecBRaQUHKGLiDL9s 995eRwWe1qiAcogCVm25kE2kzbYWnbTF0Ju8sZhE8szKPh8WOhICzQwmMMPVkts/f2aALY43xRaJu mP2xhNuQFSXElMjQDbgeVfU0LKkvVckAhA5VnX4W/pCQEfflSyqaXVXxIYDBShJ9GLYMSKg/DyZcw GKULQhe6i81EjTx+dCGA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQf9G-0007fA-0N; Fri, 01 Nov 2019 22:12:02 +0000 Received: from mail-pl1-x649.google.com ([2607:f8b0:4864:20::649]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQf9D-0007dn-3d for linux-arm-kernel@lists.infradead.org; Fri, 01 Nov 2019 22:12:00 +0000 Received: by mail-pl1-x649.google.com with SMTP id u9so1472057plq.1 for ; Fri, 01 Nov 2019 15:11:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=g+FZBFl4/jtEHUrecvJjSJDyfo/JMaY/R05FvQww7Ew=; b=H/ovEUCUoKIya9z0Y7W61l4KBSDBXc8TsOG7hs/OHvt0y9UAR+pD4g9nrrXBU0wrQq V3Xv8T9+jQ0lOgGsjG3GZJPhp4lbn2XfTTMZHHRaekqYKuH/T4m9iLr9VI7TsiDUvt4b sDDmHgKXeSxdhxsfj4H9HDNQw/ZR4RnEHqpPLuKyQIVjBOuJmpWE7s70NonvITbd0/PO MppftbHIINpO8szfUH+nJLRx0qdne3AgLd2YbZDV4fUfMivPV9oSUatIlASqWKzwXntJ HeG1Ulhlt49Dmw0e+yx1QfE3/39NeumvMU64mQDoyIjIx+F8pyhPORKjx+0AdnDSYiwm aHCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=g+FZBFl4/jtEHUrecvJjSJDyfo/JMaY/R05FvQww7Ew=; b=OZxJAdVm1/kCOyqjOqe2AULm4My9qR+Dad+5qOwdFjzPGuNW63xUl6Bx4ZohPxAZ3M 6njbLUcp4irZM2+bpWwghZ61ACdS4Iu1FWP4rSoiVeIUmrhwUn0AZdoF9Xn28jsbmPSP iEb9/rCMTUGvhQFE0QBTTeVfDxZ9BDHgpcRVbqjW8i4WpdqzHQ9qKuyNct99pgxtZSGF dMr18VggHmNE9/PuD905mb7bAonX/RCH14VCxeSX1XyoUmR3mUezDNnlT4YYcpoaGtRg uzjmOIRrB1TVZK5zyiANoTolXytXLSjO9wdXgKrBwJRQ+q7K+VE4lw6Kf5hkpVojB670 hpJA== X-Gm-Message-State: APjAAAUks6fpwg3vdWBd9tW8qE3/5u3qEJL8vyxNWYwZLthy9SABM70a 7AIMjWlAgaWAGQC3JhG40IiHQ7P96jdr4Mxi0/E= X-Google-Smtp-Source: APXvYqwWJizqznswz2NLt2QFw2UmOTSRfvX8QdaNGFlpQWjP8Kjaeosw9eI19g7pulu6RKNVswWCRLsv6MO42umNMPw= X-Received: by 2002:a63:6744:: with SMTP id b65mr15837971pgc.13.1572646317280; Fri, 01 Nov 2019 15:11:57 -0700 (PDT) Date: Fri, 1 Nov 2019 15:11:34 -0700 In-Reply-To: <20191101221150.116536-1-samitolvanen@google.com> Message-Id: <20191101221150.116536-2-samitolvanen@google.com> Mime-Version: 1.0 References: <20191018161033.261971-1-samitolvanen@google.com> <20191101221150.116536-1-samitolvanen@google.com> X-Mailer: git-send-email 2.24.0.rc1.363.gb1bccd3e3d-goog Subject: [PATCH v4 01/17] arm64: mm: avoid x18 in idmap_kpti_install_ng_mappings From: Sami Tolvanen To: Will Deacon , Catalin Marinas , Steven Rostedt , Masami Hiramatsu , Ard Biesheuvel X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20191101_151159_146465_5361F4FC X-CRM114-Status: GOOD ( 10.24 ) X-Spam-Score: -7.7 (-------) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (-7.7 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:649 listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM white-list -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.0 DKIMWL_WL_MED DKIMwl.org - Medium sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Kees Cook , Jann Horn , Masahiro Yamada , Marc Zyngier , kernel-hardening@lists.openwall.com, Nick Desaulniers , linux-kernel@vger.kernel.org, Miguel Ojeda , clang-built-linux@googlegroups.com, Sami Tolvanen , Laura Abbott , Dave Martin , linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org idmap_kpti_install_ng_mappings uses x18 as a temporary register, which will result in a conflict when x18 is reserved. Use x16 and x17 instead where needed. Signed-off-by: Sami Tolvanen Reviewed-by: Nick Desaulniers Reviewed-by: Mark Rutland --- arch/arm64/mm/proc.S | 63 ++++++++++++++++++++++---------------------- 1 file changed, 32 insertions(+), 31 deletions(-) diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index a1e0592d1fbc..fdabf40a83c8 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -250,15 +250,15 @@ ENTRY(idmap_kpti_install_ng_mappings) /* We're the boot CPU. Wait for the others to catch up */ sevl 1: wfe - ldaxr w18, [flag_ptr] - eor w18, w18, num_cpus - cbnz w18, 1b + ldaxr w17, [flag_ptr] + eor w17, w17, num_cpus + cbnz w17, 1b /* We need to walk swapper, so turn off the MMU. */ pre_disable_mmu_workaround - mrs x18, sctlr_el1 - bic x18, x18, #SCTLR_ELx_M - msr sctlr_el1, x18 + mrs x17, sctlr_el1 + bic x17, x17, #SCTLR_ELx_M + msr sctlr_el1, x17 isb /* Everybody is enjoying the idmap, so we can rewrite swapper. */ @@ -281,9 +281,9 @@ skip_pgd: isb /* We're done: fire up the MMU again */ - mrs x18, sctlr_el1 - orr x18, x18, #SCTLR_ELx_M - msr sctlr_el1, x18 + mrs x17, sctlr_el1 + orr x17, x17, #SCTLR_ELx_M + msr sctlr_el1, x17 isb /* @@ -353,46 +353,47 @@ skip_pte: b.ne do_pte b next_pmd + .unreq cpu + .unreq num_cpus + .unreq swapper_pa + .unreq cur_pgdp + .unreq end_pgdp + .unreq pgd + .unreq cur_pudp + .unreq end_pudp + .unreq pud + .unreq cur_pmdp + .unreq end_pmdp + .unreq pmd + .unreq cur_ptep + .unreq end_ptep + .unreq pte + /* Secondary CPUs end up here */ __idmap_kpti_secondary: /* Uninstall swapper before surgery begins */ - __idmap_cpu_set_reserved_ttbr1 x18, x17 + __idmap_cpu_set_reserved_ttbr1 x16, x17 /* Increment the flag to let the boot CPU we're ready */ -1: ldxr w18, [flag_ptr] - add w18, w18, #1 - stxr w17, w18, [flag_ptr] +1: ldxr w16, [flag_ptr] + add w16, w16, #1 + stxr w17, w16, [flag_ptr] cbnz w17, 1b /* Wait for the boot CPU to finish messing around with swapper */ sevl 1: wfe - ldxr w18, [flag_ptr] - cbnz w18, 1b + ldxr w16, [flag_ptr] + cbnz w16, 1b /* All done, act like nothing happened */ - offset_ttbr1 swapper_ttb, x18 + offset_ttbr1 swapper_ttb, x16 msr ttbr1_el1, swapper_ttb isb ret - .unreq cpu - .unreq num_cpus - .unreq swapper_pa .unreq swapper_ttb .unreq flag_ptr - .unreq cur_pgdp - .unreq end_pgdp - .unreq pgd - .unreq cur_pudp - .unreq end_pudp - .unreq pud - .unreq cur_pmdp - .unreq end_pmdp - .unreq pmd - .unreq cur_ptep - .unreq end_ptep - .unreq pte ENDPROC(idmap_kpti_install_ng_mappings) .popsection #endif From patchwork Fri Nov 1 22:11:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 11223807 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 54E081747 for ; Fri, 1 Nov 2019 22:12:58 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 32F4321897 for ; Fri, 1 Nov 2019 22:12:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="AfzwZrFe"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="Q0ypX2RR" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 32F4321897 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=QMzMIckDD8LXvaTnwwv1OTj6jHEPyRyDFXVu0f329RI=; b=AfzwZrFeD29qh/ 4jZxaNNnjHaZGSjB6Ejwe0fcfoL0cFO/zmH/JvBnTOh5pjnhM10ndvGkAnpUWCrJQhDca6OOiwMV5 s/+bVJaFrBCbHUd0+iY7Ef/MCgBq4M/R0i1+fSgoOL68gg0YPVeFWyH8tMiXtH2covjLw0Y/aDLV1 Zuw5X4X8J9gWAqenjlYEL4f8Y8dZ8s6FH/ThW8sWDSALIIRydDgSOqCFGJgeuvbEVZkhrd1xo19Vb vvTVUtNFurn9DXgibmH2uj4m1kDvLuK3TmE929xTM+k9Wta6o2fMaFSG5ejBALl++klzj7Rc2f2lo 5u/A8LNisic4OzBOBBAg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQfA7-00008q-QB; Fri, 01 Nov 2019 22:12:55 +0000 Received: from mail-pf1-x44a.google.com ([2607:f8b0:4864:20::44a]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQf9G-0007ey-1H for linux-arm-kernel@lists.infradead.org; Fri, 01 Nov 2019 22:12:04 +0000 Received: by mail-pf1-x44a.google.com with SMTP id r187so8428325pfc.16 for ; Fri, 01 Nov 2019 15:12:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=OH2SjYe0B8esWxTAcCgR+62MsPomBC+RzRSTMU1rFk8=; b=Q0ypX2RRPc9BCIL2STVCLvfQgO8tit2Vbhk/ba4d6hjxn/SWRm9m0P3+RgrNT2Mz6L sR+GG/DQFJrsOpiRvBMZe/oQtxBn0b/Aw4IaKtUdtT9BIZjbq98xY+73BjaIsIx8Adyg M5/Y6BEGGBeU1F0DiCWwVE0YrYEJUkeeSdYpDOR4oINiHvrg4NRPL6mRj2HcLoQWQnuD PYDoVjgxtSwP/n5hXyxm9HZngM/5zcLb8ANWcQjpR6xl/qqaKAenvDZks2pyB05/I9H1 +gQ/L2rTv3QAg1ciPxxnHi9CW/af4LKSwTNtbc9o4woUjjbzkrkmW3ha88Kih8MYyqXD oGUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=OH2SjYe0B8esWxTAcCgR+62MsPomBC+RzRSTMU1rFk8=; b=b3VrUfTY3QLjdUtVMnLCKjgS9JqdDHGA8E8jh034xCr7eQXPc0DDcUC5L7Hf6DlYOY hGL4rvJSxaGw+ekKQ+SywHlpdEhtR5I5/0ibHo9ZSeNVWO0kbA32EfFR4XNDjKv2qMtr NzefHqPjlId318nO+sr7YNHaae09Fi3uy4Ax/gDZ3NsoM8Evzd20a5ttIyUj8ykq4TE3 tjlfbke3xsMEytRBCUZ66U4eMQJxvFFbNPSUcH18XLjbfV3ZYxG8Y2AwxQhZWh55Af1I PoOMNuUfJP4UBeebDOITlTu3axDupb07xZTVUI8HoZVRGfn7XKCPoFVpOS6PFGMEu9cv sxcg== X-Gm-Message-State: APjAAAUrGC0FPdLVLhinnoah/WqwkjXU7szd2sYEBKhzuYWcZOqYafFf C1dApUybWyc5sUrodxFn0NfFnOIu8VwvDFUrLic= X-Google-Smtp-Source: APXvYqyE5ghbfjaqrUWmTezRWNfWGA0WB8HP1ZsfnYTWo7dD5ai7FSDVVzfSZssPNsNmD4T/gU2ZN0nL/1GoUaJgugM= X-Received: by 2002:a63:e454:: with SMTP id i20mr15822954pgk.319.1572646319823; Fri, 01 Nov 2019 15:11:59 -0700 (PDT) Date: Fri, 1 Nov 2019 15:11:35 -0700 In-Reply-To: <20191101221150.116536-1-samitolvanen@google.com> Message-Id: <20191101221150.116536-3-samitolvanen@google.com> Mime-Version: 1.0 References: <20191018161033.261971-1-samitolvanen@google.com> <20191101221150.116536-1-samitolvanen@google.com> X-Mailer: git-send-email 2.24.0.rc1.363.gb1bccd3e3d-goog Subject: [PATCH v4 02/17] arm64/lib: copy_page: avoid x18 register in assembler code From: Sami Tolvanen To: Will Deacon , Catalin Marinas , Steven Rostedt , Masami Hiramatsu , Ard Biesheuvel X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20191101_151202_087276_DC708E30 X-CRM114-Status: UNSURE ( 9.49 ) X-CRM114-Notice: Please train this message. X-Spam-Score: -7.7 (-------) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (-7.7 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:44a listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM white-list -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.0 DKIMWL_WL_MED DKIMwl.org - Medium sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Kees Cook , Jann Horn , Masahiro Yamada , Marc Zyngier , kernel-hardening@lists.openwall.com, Nick Desaulniers , linux-kernel@vger.kernel.org, Miguel Ojeda , clang-built-linux@googlegroups.com, Sami Tolvanen , Laura Abbott , Dave Martin , linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org From: Ard Biesheuvel Register x18 will no longer be used as a caller save register in the future, so stop using it in the copy_page() code. Link: https://patchwork.kernel.org/patch/9836869/ Signed-off-by: Ard Biesheuvel [Sami: changed the offset and bias to be explicit] Signed-off-by: Sami Tolvanen Reviewed-by: Mark Rutland --- arch/arm64/lib/copy_page.S | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/arch/arm64/lib/copy_page.S b/arch/arm64/lib/copy_page.S index bbb8562396af..290dd3c5266c 100644 --- a/arch/arm64/lib/copy_page.S +++ b/arch/arm64/lib/copy_page.S @@ -34,45 +34,45 @@ alternative_else_nop_endif ldp x14, x15, [x1, #96] ldp x16, x17, [x1, #112] - mov x18, #(PAGE_SIZE - 128) + add x0, x0, #256 add x1, x1, #128 1: - subs x18, x18, #128 + tst x0, #(PAGE_SIZE - 1) alternative_if ARM64_HAS_NO_HW_PREFETCH prfm pldl1strm, [x1, #384] alternative_else_nop_endif - stnp x2, x3, [x0] + stnp x2, x3, [x0, #-256] ldp x2, x3, [x1] - stnp x4, x5, [x0, #16] + stnp x4, x5, [x0, #16 - 256] ldp x4, x5, [x1, #16] - stnp x6, x7, [x0, #32] + stnp x6, x7, [x0, #32 - 256] ldp x6, x7, [x1, #32] - stnp x8, x9, [x0, #48] + stnp x8, x9, [x0, #48 - 256] ldp x8, x9, [x1, #48] - stnp x10, x11, [x0, #64] + stnp x10, x11, [x0, #64 - 256] ldp x10, x11, [x1, #64] - stnp x12, x13, [x0, #80] + stnp x12, x13, [x0, #80 - 256] ldp x12, x13, [x1, #80] - stnp x14, x15, [x0, #96] + stnp x14, x15, [x0, #96 - 256] ldp x14, x15, [x1, #96] - stnp x16, x17, [x0, #112] + stnp x16, x17, [x0, #112 - 256] ldp x16, x17, [x1, #112] add x0, x0, #128 add x1, x1, #128 - b.gt 1b + b.ne 1b - stnp x2, x3, [x0] - stnp x4, x5, [x0, #16] - stnp x6, x7, [x0, #32] - stnp x8, x9, [x0, #48] - stnp x10, x11, [x0, #64] - stnp x12, x13, [x0, #80] - stnp x14, x15, [x0, #96] - stnp x16, x17, [x0, #112] + stnp x2, x3, [x0, #-256] + stnp x4, x5, [x0, #16 - 256] + stnp x6, x7, [x0, #32 - 256] + stnp x8, x9, [x0, #48 - 256] + stnp x10, x11, [x0, #64 - 256] + stnp x12, x13, [x0, #80 - 256] + stnp x14, x15, [x0, #96 - 256] + stnp x16, x17, [x0, #112 - 256] ret ENDPROC(copy_page) From patchwork Fri Nov 1 22:11:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 11223813 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 50A6D1668 for ; Fri, 1 Nov 2019 22:13:12 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2BFD1217D9 for ; Fri, 1 Nov 2019 22:13:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="UGG1lk5G"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="bzUzyeY2" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2BFD1217D9 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=htD+YRumA84apyX9xMQdnM59eCvTrGrZy+Qb/HO4ETw=; b=UGG1lk5GJdooQA OYrZuPRUP9TsaGulAqWbWwg3hKssdHC0LtRgZtUfhCSb/DZqtOz1l188py9u7tZI8fSo1KQrz9Lg/ 8QovTmde6VlJQxf775VLKkqFqFvRez9UBuQjyrXYXqFV61gTmkAbmPyNQquK8StzKd7Wtt0w/d0/R Xi6QJ2KNY70j82uVIRILHuB7SxEowQzgNsvkkRM/vr67qOsUO+sYOBsQ57wLdYjeXbRjHIkId8KEq TLJYn6y6bWg5Sn8gpzGwJ0cLILLncJfKVlmGzukZitpgXqBp56R5yC3UQONMeV9JOo40Hl38pMnTO LGSLYscvX4qAfOFwS41Q==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQfAM-0000Nx-Qy; Fri, 01 Nov 2019 22:13:10 +0000 Received: from mail-pf1-x449.google.com ([2607:f8b0:4864:20::449]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQf9I-0007gU-7H for linux-arm-kernel@lists.infradead.org; Fri, 01 Nov 2019 22:12:05 +0000 Received: by mail-pf1-x449.google.com with SMTP id i187so8440154pfc.10 for ; Fri, 01 Nov 2019 15:12:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=LP5ER91yDYZoiG1DiO5iJutGGta/8W2YsIEILYrERFw=; b=bzUzyeY2Pb66Afq/YuWOrvYc+P88+Nov5leW00IV55yTXjsAAsOwV4Veo8X/xOJcJ4 ondoceMK37epaHyC4iXnp3iV0xW9o37xNMNCS5+YnzUPDYInCBHdGEQVAgM4EFYIvsze U0KH0R+6LV4HKtRGQKKP/z9BhLOXnUh8+dXJh8b3NCoa0aWnVe395JzHp4ESMni67Vsa qx0mjSBtsBjrh+oThuzhLBL/bX8C2DLD9Oj+4ukb7J9N50N4TT7aG68H5rehFDpazui3 jPyL0/ZCDdN1GkSNCkwcVAq/qh9aaSMW7e8c/YGajhmaqVmrnJc5oqhERJW4y1q7jziQ Z3bw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=LP5ER91yDYZoiG1DiO5iJutGGta/8W2YsIEILYrERFw=; b=kRtymGO4bh/sNjjN+5z8ynD38egpYkqStvG97IRs8gfjD9gogjgx4oxfTzX3RjujkN qkX5GkDiO4CzVeC3n/Okri+cWVtMZQizKJHoUXClRUI3KRwA8rArAuY+fTcuRLB4LVkD kJe7GDHNLnCOxCuAtyd/bqGhB7sEPffDIiA9senUC3j+rn46JQfyYQkF/u8dty/U/dtU gOHHZTYOY05Oyws9h/ld7dNF69oVQARuC79yY56WU3Kz097RahlpF+DKAyEyUxbOpL9Q SlPJoJnI09Adgh+VYp+11kxJdS5H3WXF/5vI6AshzaKWn5grXGLAnnJPBGIAt6SlSQvn MivA== X-Gm-Message-State: APjAAAUx//f3KhWJGZq1zDXI2cPDP8yCEwpV/UFbXzPc14j5rJomBI7J 6p1wJMLFNcoHNQLGBXrquB9CS1rAnMRtcSnXSRw= X-Google-Smtp-Source: APXvYqz206FfV2E+48L9X7nVID4/AGCaj0CfUa7XJhd+U2VCpij5TxmcTYypesImIHmvjNsozEEINFqmvJRBmlZOaoA= X-Received: by 2002:a63:535c:: with SMTP id t28mr6291818pgl.173.1572646322264; Fri, 01 Nov 2019 15:12:02 -0700 (PDT) Date: Fri, 1 Nov 2019 15:11:36 -0700 In-Reply-To: <20191101221150.116536-1-samitolvanen@google.com> Message-Id: <20191101221150.116536-4-samitolvanen@google.com> Mime-Version: 1.0 References: <20191018161033.261971-1-samitolvanen@google.com> <20191101221150.116536-1-samitolvanen@google.com> X-Mailer: git-send-email 2.24.0.rc1.363.gb1bccd3e3d-goog Subject: [PATCH v4 03/17] arm64: kvm: stop treating register x18 as caller save From: Sami Tolvanen To: Will Deacon , Catalin Marinas , Steven Rostedt , Masami Hiramatsu , Ard Biesheuvel X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20191101_151204_277765_17E84DAA X-CRM114-Status: GOOD ( 10.77 ) X-Spam-Score: -7.7 (-------) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (-7.7 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:449 listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM white-list -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.0 DKIMWL_WL_MED DKIMwl.org - Medium sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Kees Cook , Jann Horn , Masahiro Yamada , Marc Zyngier , kernel-hardening@lists.openwall.com, Nick Desaulniers , linux-kernel@vger.kernel.org, Miguel Ojeda , clang-built-linux@googlegroups.com, Sami Tolvanen , Laura Abbott , Dave Martin , linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org From: Ard Biesheuvel In preparation of reserving x18, stop treating it as caller save in the KVM guest entry/exit code. Currently, the code assumes there is no need to preserve it for the host, given that it would have been assumed clobbered anyway by the function call to __guest_enter(). Instead, preserve its value and restore it upon return. Link: https://patchwork.kernel.org/patch/9836891/ Signed-off-by: Ard Biesheuvel [Sami: updated commit message, switched from x18 to x29 for the guest context] Signed-off-by: Sami Tolvanen Reviewed-by: Kees Cook Reviewed-by: Marc Zyngier Reviewed-by: Mark Rutland Reviewed-by: Marc Zyngier --- arch/arm64/kvm/hyp/entry.S | 41 +++++++++++++++++++------------------- 1 file changed, 20 insertions(+), 21 deletions(-) diff --git a/arch/arm64/kvm/hyp/entry.S b/arch/arm64/kvm/hyp/entry.S index e5cc8d66bf53..c3c2d842c609 100644 --- a/arch/arm64/kvm/hyp/entry.S +++ b/arch/arm64/kvm/hyp/entry.S @@ -23,6 +23,7 @@ .pushsection .hyp.text, "ax" .macro save_callee_saved_regs ctxt + str x18, [\ctxt, #CPU_XREG_OFFSET(18)] stp x19, x20, [\ctxt, #CPU_XREG_OFFSET(19)] stp x21, x22, [\ctxt, #CPU_XREG_OFFSET(21)] stp x23, x24, [\ctxt, #CPU_XREG_OFFSET(23)] @@ -32,6 +33,8 @@ .endm .macro restore_callee_saved_regs ctxt + // We assume \ctxt is not x18-x28 + ldr x18, [\ctxt, #CPU_XREG_OFFSET(18)] ldp x19, x20, [\ctxt, #CPU_XREG_OFFSET(19)] ldp x21, x22, [\ctxt, #CPU_XREG_OFFSET(21)] ldp x23, x24, [\ctxt, #CPU_XREG_OFFSET(23)] @@ -48,7 +51,7 @@ ENTRY(__guest_enter) // x0: vcpu // x1: host context // x2-x17: clobbered by macros - // x18: guest context + // x29: guest context // Store the host regs save_callee_saved_regs x1 @@ -67,31 +70,28 @@ alternative_else_nop_endif ret 1: - add x18, x0, #VCPU_CONTEXT + add x29, x0, #VCPU_CONTEXT // Macro ptrauth_switch_to_guest format: // ptrauth_switch_to_guest(guest cxt, tmp1, tmp2, tmp3) // The below macro to restore guest keys is not implemented in C code // as it may cause Pointer Authentication key signing mismatch errors // when this feature is enabled for kernel code. - ptrauth_switch_to_guest x18, x0, x1, x2 + ptrauth_switch_to_guest x29, x0, x1, x2 // Restore guest regs x0-x17 - ldp x0, x1, [x18, #CPU_XREG_OFFSET(0)] - ldp x2, x3, [x18, #CPU_XREG_OFFSET(2)] - ldp x4, x5, [x18, #CPU_XREG_OFFSET(4)] - ldp x6, x7, [x18, #CPU_XREG_OFFSET(6)] - ldp x8, x9, [x18, #CPU_XREG_OFFSET(8)] - ldp x10, x11, [x18, #CPU_XREG_OFFSET(10)] - ldp x12, x13, [x18, #CPU_XREG_OFFSET(12)] - ldp x14, x15, [x18, #CPU_XREG_OFFSET(14)] - ldp x16, x17, [x18, #CPU_XREG_OFFSET(16)] - - // Restore guest regs x19-x29, lr - restore_callee_saved_regs x18 - - // Restore guest reg x18 - ldr x18, [x18, #CPU_XREG_OFFSET(18)] + ldp x0, x1, [x29, #CPU_XREG_OFFSET(0)] + ldp x2, x3, [x29, #CPU_XREG_OFFSET(2)] + ldp x4, x5, [x29, #CPU_XREG_OFFSET(4)] + ldp x6, x7, [x29, #CPU_XREG_OFFSET(6)] + ldp x8, x9, [x29, #CPU_XREG_OFFSET(8)] + ldp x10, x11, [x29, #CPU_XREG_OFFSET(10)] + ldp x12, x13, [x29, #CPU_XREG_OFFSET(12)] + ldp x14, x15, [x29, #CPU_XREG_OFFSET(14)] + ldp x16, x17, [x29, #CPU_XREG_OFFSET(16)] + + // Restore guest regs x18-x29, lr + restore_callee_saved_regs x29 // Do not touch any register after this! eret @@ -114,7 +114,7 @@ ENTRY(__guest_exit) // Retrieve the guest regs x0-x1 from the stack ldp x2, x3, [sp], #16 // x0, x1 - // Store the guest regs x0-x1 and x4-x18 + // Store the guest regs x0-x1 and x4-x17 stp x2, x3, [x1, #CPU_XREG_OFFSET(0)] stp x4, x5, [x1, #CPU_XREG_OFFSET(4)] stp x6, x7, [x1, #CPU_XREG_OFFSET(6)] @@ -123,9 +123,8 @@ ENTRY(__guest_exit) stp x12, x13, [x1, #CPU_XREG_OFFSET(12)] stp x14, x15, [x1, #CPU_XREG_OFFSET(14)] stp x16, x17, [x1, #CPU_XREG_OFFSET(16)] - str x18, [x1, #CPU_XREG_OFFSET(18)] - // Store the guest regs x19-x29, lr + // Store the guest regs x18-x29, lr save_callee_saved_regs x1 get_host_ctxt x2, x3 From patchwork Fri Nov 1 22:11:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 11223819 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5A6EF1668 for ; Fri, 1 Nov 2019 22:13:27 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 38EFB217D9 for ; Fri, 1 Nov 2019 22:13:27 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="SA7+FlNM"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="q3YTCGLY" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 38EFB217D9 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=879X++7zCJOUkWXNHENJfK4q6zBXSaZyIp+jxoYX7mY=; b=SA7+FlNM9lkHc6 2+PPJsQGwZL74PaSTK4HhxUayg4IcqivRwsbQUf/8wgmsip3eKqzPYcUj/HMyGZCAW6Dvp4TctTr0 tNq5LFCEOEjx0R3UG5DoY74naa8GJelouxB86hpLfyxGT6faoSSq5V6iCM7+PqR02fydsJZZMLufZ zmqlmEmffFMKqqm0efgt3AFPITNAJZjaQBkrlLGAMJSqVvu9WsnS90JsWeb4ccVlWXDw+6FgH2+vh wXV8HuMktPI7jynmM26JlGra/XiDxk/MDP446KBNAoqB6rBrD+Yp5VrIhbMIH11ccdYNBp//pNjwX 9GRex8Pw5risbq0Ns81Q==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQfAb-0000di-Sw; Fri, 01 Nov 2019 22:13:25 +0000 Received: from mail-pf1-x449.google.com ([2607:f8b0:4864:20::449]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQf9K-0007lN-VA for linux-arm-kernel@lists.infradead.org; Fri, 01 Nov 2019 22:12:08 +0000 Received: by mail-pf1-x449.google.com with SMTP id a14so8440644pfr.12 for ; Fri, 01 Nov 2019 15:12:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=E806Ihq7Qr5kwAxZ8O8jq/5GK0i44oON304BuEfvZ08=; b=q3YTCGLY4SwiQs7Dr523GbwXcyS1rOF4ePlC+I2FlqMLdM1nbvtx89eitGKg3+MPv/ /6kpVC7ib3nqYGqEvd/D7Aja0/Oqw9bqK4yY73IHHngPbb8yNTF80TsjwS3dip4BNdjJ vrTCQU+wyHKRY1+9qAAoEBJOivxjjzkaOrEGQr3vnrSW+SARMhsCk1i/tL9jdyhbVr9G N6J/wBNQprJN4QGbaP/iQ1MEafD71JFawBswZkK6umVAzs2YdsoYpup5A4o/8Tk1dVfR JqWRLp00wfY2sTSArmHg6PdT5yupXzlZUV+YXQhkWVfHN9eXitDPORweXPhbmCnVSWKB z5cQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=E806Ihq7Qr5kwAxZ8O8jq/5GK0i44oON304BuEfvZ08=; b=RdfhlnpyiS08mX6DKfM4ETw12EyZFex7yisSY7nmmPoA58Uu+E/EG0SNTbtcTibxhm NwcRoUjGoi2IJSOLumU4tTevZBq7HFx88Q6B66yjww3N2Pc+97O2Mb2vmtrJkursokET vKgjgapqTgq2G1uFDRVIIUVqT953MZHKjQ8wwBJjjTifVEVW/swmsppHFb6TaFAZr400 0gzNVlnR4M1TJRDlriYGN9T/X9uf4bqrY8P+yQyB4zEMOuPDSKdNnylzxci5O5cliGiO DxflTzG+/8RQZb1WzBZVLeAB+XEVDBPgNWgmHvQZsm6gp1NKe3rIGG0aGQYAqtIzOBpl Q2yA== X-Gm-Message-State: APjAAAWctNoSmCfu5b7AK9K6dsaMshs/4+KgeMbhpS4vd+lmTLxd06hK 38/4ON8KlAZrq4LocAGynuSBZnGB+LWrEqhX6vA= X-Google-Smtp-Source: APXvYqyjCtIlE+FqIkDNQuZrIZu231KKDGdgU5VhgnFa05GlKuF5lFgoXY69KDCGAPVnM5k2sylcp3A3JeY4sKUC4Q0= X-Received: by 2002:a63:7015:: with SMTP id l21mr14741976pgc.200.1572646325115; Fri, 01 Nov 2019 15:12:05 -0700 (PDT) Date: Fri, 1 Nov 2019 15:11:37 -0700 In-Reply-To: <20191101221150.116536-1-samitolvanen@google.com> Message-Id: <20191101221150.116536-5-samitolvanen@google.com> Mime-Version: 1.0 References: <20191018161033.261971-1-samitolvanen@google.com> <20191101221150.116536-1-samitolvanen@google.com> X-Mailer: git-send-email 2.24.0.rc1.363.gb1bccd3e3d-goog Subject: [PATCH v4 04/17] arm64: kernel: avoid x18 __cpu_soft_restart From: Sami Tolvanen To: Will Deacon , Catalin Marinas , Steven Rostedt , Masami Hiramatsu , Ard Biesheuvel X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20191101_151207_026089_58E909F4 X-CRM114-Status: UNSURE ( 8.99 ) X-CRM114-Notice: Please train this message. X-Spam-Score: -7.7 (-------) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (-7.7 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:449 listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM white-list -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.0 DKIMWL_WL_MED DKIMwl.org - Medium sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Kees Cook , Jann Horn , Masahiro Yamada , Marc Zyngier , kernel-hardening@lists.openwall.com, Nick Desaulniers , linux-kernel@vger.kernel.org, Miguel Ojeda , clang-built-linux@googlegroups.com, Sami Tolvanen , Laura Abbott , Dave Martin , linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org From: Ard Biesheuvel The code in __cpu_soft_restart() uses x18 as an arbitrary temp register, which will shortly be disallowed. So use x8 instead. Link: https://patchwork.kernel.org/patch/9836877/ Signed-off-by: Ard Biesheuvel [Sami: updated commit message] Signed-off-by: Sami Tolvanen Reviewed-by: Mark Rutland Reviewed-by: Kees Cook --- arch/arm64/kernel/cpu-reset.S | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kernel/cpu-reset.S b/arch/arm64/kernel/cpu-reset.S index 6ea337d464c4..32c7bf858dd9 100644 --- a/arch/arm64/kernel/cpu-reset.S +++ b/arch/arm64/kernel/cpu-reset.S @@ -42,11 +42,11 @@ ENTRY(__cpu_soft_restart) mov x0, #HVC_SOFT_RESTART hvc #0 // no return -1: mov x18, x1 // entry +1: mov x8, x1 // entry mov x0, x2 // arg0 mov x1, x3 // arg1 mov x2, x4 // arg2 - br x18 + br x8 ENDPROC(__cpu_soft_restart) .popsection From patchwork Fri Nov 1 22:11:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 11223823 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4904314DB for ; Fri, 1 Nov 2019 22:13:43 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 23C95218AC for ; Fri, 1 Nov 2019 22:13:43 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="W/6nOwPy"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="mktcrk7B" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 23C95218AC Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=5lEgqMumJwG+d8Qg+JFok4U9urOcZ8YbrK29ZRSCgnw=; b=W/6nOwPymZ9BIN WHnvYdHHOTrXlPuuCU4JjjvrgHqskkvOPeSE1FwQdI3uZOsDzqoqHoh1ElWzgC50Wvb9hwVSmfmwY xaVymEfJoJPSLH2B2LXBh1FdfQBpdF9ioZXhVHafYQ2yVKVCz8pw/CAKNJn+spBqHH086jG4qSgOJ /59EYmmIjvNoMfc4aMMUO7jz3xd1vD7qCIeWdlUFHcJsiI7G92uEWzUns0GNOhfxocnQNIiXY2oxI fZkO8sKma3M1wp0ATB1VufFbnyNJXDYnUKN71D/UOqpzGjoCwxsNz3YCLoZN17zIrKlpQmUpvcSOn TtKZ3c78lewScgKu3INQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQfAr-0000tV-64; Fri, 01 Nov 2019 22:13:41 +0000 Received: from mail-pf1-x44a.google.com ([2607:f8b0:4864:20::44a]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQf9O-0007ou-55 for linux-arm-kernel@lists.infradead.org; Fri, 01 Nov 2019 22:12:13 +0000 Received: by mail-pf1-x44a.google.com with SMTP id a14so8440754pfr.12 for ; Fri, 01 Nov 2019 15:12:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=qjKhzVcl5kAZVlPLgMUk9d1eGUxOad0I4GHT70kbrwg=; b=mktcrk7BOmyYZwBvNmXLoB6Io2Uxp/1zEGR2G+M+Hdendd04+eF2uW9TLbwpV4ub17 tRnMG0Tl3hS1lIr7TbgTqoQPm4WNN699Qy3OMQD/KbRRUXzxvj3rYyRL4E6EvL2K1/gd mEpLAld2ZRmqyE+ivBRUkHNiHksf5BC7sDrvXWhPV59wRYM80rbYu79a9pPj8rXOKtqI 4KiJEHwqv/vPymSWkRK22BGootaOY8kfn0t0b1/GXeY7Yj7oFHYxKy4ZNvo9HMW8XZ15 8w/iSZDhvEq5N5UVEcCyPQNaaFHPXiSzWkP01cqpNT6YFXeZPxM5GnxgmW+0iscFxOfO jm6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=qjKhzVcl5kAZVlPLgMUk9d1eGUxOad0I4GHT70kbrwg=; b=Hei48Qlo9SrthT+WNxhC0SS0XqxIlu5srnTb4mWO3nOFtX0phofDHmv8nhflPZCWVU uad+EkglSD4vUbsSuYm0EdVZllMulHjl5DJJuuVHfFICH6lPZYeI/GRFoth24rBZTpoT WLAMTYAJFhczStOjjQx/Hw84AAqh/++ZdbSpMNqcVx92DzxUN5BGkKd/vWjjY9cO5O9v 2UhwSB4mgysYvYgQW2EmefSWiUQcCbDs5cllm6NtzW4PKaKjI4DuIxvYx5Qg8jYUlRnI B0vTZ6K5dHRVDUkd3AMOmAIrw90tie5cJBDCoKsmRxZRHFsPpvXHBrykprLoXoX48V/k k7nw== X-Gm-Message-State: APjAAAVro2IG3gYaMJdCJ3lkoECWQbzulYnmTumdB3wvgPeEO0awg/Yi tTjCHt6jpWK7/J4f67Enl9tTjDDIZiJDkSgNO1U= X-Google-Smtp-Source: APXvYqyLARmNaH8S8O6twJdW8Y7WdqRBsyPfZfSSykSct+riROectU7m54wTkermQdBGn6v+XtLVroHFv4sT3rND/xo= X-Received: by 2002:a63:c445:: with SMTP id m5mr16021319pgg.211.1572646327753; Fri, 01 Nov 2019 15:12:07 -0700 (PDT) Date: Fri, 1 Nov 2019 15:11:38 -0700 In-Reply-To: <20191101221150.116536-1-samitolvanen@google.com> Message-Id: <20191101221150.116536-6-samitolvanen@google.com> Mime-Version: 1.0 References: <20191018161033.261971-1-samitolvanen@google.com> <20191101221150.116536-1-samitolvanen@google.com> X-Mailer: git-send-email 2.24.0.rc1.363.gb1bccd3e3d-goog Subject: [PATCH v4 05/17] add support for Clang's Shadow Call Stack (SCS) From: Sami Tolvanen To: Will Deacon , Catalin Marinas , Steven Rostedt , Masami Hiramatsu , Ard Biesheuvel X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20191101_151210_536101_9198EAD7 X-CRM114-Status: GOOD ( 26.77 ) X-Spam-Score: -7.7 (-------) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (-7.7 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:44a listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM white-list -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.0 DKIMWL_WL_MED DKIMwl.org - Medium sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Kees Cook , Jann Horn , Masahiro Yamada , Marc Zyngier , kernel-hardening@lists.openwall.com, Nick Desaulniers , linux-kernel@vger.kernel.org, Miguel Ojeda , clang-built-linux@googlegroups.com, Sami Tolvanen , Laura Abbott , Dave Martin , linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org This change adds generic support for Clang's Shadow Call Stack, which uses a shadow stack to protect return addresses from being overwritten by an attacker. Details are available here: https://clang.llvm.org/docs/ShadowCallStack.html Note that security guarantees in the kernel differ from the ones documented for user space. The kernel must store addresses of shadow stacks used by other tasks and interrupt handlers in memory, which means an attacker capable reading and writing arbitrary memory may be able to locate them and hijack control flow by modifying shadow stacks that are not currently in use. Signed-off-by: Sami Tolvanen Reviewed-by: Kees Cook Reviewed-by: Miguel Ojeda --- Makefile | 6 ++ arch/Kconfig | 33 +++++++ include/linux/compiler-clang.h | 6 ++ include/linux/compiler_types.h | 4 + include/linux/scs.h | 57 +++++++++++ init/init_task.c | 8 ++ kernel/Makefile | 1 + kernel/fork.c | 9 ++ kernel/sched/core.c | 2 + kernel/sched/sched.h | 1 + kernel/scs.c | 169 +++++++++++++++++++++++++++++++++ 11 files changed, 296 insertions(+) create mode 100644 include/linux/scs.h create mode 100644 kernel/scs.c diff --git a/Makefile b/Makefile index 79be70bf2899..e6337314f8fb 100644 --- a/Makefile +++ b/Makefile @@ -846,6 +846,12 @@ ifdef CONFIG_LIVEPATCH KBUILD_CFLAGS += $(call cc-option, -flive-patching=inline-clone) endif +ifdef CONFIG_SHADOW_CALL_STACK +CC_FLAGS_SCS := -fsanitize=shadow-call-stack +KBUILD_CFLAGS += $(CC_FLAGS_SCS) +export CC_FLAGS_SCS +endif + # arch Makefile may override CC so keep this after arch Makefile is included NOSTDINC_FLAGS += -nostdinc -isystem $(shell $(CC) -print-file-name=include) diff --git a/arch/Kconfig b/arch/Kconfig index 5f8a5d84dbbe..5e34cbcd8d6a 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -521,6 +521,39 @@ config STACKPROTECTOR_STRONG about 20% of all kernel functions, which increases the kernel code size by about 2%. +config ARCH_SUPPORTS_SHADOW_CALL_STACK + bool + help + An architecture should select this if it supports Clang's Shadow + Call Stack, has asm/scs.h, and implements runtime support for shadow + stack switching. + +config SHADOW_CALL_STACK_VMAP + bool + depends on SHADOW_CALL_STACK + help + Use virtually mapped shadow call stacks. Selecting this option + provides better stack exhaustion protection, but increases per-thread + memory consumption as a full page is allocated for each shadow stack. + +config SHADOW_CALL_STACK + bool "Clang Shadow Call Stack" + depends on ARCH_SUPPORTS_SHADOW_CALL_STACK + help + This option enables Clang's Shadow Call Stack, which uses a + shadow stack to protect function return addresses from being + overwritten by an attacker. More information can be found from + Clang's documentation: + + https://clang.llvm.org/docs/ShadowCallStack.html + + Note that security guarantees in the kernel differ from the ones + documented for user space. The kernel must store addresses of shadow + stacks used by other tasks and interrupt handlers in memory, which + means an attacker capable reading and writing arbitrary memory may + be able to locate them and hijack control flow by modifying shadow + stacks that are not currently in use. + config HAVE_ARCH_WITHIN_STACK_FRAMES bool help diff --git a/include/linux/compiler-clang.h b/include/linux/compiler-clang.h index 333a6695a918..18fc4d29ef27 100644 --- a/include/linux/compiler-clang.h +++ b/include/linux/compiler-clang.h @@ -42,3 +42,9 @@ * compilers, like ICC. */ #define barrier() __asm__ __volatile__("" : : : "memory") + +#if __has_feature(shadow_call_stack) +# define __noscs __attribute__((__no_sanitize__("shadow-call-stack"))) +#else +# define __noscs +#endif diff --git a/include/linux/compiler_types.h b/include/linux/compiler_types.h index 72393a8c1a6c..be5d5be4b1ae 100644 --- a/include/linux/compiler_types.h +++ b/include/linux/compiler_types.h @@ -202,6 +202,10 @@ struct ftrace_likely_data { # define randomized_struct_fields_end #endif +#ifndef __noscs +# define __noscs +#endif + #ifndef asm_volatile_goto #define asm_volatile_goto(x...) asm goto(x) #endif diff --git a/include/linux/scs.h b/include/linux/scs.h new file mode 100644 index 000000000000..bd5ef4278b91 --- /dev/null +++ b/include/linux/scs.h @@ -0,0 +1,57 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Shadow Call Stack support. + * + * Copyright (C) 2019 Google LLC + */ + +#ifndef _LINUX_SCS_H +#define _LINUX_SCS_H + +#include +#include +#include + +#ifdef CONFIG_SHADOW_CALL_STACK + +/* + * In testing, 1 KiB shadow stack size (i.e. 128 stack frames on a 64-bit + * architecture) provided ~40% safety margin on stack usage while keeping + * memory allocation overhead reasonable. + */ +#define SCS_SIZE 1024 +#define GFP_SCS (GFP_KERNEL | __GFP_ZERO) + +/* + * A random number outside the kernel's virtual address space to mark the + * end of the shadow stack. + */ +#define SCS_END_MAGIC 0xaf0194819b1635f6UL + +#define task_scs(tsk) (task_thread_info(tsk)->shadow_call_stack) + +static inline void task_set_scs(struct task_struct *tsk, void *s) +{ + task_scs(tsk) = s; +} + +extern void scs_init(void); +extern void scs_task_reset(struct task_struct *tsk); +extern int scs_prepare(struct task_struct *tsk, int node); +extern bool scs_corrupted(struct task_struct *tsk); +extern void scs_release(struct task_struct *tsk); + +#else /* CONFIG_SHADOW_CALL_STACK */ + +#define task_scs(tsk) NULL + +static inline void task_set_scs(struct task_struct *tsk, void *s) {} +static inline void scs_init(void) {} +static inline void scs_task_reset(struct task_struct *tsk) {} +static inline int scs_prepare(struct task_struct *tsk, int node) { return 0; } +static inline bool scs_corrupted(struct task_struct *tsk) { return false; } +static inline void scs_release(struct task_struct *tsk) {} + +#endif /* CONFIG_SHADOW_CALL_STACK */ + +#endif /* _LINUX_SCS_H */ diff --git a/init/init_task.c b/init/init_task.c index 9e5cbe5eab7b..cbd40460e903 100644 --- a/init/init_task.c +++ b/init/init_task.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include @@ -184,6 +185,13 @@ struct task_struct init_task }; EXPORT_SYMBOL(init_task); +#ifdef CONFIG_SHADOW_CALL_STACK +unsigned long init_shadow_call_stack[SCS_SIZE / sizeof(long)] __init_task_data + __aligned(SCS_SIZE) = { + [(SCS_SIZE / sizeof(long)) - 1] = SCS_END_MAGIC +}; +#endif + /* * Initial thread structure. Alignment of this is handled by a special * linker map entry. diff --git a/kernel/Makefile b/kernel/Makefile index daad787fb795..313dbd44d576 100644 --- a/kernel/Makefile +++ b/kernel/Makefile @@ -102,6 +102,7 @@ obj-$(CONFIG_TRACEPOINTS) += trace/ obj-$(CONFIG_IRQ_WORK) += irq_work.o obj-$(CONFIG_CPU_PM) += cpu_pm.o obj-$(CONFIG_BPF) += bpf/ +obj-$(CONFIG_SHADOW_CALL_STACK) += scs.o obj-$(CONFIG_PERF_EVENTS) += events/ diff --git a/kernel/fork.c b/kernel/fork.c index bcdf53125210..3fa7ba64c62d 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -94,6 +94,7 @@ #include #include #include +#include #include #include @@ -451,6 +452,8 @@ void put_task_stack(struct task_struct *tsk) void free_task(struct task_struct *tsk) { + scs_release(tsk); + #ifndef CONFIG_THREAD_INFO_IN_TASK /* * The task is finally done with both the stack and thread_info, @@ -834,6 +837,8 @@ void __init fork_init(void) NULL, free_vm_stack_cache); #endif + scs_init(); + lockdep_init_task(&init_task); uprobes_init(); } @@ -893,6 +898,10 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node) if (err) goto free_stack; + err = scs_prepare(tsk, node); + if (err) + goto free_stack; + #ifdef CONFIG_SECCOMP /* * We must handle setting up seccomp filters once we're under diff --git a/kernel/sched/core.c b/kernel/sched/core.c index dd05a378631a..e7faeb383008 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6013,6 +6013,8 @@ void init_idle(struct task_struct *idle, int cpu) raw_spin_lock_irqsave(&idle->pi_lock, flags); raw_spin_lock(&rq->lock); + scs_task_reset(idle); + __sched_fork(0, idle); idle->state = TASK_RUNNING; idle->se.exec_start = sched_clock(); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 0db2c1b3361e..c153003a011c 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -58,6 +58,7 @@ #include #include #include +#include #include #include #include diff --git a/kernel/scs.c b/kernel/scs.c new file mode 100644 index 000000000000..7c1a40020754 --- /dev/null +++ b/kernel/scs.c @@ -0,0 +1,169 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Shadow Call Stack support. + * + * Copyright (C) 2019 Google LLC + */ + +#include +#include +#include +#include +#include +#include +#include + +static inline void *__scs_base(struct task_struct *tsk) +{ + /* + * We allow architectures to use the shadow_call_stack field in + * struct thread_info to store the current shadow stack pointer + * during context switches. + * + * This allows the implementation to also clear the field when + * the task is active to avoid keeping pointers to the current + * task's shadow stack in memory. This can make it harder for an + * attacker to locate the shadow stack, but also requires us to + * compute the base address when needed. + * + * We assume the stack is aligned to SCS_SIZE. + */ + return (void *)((uintptr_t)task_scs(tsk) & ~(SCS_SIZE - 1)); +} + +#ifdef CONFIG_SHADOW_CALL_STACK_VMAP + +/* Keep a cache of shadow stacks */ +#define SCS_CACHE_SIZE 2 +static DEFINE_PER_CPU(void *, scs_cache[SCS_CACHE_SIZE]); + +static void *scs_alloc(int node) +{ + int i; + + for (i = 0; i < SCS_CACHE_SIZE; i++) { + void *s; + + s = this_cpu_xchg(scs_cache[i], NULL); + if (s) { + memset(s, 0, SCS_SIZE); + return s; + } + } + + /* + * We allocate a full page for the shadow stack, which should be + * more than we need. Check the assumption nevertheless. + */ + BUILD_BUG_ON(SCS_SIZE > PAGE_SIZE); + + return __vmalloc_node_range(PAGE_SIZE, SCS_SIZE, + VMALLOC_START, VMALLOC_END, + GFP_SCS, PAGE_KERNEL, 0, + node, __builtin_return_address(0)); +} + +static void scs_free(void *s) +{ + int i; + + for (i = 0; i < SCS_CACHE_SIZE; i++) + if (this_cpu_cmpxchg(scs_cache[i], 0, s) == 0) + return; + + vfree_atomic(s); +} + +static int scs_cleanup(unsigned int cpu) +{ + int i; + void **cache = per_cpu_ptr(scs_cache, cpu); + + for (i = 0; i < SCS_CACHE_SIZE; i++) { + vfree(cache[i]); + cache[i] = NULL; + } + + return 0; +} + +void __init scs_init(void) +{ + cpuhp_setup_state(CPUHP_BP_PREPARE_DYN, "scs:scs_cache", NULL, + scs_cleanup); +} + +#else /* !CONFIG_SHADOW_CALL_STACK_VMAP */ + +static struct kmem_cache *scs_cache; + +static inline void *scs_alloc(int node) +{ + return kmem_cache_alloc_node(scs_cache, GFP_SCS, node); +} + +static inline void scs_free(void *s) +{ + kmem_cache_free(scs_cache, s); +} + +void __init scs_init(void) +{ + scs_cache = kmem_cache_create("scs_cache", SCS_SIZE, SCS_SIZE, + 0, NULL); + WARN_ON(!scs_cache); +} + +#endif /* CONFIG_SHADOW_CALL_STACK_VMAP */ + +static inline unsigned long *scs_magic(struct task_struct *tsk) +{ + return (unsigned long *)(__scs_base(tsk) + SCS_SIZE) - 1; +} + +static inline void scs_set_magic(struct task_struct *tsk) +{ + *scs_magic(tsk) = SCS_END_MAGIC; +} + +void scs_task_reset(struct task_struct *tsk) +{ + /* + * Reset the shadow stack to the base address in case the task + * is reused. + */ + task_set_scs(tsk, __scs_base(tsk)); +} + +int scs_prepare(struct task_struct *tsk, int node) +{ + void *s; + + s = scs_alloc(node); + if (!s) + return -ENOMEM; + + task_set_scs(tsk, s); + scs_set_magic(tsk); + + return 0; +} + +bool scs_corrupted(struct task_struct *tsk) +{ + return *scs_magic(tsk) != SCS_END_MAGIC; +} + +void scs_release(struct task_struct *tsk) +{ + void *s; + + s = __scs_base(tsk); + if (!s) + return; + + WARN_ON(scs_corrupted(tsk)); + + task_set_scs(tsk, NULL); + scs_free(s); +} From patchwork Fri Nov 1 22:11:39 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 11223829 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E75C11668 for ; Fri, 1 Nov 2019 22:14:00 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AFFA721897 for ; Fri, 1 Nov 2019 22:14:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="fgz10k7O"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="lcoMeZZt" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AFFA721897 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=zW0wNC9IaW5WQrzqwVrFnKNhXGtvzXxPou3D7JKJrek=; b=fgz10k7OAsmdYR v0fE+7acNKQtjkBw1xh+ZO7JOgA6WUopnnAau6zts/xAxtXMxcNfX0eCWkdG1RoQ9J5Gnr3TmjZJl a2CRvnnvSyxOwC0/virKbhHCGUbrBfFkxw55/4wb53DOdvebQoAX5XISPjM77oA38kofHFuIwpZie jH9l4cDPRH3i8u/jELeRPRvnTq20PLsQQYGP1SZ+DDQ/Uk7PtY9W5bcFB7cMqFJH3lZR48VbD7j1r 4ZD0ijAdemk87pe3Sksp1Q5U2OeV7X4cz40oEmpXlN57Qs0aVkVQbsxV63BCpDrXnaKqTrA/1QRn+ yk92xtHcNAz2ZZBUQOpA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQfB7-0001A7-NF; Fri, 01 Nov 2019 22:13:57 +0000 Received: from mail-pl1-x649.google.com ([2607:f8b0:4864:20::649]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQf9R-0007t9-Hy for linux-arm-kernel@lists.infradead.org; Fri, 01 Nov 2019 22:12:15 +0000 Received: by mail-pl1-x649.google.com with SMTP id x8so7146482plo.17 for ; Fri, 01 Nov 2019 15:12:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=LwwDQXrIVlzkqkUkZpiUVrrf7w5cRtF+1klTx8yKO9A=; b=lcoMeZZtwO2UeiAw6gwHOcpuzrLjJt/z961haRlQmmwGIb/uUV2rh0vaq632ZkXbV7 hut87q9satXc/Kx0j37VBXcIzpYq0YrYOme0xR8/v1v4oO0bR59ntdRjOoXTNoTfhAk7 iZsFkCCGM+Q1RWqkgVBDvNxfUIb+npHcqmP182zf+X9fhJaudGjB9a4fr/gxLfc86XMe akBm2CQiulJwSDGwkU1vqHwF8dx1T2qjF/kankr80Vz0fQZngw/bwnJp6GsbEQALRndI UY9fwOIy7+hrcoVyQRL7Q3msR5yhW1tBagHCfCjFXPcIFkOHRX4YqxAl19nxkpR+b6SO C6Gg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=LwwDQXrIVlzkqkUkZpiUVrrf7w5cRtF+1klTx8yKO9A=; b=EC4/qrglEM5tmyPUUHXJk9qWXR5pb7kYwotRK914mPtjTnGi3uWPfQSanbAe8Xy443 9mvExZZ5eydhBx4MMXDoipNcbAxhT65Pa+4wca7Ei9OPNcANLO7CJNG9ung0BdoIFiJf wYlzdORp7AkmS4GcWgDNFxedqEAsC8D0bqkYUnzzLZHDkZUzLwyE5xvfg47OvWubl2Zy pGAr6+ygVHkdPDWFmn+fGicmrQdno6QOceM+fnmwZRSuZjBgVES/g5us3MUoavt7RZBG BDeENlafJjATKXsw5RmlCrksF1WmSIRoeqIEOyWHo2W5f9XmxsVDu0F1NrGPD3/JMMSY 8R8Q== X-Gm-Message-State: APjAAAX/9/HuLHtNkQ1qxMgm6oOiSB6iPKyVSUkbvaYwhkMDRshXyU9M Ecdj2uDftbhhi9zTv8mHcu540NeCwarfMxI/TV8= X-Google-Smtp-Source: APXvYqwGEhPTT/37Fn+hD11OHFroFTMLSG+ji/6EXx0DVVani+DjabSvz02NfmPZFs5Q+FFELHaCTGTgVsX8f2KrSBM= X-Received: by 2002:a63:934d:: with SMTP id w13mr15550340pgm.185.1572646330528; Fri, 01 Nov 2019 15:12:10 -0700 (PDT) Date: Fri, 1 Nov 2019 15:11:39 -0700 In-Reply-To: <20191101221150.116536-1-samitolvanen@google.com> Message-Id: <20191101221150.116536-7-samitolvanen@google.com> Mime-Version: 1.0 References: <20191018161033.261971-1-samitolvanen@google.com> <20191101221150.116536-1-samitolvanen@google.com> X-Mailer: git-send-email 2.24.0.rc1.363.gb1bccd3e3d-goog Subject: [PATCH v4 06/17] scs: add accounting From: Sami Tolvanen To: Will Deacon , Catalin Marinas , Steven Rostedt , Masami Hiramatsu , Ard Biesheuvel X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20191101_151213_647842_10230087 X-CRM114-Status: GOOD ( 12.63 ) X-Spam-Score: -7.7 (-------) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (-7.7 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:649 listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM white-list -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.0 DKIMWL_WL_MED DKIMwl.org - Medium sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Kees Cook , Jann Horn , Masahiro Yamada , Marc Zyngier , kernel-hardening@lists.openwall.com, Nick Desaulniers , linux-kernel@vger.kernel.org, Miguel Ojeda , clang-built-linux@googlegroups.com, Sami Tolvanen , Laura Abbott , Dave Martin , linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org This change adds accounting for the memory allocated for shadow stacks. Signed-off-by: Sami Tolvanen Reviewed-by: Kees Cook --- drivers/base/node.c | 6 ++++++ fs/proc/meminfo.c | 4 ++++ include/linux/mmzone.h | 3 +++ kernel/scs.c | 19 +++++++++++++++++++ mm/page_alloc.c | 6 ++++++ mm/vmstat.c | 3 +++ 6 files changed, 41 insertions(+) diff --git a/drivers/base/node.c b/drivers/base/node.c index 296546ffed6c..111e58ec231e 100644 --- a/drivers/base/node.c +++ b/drivers/base/node.c @@ -415,6 +415,9 @@ static ssize_t node_read_meminfo(struct device *dev, "Node %d AnonPages: %8lu kB\n" "Node %d Shmem: %8lu kB\n" "Node %d KernelStack: %8lu kB\n" +#ifdef CONFIG_SHADOW_CALL_STACK + "Node %d ShadowCallStack:%8lu kB\n" +#endif "Node %d PageTables: %8lu kB\n" "Node %d NFS_Unstable: %8lu kB\n" "Node %d Bounce: %8lu kB\n" @@ -438,6 +441,9 @@ static ssize_t node_read_meminfo(struct device *dev, nid, K(node_page_state(pgdat, NR_ANON_MAPPED)), nid, K(i.sharedram), nid, sum_zone_node_page_state(nid, NR_KERNEL_STACK_KB), +#ifdef CONFIG_SHADOW_CALL_STACK + nid, sum_zone_node_page_state(nid, NR_KERNEL_SCS_BYTES) / 1024, +#endif nid, K(sum_zone_node_page_state(nid, NR_PAGETABLE)), nid, K(node_page_state(pgdat, NR_UNSTABLE_NFS)), nid, K(sum_zone_node_page_state(nid, NR_BOUNCE)), diff --git a/fs/proc/meminfo.c b/fs/proc/meminfo.c index 8c1f1bb1a5ce..49768005a79e 100644 --- a/fs/proc/meminfo.c +++ b/fs/proc/meminfo.c @@ -103,6 +103,10 @@ static int meminfo_proc_show(struct seq_file *m, void *v) show_val_kb(m, "SUnreclaim: ", sunreclaim); seq_printf(m, "KernelStack: %8lu kB\n", global_zone_page_state(NR_KERNEL_STACK_KB)); +#ifdef CONFIG_SHADOW_CALL_STACK + seq_printf(m, "ShadowCallStack:%8lu kB\n", + global_zone_page_state(NR_KERNEL_SCS_BYTES) / 1024); +#endif show_val_kb(m, "PageTables: ", global_zone_page_state(NR_PAGETABLE)); diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index bda20282746b..fcb8c1708f9e 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -200,6 +200,9 @@ enum zone_stat_item { NR_MLOCK, /* mlock()ed pages found and moved off LRU */ NR_PAGETABLE, /* used for pagetables */ NR_KERNEL_STACK_KB, /* measured in KiB */ +#if IS_ENABLED(CONFIG_SHADOW_CALL_STACK) + NR_KERNEL_SCS_BYTES, /* measured in bytes */ +#endif /* Second 128 byte cacheline */ NR_BOUNCE, #if IS_ENABLED(CONFIG_ZSMALLOC) diff --git a/kernel/scs.c b/kernel/scs.c index 7c1a40020754..7780fc4e29ac 100644 --- a/kernel/scs.c +++ b/kernel/scs.c @@ -11,6 +11,7 @@ #include #include #include +#include #include static inline void *__scs_base(struct task_struct *tsk) @@ -74,6 +75,11 @@ static void scs_free(void *s) vfree_atomic(s); } +static struct page *__scs_page(struct task_struct *tsk) +{ + return vmalloc_to_page(__scs_base(tsk)); +} + static int scs_cleanup(unsigned int cpu) { int i; @@ -107,6 +113,11 @@ static inline void scs_free(void *s) kmem_cache_free(scs_cache, s); } +static struct page *__scs_page(struct task_struct *tsk) +{ + return virt_to_page(__scs_base(tsk)); +} + void __init scs_init(void) { scs_cache = kmem_cache_create("scs_cache", SCS_SIZE, SCS_SIZE, @@ -135,6 +146,12 @@ void scs_task_reset(struct task_struct *tsk) task_set_scs(tsk, __scs_base(tsk)); } +static void scs_account(struct task_struct *tsk, int account) +{ + mod_zone_page_state(page_zone(__scs_page(tsk)), NR_KERNEL_SCS_BYTES, + account * SCS_SIZE); +} + int scs_prepare(struct task_struct *tsk, int node) { void *s; @@ -145,6 +162,7 @@ int scs_prepare(struct task_struct *tsk, int node) task_set_scs(tsk, s); scs_set_magic(tsk); + scs_account(tsk, 1); return 0; } @@ -164,6 +182,7 @@ void scs_release(struct task_struct *tsk) WARN_ON(scs_corrupted(tsk)); + scs_account(tsk, -1); task_set_scs(tsk, NULL); scs_free(s); } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index ecc3dbad606b..fe17d69d98a7 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -5361,6 +5361,9 @@ void show_free_areas(unsigned int filter, nodemask_t *nodemask) " managed:%lukB" " mlocked:%lukB" " kernel_stack:%lukB" +#ifdef CONFIG_SHADOW_CALL_STACK + " shadow_call_stack:%lukB" +#endif " pagetables:%lukB" " bounce:%lukB" " free_pcp:%lukB" @@ -5382,6 +5385,9 @@ void show_free_areas(unsigned int filter, nodemask_t *nodemask) K(zone_managed_pages(zone)), K(zone_page_state(zone, NR_MLOCK)), zone_page_state(zone, NR_KERNEL_STACK_KB), +#ifdef CONFIG_SHADOW_CALL_STACK + zone_page_state(zone, NR_KERNEL_SCS_BYTES) / 1024, +#endif K(zone_page_state(zone, NR_PAGETABLE)), K(zone_page_state(zone, NR_BOUNCE)), K(free_pcp), diff --git a/mm/vmstat.c b/mm/vmstat.c index 6afc892a148a..9fe4afe670fe 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1118,6 +1118,9 @@ const char * const vmstat_text[] = { "nr_mlock", "nr_page_table_pages", "nr_kernel_stack", +#if IS_ENABLED(CONFIG_SHADOW_CALL_STACK) + "nr_shadow_call_stack_bytes", +#endif "nr_bounce", #if IS_ENABLED(CONFIG_ZSMALLOC) "nr_zspages", From patchwork Fri Nov 1 22:11:40 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 11223833 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CFC0C1668 for ; Fri, 1 Nov 2019 22:14:14 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6F6F1217D9 for ; Fri, 1 Nov 2019 22:14:14 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="sLNiC42h"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="TUOCiu9g" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6F6F1217D9 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=laVY+pxokHIUVlCpyho9t2hkqHdeJJBmH//J3vGTIGI=; b=sLNiC42hgAx/b4 NofGu5kLRHoMG2GYDDt+S5Pt6UbM0/kFr96aEPdb6/N3vbKWmZ4Q/X0C6yGcumuKBQy3T7wWtZHWB hNs+SdztHctvnrCn2eRBnbmYBAuX4ZBzQfOTFDTOFoEd+jt4n6kzas1I3U6ghlM9kIdwGOeuwFhKB RKtDciYGfOhLLkT/z4APpb/3FaLVJUMoW4uGl5Rm1Dh7bzjfC7dBdkEa1KaT1z2Avl6llVwZwINTD 98QXGG/3/h2csPvSSnOUXl4ytoJCfu7D15HOn6/zfR1LmgREyCDBbVIS+pqSP56dS0BO0Ct6gwPbP r4mxTOX2DWjnhsNgka5A==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQfBN-0001Qy-6c; Fri, 01 Nov 2019 22:14:13 +0000 Received: from mail-pf1-x449.google.com ([2607:f8b0:4864:20::449]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQf9T-0007vm-Hv for linux-arm-kernel@lists.infradead.org; Fri, 01 Nov 2019 22:12:17 +0000 Received: by mail-pf1-x449.google.com with SMTP id i6so8428963pfd.14 for ; Fri, 01 Nov 2019 15:12:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=bw/qAOzuJX7ixoBqz2PQ7jd7OYMX5ZXLMotdDh+Bihs=; b=TUOCiu9gLsLFy3Xp5OD9ee3T44RBZ9XBgIe7yt7CvNj98C6CBu9wAFiKO2vk8DcyhT Ji59uJo30gvV4E1Mc6+k9TAjV16C8IiKMXK7G0oVHeV/PAz2/fCEaRJcdINYSGhgESYR uoDTe09Jv1qncBGnVBU6OT6bb8ZxBNJKvMf1wcI0ooElnCm2FCe3Y7+zQ6a97/j39xPN Mu+yrOgb1hl2vY4VIkbI6qMmyB8qwY3HwJfeslpY+sqKd7Pkx4hNMy+qAjpsXmuINSXH Hbqb76hzhJWPhHLLM3oT+0GxKRDzWsCq1m6N3+XuHnyEuX9+4K1XJa7mKQznVdZHeHhv 741g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=bw/qAOzuJX7ixoBqz2PQ7jd7OYMX5ZXLMotdDh+Bihs=; b=pjYTul+rQIaFJSSr5VaWtI6Fjezsy/MZCZZ0IjLcGgeNENdRs7LjttJDiYk/3ha6j/ j1l4URg3vXWPqf403xNuH07qnNTTn99nVt3yL4fcZBe5PvwhSyFYdyJMMKJbnqnT8n4n QEPDV++EtEprbJf+/7xP5pKz3lGGNrEg2/2ttvhz8jJVz1DWmDlxXGmahtH4hLWm0es6 p0BQE8UPMSNgMPuPjlXga6C3eyMO0eyvUUoWpfBR/SnSxNwSO0rK20pyEaunijxMOWrx laHbJvyI/lrbQommQEiwvesg653UFGjhwOqi3R/NMIGOfQc0A/4BaiEZd0arT++Xl+gr Q6hA== X-Gm-Message-State: APjAAAVA5I00YE7f8sE9m+mH0XKxLeWUy72KAVRkKdaMy4SfGjVwLA4h CRHB8Qtk/pq9gF3AeX7PCGaPrkMtkSobIrFPkbI= X-Google-Smtp-Source: APXvYqx5AqsiV5gsrjTxTPakMbZqpZkpOWKFfa4kZ0m+C6pXJ6/DO3wHTK1s5AzNd4GAvsVbgqAkyBlJLNLHZ9Ye4kg= X-Received: by 2002:a63:5762:: with SMTP id h34mr16176849pgm.235.1572646333610; Fri, 01 Nov 2019 15:12:13 -0700 (PDT) Date: Fri, 1 Nov 2019 15:11:40 -0700 In-Reply-To: <20191101221150.116536-1-samitolvanen@google.com> Message-Id: <20191101221150.116536-8-samitolvanen@google.com> Mime-Version: 1.0 References: <20191018161033.261971-1-samitolvanen@google.com> <20191101221150.116536-1-samitolvanen@google.com> X-Mailer: git-send-email 2.24.0.rc1.363.gb1bccd3e3d-goog Subject: [PATCH v4 07/17] scs: add support for stack usage debugging From: Sami Tolvanen To: Will Deacon , Catalin Marinas , Steven Rostedt , Masami Hiramatsu , Ard Biesheuvel X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20191101_151215_638350_39209C08 X-CRM114-Status: GOOD ( 10.37 ) X-Spam-Score: -7.7 (-------) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (-7.7 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:449 listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM white-list -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.0 DKIMWL_WL_MED DKIMwl.org - Medium sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Kees Cook , Jann Horn , Masahiro Yamada , Marc Zyngier , kernel-hardening@lists.openwall.com, Nick Desaulniers , linux-kernel@vger.kernel.org, Miguel Ojeda , clang-built-linux@googlegroups.com, Sami Tolvanen , Laura Abbott , Dave Martin , linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Implements CONFIG_DEBUG_STACK_USAGE for shadow stacks. When enabled, also prints out the highest shadow stack usage per process. Signed-off-by: Sami Tolvanen Reviewed-by: Kees Cook --- kernel/scs.c | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/kernel/scs.c b/kernel/scs.c index 7780fc4e29ac..67c43af627d1 100644 --- a/kernel/scs.c +++ b/kernel/scs.c @@ -167,6 +167,44 @@ int scs_prepare(struct task_struct *tsk, int node) return 0; } +#ifdef CONFIG_DEBUG_STACK_USAGE +static inline unsigned long scs_used(struct task_struct *tsk) +{ + unsigned long *p = __scs_base(tsk); + unsigned long *end = scs_magic(tsk); + uintptr_t s = (uintptr_t)p; + + while (p < end && *p) + p++; + + return (uintptr_t)p - s; +} + +static void scs_check_usage(struct task_struct *tsk) +{ + static DEFINE_SPINLOCK(lock); + static unsigned long highest; + unsigned long used = scs_used(tsk); + + if (used <= highest) + return; + + spin_lock(&lock); + + if (used > highest) { + pr_info("%s: highest shadow stack usage %lu bytes\n", + __func__, used); + highest = used; + } + + spin_unlock(&lock); +} +#else +static inline void scs_check_usage(struct task_struct *tsk) +{ +} +#endif + bool scs_corrupted(struct task_struct *tsk) { return *scs_magic(tsk) != SCS_END_MAGIC; @@ -181,6 +219,7 @@ void scs_release(struct task_struct *tsk) return; WARN_ON(scs_corrupted(tsk)); + scs_check_usage(tsk); scs_account(tsk, -1); task_set_scs(tsk, NULL); From patchwork Fri Nov 1 22:11:41 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 11223841 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8E9C01668 for ; Fri, 1 Nov 2019 22:14:36 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 576652085B for ; Fri, 1 Nov 2019 22:14:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="jbuHdi/p"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="VqPASQjK" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 576652085B Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=hpoJsssApv80R2RYBA2ooZGPrdam1lYzn7TqDfCosgs=; b=jbuHdi/ptcPbep RlUI5gYkIXybLOoCCFXpu42digBE1x2OqRnxluTyNH99revoM+eiJOZkrSNZDfRlxp40vDSgzhSVH /tE8rhXhXobDqqKe7a4W2KMIv3+Gn4H4seVsaFkLsA8tz0QtUD2+PHmSaKePKuuHvZFfhs7JSVRT6 2+qds95+38qssmMpzAVqYHxgQPGnk9Ds+xeekqCGMhumEXH0U1y18sL2redRGFyolN8sNWX6hJAKN DQ75OpeRVdXyzdf3PJ3eaESFMQU8n55gEdXJXwX3NnKf20qwanpeJaqUJVwencznt8beB5cXWBcIi hDm9vMgZSx+3ZLl9lzIg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQfBf-0001iY-6U; Fri, 01 Nov 2019 22:14:31 +0000 Received: from mail-pf1-x44a.google.com ([2607:f8b0:4864:20::44a]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQf9V-0007yp-O2 for linux-arm-kernel@lists.infradead.org; Fri, 01 Nov 2019 22:12:19 +0000 Received: by mail-pf1-x44a.google.com with SMTP id i6so8429035pfd.14 for ; Fri, 01 Nov 2019 15:12:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=lxookr95uMT9sOw8S1HRxQFBJ0wSUMPDNTYBScBbdzA=; b=VqPASQjKWMJdAAeJVcDeaVBQprxOwcKwHm7RWTEH9a4e15mPmrxMbW1NmgWy95nCCG gTOzFuhp1ly0goL38Kzph/3B6Z/grLfH43Q8d/pdkZ/OzdVRcdb/FYfbWEa2y3Z36z+b aTHaRyLhmFGkSCynkV7G0QI+cwcWnjK1qpx31PYhN1zT7bAET6McDo/Y5fThXHdO9Ek7 zYHqR539+PQJrPPLRkOE7jflC+yHuXbTJUVLQrKHrysQdsbsiJzOBJ0HDbVrb0s28p6t kflfPrlWaRQdM1RMKSTS9mn+78aoixWKDx0joRXcRCUOTMxiomPR6k+4etm9NwgxSA31 215g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=lxookr95uMT9sOw8S1HRxQFBJ0wSUMPDNTYBScBbdzA=; b=NMItMeW6YUqGgDBE7VQ2vJ1muLoer3jEzcJsq5lW951uNH4qlPYUbkd9fSrRtOl27/ 73DL3TMuif10G2cDfPfTQI//rKychtbBdTAOwoRbSbgp1VwOo1RhESFn0avM9RM6stSh zx0V+sC2ZFnvRHrRoO2OfCiM1xhxUYPOkk2FTU6goo8JPR5Te0PHcJupwmdXxhBeD+v/ AbUUeNlawyB67UyI/UBcmezpmfeIgF/ZkMcORmwEN+KOy/naDya7Il00g3IqHGMs2abM DUEOTZZl3tHj6e88q76tQMxS0io2md18oGfzVc98yWOr/3i6AylJoJOTq4kh5cDP9M7v VeLQ== X-Gm-Message-State: APjAAAVM289oTGnQVt9fnZG+hXWW+ae3PeREnEscpkkVGG6+M2c0Sr9L YVupGVp7FkvaLuvsvLdPgxPrcBYOQHupknQ7/I8= X-Google-Smtp-Source: APXvYqy5WQVZIH4BmgA2vfXXE0nLc6aKF5v7DRRaYQ9YrQPB5qUQFSJiOmM2MmnfKMB9bJuYwXz6LVby2vjMPQ9mbfI= X-Received: by 2002:a63:d306:: with SMTP id b6mr15679209pgg.338.1572646336234; Fri, 01 Nov 2019 15:12:16 -0700 (PDT) Date: Fri, 1 Nov 2019 15:11:41 -0700 In-Reply-To: <20191101221150.116536-1-samitolvanen@google.com> Message-Id: <20191101221150.116536-9-samitolvanen@google.com> Mime-Version: 1.0 References: <20191018161033.261971-1-samitolvanen@google.com> <20191101221150.116536-1-samitolvanen@google.com> X-Mailer: git-send-email 2.24.0.rc1.363.gb1bccd3e3d-goog Subject: [PATCH v4 08/17] kprobes: fix compilation without CONFIG_KRETPROBES From: Sami Tolvanen To: Will Deacon , Catalin Marinas , Steven Rostedt , Masami Hiramatsu , Ard Biesheuvel X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20191101_151217_950863_617C854D X-CRM114-Status: GOOD ( 10.93 ) X-Spam-Score: -7.7 (-------) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (-7.7 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:44a listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM white-list -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.0 DKIMWL_WL_MED DKIMwl.org - Medium sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Kees Cook , Jann Horn , Masahiro Yamada , Marc Zyngier , kernel-hardening@lists.openwall.com, Nick Desaulniers , linux-kernel@vger.kernel.org, Miguel Ojeda , clang-built-linux@googlegroups.com, Sami Tolvanen , Laura Abbott , Dave Martin , linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org kprobe_on_func_entry and arch_kprobe_on_func_entry need to be available even if CONFIG_KRETPROBES is not selected. Signed-off-by: Sami Tolvanen Acked-by: Masami Hiramatsu Reviewed-by: Kees Cook Acked-by: Steven Rostedt (VMware) --- kernel/kprobes.c | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/kernel/kprobes.c b/kernel/kprobes.c index 53534aa258a6..b5e20a4669b8 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -1829,6 +1829,25 @@ unsigned long __weak arch_deref_entry_point(void *entry) return (unsigned long)entry; } +bool __weak arch_kprobe_on_func_entry(unsigned long offset) +{ + return !offset; +} + +bool kprobe_on_func_entry(kprobe_opcode_t *addr, const char *sym, unsigned long offset) +{ + kprobe_opcode_t *kp_addr = _kprobe_addr(addr, sym, offset); + + if (IS_ERR(kp_addr)) + return false; + + if (!kallsyms_lookup_size_offset((unsigned long)kp_addr, NULL, &offset) || + !arch_kprobe_on_func_entry(offset)) + return false; + + return true; +} + #ifdef CONFIG_KRETPROBES /* * This kprobe pre_handler is registered with every kretprobe. When probe @@ -1885,25 +1904,6 @@ static int pre_handler_kretprobe(struct kprobe *p, struct pt_regs *regs) } NOKPROBE_SYMBOL(pre_handler_kretprobe); -bool __weak arch_kprobe_on_func_entry(unsigned long offset) -{ - return !offset; -} - -bool kprobe_on_func_entry(kprobe_opcode_t *addr, const char *sym, unsigned long offset) -{ - kprobe_opcode_t *kp_addr = _kprobe_addr(addr, sym, offset); - - if (IS_ERR(kp_addr)) - return false; - - if (!kallsyms_lookup_size_offset((unsigned long)kp_addr, NULL, &offset) || - !arch_kprobe_on_func_entry(offset)) - return false; - - return true; -} - int register_kretprobe(struct kretprobe *rp) { int ret = 0; From patchwork Fri Nov 1 22:11:42 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 11223843 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C4D0614DB for ; Fri, 1 Nov 2019 22:14:43 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8DA6721897 for ; Fri, 1 Nov 2019 22:14:43 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="hqtdT2ru"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="oC92Xyk4" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8DA6721897 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=hhsdEGLG1WjYd1F1gI3KUetOtPvYMBYXvZKv/omyacM=; b=hqtdT2ruWy7gY+ GUvfC29QPekI9UrElTfhQoUQHEEhUJv+2p12hKGO/PEvwk6L/B6WoePS7oP4GCprjli1La407h5Cf lln0aqA+3FuH/3FXwxkZ0eTwYmmzH+iNfIVq9wv8J9pg6Vlu+yZvhwN6ewgoJeTrGoyVuWCY8PWV6 noFeXR5m9GHbkUZ4qddP11vE2bOIMY8mykCyRSCDcbOG0dDTbAWepqxdPTqbYkhmhDJb2uWgI++jc AyI2WKGUYml7rtdtBERLE1lT4euwQ1KLbdp5mBcXVAmoo9SIYOI9rXTh1sZMkEgT0PNblz1lTo9En s1c4QpssTaqFG/hp95xg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQfBq-0001u5-6b; Fri, 01 Nov 2019 22:14:42 +0000 Received: from mail-pl1-x64a.google.com ([2607:f8b0:4864:20::64a]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQf9Y-00081H-Oi for linux-arm-kernel@lists.infradead.org; Fri, 01 Nov 2019 22:12:22 +0000 Received: by mail-pl1-x64a.google.com with SMTP id p14so7168657plq.2 for ; Fri, 01 Nov 2019 15:12:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=YuceP/m4Sv7I7vrWsOPJ1pSb3CxvhwOwQ1OQoGSTS4o=; b=oC92Xyk4xx4OFjCywUW0xRGLKJPopJyks6H3DIWfvMcqM41QRHY8A6HpLSO2oO1inw 7nVZsYb0ejRziH+5dLCL9D9wKeQQKeMEb1VUrkYc1cMPvyFcm3It+tNXG+Hwcj7gshxb 6nHSjR+gi4hodQFSJLJDhZIR1+Kzve8eUQ6q7dVUcbIISZ2N2+mSkGwMJQvydH28gjMd oRMh4SmcetuL5E9614gPeg7eJPENTgrxaRMgcD0aWe9YQNHjd9wnZeZVxQ0kRl59z7of 8623sjz/SPs/pQv7SS4ugqqYKNBb1daNI8KHgf/+dBsRhEhkLc4Q35KQzw9rmfiSpNPD e0Yw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=YuceP/m4Sv7I7vrWsOPJ1pSb3CxvhwOwQ1OQoGSTS4o=; b=ltHgV7aGUDYSdJYqiyWxgJU99vc+cqgONydll7i0Bul+XT3IAoizVhWMgESk5YBqfk T3UCoezJG+uGA1QXLKQtKZiw/DBB3FCwfly2wq0pa0RU4L6foIgDPAEVF5jv4czKlnjB 8MLeicS8Y3va6ntJ9NhYbkFNPKxdKWRM4OPNMcLoPORWrFUQM1I+LJpLiwciau5xiOyP whv5238ILHM3tm5gt+SttCDRKx6ErcbWDH6kWt+yRMzheZN92ydvgf9vbBnQNEh/exQJ gBjLek8lzd6ihjH/5UG2INHHw+ApuBoWzqJC9yUNCfBwEfcVhnSrnG5CdeQbll7qZjs/ HGsA== X-Gm-Message-State: APjAAAXLiDfLZG8Zd5Q05pBV9GbnkxXimvdj0QmSFavvW4TMw3fwqUW3 Tyvgnl6VctOymJsuEs8TY8yyYCoXTGCygA7DY8s= X-Google-Smtp-Source: APXvYqyESI1nj79EQYAfPNp7QT9azRRC93peb8wyxMPp245spxyPTFrWPJ4dhblj2gZ8cJDr8g3hFVKQrrU1shqGWxs= X-Received: by 2002:a63:2b8e:: with SMTP id r136mr2674046pgr.103.1572646338730; Fri, 01 Nov 2019 15:12:18 -0700 (PDT) Date: Fri, 1 Nov 2019 15:11:42 -0700 In-Reply-To: <20191101221150.116536-1-samitolvanen@google.com> Message-Id: <20191101221150.116536-10-samitolvanen@google.com> Mime-Version: 1.0 References: <20191018161033.261971-1-samitolvanen@google.com> <20191101221150.116536-1-samitolvanen@google.com> X-Mailer: git-send-email 2.24.0.rc1.363.gb1bccd3e3d-goog Subject: [PATCH v4 09/17] arm64: kprobes: fix kprobes without CONFIG_KRETPROBES From: Sami Tolvanen To: Will Deacon , Catalin Marinas , Steven Rostedt , Masami Hiramatsu , Ard Biesheuvel X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20191101_151220_820834_E02685CB X-CRM114-Status: UNSURE ( 8.89 ) X-CRM114-Notice: Please train this message. X-Spam-Score: -7.7 (-------) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (-7.7 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:64a listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM white-list -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.0 DKIMWL_WL_MED DKIMwl.org - Medium sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Kees Cook , Jann Horn , Masahiro Yamada , Marc Zyngier , kernel-hardening@lists.openwall.com, Nick Desaulniers , linux-kernel@vger.kernel.org, Miguel Ojeda , clang-built-linux@googlegroups.com, Sami Tolvanen , Laura Abbott , Dave Martin , linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org This allows CONFIG_KRETPROBES to be disabled without disabling kprobes entirely. Signed-off-by: Sami Tolvanen Reviewed-by: Kees Cook --- arch/arm64/kernel/probes/kprobes.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/arm64/kernel/probes/kprobes.c b/arch/arm64/kernel/probes/kprobes.c index c4452827419b..98230ae979ca 100644 --- a/arch/arm64/kernel/probes/kprobes.c +++ b/arch/arm64/kernel/probes/kprobes.c @@ -551,6 +551,7 @@ void __kprobes __used *trampoline_probe_handler(struct pt_regs *regs) return (void *)orig_ret_address; } +#ifdef CONFIG_KRETPROBES void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, struct pt_regs *regs) { @@ -564,6 +565,7 @@ int __kprobes arch_trampoline_kprobe(struct kprobe *p) { return 0; } +#endif int __init arch_init_kprobes(void) { From patchwork Fri Nov 1 22:11:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 11223847 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3B06D14DB for ; Fri, 1 Nov 2019 22:14:58 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1616F2085B for ; Fri, 1 Nov 2019 22:14:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="VJsUzCC6"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="WX+ZlyX9" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1616F2085B Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=NZzhBmRoflFpvwu7TjTVCDX6bmawLdOmKZDLoHM/lbw=; b=VJsUzCC6tyDM8L 0K3netStWclySjAg3gUhIcQxiqRxyqFnRJOlZzsT0fNrpA8phX/g68pkJFoUW5NYLww8O8uMQKOcZ Zh5PDk/UQwP/HnngtR0X+rtOYa0Cp3bEOyLqIv2mxJ3WGFUZu0dxF/t0bOTw/mUjzj84BX4c0ut9L OzH4ttlP+8mcApBmsJgtqH1xoiuJSDOJP3rKj2u0atJu3hK+QNFn2nxmL/nyQGUOnb3xi0pAR6YKw RyD37IpRS0JR4qOPxzCyOhbiWdrixSrZqlriCCakGzpYVb9ALKNQzPO1Pgdy7Og4BQwRdJT4yfDM5 PYafC8mdY9agodoQkKww==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQfC5-000292-1L; Fri, 01 Nov 2019 22:14:57 +0000 Received: from mail-pl1-x649.google.com ([2607:f8b0:4864:20::649]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQf9b-00083Q-AH for linux-arm-kernel@lists.infradead.org; Fri, 01 Nov 2019 22:12:24 +0000 Received: by mail-pl1-x649.google.com with SMTP id g7so7167844plo.5 for ; Fri, 01 Nov 2019 15:12:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=TmDnfQN5KSEC3g/578TQjZ7QNjxfyUPcu+BQjF9Yuvo=; b=WX+ZlyX9au7TTis+HDd7Dbd6apXsSxGT4BjPuLcLwhq8XdU1hWXuu+jMmXXpmqn/ly JnciDuOZ2YKOinrTMYHNYjglGY6CBM9JRa6UwI5ny+82cASaJNLEYq03XkrGGMRD/45C 080BYykDqPbpaWcEFlq6ukx4uUgLPbDYn8rQoXBjaKsPYUg8U6UVKNYt0gsSY+aCZh6t sDQ/hg4YUpFcnNlIJocP1LkLyLwN/E6CPVe3fwUCeFBsqzz656n40juIl8qkmBDC5VxV leO2fOlOj+kIu6XcB0vIjnzmxkmxh31gkv9F3DRvB4cNzr9altIj/aQ4Xeb30TvjIhYS CEeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=TmDnfQN5KSEC3g/578TQjZ7QNjxfyUPcu+BQjF9Yuvo=; b=X5DbT8DFs4zsIXYUoOQjur1saxUbzUMqHCAzGbn+jwkAZ1UIzcmXWC2TiRJSnCbqeM NB3NFTop2z0CtU4valyLjfoplLqm8dFmtUFocpoUOf5Mrce0pr0qjmc7aeaXwJxZdumG UxDx0VT1sHk5mwZvkNyO565UctAasFbo1ZzDt3TFviZV/Llk61cTu7IHYdFFhP2JDblg sbUgbSGtooixHDR4mkrDbv6yUBGNn64V0iDONk3Cgy//ZsDtFbuAYIriHQseycdScV6/ rB7uMuLAd+GH6R2qbKonXOvBFlFXcvYxG/RWttfsNg/cbmR/8v6l7lJTv0IhkAA5Z5ZQ m6OA== X-Gm-Message-State: APjAAAXm3g0BEipZGigP1Jfq+eu+lTzxD7+yMcw1+1USy4JJhPym+qXW sLjpeXHxVs5e5QU+L/aIK4wgJD8aJ8WgeJSkSqo= X-Google-Smtp-Source: APXvYqzVngOFcUfwiWQ8eDwuiV8sr1oY1W9Y7iX+20vU9uQZSIlJayld9jxLG7yCiTZmX5C4MT0to41bsvZhb6iCe1E= X-Received: by 2002:a63:364d:: with SMTP id d74mr15884929pga.408.1572646341266; Fri, 01 Nov 2019 15:12:21 -0700 (PDT) Date: Fri, 1 Nov 2019 15:11:43 -0700 In-Reply-To: <20191101221150.116536-1-samitolvanen@google.com> Message-Id: <20191101221150.116536-11-samitolvanen@google.com> Mime-Version: 1.0 References: <20191018161033.261971-1-samitolvanen@google.com> <20191101221150.116536-1-samitolvanen@google.com> X-Mailer: git-send-email 2.24.0.rc1.363.gb1bccd3e3d-goog Subject: [PATCH v4 10/17] arm64: disable kretprobes with SCS From: Sami Tolvanen To: Will Deacon , Catalin Marinas , Steven Rostedt , Masami Hiramatsu , Ard Biesheuvel X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20191101_151223_511056_EC66A86B X-CRM114-Status: GOOD ( 10.72 ) X-Spam-Score: -7.7 (-------) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (-7.7 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:649 listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM white-list -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.0 DKIMWL_WL_MED DKIMwl.org - Medium sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Kees Cook , Jann Horn , Masahiro Yamada , Marc Zyngier , kernel-hardening@lists.openwall.com, Nick Desaulniers , linux-kernel@vger.kernel.org, Miguel Ojeda , clang-built-linux@googlegroups.com, Sami Tolvanen , Laura Abbott , Dave Martin , linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org With CONFIG_KRETPROBES, function return addresses are modified to redirect control flow to kretprobe_trampoline. This is incompatible with SCS. Signed-off-by: Sami Tolvanen Reviewed-by: Kees Cook --- arch/arm64/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 3f047afb982c..e7b57a8a5531 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -165,7 +165,7 @@ config ARM64 select HAVE_STACKPROTECTOR select HAVE_SYSCALL_TRACEPOINTS select HAVE_KPROBES - select HAVE_KRETPROBES + select HAVE_KRETPROBES if !SHADOW_CALL_STACK select HAVE_GENERIC_VDSO select IOMMU_DMA if IOMMU_SUPPORT select IRQ_DOMAIN From patchwork Fri Nov 1 22:11:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 11223851 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9542D13BD for ; Fri, 1 Nov 2019 22:15:34 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 73B2C2085B for ; Fri, 1 Nov 2019 22:15:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="SJlLqym1"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="o1wmzs+q" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 73B2C2085B Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=9PxZDuCy1Fbh5I5OXQbqM1GBASA+9neSuPBa2J+gNWU=; b=SJlLqym15uuMbB hy3w+V2ZLKcqalx8CXwe6E/0gQDiVj/1lEUMxWzKkrRawvjYS8rZ8vGuD+TIlWO72cIUJqhIYKMOo vtiab1jYlotlGGOjYfGt2qds+Nhjm4ye8nqgRBS1mRC88bA6EJhm/ekchWDzayuLwXk/lBcaHtL/D fnS/2mfHohvlAwlVJDgzAK/cIzGuqB+/qJEXrO9SYtRF25a+cs1/U629ai7lDayJUG1DgjMvoRwFY vxYjZuhmHnUTzBz7WXvgeLKX9ogsXl1qufQDCiq8b8j+xKqLfNF+oGtvTjfdYHjZaPOW3kRXo7UM+ rIop6A7kKcivWIH9f/mw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQfCg-0003sg-0E; Fri, 01 Nov 2019 22:15:34 +0000 Received: from mail-pf1-x44a.google.com ([2607:f8b0:4864:20::44a]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQf9d-00085p-W5 for linux-arm-kernel@lists.infradead.org; Fri, 01 Nov 2019 22:12:30 +0000 Received: by mail-pf1-x44a.google.com with SMTP id r187so8429134pfc.16 for ; Fri, 01 Nov 2019 15:12:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=RsnQ6FRofmXv+RSRylCILajHjnEectXS/G9SmKbukAo=; b=o1wmzs+qUKyvyNRJXp3X9qKqVJBTuvDks2cY0wcDPQb7bIkAGC906DDkR30LW9eQqJ zMXXLC1USMOZgspHImZJpgDDlfNKWn8CCmJRaf8YaWLWZjkkBOjbdmCWqR0xiz2Z1E8M IMXajWsM2ZNytq3YJD57+AcO0W4a4BAeYhyFET+L/KsEcCvEEvcQIkNw9qARaUd3IBj+ ++FoWxR12a+dlSA/MO5FA1lV1Ulde6ProNs5GbCiHu4RxgZbton9GSSH46BCYxqtL4td jIGaL3F6B2fs+2u9hLh0hFh84welR9hB0LYDeiGk0rmXfYPzPJvvptVVG6q7/+epyQDG 9jJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=RsnQ6FRofmXv+RSRylCILajHjnEectXS/G9SmKbukAo=; b=CEwA3j1Mvqt5hkRz2pgT1/KLt2UBfHo8XpOt50p8EaYENBP9R3ithu7BgxGo1slem1 YPRbgOrWgStddcyYJ6Uah9xXvGN/2OtsYIlfGf7y6aKq1RqBQT7F+CdOK1JtURsDSO9n 5vqLV6dSqoFtuTHe7px5zkdB5Ktu46ttZo2mbhuAxPux6rLZotvnwA7dlqVmmoUjH1LU fg/4vcb+rhbs91Doiy/IL03re8fzk9SLe0Nq8En+SpVqk0PFcYcQnYm/OufV0xfNeQPT g5dfChxoZZLUMRFtuxFGx3qDa35ugUATFYV/4ngHtYNjEslJ+7kCSajZaz9W/yBVr735 FZug== X-Gm-Message-State: APjAAAVQT576y7Cwk2xu4G156lQSTRyLLPwbN411jTV/6GwdP1YTpgOD P/CWTgLXaRnK1I+UZarbnR44Jb2XrKQEJmMn8eE= X-Google-Smtp-Source: APXvYqy8xknc+AggluM5pslguFgR4jRQWMnOnFZJmuiy7zcx6A+OMKobqQMh8kuVoFD8eIIYYvO3uXQeHJhmKYfS7dA= X-Received: by 2002:a63:8f12:: with SMTP id n18mr1357176pgd.340.1572646343780; Fri, 01 Nov 2019 15:12:23 -0700 (PDT) Date: Fri, 1 Nov 2019 15:11:44 -0700 In-Reply-To: <20191101221150.116536-1-samitolvanen@google.com> Message-Id: <20191101221150.116536-12-samitolvanen@google.com> Mime-Version: 1.0 References: <20191018161033.261971-1-samitolvanen@google.com> <20191101221150.116536-1-samitolvanen@google.com> X-Mailer: git-send-email 2.24.0.rc1.363.gb1bccd3e3d-goog Subject: [PATCH v4 11/17] arm64: disable function graph tracing with SCS From: Sami Tolvanen To: Will Deacon , Catalin Marinas , Steven Rostedt , Masami Hiramatsu , Ard Biesheuvel X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20191101_151226_070760_529073E5 X-CRM114-Status: GOOD ( 10.63 ) X-Spam-Score: -7.7 (-------) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (-7.7 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:44a listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM white-list -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.0 DKIMWL_WL_MED DKIMwl.org - Medium sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Kees Cook , Jann Horn , Masahiro Yamada , Marc Zyngier , kernel-hardening@lists.openwall.com, Nick Desaulniers , linux-kernel@vger.kernel.org, Miguel Ojeda , clang-built-linux@googlegroups.com, Sami Tolvanen , Laura Abbott , Dave Martin , linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org With CONFIG_FUNCTION_GRAPH_TRACER, function return addresses are modified in ftrace_graph_caller and prepare_ftrace_return to redirect control flow to ftrace_return_to_handler. This is incompatible with SCS. Signed-off-by: Sami Tolvanen Reviewed-by: Kees Cook --- arch/arm64/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index e7b57a8a5531..42867174920f 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -148,7 +148,7 @@ config ARM64 select HAVE_FTRACE_MCOUNT_RECORD select HAVE_FUNCTION_TRACER select HAVE_FUNCTION_ERROR_INJECTION - select HAVE_FUNCTION_GRAPH_TRACER + select HAVE_FUNCTION_GRAPH_TRACER if !SHADOW_CALL_STACK select HAVE_GCC_PLUGINS select HAVE_HW_BREAKPOINT if PERF_EVENTS select HAVE_IRQ_TIME_ACCOUNTING From patchwork Fri Nov 1 22:11:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 11223849 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BAF8B1668 for ; Fri, 1 Nov 2019 22:15:25 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 98E122085B for ; Fri, 1 Nov 2019 22:15:25 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="izRT1yN0"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="iJRBiSEs" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 98E122085B Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=q/us5YYoN6NcpVPm//s9pMAcFkF2nW12G8DHM3ancRg=; b=izRT1yN0ELOLk6 lSuWJ9bEIQceoor2H3AIfElfhKkAaDik07F6GjVv1+rR8QKdDutiOff8AKkow9spcYLeX4SfR2woP rVuXK+oFHtJVeLF1nqMx1oI/ZVQ7ziNb/su1Pr7oZMV7E/SE2m1iUTSf9DlKeW+KzgvTkpQ3yRQrK /WVjnf15AiBBio8RTouYbsgKBgjqsoce4WDopyR0xM/1kJ8LRhJViDtWQdQjwiL1pYwIxhC6GRd+U BJFycJbpOcFWIWZabketUBrX1Jei4nelDAJ7u1vCKDXsvqsASuLxYl51PDvwvdgSYiJkU97EG1RR7 PGaWTrTwjHWauApmj1Zg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQfCV-0003gW-AI; Fri, 01 Nov 2019 22:15:23 +0000 Received: from mail-pg1-x549.google.com ([2607:f8b0:4864:20::549]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQf9h-00088I-3N for linux-arm-kernel@lists.infradead.org; Fri, 01 Nov 2019 22:12:31 +0000 Received: by mail-pg1-x549.google.com with SMTP id v10so8033328pge.12 for ; Fri, 01 Nov 2019 15:12:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=xr3X122JfUZTOJ9HNPiJpW+Iyor6r4hGY/b4sjNxJZA=; b=iJRBiSEsCdhJICeToYTjV6/+ZJ3W2xifgYKJfjcL131hC2BB0cUCRbkywESPVzBlrI 8VlR5cC+N7pMsU5+x22Aw5ne1YAMEzfVAV/3q/b7V7ZoG9tAVIz0AEgVSuh4hW3Zbj6L 8aT4Az4vTNPYUadKRN4vE4k+cTQ9+QjgE+sNFrGD+oYFu80MiqRx6LYVU3nA2t8rpdAu rt0Xtr9hynaTlZ4tY88Emujp4zPpcari/k/MzTq60ei+HHRB6JOsgOHkUmAALjYf3xWu 4VQzOKmPKCT2h1o9LllZnsSb98KxsZJ03JbNAV7PqwK8Ovk06fBR7TYZNq1DCfB0qZBz hFDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=xr3X122JfUZTOJ9HNPiJpW+Iyor6r4hGY/b4sjNxJZA=; b=gseJjPYiTcS4QszwAoPX9zE3+DImGLgPswc7LZdCIilLfctV+6OVuRk3j98F+7eysj fDlJj1zMBgCR56FHkgt6GOx63fs5Rs6CszLiJNWdnfnllZ0E7G4to1ihbDihnufSL89m Uj9htpTh2lHFsc1NKpAhR2n2jn/ZM6edfz9hw7VhzdTZOUn30BEU+Qlf9rwTw1dqciLf pnWJltjREEw/1spd0Sxjdtv+5jYT2lh8yztHDCEg6ZiBk5srv4S5N/bOYGa50m5oj767 gncz3xxG1AYA/9NFjOzS5+b02NgaTPDgaQwPh3yy+J+nYwhxrCp2ldIPza0rGPsm/AAc hfFg== X-Gm-Message-State: APjAAAVMoXLmdp1zW7BH9ZfQ5AQ0kQjkLe+gPNSax89vBaW52Ii97yBO 9vWzIbkhAxNPpQSqKrHBWa1I5D4lH8VYiACq3t4= X-Google-Smtp-Source: APXvYqyCFSFt+5MiVYAedclgUa3YVl6NHt+7PUMsEM4gdNt4exvKeZ9B7bS1m6sTTxVqCs+LwHo1QQUcaSTSyZAlU8w= X-Received: by 2002:a65:47cd:: with SMTP id f13mr15511166pgs.356.1572646346250; Fri, 01 Nov 2019 15:12:26 -0700 (PDT) Date: Fri, 1 Nov 2019 15:11:45 -0700 In-Reply-To: <20191101221150.116536-1-samitolvanen@google.com> Message-Id: <20191101221150.116536-13-samitolvanen@google.com> Mime-Version: 1.0 References: <20191018161033.261971-1-samitolvanen@google.com> <20191101221150.116536-1-samitolvanen@google.com> X-Mailer: git-send-email 2.24.0.rc1.363.gb1bccd3e3d-goog Subject: [PATCH v4 12/17] arm64: reserve x18 from general allocation with SCS From: Sami Tolvanen To: Will Deacon , Catalin Marinas , Steven Rostedt , Masami Hiramatsu , Ard Biesheuvel X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20191101_151229_281038_DCBB7DDD X-CRM114-Status: UNSURE ( 8.84 ) X-CRM114-Notice: Please train this message. X-Spam-Score: -7.7 (-------) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (-7.7 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:549 listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM white-list -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.0 DKIMWL_WL_MED DKIMwl.org - Medium sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Kees Cook , Jann Horn , Masahiro Yamada , Marc Zyngier , kernel-hardening@lists.openwall.com, Nick Desaulniers , linux-kernel@vger.kernel.org, Miguel Ojeda , clang-built-linux@googlegroups.com, Sami Tolvanen , Laura Abbott , Dave Martin , linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Reserve the x18 register from general allocation when SCS is enabled, because the compiler uses the register to store the current task's shadow stack pointer. Note that all external kernel modules must also be compiled with -ffixed-x18 if the kernel has SCS enabled. Signed-off-by: Sami Tolvanen Reviewed-by: Nick Desaulniers Reviewed-by: Kees Cook --- arch/arm64/Makefile | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile index 2c0238ce0551..ef76101201b2 100644 --- a/arch/arm64/Makefile +++ b/arch/arm64/Makefile @@ -72,6 +72,10 @@ stack_protector_prepare: prepare0 include/generated/asm-offsets.h)) endif +ifeq ($(CONFIG_SHADOW_CALL_STACK), y) +KBUILD_CFLAGS += -ffixed-x18 +endif + ifeq ($(CONFIG_CPU_BIG_ENDIAN), y) KBUILD_CPPFLAGS += -mbig-endian CHECKFLAGS += -D__AARCH64EB__ From patchwork Fri Nov 1 22:11:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 11223853 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 50EB913BD for ; Fri, 1 Nov 2019 22:15:54 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 09E7420659 for ; Fri, 1 Nov 2019 22:15:54 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="EKv/r2Ba"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="LtAl7Dty" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 09E7420659 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=jSFIuWRWi5sAZJc4zYZPj20GSXSzq1zG+GUirAebXsA=; b=EKv/r2BaYpF7qI VNRYgnnuCDZoXWk1PfUUe82zAaOEmvz/xy4vRvZ73oyZYXPm6yg2C8Sl/kRdt/TogS/LBCxCyZLct VrGKW6UOSSy4TFckAIFOO6prrIAHQj9IhHT5QcMAlUofrWXP0sQNcTbo5cM7S1feHNvgPiwWmcwsw 4HsI8TVFUlwWRHCvzqHq02bvz+6PpxyorWEGRi4lc5SaWGzj6wxaL0Cxy8hDqMDQzs4AXzxr0uTif a+cShHjrIHEJj/jlc3Ga9YcBXGBa24noH5c3swhbuhK6MTuAWanThnD9zPo97OSFj9xWD43bBIC3U DU/AGlHHmJtCjKuj6TQA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQfCw-00049p-OZ; Fri, 01 Nov 2019 22:15:50 +0000 Received: from mail-vk1-xa4a.google.com ([2607:f8b0:4864:20::a4a]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQf9k-0008BT-1Q for linux-arm-kernel@lists.infradead.org; Fri, 01 Nov 2019 22:12:33 +0000 Received: by mail-vk1-xa4a.google.com with SMTP id s17so4464356vkb.5 for ; Fri, 01 Nov 2019 15:12:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=oPoEfiJ829HFiTJF4c1zGL5TdlrTqAGNgCQhkPxdZv8=; b=LtAl7DtyP3CWOemlPwjpOVNnzn34lyrO0lR489D5hiJ2VdFoF7EvYDq2wfAcUETMUj AJJV9M7q6HlU6FTXIdD1nVIYYwLxTKh17kUwJIBUkK7g2ckd+p7WgRLwUL+U4LhnUrbS 88BQeTCyX9oSneGkX/XIJMANH1JIMKlq987kvfi9Jwu3IO8heBNjQe3JNyCoyIBcqE5C r14KyfVUCK2DPs+EL3vCGkIShUp8wmIdLEHOjtbfUl144pnEGGFEFKJlhFi/qSds+L4C EczTWrE0OgWD8kqUXRcCY/eBf8by4lrn14g7ldLKqA66pCBxbbWJBFgW5xjq+nWP9ZjC NtIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=oPoEfiJ829HFiTJF4c1zGL5TdlrTqAGNgCQhkPxdZv8=; b=YwuGfprfys0+MM15f356l0LPoNODh8Bw60YPT+DSvySx1ulYuoZJRaiYCtUkJ6C31L l6IWPJok7wfKA0/fA+7WqyjFIDAc7Y+mnAeBzKK38gmeUsMZVYuElEMqNoHR5V9n0bFJ fKfux3byEg3cblUvrCn5vcQ59P0cTdwuoVg9KfvQC6/wQBXU0X+8rtAemSPY72zfqsYG TIyH0ltSpYQGKT42g6tN8ygvIQA2eK0m4do/S8yaECL6Eh+uobUDuq3+9sj8QdXSKI3q p3kMqfvx48oKXw22svIUTANxOdi8M6LFPM8PVLEEZr+ZlLaP1ky9Sj9bRJts2FqEFo6d mVcg== X-Gm-Message-State: APjAAAVW3abebGEtokStDoEacH0jDjk5sn5aLcNNg7DOnP+EdoG9/n2j M1j+DGrvwlF0aeLX6ffRqbCiCl/uA9Qb3D9gsLw= X-Google-Smtp-Source: APXvYqxf6pyuy2b0C7kGwVb9cMM9HBx/dT4rTM6z4dJTmBDml32z4MNPrDuebbuZ9ExwOLM18YhheN7oDa5AKWC4xdM= X-Received: by 2002:a67:f7d0:: with SMTP id a16mr2505843vsp.108.1572646349491; Fri, 01 Nov 2019 15:12:29 -0700 (PDT) Date: Fri, 1 Nov 2019 15:11:46 -0700 In-Reply-To: <20191101221150.116536-1-samitolvanen@google.com> Message-Id: <20191101221150.116536-14-samitolvanen@google.com> Mime-Version: 1.0 References: <20191018161033.261971-1-samitolvanen@google.com> <20191101221150.116536-1-samitolvanen@google.com> X-Mailer: git-send-email 2.24.0.rc1.363.gb1bccd3e3d-goog Subject: [PATCH v4 13/17] arm64: preserve x18 when CPU is suspended From: Sami Tolvanen To: Will Deacon , Catalin Marinas , Steven Rostedt , Masami Hiramatsu , Ard Biesheuvel X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20191101_151232_154447_66EB763A X-CRM114-Status: GOOD ( 10.77 ) X-Spam-Score: -7.7 (-------) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (-7.7 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:a4a listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM white-list -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.0 DKIMWL_WL_MED DKIMwl.org - Medium sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Kees Cook , Jann Horn , Masahiro Yamada , Marc Zyngier , kernel-hardening@lists.openwall.com, Nick Desaulniers , linux-kernel@vger.kernel.org, Miguel Ojeda , clang-built-linux@googlegroups.com, Sami Tolvanen , Laura Abbott , Dave Martin , linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Don't lose the current task's shadow stack when the CPU is suspended. Signed-off-by: Sami Tolvanen Reviewed-by: Nick Desaulniers Reviewed-by: Kees Cook --- arch/arm64/include/asm/suspend.h | 2 +- arch/arm64/mm/proc.S | 10 ++++++++++ 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/suspend.h b/arch/arm64/include/asm/suspend.h index 8939c87c4dce..0cde2f473971 100644 --- a/arch/arm64/include/asm/suspend.h +++ b/arch/arm64/include/asm/suspend.h @@ -2,7 +2,7 @@ #ifndef __ASM_SUSPEND_H #define __ASM_SUSPEND_H -#define NR_CTX_REGS 12 +#define NR_CTX_REGS 13 #define NR_CALLEE_SAVED_REGS 12 /* diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index fdabf40a83c8..5616dc52a033 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -49,6 +49,8 @@ * cpu_do_suspend - save CPU registers context * * x0: virtual address of context pointer + * + * This must be kept in sync with struct cpu_suspend_ctx in . */ ENTRY(cpu_do_suspend) mrs x2, tpidr_el0 @@ -73,6 +75,9 @@ alternative_endif stp x8, x9, [x0, #48] stp x10, x11, [x0, #64] stp x12, x13, [x0, #80] +#ifdef CONFIG_SHADOW_CALL_STACK + str x18, [x0, #96] +#endif ret ENDPROC(cpu_do_suspend) @@ -89,6 +94,11 @@ ENTRY(cpu_do_resume) ldp x9, x10, [x0, #48] ldp x11, x12, [x0, #64] ldp x13, x14, [x0, #80] +#ifdef CONFIG_SHADOW_CALL_STACK + ldr x18, [x0, #96] + /* Clear the SCS pointer from the state buffer */ + str xzr, [x0, #96] +#endif msr tpidr_el0, x2 msr tpidrro_el0, x3 msr contextidr_el1, x4 From patchwork Fri Nov 1 22:11:47 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 11223855 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 277BC1668 for ; Fri, 1 Nov 2019 22:16:04 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E56F120659 for ; Fri, 1 Nov 2019 22:16:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="ci84wJ59"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="odv3zPa4" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E56F120659 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=wgg+ynFX765AsTUmjlJn9/2mlvlVDI5jTY+W9EuxF1k=; b=ci84wJ5904ZVpZ yLTtaj9HGVCLUThuZElQwEeLk/jb6J1XvyQ3qCKASOhvKzDNQWBLzjmAWxCL+gBy5Ofl0AEDVu/xP md14iPYnBrtSNgyXoZ/hp5n//xVFxbXAIDixdkxPFQ3W7Hpn6M944Xg79v52WE7qkqkyp3OCbTJU+ oXHhx/F1K882Vx5+N6IPQogONbDryY2lxzSOUyv6gEo8Po73Ikpob3hWGt4GTb1sZeC+muLynOBqN NMLyjCioyP9xZuSKKyxotLzmFVl99GSeHNuHiXtk6kymbCuDiozXb3C6SHCh9OEEky9T1Fg/sINi1 9v1wfmchTCjG0/qH51BA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQfD9-0004OD-Bm; Fri, 01 Nov 2019 22:16:03 +0000 Received: from mail-pg1-x54a.google.com ([2607:f8b0:4864:20::54a]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQf9n-0008E0-1M for linux-arm-kernel@lists.infradead.org; Fri, 01 Nov 2019 22:12:36 +0000 Received: by mail-pg1-x54a.google.com with SMTP id w13so7993675pge.15 for ; Fri, 01 Nov 2019 15:12:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=uZ/kQt4EYM2LvpfCusnWoxftHyD2DxgcHTICv7X7dqw=; b=odv3zPa4RgQ8s6dsoDxvbsmHF3EENGYKEFkki0g9WgsTtiUy1PWOo4QagxAFwNjlM5 UaXo+6ESXS0tfN9t/uaTyImKPgoUdkbU5/6mXIU+WDuhuaMZVZlToQjZBZjUeJ+XCKaN AZaYtQv4cn3jcrlvgSVPiMAL2aX2o/kWtvg0ZhsbFx5XNHfG27Zn+lw1m2rgEzTuN0Qx AhXuI7DUwTtXfZmNooQs1nB+s/Ns/DX1LvGMoiAbd56F3AJ+mJtqGL8w4imrJ9czieHh o3BNgiNlRATQnbvYsOjBOz35StnAU9Usdu1hP8mnieHo1tIou+WOWs5WVmR96RMvFpMw BkXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=uZ/kQt4EYM2LvpfCusnWoxftHyD2DxgcHTICv7X7dqw=; b=oIXae33PstjqHSqFeZ/VpvIisQ9zmP6RG7KnAxQjcEIFAuH10ZNfgBcYYjIOBXHM7C 2yar5lEZnlRemW2x/lEqFOrTCSVHKbp8cHl092gS2kGYZBotAXijabcOb88xkGQ5Qfed QOLsNhSChlCq3emQ1g/BHLeHHtBwWiCCap1/VmVW2I/cyD0soU/QpELWXPsWrhEv619D veBHK2QJ8THNMWuepiSw7G+MDsR6C+ul85taZYA/6NmlhgQyT2fhv7wxjsPK590HB+cE EkkrZ03orVedW9CdI/atlN4PnVhHnO2pOz8fCIld+7NEC0TkZAAwxOUdm2yuhYN/e2dL aw8g== X-Gm-Message-State: APjAAAU9k4rjOQ1QJmtPnVHai4zRrdZKAVixxFpz1upxpjxnPNwRhO8G m4xGP70hEb/QmVU7hYSFD/91JhsLP7yjZXwd19E= X-Google-Smtp-Source: APXvYqxnvjccMKGZa+EMFNvyVvl4JQq0YHBc8XNC8xBIuIHK+TPnyLwR96cD6M+G58PTtTiYueuKAoyE/cNOXhTJ/L8= X-Received: by 2002:a63:3203:: with SMTP id y3mr15810585pgy.437.1572646351983; Fri, 01 Nov 2019 15:12:31 -0700 (PDT) Date: Fri, 1 Nov 2019 15:11:47 -0700 In-Reply-To: <20191101221150.116536-1-samitolvanen@google.com> Message-Id: <20191101221150.116536-15-samitolvanen@google.com> Mime-Version: 1.0 References: <20191018161033.261971-1-samitolvanen@google.com> <20191101221150.116536-1-samitolvanen@google.com> X-Mailer: git-send-email 2.24.0.rc1.363.gb1bccd3e3d-goog Subject: [PATCH v4 14/17] arm64: efi: restore x18 if it was corrupted From: Sami Tolvanen To: Will Deacon , Catalin Marinas , Steven Rostedt , Masami Hiramatsu , Ard Biesheuvel X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20191101_151235_131149_8BCB2142 X-CRM114-Status: UNSURE ( 9.43 ) X-CRM114-Notice: Please train this message. X-Spam-Score: -7.7 (-------) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (-7.7 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:54a listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM white-list -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.0 DKIMWL_WL_MED DKIMwl.org - Medium sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Kees Cook , Jann Horn , Masahiro Yamada , Marc Zyngier , kernel-hardening@lists.openwall.com, Nick Desaulniers , linux-kernel@vger.kernel.org, Miguel Ojeda , clang-built-linux@googlegroups.com, Sami Tolvanen , Laura Abbott , Dave Martin , linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org If we detect a corrupted x18 and SCS is enabled, restore the register before jumping back to instrumented code. This is safe, because the wrapper is called with preemption disabled and a separate shadow stack is used for interrupt handling. Signed-off-by: Sami Tolvanen Reviewed-by: Kees Cook --- arch/arm64/kernel/efi-rt-wrapper.S | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kernel/efi-rt-wrapper.S b/arch/arm64/kernel/efi-rt-wrapper.S index 3fc71106cb2b..945744f16086 100644 --- a/arch/arm64/kernel/efi-rt-wrapper.S +++ b/arch/arm64/kernel/efi-rt-wrapper.S @@ -34,5 +34,10 @@ ENTRY(__efi_rt_asm_wrapper) ldp x29, x30, [sp], #32 b.ne 0f ret -0: b efi_handle_corrupted_x18 // tail call +0: +#ifdef CONFIG_SHADOW_CALL_STACK + /* Restore x18 before returning to instrumented code. */ + mov x18, x2 +#endif + b efi_handle_corrupted_x18 // tail call ENDPROC(__efi_rt_asm_wrapper) From patchwork Fri Nov 1 22:11:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 11223857 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 96CB413BD for ; Fri, 1 Nov 2019 22:16:23 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 72CEF217D9 for ; Fri, 1 Nov 2019 22:16:23 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="WwuBK1s8"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="mLcd5Ko+" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 72CEF217D9 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=EDLkDIRY8EbcoQ0omMi5wbuWxlquPR1aWUydVyB+3bo=; b=WwuBK1s84GPEdD 7WIAktzZLFwEV2aHm/q3YK62rhgyO5dsAp1YohXUUbIAMSVvL4GFSewX9v/YmOcb5/Jz9ZvXnGInd V00XjZ3TaZ8Zp3Mh32f91P8baHdhRJ55b2aLcYhVrvdM8hGPK/1AavN0lvq69Po/Htb/SFRUrhwZB 9g2fMcRkrkC39ms9X4KhJ+fOJXCCY8//jCie/rqryibOTPhBikIE3/RzUMmeemHuJ70quaQ/4eqcI +nUyMGfDuBYdjKmqMFx08p/zH39vgA83UktSmIFzH2tuxHqkzQY/GJyJaPFCOzbKabu5dTVPdBV2n 6lzlJznfa6SO1QxswEww==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQfDR-0004fI-0G; Fri, 01 Nov 2019 22:16:21 +0000 Received: from mail-pg1-x549.google.com ([2607:f8b0:4864:20::549]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQf9o-0008Go-Eh for linux-arm-kernel@lists.infradead.org; Fri, 01 Nov 2019 22:12:37 +0000 Received: by mail-pg1-x549.google.com with SMTP id o3so5702776pgb.16 for ; Fri, 01 Nov 2019 15:12:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=oB4lIvjzdBiHgRAUphuiQVPgI+jp2K5bMoUCGUiH4hk=; b=mLcd5Ko+fJmhjxDMoBpGZ3EPC3b/VP/PBLdcmlypMJNdCOc5u9jhjSLWUNRMYr69hx U2P5JSWVYdV6u8EzfA94UAbX9w5Q1ZrloIyJVyjdf/ObF5XN+RiQraEntwZF84tiJw4A SKbzKEtQORywvZdxPhMYx0LHvfikuCCPY9XvjfKtjRiin1Jn0QFeomQACCaJo3jt8lCW KyuDz5lfuSMpAGWUw7ut45j+6bJefZga8POnR6VI0FVyWw/cL8vyvxBQD9C59E1UjFOA 7acbvIVDECPMMh62wKUfHOE3f3OZYuymQJbmahdFgydr8nJ0p/3u8CSBUyJzVVefqNG+ Z8oA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=oB4lIvjzdBiHgRAUphuiQVPgI+jp2K5bMoUCGUiH4hk=; b=BS9UzqKVSSzpXRwCW6d7WJOwik4OBPQRqp7lc6VdSquRCi7TTHMMDoHc9MZ7PhXKVR c0jSOc7vSv1cvEyVfc+ms0QEQaQH5Dg5TWmxMS65LFN90rMbEW1KUCM/AjJg4AN3YXZk AuhyxHPpEBcFkVynKoCd4ncXbActoGPGXol5R320QXfCMp1cHHLfUA1mU1YSaeLmKrxW VVums/Y5lfDTtvNwXCHr98xNn5dQ375IyAHdsoYYOVSI5PeyiVR2wJBot1xtQNc5kA/4 LiW2HbwXkSW+SPqFQCKgMlV2KpXrNz/3hYFn/1YBemMAS+/1im25Q2XpJ8uPBKLdSn5B ukow== X-Gm-Message-State: APjAAAXWs5mZeEBPkksVx58pd4AZw1HRi+RCXRBAcu170nq2aP4XDYpF vjMzVg9bn5/Mk4DlOooRymXXfjkKd5jibXOYtlQ= X-Google-Smtp-Source: APXvYqxeW4tUSlj5Rjy/RV2utncaJwaQyF3tWO3lBVE2VevumOyR70VNIZ9Nj/gJhm+HF4+zxBXbctrnYgLnOQF1Wjw= X-Received: by 2002:a63:d20c:: with SMTP id a12mr14724175pgg.402.1572646354642; Fri, 01 Nov 2019 15:12:34 -0700 (PDT) Date: Fri, 1 Nov 2019 15:11:48 -0700 In-Reply-To: <20191101221150.116536-1-samitolvanen@google.com> Message-Id: <20191101221150.116536-16-samitolvanen@google.com> Mime-Version: 1.0 References: <20191018161033.261971-1-samitolvanen@google.com> <20191101221150.116536-1-samitolvanen@google.com> X-Mailer: git-send-email 2.24.0.rc1.363.gb1bccd3e3d-goog Subject: [PATCH v4 15/17] arm64: vdso: disable Shadow Call Stack From: Sami Tolvanen To: Will Deacon , Catalin Marinas , Steven Rostedt , Masami Hiramatsu , Ard Biesheuvel X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20191101_151236_514708_07EB3DEF X-CRM114-Status: GOOD ( 10.41 ) X-Spam-Score: -7.7 (-------) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (-7.7 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:549 listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM white-list -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.0 DKIMWL_WL_MED DKIMwl.org - Medium sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Kees Cook , Jann Horn , Masahiro Yamada , Marc Zyngier , kernel-hardening@lists.openwall.com, Nick Desaulniers , linux-kernel@vger.kernel.org, Miguel Ojeda , clang-built-linux@googlegroups.com, Sami Tolvanen , Laura Abbott , Dave Martin , linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Shadow stacks are only available in the kernel, so disable SCS instrumentation for the vDSO. Signed-off-by: Sami Tolvanen Reviewed-by: Nick Desaulniers Reviewed-by: Kees Cook --- arch/arm64/kernel/vdso/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/arm64/kernel/vdso/Makefile b/arch/arm64/kernel/vdso/Makefile index dd2514bb1511..a87a4f11724e 100644 --- a/arch/arm64/kernel/vdso/Makefile +++ b/arch/arm64/kernel/vdso/Makefile @@ -25,7 +25,7 @@ ccflags-y += -DDISABLE_BRANCH_PROFILING VDSO_LDFLAGS := -Bsymbolic -CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE) -Os +CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE) -Os $(CC_FLAGS_SCS) KBUILD_CFLAGS += $(DISABLE_LTO) KASAN_SANITIZE := n UBSAN_SANITIZE := n From patchwork Fri Nov 1 22:11:49 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 11223859 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7A4621668 for ; Fri, 1 Nov 2019 22:16:41 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 53BA521897 for ; Fri, 1 Nov 2019 22:16:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="NPDSgI+M"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="jD7UphcO" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 53BA521897 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=web2MIqdaQaZlmbioBDkjzWbHxP3CjuG09uVLcrnlVY=; b=NPDSgI+MI/TCTm 8sSoVbO3CGRVFdIHO17oM0dAcJQnXk98gm4ASKPabve3tACZFWUyvYy1VQf/ztaR1z8xZiPvxGVdv W2EcKun4lHxa78Ux3EG57RNk+CXWbPLbWLlqS0J3nUogU5CSjkloU74eC4Zg9mCiuWOq+9kh/ehDY 4b7p3cQFwqDeFVi2Q0c6qczVMuFEGe4riuWFKfENtkvzlWXiuQ/twHwapq5HAFlv9BWaCiLDaqaQj jU7CmKVZR6R+KpTayMdu+hSi8xOKS4AZCwlPh08aTRfFGQqY2aoll0I5KrnXNpTDVdpCZpN119tz0 TOBXm41kAKVs05P9F6Aw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQfDi-0004xl-7U; Fri, 01 Nov 2019 22:16:38 +0000 Received: from mail-pf1-x449.google.com ([2607:f8b0:4864:20::449]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQf9s-0008Jx-Jm for linux-arm-kernel@lists.infradead.org; Fri, 01 Nov 2019 22:12:42 +0000 Received: by mail-pf1-x449.google.com with SMTP id w2so8449000pfi.7 for ; Fri, 01 Nov 2019 15:12:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=SBPjjox/K0nQBp9p/UReaX06mOCKZA5VG5oKrHSRp3U=; b=jD7UphcOofzvzp+Fej+FGAYX3qlgaInC3q6riheK3W8LCKmKWtfTlTrJ/vzD4a19an +Hcrv+tkq9TdRVKgDOaa6Ic5PZn6XxN9+vLYJmvkleDiug2F9Tcj89ldzM4/Wx0mDp22 mgzUKliuIEQOx7CwzvE6V9myvgkQNgKcVgCutPLeSHS/wibMi+2JvOa+OCv75/Nwje2O 8oWep5ZzzzdxjS5LcG8wINg1XSm4eYtuiQytGXDC4/Up/8zjkIUOQ+XAAMs0f9WdxGw6 GvK3uzJndPpvEwWoClt/0ppEIta5+qJS7qlUvBidggYVVp60p9g8sBe1C+92bc/icYtq Aalg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=SBPjjox/K0nQBp9p/UReaX06mOCKZA5VG5oKrHSRp3U=; b=NBNDX7x2Rmq2zBsN9JR01GPYAQj/JncqKFxqr6MOFVIR6hJHQsaEMC2DE8tzhZsUUF QjOK8p6gq4n5IhFrqU7gl34acCwWJeRtBXbsoly82u5c+EquBkr+9Vxao1i42JeuXuM8 qVkGm9Vw18WfY+iBtIv2eejKeXC7E+89SKiYvw7eJ23hrZEJTEaSYRIp2H7MKCkh7baM LXLRTdSSXdPLKf8AlkOkiia5E5yEurIq/e/AT5MwQKvBO7MZuTRMfw/R4T0eGRBTsS+8 LkMa7KWZ2XPRyAQO0mgXRukMZje3FiifLC0fkwaQQZnOQSVvhlZtkn/wwRDkXK5khtq0 8i5w== X-Gm-Message-State: APjAAAVad3ZNyLJ4NL1ZaSt35kCTRXfMlZ6pZEKryjiWPE2K/oF8xgwk acV63g6nB0Svjn6ySQQz2CkNLZrmtLGgScwGj70= X-Google-Smtp-Source: APXvYqxqSMUGI9UyircMqSL5QLFPtKAtuhwEXcqXTbAsf4XrCTVNUwTxLrj6ufv4Kgs0IKAfQNkuo4iX4Ohmpxbq4g8= X-Received: by 2002:a63:4525:: with SMTP id s37mr16212936pga.148.1572646357518; Fri, 01 Nov 2019 15:12:37 -0700 (PDT) Date: Fri, 1 Nov 2019 15:11:49 -0700 In-Reply-To: <20191101221150.116536-1-samitolvanen@google.com> Message-Id: <20191101221150.116536-17-samitolvanen@google.com> Mime-Version: 1.0 References: <20191018161033.261971-1-samitolvanen@google.com> <20191101221150.116536-1-samitolvanen@google.com> X-Mailer: git-send-email 2.24.0.rc1.363.gb1bccd3e3d-goog Subject: [PATCH v4 16/17] arm64: disable SCS for hypervisor code From: Sami Tolvanen To: Will Deacon , Catalin Marinas , Steven Rostedt , Masami Hiramatsu , Ard Biesheuvel X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20191101_151240_885443_812EACED X-CRM114-Status: UNSURE ( 9.26 ) X-CRM114-Notice: Please train this message. X-Spam-Score: -7.7 (-------) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (-7.7 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:449 listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM white-list -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.0 DKIMWL_WL_MED DKIMwl.org - Medium sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Kees Cook , Jann Horn , Masahiro Yamada , Marc Zyngier , kernel-hardening@lists.openwall.com, Nick Desaulniers , linux-kernel@vger.kernel.org, Miguel Ojeda , clang-built-linux@googlegroups.com, Sami Tolvanen , Laura Abbott , Dave Martin , linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Filter out CC_FLAGS_SCS for code that runs at a different exception level. Suggested-by: Steven Rostedt (VMware) Signed-off-by: Sami Tolvanen Reviewed-by: Kees Cook --- arch/arm64/kvm/hyp/Makefile | 3 +++ 1 file changed, 3 insertions(+) diff --git a/arch/arm64/kvm/hyp/Makefile b/arch/arm64/kvm/hyp/Makefile index ea710f674cb6..17ea3da325e9 100644 --- a/arch/arm64/kvm/hyp/Makefile +++ b/arch/arm64/kvm/hyp/Makefile @@ -28,3 +28,6 @@ GCOV_PROFILE := n KASAN_SANITIZE := n UBSAN_SANITIZE := n KCOV_INSTRUMENT := n + +# remove the SCS flags from all objects in this directory +KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_SCS), $(KBUILD_CFLAGS)) From patchwork Fri Nov 1 22:11:50 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 11223861 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9271F1668 for ; Fri, 1 Nov 2019 22:16:55 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6907320659 for ; Fri, 1 Nov 2019 22:16:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="ogZ4yqvo"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="oFYpNtWp" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6907320659 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=wUKyN0X5MG9Z8l/7SFdZbixw9KcXeN02GUk2jItEqiU=; b=ogZ4yqvod51XbI L9yXUGkeeCugMkpB5OoPdCAJrwD2YVG4mjjB98Eu0k7I7i+8Ketvsel6AQM2J+qzUbgGrGxQGC3Ye 3T1nswZatz5ivVdb7yNW4iA4gSJ+1yq6Rm06T4ySeA1mg5wqVsiwTY2950hixvDbbP4TUgMOBJdN2 iiRoCgXd5z0Pc8IadYNz/66qFqz+eB5dAKtn99Mwu8ajdqMgNZB/LejEjENqKxlquIY7EFmbKSx7a 8txRtvldskqZsozLYKGSQ66wp33K9QRtNOvsbtgj1vBY5buVRP1JuoFMbWrvf1h69xnYJIQU6XHvG ASqNnUvzwFcNiu3oy8rQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQfDy-0005C5-KG; Fri, 01 Nov 2019 22:16:54 +0000 Received: from mail-pl1-x64a.google.com ([2607:f8b0:4864:20::64a]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1iQf9x-0008PX-96 for linux-arm-kernel@lists.infradead.org; Fri, 01 Nov 2019 22:12:47 +0000 Received: by mail-pl1-x64a.google.com with SMTP id g7so7168496plo.5 for ; Fri, 01 Nov 2019 15:12:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=die3x67ovmzWsWhvopUJvu6mU5I+GEuDo3qNBzEJCbM=; b=oFYpNtWp8IVtnl4tSZnpqO/2yUP+BEVmb3I/w9P/l7O0sI/RSB7j5R6VsFUYzjK8Ar oHGwre/O4pLHBzABM29Tf5rk4gomHAaTA5lJZtiLwYJoIsDz/o3NCvI9JPntwCUSEm0v fxkCSUnGt9W7sX4B79fro0qdhcKgmoFf2guUypNPf0Nf/1vY6NXf/uDWF9EteDvv8yz+ bRrCagTPSIykzT/fiZPT5tJQDFTScoAOWBZQz04svyzYQoxwzmfCR7xOifQQ4PbGni06 gbfmweTHedc8VvpNpuRaFwWTDy3OsYqolqxDdUe7s6fz3i6sUAI09KFlV7tIypxg86uv Uq0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=die3x67ovmzWsWhvopUJvu6mU5I+GEuDo3qNBzEJCbM=; b=oIESH03uWW+eD0oJ45R81s5jlEDCCdfXp3kXfI/PSYI51L4koKTa5lSLPPwHIHSOKw 0Ly9pY/h+JaOzzQkirYFgPZEulgYxwHExaQz441vostxmcxlfilTigoMwn240Ftl7ory jpbfxOqotV+6HohLwtVLqJKV52+XNsMA8WAXpNAwnDbCDlmM2jsqCDl1RNe9+F4r8ftB RIWq5Q+fYNfuMY2oA0EBX6Pa4vkh8ETmiXEw0BjrQTcP0lAf87wE8Yr4Xcjd+vYO3Owl lOyy4hvO6Ot+vKHG0t88gPaA9DyopRSdqo+4X3YsSzcVJeb+wRqeuZnhNe/XiHDVelfx aYtQ== X-Gm-Message-State: APjAAAUKJ0beCs1oKIfpoJbWaXSHpXCHPPcU9K5roaUy4VJQospXYbV/ 3FlJ1zMT7BSi/clfnaPvHO4edTdA3edCPOtjY64= X-Google-Smtp-Source: APXvYqzB6uUGqkuzOL6z7LyH53TR06l8k2phfWBLsMkqVxMrZpFDw+a5MeU2xONzAMSK5KNI5p6R8FP5Qbp12Jf2xC8= X-Received: by 2002:a63:eb47:: with SMTP id b7mr15595265pgk.179.1572646363390; Fri, 01 Nov 2019 15:12:43 -0700 (PDT) Date: Fri, 1 Nov 2019 15:11:50 -0700 In-Reply-To: <20191101221150.116536-1-samitolvanen@google.com> Message-Id: <20191101221150.116536-18-samitolvanen@google.com> Mime-Version: 1.0 References: <20191018161033.261971-1-samitolvanen@google.com> <20191101221150.116536-1-samitolvanen@google.com> X-Mailer: git-send-email 2.24.0.rc1.363.gb1bccd3e3d-goog Subject: [PATCH v4 17/17] arm64: implement Shadow Call Stack From: Sami Tolvanen To: Will Deacon , Catalin Marinas , Steven Rostedt , Masami Hiramatsu , Ard Biesheuvel X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20191101_151245_349769_EECE7BD2 X-CRM114-Status: GOOD ( 17.86 ) X-Spam-Score: -7.7 (-------) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (-7.7 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:64a listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM white-list -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.0 DKIMWL_WL_MED DKIMwl.org - Medium sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Kees Cook , Jann Horn , Masahiro Yamada , Marc Zyngier , kernel-hardening@lists.openwall.com, Nick Desaulniers , linux-kernel@vger.kernel.org, Miguel Ojeda , clang-built-linux@googlegroups.com, Sami Tolvanen , Laura Abbott , Dave Martin , linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org This change implements shadow stack switching, initial SCS set-up, and interrupt shadow stacks for arm64. Signed-off-by: Sami Tolvanen Reviewed-by: Kees Cook --- arch/arm64/Kconfig | 5 ++++ arch/arm64/include/asm/scs.h | 37 ++++++++++++++++++++++++++ arch/arm64/include/asm/stacktrace.h | 4 +++ arch/arm64/include/asm/thread_info.h | 3 +++ arch/arm64/kernel/Makefile | 1 + arch/arm64/kernel/asm-offsets.c | 3 +++ arch/arm64/kernel/entry.S | 28 ++++++++++++++++++++ arch/arm64/kernel/head.S | 9 +++++++ arch/arm64/kernel/irq.c | 2 ++ arch/arm64/kernel/process.c | 2 ++ arch/arm64/kernel/scs.c | 39 ++++++++++++++++++++++++++++ arch/arm64/kernel/smp.c | 4 +++ 12 files changed, 137 insertions(+) create mode 100644 arch/arm64/include/asm/scs.h create mode 100644 arch/arm64/kernel/scs.c diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 42867174920f..f4c94c5e8012 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -66,6 +66,7 @@ config ARM64 select ARCH_USE_QUEUED_RWLOCKS select ARCH_USE_QUEUED_SPINLOCKS select ARCH_SUPPORTS_MEMORY_FAILURE + select ARCH_SUPPORTS_SHADOW_CALL_STACK if CC_HAVE_SHADOW_CALL_STACK select ARCH_SUPPORTS_ATOMIC_RMW select ARCH_SUPPORTS_INT128 if GCC_VERSION >= 50000 || CC_IS_CLANG select ARCH_SUPPORTS_NUMA_BALANCING @@ -948,6 +949,10 @@ config ARCH_HAS_CACHE_LINE_SIZE config ARCH_ENABLE_SPLIT_PMD_PTLOCK def_bool y if PGTABLE_LEVELS > 2 +# Supported by clang >= 7.0 +config CC_HAVE_SHADOW_CALL_STACK + def_bool $(cc-option, -fsanitize=shadow-call-stack -ffixed-x18) + config SECCOMP bool "Enable seccomp to safely compute untrusted bytecode" ---help--- diff --git a/arch/arm64/include/asm/scs.h b/arch/arm64/include/asm/scs.h new file mode 100644 index 000000000000..c50d2b0c6c5f --- /dev/null +++ b/arch/arm64/include/asm/scs.h @@ -0,0 +1,37 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_SCS_H +#define _ASM_SCS_H + +#ifndef __ASSEMBLY__ + +#include + +#ifdef CONFIG_SHADOW_CALL_STACK + +extern void scs_init_irq(void); + +static __always_inline void scs_save(struct task_struct *tsk) +{ + void *s; + + asm volatile("mov %0, x18" : "=r" (s)); + task_set_scs(tsk, s); +} + +static inline void scs_overflow_check(struct task_struct *tsk) +{ + if (unlikely(scs_corrupted(tsk))) + panic("corrupted shadow stack detected inside scheduler\n"); +} + +#else /* CONFIG_SHADOW_CALL_STACK */ + +static inline void scs_init_irq(void) {} +static inline void scs_save(struct task_struct *tsk) {} +static inline void scs_overflow_check(struct task_struct *tsk) {} + +#endif /* CONFIG_SHADOW_CALL_STACK */ + +#endif /* __ASSEMBLY __ */ + +#endif /* _ASM_SCS_H */ diff --git a/arch/arm64/include/asm/stacktrace.h b/arch/arm64/include/asm/stacktrace.h index 4d9b1f48dc39..b6cf32fb4efe 100644 --- a/arch/arm64/include/asm/stacktrace.h +++ b/arch/arm64/include/asm/stacktrace.h @@ -68,6 +68,10 @@ extern void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk); DECLARE_PER_CPU(unsigned long *, irq_stack_ptr); +#ifdef CONFIG_SHADOW_CALL_STACK +DECLARE_PER_CPU(unsigned long *, irq_shadow_call_stack_ptr); +#endif + static inline bool on_irq_stack(unsigned long sp, struct stack_info *info) { diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h index f0cec4160136..8c73764b9ed2 100644 --- a/arch/arm64/include/asm/thread_info.h +++ b/arch/arm64/include/asm/thread_info.h @@ -41,6 +41,9 @@ struct thread_info { #endif } preempt; }; +#ifdef CONFIG_SHADOW_CALL_STACK + void *shadow_call_stack; +#endif }; #define thread_saved_pc(tsk) \ diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index 478491f07b4f..b3995329d9e5 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -63,6 +63,7 @@ obj-$(CONFIG_CRASH_CORE) += crash_core.o obj-$(CONFIG_ARM_SDE_INTERFACE) += sdei.o obj-$(CONFIG_ARM64_SSBD) += ssbd.o obj-$(CONFIG_ARM64_PTR_AUTH) += pointer_auth.o +obj-$(CONFIG_SHADOW_CALL_STACK) += scs.o obj-y += vdso/ probes/ obj-$(CONFIG_COMPAT_VDSO) += vdso32/ diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 214685760e1c..f6762b9ae1e1 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -33,6 +33,9 @@ int main(void) DEFINE(TSK_TI_ADDR_LIMIT, offsetof(struct task_struct, thread_info.addr_limit)); #ifdef CONFIG_ARM64_SW_TTBR0_PAN DEFINE(TSK_TI_TTBR0, offsetof(struct task_struct, thread_info.ttbr0)); +#endif +#ifdef CONFIG_SHADOW_CALL_STACK + DEFINE(TSK_TI_SCS, offsetof(struct task_struct, thread_info.shadow_call_stack)); #endif DEFINE(TSK_STACK, offsetof(struct task_struct, stack)); #ifdef CONFIG_STACKPROTECTOR diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index cf3bd2976e57..1eff08c71403 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -172,6 +172,10 @@ alternative_cb_end apply_ssbd 1, x22, x23 +#ifdef CONFIG_SHADOW_CALL_STACK + ldr x18, [tsk, #TSK_TI_SCS] // Restore shadow call stack + str xzr, [tsk, #TSK_TI_SCS] // Limit visibility of saved SCS +#endif .else add x21, sp, #S_FRAME_SIZE get_current_task tsk @@ -278,6 +282,12 @@ alternative_else_nop_endif ct_user_enter .endif +#ifdef CONFIG_SHADOW_CALL_STACK + .if \el == 0 + str x18, [tsk, #TSK_TI_SCS] // Save shadow call stack + .endif +#endif + #ifdef CONFIG_ARM64_SW_TTBR0_PAN /* * Restore access to TTBR0_EL1. If returning to EL0, no need for SPSR @@ -383,6 +393,9 @@ alternative_insn eret, nop, ARM64_UNMAP_KERNEL_AT_EL0 .macro irq_stack_entry mov x19, sp // preserve the original sp +#ifdef CONFIG_SHADOW_CALL_STACK + mov x20, x18 // preserve the original shadow stack +#endif /* * Compare sp with the base of the task stack. @@ -400,6 +413,12 @@ alternative_insn eret, nop, ARM64_UNMAP_KERNEL_AT_EL0 /* switch to the irq stack */ mov sp, x26 + +#ifdef CONFIG_SHADOW_CALL_STACK + /* also switch to the irq shadow stack */ + ldr_this_cpu x18, irq_shadow_call_stack_ptr, x26 +#endif + 9998: .endm @@ -409,6 +428,10 @@ alternative_insn eret, nop, ARM64_UNMAP_KERNEL_AT_EL0 */ .macro irq_stack_exit mov sp, x19 +#ifdef CONFIG_SHADOW_CALL_STACK + /* x20 is also preserved */ + mov x18, x20 +#endif .endm /* GPRs used by entry code */ @@ -1155,6 +1178,11 @@ ENTRY(cpu_switch_to) ldr lr, [x8] mov sp, x9 msr sp_el0, x1 +#ifdef CONFIG_SHADOW_CALL_STACK + str x18, [x0, #TSK_TI_SCS] + ldr x18, [x1, #TSK_TI_SCS] + str xzr, [x1, #TSK_TI_SCS] // limit visibility of saved SCS +#endif ret ENDPROC(cpu_switch_to) NOKPROBE(cpu_switch_to) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 989b1944cb71..2be977c6496f 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -27,6 +27,7 @@ #include #include #include +#include #include #include #include @@ -424,6 +425,10 @@ __primary_switched: stp xzr, x30, [sp, #-16]! mov x29, sp +#ifdef CONFIG_SHADOW_CALL_STACK + adr_l x18, init_shadow_call_stack // Set shadow call stack +#endif + str_l x21, __fdt_pointer, x5 // Save FDT pointer ldr_l x4, kimage_vaddr // Save the offset between @@ -731,6 +736,10 @@ __secondary_switched: ldr x2, [x0, #CPU_BOOT_TASK] cbz x2, __secondary_too_slow msr sp_el0, x2 +#ifdef CONFIG_SHADOW_CALL_STACK + ldr x18, [x2, #TSK_TI_SCS] // Set shadow call stack + str xzr, [x2, #TSK_TI_SCS] +#endif mov x29, #0 mov x30, #0 b secondary_start_kernel diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c index 04a327ccf84d..fe0ca522ff60 100644 --- a/arch/arm64/kernel/irq.c +++ b/arch/arm64/kernel/irq.c @@ -21,6 +21,7 @@ #include #include #include +#include unsigned long irq_err_count; @@ -63,6 +64,7 @@ static void init_irq_stacks(void) void __init init_IRQ(void) { init_irq_stacks(); + scs_init_irq(); irqchip_init(); if (!handle_arch_irq) panic("No interrupt controller found."); diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index 71f788cd2b18..5f0aec285848 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -52,6 +52,7 @@ #include #include #include +#include #include #if defined(CONFIG_STACKPROTECTOR) && !defined(CONFIG_STACKPROTECTOR_PER_TASK) @@ -507,6 +508,7 @@ __notrace_funcgraph struct task_struct *__switch_to(struct task_struct *prev, uao_thread_switch(next); ptrauth_thread_switch(next); ssbs_thread_switch(next); + scs_overflow_check(next); /* * Complete any pending TLB or cache maintenance on this CPU in case diff --git a/arch/arm64/kernel/scs.c b/arch/arm64/kernel/scs.c new file mode 100644 index 000000000000..6f255072c9a9 --- /dev/null +++ b/arch/arm64/kernel/scs.c @@ -0,0 +1,39 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Shadow Call Stack support. + * + * Copyright (C) 2019 Google LLC + */ + +#include +#include +#include + +DEFINE_PER_CPU(unsigned long *, irq_shadow_call_stack_ptr); + +#ifndef CONFIG_SHADOW_CALL_STACK_VMAP +DEFINE_PER_CPU(unsigned long [SCS_SIZE/sizeof(long)], irq_shadow_call_stack) + __aligned(SCS_SIZE); +#endif + +void scs_init_irq(void) +{ + int cpu; + + for_each_possible_cpu(cpu) { +#ifdef CONFIG_SHADOW_CALL_STACK_VMAP + unsigned long *p; + + p = __vmalloc_node_range(SCS_SIZE, SCS_SIZE, + VMALLOC_START, VMALLOC_END, + SCS_GFP, PAGE_KERNEL, + 0, cpu_to_node(cpu), + __builtin_return_address(0)); + + per_cpu(irq_shadow_call_stack_ptr, cpu) = p; +#else + per_cpu(irq_shadow_call_stack_ptr, cpu) = + per_cpu(irq_shadow_call_stack, cpu); +#endif /* CONFIG_SHADOW_CALL_STACK_VMAP */ + } +} diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index dc9fe879c279..cc1938a585d2 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -44,6 +44,7 @@ #include #include #include +#include #include #include #include @@ -357,6 +358,9 @@ void cpu_die(void) { unsigned int cpu = smp_processor_id(); + /* Save the shadow stack pointer before exiting the idle task */ + scs_save(current); + idle_task_exit(); local_daif_mask();