From patchwork Fri Nov 19 06:09:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Reiji Watanabe X-Patchwork-Id: 12693103 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88411C433F5 for ; Fri, 19 Nov 2021 06:11:22 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5077F61A6C for ; Fri, 19 Nov 2021 06:11:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 5077F61A6C Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=fB0NlHpLsEMFSS/XrhMpfl+DrvrdjqJg9NdbMKAxuAk=; b=LQb8UmsnB16bOdAIC9xcZgffaJ VtAfOnM8sDF/tfWBPckIZMRpxVFwoHZdQEFrSyUnhQtKYk6jPqeEZQ5WbcA5USKTimprFl/LrIS65 6TVuxcVFRVTBRTrlr77uFIGS8KGc0K9QhRyMwW1fovjC/X1t20oEm7uOByWybDkFx2zqmsptozjrR C1UhVtl7Hh1LhsKVoh1IqnYRd5TMZeLI5SEmjPir4VyqRLUjObq5m5NOB7Kt9kfSRRX8GavAiBmeY vuqFzc0SZFds/qtx/J3uiq8phq0LbLhB0tTzXD3/c6gaHA7qh1A9uQ+1Gw0KGph7URUToeoojnBI/ 963lT11g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mnx5p-009W1o-4U; Fri, 19 Nov 2021 06:09:49 +0000 Received: from mail-pf1-x44a.google.com ([2607:f8b0:4864:20::44a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mnx5l-009W1I-IE for linux-arm-kernel@lists.infradead.org; Fri, 19 Nov 2021 06:09:46 +0000 Received: by mail-pf1-x44a.google.com with SMTP id w2-20020a627b02000000b0049fa951281fso5084345pfc.9 for ; Thu, 18 Nov 2021 22:09:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=+0epaIan2LhTQH+KABtB/Z1kC0F9655cUhxxE8sPGSA=; b=IJgSbuPiRzRdNY/qCEUJEznXOV+ZtHnQT2hOlWd4ME+Qv+Wcw9SM2CdO/04ggAgFhD 6qC39bO3RQuofEFSVtMgg80Uu6UhcZM4PmMW0RCkZTc41rcAGOdYwCg3o5bhax2MXrEc 6dE7e8av7am+B+49CLLoIYRDtL/j3IqhvB+MhumNDh7Ed6k9jE5zPytwEiSA7zQpn8JT oeVlFcal4hwG+Xuaz7X/0leCDrsJ659qdpKkW6pMnTYbSW4Dc6G9b/sJJp2nNFpm6coX cCdKzRx8umqwPtwroYLD+O+hwvZ4aCDDRyZkyMBmrYvXY7GAjOZbm8K333KBdqouVTc6 dX5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=+0epaIan2LhTQH+KABtB/Z1kC0F9655cUhxxE8sPGSA=; b=LjdgbpaVBi39efvSFPAHZPwhjOZquQHQfuujaw1hxD74cb/DwSvOGRflZ4mtWhgP2b OklrPjHcgZw7gqab486bYLcUJqUrZYxcWDaFSIzXpigYgmPAX5OmkUpE9tUplURRDF9f kME4NlbhgW//i9LIy9L2IqIzDWWlO3S8MNuVOokDz5qQN5WNwm+evhyJxJsEnM2gNLND HtXa1EE8SYOicoBvfuRy05qjEW3/fsSY+T8EMDvqN1PsFt2JcndrFtw/3I1KRdbRL3MB fOwKKcVlRpu3lZqWzZWBPUdgFYWSKYwKkuKIlFvxKuqSA4eRfQwlMWfjhFlDbQj4TEWw lJ2Q== X-Gm-Message-State: AOAM531leJt7TLGTLjfwLRGfLVrkzw7pzkSFcMHlIG5X3yUYuIPxZ1MS dcu/+ynibeeKSjaMkAlr7zhDOC2Xlzw= X-Google-Smtp-Source: ABdhPJynkFFLMGDkiH7vTZk0Amnu9bHC1R/P5sVomu5sEe3QtHvmWQpsYa3nwo0Jkg1yXeyINZVmRM2/8mg= X-Received: from reiji-vws-sp.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3d59]) (user=reijiw job=sendgmr) by 2002:a17:902:7b82:b0:143:a6d6:34ab with SMTP id w2-20020a1709027b8200b00143a6d634abmr69475908pll.30.1637302183791; Thu, 18 Nov 2021 22:09:43 -0800 (PST) Date: Thu, 18 Nov 2021 22:09:26 -0800 In-Reply-To: <20211119060927.3185381-1-reijiw@google.com> Message-Id: <20211119060927.3185381-2-reijiw@google.com> Mime-Version: 1.0 References: <20211119060927.3185381-1-reijiw@google.com> X-Mailer: git-send-email 2.34.0.rc2.393.gf8c9666880-goog Subject: [PATCH v3 1/2] arm64: clear_page() shouldn't use DC ZVA when DCZID_EL0.DZP == 1 From: Reiji Watanabe To: Catalin Marinas , Will Deacon Cc: Marc Zyngier , Robin Murphy , Mark Rutland , Peter Collingbourne , Evgenii Stepanov , linux-arm-kernel@lists.infradead.org, guanghuifeng@linux.alibaba.com, Peter Shier , Ricardo Koller , Oliver Upton , Jing Zhang , Raghavendra Rao Anata , Reiji Watanabe X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211118_220945_644265_7946DC2D X-CRM114-Status: UNSURE ( 8.78 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, clear_page() uses DC ZVA instruction unconditionally. But it should make sure that DCZID_EL0.DZP, which indicates whether or not use of DC ZVA instruction is prohibited, is zero when using the instruction. Use STNP instead when DCZID_EL0.DZP == 1. Fixes: f27bb139c387 ("arm64: Miscellaneous library functions") Signed-off-by: Reiji Watanabe Reviewed-by: Robin Murphy --- arch/arm64/lib/clear_page.S | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/arch/arm64/lib/clear_page.S b/arch/arm64/lib/clear_page.S index b84b179edba3..1fd5d790ab80 100644 --- a/arch/arm64/lib/clear_page.S +++ b/arch/arm64/lib/clear_page.S @@ -16,6 +16,7 @@ */ SYM_FUNC_START_PI(clear_page) mrs x1, dczid_el0 + tbnz x1, #4, 2f /* Branch if DC ZVA is prohibited */ and w1, w1, #0xf mov x2, #4 lsl x1, x2, x1 @@ -25,5 +26,14 @@ SYM_FUNC_START_PI(clear_page) tst x0, #(PAGE_SIZE - 1) b.ne 1b ret + +2: stnp xzr, xzr, [x0] + stnp xzr, xzr, [x0, #16] + stnp xzr, xzr, [x0, #32] + stnp xzr, xzr, [x0, #48] + add x0, x0, #64 + tst x0, #(PAGE_SIZE - 1) + b.ne 2b + ret SYM_FUNC_END_PI(clear_page) EXPORT_SYMBOL(clear_page) From patchwork Fri Nov 19 06:09:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Reiji Watanabe X-Patchwork-Id: 12693104 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 715D0C433F5 for ; Fri, 19 Nov 2021 06:11:25 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3D0BE61A6C for ; Fri, 19 Nov 2021 06:11:25 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 3D0BE61A6C Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=PPZL9l+XvuYCquGrWUsRvStj1FXYHZCkA7qj81HPehc=; b=KdHblHgT2WGgGmH29DDyBTSijZ b+SQkg/qiIsNExH98J0k3TLAYIapJyHQrCp+NcKWjSgpiGZg1SXUs4UG9eJP4/CR0ZoaPETvnlbVW E7wcgO75FB3Kq7Bho+q0KoqcuxZY4OveCWpyA891qmJd4vJs6SDID5Il6OdglGaawLY9MJ5GM7xZ8 WEc7rpQKmLKyD/6w3tzQ/cPKyc1dRMg2Vodq3litIV3JXc3nwSM2XwLvWdyXyskj2MJNbR2zDe0+V y3P+Lp1I/bj67drGRFdGFb8RdARrYeFuhlVrvuVeAE6r1T2Wfp9EBVYfKsEhC0enRC2s87/apQD3G ju/7uSvg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mnx61-009W3F-0b; Fri, 19 Nov 2021 06:10:01 +0000 Received: from mail-pf1-x44a.google.com ([2607:f8b0:4864:20::44a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mnx5x-009W2Z-JW for linux-arm-kernel@lists.infradead.org; Fri, 19 Nov 2021 06:09:58 +0000 Received: by mail-pf1-x44a.google.com with SMTP id 184-20020a6217c1000000b0049f9aad0040so5056347pfx.21 for ; Thu, 18 Nov 2021 22:09:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=fcckAcmgEQbovsGgGLuMRFH/grtBIDbLvh2T+PSrY5U=; b=gcHbxi7tqCg0M8sof+1DwqiDeWRQOxX7DjUts1IvXWBONHNvCB4WsZ9nWA3ZbCO4Gj QxEYmrntmKicPAE3daqJO9eamcQYYyr4hLZAeUZRgjAp+q3GTuakL+ULzbn/HwY2adZe AxrCvdlI3wFTEmnWuUu1gjcOkpRu3TZHuW0EbCm1FiI+VqYfCnpYUx+CX3sE4PeXN9Sw KAUmE5sFVIEbm1ALgl4ZCLCV9D/c3bAyuff8mjQZ9IiEqiAnnTvRMZNXm3y6wBBrwS5t Is1Kztak3YI2uBXxPDMBQnPbLZY5J3TC4GX2N4xnsSPPd+vR/GUgoFi3Qp9PVWru9XMK UdJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=fcckAcmgEQbovsGgGLuMRFH/grtBIDbLvh2T+PSrY5U=; b=Y39W24chUs5qSv1sekMt3xJ4i5w+Amn/LUse+t+xejC5CLi0bdczNv/McRGMFQ7w2b cBtIz7QF3fixCQD7r7rgQ88CHpUDNOclRZLY0DQOU+lBlIbe915ySDuZvN0cUIYs39IG 1/44cNqaU+6qnnNf5kf5lqUttXkbofm9JuaYXskC7TrGctFBtVmbYhF1PrkQZ3zieG2H PCRi4GgCvGShENiNHPiQepWvV2Uls03L7PfLxclgV6k/ATZj2hn3lMiAvGoUEkZamMmP /EdjSiMhPCZI0BicIdruj90PHp0/0do14ZJeZRxPAJRMKoeEe57Ow/N5uYn1rtyb+xzc ghrA== X-Gm-Message-State: AOAM532eKetCqnTbhCEd3fyTAWoHAJl/MOylXV6ikb2UcqscaFarPsIb PSnDIVyoAqmfvSxMS+YLHpMEhY0owdQ= X-Google-Smtp-Source: ABdhPJzFXX4Xk5uYFpfMpyJJsoCcHY+O6x2k6zcowYF3iwK+5iR2utuIb2fGErLrP+349WkBpKhJzKfDyQc= X-Received: from reiji-vws-sp.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3d59]) (user=reijiw job=sendgmr) by 2002:a05:6a00:10d2:b0:44d:f03e:46c7 with SMTP id d18-20020a056a0010d200b0044df03e46c7mr61693766pfu.0.1637302196285; Thu, 18 Nov 2021 22:09:56 -0800 (PST) Date: Thu, 18 Nov 2021 22:09:27 -0800 In-Reply-To: <20211119060927.3185381-1-reijiw@google.com> Message-Id: <20211119060927.3185381-3-reijiw@google.com> Mime-Version: 1.0 References: <20211119060927.3185381-1-reijiw@google.com> X-Mailer: git-send-email 2.34.0.rc2.393.gf8c9666880-goog Subject: [PATCH v3 2/2] arm64: mte: DC {GVA,GZVA} shouldn't be used when DCZID_EL0.DZP == 1 From: Reiji Watanabe To: Catalin Marinas , Will Deacon Cc: Marc Zyngier , Robin Murphy , Mark Rutland , Peter Collingbourne , Evgenii Stepanov , linux-arm-kernel@lists.infradead.org, guanghuifeng@linux.alibaba.com, Peter Shier , Ricardo Koller , Oliver Upton , Jing Zhang , Raghavendra Rao Anata , Reiji Watanabe X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211118_220957_672028_682FCAAC X-CRM114-Status: GOOD ( 12.95 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, mte_set_mem_tag_range() and mte_zero_clear_page_tags() use DC {GVA,GZVA} unconditionally. But, they should make sure that DCZID_EL0.DZP, which indicates whether or not use of those instructions is prohibited, is zero when using those instructions. Use ST{G,ZG} instead when DCZID_EL0.DZP == 1. Fixes: 013bb59dbb7c ("arm64: mte: handle tags zeroing at page allocation time") Fixes: 3d0cca0b02ac ("kasan: speed up mte_set_mem_tag_range") Signed-off-by: Reiji Watanabe --- arch/arm64/include/asm/mte-kasan.h | 8 +++++--- arch/arm64/lib/mte.S | 8 +++++++- 2 files changed, 12 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/mte-kasan.h b/arch/arm64/include/asm/mte-kasan.h index 478b9bcf69ad..e4704a403237 100644 --- a/arch/arm64/include/asm/mte-kasan.h +++ b/arch/arm64/include/asm/mte-kasan.h @@ -84,10 +84,12 @@ static inline void __dc_gzva(u64 p) static inline void mte_set_mem_tag_range(void *addr, size_t size, u8 tag, bool init) { - u64 curr, mask, dczid_bs, end1, end2, end3; + u64 curr, mask, dczid, dczid_bs, dczid_dzp, end1, end2, end3; /* Read DC G(Z)VA block size from the system register. */ - dczid_bs = 4ul << (read_cpuid(DCZID_EL0) & 0xf); + dczid = read_cpuid(DCZID_EL0); + dczid_bs = 4ul << (dczid & 0xf); + dczid_dzp = (dczid >> 4) & 1; curr = (u64)__tag_set(addr, tag); mask = dczid_bs - 1; @@ -106,7 +108,7 @@ static inline void mte_set_mem_tag_range(void *addr, size_t size, u8 tag, */ #define SET_MEMTAG_RANGE(stg_post, dc_gva) \ do { \ - if (size >= 2 * dczid_bs) { \ + if (!dczid_dzp && size >= 2 * dczid_bs) {\ do { \ curr = stg_post(curr); \ } while (curr < end1); \ diff --git a/arch/arm64/lib/mte.S b/arch/arm64/lib/mte.S index e83643b3995f..e62c048af337 100644 --- a/arch/arm64/lib/mte.S +++ b/arch/arm64/lib/mte.S @@ -43,17 +43,23 @@ SYM_FUNC_END(mte_clear_page_tags) * x0 - address to the beginning of the page */ SYM_FUNC_START(mte_zero_clear_page_tags) + and x0, x0, #(1 << MTE_TAG_SHIFT) - 1 // clear the tag mrs x1, dczid_el0 + tbnz x1, #4, 2f // Branch if DC GZVA is prohibited and w1, w1, #0xf mov x2, #4 lsl x1, x2, x1 - and x0, x0, #(1 << MTE_TAG_SHIFT) - 1 // clear the tag 1: dc gzva, x0 add x0, x0, x1 tst x0, #(PAGE_SIZE - 1) b.ne 1b ret + +2: stzg x0, [x0], #16 + tst x0, #(PAGE_SIZE - 1) + b.ne 2b + ret SYM_FUNC_END(mte_zero_clear_page_tags) /*