From patchwork Thu Jun 28 14:50:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wei Xu X-Patchwork-Id: 10494193 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id CB6C160230 for ; Thu, 28 Jun 2018 14:51:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B3FEE29DAE for ; Thu, 28 Jun 2018 14:51:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A7D1C29D84; Thu, 28 Jun 2018 14:51:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id B7E542A3FA for ; Thu, 28 Jun 2018 14:51:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date: Message-ID:From:References:To:Subject:Reply-To:Content-ID:Content-Description :Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=3Z7F6xgTaJ3I2BukwCIsQU/kpD/GUKGF1ok41WO4NgI=; b=m+0el6mi+FglE8 EKcvcGuwXjAoKNxRE1rmFsbCxh1TMC5D1K522fiThD3NXC2hUoPHXJrWHxohV9MZUVNWAh410SDa5 vukZRA5y4zTehB83tY+uBre81GMs5rx7uxPhWPZuUBiiF9FGXKLKtfZEDJ3cq6FBw/M1eHgpIteed KNzO3FfREvkyTzhmDZYLwF6iw3oBVdWsA74JwNlJ/J1enmUa70GNcYnPMYfRxVHufaScS5bPoW0pC FZdT2Ama+4DE7728Rs8DWvkGzX4uSyt3PyNcsE04RWMEjRqgvFhGkc/Uxf/4tyksRFrrZImYL0vjl U9O/OUvb5HH3hKRYkYww==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1fYYGk-0001NG-7f; Thu, 28 Jun 2018 14:51:34 +0000 Received: from szxga04-in.huawei.com ([45.249.212.190] helo=huawei.com) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1fYYGh-0001Lc-Hb for linux-arm-kernel@lists.infradead.org; Thu, 28 Jun 2018 14:51:33 +0000 Received: from DGGEMS413-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id DBF63442F0AD3; Thu, 28 Jun 2018 22:50:57 +0800 (CST) Received: from [127.0.0.1] (10.202.226.42) by DGGEMS413-HUB.china.huawei.com (10.3.19.213) with Microsoft SMTP Server id 14.3.382.0; Thu, 28 Jun 2018 22:50:52 +0800 Subject: Re: KVM guest sometimes failed to boot because of kernel stack overflow if KPTI is enabled on a hisilicon ARM64 platform. To: Will Deacon References: <5B2A6218.3030201@hisilicon.com> <20180620144257.GB27776@arm.com> <5B2A7832.4010502@hisilicon.com> <5B2A7FE1.5040607@hisilicon.com> <5B2B6DEA.2090100@hisilicon.com> <5B3274FC.7000206@hisilicon.com> <20180626174746.GO23375@arm.com> <5B338F7B.9070500@hisilicon.com> <20180627132826.GB30631@arm.com> From: Wei Xu Message-ID: <5B34F5C0.9090001@hisilicon.com> Date: Thu, 28 Jun 2018 15:50:40 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 MIME-Version: 1.0 In-Reply-To: <20180627132826.GB30631@arm.com> X-Originating-IP: [10.202.226.42] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180628_075131_756937_AEA93DBA X-CRM114-Status: GOOD ( 16.88 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, zhangxiquan@hisilicon.com, suzuki.poulose@arm.com, marc.zyngier@arm.com, catalin.marinas@arm.com, wxf.wang@hisilicon.com, Linuxarm , linux-kernel@vger.kernel.org, dingshuai1@huawei.com, James Morse , "Xiongfanggou \(James\)" , Hanjun Guo , "Liyuan \(Larry, Turing Solution\)" , libeijian@hisilicon.com, "Liguozhu \(Kenneth\)" , Zhangyi ac , linux-arm-kernel@lists.infradead.org, dave.martin@arm.com Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Hi Will, On 2018/6/27 14:28, Will Deacon wrote: > On Wed, Jun 27, 2018 at 02:22:03PM +0100, Wei Xu wrote: >> On 2018/6/26 18:47, Will Deacon wrote: >>> If you look at the __idmap_kpti_put_pgtable_ent_ng asm macro, can you try >>> replacing: >>> >>> dc civac, cur_\()\type\()p >>> >>> with: >>> >>> dc ivac, cur_\()\type\()p >>> >>> please? Only do this for the guest kernel, not the host. KVM will upgrade >>> the clean to a clean+invalidate, so it's interesting to see if this has >>> an effect on the behaviour. >> >> Only changed the guest kernel, the guest still failed to boot and the log >> is same with the last mail. >> >> But if I changed to cvac as below for the guest, it is kind of stable. >> dc cvac, cur_\()\type\()p >> >> I have synced with our SoC guys about this and hope we can find the reason. >> Do you have any more suggestion? > > Unfortunately, not. It looks like somehow clean+invalidate is behaving > just as an invalidate, and we're corrupting the page table as a result. > > Hopefully the SoC guys will figure it out. After replaced the dmb with dsb in both __idmap_kpti_get_pgtable_ent and __idmap_kpti_put_pgtable_ent_ng, we tested 20 times and we can not reproduce the issue. Today we will continue to do the stress testing and will update the result tomorrow. The dsb in __idmap_kpti_get_pgtable_ent is to make sure the dc has been done and the following ldr can get the latest data. The dsb in __idmap_kpti_put_pgtable_ent_ng is to make sure the str will be done before dc. Although dmb can guarantee the order of the str and dc on the L2 cache, dmb can not guarantee the order on the bus. How do you think about it? Thanks! ---- > > Will > > . > diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 03646e6..bb767ea 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -209,7 +209,7 @@ ENDPROC(idmap_cpu_replace_ttbr1) .macro __idmap_kpti_get_pgtable_ent, type dc cvac, cur_\()\type\()p // Ensure any existing dirty - dmb sy // lines are written back before + dsb sy // lines are written back before ldr \type, [cur_\()\type\()p] // loading the entry tbz \type, #0, skip_\()\type // Skip invalid and tbnz \type, #11, skip_\()\type // non-global entries @@ -218,8 +218,9 @@ ENDPROC(idmap_cpu_replace_ttbr1) .macro __idmap_kpti_put_pgtable_ent_ng, type orr \type, \type, #PTE_NG // Same bit for blocks and pages str \type, [cur_\()\type\()p] // Update the entry and ensure - dmb sy // that it is visible to all + dsb sy // that it is visible to all dc civac, cur_\()\type\()p // CPUs. Best Regards, Wei