From patchwork Fri Jan 15 20:02:48 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chris Metcalf X-Patchwork-Id: 8044441 Return-Path: X-Original-To: patchwork-linux-arm@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id B082C9F1C0 for ; Fri, 15 Jan 2016 20:05:06 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 8FE5220390 for ; Fri, 15 Jan 2016 20:05:05 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.9]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B3F4320386 for ; Fri, 15 Jan 2016 20:05:03 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1aKAak-0006Kj-IC; Fri, 15 Jan 2016 20:03:26 +0000 Received: from mail-am1on0078.outbound.protection.outlook.com ([157.56.112.78] helo=emea01-am1-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1aKAag-0006Fx-9I for linux-arm-kernel@lists.infradead.org; Fri, 15 Jan 2016 20:03:24 +0000 Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=cmetcalf@ezchip.com; Received: from [10.7.0.41] (12.216.194.146) by AM3PR02MB113.eurprd02.prod.outlook.com (2a01:111:e400:8807::12) with Microsoft SMTP Server (TLS) id 15.1.361.13; Fri, 15 Jan 2016 20:02:57 +0000 To: Marc Zyngier , Christoffer Dall , , From: Chris Metcalf Subject: [PATCH] help guest boot up on AArch64 host with GICv2 Message-ID: <56995068.9050207@ezchip.com> Date: Fri, 15 Jan 2016 15:02:48 -0500 User-Agent: Mozilla/5.0 (X11; Linux i686 on x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 X-Originating-IP: [12.216.194.146] X-ClientProxiedBy: BL2PR01CA0023.prod.exchangelabs.com (2a01:111:e400:c1b::23) To AM3PR02MB113.eurprd02.prod.outlook.com (2a01:111:e400:8807::12) X-Microsoft-Exchange-Diagnostics: 1; AM3PR02MB113; 2:W2CshyKbQjr1ofj0865zfHixlrqFL9TgkWtmxxMCdShRwgX0y/a6t0M69Vsr+N6qkqYNOXxhNQUKuxtW0G1q4Pn5fZgHkJaTfxY6Gz+Xj0EGeR8av6ThMs/c85rrYc/S/LboFAx6Ktf3FPDeYVWtRQ==; 3:kASnObNsMSPflEwtvpmItZOCB+TOfoArK7MlBQFvmN6Ncu3Xanuo9QWBZE4iKBexJ1HYakWTd7dJeBaKFLIr8q1a1rdvAaN97uReobMpo7Asu8EcwIp0vLY3EW/oGPCH; 25:M6F1o0PlpdcWZgNTLxtO5hk2tGfWwWGollH9+DFDIfS+1M6+adrpKxZT3yzzEBYwwu/R1ZWiSdlyH2slcYG5vS9qBe4Lyi8nQEzAjnucOeYMnlcrYcusovlMWKjVClt8Be1sHGut4mx3zUJpyBbU59joJ2/WjBzdt/HQq10OKxqq1sCNfEaEAfTZU1q0u3JHo+D42xYQ/w+GQZZaOq92ZMppYrc0Vc2oAL8uzcyN+CNY3JisCd2Azr5Kf0a0JGy7; 20:7XSXeaqQaFEf0CI4/QSrkhIbxQaPkBMYQuOMLy5+fWTyLvyJZoH2vJ8jcZzmEpS1365wJ3gB03l943dhDKZ0iGhKJZHllA8+foCGSY1xRMjDBxX7GF6ghTj7EJMCAJEtuaIJo2ALvEIeduxiyUlG5+fpKMeTsDQsA9WEyWlqgDg= X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:AM3PR02MB113; X-MS-Office365-Filtering-Correlation-Id: 094e0d8a-92ce-4752-5ee9-08d31de6dcf9 X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(121898900299872); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(102615245)(601004)(2401047)(520078)(5005006)(8121501046)(10201501046)(3002001); SRVR:AM3PR02MB113; BCL:0; PCL:0; RULEID:; SRVR:AM3PR02MB113; X-Microsoft-Exchange-Diagnostics: 1; AM3PR02MB113; 4:4YB5+2C12GYVGAP/jP3oxWK6Ze4ItoxhClEyv3d2TcsLCENaKoq7Vuw5sjojupdTeoHNsabJMVR3DCVdvsSTp6ufskD0hnwPwAtYJQ92NjdocJXg9TEaYDVhf+KpGe3gYc7Y+CGWy2dSB93W1dUF3In/whnTlsE6CbTUR8umJBW0FvHeP+ySmN9ylbTJ3LlQXKJe+Czxx7L49dQ+PXJp1ct5xG4sunzD4sUXpF4YvCnXld7M5k7RxUdeq9kiRxwChaM2woin0DxFZw/wyxmJyGJw1FABnx4ZTh8K9K7HannqOs/49VAZ0g5vQF7exS/pFAPLeZ+4l3UbK7LHYK7ZLx06xQqi6W4NMFnjyA/G4ulvvjE6VoRQoWsEymqYbkE+8sM4ULWF5iz+aM5ed6uykpc3fdnN8agdHtPycA6c71LYZOoMbFXmYD2b6dWfe1GZmpAi0fnbirBeVx1yAEAxFA== X-Forefront-PRVS: 08220FA8D6 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6049001)(6009001)(189002)(199003)(15975445007)(36756003)(512874002)(83506001)(16601075003)(77096005)(86362001)(229853001)(122386002)(84326002)(101416001)(87976001)(4610100001)(5004730100002)(2201001)(2171001)(568964002)(3846002)(586003)(5008740100001)(92566002)(42186005)(107886002)(6116002)(19580395003)(189998001)(105586002)(99136001)(65806001)(5890100001)(81156007)(106356001)(2476003)(5001960100002)(66066001)(97736004)(4001350100001)(65816999)(54356999)(40100003)(87266999)(59896002)(270700001)(65956001)(50986999)(1096002)(5001770100001)(2906002)(117636001)(64126003)(62816006); DIR:OUT; SFP:1101; SCL:1; SRVR:AM3PR02MB113; H:[10.7.0.41]; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: ezchip.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; AM3PR02MB113; 23:jdQsSQvnC/2BbNPRDq145dZFktHDH9mNE9geiXnWb8?= =?us-ascii?Q?oZoxwLdcFKk4fE+r8BZtUgfIa+SDmatyOcFf/Ed3YnJsB/6ATJQWNFERSMHq?= =?us-ascii?Q?2MIG6jMfdwAxMXxWcfmQmDOtfL+5GDZ9/HwS6oj1BdHtC3rwUcFtY1MTvKuY?= =?us-ascii?Q?Owq+tRQuDXV8ICY7HAtNbi83P4OnRRIXRsj1NrGoXSpzykEdfsklxF+xW/m7?= =?us-ascii?Q?YuoXXaSQp8SxkCq7jNjq/pmexqu9OfWoUCv5m/IMQOPPl1fZIZOjzYHuhjtj?= =?us-ascii?Q?F8DFx0QkLZgpxZzK9HwJATvR4OYjM9juN+GIbOnVmtPH+ofJ1q4q1wTURwlG?= =?us-ascii?Q?9tftYYWXnjDy/wtBhcN4Nst3+Vcq6Zhi8uv8nRMIt8ukQdiEEA0VP5XJ55ze?= =?us-ascii?Q?Q0RlSH27Ce+T5nI6oGUGLxf4pRGV2rYYUkL8nAcC0xM6QLGPMk1/LxDUfRB2?= =?us-ascii?Q?DykZAvtMkFdx99xr3hbm6PZiP/h+x/MxDYizZP9fOCgfRsPFG6RZeXyNz9+l?= =?us-ascii?Q?ZV37a2l/N1iqVi3l8YluH0RglEMwXX7afvyY8VeB/Z8n41pa0ZPSOJJCxyXj?= =?us-ascii?Q?HS10NtLzSiocLsQKfrtPCiyFquNiGl0LNjmMMuvIbOGU4QA4yNTWbelqoE1R?= =?us-ascii?Q?5TnSVGSeWPjnvMV9YCLp8ssyU87AH6VfLEcmuWgGGlsC1qQTGkzsRZYqqaOQ?= =?us-ascii?Q?kaRHPrz+gntsToVn5u0AddPg6MpRHpLgOKq626/SvdY895y0sVtpta6c+J1E?= =?us-ascii?Q?9Tlihlqxm5GG9qxbtHM/ca2t/X+vZEBJ+r0Vxw+0nYbB5IlHzb7iOYGP5z2z?= =?us-ascii?Q?+Y13QmSlfSDxxWubyq5AtG9nd8TjAp+/DOumMF1eaDC2NMky+otLZsQFETp4?= =?us-ascii?Q?bst3djdS13RGJedbyqimU9bagpW0ZM9C47FDvZ0Pb5YLhv8sMOZ9vTmmM/LN?= =?us-ascii?Q?XHB6m2eSxB7jR05J05crz7NlwrCZp8LoMxy7drGPJE8VXtTSIR5ZKIuPdj5V?= =?us-ascii?Q?9L9RU+AAEHjiBr4lmtsYSeX5BjOd07RpTP6sSfXS3TKdy9PHZ4kxw/CGJjQU?= =?us-ascii?Q?PGZ4ma3dwu4bqtNPznCwlfQKXIdoZZvErqr8ypQTtRVEN1dCYjBm2GLdleY+?= =?us-ascii?Q?DYCn6XNTFA96GfeKclp16H9pWUv2qqwmbVNhyJOT8OQyRPu0MOYBJQwDPRJA?= =?us-ascii?Q?TtCOIxO3rAiqnh2BFl3U1nFZv9uHSJ1P+l1Sn7lRCP1Es4GAX7SqrIeYNm7m?= =?us-ascii?Q?2N3VvnBZ56jMDxYLylBGffMk3T44OnR7nwW5vEuCiTuVxO+FRpvMbCKQEN4m?= =?us-ascii?Q?3FlUhBN/AM4P5PdFjN9e8+iyBdf1B1fNhM9F8H1imRPqvV9RjIh6xopWInId?= =?us-ascii?Q?3VnhetIoNi2rz+WoA6QfIEnEl4KUhfqhNZk8IpoSQ7R1tj?= X-Microsoft-Exchange-Diagnostics: 1; AM3PR02MB113; 5:Rp+Z3Jxw0U5XxOBBaV7RYT50pexsOX9IxO62GGJFk7Njnt9XQoL77h88XV7aj06Aas38Qr+WIIxZSx19Y0TDrIcjCvJSj72+dLHAVQ+RL/HS8C6zwlcKAtohOotM0eFzDv9JWI8jwATUygwO2iyo/Q==; 24:gTAcbvQUMpC+xLbHYSBIbKSxlsraHXkyIhmeZhVn1aHKv9yesUUywjj3S0lVIa9yQsDHZaQMEAa9BI24yw3PdGNmGZnT4Y4P9uFyecaoskY= SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: ezchip.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jan 2016 20:02:57.0301 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM3PR02MB113 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20160115_120322_819219_70183C66 X-CRM114-Status: GOOD ( 19.73 ) X-Spam-Score: -1.9 (-) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Spam-Status: No, score=-4.2 required=5.0 tests=BAD_ENC_HEADER,BAYES_00, RCVD_IN_DNSWL_MED, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP We are using GICv2 compatibility mode in the Fast Models/Foundation Models simulations we are running because the boot code (ATF/UEFI) doesn't support GICv3 in our system at the moment. However, starting with kernel 4.2, the guest couldn't boot up because it wasn't getting timer interrupts. I tracked this down to a kernel commit that switched to using the "alternatives" mechanism -- rather than seeing either a GICv2 or GICv3 and configuring appropriately, the KVM code just configured the code that saves/restores the vgic state based on the presence of the system register interface to the GIC CPU interface. See the attached patch for a fix that manages this differently and allows me to boot up the guest in this configuration. However, even assuming this patch can be taken into an upstream tree, I still have a couple of additional problems: - I can boot up with the Foundation Models using this change, but not with the Fast Models (again, using a v3 GIC but in v2 compatibility mode in the device tree). The Fast Models dts looks like it has the same configuration for the GIC and the timers so I'm not sure what's going on here. Any suggestions appreciated. - Without this change, I could only boot kernels up to 4.1. With the change, I can boot kernels up to 4.3. But 4.4 won't boot for me either; I haven't bisected it down yet. So any suggestions on what might be going wrong here would also be appreciated. We are planning to eventually use GICv3 mode in our software stack but for the time being I assume it is interesting to resolve issues with GIC v2 compatibility mode on GIC v3. From 3dcb529de23adb918b9a4d6eca717c737f380bc3 Mon Sep 17 00:00:00 2001 From: Chris Metcalf Date: Fri, 15 Jan 2016 13:18:06 -0500 Subject: [PATCH] gic: update save/restore pointers only when gic v3 detected The original code set up the VGIC save/restore calls in __kvm_vcpu_run() based on whether the GIC had been detected as v2 or v3. Commit 8a14849b4a35 ("arm64: KVM: Switch vgic save/restore to alternative_insn") switched to making that choice based on whether the processor feature register reports that the system register interface to the GIC CPU interface is supported. However, booting up with the GIC v3 in v2 compatibility mode (in this case on the Linaro Foundation Model simulator) we find that the v3 save/restore isn't the right thing, since we end up with no timer interrupts being delivered to the KVM guest. Reverting to a model where we set up the VGIC save/restore calls based on the actual GIC type fixes this. To do this and still keep the simplicity of the "alternatives" model, we instead leave the v2 branch-and-link instruction in place, but patch it dynamically to be a branch-and-link to the v3 routines if we detect a v3 GIC. Signed-off-by: Chris Metcalf --- arch/arm/include/asm/kvm_host.h | 5 +++++ arch/arm64/include/asm/kvm_host.h | 37 +++++++++++++++++++++++++++++++++++++ arch/arm64/kvm/hyp.S | 14 ++++---------- virt/kvm/arm/vgic.c | 3 +++ 4 files changed, 49 insertions(+), 10 deletions(-) diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h index c4072d9f32c7..a34ce4d73498 100644 --- a/arch/arm/include/asm/kvm_host.h +++ b/arch/arm/include/asm/kvm_host.h @@ -216,6 +216,11 @@ static inline int kvm_arch_dev_ioctl_check_extension(long ext) return 0; } +static inline void vgic_arch_setup(const struct vgic_params *vgic) +{ + BUG_ON(vgic->type != VGIC_V2); +} + int kvm_perf_init(void); int kvm_perf_teardown(void); diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index ed039688c221..9ce98e69b5ec 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -27,6 +27,7 @@ #include #include #include +#include #define __KVM_HAVE_ARCH_INTC_INITIALIZED @@ -244,6 +245,42 @@ static inline void __cpu_init_hyp_mode(phys_addr_t boot_pgd_ptr, hyp_stack_ptr, vector_ptr); } +#ifdef CONFIG_ARM_GIC_V3 +/* Write a 'bl FUNC' instruction at address CALLSITE. */ +static inline void vgic_patch(char *callsite, char *func) +{ + aarch64_insn_patch_text_nosync( + callsite, + aarch64_insn_gen_branch_imm((long)callsite, (long)func, + AARCH64_INSN_BRANCH_LINK)); +} +#endif + +static inline void vgic_arch_setup(const struct vgic_params *vgic) +{ + switch(vgic->type) + { + case VGIC_V2: + break; + +#ifdef CONFIG_ARM_GIC_V3 + case VGIC_V3: + { + extern char __save_vgic_state_insn[]; + extern char __save_vgic_v3_state[]; + extern char __restore_vgic_state_insn[]; + extern char __restore_vgic_v3_state[]; + vgic_patch(__save_vgic_state_insn, __save_vgic_v3_state); + vgic_patch(__restore_vgic_state_insn, __restore_vgic_v3_state); + break; + } +#endif + + default: + BUG(); + } +} + static inline void kvm_arch_hardware_disable(void) {} static inline void kvm_arch_hardware_unsetup(void) {} static inline void kvm_arch_sync_events(struct kvm *kvm) {} diff --git a/arch/arm64/kvm/hyp.S b/arch/arm64/kvm/hyp.S index e5836138ec42..6b1eded53051 100644 --- a/arch/arm64/kvm/hyp.S +++ b/arch/arm64/kvm/hyp.S @@ -518,11 +518,8 @@ * Call into the vgic backend for state saving */ .macro save_vgic_state -alternative_if_not ARM64_HAS_SYSREG_GIC_CPUIF - bl __save_vgic_v2_state -alternative_else - bl __save_vgic_v3_state -alternative_endif +ENTRY(__save_vgic_state_insn) + bl __save_vgic_v2_state // may update to __save_vgic_v3_state mrs x24, hcr_el2 mov x25, #HCR_INT_OVERRIDE neg x25, x25 @@ -539,11 +536,8 @@ alternative_endif orr x24, x24, #HCR_INT_OVERRIDE orr x24, x24, x25 msr hcr_el2, x24 -alternative_if_not ARM64_HAS_SYSREG_GIC_CPUIF - bl __restore_vgic_v2_state -alternative_else - bl __restore_vgic_v3_state -alternative_endif +ENTRY(__restore_vgic_state_insn) + bl __restore_vgic_v2_state // may update to __restore_vgic_v3_state .endm .macro save_timer_state diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c index 66c66165e712..8b4215414d11 100644 --- a/virt/kvm/arm/vgic.c +++ b/virt/kvm/arm/vgic.c @@ -2502,6 +2502,9 @@ int kvm_vgic_hyp_init(void) goto out_free_irq; } + /* Callback into for arch code for setup */ + vgic_arch_setup(vgic); + on_each_cpu(vgic_init_maintenance_interrupt, NULL, 1); return 0; -- 2.1.2