From patchwork Fri Mar 11 17:57:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 12778503 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6D343C433EF for ; Fri, 11 Mar 2022 18:00:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=/pqEHmqvvpXDRSzOvtkQIIVIAkGAjgAyBaoaKV7Xt68=; b=Acp00qBx8nE5xIgFe138btquAn pKsJOJaQxdOb3e2Sl3mjPgLBesPQUwWUn0/JU+438rS9Au+NeFADiPYzSZep8/7fLJXHK1nIL7rJm YejW0acB1IFv6ZkyMv/8yXhyYyIrcZ/p/W0kCJMYRZD5nuDS/LnLnEgdDOh67eH9LyOcD14MoZMcU vbZLB0p0izgaqZTQ9e1dCeusrW65Bb5ceSwwi1rk1HPmB7jvRklKV1cF6JAAecwbYBu8VwOWqNEE+ /tLnG2hj4XoEHYiZNyNY/x/8EnpEKkPzSIOol5zeFcYKOuuUGmHeiDnLFEdmaUYBFZSKLJVeFPD/W ZKzid0IA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nSjX9-00HYgm-TO; Fri, 11 Mar 2022 17:58:36 +0000 Received: from mail-io1-xd4a.google.com ([2607:f8b0:4864:20::d4a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nSjWN-00HYKG-PK for linux-arm-kernel@lists.infradead.org; Fri, 11 Mar 2022 17:57:52 +0000 Received: by mail-io1-xd4a.google.com with SMTP id w28-20020a05660205dc00b00645d3cdb0f7so6762092iox.10 for ; Fri, 11 Mar 2022 09:57:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=4Pv27JBfLZfH/CmvIWK/wVlW7nXuMPk8mN4kvQnrF8Y=; b=LkJFfaF+QDSe8RcIrq5JqVH3HGsEBs6ubj8L5VyPFuDtXYYvW5BAzKcm+lxvvq4MKL iw6Ci+dRx3gDW6Il7hzdXHUXBSOu2ZR3LpsAvK3ISNBGxKKNzwD7mB11XiEaz+Ef7jIh vTpUPJM1SnQPELYHqYezGl0SjuOO9A1V0ezaQ8f+PgJnQ+WwCGIW4wG5me1oNdrmXwxI DmBiVhMsWpXaKbTiNZMqkwamgxdJR2++mu51KA23u7F7k7p9sdvSrPnB/xpF/QzHDKy/ d8EFpp61p/23yPk7SAe/1JkL3dRSgNU0l0v5RNMsOVmyBmEaHQstYer3GWULVNLnaz72 +kvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=4Pv27JBfLZfH/CmvIWK/wVlW7nXuMPk8mN4kvQnrF8Y=; b=4PSx4Hk0nTNAWiLWErsOBrZAy60XhpzCAczgagvxgCChT6s/rNx8XB552jvm0CtHlw xdXdG1r+HUzL8Mq3uJlSpcLAoVUkOFrUtTSBt6kW9Ea6JiJ1YVyslVPI/PfR+wyUBfth g8p5s2wF/Y5KCXs4BOnDIaXOTvsiej1QddSX57prNC4aOGRDGZkdT875EBOfDbH/b+hq nYc+MXwbuYNIXMB4QqjZRbYSzIDKT74NSDCd+JsPy0QWueopZJdRSjplEz55Ylx38NWH 0dGV+bkhrvMsAUsfVbrtJu7XJT3o66z6MfuKp9eo7wpgMM1JAYElAvZEEuV1y3i4AA6K qkzw== X-Gm-Message-State: AOAM530dN5CGneSOxsxeojauoJgd4RqwTUt+s4pA/mT3NX4KxuD7Eyd2 dZQa6/uihvEITeckIoTej3OyjyfS3aM= X-Google-Smtp-Source: ABdhPJxMl3lNbjrYiw3dPTWd6Qtur4nKZbyrPolqLB3Wtk5CoxQO4h6cUg3DgclyKsOnvihxhaP1UAZjIDM= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a05:6638:d93:b0:317:ca63:2d38 with SMTP id l19-20020a0566380d9300b00317ca632d38mr3186286jaj.171.1647021466505; Fri, 11 Mar 2022 09:57:46 -0800 (PST) Date: Fri, 11 Mar 2022 17:57:17 +0000 In-Reply-To: <20220311175717.616958-1-oupton@google.com> Message-Id: <20220311175717.616958-6-oupton@google.com> Mime-Version: 1.0 References: <20220311174001.605719-1-oupton@google.com> <20220311175717.616958-1-oupton@google.com> X-Mailer: git-send-email 2.35.1.723.g4982287a31-goog Subject: [RFC PATCH kvmtool 5/5] ARM: Implement PSCI SYSTEM_SUSPEND From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , linux-arm-kernel@lists.infradead.org, Peter Shier , Ricardo Koller , Reiji Watanabe , Will Deacon , Julien Thierry , Oliver Upton X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220311_095747_873147_DA6C6A8F X-CRM114-Status: GOOD ( 17.27 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org KVM_CAP_ARM_SYSTEM_SUSPEND allows VMMs to trap guest attempts to use the PSCI SYSTEM_SUSPEND hypercall. Make use of that capability in KVM tool to implement guest suspend support. Add some minimal SMCCC register handling (params, return values) for AArch32 and AArch64. Perform only the required sanity check before suspending the VM by ensuring all other vCPUs besides the caller are powered off. Leverage KVM_MP_STATE_SUSPENDED to emulate the suspend as an architectural WFI. Signed-off-by: Oliver Upton --- arm/aarch32/kvm-cpu.c | 72 ++++++++++++++++++++++++ arm/aarch64/kvm-cpu.c | 66 ++++++++++++++++++++++ arm/include/arm-common/kvm-cpu-arch.h | 5 ++ arm/kvm-cpu.c | 81 +++++++++++++++++++++++++++ arm/kvm.c | 9 +++ 5 files changed, 233 insertions(+) diff --git a/arm/aarch32/kvm-cpu.c b/arm/aarch32/kvm-cpu.c index 95fb1da..dd5f70f 100644 --- a/arm/aarch32/kvm-cpu.c +++ b/arm/aarch32/kvm-cpu.c @@ -130,3 +130,75 @@ void kvm_cpu__show_registers(struct kvm_cpu *vcpu) die("KVM_GET_ONE_REG failed (LR_svc)"); dprintf(debug_fd, " LR_svc: 0x%x\n", data); } + +u64 kvm_cpu__smccc_get_arg1(struct kvm_vcpu *vcpu) +{ + struct kvm_one_reg reg; + u32 data; + + reg.addr = (u64)&data; + reg.id = ARM_CORE_REG(usr_regs.ARM_r1); + if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, ®) < 0) + die("KVM_GET_ONE_REG failed (r1)"); + + return data; +} + +u64 kvm_cpu__smccc_get_arg2(struct kvm_vcpu *vcpu) +{ + struct kvm_one_reg reg; + u32 data; + + reg.addr = (u64)&data; + reg.id = ARM_CORE_REG(usr_regs.ARM_r2); + if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, ®) < 0) + die("KVM_GET_ONE_REG failed (r2)"); + + return data; +} + +void kvm_cpu__smccc_return(struct kvm_cpu *vcpu, u64 a0, u64 a1, u64 a2, u64 a3) +{ + struct kvm_one_reg reg; + u32 data; + + reg.addr = (u64)&data; + + data = (u32)a0; + reg.id = ARM_CORE_REG(usr_regs.ARM_r0); + if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, ®) < 0) + die("KVM_SET_ONE_REG failed (r0)"); + + data = (u32)a1; + reg.id = ARM_CORE_REG(usr_regs.ARM_r1); + if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, ®) < 0) + die("KVM_SET_ONE_REG failed (r1)"); + + data = (u32)a2; + reg.id = ARM_CORE_REG(usr_regs.ARM_r2); + if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, ®) < 0) + die("KVM_SET_ONE_REG failed (r2)"); + + data = (u32)a3; + reg.id = ARM_CORE_REG(usr_regs.ARM_r3); + if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, ®) < 0) + die("KVM_SET_ONE_REG failed (r3)"); +} + +void kvm_cpu__psci_set_entry(struct kvm_cpu *vcpu, u64 entry_addr, u64 context_id) +{ + struct kvm_one_reg reg; + u32 data; + + reg.addr = (u64)&data; + + data = (u32)entry_addr; + reg.id = ARM_CORE_REG(usr_regs.ARM_pc); + if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, ®) < 0) + die("KVM_SET_ONE_REG failed (pc)"); + + data = (u32)context_id; + reg.id = ARM64_CORE_REG(usr_regs.ARM_r0); + if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, ®) < 0) + die("KVM_SET_ONE_REG failed (r0)"); +} diff --git a/arm/aarch64/kvm-cpu.c b/arm/aarch64/kvm-cpu.c index 9f3e858..3118c54 100644 --- a/arm/aarch64/kvm-cpu.c +++ b/arm/aarch64/kvm-cpu.c @@ -254,3 +254,69 @@ void kvm_cpu__show_registers(struct kvm_cpu *vcpu) die("KVM_GET_ONE_REG failed (lr)"); dprintf(debug_fd, " LR: 0x%lx\n", data); } + +u64 kvm_cpu__smccc_get_arg1(struct kvm_cpu *vcpu) +{ + struct kvm_one_reg reg; + u64 data; + + reg.addr = (u64)&data; + reg.id = ARM64_CORE_REG(regs.regs[1]); + if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, ®) < 0) + die("KVM_GET_ONE_REG failed (x1)"); + + return data; +} + +u64 kvm_cpu__smccc_get_arg2(struct kvm_cpu *vcpu) +{ + struct kvm_one_reg reg; + u64 data; + + reg.addr = (u64)&data; + reg.id = ARM64_CORE_REG(regs.regs[2]); + if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, ®) < 0) + die("KVM_GET_ONE_REG failed (x2)"); + + return data; +} + +void kvm_cpu__smccc_return(struct kvm_cpu *vcpu, u64 a0, u64 a1, u64 a2, u64 a3) +{ + struct kvm_one_reg reg; + + reg.addr = (u64)&a0; + reg.id = ARM64_CORE_REG(regs.regs[0]); + if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, ®) < 0) + die("KVM_SET_ONE_REG failed (x0)"); + + reg.addr = (u64)&a1; + reg.id = ARM64_CORE_REG(regs.regs[1]); + if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, ®) < 0) + die("KVM_SET_ONE_REG failed (x1)"); + + reg.addr = (u64)&a2; + reg.id = ARM64_CORE_REG(regs.regs[2]); + if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, ®) < 0) + die("KVM_SET_ONE_REG failed (x2)"); + + reg.addr = (u64)&a3; + reg.id = ARM64_CORE_REG(regs.regs[3]); + if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, ®) < 0) + die("KVM_SET_ONE_REG failed (x3)"); +} + +void kvm_cpu__psci_set_entry(struct kvm_cpu *vcpu, u64 entry_addr, u64 context_id) +{ + struct kvm_one_reg reg; + + reg.addr = (u64)&entry_addr; + reg.id = ARM64_CORE_REG(regs.pc); + if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, ®) < 0) + die("KVM_SET_ONE_REG failed (pc)"); + + reg.addr = (u64)&context_id; + reg.id = ARM64_CORE_REG(regs.regs[0]); + if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, ®) < 0) + die("KVM_SET_ONE_REG failed (x0)"); +} diff --git a/arm/include/arm-common/kvm-cpu-arch.h b/arm/include/arm-common/kvm-cpu-arch.h index 4027afe..17179c2 100644 --- a/arm/include/arm-common/kvm-cpu-arch.h +++ b/arm/include/arm-common/kvm-cpu-arch.h @@ -59,4 +59,9 @@ static inline bool kvm_cpu__emulate_mmio(struct kvm_cpu *vcpu, u64 phys_addr, unsigned long kvm_cpu__get_vcpu_mpidr(struct kvm_cpu *vcpu); +u64 kvm_cpu__smccc_get_arg1(struct kvm_cpu *vcpu); +u64 kvm_cpu__smccc_get_arg2(struct kvm_cpu *vcpu); +void kvm_cpu__smccc_return(struct kvm_cpu *vcpu, u64 a0, u64 a1, u64 a2, u64 a3); +void kvm_cpu__psci_set_entry(struct kvm_cpu *vcpu, u64 entry_addr, u64 context_id); + #endif /* ARM_COMMON__KVM_CPU_ARCH_H */ diff --git a/arm/kvm-cpu.c b/arm/kvm-cpu.c index 164e399..677f8bb 100644 --- a/arm/kvm-cpu.c +++ b/arm/kvm-cpu.c @@ -163,3 +163,84 @@ bool kvm_cpu__handle_exit(struct kvm_cpu *vcpu) void kvm_cpu__show_page_tables(struct kvm_cpu *vcpu) { } + +static void kvm_cpu__arm_suspend(struct kvm_cpu *vcpu) +{ + struct kvm_mp_state mp_state = { + .mp_state = KVM_MP_STATE_SUSPENDED, + }; + + if (ioctl(vcpu->vcpu_fd, KVM_SET_MP_STATE, &mp_state) < 0) + die("KVM_SET_MP_STATE failed"); +} + +static void kvm_cpu__wakeup(struct kvm_cpu *vcpu) +{ + u64 entry_addr, context_id; + + entry_addr = kvm_cpu__smccc_get_arg1(vcpu); + context_id = kvm_cpu__smccc_get_arg2(vcpu); + + /* + * The resuming CPU could have been a secondary CPU at boot. Ensure the + * vCPU is made runnable. + */ + vcpu->init.features[0] &= ~(1ul << KVM_ARM_VCPU_POWER_OFF); + + kvm_cpu__arch_reinit(vcpu); + kvm_cpu__reset_vcpu(vcpu); + kvm_cpu__psci_set_entry(vcpu, entry_addr, context_id); +} + +static void kvm_cpu__psci_system_suspend(struct kvm_cpu *vcpu) +{ + struct kvm *kvm = vcpu->kvm; + bool denied = false; + int i; + + /* + * Mark the caller as paused before actually pausing the VM. This avoids + * the hazard of attempting to acquire the pause_lock in the SIGKVMPAUSE + * handler from the thread that already holds it. + */ + vcpu->paused = 1; + + kvm__pause(kvm); + for (i = 0; i < kvm->nrcpus; i++) { + struct kvm_cpu *tmp = kvm->cpus[i]; + struct kvm_mp_state mp_state; + + if (vcpu == tmp) + continue; + + if (ioctl(tmp->vcpu_fd, KVM_GET_MP_STATE, &mp_state) < 0) + die("KVM_GET_MP_STATE failed"); + + if (mp_state.mp_state != KVM_MP_STATE_STOPPED) { + denied = true; + break; + } + } + + if (!denied) + kvm_cpu__arm_suspend(vcpu); + else + kvm_cpu__smccc_return(vcpu, PSCI_RET_DENIED, 0, 0, 0); + + vcpu->paused = 0; + kvm__continue(kvm); +} + +bool kvm_cpu__arch_handle_system_event(struct kvm_cpu *vcpu) +{ + switch (vcpu->kvm_run->system_event.type) { + case KVM_SYSTEM_EVENT_SUSPEND: + kvm_cpu__psci_system_suspend(vcpu); + return true; + case KVM_SYSTEM_EVENT_WAKEUP: + kvm_cpu__wakeup(vcpu); + return true; + default: + return false; + } +} diff --git a/arm/kvm.c b/arm/kvm.c index 80d233f..8ae4711 100644 --- a/arm/kvm.c +++ b/arm/kvm.c @@ -86,6 +86,15 @@ void kvm__arch_init(struct kvm *kvm, const char *hugetlbfs_path, u64 ram_size) /* Create the virtual GIC. */ if (gic__create(kvm, kvm->cfg.arch.irqchip)) die("Failed to create virtual GIC"); + + if (kvm__supports_extension(kvm, KVM_CAP_ARM_SYSTEM_SUSPEND)) { + struct kvm_enable_cap cap = { + .cap = KVM_CAP_ARM_SYSTEM_SUSPEND, + }; + + if (ioctl(kvm->vm_fd, KVM_ENABLE_CAP, &cap) < 0) + die("Enabling KVM_CAP_ARM_SYSTEM_SUSPEND failed"); + } } #define FDT_ALIGN SZ_2M