From patchwork Thu Nov 18 08:39:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atish Kumar Patra X-Patchwork-Id: 12626303 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3EFEEC433F5 for ; Thu, 18 Nov 2021 08:40:34 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id F062261AA9 for ; Thu, 18 Nov 2021 08:40:33 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org F062261AA9 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=oTAFVbJfBzrY4aapj9h4XG4cqnUvX9zKBpMtxspIOtA=; b=4EeKitqGZtqlmR U5cT29q3zHkcOw+kP1jXFejL+8sSPA7T9ZkusTsADPTF0osQGZh/e3KDXQHQTe13x79Of0MK9O//3 GdcNPjAKJb6/JZEsS5wy1Bwn5Sf5jo3FwZB5QRM9SOhPyMAt3h5mY4AUOuFMNVZa9DGKZbYAZ96Eq /J/M+CCD8zmE5OPJVvOlH3TZL5VLd+hPoBMFzPjFvqvL827uSNknQ9T/IygVK5f6NYjVMzYFPI7Oe X3BbkHMbKjpRtR13/dKl/vB/Zj/yINc1VgiO9/byDVZyKtCh5C/n5YoHOtRb1Fy6tXw+3Kp6HbhQ0 5KnN+lhNv2cnM/VDIezw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mncy2-007DtW-8v; Thu, 18 Nov 2021 08:40:26 +0000 Received: from mail-ot1-x335.google.com ([2607:f8b0:4864:20::335]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mncxp-007DmK-Jm for linux-riscv@lists.infradead.org; Thu, 18 Nov 2021 08:40:15 +0000 Received: by mail-ot1-x335.google.com with SMTP id x19-20020a9d7053000000b0055c8b39420bso9745502otj.1 for ; Thu, 18 Nov 2021 00:40:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=89xr5l4aQSea0c59k4jjlx2ck4kW898rOwi+JsY0e98=; b=BGslt7yqfKOdo8PN4IVyYrjn8e35WOb2IMOH931zccIX8h0NyJQ/YxtoJbs6iFpi7Z QdLk/QYvOBvh7SDzgSBrVhAUeVdsKrrcQ920LEGxXxz1bMm6TmT9hdr5BRUG68RpB6Sr I4ORsLO9k0jEo+A/SWvG1fYUrSajIf+lEYPEFvXi6Ii7QVa4gPSFx8PFiiOw0nvrhqWw 05oYUTQnl+NlGCxwkYyJbC9mOtChywZriz95wIRH81zn/hhJpNgQLadahNxLWaMvQzSh jPOGxi2ceAA9W6rd/+33yYZQ7XttHOUK8J3LwVIRbSXFHTNeJLQ2GF2Oe/n0yJeiC7fJ aSMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=89xr5l4aQSea0c59k4jjlx2ck4kW898rOwi+JsY0e98=; b=xPfVPzHDJcUr1IIsBpEel9uY7XcvWOVJtFi0iCp6vJMeZFlpFJKiVHswvqQzfihDD7 evlHC9pnKKUj2oXY8ccNJmzwx1vlKIY5qQl8rNp1b7yBBN3rG/BHvLnA1BQUK9B93rnn Sv/t69jFzJJ88jvbGbIH+LYuT6nSUGQU/BDVROFsLxtt/uhGZfRae6TU0X8nmA+DYhyY T+PCtsDARQQ4L0+PnS+vGyxaHcnnzcsyjm3PDhHES73q0jaTMwk+/lSuGMYq/i+Ii4eP FhVZJ+CbAZQ3G4xDqTkyH3ujYuNo4zOgF1nEkTzEIKcMLzxYZIEQmseYz9Y5FCMO6YZW 124Q== X-Gm-Message-State: AOAM532UmsM8LKZMFlW6l1lTRVprXq/kd4/qUD6i2kiaEyetRqmb6UGM gVRgxxlatt1KVmuCinLPOUBqiQ== X-Google-Smtp-Source: ABdhPJzhsuyHIaz9OwhKmOdqpaM2QJ42EF/gciwTjJTdDhkx6KTgmhdTunV1g+b77eh91Weg0MM42A== X-Received: by 2002:a05:6830:1da:: with SMTP id r26mr19675998ota.73.1637224812342; Thu, 18 Nov 2021 00:40:12 -0800 (PST) Received: from fedora.. (99-13-229-45.lightspeed.snjsca.sbcglobal.net. [99.13.229.45]) by smtp.gmail.com with ESMTPSA id p14sm422100oov.0.2021.11.18.00.40.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 00:40:12 -0800 (PST) From: Atish Patra To: linux-kernel@vger.kernel.org Cc: Atish Patra , Anup Patel , Atish Patra , Albert Ou , Heinrich Schuchardt , Kefeng Wang , kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, linux-riscv@lists.infradead.org, Palmer Dabbelt , Paul Walmsley Subject: [PATCH v5 5/5] RISC-V: KVM: Add SBI HSM extension in KVM Date: Thu, 18 Nov 2021 00:39:12 -0800 Message-Id: <20211118083912.981995-6-atishp@rivosinc.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211118083912.981995-1-atishp@rivosinc.com> References: <20211118083912.981995-1-atishp@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211118_004013_741452_CAEB0951 X-CRM114-Status: GOOD ( 22.21 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Atish Patra SBI HSM extension allows OS to start/stop harts any time. It also allows ordered booting of harts instead of random booting. Implement SBI HSM exntesion and designate the vcpu 0 as the boot vcpu id. All other non-zero non-booting vcpus should be brought up by the OS implementing HSM extension. If the guest OS doesn't implement HSM extension, only single vcpu will be available to OS. Reviewed-by: Anup Patel Signed-off-by: Atish Patra Signed-off-by: Atish Patra --- arch/riscv/include/asm/sbi.h | 1 + arch/riscv/kvm/Makefile | 1 + arch/riscv/kvm/vcpu.c | 23 ++++++++ arch/riscv/kvm/vcpu_sbi.c | 4 ++ arch/riscv/kvm/vcpu_sbi_hsm.c | 105 ++++++++++++++++++++++++++++++++++ 5 files changed, 134 insertions(+) create mode 100644 arch/riscv/kvm/vcpu_sbi_hsm.c diff --git a/arch/riscv/include/asm/sbi.h b/arch/riscv/include/asm/sbi.h index 4f9370b6032e..79af25c45c8d 100644 --- a/arch/riscv/include/asm/sbi.h +++ b/arch/riscv/include/asm/sbi.h @@ -90,6 +90,7 @@ enum sbi_hsm_hart_status { #define SBI_ERR_INVALID_PARAM -3 #define SBI_ERR_DENIED -4 #define SBI_ERR_INVALID_ADDRESS -5 +#define SBI_ERR_ALREADY_AVAILABLE -6 extern unsigned long sbi_spec_version; struct sbiret { diff --git a/arch/riscv/kvm/Makefile b/arch/riscv/kvm/Makefile index 4757ae158bf3..aaf181a3d74b 100644 --- a/arch/riscv/kvm/Makefile +++ b/arch/riscv/kvm/Makefile @@ -26,4 +26,5 @@ kvm-y += vcpu_sbi.o kvm-$(CONFIG_RISCV_SBI_V01) += vcpu_sbi_v01.o kvm-y += vcpu_sbi_base.o kvm-y += vcpu_sbi_replace.o +kvm-y += vcpu_sbi_hsm.o kvm-y += vcpu_timer.o diff --git a/arch/riscv/kvm/vcpu.c b/arch/riscv/kvm/vcpu.c index e3d3aed46184..50158867406d 100644 --- a/arch/riscv/kvm/vcpu.c +++ b/arch/riscv/kvm/vcpu.c @@ -53,6 +53,17 @@ static void kvm_riscv_reset_vcpu(struct kvm_vcpu *vcpu) struct kvm_vcpu_csr *reset_csr = &vcpu->arch.guest_reset_csr; struct kvm_cpu_context *cntx = &vcpu->arch.guest_context; struct kvm_cpu_context *reset_cntx = &vcpu->arch.guest_reset_context; + bool loaded; + + /** + * The preemption should be disabled here because it races with + * kvm_sched_out/kvm_sched_in(called from preempt notifiers) which + * also calls vcpu_load/put. + */ + get_cpu(); + loaded = (vcpu->cpu != -1); + if (loaded) + kvm_arch_vcpu_put(vcpu); memcpy(csr, reset_csr, sizeof(*csr)); @@ -64,6 +75,11 @@ static void kvm_riscv_reset_vcpu(struct kvm_vcpu *vcpu) WRITE_ONCE(vcpu->arch.irqs_pending, 0); WRITE_ONCE(vcpu->arch.irqs_pending_mask, 0); + + /* Reset the guest CSRs for hotplug usecase */ + if (loaded) + kvm_arch_vcpu_load(vcpu, smp_processor_id()); + put_cpu(); } int kvm_arch_vcpu_precreate(struct kvm *kvm, unsigned int id) @@ -100,6 +116,13 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu) { + /** + * vcpu with id 0 is the designated boot cpu. + * Keep all vcpus with non-zero cpu id in power-off state so that they + * can brought to online using SBI HSM extension. + */ + if (vcpu->vcpu_idx != 0) + kvm_riscv_vcpu_power_off(vcpu); } void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu) diff --git a/arch/riscv/kvm/vcpu_sbi.c b/arch/riscv/kvm/vcpu_sbi.c index cf284e080f3e..f62d25bc9733 100644 --- a/arch/riscv/kvm/vcpu_sbi.c +++ b/arch/riscv/kvm/vcpu_sbi.c @@ -25,6 +25,8 @@ static int kvm_linux_err_map_sbi(int err) return SBI_ERR_INVALID_ADDRESS; case -EOPNOTSUPP: return SBI_ERR_NOT_SUPPORTED; + case -EALREADY: + return SBI_ERR_ALREADY_AVAILABLE; default: return SBI_ERR_FAILURE; }; @@ -43,6 +45,7 @@ extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_base; extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_time; extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_ipi; extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_rfence; +extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_hsm; static const struct kvm_vcpu_sbi_extension *sbi_ext[] = { &vcpu_sbi_ext_v01, @@ -50,6 +53,7 @@ static const struct kvm_vcpu_sbi_extension *sbi_ext[] = { &vcpu_sbi_ext_time, &vcpu_sbi_ext_ipi, &vcpu_sbi_ext_rfence, + &vcpu_sbi_ext_hsm, }; void kvm_riscv_vcpu_sbi_forward(struct kvm_vcpu *vcpu, struct kvm_run *run) diff --git a/arch/riscv/kvm/vcpu_sbi_hsm.c b/arch/riscv/kvm/vcpu_sbi_hsm.c new file mode 100644 index 000000000000..2e383687fa48 --- /dev/null +++ b/arch/riscv/kvm/vcpu_sbi_hsm.c @@ -0,0 +1,105 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2021 Western Digital Corporation or its affiliates. + * + * Authors: + * Atish Patra + */ + +#include +#include +#include +#include +#include +#include + +static int kvm_sbi_hsm_vcpu_start(struct kvm_vcpu *vcpu) +{ + struct kvm_cpu_context *reset_cntx; + struct kvm_cpu_context *cp = &vcpu->arch.guest_context; + struct kvm_vcpu *target_vcpu; + unsigned long target_vcpuid = cp->a0; + + target_vcpu = kvm_get_vcpu_by_id(vcpu->kvm, target_vcpuid); + if (!target_vcpu) + return -EINVAL; + if (!target_vcpu->arch.power_off) + return -EALREADY; + + reset_cntx = &target_vcpu->arch.guest_reset_context; + /* start address */ + reset_cntx->sepc = cp->a1; + /* target vcpu id to start */ + reset_cntx->a0 = target_vcpuid; + /* private data passed from kernel */ + reset_cntx->a1 = cp->a2; + kvm_make_request(KVM_REQ_VCPU_RESET, target_vcpu); + + kvm_riscv_vcpu_power_on(target_vcpu); + + return 0; +} + +static int kvm_sbi_hsm_vcpu_stop(struct kvm_vcpu *vcpu) +{ + if (vcpu->arch.power_off) + return -EINVAL; + + kvm_riscv_vcpu_power_off(vcpu); + + return 0; +} + +static int kvm_sbi_hsm_vcpu_get_status(struct kvm_vcpu *vcpu) +{ + struct kvm_cpu_context *cp = &vcpu->arch.guest_context; + unsigned long target_vcpuid = cp->a0; + struct kvm_vcpu *target_vcpu; + + target_vcpu = kvm_get_vcpu_by_id(vcpu->kvm, target_vcpuid); + if (!target_vcpu) + return -EINVAL; + if (!target_vcpu->arch.power_off) + return SBI_HSM_HART_STATUS_STARTED; + else + return SBI_HSM_HART_STATUS_STOPPED; +} + +static int kvm_sbi_ext_hsm_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, + unsigned long *out_val, + struct kvm_cpu_trap *utrap, + bool *exit) +{ + int ret = 0; + struct kvm_cpu_context *cp = &vcpu->arch.guest_context; + struct kvm *kvm = vcpu->kvm; + unsigned long funcid = cp->a6; + + switch (funcid) { + case SBI_EXT_HSM_HART_START: + mutex_lock(&kvm->lock); + ret = kvm_sbi_hsm_vcpu_start(vcpu); + mutex_unlock(&kvm->lock); + break; + case SBI_EXT_HSM_HART_STOP: + ret = kvm_sbi_hsm_vcpu_stop(vcpu); + break; + case SBI_EXT_HSM_HART_STATUS: + ret = kvm_sbi_hsm_vcpu_get_status(vcpu); + if (ret >= 0) { + *out_val = ret; + ret = 0; + } + break; + default: + ret = -EOPNOTSUPP; + } + + return ret; +} + +const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_hsm = { + .extid_start = SBI_EXT_HSM, + .extid_end = SBI_EXT_HSM, + .handler = kvm_sbi_ext_hsm_handler, +};