From patchwork Mon Jan 4 08:16:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shenming Lu X-Patchwork-Id: 11996301 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E758C433E0 for ; Mon, 4 Jan 2021 08:18:37 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3893520784 for ; Mon, 4 Jan 2021 08:18:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3893520784 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:Message-ID:Date:Subject:To:From: Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender :Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Owner; bh=GUwuAZZSBtErK4IJWGi+DDLk1gxvpj3v0KoNQnYDS1s=; b=VDeRkbXxI19xWuFJtHzyGecS32 NViJUBHdum/P0PkTvwNB+SXztw+7rOkH46gv8m97jLlb8PweYu5zNPIq71Ok1pL68C66iWAbYcytw OQWg7H2CJRGzRiegE7wPD7vv3AlR43eijiI8dh2gHF27lqqxhci/BhKlSrzVRJJBeoZEP3gOUgZGq +J5EFWGLa0D3yaFv7golKnR+gB8SoW34HCWnxt7jQpewz0tBJYRtxCQkvGVNXRTAGn+79Nf5K1AYs xdzpn6eaulkLvF/DWUDyGJl66AMRLlJ87FnP0mVdA5rC4120cakb79XfjU97UpWnpoy14xyg+DnBi 9FQqiYiA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kwL3D-0007nG-Qd; Mon, 04 Jan 2021 08:17:15 +0000 Received: from szxga05-in.huawei.com ([45.249.212.191]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kwL37-0007jV-Je for linux-arm-kernel@lists.infradead.org; Mon, 04 Jan 2021 08:17:11 +0000 Received: from DGGEMS403-HUB.china.huawei.com (unknown [172.30.72.58]) by szxga05-in.huawei.com (SkyGuard) with ESMTP id 4D8T341WJnzMF83; Mon, 4 Jan 2021 16:15:44 +0800 (CST) Received: from DESKTOP-7FEPK9S.china.huawei.com (10.174.184.196) by DGGEMS403-HUB.china.huawei.com (10.3.19.203) with Microsoft SMTP Server id 14.3.498.0; Mon, 4 Jan 2021 16:16:45 +0800 From: Shenming Lu To: Marc Zyngier , Eric Auger , "Will Deacon" , , , , Subject: [RFC PATCH v2 0/4] KVM: arm64: Add VLPI migration support on GICv4.1 Date: Mon, 4 Jan 2021 16:16:09 +0800 Message-ID: <20210104081613.100-1-lushenming@huawei.com> X-Mailer: git-send-email 2.27.0.windows.1 MIME-Version: 1.0 X-Originating-IP: [10.174.184.196] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210104_031709_968967_E2EFEA85 X-CRM114-Status: GOOD ( 11.74 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Lorenzo Pieralisi , Cornelia Huck , lushenming@huawei.com, Alex Williamson , yuzenghui@huawei.com, wanghaibin.wang@huawei.com Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In GICv4.1, migration has been supported except for (directly-injected) VLPI. And GICv4.1 Spec explicitly gives a way to get the VLPI's pending state (which was crucially missing in GICv4.0). So we make VLPI migration capable on GICv4.1 in this patch set. In order to support VLPI migration, we need to save and restore all required configuration information and pending states of VLPIs. But in fact, the configuration information of VLPIs has already been saved (or will be reallocated on the dst host...) in vgic(kvm) migration. So we only have to migrate the pending states of VLPIs specially. Below is the related workflow in migration. On the save path: In migration completion: pause all vCPUs | call each VM state change handler: pause other devices (just keep from sending interrupts, and such as VFIO migration protocol has already realized it [1]) | flush ITS tables into guest RAM | flush RDIST pending tables (also flush VLPI state here) | ... On the resume path: load each device's state: restore ITS tables (include pending tables) from guest RAM | for other (PCI) devices (paused), if configured to have VLPIs, establish the forwarding paths of their VLPIs (and transfer the pending states from kvm's vgic to VPT here) We have tested this series in VFIO migration, and found some related issues in QEMU [2]. Links: [1] vfio: UAPI for migration interface for device state: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/ commit/?id=a8a24f3f6e38103b77cf399c38eb54e1219d00d6 [2] vfio: Some fixes and optimizations for VFIO migration: https://patchwork.ozlabs.org/cover/1413263/ History: v1 -> v2: - Get the VLPI state from the KVM side. - Nit fixes. Since there seems to be no better place to transfer the pending states in patch 3, we just keep it unchanged. Thanks, Shenming Shenming Lu (3): KVM: arm64: GICv4.1: Add function to get VLPI state KVM: arm64: GICv4.1: Try to save hw pending state in save_pending_tables KVM: arm64: GICv4.1: Give a chance to save VLPI's pending state Zenghui Yu (1): KVM: arm64: GICv4.1: Restore VLPI's pending state to physical side .../virt/kvm/devices/arm-vgic-its.rst | 2 +- arch/arm64/kvm/vgic/vgic-its.c | 6 +- arch/arm64/kvm/vgic/vgic-v3.c | 58 +++++++++++++++++-- arch/arm64/kvm/vgic/vgic-v4.c | 36 ++++++++++++ arch/arm64/kvm/vgic/vgic.h | 1 + 5 files changed, 93 insertions(+), 10 deletions(-)