From patchwork Mon Mar 22 06:01:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shenming Lu X-Patchwork-Id: 12153779 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99D4AC433E0 for ; Mon, 22 Mar 2021 06:04:55 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3534B61931 for ; Mon, 22 Mar 2021 06:04:55 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3534B61931 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:Message-ID:Date:Subject:CC:To:From: Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender :Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Owner; bh=m00Zk7x+HOZ/R7l8UfU/UP1FJRU1rUga+Der/I9wSGU=; b=fEUuq0TRGtfj4CAz4uxcZkImOd 5ejHFMyhsSbqH3ser2AvW7dd9eLc2ifdP5YP+yC6G6pjwc7cVwXHit25OjmADeZv38l/NlJe09RSv fiKx7wBy9cO7vlr/LPzVhX1FiSi0QoGKt9V8biUtU4+l4x+elDIKZHbM+p7zTRPqq+cr20SK1yxU5 8VsQfRQum55dnw+rKyJtfaT9PfaN2H9O2cQsUkFN4mpY84RDFyWKxn2wzYQi0wr70HyqD8fCLBnEw dESTbdl53SdA6p8g4FZoaYkWU7E6kR+0h9y72fZBRf7/Ztp8MUzwP+1wsdviGLou30iSHv+yyy6XJ 1L4VGCDA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lODeH-00Ay1r-GY; Mon, 22 Mar 2021 06:02:45 +0000 Received: from szxga06-in.huawei.com ([45.249.212.32]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lODe7-00Ay0N-4s for linux-arm-kernel@lists.infradead.org; Mon, 22 Mar 2021 06:02:38 +0000 Received: from DGGEMS412-HUB.china.huawei.com (unknown [172.30.72.59]) by szxga06-in.huawei.com (SkyGuard) with ESMTP id 4F3kPv4k4tzkdDV; Mon, 22 Mar 2021 14:00:51 +0800 (CST) Received: from DESKTOP-7FEPK9S.china.huawei.com (10.174.184.135) by DGGEMS412-HUB.china.huawei.com (10.3.19.212) with Microsoft SMTP Server id 14.3.498.0; Mon, 22 Mar 2021 14:02:18 +0800 From: Shenming Lu To: Marc Zyngier , Eric Auger , "Will Deacon" , , , , CC: Alex Williamson , Cornelia Huck , Lorenzo Pieralisi , , , Subject: [PATCH v5 0/6] KVM: arm64: Add VLPI migration support on GICv4.1 Date: Mon, 22 Mar 2021 14:01:52 +0800 Message-ID: <20210322060158.1584-1-lushenming@huawei.com> X-Mailer: git-send-email 2.27.0.windows.1 MIME-Version: 1.0 X-Originating-IP: [10.174.184.135] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210322_060236_189894_AEBCD8C9 X-CRM114-Status: GOOD ( 12.26 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi, In GICv4.1, migration has been supported except for (directly-injected) VLPI. And GICv4.1 Spec explicitly gives a way to get the VLPI's pending state (which was crucially missing in GICv4.0). So we make VLPI migration capable on GICv4.1 in this series. In order to support VLPI migration, we need to save and restore all required configuration information and pending states of VLPIs. But in fact, the configuration information of VLPIs has already been saved (or will be reallocated on the dst host...) in vgic(kvm) migration. So we only have to migrate the pending states of VLPIs specially. Below is the related workflow in migration. On the save path: In migration completion: pause all vCPUs | call each VM state change handler: pause other devices (just keep from sending interrupts, and such as VFIO migration protocol has already realized it [1]) | flush ITS tables into guest RAM | flush RDIST pending tables (also flush VLPI pending states here) | ... On the resume path: load each device's state: restore ITS tables (include pending tables) from guest RAM | for other (PCI) devices (paused), if configured to have VLPIs, establish the forwarding paths of their VLPIs (and transfer the pending states from kvm's vgic to VPT here) We have tested this series in VFIO migration, and found some related issues in QEMU [2]. Links: [1] vfio: UAPI for migration interface for device state: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=a8a24f3f6e38103b77cf399c38eb54e1219d00d6 [2] vfio: Some fixes and optimizations for VFIO migration: https://patchwork.ozlabs.org/project/qemu-devel/cover/20210310030233.1133-1-lushenming@huawei.com/ History: v4 -> v5 - Lock the whole pending state read/write sequence. (in Patch 5, from Marc) v3 -> v4 - Nit fixes. - Add a CPU cache invalidation right after unmapping the vPE. (Patch 1) - Drop the setting of PTZ altogether. (Patch 2) - Bail out if spot !vgic_initialized(). (in Patch 4) - Communicate the state change (clear pending_latch) via vgic_queue_irq_unlock. (in Patch 5) Thanks a lot for the suggestions from Marc! v2 -> v3 - Add the vgic initialized check to ensure that the allocation and enabling of the doorbells have already been done before unmapping the vPEs. - Check all get_vlpi_state related conditions in save_pending_tables in one place. - Nit fixes. v1 -> v2: - Get the VLPI state from the KVM side. - Nit fixes. Thanks, Shenming Marc Zyngier (1): irqchip/gic-v3-its: Add a cache invalidation right after vPE unmapping Shenming Lu (4): irqchip/gic-v3-its: Drop the setting of PTZ altogether KVM: arm64: GICv4.1: Add function to get VLPI state KVM: arm64: GICv4.1: Try to save VLPI state in save_pending_tables KVM: arm64: GICv4.1: Give a chance to save VLPI state Zenghui Yu (1): KVM: arm64: GICv4.1: Restore VLPI pending state to physical side .../virt/kvm/devices/arm-vgic-its.rst | 2 +- arch/arm64/kvm/vgic/vgic-its.c | 6 +- arch/arm64/kvm/vgic/vgic-v3.c | 66 +++++++++++++++++-- arch/arm64/kvm/vgic/vgic-v4.c | 38 +++++++++++ arch/arm64/kvm/vgic/vgic.h | 1 + drivers/irqchip/irq-gic-v3-its.c | 21 +++++- 6 files changed, 122 insertions(+), 12 deletions(-)