From patchwork Fri Nov 1 22:41:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Suthikulpanit, Suravee" X-Patchwork-Id: 11223901 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8E1A61599 for ; Fri, 1 Nov 2019 22:42:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 62E58217D9 for ; Fri, 1 Nov 2019 22:42:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amdcloud.onmicrosoft.com header.i=@amdcloud.onmicrosoft.com header.b="oQ5hDg90" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727728AbfKAWl1 (ORCPT ); Fri, 1 Nov 2019 18:41:27 -0400 Received: from mail-eopbgr720065.outbound.protection.outlook.com ([40.107.72.65]:49770 "EHLO NAM05-CO1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727029AbfKAWl1 (ORCPT ); Fri, 1 Nov 2019 18:41:27 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jOBb4F5gTdTHAeHvgDDJB4YJuF9u2NOsrPsC8X4tNqf3+QGrBAXkUTi9MOwl/1n2TuJmT+ugz8LN8Yoes95XW9bWZBk2tCfSrovcodSKnaiWy4CB2Xu1aGB9FPl0tizzJem5dOMfML+1XBUrovXntJM1G+TyzBgm0PTVmmx6VkDD8JISmguQ+N1Lo7k9iniP1XA0l3df0F0GUr+n/os4I9xpvcsfAlFQU1EP1maDOTBClnam9RHjZPb4W2frFgf7HLyE6wO6F3Y2l1E7CWVy+9U9i+Cbx502UZWHfbftN/pJviodwq+vqMWOB91QfujuDxNYBpFrD5Vho3Pd2yBZcw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pIpksEIacLQcGqwrLOjQ3oi61X6LlqCS4MXDFkZ6/VY=; b=Rb2DKFicP4vhZP/3S8dHzwmZtcbWpPF0UExAXsYRR8AGdQAKm24pn45Ro96ZLRjrG+pLVwv7ru/M3oR5TdYu/OeYEM3tEmoGlru8xU0hE5EbmhpGp82DxvIayuVtybSY/IxMK/q5VOIe+3iol97QSSSurrEBmBKOe//wPrnFUtPTIVX/6UkrfJevPYUd7LZmeNhBn1rof/Q4CARSXj5tM/ZMIxrDLuH1y2s3QRajJQ7s3M/+bFPfIpefJzi95TZdOt67y7eDISp+M2RGHkmS3BesB2dLU+X0rqWyzqvyqR78oWU1wPOGtupEZFWmuBRE3WNgjpdUHfXbpCV5tZ+/9g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector2-amdcloud-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pIpksEIacLQcGqwrLOjQ3oi61X6LlqCS4MXDFkZ6/VY=; b=oQ5hDg90jh0weV1wil8fusnbTs5IQD89lzlqqrVYXTCrOvBVdMXgpZz0gEJMVSRY9w5dKmG7qHIF1SqRLheDc2qATNX7eJC9QVru9oh1KwkHkyPT3dFx94qgkdWt8lfN/ui+7OhKaR0z5qSJ0XA62c2AQjC8z7QruXrVO/8J+Og= Received: from DM6PR12MB3865.namprd12.prod.outlook.com (10.255.173.210) by DM6PR12MB3243.namprd12.prod.outlook.com (20.179.105.155) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2408.24; Fri, 1 Nov 2019 22:41:22 +0000 Received: from DM6PR12MB3865.namprd12.prod.outlook.com ([fe80::4898:93e0:3c0c:d862]) by DM6PR12MB3865.namprd12.prod.outlook.com ([fe80::4898:93e0:3c0c:d862%6]) with mapi id 15.20.2408.024; Fri, 1 Nov 2019 22:41:22 +0000 From: "Suthikulpanit, Suravee" To: "linux-kernel@vger.kernel.org" , "kvm@vger.kernel.org" CC: "pbonzini@redhat.com" , "rkrcmar@redhat.com" , "joro@8bytes.org" , "vkuznets@redhat.com" , "rkagan@virtuozzo.com" , "graf@amazon.com" , "jschoenh@amazon.de" , "karahmed@amazon.de" , "rimasluk@amazon.com" , "Grimm, Jon" , "Suthikulpanit, Suravee" Subject: [PATCH v4 00/17] kvm: x86: Support AMD SVM AVIC w/ in-kernel irqchip mode Thread-Topic: [PATCH v4 00/17] kvm: x86: Support AMD SVM AVIC w/ in-kernel irqchip mode Thread-Index: AQHVkQV87nIRXYvD70eAN6/11mAloQ== Date: Fri, 1 Nov 2019 22:41:22 +0000 Message-ID: <1572648072-84536-1-git-send-email-suravee.suthikulpanit@amd.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [165.204.78.1] x-clientproxiedby: SN1PR12CA0047.namprd12.prod.outlook.com (2603:10b6:802:20::18) To DM6PR12MB3865.namprd12.prod.outlook.com (2603:10b6:5:1c8::18) authentication-results: spf=none (sender IP is ) smtp.mailfrom=Suravee.Suthikulpanit@amd.com; x-ms-exchange-messagesentrepresentingtype: 1 x-mailer: git-send-email 1.8.3.1 x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: c877bc34-4629-4515-59ad-08d75f1c9e9b x-ms-traffictypediagnostic: DM6PR12MB3243: x-ms-exchange-purlcount: 2 x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:1247; x-forefront-prvs: 020877E0CB x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(4636009)(396003)(346002)(366004)(376002)(39860400002)(136003)(199004)(189003)(66476007)(66446008)(26005)(6506007)(386003)(86362001)(5660300002)(102836004)(4720700003)(305945005)(7736002)(7416002)(256004)(14444005)(8936002)(8676002)(186003)(52116002)(6486002)(478600001)(50226002)(71200400001)(71190400001)(81166006)(6436002)(476003)(6306002)(2906002)(3846002)(6116002)(6512007)(99286004)(4326008)(2616005)(316002)(486006)(81156014)(14454004)(66946007)(36756003)(110136005)(54906003)(66066001)(64756008)(66556008)(25786009)(2501003);DIR:OUT;SFP:1101;SCL:1;SRVR:DM6PR12MB3243;H:DM6PR12MB3865.namprd12.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;A:1;MX:1; received-spf: None (protection.outlook.com: amd.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: bWiGoPPz+pIE1csKJ52L6oFgO8RAj+s5a6E5MrY7XPdXGmUEyPXacHXmIpuTJOTV9m4JD93YfYAr2Kl+nkbI48KuFvnlC0B2mccvRKO9OIXf5BzeIWfuyh54BB490xDf4EOz/ID5P5ojUmMlcVfOJlUM4Uk1l//PTDotVZa47UqzCYvWJcd4fsxieoU3zKY5BvNWZnIn8AeVCPOwyQ8hpbz92lU18hhBWUvUFx+KVCMKj5DHxOc8XaoSEq5eFFmZ6+Zw+3ysF2KJ1MeYb/7P/Tm4oChPx2PO5nDbVHAZMDIWpWxfBG8JfoFEM0ERc9phyKfFaykN3b90LeVvnyIsUxai8E/6DnO9/NOG4LtRu/uQtXv6AuNend0ou0o3EtrZBZphZUR+otlaAlNU+urfLRkvDSLc99XuTzCW+Fdnad+JBJ/r1/8vXinFENcQAHMPHsp6sPNA8nxaatqI5Yb4iMKmy69YMTlWstgvIerUSZU= Content-ID: MIME-Version: 1.0 X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: c877bc34-4629-4515-59ad-08d75f1c9e9b X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Nov 2019 22:41:22.3257 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: D1BfOTxe7SrNjfaA0ZLQ+H1rZYU8FGZK7CipjLLlsrB56lnrIF5qqUHp2GcQcJDw7sy+FyAj7mi31QiDnzkeow== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB3243 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The 'commit 67034bb9dd5e ("KVM: SVM: Add irqchip_split() checks before enabling AVIC")' was introduced to fix miscellaneous boot-hang issues when enable AVIC. This is mainly due to AVIC hardware doest not #vmexit on write to LAPIC EOI register resulting in-kernel PIC and IOAPIC to wait and do not inject new interrupts (e.g. PIT, RTC). This limits AVIC to only work with kernel_irqchip=split mode, which is not currently enabled by default, and also required user-space to support split irqchip model, which might not be the case. The goal of this series is to enable AVIC to work in both irqchip modes, by allowing AVIC to be deactivated temporarily during runtime, and fallback to legacy interrupt injection mode (w/ vINTR and interrupt windows) when needed, and then re-enabled subsequently. Similar approach is also used to handle Hyper-V SynIC in the 'commit 5c919412fe61 ("kvm/x86: Hyper-V synthetic interrupt controller")', where APICv is permanently disabled at runtime (currently broken for AVIC, and fixed by this series). This series contains serveral parts: * Part 1: patch 1,2 Code clean up, refactor, and introduce helper funtions * Part 2: patch 3 Introduce APICv deactivate bits to keep track of APICv state for each vm. * Part 3: patch 4-9 Add support for activate/deactivate APICv at runtime * Part 4: patch 10-13: Add support various cases where APICv needs to be deactivated * Part 5: patch 14-16: Introduce in-kernel IOAPIC workaround for AVIC EOI * Part 6: path 17 Allow enable AVIC w/ kernel_irqchip=on Pre-requisite Patch: * commit b9c6ff94e43a ("iommu/amd: Re-factor guest virtual APIC (de-)activation code") (https://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu.git/commit/ ?h=next&id=b9c6ff94e43a0ee053e0c1d983fba1ac4953b762) This series has been tested against v5.3 as following: * Booting Linux and Windows Server 2019 VMs upto 240 vcpus and FreeBSD upto 128 vcpus w/ qemu option "kernel-irqchip=on" and "-no-hpet". * Pass-through Intel 10GbE NIC and run netperf in the VM. Changes from V3: (https://lkml.org/lkml/2019/9/13/871) (Per Paolo comments) * Replace struct kvm_vcpu with struct kvm in various interfaces * Replace KVM_REQ_APICV_ACTIVATE/DEACTIVATE with KVM_REQ_APICV_UPDATE request * Replace APICv state enum (introduced in V3) w/ deactivate bits to track APICv state * Remove kvm_apicv_eoi_accelerate() (introduced in V3) * Deactivate APICv when using PIT re-inject mode * Consolidate srcu_read_unlock/lock into svm_request_update_avic() Suravee Suthikulpanit (17): kvm: x86: Modify kvm_x86_ops.get_enable_apicv() to use struct kvm parameter kvm: lapic: Introduce APICv update helper function kvm: x86: Introduce APICv deactivate bits kvm: x86: Add support for activate/de-activate APICv at runtime kvm: x86: Add APICv activate/deactivate request trace points kvm: x86: svm: Add support to activate/deactivate posted interrupts svm: Add support for setup/destroy virutal APIC backing page for AVIC kvm: x86: Introduce APICv pre-update hook svm: Add support for activate/deactivate AVIC at runtime kvm: x86: hyperv: Use APICv update request interface svm: Deactivate AVIC when launching guest with nested SVM support svm: Temporary deactivate AVIC during ExtINT handling kvm: i8254: Deactivate APICv when using in-kernel PIT re-injection mode. kvm: lapic: Clean up APIC predefined macros kvm: ioapic: Refactor kvm_ioapic_update_eoi() kvm: ioapic: Lazy update IOAPIC EOI svm: Allow AVIC with in-kernel irqchip mode arch/x86/include/asm/kvm_host.h | 18 ++++- arch/x86/kvm/hyperv.c | 5 +- arch/x86/kvm/i8254.c | 10 +++ arch/x86/kvm/ioapic.c | 149 +++++++++++++++++++++++++--------------- arch/x86/kvm/lapic.c | 35 ++++++---- arch/x86/kvm/lapic.h | 2 + arch/x86/kvm/svm.c | 136 +++++++++++++++++++++++++++++++----- arch/x86/kvm/trace.h | 19 +++++ arch/x86/kvm/vmx/vmx.c | 6 +- arch/x86/kvm/x86.c | 61 +++++++++++++--- 10 files changed, 343 insertions(+), 98 deletions(-)