From patchwork Sat Nov 9 07:05:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wanpeng Li X-Patchwork-Id: 11235811 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 39DC61747 for ; Sat, 9 Nov 2019 07:06:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 0647121848 for ; Sat, 9 Nov 2019 07:06:02 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="eNKAefhx" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726194AbfKIHFm (ORCPT ); Sat, 9 Nov 2019 02:05:42 -0500 Received: from mail-pg1-f193.google.com ([209.85.215.193]:34428 "EHLO mail-pg1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725861AbfKIHFm (ORCPT ); Sat, 9 Nov 2019 02:05:42 -0500 Received: by mail-pg1-f193.google.com with SMTP id c9so966711pgc.1; Fri, 08 Nov 2019 23:05:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=ZCWySZwIFzo2R68xigCV0RDhtUpT4rtgzjsd95CcpwQ=; b=eNKAefhxCaa3zY6JByKGRa8WBVBBnoI0ZkPv9LLZz9qO3eYRoWbg5y6KdfqF9g+GLn 9RGGyjCbTguezobMDVYXOKVD0JI8ABdKhSH1J1G0dTt6ULRGAWs+cRBLimdYjjhcAQun P+0tpsXw5sbWtnQgEb41Owm+8ehVY2p+KDcK7jpepDmNlfxoW2k+XPJfWdDtbtGMQIon fB9iiJMcHa/V97mEbISMdqcCIfokDoXrGd1FFlXKlriKrZeBgiBEigv4pGzPrk1Nt/4c g/IRf6I50Pf0yt/7jQ8anvaaQTaj3ZkCkyPnh6aY7r/Ugqq0Lb0X0B+6g9Tanrjbaq/5 c1Dg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=ZCWySZwIFzo2R68xigCV0RDhtUpT4rtgzjsd95CcpwQ=; b=CC0KAcYTGhhEfZi3CSYcOonrC+F9ubmRk5sfrAjdBIDLsHsmF/zswp1v6AeJZug8V2 YUk1QK2YBbETor2PzVqoJRKmtoNu0XuGdnXIsRQFSoLf/RixuH0eel+M0AkdM+FvKLfO F5oo9mqAEK6aTWRoHI8yXxsFf6jSLJ8LZT+EF01nbFF0HeLy8wPcRq80DH6QGPM8wJoY CB09Q7kaWknNDrhhuiy44Ow2iXobC1UErO7OItGemPzdrTyFCyI1bssMB3aWB8feaF/+ tpUsOx/w3tB0pjYnCyjMKMSR9kHredpdwAlmMLdtks+JdePAhg1/QuJ10dYeb/8xFSEe /Yzg== X-Gm-Message-State: APjAAAWEP0Esf6uorwIj6ukoqNQqK8WdgqM1ymzixIprSkLZlkFonRAv y9HpHFVjjL3M2kpWyiMsVc/9sJ/N X-Google-Smtp-Source: APXvYqzrHyFC3iNi9gdl0/Mi61B9vgmW5ATsn68KqWw+xYueR5oDvgvcOpJax3cgw4PCUgZt27Dxzg== X-Received: by 2002:a63:1b41:: with SMTP id b1mr16833656pgm.335.1573283140988; Fri, 08 Nov 2019 23:05:40 -0800 (PST) Received: from localhost.localdomain ([203.205.141.123]) by smtp.googlemail.com with ESMTPSA id b200sm8337991pfb.86.2019.11.08.23.05.38 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 08 Nov 2019 23:05:40 -0800 (PST) From: Wanpeng Li X-Google-Original-From: Wanpeng Li To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: Paolo Bonzini , =?utf-8?b?UmFkaW0gS3LEjW3DocWZ?= , Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel Subject: [PATCH 1/2] KVM: X86: Single target IPI fastpath Date: Sat, 9 Nov 2019 15:05:34 +0800 Message-Id: <1573283135-5502-1-git-send-email-wanpengli@tencent.com> X-Mailer: git-send-email 2.7.4 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Wanpeng Li This patch tries to optimize x2apic physical destination mode, fixed delivery mode single target IPI by delivering IPI to receiver immediately after sender writes ICR vmexit to avoid various checks when possible. Testing on Xeon Skylake server: The virtual IPI latency from sender send to receiver receive reduces more than 330+ cpu cycles. Running hackbench(reschedule ipi) in the guest, the avg handle time of MSR_WRITE caused vmexit reduces more than 1000+ cpu cycles: Before patch: VM-EXIT Samples Samples% Time% Min Time Max Time Avg time MSR_WRITE 5417390 90.01% 16.31% 0.69us 159.60us 1.08us After patch: VM-EXIT Samples Samples% Time% Min Time Max Time Avg time MSR_WRITE 6726109 90.73% 62.18% 0.48us 191.27us 0.58us Signed-off-by: Wanpeng Li Signed-off-by: Wanpeng Li --- arch/x86/kvm/vmx/vmx.c | 39 +++++++++++++++++++++++++++++++++++++-- include/linux/kvm_host.h | 1 + 2 files changed, 38 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 5d21a4a..5c67061 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -5924,7 +5924,9 @@ static int vmx_handle_exit(struct kvm_vcpu *vcpu) } } - if (exit_reason < kvm_vmx_max_exit_handlers + if (vcpu->fast_vmexit) + return 1; + else if (exit_reason < kvm_vmx_max_exit_handlers && kvm_vmx_exit_handlers[exit_reason]) return kvm_vmx_exit_handlers[exit_reason](vcpu); else { @@ -6474,6 +6476,34 @@ void vmx_update_host_rsp(struct vcpu_vmx *vmx, unsigned long host_rsp) bool __vmx_vcpu_run(struct vcpu_vmx *vmx, unsigned long *regs, bool launched); +static int handle_ipi_fastpath(struct kvm_vcpu *vcpu) +{ + u32 index; + u64 data; + int ret = 0; + + if (lapic_in_kernel(vcpu) && apic_x2apic_mode(vcpu->arch.apic)) { + /* + * fastpath to IPI target + */ + index = kvm_rcx_read(vcpu); + data = kvm_read_edx_eax(vcpu); + + if (((index - APIC_BASE_MSR) << 4 == APIC_ICR) && + ((data & KVM_APIC_DEST_MASK) == APIC_DEST_PHYSICAL) && + ((data & APIC_MODE_MASK) == APIC_DM_FIXED)) { + + kvm_lapic_set_reg(vcpu->arch.apic, APIC_ICR2, (u32)(data >> 32)); + ret = kvm_lapic_reg_write(vcpu->arch.apic, APIC_ICR, (u32)data); + + if (ret == 0) + ret = kvm_skip_emulated_instruction(vcpu); + }; + }; + + return ret; +} + static void vmx_vcpu_run(struct kvm_vcpu *vcpu) { struct vcpu_vmx *vmx = to_vmx(vcpu); @@ -6615,6 +6645,12 @@ static void vmx_vcpu_run(struct kvm_vcpu *vcpu) | (1 << VCPU_EXREG_CR3)); vcpu->arch.regs_dirty = 0; + vmx->exit_reason = vmx->fail ? 0xdead : vmcs_read32(VM_EXIT_REASON); + vcpu->fast_vmexit = false; + if (!is_guest_mode(vcpu) && + vmx->exit_reason == EXIT_REASON_MSR_WRITE) + vcpu->fast_vmexit = handle_ipi_fastpath(vcpu); + pt_guest_exit(vmx); /* @@ -6634,7 +6670,6 @@ static void vmx_vcpu_run(struct kvm_vcpu *vcpu) vmx->nested.nested_run_pending = 0; vmx->idt_vectoring_info = 0; - vmx->exit_reason = vmx->fail ? 0xdead : vmcs_read32(VM_EXIT_REASON); if ((u16)vmx->exit_reason == EXIT_REASON_MCE_DURING_VMENTRY) kvm_machine_check(); diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 719fc3e..7a7358b 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -319,6 +319,7 @@ struct kvm_vcpu { #endif bool preempted; bool ready; + bool fast_vmexit; struct kvm_vcpu_arch arch; struct dentry *debugfs_dentry; }; From patchwork Sat Nov 9 07:05:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wanpeng Li X-Patchwork-Id: 11235813 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 633D71864 for ; Sat, 9 Nov 2019 07:06:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 3CEDE21848 for ; Sat, 9 Nov 2019 07:06:02 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="nnha3lfJ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726383AbfKIHFp (ORCPT ); Sat, 9 Nov 2019 02:05:45 -0500 Received: from mail-pl1-f196.google.com ([209.85.214.196]:44821 "EHLO mail-pl1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725861AbfKIHFp (ORCPT ); Sat, 9 Nov 2019 02:05:45 -0500 Received: by mail-pl1-f196.google.com with SMTP id az9so4462778plb.11; Fri, 08 Nov 2019 23:05:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ZKFRaOW+bpKPizxwZqQQG7355BbFkJUXxhEyGAxMrv0=; b=nnha3lfJRQmONkwwgg50/QP/4dlQvjSaO5o5IPqzM9GS54CqrHf4BLZ4dLPQfM3Zen OF5a7jJpXT27bbSJDVzSnE++OQXlx9aNhCMbPPAnX6Utp4RK9hLGVSjxq0qV5SVpjh56 PTH3AQpujuhK0IyRwGCv6E5fV78vHIjZC0C7NCkz58aUQVJcuNwd7FThsXqdSwpVFU8f zWI4iEL3S3pmXvWc/ew0fu1YaHJ4QBiDzonEtPzKMfV1dO04bflyRjn9JqfSVKPYwF7+ 9Rz06tTW9w2ffLF7y9TPFi02IoLNqvIlLW5Lz7IB20ScIVyfjMFammz+TseXzEIthUOW ACDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ZKFRaOW+bpKPizxwZqQQG7355BbFkJUXxhEyGAxMrv0=; b=tsYwt6jbJeGOgF1rqIZ8uQwtB3opZ2x9aoUsbLJbVvUJZYAs/znHYzllt7xawp779u XD53a342zkId8ohXhsQ70sigURcyJhX3zfHj8zeDf3cnsBghHs0o+3HxzfmRvEhNko2y dIvUCEpP+/34PlCHGL1vhtRqLIyPv9DDV/WTO/IM6Y1fKjoUWBLFqnY5enpsFZAduMaE /dJ62QnxIZXnmnBeOrxEVZUgNiqtDs0PQ2kL6cxW65ZvAF69QU3fP48NkVP514rEnzvJ OQCog4aKUjdORP8F3bjCf+cL5Kc7U/0jzwaXLf99JmrZVpQ09oVgEAJk+HuDwzmix+8p iyzg== X-Gm-Message-State: APjAAAXqFY00slSwtXh7AYfNav4/u3REUVwzqDf5qdInF1COkBRZCc8W CvKYfAUuObhH8mmDHsLGyoAPJiDp X-Google-Smtp-Source: APXvYqxvLpU0EGwB6Ipl7KdzI3cos3MwFxaj9xVmqoY7KF8XHkgILa0n1a3yb7lFMz5SW+7F5l1MLw== X-Received: by 2002:a17:902:b585:: with SMTP id a5mr14956646pls.191.1573283143879; Fri, 08 Nov 2019 23:05:43 -0800 (PST) Received: from localhost.localdomain ([203.205.141.123]) by smtp.googlemail.com with ESMTPSA id b200sm8337991pfb.86.2019.11.08.23.05.41 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 08 Nov 2019 23:05:43 -0800 (PST) From: Wanpeng Li X-Google-Original-From: Wanpeng Li To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: Paolo Bonzini , =?utf-8?b?UmFkaW0gS3LEjW3DocWZ?= , Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel Subject: [PATCH 2/2] KVM: LAPIC: micro-optimize fixed mode ipi delivery Date: Sat, 9 Nov 2019 15:05:35 +0800 Message-Id: <1573283135-5502-2-git-send-email-wanpengli@tencent.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1573283135-5502-1-git-send-email-wanpengli@tencent.com> References: <1573283135-5502-1-git-send-email-wanpengli@tencent.com> Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Wanpeng Li After disabling mwait/halt/pause vmexits, RESCHEDULE_VECTOR and CALL_FUNCTION_SINGLE_VECTOR etc IPI is one of the main remaining cause of vmexits observed in product environment which can't be optimized by PV IPIs. This patch is the follow-up on commit 0e6d242eccdb (KVM: LAPIC: Micro optimize IPI latency), to optimize redundancy logic before fixed mode ipi is delivered in the fast path. - broadcast handling needs to go slow path, so the delivery mode repair can be delayed to before slow path. - self-IPI will not be intervened by hypervisor any more after APICv is introduced and the boxes support APICv are popular now. In addition, kvm_apic_map_get_dest_lapic() can handle the self-IPI, so there is no need a shortcut for the non-APICv case. Signed-off-by: Wanpeng Li --- arch/x86/kvm/irq_comm.c | 6 +++--- arch/x86/kvm/lapic.c | 5 ----- 2 files changed, 3 insertions(+), 8 deletions(-) diff --git a/arch/x86/kvm/irq_comm.c b/arch/x86/kvm/irq_comm.c index 8ecd48d..aa88156 100644 --- a/arch/x86/kvm/irq_comm.c +++ b/arch/x86/kvm/irq_comm.c @@ -52,15 +52,15 @@ int kvm_irq_delivery_to_apic(struct kvm *kvm, struct kvm_lapic *src, unsigned long dest_vcpu_bitmap[BITS_TO_LONGS(KVM_MAX_VCPUS)]; unsigned int dest_vcpus = 0; + if (kvm_irq_delivery_to_apic_fast(kvm, src, irq, &r, dest_map)) + return r; + if (irq->dest_mode == 0 && irq->dest_id == 0xff && kvm_lowest_prio_delivery(irq)) { printk(KERN_INFO "kvm: apic: phys broadcast and lowest prio\n"); irq->delivery_mode = APIC_DM_FIXED; } - if (kvm_irq_delivery_to_apic_fast(kvm, src, irq, &r, dest_map)) - return r; - memset(dest_vcpu_bitmap, 0, sizeof(dest_vcpu_bitmap)); kvm_for_each_vcpu(i, vcpu, kvm) { diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index b29d00b..ea936fa 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -951,11 +951,6 @@ bool kvm_irq_delivery_to_apic_fast(struct kvm *kvm, struct kvm_lapic *src, *r = -1; - if (irq->shorthand == APIC_DEST_SELF) { - *r = kvm_apic_set_irq(src->vcpu, irq, dest_map); - return true; - } - rcu_read_lock(); map = rcu_dereference(kvm->arch.apic_map);