From patchwork Thu Mar 2 01:49:49 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chao Gao X-Patchwork-Id: 9599721 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 1765660453 for ; Thu, 2 Mar 2017 08:54:36 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DFC9328576 for ; Thu, 2 Mar 2017 08:54:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D296528587; Thu, 2 Mar 2017 08:54:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=2.0 tests=BAYES_00, DATE_IN_PAST_06_12, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id AE91E28576 for ; Thu, 2 Mar 2017 08:54:34 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cjMT1-0003AR-QM; Thu, 02 Mar 2017 08:52:07 +0000 Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cjMT0-00039c-41 for xen-devel@lists.xen.org; Thu, 02 Mar 2017 08:52:06 +0000 Received: from [85.158.139.211] by server-12.bemta-5.messagelabs.com id F7/4F-13179-53DD7B85; Thu, 02 Mar 2017 08:52:05 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrILMWRWlGSWpSXmKPExsXS1tbhqGtyd3u EwctVxhZLPi5mcWD0OLr7N1MAYxRrZl5SfkUCa8bjmx3sBTO0Ky6ePsXYwHhWoYuRk0NIYBqj xMTPciC2hACvxJFlM1i7GDmAbD+J9/2VECXlEu37lzGB2GwCyhIXv/aygdgiAtIS1z5fZuxi5 OJgFpjBJHFg9jN2kISwgJNE49WNLCA2i4CqxKTJq8AaeAUcJb5snc0EsUtBYsrD98wTGLkXMD KsYtQoTi0qSy3SNTLSSyrKTM8oyU3MzNE1NDDVy00tLk5MT81JTCrWS87P3cQI9G49AwPjDsY 97X6HGCU5mJREeQ9f3x4hxJeUn1KZkVicEV9UmpNafIhRhoNDSYI36Q5QTrAoNT21Ii0zBxhm MGkJDh4lEd5mkDRvcUFibnFmOkTqFKMux5zZu98wCbHk5eelSonzSoEUCYAUZZTmwY2AhfwlR lkpYV5GBgYGIZ6C1KLczBJU+VeM4hyMSsK8ASBTeDLzSuA2vQI6ggnoiBcqW0GOKElESEk1MF YwrbnOe6Dn7YdQ9xVt2xe9ZY5fya3kLK+h6S7bf9Hu4X5FuWtrZ9hunKG4suPffB/B5E+mhts Ujxew6nm/llxlL6p4/1rI9jkTW/fdeWpz1elgy6M9iQ/KLMQuzdk6mecc04K0yx3TH73o+ah7 LH7aSna/rP+9kyzuzQ+YumPNppCut1UmyS1KLMUZiYZazEXFiQCuUvApdAIAAA== X-Env-Sender: chao.gao@intel.com X-Msg-Ref: server-7.tower-206.messagelabs.com!1488444722!84574181!1 X-Originating-IP: [134.134.136.65] X-SpamReason: No, hits=0.8 required=7.0 tests=DATE_IN_PAST_06_12 X-StarScan-Received: X-StarScan-Version: 9.4.3; banners=-,-,- X-VirusChecked: Checked Received: (qmail 40945 invoked from network); 2 Mar 2017 08:52:04 -0000 Received: from mga03.intel.com (HELO mga03.intel.com) (134.134.136.65) by server-7.tower-206.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 2 Mar 2017 08:52:04 -0000 Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 02 Mar 2017 00:52:02 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.35,230,1484035200"; d="scan'208";a="63515989" Received: from skl-2s3.sh.intel.com ([10.239.48.35]) by orsmga004.jf.intel.com with ESMTP; 02 Mar 2017 00:52:00 -0800 From: Chao Gao To: xen-devel@lists.xen.org Date: Thu, 2 Mar 2017 09:49:49 +0800 Message-Id: <1488419389-8016-1-git-send-email-chao.gao@intel.com> X-Mailer: git-send-email 1.8.3.1 Cc: yang.zhang.wz@gmail.com, "Xuquan \(Quan Xu\)" , quan.xu0@gmail.com, Jun Nakajima , Andrew Cooper , Kevin Tian , Jan Beulich , Chao Gao Subject: [Xen-devel] [PATCH v3] x86/apicv: Enhance posted-interrupt processing X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP __vmx_deliver_posted_interrupt() wrongly used a softirq bit to decide whether to suppress an IPI. Its logic was: the first time an IPI was sent, we set the softirq bit. Next time, we would check that softirq bit before sending another IPI. If the 1st IPI arrived at the pCPU which was in non-root mode, the hardware would consume the IPI and sync PIR to vIRR. During the process, no one (both hardware and software) will clear the softirq bit. As a result, the following IPI would be wrongly suppressed. This patch discards the suppression check, always sending IPI. The softirq also need to be raised. But there is a little change. This patch moves the place where we raise a softirq for 'cpu != smp_processor_id()' case to the IPI interrupt handler. Namely, don't raise a softirq for this case and set the interrupt handler to pi_notification_interrupt()(in which a softirq is raised) regardless of posted interrupt enabled or not. The only difference is when the IPI arrives at the pCPU which is happened in non-root mode, the patch will not raise a useless softirq since the IPI is consumed by hardware rather than raise a softirq unconditionally. Quan doesn't have enough time to upstream this fix patch. He asks me to do this. Merge another his related patch (https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg02885.html). Signed-off-by: Quan Xu Signed-off-by: Chao Gao Acked-by: Kevin Tian --- xen/arch/x86/hvm/vmx/vmx.c | 56 ++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 49 insertions(+), 7 deletions(-) diff --git a/xen/arch/x86/hvm/vmx/vmx.c b/xen/arch/x86/hvm/vmx/vmx.c index 5b1717d..9db4bd0 100644 --- a/xen/arch/x86/hvm/vmx/vmx.c +++ b/xen/arch/x86/hvm/vmx/vmx.c @@ -1842,13 +1842,59 @@ static void __vmx_deliver_posted_interrupt(struct vcpu *v) bool_t running = v->is_running; vcpu_unblock(v); + /* + * The underlying 'IF' excludes two cases which we don't need further + * operation to make sure the target vCPU (@v) will sync PIR to vIRR ASAP. + * Specifically, the two cases are: + * 1. The target vCPU is not running, meaning it is blocked or runnable. + * Since we have unblocked the target vCPU above, the target vCPU will + * sync PIR to vIRR when it is chosen to run. + * 2. The target vCPU is the current vCPU and in_irq() is false. It means + * the function is called in noninterrupt context. Since we don't call + * the function in noninterrupt context after the last time a vCPU syncs + * PIR to vIRR, excluding this case is also safe. + */ if ( running && (in_irq() || (v != current)) ) { + /* + * Note: Only two cases will reach here: + * 1. The target vCPU is running on other pCPU. + * 2. The target vCPU is running on the same pCPU with the current + * vCPU and the current vCPU is in interrupt context. That's to say, + * the target vCPU is the current vCPU. + * + * Note2: Don't worry the v->processor may change since at least when + * the target vCPU is chosen to run or be blocked, there is a chance + * to sync PIR to vIRR. + */ unsigned int cpu = v->processor; - if ( !test_and_set_bit(VCPU_KICK_SOFTIRQ, &softirq_pending(cpu)) - && (cpu != smp_processor_id()) ) + /* + * For case 1, we send a IPI to the pCPU. When the IPI arrives, the + * target vCPU maybe is running in non-root mode, running in root + * mode, runnable or blocked. If the target vCPU is running in + * non-root mode, the hardware will sync PIR to vIRR for the IPI + * vector is special to the pCPU. If the target vCPU is running in + * root-mode, the interrupt handler starts to run. In order to make + * sure the target vCPU could go back to vmx_intr_assist(), the + * interrupt handler should raise a softirq if no pending softirq. + * If the target vCPU is runnable, it will sync PIR to vIRR next time + * it is chose to run. In this case, a IPI and a softirq is sent to + * a wrong vCPU which we think it is not a big issue. If the target + * vCPU is blocked, since vcpu_block() checks whether there is an + * event to be delivered through local_events_need_delivery() just + * after blocking, the vCPU must have synced PIR to vIRR. Similarly, + * there is a IPI and a softirq sent to a wrong vCPU. + */ + if ( cpu != smp_process_id() ) send_IPI_mask(cpumask_of(cpu), posted_intr_vector); + /* + * For case 2, raising a softirq can cause vmx_intr_assist() where PIR + * has a chance to be synced to vIRR to be called. As an optimization, + * We only need to raise a softirq when no pending softirq. + */ + else if ( !softirq_pending(cpu) ) + raise_softirq(VCPU_KICK_SOFTIRQ); } } @@ -2281,13 +2327,9 @@ const struct hvm_function_table * __init start_vmx(void) if ( cpu_has_vmx_posted_intr_processing ) { + alloc_direct_apic_vector(&posted_intr_vector, pi_notification_interrupt); if ( iommu_intpost ) - { - alloc_direct_apic_vector(&posted_intr_vector, pi_notification_interrupt); alloc_direct_apic_vector(&pi_wakeup_vector, pi_wakeup_interrupt); - } - else - alloc_direct_apic_vector(&posted_intr_vector, event_check_interrupt); } else {