From patchwork Fri Feb 27 17:54:31 2009 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 9212 Received: from vger.kernel.org (vger.kernel.org [209.132.176.167]) by demeter.kernel.org (8.14.2/8.14.2) with ESMTP id n1RHtOs0027752 for ; Fri, 27 Feb 2009 17:55:24 GMT Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756098AbZB0RzW (ORCPT ); Fri, 27 Feb 2009 12:55:22 -0500 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1756243AbZB0RzW (ORCPT ); Fri, 27 Feb 2009 12:55:22 -0500 Received: from mx2.redhat.com ([66.187.237.31]:51362 "EHLO mx2.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756098AbZB0RzV (ORCPT ); Fri, 27 Feb 2009 12:55:21 -0500 Received: from int-mx2.corp.redhat.com (int-mx2.corp.redhat.com [172.16.27.26]) by mx2.redhat.com (8.13.8/8.13.8) with ESMTP id n1RHtIeJ017116; Fri, 27 Feb 2009 12:55:18 -0500 Received: from ns3.rdu.redhat.com (ns3.rdu.redhat.com [10.11.255.199]) by int-mx2.corp.redhat.com (8.13.1/8.13.1) with ESMTP id n1RHtIJO002253; Fri, 27 Feb 2009 12:55:19 -0500 Received: from amt.cnet (vpn-10-52.str.redhat.com [10.32.10.52]) by ns3.rdu.redhat.com (8.13.8/8.13.8) with ESMTP id n1RHtGsj002771; Fri, 27 Feb 2009 12:55:17 -0500 Received: from amt.cnet (amt.cnet [127.0.0.1]) by amt.cnet (Postfix) with ESMTP id 40C66680001; Fri, 27 Feb 2009 14:54:39 -0300 (BRT) Received: (from marcelo@localhost) by amt.cnet (8.14.3/8.14.3/Submit) id n1RHsV5b010852; Fri, 27 Feb 2009 14:54:31 -0300 Date: Fri, 27 Feb 2009 14:54:31 -0300 From: Marcelo Tosatti To: "Yang, Sheng" Cc: "kvm@vger.kernel.org" , Avi Kivity , Chris Wright Subject: Re: KVM: protect assigned dev workqueue, int handler and irq acker Message-ID: <20090227175431.GA10839@amt.cnet> References: <20090226235054.GB4188@amt.cnet> <200902271217.01918.sheng.yang@intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <200902271217.01918.sheng.yang@intel.com> User-Agent: Mutt/1.5.18 (2008-05-17) X-Scanned-By: MIMEDefang 2.58 on 172.16.27.26 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org On Fri, Feb 27, 2009 at 12:17:01PM +0800, Yang, Sheng wrote: > On Friday 27 February 2009 07:50:54 Marcelo Tosatti wrote: > > Can someone with HW test this please? > > Good catch! The patch works fine on my side. > > Can it be a per-device lock? One big lock for all assigned device seems > restrict scalability. Since all state is per-device, yes. Can you please review, test and ack the patch below? Thanks. > > ----- > > > > kvm_assigned_dev_ack_irq is vulnerable to a race condition with the > > interrupt handler function. It does: > > > > if (dev->host_irq_disabled) { > > enable_irq(dev->host_irq); > > dev->host_irq_disabled = false; > > } > > > > If an interrupt triggers before the host->dev_irq_disabled assignment, > > it will disable the interrupt and set dev->host_irq_disabled to true. > > > > On return to kvm_assigned_dev_ack_irq, dev->host_irq_disabled is set to > > false, and the next kvm_assigned_dev_ack_irq call will fail to reenable > > it. > > > > Other than that, having the interrupt handler and work handlers run in > > parallel sounds like asking for trouble (could not spot any obvious > > problem, but better not have to, its fragile). > > Well, my original purpose is a FIFO between interrupt handler and work(for > MSI-X), but seems too complex... And I also don't see any problem for now... > > -- > regards > Yang, Sheng --- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 3832243..16bf60b 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -349,6 +349,7 @@ struct kvm_assigned_dev_kernel { int flags; struct pci_dev *dev; struct kvm *kvm; + spinlock_t assigned_dev_lock; }; struct kvm_irq_mask_notifier { diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 4d2be16..b7d060f 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -41,6 +41,7 @@ #include #include #include +#include #include #include @@ -132,6 +133,7 @@ static void kvm_assigned_dev_interrupt_work_handler(struct work_struct *work) * finer-grained lock, update this */ mutex_lock(&kvm->lock); + spin_lock_irq(&assigned_dev->assigned_dev_lock); if (assigned_dev->irq_requested_type & KVM_ASSIGNED_DEV_MSIX) { struct kvm_guest_msix_entry *guest_entries = assigned_dev->guest_msix_entries; @@ -158,18 +160,21 @@ static void kvm_assigned_dev_interrupt_work_handler(struct work_struct *work) } } + spin_unlock_irq(&assigned_dev->assigned_dev_lock); mutex_unlock(&assigned_dev->kvm->lock); } static irqreturn_t kvm_assigned_dev_intr(int irq, void *dev_id) { + unsigned long flags; struct kvm_assigned_dev_kernel *assigned_dev = (struct kvm_assigned_dev_kernel *) dev_id; + spin_lock_irqsave(&assigned_dev->assigned_dev_lock, flags); if (assigned_dev->irq_requested_type == KVM_ASSIGNED_DEV_MSIX) { int index = find_index_from_host_irq(assigned_dev, irq); if (index < 0) - return IRQ_HANDLED; + goto out; assigned_dev->guest_msix_entries[index].flags |= KVM_ASSIGNED_MSIX_PENDING; } @@ -179,6 +184,8 @@ static irqreturn_t kvm_assigned_dev_intr(int irq, void *dev_id) disable_irq_nosync(irq); assigned_dev->host_irq_disabled = true; +out: + spin_unlock_irqrestore(&assigned_dev->assigned_dev_lock, flags); return IRQ_HANDLED; } @@ -186,6 +193,7 @@ static irqreturn_t kvm_assigned_dev_intr(int irq, void *dev_id) static void kvm_assigned_dev_ack_irq(struct kvm_irq_ack_notifier *kian) { struct kvm_assigned_dev_kernel *dev; + unsigned long flags; if (kian->gsi == -1) return; @@ -198,10 +206,12 @@ static void kvm_assigned_dev_ack_irq(struct kvm_irq_ack_notifier *kian) /* The guest irq may be shared so this ack may be * from another device. */ + spin_lock_irqsave(&dev->assigned_dev_lock, flags); if (dev->host_irq_disabled) { enable_irq(dev->host_irq); dev->host_irq_disabled = false; } + spin_unlock_irqrestore(&dev->assigned_dev_lock, flags); } /* The function implicit hold kvm->lock mutex due to cancel_work_sync() */ @@ -604,6 +614,7 @@ static int kvm_vm_ioctl_assign_device(struct kvm *kvm, match->dev = dev; match->irq_source_id = -1; match->kvm = kvm; + spin_lock_init(&match->assigned_dev_lock); list_add(&match->list, &kvm->arch.assigned_dev_head);