From patchwork Mon Jan 9 19:30:37 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefano Stabellini X-Patchwork-Id: 9505809 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 89B68606E1 for ; Mon, 9 Jan 2017 19:33:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 896F6284F8 for ; Mon, 9 Jan 2017 19:33:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7DE4E284FC; Mon, 9 Jan 2017 19:33:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 88E89284F8 for ; Mon, 9 Jan 2017 19:33:20 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cQfec-0003SK-J2; Mon, 09 Jan 2017 19:30:50 +0000 Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cQfeb-0003Rz-Ln for xen-devel@lists.xenproject.org; Mon, 09 Jan 2017 19:30:49 +0000 Received: from [85.158.139.211] by server-4.bemta-5.messagelabs.com id EE/1D-25360-8E4E3785; Mon, 09 Jan 2017 19:30:48 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrMIsWRWlGSWpSXmKPExsVybKJsh+6zJ8U RBqcPmVp83zKZyYHR4/CHKywBjFGsmXlJ+RUJrBl7f3UxFvSEVaw9z9XAuMypi5GLQ0hgKqPE +fV9zBDObCaJUxu3s3cxcnKwCGhJnDu/kxXEZhMwlPj7ZBNbFyMHhwSQveQzB0hYRMBIovPOZ RaQXmaBvUwSP7rmg/UKCyRJ3N3+kA3E5hSwkJjzYheYzSvgJdH+/RMrxLIJTBI7vrcxgiREBX QlDv37A1UkKHFy5hMWEJsZ6Ijl07eB2RICGRLzeuawQtheEotuXIKy1SSuntvEPIFRcBaS9ll I2hcwMq1i1ChOLSpLLdI1NNFLKspMzyjJTczM0TU0MNXLTS0uTkxPzUlMKtZLzs/dxAgMUAYg 2MF49rTnIUZJDiYlUV6DXcURQnxJ+SmVGYnFGfFFpTmpxYcYZTg4lCR4vz8GygkWpaanVqRl5 gBjBSYtwcGjJML7ACTNW1yQmFucmQ6ROsWoyzHt2eKnTEIsefl5qVLivMdAigRAijJK8+BGwO L2EqOslDAvI9BRQjwFqUW5mSWo8q8YxTkYlYR5q0Cm8GTmlcBtegV0BBPQEZF2YEeUJCKkpBo Yl58VUjmxnnWeDMM0nm2XNyUZ6azpCzC+JMNmuebqnXNRgv/37g5nut9aLKl+2Yq9uqZY57bc b2NxydepzxzWLnPT7Dj3yfLi19gtNz7/9tj31TTk8Jtw2fdXc2z03pasb9UJnvXe4rGmUXby5 cycgkC3bL5jLM0e7bmW/tlP/2Sa7ZNK2vNDiaU4I9FQi7moOBEAKNvZ7NYCAAA= X-Env-Sender: sstabellini@kernel.org X-Msg-Ref: server-11.tower-206.messagelabs.com!1483990245!66934876!1 X-Originating-IP: [198.145.29.136] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 9.1.1; banners=-,-,- X-VirusChecked: Checked Received: (qmail 47228 invoked from network); 9 Jan 2017 19:30:46 -0000 Received: from mail.kernel.org (HELO mail.kernel.org) (198.145.29.136) by server-11.tower-206.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 9 Jan 2017 19:30:46 -0000 Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 327CE20375; Mon, 9 Jan 2017 19:30:43 +0000 (UTC) Received: from [10.1.10.56] (96-82-76-110-static.hfc.comcastbusiness.net [96.82.76.110]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id E17942035B; Mon, 9 Jan 2017 19:30:39 +0000 (UTC) Date: Mon, 9 Jan 2017 11:30:37 -0800 (PST) From: Stefano Stabellini X-X-Sender: sstabellini@sstabellini-ThinkPad-X260 To: Konrad Rzeszutek Wilk In-Reply-To: <20170109155929.GA10991@char.us.oracle.com> Message-ID: References: <20170105192856.25559-1-dan.streetman@canonical.com> <20170107010651.GV16608@char.us.oracle.com> <42d912c2-596e-29f6-8385-dc82a891895c@oracle.com> <20170109155929.GA10991@char.us.oracle.com> User-Agent: Alpine 2.10 (DEB 1266 2009-07-14) MIME-Version: 1.0 X-Virus-Scanned: ClamAV using ClamSMTP Cc: stefano@char.us.oracle.com, linux-kernel , linux-pci@vger.kernel.org, Bjorn Helgaas , xen-devel@lists.xenproject.org, Boris Ostrovsky , Dan Streetman , Dan Streetman Subject: Re: [Xen-devel] [PATCH] xen: do not re-use pirq number cached in pci device msi msg data X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP On Mon, 9 Jan 2017, Konrad Rzeszutek Wilk wrote: > On Mon, Jan 09, 2017 at 10:42:41AM -0500, Dan Streetman wrote: > > On Mon, Jan 9, 2017 at 9:59 AM, Boris Ostrovsky > > wrote: > > > On 01/06/2017 08:06 PM, Konrad Rzeszutek Wilk wrote: > > >> On Thu, Jan 05, 2017 at 02:28:56PM -0500, Dan Streetman wrote: > > >>> Do not read a pci device's msi message data to see if a pirq was > > >>> previously configured for the device's msi/msix, as the old pirq was > > >>> unmapped and may now be in use by another pci device. The previous > > >>> pirq should never be re-used; instead a new pirq should always be > > >>> allocated from the hypervisor. > > >> Won't this cause a starvation problem? That is we will run out of PIRQs > > >> as we are not reusing them? > > > > > > Don't we free the pirq when we unmap it? > > > > I think this is actually a bit worse than I initially thought. After > > looking a bit closer, and I think there's an asymmetry with pirq > > allocation: > > Lets include Stefano, > > Thank you for digging in this! This has quite the deja-vu > feeling as I believe I hit this at some point in the past and > posted some possible ways of fixing this. But sadly I can't > find the thread. This issue seems to be caused by: commit af42b8d12f8adec6711cb824549a0edac6a4ae8f Author: Stefano Stabellini Date: Wed Dec 1 14:51:44 2010 +0000 xen: fix MSI setup and teardown for PV on HVM guests which was a fix to a bug: This fixes a bug in xen_hvm_setup_msi_irqs that manifests itself when trying to enable the same MSI for the second time: the old MSI to pirq mapping is still valid at this point but xen_hvm_setup_msi_irqs would try to assign a new pirq anyway. A simple way to reproduce this bug is to assign an MSI capable network card to a PV on HVM guest, if the user brings down the corresponding ethernet interface and up again, Linux would fail to enable MSIs on the device. I don't remember any of the details. From the description of this bug, it seems that Xen changed behavior in the past few years: before it used to keep the pirq-MSI mapping, while today it doesn't. If I wrote "the old MSI to pirq mapping is still valid at this point", the pirq couldn't have been completely freed, then reassigned to somebody else the way it is described in this email. I think we should indentify the changeset or Xen version that introduced the new behavior. If it is old enough, we might be able to just revert af42b8d12f8adec6711cb824549a0edac6a4ae8f. Otherwise we could make the behavior conditional to the Xen version. > > tl;dr: > > > > pci_enable_msix_range() -> each MSIX (or MSI) now has a pirq > > allocated, and reserved in the hypervisor > > > > request_irq() -> an event channel is opened for the specific pirq, and > > maps the pirq with the hypervisor > > > > free_irq() -> the event channel is closed, and the pirq is unmapped, > > but that unmap function also frees the pirq! The hypervisor can/will > > give it away to the next call to get_free_pirq. However, the pci > > msi/msix data area still contains the pirq number, and the next call > > to request_irq() will re-use the pirq. > > > > pci_disable_msix() -> this has no effect on the pirq in the hypervisor > > (it's already been freed), and it also doesn't clear anything from the > > msi/msix data area, so the pirq is still cached there. > > > > > > It seems like the hypervisor needs to be fixed to *not* unmap the pirq > > when the event channel is closed - or at least, not to change it to > > IRQ_UNBOUND state? And, the pci_disable_msix call should eventually > > call into something in the Xen guest kernel that actually does the > > pirq unmapping, and clear it from the msi data area (i.e. > > pci_write_msi_msg) > > The problem is that Xen changes have sailed a long long time ago. > > > > Alternately, if the hypervisor doesn't change, then the call into the > > hypervisor to actually allocate a pirq needs to move from the > > pci_enable_msix_range() call to the request_irq() call? So that when > > the msi/msix range is enabled, it doesn't actually reserve any pirq's > > for any of the vectors; each request_irq/free_irq pair do the pirq > > allocate-and-map/unmap... > > > Or a third one: We keep an pirq->device lookup and inhibit free_irq() > from actually calling evtchn_close() until the pci_disable_msix() is done? I think that's a reasonable alternative: we mask the evtchn, but do not call xen_evtchn_close in shutdown_pirq for PV on HVM guests. Something like (not compiled, untested): We want to keep the pirq allocated to the device - not closing the evtchn seems like the right thing to do. I suggest we test for the original bug too: enable/disable the network interface of an MSI capable network card. > > longer details: > > > > The chain of function calls starts in the initial call to configure > > the msi vectors, which eventually calls __pci_enable_msix_range (or > > msi_) which then eventually reaches xen_hvm_setup_msi_irqs(), which > > either tries to re-use any cached pirq in the MSI data area, or (for > > the first time setup) allocates a new pirq from the hypervisor via > > PHYSDEVOP_get_free_pirq. That pirq is then reserved from the > > hypervisor's perspective, but it's not mapped to anything in the guest > > kernel. > > > > Then, the driver calls request_irq to actually start using the irq, > > which calls __setup_irq to irq_startup to startup_pirq. The > > startup_pirq call actually creates the evtchn and binds it to the > > previously allocated pirq via EVTCHNOP_bind_pirq. > > > > At this point, the pirq is bound to a guest kernel evtchn (and irq) > > and is in use. But then, when the driver doesn't want it anymore, it > > calls free_irq, and that calls irq_shutdown to shutdown_pirq; and that > > function closes the evtchn via EVTCHNOP_close. > > > > Inside the hypervisor, in xen/common/event_channel.c in > > evtchn_close(), if the channel is type ECS_PIRQ (which our pirq > > channel is) then it unmaps the pirq mapping via > > unmap_domain_pirq_emuirq. This unmaps the pirq, but also puts it back > > to state IRQ_UNBOUND, which makes it available for the hypervisor to > > give away to anyone requesting a new pirq! > > > > > > > > > > > > > > > > -boris > > > > > >>> The xen_hvm_setup_msi_irqs() function currently checks the pci device's > > >>> msi descriptor message data for each msi/msix vector it sets up, and if > > >>> it finds the vector was previously configured with a pirq, and that pirq > > >>> is mapped to an irq, it re-uses the pirq instead of requesting a new pirq > > >>> from the hypervisor. However, that pirq was unmapped when the pci device > > >>> disabled its msi/msix, and it cannot be re-used; it may have been given > > >>> to a different pci device. > > >> Hm, but this implies that we do keep track of it. > > >> > > >> > > >> while (true) > > >> do > > >> rmmod nvme > > >> modprobe nvme > > >> done > > >> > > >> Things go boom without this patch. But with this patch does this > > >> still work? As in we don't run out of PIRQs? > > >> > > >> Thanks. > > >>> This exact situation is happening in a Xen guest where multiple NVMe > > >>> controllers (pci devices) are present. The NVMe driver configures each > > >>> pci device's msi/msix twice; first to configure a single vector (to > > >>> talk to the controller for its configuration info), and then it disables > > >>> that msi/msix and re-configures with all the msi/msix it needs. When > > >>> multiple NVMe controllers are present, this happens concurrently on all > > >>> of them, and in the time between controller A calling pci_disable_msix() > > >>> and then calling pci_enable_msix_range(), controller B enables its msix > > >>> and gets controller A's pirq allocated from the hypervisor. Then when > > >>> controller A re-configures its msix, its first vector tries to re-use > > >>> the same pirq that it had before; but that pirq was allocated to > > >>> controller B, and thus the Xen event channel for controller A's re-used > > >>> pirq fails to map its irq to that pirq; the hypervisor already has the > > >>> pirq mapped elsewhere. > > >>> > > >>> Signed-off-by: Dan Streetman > > >>> --- > > >>> arch/x86/pci/xen.c | 23 +++++++---------------- > > >>> 1 file changed, 7 insertions(+), 16 deletions(-) > > >>> > > >>> diff --git a/arch/x86/pci/xen.c b/arch/x86/pci/xen.c > > >>> index bedfab9..a00a6c0 100644 > > >>> --- a/arch/x86/pci/xen.c > > >>> +++ b/arch/x86/pci/xen.c > > >>> @@ -234,23 +234,14 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type) > > >>> return 1; > > >>> > > >>> for_each_pci_msi_entry(msidesc, dev) { > > >>> - __pci_read_msi_msg(msidesc, &msg); > > >>> - pirq = MSI_ADDR_EXT_DEST_ID(msg.address_hi) | > > >>> - ((msg.address_lo >> MSI_ADDR_DEST_ID_SHIFT) & 0xff); > > >>> - if (msg.data != XEN_PIRQ_MSI_DATA || > > >>> - xen_irq_from_pirq(pirq) < 0) { > > >>> - pirq = xen_allocate_pirq_msi(dev, msidesc); > > >>> - if (pirq < 0) { > > >>> - irq = -ENODEV; > > >>> - goto error; > > >>> - } > > >>> - xen_msi_compose_msg(dev, pirq, &msg); > > >>> - __pci_write_msi_msg(msidesc, &msg); > > >>> - dev_dbg(&dev->dev, "xen: msi bound to pirq=%d\n", pirq); > > >>> - } else { > > >>> - dev_dbg(&dev->dev, > > >>> - "xen: msi already bound to pirq=%d\n", pirq); > > >>> + pirq = xen_allocate_pirq_msi(dev, msidesc); > > >>> + if (pirq < 0) { > > >>> + irq = -ENODEV; > > >>> + goto error; > > >>> } > > >>> + xen_msi_compose_msg(dev, pirq, &msg); > > >>> + __pci_write_msi_msg(msidesc, &msg); > > >>> + dev_dbg(&dev->dev, "xen: msi bound to pirq=%d\n", pirq); > > >>> irq = xen_bind_pirq_msi_to_irq(dev, msidesc, pirq, > > >>> (type == PCI_CAP_ID_MSI) ? nvec : 1, > > >>> (type == PCI_CAP_ID_MSIX) ? > > >>> -- > > >>> 2.9.3 > > >>> > > > > > > > > > _______________________________________________ > > > Xen-devel mailing list > > > Xen-devel@lists.xen.org > > > https://lists.xen.org/xen-devel > diff --git a/drivers/xen/events/events_base.c b/drivers/xen/events/events_base.c index 137bd0e..3174923 100644 --- a/drivers/xen/events/events_base.c +++ b/drivers/xen/events/events_base.c @@ -577,7 +577,8 @@ static void shutdown_pirq(struct irq_data *data) return; mask_evtchn(evtchn); - xen_evtchn_close(evtchn); + if (!xen_hvm_domain()) + xen_evtchn_close(evtchn); xen_irq_info_cleanup(info); }