From patchwork Tue Dec 9 14:35:56 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiang Liu X-Patchwork-Id: 5463431 Return-Path: X-Original-To: patchwork-linux-arm@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 11B07BEEA8 for ; Tue, 9 Dec 2014 14:39:44 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 1857720166 for ; Tue, 9 Dec 2014 14:39:43 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.9]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B27AF20121 for ; Tue, 9 Dec 2014 14:39:41 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1XyLuP-0000u3-Vp; Tue, 09 Dec 2014 14:37:01 +0000 Received: from mga03.intel.com ([134.134.136.65]) by bombadil.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1XyLuM-0000sO-PO for linux-arm-kernel@lists.infradead.org; Tue, 09 Dec 2014 14:36:59 +0000 Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga103.jf.intel.com with ESMTP; 09 Dec 2014 06:34:05 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.04,691,1406617200"; d="scan'208";a="496048256" Received: from plu1-mobl.ccr.corp.intel.com (HELO [10.255.27.35]) ([10.255.27.35]) by orsmga003.jf.intel.com with ESMTP; 09 Dec 2014 06:32:15 -0800 Message-ID: <548708CC.5000405@linux.intel.com> Date: Tue, 09 Dec 2014 22:35:56 +0800 From: Jiang Liu Organization: Intel User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: Marc Zyngier , Yijing Wang , Bjorn Helgaas , Thomas Gleixner Subject: Re: [PATCH 2/6] PCI/MSI: add hooks to populate the msi_domain field References: <1418069543-21969-1-git-send-email-marc.zyngier@arm.com> <1418069543-21969-3-git-send-email-marc.zyngier@arm.com> <5486585B.40000@huawei.com> <5486C8BA.8030608@arm.com> <5486E3BF.70703@huawei.com> <5486E747.4010804@arm.com> <5486EF4E.5020704@linux.intel.com> <5487012B.4040505@arm.com> <54870302.5050703@linux.intel.com> <548706E9.8080701@arm.com> In-Reply-To: <548706E9.8080701@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20141209_063658_872143_16804A6E X-CRM114-Status: GOOD ( 23.02 ) X-Spam-Score: -5.0 (-----) Cc: "linux-pci@vger.kernel.org" , "suravee.suthikulpanit@amd.com" , Will Deacon , "linux-kernel@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_LOW, T_RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On 2014/12/9 22:27, Marc Zyngier wrote: > On 09/12/14 14:11, Jiang Liu wrote: >> On 2014/12/9 22:03, Marc Zyngier wrote: >>> Hi Gerry, >>> >>> On 09/12/14 12:47, Jiang Liu wrote: >>>> On 2014/12/9 20:12, Marc Zyngier wrote: >>>>> Yijing, >>>>> >>>>> On 09/12/14 11:57, Yijing Wang wrote: >>>>>>>>> +void __weak pcibios_set_phb_msi_domain(struct pci_bus *bus) >>>>>>>>> +{ >>>>>>>>> +} >>>>>>>>> + >>>>>>>>> +static void pci_set_bus_msi_domain(struct pci_bus *bus) >>>>>>>>> +{ >>>>>>>>> + struct pci_dev *bridge = bus->self; >>>>>>>>> + >>>>>>>>> + if (!bridge) >>>>>>>>> + pcibios_set_phb_msi_domain(bus); >>>>>>>>> + else >>>>>>>>> + dev_set_msi_domain(&bus->dev, dev_get_msi_domain(&bridge->dev)); >>>>>>>>> +} >>>>>>>> >>>>>>>> >>>>>>>> Hi Marc, we can not assume pci devices under same phb share the same msi irq domain, >>>>>>>> now in x86, pci devices under the same phb may associate different msi irq domain. >>>>>> >>>>>> Hi Marc, >>>>>> >>>>>>> >>>>>>> Well, this is not supposed to be a perfect solution yet, but instead a >>>>>>> basis for discussion. What I'd like to find out is: >>>>>>> >>>>>>> - What is the minimum granularity for associating a device with its MSI >>>>>>> domain in existing platforms? >>>>>> >>>>>> PCI device, after Gerry's msi irq domain patchset which now in linux-next, >>>>>> in x86, we will find msi irq domain by pci_dev. >>>>> >>>>> Are you *really* associating the MSI domain on a per pci-device basis? >>>>> That is, you have devices on the same PCI bus talking to different MSI hw? >>>> Hi Marc, >>>> This is a little wild:( >>>> On x86 platform with Intel VT-d(not the case for AMD-v), >>>> interrupt remapping is tight to DMA remapping (IOMMU) unit. >>>> For most common cases, IOMMU unit manages PCI bus and its sub-hierarchy. >>>> But it may also manage a specific PCI device. This is typically used to >>>> provide QoS for audio device by using dedicated IOMMU unit to avoid >>>> resource contention on DMA remapping tables. BIOS uses ACPI table to >>>> report PCI bus/device to IOMMU unit mapping relationship. (To be honest, >>>> I have no really experience with such a hardware platform yet, just for >>>> theoretical analysis) >>>> On the other hand, we now support hierarchy irqdomain. So to >>>> support per-PCI IOMMU unit case, we need maintain irqdomain at PCI >>>> device level. >>>> This piece of code from your [4/6] is flexible enough, which >>>> retrieves msi_domain from PCI device, then fallback to PCI bus, >>>> then fallback to platform specific method. >>>> domain = dev_get_msi_domain(&dev->dev); >>>> if (!domain && dev->bus->msi) >>>> domain = dev->bus->msi->domain; >>>> if (!domain) >>>> domain = arch_get_pci_msi_domain(dev); >>> >>> OK. But what I'd really like to see is a way to setup the >>> device<->domain binding as early as possible, without having to use more >>> conditional code in pci_msi_get_domain. >>> >>> IOW, can we do something similar to what pci_set_bus_msi_domain and >>> pci_set_msi_domain do in this patch? >> Hi Marc, >> I have checked x86 code, we could set pci_dev->msi_domain >> when creating PCI devices, just need to find some hook points >> into PCI core next step. If arch code doesn't set pci_dev->msi_domain, >> PCI MSI core may provide a default way to set pci_dev->msi_domain. >> This may make the implementation simpler, I guess:) > > Right. So following your earlier suggestion, I could make > pci_set_msi_domain a weak symbol and let arch code override this. > > My preference would have been to have arch code to create a set of > arch-independent data structures describing the topology, and use that > for everything, but maybe that's a bit ambitious for a start. > > I'll rework the series to make the symbols weak. Hi Marc, I think we may not need the weak symbol at all. With following draft patch, the PCI MSI core may simply do: if (pci_dev->dev.msi_domain == NULL) dev_set_msi_domain(&dev->dev, dev_get_msi_domain(&dev->bus->dev)); ----------------------------------------------------------------------- Note: the patch won't pass compilation, just to show the key idea:) > > Thanks, > > M. > diff --git a/arch/x86/kernel/apic/msi.c b/arch/x86/kernel/apic/msi.c index da163da5fdee..8147d25d4349 100644 --- a/arch/x86/kernel/apic/msi.c +++ b/arch/x86/kernel/apic/msi.c @@ -67,6 +67,23 @@ static struct irq_chip pci_msi_controller = { .flags = IRQCHIP_SKIP_SET_WAKE, }; +struct irq_domain *x86_get_pci_msi_domain(struct pci_dev *dev) +{ + struct irq_domain *domain; + struct irq_alloc_info info; + + init_irq_alloc_info(&info, NULL); + info.type = X86_IRQ_ALLOC_TYPE_MSI; + info.msi_dev = dev; + domain = irq_remapping_get_irq_domain(&info); + if (domain == NULL) + domain = msi_default_domain; + if (domain == NULL) + domain = ERR_PTR(-ENOSYS); + + return domain; +} + int native_setup_msi_irqs(struct pci_dev *dev, int nvec, int type) { struct irq_domain *domain; diff --git a/arch/x86/pci/common.c b/arch/x86/pci/common.c index 7b20bccf3648..a26f30a8bb8f 100644 --- a/arch/x86/pci/common.c +++ b/arch/x86/pci/common.c @@ -652,6 +652,9 @@ int pcibios_add_device(struct pci_dev *dev) pa_data = data->next; iounmap(data); } + + dev->dev.msi_domain = x86_get_pci_msi_domain(dev); + return 0; }