[RFC,v2,03/18] irq/dev-msi: Create IR-DEV-MSI irq domain

Commit Message

Dave Jiang July 21, 2020, 4:02 p.m. UTC

From: Megha Dey <[e-mail]>

When DEV_MSI is enabled, the dev_msi_default_domain is updated to the
base DEV-MSI irq  domain. If interrupt remapping is enabled, we create
a new IR-DEV-MSI irq domain and update the dev_msi_default domain to
the same.

For X86, introduce a new irq_alloc_type which will be used by the
interrupt remapping driver.

Reviewed-by: Dan Williams <[e-mail]>
Signed-off-by: Megha Dey <[e-mail]>
Signed-off-by: Dave Jiang <[e-mail]>
---
 arch/x86/include/asm/hw_irq.h       |    1 +
 arch/x86/kernel/apic/msi.c          |   12 ++++++
 drivers/base/dev-msi.c              |   66 +++++++++++++++++++++++++++++++----
 drivers/iommu/intel/irq_remapping.c |   11 +++++-
 include/linux/intel-iommu.h         |    1 +
 include/linux/irqdomain.h           |   11 ++++++
 include/linux/msi.h                 |    3 ++
 7 files changed, 96 insertions(+), 9 deletions(-)

Comments

Jason Gunthorpe July 21, 2020, 4:21 p.m. UTC | #1

On Tue, Jul 21, 2020 at 09:02:35AM -0700, Dave Jiang wrote:
> From: Megha Dey <[e-mail]>
> 
> When DEV_MSI is enabled, the dev_msi_default_domain is updated to the
> base DEV-MSI irq  domain. If interrupt remapping is enabled, we create
> a new IR-DEV-MSI irq domain and update the dev_msi_default domain to
> the same.
> 
> For X86, introduce a new irq_alloc_type which will be used by the
> interrupt remapping driver.

Why? Shouldn't this by symmetrical with normal MSI? Does MSI do this?

I would have thought you'd want to switch to this remapping mode as
part of vfio or something like current cases.

> +struct irq_domain *create_remap_dev_msi_irq_domain(struct irq_domain *parent,
> +						   const char *name)
> +{
> +	struct fwnode_handle *fn;
> +	struct irq_domain *domain;
> +
> +	fn = irq_domain_alloc_named_fwnode(name);
> +	if (!fn)
> +		return NULL;
> +
> +	domain = msi_create_irq_domain(fn, &dev_msi_ir_domain_info, parent);
> +	if (!domain) {
> +		pr_warn("failed to initialize irqdomain for IR-DEV-MSI.\n");
> +		return ERR_PTR(-ENXIO);
> +	}
> +
> +	irq_domain_update_bus_token(domain, DOMAIN_BUS_PLATFORM_MSI);
> +
> +	if (!dev_msi_default_domain)
> +		dev_msi_default_domain = domain;
> +
> +	return domain;
> +}

What about this code creates a "remap" ? ie why is the function called
"create_remap" ?

> diff --git a/include/linux/msi.h b/include/linux/msi.h
> index 1da97f905720..7098ba566bcd 100644
> +++ b/include/linux/msi.h
> @@ -378,6 +378,9 @@ void *platform_msi_get_host_data(struct irq_domain *domain);
>  void platform_msi_write_msg(struct irq_data *data, struct msi_msg *msg);
>  void platform_msi_unmask_irq(struct irq_data *data);
>  void platform_msi_mask_irq(struct irq_data *data);
> +
> +int dev_msi_prepare(struct irq_domain *domain, struct device *dev,
> +                           int nvec, msi_alloc_info_t *arg);

I wonder if this should use the popular #ifdef dev_msi_prepare scheme
instead of a weak symbol?

Jason

Dey, Megha July 22, 2020, 5:03 p.m. UTC | #2

Hi Dan,

On 7/21/2020 9:21 AM, Jason Gunthorpe wrote:
> On Tue, Jul 21, 2020 at 09:02:35AM -0700, Dave Jiang wrote:
>> From: Megha Dey <[e-mail]>
>>
>> When DEV_MSI is enabled, the dev_msi_default_domain is updated to the
>> base DEV-MSI irq  domain. If interrupt remapping is enabled, we create
>> a new IR-DEV-MSI irq domain and update the dev_msi_default domain to
>> the same.
>>
>> For X86, introduce a new irq_alloc_type which will be used by the
>> interrupt remapping driver.
> 
> Why? Shouldn't this by symmetrical with normal MSI? Does MSI do this?

Since I am introducing the new dev msi domain for the case when IR_REMAP 
is turned on, I have introduced the new type in this patch.

MSI/MSIX have their own irq alloc types which are also only used by the 
intel remapping driver..

> 
> I would have thought you'd want to switch to this remapping mode as
> part of vfio or something like current cases.

Can you let me know what current case you are referring to?
> 
>> +struct irq_domain *create_remap_dev_msi_irq_domain(struct irq_domain *parent,
>> +						   const char *name)
>> +{
>> +	struct fwnode_handle *fn;
>> +	struct irq_domain *domain;
>> +
>> +	fn = irq_domain_alloc_named_fwnode(name);
>> +	if (!fn)
>> +		return NULL;
>> +
>> +	domain = msi_create_irq_domain(fn, &dev_msi_ir_domain_info, parent);
>> +	if (!domain) {
>> +		pr_warn("failed to initialize irqdomain for IR-DEV-MSI.\n");
>> +		return ERR_PTR(-ENXIO);
>> +	}
>> +
>> +	irq_domain_update_bus_token(domain, DOMAIN_BUS_PLATFORM_MSI);
>> +
>> +	if (!dev_msi_default_domain)
>> +		dev_msi_default_domain = domain;
>> +
>> +	return domain;
>> +}
> 
> What about this code creates a "remap" ? ie why is the function called
> "create_remap" ?

Well, this function creates a new domain for the case when IR_REMAP is 
enabled, hence I called it create_remap...

> 
>> diff --git a/include/linux/msi.h b/include/linux/msi.h
>> index 1da97f905720..7098ba566bcd 100644
>> +++ b/include/linux/msi.h
>> @@ -378,6 +378,9 @@ void *platform_msi_get_host_data(struct irq_domain *domain);
>>   void platform_msi_write_msg(struct irq_data *data, struct msi_msg *msg);
>>   void platform_msi_unmask_irq(struct irq_data *data);
>>   void platform_msi_mask_irq(struct irq_data *data);
>> +
>> +int dev_msi_prepare(struct irq_domain *domain, struct device *dev,
>> +                           int nvec, msi_alloc_info_t *arg);
> 
> I wonder if this should use the popular #ifdef dev_msi_prepare scheme
> instead of a weak symbol?

Ok, I will look into the #ifdef option.
> 
> Jason
>

Jason Gunthorpe July 22, 2020, 5:33 p.m. UTC | #3

On Wed, Jul 22, 2020 at 10:03:45AM -0700, Dey, Megha wrote:
> Hi Dan,
> 
> On 7/21/2020 9:21 AM, Jason Gunthorpe wrote:
> > On Tue, Jul 21, 2020 at 09:02:35AM -0700, Dave Jiang wrote:
> > > From: Megha Dey <[e-mail]>
> > > 
> > > When DEV_MSI is enabled, the dev_msi_default_domain is updated to the
> > > base DEV-MSI irq  domain. If interrupt remapping is enabled, we create
> > > a new IR-DEV-MSI irq domain and update the dev_msi_default domain to
> > > the same.
> > > 
> > > For X86, introduce a new irq_alloc_type which will be used by the
> > > interrupt remapping driver.
> > 
> > Why? Shouldn't this by symmetrical with normal MSI? Does MSI do this?
> 
> Since I am introducing the new dev msi domain for the case when IR_REMAP is
> turned on, I have introduced the new type in this patch.
>
> MSI/MSIX have their own irq alloc types which are also only used by the
> intel remapping driver..
>
> > 
> > I would have thought you'd want to switch to this remapping mode as
> > part of vfio or something like current cases.
> 
> Can you let me know what current case you are referring to?

My mistake, I see Intel unconditionally globally enables IR, so this
seems consistent with Intel's MSI

> > > +struct irq_domain *create_remap_dev_msi_irq_domain(struct irq_domain *parent,
> > > +						   const char *name)
> > > +{
> > > +	struct fwnode_handle *fn;
> > > +	struct irq_domain *domain;
> > > +
> > > +	fn = irq_domain_alloc_named_fwnode(name);
> > > +	if (!fn)
> > > +		return NULL;
> > > +
> > > +	domain = msi_create_irq_domain(fn, &dev_msi_ir_domain_info, parent);
> > > +	if (!domain) {
> > > +		pr_warn("failed to initialize irqdomain for IR-DEV-MSI.\n");
> > > +		return ERR_PTR(-ENXIO);
> > > +	}
> > > +
> > > +	irq_domain_update_bus_token(domain, DOMAIN_BUS_PLATFORM_MSI);
> > > +
> > > +	if (!dev_msi_default_domain)
> > > +		dev_msi_default_domain = domain;
> > > +
> > > +	return domain;
> > > +}
> > 
> > What about this code creates a "remap" ? ie why is the function called
> > "create_remap" ?
> 
> Well, this function creates a new domain for the case when IR_REMAP is
> enabled, hence I called it create_remap...

This looks like it just creates a new domain - the thing that makes it
remapping is the caller putting it under the ir_domain - so this code
here in base shouldn't have the word 'remap' in it, this is just
creating a generic domain.

It also kinda looks like create_dev_msi_domain() can just call the
above directly instead of duplicating everything - eg why do we need
two identical dev_msi_ir_controller vs dev_msi_controller just to have
the irq_set_vcpu_affinity difference?

Jason

Thomas Gleixner July 22, 2020, 8:44 p.m. UTC | #4

Dave Jiang <[e-mail]> writes:
> From: Megha Dey <[e-mail]>
>
> When DEV_MSI is enabled, the dev_msi_default_domain is updated to the
> base DEV-MSI irq  domain. If interrupt remapping is enabled, we create

s/we//

> a new IR-DEV-MSI irq domain and update the dev_msi_default domain to
> the same.
>
> For X86, introduce a new irq_alloc_type which will be used by the
> interrupt remapping driver.
>
> Reviewed-by: Dan Williams <[e-mail]>
> Signed-off-by: Megha Dey <[e-mail]>
> Signed-off-by: Dave Jiang <[e-mail]>
> ---
>  arch/x86/include/asm/hw_irq.h       |    1 +
>  arch/x86/kernel/apic/msi.c          |   12 ++++++
>  drivers/base/dev-msi.c              |   66 +++++++++++++++++++++++++++++++----
>  drivers/iommu/intel/irq_remapping.c |   11 +++++-
>  include/linux/intel-iommu.h         |    1 +
>  include/linux/irqdomain.h           |   11 ++++++
>  include/linux/msi.h                 |    3 ++

Why is this mixing generic code, x86 core code and intel specific driver
code? This is new functionality so:

      1) Provide the infrastructure
      2) Add support to architecture specific parts
      3) Enable it

> +
> +#ifdef CONFIG_DEV_MSI
> +int dev_msi_prepare(struct irq_domain *domain, struct device *dev,
> +			   int nvec, msi_alloc_info_t *arg)
> +{
> +	memset(arg, 0, sizeof(*arg));
> +
> +	arg->type = X86_IRQ_ALLOC_TYPE_DEV_MSI;
> +
> +	return 0;
> +}
> +#endif

What is this? Tons of new lines for taking up more space and not a
single comment.

> -static int dev_msi_prepare(struct irq_domain *domain, struct device *dev,
> +int __weak dev_msi_prepare(struct irq_domain *domain, struct device *dev,
>  			   int nvec, msi_alloc_info_t *arg)
>  {
>  	memset(arg, 0, sizeof(*arg));

Oh well. So every architecure which needs to override this and I assume
all which are eventually going to support it need to do the memset() in
their override.

       memset(arg,,,);
       arch_dev_msi_prepare();


> -	dev_msi_default_domain = msi_create_irq_domain(fn, &dev_msi_domain_info, parent);
> +	/*
> +	 * This initcall may come after remap code is initialized. Ensure that
> +	 * dev_msi_default domain is updated correctly.

What? No, this is a disgusting hack. Get your ordering straight, that's
not rocket science.

> +#ifdef CONFIG_IRQ_REMAP

IRQ_REMAP is x86 specific. Is this file x86 only or intended to be for
general use? If it's x86 only, then this should be clearly
documented. If not, then these x86'isms have no place here.

> +struct irq_domain *create_remap_dev_msi_irq_domain(struct irq_domain *parent,
> +						   const char *name)

So we have msi_create_irq_domain() and this is about dev_msi, right? So
can you please stick with a consistent naming scheme?

> +{
> +	struct fwnode_handle *fn;
> +	struct irq_domain *domain;
> +
> +	fn = irq_domain_alloc_named_fwnode(name);
> +	if (!fn)
> +		return NULL;
> +
> +	domain = msi_create_irq_domain(fn, &dev_msi_ir_domain_info, parent);
> +	if (!domain) {
> +		pr_warn("failed to initialize irqdomain for IR-DEV-MSI.\n");
> +		return ERR_PTR(-ENXIO);
> +	}
> +
> +	irq_domain_update_bus_token(domain, DOMAIN_BUS_PLATFORM_MSI);
> +
> +	if (!dev_msi_default_domain)
> +		dev_msi_default_domain = domain;

Can this be called several times? If so, then this lacks a comment. If
not, then this condition is useless.

Thanks,

        tglx

Dey, Megha Aug. 5, 2020, 7:02 p.m. UTC | #5

Hi Thomas,

> -----Original Message-----
> From: Thomas Gleixner <[e-mail]>
> Sent: Wednesday, July 22, 2020 1:45 PM
> To: Jiang, Dave <[e-mail]>; [e-mail]; Dey, Megha
> <[e-mail]>; [e-mail]; [e-mail];
> [e-mail]; [e-mail]; [e-mail];
> [e-mail]; Pan, Jacob jun <[e-mail]>; Raj,
> Ashok <[e-mail]>; [e-mail]; Liu, Yi L <[e-mail]>;
> Lu, Baolu <[e-mail]>; Tian, Kevin <[e-mail]>; Kumar,
> Sanjay K <[e-mail]>; Luck, Tony <[e-mail]>; Lin,
> Jing <[e-mail]>; Williams, Dan J <[e-mail]>;
> [e-mail]; [e-mail]; [e-mail];
> [e-mail]; [e-mail]; Hansen, Dave
> <[e-mail]>; [e-mail]; [e-mail];
> [e-mail]; [e-mail]; Ortiz, Samuel
> <[e-mail]>; Hossain, Mona <[e-mail]>
> Cc: [e-mail]; [e-mail];
> [e-mail]; [e-mail]; [e-mail]
> Subject: Re: [PATCH RFC v2 03/18] irq/dev-msi: Create IR-DEV-MSI irq domain
> 
> Dave Jiang <[e-mail]> writes:
> > From: Megha Dey <[e-mail]>
> >
> > When DEV_MSI is enabled, the dev_msi_default_domain is updated to the
> > base DEV-MSI irq  domain. If interrupt remapping is enabled, we create
> 
> s/we//

ok
> 
> > a new IR-DEV-MSI irq domain and update the dev_msi_default domain to
> > the same.
> >
> > For X86, introduce a new irq_alloc_type which will be used by the
> > interrupt remapping driver.
> >
> > Reviewed-by: Dan Williams <[e-mail]>
> > Signed-off-by: Megha Dey <[e-mail]>
> > Signed-off-by: Dave Jiang <[e-mail]>
> > ---
> >  arch/x86/include/asm/hw_irq.h       |    1 +
> >  arch/x86/kernel/apic/msi.c          |   12 ++++++
> >  drivers/base/dev-msi.c              |   66 +++++++++++++++++++++++++++++++----
> >  drivers/iommu/intel/irq_remapping.c |   11 +++++-
> >  include/linux/intel-iommu.h         |    1 +
> >  include/linux/irqdomain.h           |   11 ++++++
> >  include/linux/msi.h                 |    3 ++
> 
> Why is this mixing generic code, x86 core code and intel specific driver code?
> This is new functionality so:
> 
>       1) Provide the infrastructure
>       2) Add support to architecture specific parts
>       3) Enable it

Ok, I will try to adhere to the layering next time around..
> 
> > +
> > +#ifdef CONFIG_DEV_MSI
> > +int dev_msi_prepare(struct irq_domain *domain, struct device *dev,
> > +			   int nvec, msi_alloc_info_t *arg) {
> > +	memset(arg, 0, sizeof(*arg));
> > +
> > +	arg->type = X86_IRQ_ALLOC_TYPE_DEV_MSI;
> > +
> > +	return 0;
> > +}
> > +#endif
> 
> What is this? Tons of new lines for taking up more space and not a single
> comment.

Hmm, I will add a comment..
> 
> > -static int dev_msi_prepare(struct irq_domain *domain, struct device
> > *dev,
> > +int __weak dev_msi_prepare(struct irq_domain *domain, struct device
> > +*dev,
> >  			   int nvec, msi_alloc_info_t *arg)  {
> >  	memset(arg, 0, sizeof(*arg));
> 
> Oh well. So every architecure which needs to override this and I assume all
> which are eventually going to support it need to do the memset() in their
> override.
> 
>        memset(arg,,,);
>        arch_dev_msi_prepare();
> 
> 
Per you suggestion, I have introduced arch_dev_msi_prepare which returns 0 by default unless
overridden by arch code in the next patch set.

> > -	dev_msi_default_domain = msi_create_irq_domain(fn,
> &dev_msi_domain_info, parent);
> > +	/*
> > +	 * This initcall may come after remap code is initialized. Ensure that
> > +	 * dev_msi_default domain is updated correctly.
> 
> What? No, this is a disgusting hack. Get your ordering straight, that's not rocket
> science.
> 

Hmm yeah, actually I realized we don't really need to have 2 new IRQ domains for dev-msi 
(with and without interrupt remapping enabled). Hence all this will go away in the next round
of patches.

> > +#ifdef CONFIG_IRQ_REMAP
> 
> IRQ_REMAP is x86 specific. Is this file x86 only or intended to be for general use?
> If it's x86 only, then this should be clearly documented. If not, then these
> x86'isms have no place here.

True, I will take care of this in the next patch set.
> 
> > +struct irq_domain *create_remap_dev_msi_irq_domain(struct irq_domain
> *parent,
> > +						   const char *name)
> 
> So we have msi_create_irq_domain() and this is about dev_msi, right? So can
> you please stick with a consistent naming scheme?

sure
> 
> > +{
> > +	struct fwnode_handle *fn;
> > +	struct irq_domain *domain;
> > +
> > +	fn = irq_domain_alloc_named_fwnode(name);
> > +	if (!fn)
> > +		return NULL;
> > +
> > +	domain = msi_create_irq_domain(fn, &dev_msi_ir_domain_info,
> parent);
> > +	if (!domain) {
> > +		pr_warn("failed to initialize irqdomain for IR-DEV-MSI.\n");
> > +		return ERR_PTR(-ENXIO);
> > +	}
> > +
> > +	irq_domain_update_bus_token(domain,
> DOMAIN_BUS_PLATFORM_MSI);
> > +
> > +	if (!dev_msi_default_domain)
> > +		dev_msi_default_domain = domain;
> 
> Can this be called several times? If so, then this lacks a comment. If not, then
> this condition is useless.

Hmm this will go way in the next patch set, thank you for your input!
> 
> Thanks,
> 
>         tglx

Patch

diff --git a/arch/x86/include/asm/hw_irq.h b/arch/x86/include/asm/hw_irq.h
index 8ecd7570589d..bdddd63add41 100644
--- a/arch/x86/include/asm/hw_irq.h
+++ b/arch/x86/include/asm/hw_irq.h
@@ -40,6 +40,7 @@  enum irq_alloc_type {
 	X86_IRQ_ALLOC_TYPE_MSIX,
 	X86_IRQ_ALLOC_TYPE_DMAR,
 	X86_IRQ_ALLOC_TYPE_UV,
+	X86_IRQ_ALLOC_TYPE_DEV_MSI,
 };
 
 struct irq_alloc_info {
diff --git a/arch/x86/kernel/apic/msi.c b/arch/x86/kernel/apic/msi.c
index 5cbaca58af95..8b25cadbae09 100644
--- a/arch/x86/kernel/apic/msi.c
+++ b/arch/x86/kernel/apic/msi.c
@@ -507,3 +507,15 @@  int hpet_assign_irq(struct irq_domain *domain, struct hpet_channel *hc,
 	return irq_domain_alloc_irqs(domain, 1, NUMA_NO_NODE, &info);
 }
 #endif
+
+#ifdef CONFIG_DEV_MSI
+int dev_msi_prepare(struct irq_domain *domain, struct device *dev,
+			   int nvec, msi_alloc_info_t *arg)
+{
+	memset(arg, 0, sizeof(*arg));
+
+	arg->type = X86_IRQ_ALLOC_TYPE_DEV_MSI;
+
+	return 0;
+}
+#endif
diff --git a/drivers/base/dev-msi.c b/drivers/base/dev-msi.c
index 240ccc353933..43d6ed3ba10f 100644
--- a/drivers/base/dev-msi.c
+++ b/drivers/base/dev-msi.c
@@ -5,6 +5,7 @@ 
  * Author: Megha Dey <[e-mail]>
  */
 
+#include <linux/device.h>
 #include <linux/irq.h>
 #include <linux/irqdomain.h>
 #include <linux/msi.h>
@@ -32,7 +33,7 @@  static void dev_msi_set_desc(msi_alloc_info_t *arg, struct msi_desc *desc)
 	arg->hwirq = dev_msi_calc_hwirq(desc);
 }
 
-static int dev_msi_prepare(struct irq_domain *domain, struct device *dev,
+int __weak dev_msi_prepare(struct irq_domain *domain, struct device *dev,
 			   int nvec, msi_alloc_info_t *arg)
 {
 	memset(arg, 0, sizeof(*arg));
@@ -81,15 +82,66 @@  static int __init create_dev_msi_domain(void)
 	if (!fn)
 		return -ENXIO;
 
-	dev_msi_default_domain = msi_create_irq_domain(fn, &dev_msi_domain_info, parent);
+	/*
+	 * This initcall may come after remap code is initialized. Ensure that
+	 * dev_msi_default domain is updated correctly.
+	 */
 	if (!dev_msi_default_domain) {
-		pr_warn("failed to initialize irqdomain for DEV-MSI.\n");
-		return -ENXIO;
+		dev_msi_default_domain = msi_create_irq_domain(fn, &dev_msi_domain_info, parent);
+		if (!dev_msi_default_domain) {
+			pr_warn("failed to initialize irqdomain for DEV-MSI.\n");
+			return -ENXIO;
+		}
+
+		irq_domain_update_bus_token(dev_msi_default_domain, DOMAIN_BUS_PLATFORM_MSI);
+		irq_domain_free_fwnode(fn);
 	}
 
-	irq_domain_update_bus_token(dev_msi_default_domain, DOMAIN_BUS_PLATFORM_MSI);
-	irq_domain_free_fwnode(fn);
-
 	return 0;
 }
 device_initcall(create_dev_msi_domain);
+
+#ifdef CONFIG_IRQ_REMAP
+static struct irq_chip dev_msi_ir_controller = {
+	.name			= "IR-DEV-MSI",
+	.irq_unmask		= platform_msi_unmask_irq,
+	.irq_mask		= platform_msi_mask_irq,
+	.irq_write_msi_msg	= platform_msi_write_msg,
+	.irq_ack		= irq_chip_ack_parent,
+	.irq_retrigger		= irq_chip_retrigger_hierarchy,
+	.irq_set_vcpu_affinity	= irq_chip_set_vcpu_affinity_parent,
+	.flags			= IRQCHIP_SKIP_SET_WAKE,
+};
+
+static struct msi_domain_info dev_msi_ir_domain_info = {
+	.flags		= MSI_FLAG_USE_DEF_DOM_OPS | MSI_FLAG_USE_DEF_CHIP_OPS,
+	.ops		= &dev_msi_domain_ops,
+	.chip		= &dev_msi_ir_controller,
+	.handler	= handle_edge_irq,
+	.handler_name	= "edge",
+};
+
+struct irq_domain *create_remap_dev_msi_irq_domain(struct irq_domain *parent,
+						   const char *name)
+{
+	struct fwnode_handle *fn;
+	struct irq_domain *domain;
+
+	fn = irq_domain_alloc_named_fwnode(name);
+	if (!fn)
+		return NULL;
+
+	domain = msi_create_irq_domain(fn, &dev_msi_ir_domain_info, parent);
+	if (!domain) {
+		pr_warn("failed to initialize irqdomain for IR-DEV-MSI.\n");
+		return ERR_PTR(-ENXIO);
+	}
+
+	irq_domain_update_bus_token(domain, DOMAIN_BUS_PLATFORM_MSI);
+
+	if (!dev_msi_default_domain)
+		dev_msi_default_domain = domain;
+
+	return domain;
+}
+#endif
diff --git a/drivers/iommu/intel/irq_remapping.c b/drivers/iommu/intel/irq_remapping.c
index 7f8769800815..51872aabe5f8 100644
--- a/drivers/iommu/intel/irq_remapping.c
+++ b/drivers/iommu/intel/irq_remapping.c
@@ -573,6 +573,10 @@  static int intel_setup_irq_remapping(struct intel_iommu *iommu)
 						 "INTEL-IR-MSI",
 						 iommu->seq_id);
 
+	iommu->ir_dev_msi_domain =
+		create_remap_dev_msi_irq_domain(iommu->ir_domain,
+						"INTEL-IR-DEV-MSI");
+
 	ir_table->base = page_address(pages);
 	ir_table->bitmap = bitmap;
 	iommu->ir_table = ir_table;
@@ -1299,9 +1303,10 @@  static void intel_irq_remapping_prepare_irte(struct intel_ir_data *data,
 	case X86_IRQ_ALLOC_TYPE_HPET:
 	case X86_IRQ_ALLOC_TYPE_MSI:
 	case X86_IRQ_ALLOC_TYPE_MSIX:
+	case X86_IRQ_ALLOC_TYPE_DEV_MSI:
 		if (info->type == X86_IRQ_ALLOC_TYPE_HPET)
 			set_hpet_sid(irte, info->hpet_id);
-		else
+		else if (info->type != X86_IRQ_ALLOC_TYPE_DEV_MSI)
 			set_msi_sid(irte, info->msi_dev);
 
 		msg->address_hi = MSI_ADDR_BASE_HI;
@@ -1353,8 +1358,10 @@  static int intel_irq_remapping_alloc(struct irq_domain *domain,
 
 	if (!info || !iommu)
 		return -EINVAL;
+
 	if (nr_irqs > 1 && info->type != X86_IRQ_ALLOC_TYPE_MSI &&
-	    info->type != X86_IRQ_ALLOC_TYPE_MSIX)
+	    info->type != X86_IRQ_ALLOC_TYPE_MSIX &&
+	    info->type != X86_IRQ_ALLOC_TYPE_DEV_MSI)
 		return -EINVAL;
 
 	/*
diff --git a/include/linux/intel-iommu.h b/include/linux/intel-iommu.h
index d129baf7e0b8..3b868d1c43df 100644
--- a/include/linux/intel-iommu.h
+++ b/include/linux/intel-iommu.h
@@ -596,6 +596,7 @@  struct intel_iommu {
 	struct ir_table *ir_table;	/* Interrupt remapping info */
 	struct irq_domain *ir_domain;
 	struct irq_domain *ir_msi_domain;
+	struct irq_domain *ir_dev_msi_domain;
 #endif
 	struct iommu_device iommu;  /* IOMMU core code handle */
 	int		node;
diff --git a/include/linux/irqdomain.h b/include/linux/irqdomain.h
index b37350c4fe37..e537d7b50cee 100644
--- a/include/linux/irqdomain.h
+++ b/include/linux/irqdomain.h
@@ -589,6 +589,17 @@  irq_domain_hierarchical_is_msi_remap(struct irq_domain *domain)
 }
 #endif	/* CONFIG_IRQ_DOMAIN_HIERARCHY */
 
+#if defined(CONFIG_DEV_MSI) && defined(CONFIG_IRQ_REMAP)
+extern struct irq_domain *create_remap_dev_msi_irq_domain(struct irq_domain *parent,
+							  const char *name);
+#else
+static inline struct irq_domain *create_remap_dev_msi_irq_domain(struct irq_domain *parent,
+								 const char *name)
+{
+	return NULL;
+}
+#endif
+
 #else /* CONFIG_IRQ_DOMAIN */
 static inline void irq_dispose_mapping(unsigned int virq) { }
 static inline struct irq_domain *irq_find_matching_fwnode(
diff --git a/include/linux/msi.h b/include/linux/msi.h
index 1da97f905720..7098ba566bcd 100644
--- a/include/linux/msi.h
+++ b/include/linux/msi.h
@@ -378,6 +378,9 @@  void *platform_msi_get_host_data(struct irq_domain *domain);
 void platform_msi_write_msg(struct irq_data *data, struct msi_msg *msg);
 void platform_msi_unmask_irq(struct irq_data *data);
 void platform_msi_mask_irq(struct irq_data *data);
+
+int dev_msi_prepare(struct irq_domain *domain, struct device *dev,
+                           int nvec, msi_alloc_info_t *arg);
 #endif /* CONFIG_GENERIC_MSI_IRQ_DOMAIN */
 
 #ifdef CONFIG_PCI_MSI_IRQ_DOMAIN