| Message ID | 20240505145318.398135-2-shayd@nvidia.com (mailing list archive) |
|---|---|
| State | Superseded |
| Delegated to: | Netdev Maintainers |
| Headers | show |
| Series | Introduce auxiliary bus IRQs sysfs | expand |
> From: Shay Drori <shayd@nvidia.com> > Sent: Sunday, May 5, 2024 7:53 AM > diff --git a/include/linux/auxiliary_bus.h b/include/linux/auxiliary_bus.h > index de21d9d24a95..fe2c438c0217 100644 > --- a/include/linux/auxiliary_bus.h > +++ b/include/linux/auxiliary_bus.h > @@ -58,6 +58,9 @@ > * in > * @name: Match name found by the auxiliary device driver, > * @id: unique identitier if multiple devices of the same name are exported, > + * @irqs: irqs xarray contains irq indices which are used by the > + device, > + * @groups: first group is for irqs sysfs directory; it is a NULL terminated > + * array, > * > * An auxiliary_device represents a part of its parent device's functionality. > * It is given a name that, combined with the registering drivers @@ -138,6 > +141,8 @@ struct auxiliary_device { > struct device dev; > const char *name; > + struct xarray irqs; > + const struct attribute_group *groups[2]; > u32 id; > }; > > @@ -209,8 +214,19 @@ static inline struct auxiliary_driver > *to_auxiliary_drv(struct device_driver *dr } > > int auxiliary_device_init(struct auxiliary_device *auxdev); -int > __auxiliary_device_add(struct auxiliary_device *auxdev, const char > *modname); -#define auxiliary_device_add(auxdev) > __auxiliary_device_add(auxdev, KBUILD_MODNAME) > +int __auxiliary_device_add(struct auxiliary_device *auxdev, const char > *modname, > + bool irqs_sysfs_enable); > +#define auxiliary_device_add(auxdev) __auxiliary_device_add(auxdev, > +KBUILD_MODNAME, false) #define auxiliary_device_add_with_irqs(auxdev) > \ > + __auxiliary_device_add(auxdev, KBUILD_MODNAME, true) > + > +#ifdef CONFIG_SYSFS > +int auxiliary_device_sysfs_irq_add(struct auxiliary_device *auxdev, int > +irq); void auxiliary_device_sysfs_irq_remove(struct auxiliary_device > +*auxdev, int irq); #else /* CONFIG_SYSFS */ int > +auxiliary_device_sysfs_irq_add(struct auxiliary_device *auxdev, int > +irq) {return 0; } void auxiliary_device_sysfs_irq_remove(struct > +auxiliary_device *auxdev, int irq) {} #endif > Above definitions need to be static inline and should under 80 characters. Please fix them.
On 5/5/24 16:53, Shay Drory wrote: > PCI subfunctions (SF) are anchored on the auxiliary bus. PCI physical > and virtual functions are anchored on the PCI bus; the irq information > of each such function is visible to users via sysfs directory "msi_irqs" > containing file for each irq entry. However, for PCI SFs such information > is unavailable. Due to this users have no visibility on IRQs used by the > SFs. > Secondly, an SF is a multi function device supporting rdma, netdevice > and more. Without irq information at the bus level, the user is unable > to view or use the affinity of the SF IRQs. > > Hence to match to the equivalent PCI PFs and VFs, add "irqs" directory, > for supporting auxiliary devices, containing file for each irq entry. > > Additionally, the PCI SFs sometimes share the IRQs with peer SFs. This > information is also not available to the users. To overcome this > limitation, each irq sysfs entry shows if irq is exclusive or shared. > > For example: > $ ls /sys/bus/auxiliary/devices/mlx5_core.sf.1/irqs/ > 50 51 52 53 54 55 56 57 58 > $ cat /sys/bus/auxiliary/devices/mlx5_core.sf.1/irqs/52 > exclusive > > Reviewed-by: Parav Pandit <parav@nvidia.com> > Signed-off-by: Shay Drory <shayd@nvidia.com> > > --- > v1->v2: > - move #ifdefs from drivers/base/auxiliary.c to > include/linux/auxiliary_bus.h (Greg) > - use EXPORT_SYMBOL_GPL instead of EXPORT_SYMBOL (Greg) > - Fix kzalloc(ref) to kzalloc(*ref) (Simon) > - Add return description in auxiliary_device_sysfs_irq_add() kdoc (Simon) > - Fix auxiliary_irq_mode_show doc (kernel test boot) > --- > Documentation/ABI/testing/sysfs-bus-auxiliary | 14 ++ > drivers/base/auxiliary.c | 167 +++++++++++++++++- > include/linux/auxiliary_bus.h | 20 ++- > 3 files changed, 198 insertions(+), 3 deletions(-) > create mode 100644 Documentation/ABI/testing/sysfs-bus-auxiliary > > diff --git a/Documentation/ABI/testing/sysfs-bus-auxiliary b/Documentation/ABI/testing/sysfs-bus-auxiliary > new file mode 100644 > index 000000000000..3b8299d49d9e > --- /dev/null > +++ b/Documentation/ABI/testing/sysfs-bus-auxiliary > @@ -0,0 +1,14 @@ > +What: /sys/bus/auxiliary/devices/.../irqs/ > +Date: April, 2024 > +Contact: Shay Drory <shayd@nvidia.com> > +Description: > + The /sys/devices/.../irqs directory contains a variable set of > + files, with each file is named as irq number similar to PCI PF > + or VF's irq number located in msi_irqs directory. > + > +What: /sys/bus/auxiliary/devices/.../irqs/<N> > +Date: April, 2024 > +Contact: Shay Drory <shayd@nvidia.com> > +Description: > + auxiliary devices can share IRQs. This attribute indicates if > + the irq is shared with other SFs or exclusively used by the SF. > diff --git a/drivers/base/auxiliary.c b/drivers/base/auxiliary.c > index d3a2c40c2f12..43d12a147f1f 100644 > --- a/drivers/base/auxiliary.c > +++ b/drivers/base/auxiliary.c > @@ -158,6 +158,164 @@ > * }; > */ > > +#ifdef CONFIG_SYSFS > +/* Xarray of irqs to determine if irq is exclusive or shared. */ > +static DEFINE_XARRAY(irqs); > +/* Protects insertions into the irtqs xarray. */ > +static DEFINE_MUTEX(irqs_lock); > + > +struct auxiliary_irq_info { > + struct device_attribute sysfs_attr; > + int irq; > +}; > + > +static struct attribute *auxiliary_irq_attrs[] = { > + NULL > +}; > + > +static const struct attribute_group auxiliary_irqs_group = { > + .name = "irqs", > + .attrs = auxiliary_irq_attrs, > +}; > + > +/* Auxiliary devices can share IRQs. Expose to user whether the provided IRQ is > + * shared or exclusive. > + */ > +static ssize_t auxiliary_irq_mode_show(struct device *dev, > + struct device_attribute *attr, char *buf) > +{ > + struct auxiliary_irq_info *info = > + container_of(attr, struct auxiliary_irq_info, sysfs_attr); > + > + if (refcount_read(xa_load(&irqs, info->irq)) > 1) > + return sysfs_emit(buf, "%s\n", "shared"); > + else > + return sysfs_emit(buf, "%s\n", "exclusive"); > +} > + > +static void auxiliary_irq_destroy(int irq) > +{ > + refcount_t *ref; > + > + xa_lock(&irqs); > + ref = xa_load(&irqs, irq); > + if (refcount_dec_and_test(ref)) { > + __xa_erase(&irqs, irq); > + kfree(ref); > + } > + xa_unlock(&irqs); > +} > + > +static int auxiliary_irq_create(int irq) > +{ > + refcount_t *ref; > + int ret = 0; > + > + mutex_lock(&irqs_lock); > + ref = xa_load(&irqs, irq); > + if (ref && refcount_inc_not_zero(ref)) > + goto out; > + > + ref = kzalloc(sizeof(*ref), GFP_KERNEL); > + if (!ref) { > + ret = -ENOMEM; > + goto out; > + } > + > + refcount_set(ref, 1); > + ret = xa_insert(&irqs, irq, ref, GFP_KERNEL); > + if (ret) > + kfree(ref); > + > +out: > + mutex_unlock(&irqs_lock); > + return ret; > +} > + > +/** > + * auxiliary_device_sysfs_irq_add - add a sysfs entry for the given IRQ > + * @auxdev: auxiliary bus device to add the sysfs entry. > + * @irq: The associated Linux interrupt number. > + * > + * This function should be called after auxiliary device have successfully > + * received the irq. s/received/registered/? > + * > + * Return: zero on success or an error code on failure. > + */ > +int auxiliary_device_sysfs_irq_add(struct auxiliary_device *auxdev, int irq) > +{ > + struct device *dev = &auxdev->dev; > + struct auxiliary_irq_info *info; > + int ret; > + > + ret = auxiliary_irq_create(irq); > + if (ret) > + return ret; > + > + info = kzalloc(sizeof(*info), GFP_KERNEL); > + if (!info) { > + ret = -ENOMEM; > + goto info_err; > + } > + > + sysfs_attr_init(&info->sysfs_attr.attr); > + info->sysfs_attr.attr.name = kasprintf(GFP_KERNEL, "%d", irq); > + if (!info->sysfs_attr.attr.name) { > + ret = -ENOMEM; > + goto name_err; > + } > + info->irq = irq; > + info->sysfs_attr.attr.mode = 0444; > + info->sysfs_attr.show = auxiliary_irq_mode_show; > + > + ret = xa_insert(&auxdev->irqs, irq, info, GFP_KERNEL); > + if (ret) > + goto auxdev_xa_err; > + > + ret = sysfs_add_file_to_group(&dev->kobj, &info->sysfs_attr.attr, > + auxiliary_irqs_group.name); > + if (ret) > + goto sysfs_add_err; > + > + return 0; > + > +sysfs_add_err: > + xa_erase(&auxdev->irqs, irq); > +auxdev_xa_err: > + kfree(info->sysfs_attr.attr.name); > +name_err: > + kfree(info); > +info_err: > + auxiliary_irq_destroy(irq); > + return ret; > +} > +EXPORT_SYMBOL_GPL(auxiliary_device_sysfs_irq_add); > + > +/** > + * auxiliary_device_sysfs_irq_remove - remove a sysfs entry for the given IRQ > + * @auxdev: auxiliary bus device to add the sysfs entry. > + * @irq: the IRQ to remove. > + * > + * This function should be called to remove an IRQ sysfs entry. > + */ > +void auxiliary_device_sysfs_irq_remove(struct auxiliary_device *auxdev, int irq) (not an issue, just a question) do you need to select IRQ to remove? ... > +{ > + struct auxiliary_irq_info *info = xa_load(&auxdev->irqs, irq); > + struct device *dev = &auxdev->dev; > + > + if (WARN_ON(!info)) > + return; > + > + sysfs_remove_file_from_group(&dev->kobj, &info->sysfs_attr.attr, > + auxiliary_irqs_group.name); ... because there is an option to remove whole group at once > + xa_erase(&auxdev->irqs, irq); > + kfree(info->sysfs_attr.attr.name); > + kfree(info); > + auxiliary_irq_destroy(irq); > +} > +EXPORT_SYMBOL_GPL(auxiliary_device_sysfs_irq_remove); > +#endif > + > static const struct auxiliary_device_id *auxiliary_match_id(const struct auxiliary_device_id *id, > const struct auxiliary_device *auxdev) > { > @@ -295,6 +453,7 @@ EXPORT_SYMBOL_GPL(auxiliary_device_init); > * __auxiliary_device_add - add an auxiliary bus device > * @auxdev: auxiliary bus device to add to the bus > * @modname: name of the parent device's driver module > + * @irqs_sysfs_enable: whether to enable IRQs sysfs > * > * This is the third step in the three-step process to register an > * auxiliary_device. > @@ -310,7 +469,8 @@ EXPORT_SYMBOL_GPL(auxiliary_device_init); > * parameter. Only if a user requires a custom name would this version be > * called directly. > */ > -int __auxiliary_device_add(struct auxiliary_device *auxdev, const char *modname) > +int __auxiliary_device_add(struct auxiliary_device *auxdev, const char *modname, > + bool irqs_sysfs_enable) > { > struct device *dev = &auxdev->dev; > int ret; > @@ -325,6 +485,11 @@ int __auxiliary_device_add(struct auxiliary_device *auxdev, const char *modname) > dev_err(dev, "auxiliary device dev_set_name failed: %d\n", ret); > return ret; > } > + if (irqs_sysfs_enable) { > + auxdev->groups[0] = &auxiliary_irqs_group; I would remove this array ... > + xa_init(&auxdev->irqs); > + dev->groups = auxdev->groups; ... and use &auxiliary_irqs_group directly here (you will need to change it to 2 elem array though) > + } > > ret = device_add(dev); > if (ret) > diff --git a/include/linux/auxiliary_bus.h b/include/linux/auxiliary_bus.h > index de21d9d24a95..fe2c438c0217 100644 > --- a/include/linux/auxiliary_bus.h > +++ b/include/linux/auxiliary_bus.h > @@ -58,6 +58,9 @@ > * in > * @name: Match name found by the auxiliary device driver, > * @id: unique identitier if multiple devices of the same name are exported, > + * @irqs: irqs xarray contains irq indices which are used by the device, > + * @groups: first group is for irqs sysfs directory; it is a NULL terminated > + * array, > * > * An auxiliary_device represents a part of its parent device's functionality. > * It is given a name that, combined with the registering drivers > @@ -138,6 +141,8 @@ > struct auxiliary_device { > struct device dev; > const char *name; > + struct xarray irqs; > + const struct attribute_group *groups[2]; > u32 id; > }; > > @@ -209,8 +214,19 @@ static inline struct auxiliary_driver *to_auxiliary_drv(struct device_driver *dr > } > > int auxiliary_device_init(struct auxiliary_device *auxdev); > -int __auxiliary_device_add(struct auxiliary_device *auxdev, const char *modname); > -#define auxiliary_device_add(auxdev) __auxiliary_device_add(auxdev, KBUILD_MODNAME) > +int __auxiliary_device_add(struct auxiliary_device *auxdev, const char *modname, > + bool irqs_sysfs_enable); > +#define auxiliary_device_add(auxdev) __auxiliary_device_add(auxdev, KBUILD_MODNAME, false) > +#define auxiliary_device_add_with_irqs(auxdev) \ > + __auxiliary_device_add(auxdev, KBUILD_MODNAME, true) > + > +#ifdef CONFIG_SYSFS > +int auxiliary_device_sysfs_irq_add(struct auxiliary_device *auxdev, int irq); > +void auxiliary_device_sysfs_irq_remove(struct auxiliary_device *auxdev, int irq); > +#else /* CONFIG_SYSFS */ > +int auxiliary_device_sysfs_irq_add(struct auxiliary_device *auxdev, int irq) {return 0; } > +void auxiliary_device_sysfs_irq_remove(struct auxiliary_device *auxdev, int irq) {} > +#endif > > static inline void auxiliary_device_uninit(struct auxiliary_device *auxdev) > {
On 06/05/2024 18:15, Parav Pandit wrote: > >> From: Shay Drori <shayd@nvidia.com> >> Sent: Sunday, May 5, 2024 7:53 AM > > >> diff --git a/include/linux/auxiliary_bus.h b/include/linux/auxiliary_bus.h >> index de21d9d24a95..fe2c438c0217 100644 >> --- a/include/linux/auxiliary_bus.h >> +++ b/include/linux/auxiliary_bus.h >> @@ -58,6 +58,9 @@ >> * in >> * @name: Match name found by the auxiliary device driver, >> * @id: unique identitier if multiple devices of the same name are exported, >> + * @irqs: irqs xarray contains irq indices which are used by the >> + device, >> + * @groups: first group is for irqs sysfs directory; it is a NULL terminated >> + * array, >> * >> * An auxiliary_device represents a part of its parent device's functionality. >> * It is given a name that, combined with the registering drivers @@ -138,6 >> +141,8 @@ struct auxiliary_device { >> struct device dev; >> const char *name; >> + struct xarray irqs; >> + const struct attribute_group *groups[2]; >> u32 id; >> }; >> >> @@ -209,8 +214,19 @@ static inline struct auxiliary_driver >> *to_auxiliary_drv(struct device_driver *dr } >> >> int auxiliary_device_init(struct auxiliary_device *auxdev); -int >> __auxiliary_device_add(struct auxiliary_device *auxdev, const char >> *modname); -#define auxiliary_device_add(auxdev) >> __auxiliary_device_add(auxdev, KBUILD_MODNAME) >> +int __auxiliary_device_add(struct auxiliary_device *auxdev, const char >> *modname, >> + bool irqs_sysfs_enable); >> +#define auxiliary_device_add(auxdev) __auxiliary_device_add(auxdev, >> +KBUILD_MODNAME, false) #define auxiliary_device_add_with_irqs(auxdev) >> \ >> + __auxiliary_device_add(auxdev, KBUILD_MODNAME, true) >> + > >> +#ifdef CONFIG_SYSFS >> +int auxiliary_device_sysfs_irq_add(struct auxiliary_device *auxdev, int >> +irq); void auxiliary_device_sysfs_irq_remove(struct auxiliary_device >> +*auxdev, int irq); #else /* CONFIG_SYSFS */ int >> +auxiliary_device_sysfs_irq_add(struct auxiliary_device *auxdev, int >> +irq) {return 0; } void auxiliary_device_sysfs_irq_remove(struct >> +auxiliary_device *auxdev, int irq) {} #endif >> > Above definitions need to be static inline and should under 80 characters. > Please fix them. will fix in v3.
On 07/05/2024 11:12, Przemek Kitszel wrote: > External email: Use caution opening links or attachments > > > On 5/5/24 16:53, Shay Drory wrote: >> PCI subfunctions (SF) are anchored on the auxiliary bus. PCI physical >> and virtual functions are anchored on the PCI bus; the irq information >> of each such function is visible to users via sysfs directory "msi_irqs" >> containing file for each irq entry. However, for PCI SFs such information >> is unavailable. Due to this users have no visibility on IRQs used by the >> SFs. >> Secondly, an SF is a multi function device supporting rdma, netdevice >> and more. Without irq information at the bus level, the user is unable >> to view or use the affinity of the SF IRQs. >> >> Hence to match to the equivalent PCI PFs and VFs, add "irqs" directory, >> for supporting auxiliary devices, containing file for each irq entry. >> >> Additionally, the PCI SFs sometimes share the IRQs with peer SFs. This >> information is also not available to the users. To overcome this >> limitation, each irq sysfs entry shows if irq is exclusive or shared. >> >> For example: >> $ ls /sys/bus/auxiliary/devices/mlx5_core.sf.1/irqs/ >> 50 51 52 53 54 55 56 57 58 >> $ cat /sys/bus/auxiliary/devices/mlx5_core.sf.1/irqs/52 >> exclusive >> >> Reviewed-by: Parav Pandit <parav@nvidia.com> >> Signed-off-by: Shay Drory <shayd@nvidia.com> >> >> --- >> v1->v2: >> - move #ifdefs from drivers/base/auxiliary.c to >> include/linux/auxiliary_bus.h (Greg) >> - use EXPORT_SYMBOL_GPL instead of EXPORT_SYMBOL (Greg) >> - Fix kzalloc(ref) to kzalloc(*ref) (Simon) >> - Add return description in auxiliary_device_sysfs_irq_add() kdoc (Simon) >> - Fix auxiliary_irq_mode_show doc (kernel test boot) >> --- >> Documentation/ABI/testing/sysfs-bus-auxiliary | 14 ++ >> drivers/base/auxiliary.c | 167 +++++++++++++++++- >> include/linux/auxiliary_bus.h | 20 ++- >> 3 files changed, 198 insertions(+), 3 deletions(-) >> create mode 100644 Documentation/ABI/testing/sysfs-bus-auxiliary >> >> diff --git a/Documentation/ABI/testing/sysfs-bus-auxiliary >> b/Documentation/ABI/testing/sysfs-bus-auxiliary >> new file mode 100644 >> index 000000000000..3b8299d49d9e >> --- /dev/null >> +++ b/Documentation/ABI/testing/sysfs-bus-auxiliary >> @@ -0,0 +1,14 @@ >> +What: /sys/bus/auxiliary/devices/.../irqs/ >> +Date: April, 2024 >> +Contact: Shay Drory <shayd@nvidia.com> >> +Description: >> + The /sys/devices/.../irqs directory contains a variable >> set of >> + files, with each file is named as irq number similar to >> PCI PF >> + or VF's irq number located in msi_irqs directory. >> + >> +What: /sys/bus/auxiliary/devices/.../irqs/<N> >> +Date: April, 2024 >> +Contact: Shay Drory <shayd@nvidia.com> >> +Description: >> + auxiliary devices can share IRQs. This attribute >> indicates if >> + the irq is shared with other SFs or exclusively used by >> the SF. >> diff --git a/drivers/base/auxiliary.c b/drivers/base/auxiliary.c >> index d3a2c40c2f12..43d12a147f1f 100644 >> --- a/drivers/base/auxiliary.c >> +++ b/drivers/base/auxiliary.c >> @@ -158,6 +158,164 @@ >> * }; >> */ >> >> +#ifdef CONFIG_SYSFS >> +/* Xarray of irqs to determine if irq is exclusive or shared. */ >> +static DEFINE_XARRAY(irqs); >> +/* Protects insertions into the irtqs xarray. */ >> +static DEFINE_MUTEX(irqs_lock); >> + >> +struct auxiliary_irq_info { >> + struct device_attribute sysfs_attr; >> + int irq; >> +}; >> + >> +static struct attribute *auxiliary_irq_attrs[] = { >> + NULL >> +}; >> + >> +static const struct attribute_group auxiliary_irqs_group = { >> + .name = "irqs", >> + .attrs = auxiliary_irq_attrs, >> +}; >> + >> +/* Auxiliary devices can share IRQs. Expose to user whether the >> provided IRQ is >> + * shared or exclusive. >> + */ >> +static ssize_t auxiliary_irq_mode_show(struct device *dev, >> + struct device_attribute *attr, >> char *buf) >> +{ >> + struct auxiliary_irq_info *info = >> + container_of(attr, struct auxiliary_irq_info, sysfs_attr); >> + >> + if (refcount_read(xa_load(&irqs, info->irq)) > 1) >> + return sysfs_emit(buf, "%s\n", "shared"); >> + else >> + return sysfs_emit(buf, "%s\n", "exclusive"); >> +} >> + >> +static void auxiliary_irq_destroy(int irq) >> +{ >> + refcount_t *ref; >> + >> + xa_lock(&irqs); >> + ref = xa_load(&irqs, irq); >> + if (refcount_dec_and_test(ref)) { >> + __xa_erase(&irqs, irq); >> + kfree(ref); >> + } >> + xa_unlock(&irqs); >> +} >> + >> +static int auxiliary_irq_create(int irq) >> +{ >> + refcount_t *ref; >> + int ret = 0; >> + >> + mutex_lock(&irqs_lock); >> + ref = xa_load(&irqs, irq); >> + if (ref && refcount_inc_not_zero(ref)) >> + goto out; >> + >> + ref = kzalloc(sizeof(*ref), GFP_KERNEL); >> + if (!ref) { >> + ret = -ENOMEM; >> + goto out; >> + } >> + >> + refcount_set(ref, 1); >> + ret = xa_insert(&irqs, irq, ref, GFP_KERNEL); >> + if (ret) >> + kfree(ref); >> + >> +out: >> + mutex_unlock(&irqs_lock); >> + return ret; >> +} >> + >> +/** >> + * auxiliary_device_sysfs_irq_add - add a sysfs entry for the given IRQ >> + * @auxdev: auxiliary bus device to add the sysfs entry. >> + * @irq: The associated Linux interrupt number. >> + * >> + * This function should be called after auxiliary device have >> successfully >> + * received the irq. > > s/received/registered/? I used received on purpose. as mention in the commit message: "the PCI SFs sometimes share the IRQs with peer SFs." This means some SFs won't register the IRQ. > >> + * >> + * Return: zero on success or an error code on failure. >> + */ >> +int auxiliary_device_sysfs_irq_add(struct auxiliary_device *auxdev, >> int irq) >> +{ >> + struct device *dev = &auxdev->dev; >> + struct auxiliary_irq_info *info; >> + int ret; >> + >> + ret = auxiliary_irq_create(irq); >> + if (ret) >> + return ret; >> + >> + info = kzalloc(sizeof(*info), GFP_KERNEL); >> + if (!info) { >> + ret = -ENOMEM; >> + goto info_err; >> + } >> + >> + sysfs_attr_init(&info->sysfs_attr.attr); >> + info->sysfs_attr.attr.name = kasprintf(GFP_KERNEL, "%d", irq); >> + if (!info->sysfs_attr.attr.name) { >> + ret = -ENOMEM; >> + goto name_err; >> + } >> + info->irq = irq; >> + info->sysfs_attr.attr.mode = 0444; >> + info->sysfs_attr.show = auxiliary_irq_mode_show; >> + >> + ret = xa_insert(&auxdev->irqs, irq, info, GFP_KERNEL); >> + if (ret) >> + goto auxdev_xa_err; >> + >> + ret = sysfs_add_file_to_group(&dev->kobj, &info->sysfs_attr.attr, >> + auxiliary_irqs_group.name); >> + if (ret) >> + goto sysfs_add_err; >> + >> + return 0; >> + >> +sysfs_add_err: >> + xa_erase(&auxdev->irqs, irq); >> +auxdev_xa_err: >> + kfree(info->sysfs_attr.attr.name); >> +name_err: >> + kfree(info); >> +info_err: >> + auxiliary_irq_destroy(irq); >> + return ret; >> +} >> +EXPORT_SYMBOL_GPL(auxiliary_device_sysfs_irq_add); >> + >> +/** >> + * auxiliary_device_sysfs_irq_remove - remove a sysfs entry for the >> given IRQ >> + * @auxdev: auxiliary bus device to add the sysfs entry. >> + * @irq: the IRQ to remove. >> + * >> + * This function should be called to remove an IRQ sysfs entry. >> + */ >> +void auxiliary_device_sysfs_irq_remove(struct auxiliary_device >> *auxdev, int irq) > > (not an issue, just a question) > do you need to select IRQ to remove? ... yes. in order to keep a symetry between add and remove flows. > >> +{ >> + struct auxiliary_irq_info *info = xa_load(&auxdev->irqs, irq); >> + struct device *dev = &auxdev->dev; >> + >> + if (WARN_ON(!info)) >> + return; >> + >> + sysfs_remove_file_from_group(&dev->kobj, &info->sysfs_attr.attr, >> + auxiliary_irqs_group.name); > > ... because there is an option to remove whole group at once > >> + xa_erase(&auxdev->irqs, irq); >> + kfree(info->sysfs_attr.attr.name); >> + kfree(info); >> + auxiliary_irq_destroy(irq); >> +} >> +EXPORT_SYMBOL_GPL(auxiliary_device_sysfs_irq_remove); >> +#endif >> + >> static const struct auxiliary_device_id *auxiliary_match_id(const >> struct auxiliary_device_id *id, >> const struct >> auxiliary_device *auxdev) >> { >> @@ -295,6 +453,7 @@ EXPORT_SYMBOL_GPL(auxiliary_device_init); >> * __auxiliary_device_add - add an auxiliary bus device >> * @auxdev: auxiliary bus device to add to the bus >> * @modname: name of the parent device's driver module >> + * @irqs_sysfs_enable: whether to enable IRQs sysfs >> * >> * This is the third step in the three-step process to register an >> * auxiliary_device. >> @@ -310,7 +469,8 @@ EXPORT_SYMBOL_GPL(auxiliary_device_init); >> * parameter. Only if a user requires a custom name would this >> version be >> * called directly. >> */ >> -int __auxiliary_device_add(struct auxiliary_device *auxdev, const >> char *modname) >> +int __auxiliary_device_add(struct auxiliary_device *auxdev, const >> char *modname, >> + bool irqs_sysfs_enable) >> { >> struct device *dev = &auxdev->dev; >> int ret; >> @@ -325,6 +485,11 @@ int __auxiliary_device_add(struct >> auxiliary_device *auxdev, const char *modname) >> dev_err(dev, "auxiliary device dev_set_name failed: >> %d\n", ret); >> return ret; >> } >> + if (irqs_sysfs_enable) { >> + auxdev->groups[0] = &auxiliary_irqs_group; > > I would remove this array ... > >> + xa_init(&auxdev->irqs); >> + dev->groups = auxdev->groups; > > ... and use &auxiliary_irqs_group directly here > (you will need to change it to 2 elem array though) thanks. will change in v3. > >> + } >> >> ret = device_add(dev); >> if (ret) >> diff --git a/include/linux/auxiliary_bus.h >> b/include/linux/auxiliary_bus.h >> index de21d9d24a95..fe2c438c0217 100644 >> --- a/include/linux/auxiliary_bus.h >> +++ b/include/linux/auxiliary_bus.h >> @@ -58,6 +58,9 @@ >> * in >> * @name: Match name found by the auxiliary device driver, >> * @id: unique identitier if multiple devices of the same name are >> exported, >> + * @irqs: irqs xarray contains irq indices which are used by the device, >> + * @groups: first group is for irqs sysfs directory; it is a NULL >> terminated >> + * array, >> * >> * An auxiliary_device represents a part of its parent device's >> functionality. >> * It is given a name that, combined with the registering drivers >> @@ -138,6 +141,8 @@ >> struct auxiliary_device { >> struct device dev; >> const char *name; >> + struct xarray irqs; >> + const struct attribute_group *groups[2]; >> u32 id; >> }; >> >> @@ -209,8 +214,19 @@ static inline struct auxiliary_driver >> *to_auxiliary_drv(struct device_driver *dr >> } >> >> int auxiliary_device_init(struct auxiliary_device *auxdev); >> -int __auxiliary_device_add(struct auxiliary_device *auxdev, const >> char *modname); >> -#define auxiliary_device_add(auxdev) __auxiliary_device_add(auxdev, >> KBUILD_MODNAME) >> +int __auxiliary_device_add(struct auxiliary_device *auxdev, const >> char *modname, >> + bool irqs_sysfs_enable); >> +#define auxiliary_device_add(auxdev) __auxiliary_device_add(auxdev, >> KBUILD_MODNAME, false) >> +#define auxiliary_device_add_with_irqs(auxdev) \ >> + __auxiliary_device_add(auxdev, KBUILD_MODNAME, true) >> + >> +#ifdef CONFIG_SYSFS >> +int auxiliary_device_sysfs_irq_add(struct auxiliary_device *auxdev, >> int irq); >> +void auxiliary_device_sysfs_irq_remove(struct auxiliary_device >> *auxdev, int irq); >> +#else /* CONFIG_SYSFS */ >> +int auxiliary_device_sysfs_irq_add(struct auxiliary_device *auxdev, >> int irq) {return 0; } >> +void auxiliary_device_sysfs_irq_remove(struct auxiliary_device >> *auxdev, int irq) {} >> +#endif >> >> static inline void auxiliary_device_uninit(struct auxiliary_device >> *auxdev) >> { >
diff --git a/Documentation/ABI/testing/sysfs-bus-auxiliary b/Documentation/ABI/testing/sysfs-bus-auxiliary new file mode 100644 index 000000000000..3b8299d49d9e --- /dev/null +++ b/Documentation/ABI/testing/sysfs-bus-auxiliary @@ -0,0 +1,14 @@ +What: /sys/bus/auxiliary/devices/.../irqs/ +Date: April, 2024 +Contact: Shay Drory <shayd@nvidia.com> +Description: + The /sys/devices/.../irqs directory contains a variable set of + files, with each file is named as irq number similar to PCI PF + or VF's irq number located in msi_irqs directory. + +What: /sys/bus/auxiliary/devices/.../irqs/<N> +Date: April, 2024 +Contact: Shay Drory <shayd@nvidia.com> +Description: + auxiliary devices can share IRQs. This attribute indicates if + the irq is shared with other SFs or exclusively used by the SF. diff --git a/drivers/base/auxiliary.c b/drivers/base/auxiliary.c index d3a2c40c2f12..43d12a147f1f 100644 --- a/drivers/base/auxiliary.c +++ b/drivers/base/auxiliary.c @@ -158,6 +158,164 @@ * }; */ +#ifdef CONFIG_SYSFS +/* Xarray of irqs to determine if irq is exclusive or shared. */ +static DEFINE_XARRAY(irqs); +/* Protects insertions into the irtqs xarray. */ +static DEFINE_MUTEX(irqs_lock); + +struct auxiliary_irq_info { + struct device_attribute sysfs_attr; + int irq; +}; + +static struct attribute *auxiliary_irq_attrs[] = { + NULL +}; + +static const struct attribute_group auxiliary_irqs_group = { + .name = "irqs", + .attrs = auxiliary_irq_attrs, +}; + +/* Auxiliary devices can share IRQs. Expose to user whether the provided IRQ is + * shared or exclusive. + */ +static ssize_t auxiliary_irq_mode_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct auxiliary_irq_info *info = + container_of(attr, struct auxiliary_irq_info, sysfs_attr); + + if (refcount_read(xa_load(&irqs, info->irq)) > 1) + return sysfs_emit(buf, "%s\n", "shared"); + else + return sysfs_emit(buf, "%s\n", "exclusive"); +} + +static void auxiliary_irq_destroy(int irq) +{ + refcount_t *ref; + + xa_lock(&irqs); + ref = xa_load(&irqs, irq); + if (refcount_dec_and_test(ref)) { + __xa_erase(&irqs, irq); + kfree(ref); + } + xa_unlock(&irqs); +} + +static int auxiliary_irq_create(int irq) +{ + refcount_t *ref; + int ret = 0; + + mutex_lock(&irqs_lock); + ref = xa_load(&irqs, irq); + if (ref && refcount_inc_not_zero(ref)) + goto out; + + ref = kzalloc(sizeof(*ref), GFP_KERNEL); + if (!ref) { + ret = -ENOMEM; + goto out; + } + + refcount_set(ref, 1); + ret = xa_insert(&irqs, irq, ref, GFP_KERNEL); + if (ret) + kfree(ref); + +out: + mutex_unlock(&irqs_lock); + return ret; +} + +/** + * auxiliary_device_sysfs_irq_add - add a sysfs entry for the given IRQ + * @auxdev: auxiliary bus device to add the sysfs entry. + * @irq: The associated Linux interrupt number. + * + * This function should be called after auxiliary device have successfully + * received the irq. + * + * Return: zero on success or an error code on failure. + */ +int auxiliary_device_sysfs_irq_add(struct auxiliary_device *auxdev, int irq) +{ + struct device *dev = &auxdev->dev; + struct auxiliary_irq_info *info; + int ret; + + ret = auxiliary_irq_create(irq); + if (ret) + return ret; + + info = kzalloc(sizeof(*info), GFP_KERNEL); + if (!info) { + ret = -ENOMEM; + goto info_err; + } + + sysfs_attr_init(&info->sysfs_attr.attr); + info->sysfs_attr.attr.name = kasprintf(GFP_KERNEL, "%d", irq); + if (!info->sysfs_attr.attr.name) { + ret = -ENOMEM; + goto name_err; + } + info->irq = irq; + info->sysfs_attr.attr.mode = 0444; + info->sysfs_attr.show = auxiliary_irq_mode_show; + + ret = xa_insert(&auxdev->irqs, irq, info, GFP_KERNEL); + if (ret) + goto auxdev_xa_err; + + ret = sysfs_add_file_to_group(&dev->kobj, &info->sysfs_attr.attr, + auxiliary_irqs_group.name); + if (ret) + goto sysfs_add_err; + + return 0; + +sysfs_add_err: + xa_erase(&auxdev->irqs, irq); +auxdev_xa_err: + kfree(info->sysfs_attr.attr.name); +name_err: + kfree(info); +info_err: + auxiliary_irq_destroy(irq); + return ret; +} +EXPORT_SYMBOL_GPL(auxiliary_device_sysfs_irq_add); + +/** + * auxiliary_device_sysfs_irq_remove - remove a sysfs entry for the given IRQ + * @auxdev: auxiliary bus device to add the sysfs entry. + * @irq: the IRQ to remove. + * + * This function should be called to remove an IRQ sysfs entry. + */ +void auxiliary_device_sysfs_irq_remove(struct auxiliary_device *auxdev, int irq) +{ + struct auxiliary_irq_info *info = xa_load(&auxdev->irqs, irq); + struct device *dev = &auxdev->dev; + + if (WARN_ON(!info)) + return; + + sysfs_remove_file_from_group(&dev->kobj, &info->sysfs_attr.attr, + auxiliary_irqs_group.name); + xa_erase(&auxdev->irqs, irq); + kfree(info->sysfs_attr.attr.name); + kfree(info); + auxiliary_irq_destroy(irq); +} +EXPORT_SYMBOL_GPL(auxiliary_device_sysfs_irq_remove); +#endif + static const struct auxiliary_device_id *auxiliary_match_id(const struct auxiliary_device_id *id, const struct auxiliary_device *auxdev) { @@ -295,6 +453,7 @@ EXPORT_SYMBOL_GPL(auxiliary_device_init); * __auxiliary_device_add - add an auxiliary bus device * @auxdev: auxiliary bus device to add to the bus * @modname: name of the parent device's driver module + * @irqs_sysfs_enable: whether to enable IRQs sysfs * * This is the third step in the three-step process to register an * auxiliary_device. @@ -310,7 +469,8 @@ EXPORT_SYMBOL_GPL(auxiliary_device_init); * parameter. Only if a user requires a custom name would this version be * called directly. */ -int __auxiliary_device_add(struct auxiliary_device *auxdev, const char *modname) +int __auxiliary_device_add(struct auxiliary_device *auxdev, const char *modname, + bool irqs_sysfs_enable) { struct device *dev = &auxdev->dev; int ret; @@ -325,6 +485,11 @@ int __auxiliary_device_add(struct auxiliary_device *auxdev, const char *modname) dev_err(dev, "auxiliary device dev_set_name failed: %d\n", ret); return ret; } + if (irqs_sysfs_enable) { + auxdev->groups[0] = &auxiliary_irqs_group; + xa_init(&auxdev->irqs); + dev->groups = auxdev->groups; + } ret = device_add(dev); if (ret) diff --git a/include/linux/auxiliary_bus.h b/include/linux/auxiliary_bus.h index de21d9d24a95..fe2c438c0217 100644 --- a/include/linux/auxiliary_bus.h +++ b/include/linux/auxiliary_bus.h @@ -58,6 +58,9 @@ * in * @name: Match name found by the auxiliary device driver, * @id: unique identitier if multiple devices of the same name are exported, + * @irqs: irqs xarray contains irq indices which are used by the device, + * @groups: first group is for irqs sysfs directory; it is a NULL terminated + * array, * * An auxiliary_device represents a part of its parent device's functionality. * It is given a name that, combined with the registering drivers @@ -138,6 +141,8 @@ struct auxiliary_device { struct device dev; const char *name; + struct xarray irqs; + const struct attribute_group *groups[2]; u32 id; }; @@ -209,8 +214,19 @@ static inline struct auxiliary_driver *to_auxiliary_drv(struct device_driver *dr } int auxiliary_device_init(struct auxiliary_device *auxdev); -int __auxiliary_device_add(struct auxiliary_device *auxdev, const char *modname); -#define auxiliary_device_add(auxdev) __auxiliary_device_add(auxdev, KBUILD_MODNAME) +int __auxiliary_device_add(struct auxiliary_device *auxdev, const char *modname, + bool irqs_sysfs_enable); +#define auxiliary_device_add(auxdev) __auxiliary_device_add(auxdev, KBUILD_MODNAME, false) +#define auxiliary_device_add_with_irqs(auxdev) \ + __auxiliary_device_add(auxdev, KBUILD_MODNAME, true) + +#ifdef CONFIG_SYSFS +int auxiliary_device_sysfs_irq_add(struct auxiliary_device *auxdev, int irq); +void auxiliary_device_sysfs_irq_remove(struct auxiliary_device *auxdev, int irq); +#else /* CONFIG_SYSFS */ +int auxiliary_device_sysfs_irq_add(struct auxiliary_device *auxdev, int irq) {return 0; } +void auxiliary_device_sysfs_irq_remove(struct auxiliary_device *auxdev, int irq) {} +#endif static inline void auxiliary_device_uninit(struct auxiliary_device *auxdev) {