| Message ID | 20240429104633.11060-3-ilpo.jarvinen@linux.intel.com (mailing list archive) |
|---|---|
| State | Changes Requested |
| Headers | show |
| Series | PCI: Add generic Conf Type 0/1 helpers | expand |
On Monday 29 April 2024 13:46:25 Ilpo Järvinen wrote: > Many places in arch and PCI controller code need to calculate PCI > Configuration Space Addresses for Type 0/1 accesses. There are small > variations between archs when it comes to bits outside of [10:2] (Type > 0) and [24:2] (Type 1) but the basic calculation can still be > generalized. > > drivers/pci/pci.h has PCI_CONF1{,_EXT}_ADDRESS() but due to their > location the use is limited to PCI subsys and the also always enable > PCI_CONF1_ENABLE which is not what all the callers want. > > Add generic pci_conf{0,1}_addr() and pci_conf1_ext_addr() helpers into > include/linux/pci.h which can be reused by various parts of the kernel > that have to calculate PCI Conf Type 0/1 addresses. > > The PCI_CONF* defines are needed by the new helpers so move also them > to include/linux/pci.h. The new helpers use true bitmasks and > FIELD_PREP() instead of open coded masking and shifting so adjust > PCI_CONF* definitions to match that. I introduced these PCI_CONF* macros years ago. At that time I studied more sources and drivers what they use. To make it clear I will first try to explain few things. Hopefully I do not write mistakes here, it is longer time. Configuration mechanism is a SW API which allows access config space. Configuration access command is on-wire "format" which allows HW bus to access config space. There are two standardized configuration mechanisms #1 and #2. #2 was removed in PCI 3.0 spec and is out of scope (there are no macros for them). #1 uses pair of I/O registers (on x86 they are CF8 and CFC; on ARM they are whatever vendor invented). There is an extension of #1 which uses few reserved bits to describe config space registers above 255. Then there is standardized PCIe ECAM. And then there are lot of proprietary vendor specific ways. Proprietary vendor way can be for example composing PCIe packet manually or composing configuration access command manually. There are two configuration space access commands: type 0 and type 1. It is required to issue correct command type based on topology of the endpoint device. When mechanism #1 is used then it is HW who generate these commands and it takes care to correctly choose type 0 or 1. But if some vendor specific mechanism is used which requires SW to compose access command manually then SW is responsible for correct choose. In your change you have mixed configuration mechanism #1 together with configuration command for type 0. This is really a problem as it makes the code harder to understand and even hard to figure out how to write a new driver (e.g. how to compose configuration command for type 1?). Also it took me a little bit more time to understand your change. Format of command for type 1 and format of configuration mechanism #1 really looks very similar. So there can be a confusion. Bit 31 is a key there. I understand that you want to unify drivers, so I would suggest to not change existing macros for configuration mechanism #1 and #1-ext. And rather create new macros (or functions) for composing configuration commands. This makes it explicitly clear that particular PCI or PCIe controller HW requires SW driver to compose configuration command (type 0 and 1). Or that HW requires from SW to compose format of configuration mechanism #1 (or #1-ext). Also it would make pci controller drivers more readable as from the macro/function it would be easy to understand what it is doing. Also important is that if you are in pci controller driver composing command for type 0 then with very high probability the HW is designed in a way that it requires from SW to also compose command type 1. And it would be an mistake if the driver compose only type 0 or type 1. I remember from the past an issue: why endpoint PCIe card is working, but it is not working if it is connected behind PCIe switch. (mistake was: HW was always sending type 0 command due to SW issue) > Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> > --- > drivers/pci/pci.h | 43 ++--------------------- > include/linux/pci.h | 85 +++++++++++++++++++++++++++++++++++++++++++++ > 2 files changed, 88 insertions(+), 40 deletions(-) > > diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h > index 17fed1846847..cf0530a60105 100644 > --- a/drivers/pci/pci.h > +++ b/drivers/pci/pci.h > @@ -833,49 +833,12 @@ struct pci_devres { > > struct pci_devres *find_pci_dr(struct pci_dev *pdev); > > -/* > - * Config Address for PCI Configuration Mechanism #1 > - * > - * See PCI Local Bus Specification, Revision 3.0, > - * Section 3.2.2.3.2, Figure 3-2, p. 50. > - */ > - > -#define PCI_CONF1_BUS_SHIFT 16 /* Bus number */ > -#define PCI_CONF1_DEV_SHIFT 11 /* Device number */ > -#define PCI_CONF1_FUNC_SHIFT 8 /* Function number */ > - > -#define PCI_CONF1_BUS_MASK 0xff > -#define PCI_CONF1_DEV_MASK 0x1f > -#define PCI_CONF1_FUNC_MASK 0x7 > -#define PCI_CONF1_REG_MASK 0xfc /* Limit aligned offset to a maximum of 256B */ > - > -#define PCI_CONF1_ENABLE BIT(31) > -#define PCI_CONF1_BUS(x) (((x) & PCI_CONF1_BUS_MASK) << PCI_CONF1_BUS_SHIFT) > -#define PCI_CONF1_DEV(x) (((x) & PCI_CONF1_DEV_MASK) << PCI_CONF1_DEV_SHIFT) > -#define PCI_CONF1_FUNC(x) (((x) & PCI_CONF1_FUNC_MASK) << PCI_CONF1_FUNC_SHIFT) > -#define PCI_CONF1_REG(x) ((x) & PCI_CONF1_REG_MASK) > - > #define PCI_CONF1_ADDRESS(bus, dev, func, reg) \ > (PCI_CONF1_ENABLE | \ > - PCI_CONF1_BUS(bus) | \ > - PCI_CONF1_DEV(dev) | \ > - PCI_CONF1_FUNC(func) | \ > - PCI_CONF1_REG(reg)) > - > -/* > - * Extension of PCI Config Address for accessing extended PCIe registers > - * > - * No standardized specification, but used on lot of non-ECAM-compliant ARM SoCs > - * or on AMD Barcelona and new CPUs. Reserved bits [27:24] of PCI Config Address > - * are used for specifying additional 4 high bits of PCI Express register. > - */ > - > -#define PCI_CONF1_EXT_REG_SHIFT 16 > -#define PCI_CONF1_EXT_REG_MASK 0xf00 > -#define PCI_CONF1_EXT_REG(x) (((x) & PCI_CONF1_EXT_REG_MASK) << PCI_CONF1_EXT_REG_SHIFT) > + pci_conf1_addr(bus, PCI_DEVFN(dev, func), reg & ~0x3U)) > > #define PCI_CONF1_EXT_ADDRESS(bus, dev, func, reg) \ > - (PCI_CONF1_ADDRESS(bus, dev, func, reg) | \ > - PCI_CONF1_EXT_REG(reg)) > + (PCI_CONF1_ENABLE | \ > + pci_conf1_ext_addr(bus, PCI_DEVFN(dev, func), reg & ~0x3U)) > > #endif /* DRIVERS_PCI_H */ > diff --git a/include/linux/pci.h b/include/linux/pci.h > index 16493426a04f..4c4e3bb52a0a 100644 > --- a/include/linux/pci.h > +++ b/include/linux/pci.h > @@ -26,6 +26,8 @@ > #include <linux/args.h> > #include <linux/mod_devicetable.h> > > +#include <linux/bits.h> > +#include <linux/bitfield.h> > #include <linux/types.h> > #include <linux/init.h> > #include <linux/ioport.h> > @@ -1183,6 +1185,89 @@ void pci_sort_breadthfirst(void); > #define dev_is_pci(d) ((d)->bus == &pci_bus_type) > #define dev_is_pf(d) ((dev_is_pci(d) ? to_pci_dev(d)->is_physfn : false)) > > +/* > + * Config Address for PCI Configuration Mechanism #0/1 > + * > + * See PCI Local Bus Specification, Revision 3.0, > + * Section 3.2.2.3.2, Figure 3-1 and 3-2, p. 48-50. > + */ > +#define PCI_CONF_REG 0x000000ffU /* common for Type 0/1 */ > +#define PCI_CONF_FUNC 0x00000700U /* common for Type 0/1 */ > +#define PCI_CONF1_DEV 0x0000f800U > +#define PCI_CONF1_BUS 0x00ff0000U > +#define PCI_CONF1_ENABLE BIT(31) > + > +/** > + * pci_conf0_addr - PCI Base Configuration Space address for Type 0 access > + * @devfn: Device and function numbers (device number will be ignored) > + * @reg: Base configuration space offset > + * > + * Calculates the PCI Configuration Space address for Type 0 accesses. > + * > + * Note: the caller is responsible for adding the bits outside of [10:0]. > + * > + * Return: Base Configuration Space address. > + */ > +static inline u32 pci_conf0_addr(u8 devfn, u8 reg) > +{ > + return FIELD_PREP(PCI_CONF_FUNC, PCI_FUNC(devfn)) | > + FIELD_PREP(PCI_CONF_REG, reg & ~3); > +} > + > +/** > + * pci_conf1_addr - PCI Base Configuration Space address for Type 1 access > + * @bus: Bus number of the device > + * @devfn: Device and function numbers > + * @reg: Base configuration space offset > + * @enable: Assert enable bit (bit 31) > + * > + * Calculates the PCI Base Configuration Space (first 256 bytes) address for > + * Type 1 accesses. > + * > + * Note: the caller is responsible for adding the bits outside of [24:2] > + * and enable bit. > + * > + * Return: PCI Base Configuration Space address. > + */ > +static inline u32 pci_conf1_addr(u8 bus, u8 devfn, u8 reg, bool enable) > +{ > + return (enable ? PCI_CONF1_ENABLE : 0) | > + FIELD_PREP(PCI_CONF1_BUS, bus) | > + FIELD_PREP(PCI_CONF1_DEV | PCI_CONF_FUNC, devfn) | > + FIELD_PREP(PCI_CONF_REG, reg & ~3); > +} > + > +/* > + * Extension of PCI Config Address for accessing extended PCIe registers > + * > + * No standardized specification, but used on lot of non-ECAM-compliant ARM SoCs > + * or on AMD Barcelona and new CPUs. Reserved bits [27:24] of PCI Config Address > + * are used for specifying additional 4 high bits of PCI Express register. > + */ > +#define PCI_CONF1_EXT_REG 0x0f000000UL > + > +/** > + * pci_conf1_ext_addr - PCI Configuration Space address for Type 1 access > + * @bus: Bus number of the device > + * @devfn: Device and function numbers > + * @reg: Base or Extended Configuration space offset > + * @enable: Assert enable bit (bit 31) > + * > + * Calculates the PCI Base and Extended (4096 bytes per PCI function) > + * Configuration Space address for Type 1 accesses. This function assumes > + * the Extended Conguration Space is using the reserved bits [27:24]. > + * > + * Note: the caller is responsible for adding the bits outside of [27:2] and > + * enable bit. > + * > + * Return: PCI Configuration Space address. > + */ > +static inline u32 pci_conf1_ext_addr(u8 bus, u8 devfn, u16 reg, bool enable) > +{ > + return FIELD_PREP(PCI_CONF1_EXT_REG, (reg & 0xf00) >> 8) | > + pci_conf1_addr(bus, devfn, reg & 0xff, enable); > +} > + > /* Generic PCI functions exported to card drivers */ > > u8 pci_bus_find_capability(struct pci_bus *bus, unsigned int devfn, int cap); > -- > 2.39.2 >
On Mon, 29 Apr 2024, Pali Rohár wrote: > On Monday 29 April 2024 13:46:25 Ilpo Järvinen wrote: > > Many places in arch and PCI controller code need to calculate PCI > > Configuration Space Addresses for Type 0/1 accesses. There are small > > variations between archs when it comes to bits outside of [10:2] (Type > > 0) and [24:2] (Type 1) but the basic calculation can still be > > generalized. > > > > drivers/pci/pci.h has PCI_CONF1{,_EXT}_ADDRESS() but due to their > > location the use is limited to PCI subsys and the also always enable > > PCI_CONF1_ENABLE which is not what all the callers want. > > > > Add generic pci_conf{0,1}_addr() and pci_conf1_ext_addr() helpers into > > include/linux/pci.h which can be reused by various parts of the kernel > > that have to calculate PCI Conf Type 0/1 addresses. > > > > The PCI_CONF* defines are needed by the new helpers so move also them > > to include/linux/pci.h. The new helpers use true bitmasks and > > FIELD_PREP() instead of open coded masking and shifting so adjust > > PCI_CONF* definitions to match that. > > I introduced these PCI_CONF* macros years ago. At that time I studied > more sources and drivers what they use. To make it clear I will first > try to explain few things. Hopefully I do not write mistakes here, it is > longer time. > > Configuration mechanism is a SW API which allows access config space. > Configuration access command is on-wire "format" which allows HW bus to > access config space. > > There are two standardized configuration mechanisms #1 and #2. #2 was > removed in PCI 3.0 spec and is out of scope (there are no macros for > them). #1 uses pair of I/O registers (on x86 they are CF8 and CFC; on > ARM they are whatever vendor invented). There is an extension of #1 > which uses few reserved bits to describe config space registers above > 255. Then there is standardized PCIe ECAM. And then there are lot of > proprietary vendor specific ways. Proprietary vendor way can be for > example composing PCIe packet manually or composing configuration access > command manually. > > There are two configuration space access commands: type 0 and type 1. > It is required to issue correct command type based on topology of the > endpoint device. When mechanism #1 is used then it is HW who generate > these commands and it takes care to correctly choose type 0 or 1. But if > some vendor specific mechanism is used which requires SW to compose > access command manually then SW is responsible for correct choose. > > > In your change you have mixed configuration mechanism #1 together with > configuration command for type 0. This is really a problem as it makes > the code harder to understand and even hard to figure out how to write a > new driver (e.g. how to compose configuration command for type 1?). > > Also it took me a little bit more time to understand your change. > Format of command for type 1 and format of configuration mechanism #1 > really looks very similar. So there can be a confusion. Bit 31 is a key > there. Thanks a lot for chimming in! In practice, the calculation done is very similar for many archs. I initially was planning to only convert things in drivers/pci/pci.h to FIELD_PREP() and be done with it, but then I came across this (a rough and incomplete list): $ git grep -e 'bus.*<<.*16' -B1 -A3 arch So I ended up extending the scope and trying to find a common ground, which was to make functions into include/linux/pci.h to cover the main calculations. This should explain the placement which you asked in the other reply. I don't think drivers/pci/pci.h is helpful when arch/ code has this very calculation repeated n times. To me it looked obvious that those calculations relate to what is described in PCI Local Bus Spec r3.0 sec 3.2.2.3.1 (some even explicitly indicate type 0 or type 1). I still had to find some name for the functions and didn't see any reason why I couldn't just use type 0 and type 1 as in that spec. To me, it's hardly an accident that mechanism #1 fields are 1:1 copy of type 1 calculation but I think I can still see it also from your viewpoint. I perhaps just look this from more practical standpoint than you (no offence meant in any way). I don't know if the vendor specific part is even relevant to this series or if you just noted it for completeness. I'm looking for common ground here and if vendor's copying the existing format where they could have invented entirely custom formatting, but is that good enough reason to name the functions differently? Or falsely claim it's vendor specific when it's obviously nothing but copying the existing way (i.e., the fields from type 1)? Perhaps the main wrong here was in the naming step or in the terminology I used in the commit messages, or do you think those calcs done under arch/ do not related to type 0/1 in any way (despite being 1:1 copy!)? Lets take a look at a real function in kernel (this is not the only example, there are more similar ones under arch/, e.g., in arch/alpha/kernel/core_lca.c): static u32 ixp4xx_config_addr(u8 bus_num, u16 devfn, int where) { /* Root bus is always 0 in this hardware */ if (bus_num == 0) { /* type 0 */ return (PCI_CONF1_ADDRESS(0, 0, PCI_FUNC(devfn), where) & ~PCI_CONF1_ENABLE) | BIT(32-PCI_SLOT(devfn)); } else { /* type 1 */ return (PCI_CONF1_ADDRESS(bus_num, PCI_SLOT(devfn), PCI_FUNC(devfn), where) & ~PCI_CONF1_ENABLE) | 1; } } So is this "mixing" #1 and type 0 together? Or how you categorize this? I suspect, by your definition, it's actually abusing what is meant for mechanism #1 to calculate type 1 (obviously because the main calculation is undeniably very much the same :-))? > I understand that you want to unify drivers, so I would suggest to not > change existing macros for configuration mechanism #1 and #1-ext. > And rather create new macros (or functions) for composing configuration > commands. And what about code under arch/? I see bits 24-27 and bit #31 being enabled there as well. Or do those categorize as "vendor specific" methods so nothing can be done to them?!? > This makes it explicitly clear that particular PCI or PCIe controller HW > requires SW driver to compose configuration command (type 0 and 1). Or > that HW requires from SW to compose format of configuration mechanism #1 > (or #1-ext). Also it would make pci controller drivers more readable as > from the macro/function it would be easy to understand what it is doing. So does this effectively boil down to instead of having a boolean enable parameter for bit #31, I should create two functions with different names (well, reusing the existing ones perhaps for one of them but the same idea)? Couldn't I just name the function and that enable parameter differently and/or document things better without having two functions? It's not like these these two things came to be mostly the same by chance, they're very much related. > Also important is that if you are in pci controller driver composing > command for type 0 then with very high probability the HW is designed in > a way that it requires from SW to also compose command type 1. And it > would be an mistake if the driver compose only type 0 or type 1. I > remember from the past an issue: why endpoint PCIe card is working, but > it is not working if it is connected behind PCIe switch. (mistake was: > HW was always sending type 0 command due to SW issue) Do you think this would warrant something that actually combines the type 0/1 calculations inside? Just asking your opinion, I'm not sure how easy the arch code would be to adapt such a bigger change because of the variations.
On Tuesday 30 April 2024 13:21:23 Ilpo Järvinen wrote: > On Mon, 29 Apr 2024, Pali Rohár wrote: > > > On Monday 29 April 2024 13:46:25 Ilpo Järvinen wrote: > > > Many places in arch and PCI controller code need to calculate PCI > > > Configuration Space Addresses for Type 0/1 accesses. There are small > > > variations between archs when it comes to bits outside of [10:2] (Type > > > 0) and [24:2] (Type 1) but the basic calculation can still be > > > generalized. > > > > > > drivers/pci/pci.h has PCI_CONF1{,_EXT}_ADDRESS() but due to their > > > location the use is limited to PCI subsys and the also always enable > > > PCI_CONF1_ENABLE which is not what all the callers want. > > > > > > Add generic pci_conf{0,1}_addr() and pci_conf1_ext_addr() helpers into > > > include/linux/pci.h which can be reused by various parts of the kernel > > > that have to calculate PCI Conf Type 0/1 addresses. > > > > > > The PCI_CONF* defines are needed by the new helpers so move also them > > > to include/linux/pci.h. The new helpers use true bitmasks and > > > FIELD_PREP() instead of open coded masking and shifting so adjust > > > PCI_CONF* definitions to match that. > > > > I introduced these PCI_CONF* macros years ago. At that time I studied > > more sources and drivers what they use. To make it clear I will first > > try to explain few things. Hopefully I do not write mistakes here, it is > > longer time. > > > > Configuration mechanism is a SW API which allows access config space. > > Configuration access command is on-wire "format" which allows HW bus to > > access config space. > > > > There are two standardized configuration mechanisms #1 and #2. #2 was > > removed in PCI 3.0 spec and is out of scope (there are no macros for > > them). #1 uses pair of I/O registers (on x86 they are CF8 and CFC; on > > ARM they are whatever vendor invented). There is an extension of #1 > > which uses few reserved bits to describe config space registers above > > 255. Then there is standardized PCIe ECAM. And then there are lot of > > proprietary vendor specific ways. Proprietary vendor way can be for > > example composing PCIe packet manually or composing configuration access > > command manually. > > > > There are two configuration space access commands: type 0 and type 1. > > It is required to issue correct command type based on topology of the > > endpoint device. When mechanism #1 is used then it is HW who generate > > these commands and it takes care to correctly choose type 0 or 1. But if > > some vendor specific mechanism is used which requires SW to compose > > access command manually then SW is responsible for correct choose. > > > > > > In your change you have mixed configuration mechanism #1 together with > > configuration command for type 0. This is really a problem as it makes > > the code harder to understand and even hard to figure out how to write a > > new driver (e.g. how to compose configuration command for type 1?). > > > > Also it took me a little bit more time to understand your change. > > Format of command for type 1 and format of configuration mechanism #1 > > really looks very similar. So there can be a confusion. Bit 31 is a key > > there. > > Thanks a lot for chimming in! > > In practice, the calculation done is very similar for many archs. I > initially was planning to only convert things in drivers/pci/pci.h to > FIELD_PREP() and be done with it, but then I came across this (a rough and > incomplete list): > > $ git grep -e 'bus.*<<.*16' -B1 -A3 arch > > So I ended up extending the scope and trying to find a common ground, > which was to make functions into include/linux/pci.h to cover the main > calculations. This should explain the placement which you asked in the > other reply. I don't think drivers/pci/pci.h is helpful when arch/ code > has this very calculation repeated n times. I know. In past I have called similar git grep commands and at that time I also come to similar conclusion and that is why I originally also suggested to provide macros in include/linux/*.h > To me it looked obvious that those calculations relate to what is > described in PCI Local Bus Spec r3.0 sec 3.2.2.3.1 (some even explicitly > indicate type 0 or type 1). It is important to see that there are sections 3.2.2.3.1. and 3.2.2.3.2. which describes different things. > I still had to find some name for the functions and didn't see any reason > why I couldn't just use type 0 and type 1 as in that spec. To me, it's > hardly an accident that mechanism #1 fields are 1:1 copy of type 1 > calculation but I think I can still see it also from your viewpoint. I > perhaps just look this from more practical standpoint than you (no offence > meant in any way). The best names of the functions is to use terminology from the standards (instead of inventing new custom terminology). Because standards are "standards", so anybody from the specific industry would be able to better understand the code. > I don't know if the vendor specific part is even relevant to this series > or if you just noted it for completeness. For completeness because it is very common and it is important to know about it. Also mechanism #1-ext is vendor specific but somehow used by lot of vendors. > I'm looking for common ground > here and if vendor's copying the existing format where they could have > invented entirely custom formatting, but is that good enough reason to > name the functions differently? Or falsely claim it's vendor specific when > it's obviously nothing but copying the existing way (i.e., the fields from > type 1)? "type 1 command" has exact definition and well-defined usage. It is used for communication with device behind the PCI-to-PCI bridge. PCIe endpoint device has to reply with status unsupported request. Using "type 1 command" in function name which sends arbitrary command (including type 0; e.g. for filling mechanism #1 format) is misleading and would open questions like: why is using function "type 1 command" for sending "type 0 command"? Reason that some bits in the formats have same meaning does not mean to use wrong names. > Perhaps the main wrong here was in the naming step or in the terminology > I used in the commit messages, or do you think those calcs done under > arch/ do not related to type 0/1 in any way (despite being 1:1 copy!)? For example what is wrong is this documentation: "pci_conf1_ext_addr - PCI Configuration Space address for Type 1 access" This function does not generate Type 1 command as it does not sets BIT(0) to 1. It let BIT(0) as 0, which means that it generates Type 0 command. It needs to be properly investigated if the HW which targets code in arch/ generates type 0/1 commands in SW or in HW. > Lets take a look at a real function in kernel (this is not the only > example, there are more similar ones under arch/, e.g., in > arch/alpha/kernel/core_lca.c): > > static u32 ixp4xx_config_addr(u8 bus_num, u16 devfn, int where) > { > /* Root bus is always 0 in this hardware */ > if (bus_num == 0) { > /* type 0 */ > return (PCI_CONF1_ADDRESS(0, 0, PCI_FUNC(devfn), where) & > ~PCI_CONF1_ENABLE) | BIT(32-PCI_SLOT(devfn)); > } else { > /* type 1 */ > return (PCI_CONF1_ADDRESS(bus_num, PCI_SLOT(devfn), > PCI_FUNC(devfn), where) & > ~PCI_CONF1_ENABLE) | 1; > } > } > > So is this "mixing" #1 and type 0 together? Or how you categorize this? > I suspect, by your definition, it's actually abusing what is meant for > mechanism #1 to calculate type 1 (obviously because the main calculation > is undeniably very much the same :-))? So first is: Is this code correct? Are not you just trying to create some new functions and abstractions for something which is wrong? As I wrote in previous email, I have an experience with wrongly written SW which incorrectly generated config requests. The example is for some alpha HW, I guess something not commonly used, so if there is a bug in that code, there is very little chance that somebody will report it. The worst thing which can happen is to design some abstraction based on wrong assumptions (or wrong code). As it would lead to new bugs in newly written code. New designed abstraction should prevent it. > > I understand that you want to unify drivers, so I would suggest to not > > change existing macros for configuration mechanism #1 and #1-ext. > > And rather create new macros (or functions) for composing configuration > > commands. > > And what about code under arch/? I see bits 24-27 and bit #31 being > enabled there as well. Or do those categorize as "vendor specific" methods > so nothing can be done to them?!? Do you mean above ixp4xx_config_addr? The correct answer is to look into HW documentation. It is hard to predict but I guess that it is constructions of the format for raw commands. When bus_num is not 0 then is is type 1 command. And if bus_num is 0 then bits 11-31 are reserved (in HW can be hardwired to zero) so BIT(32-PCI_SLOT(devfn)) can do nothing. But this is only my assumption and reality can be different. It is really common do to such mistake if you do not read documentation properly or have assumptions from wrong input information (like mixing formats of type 0 command and mech #1). > > This makes it explicitly clear that particular PCI or PCIe controller HW > > requires SW driver to compose configuration command (type 0 and 1). Or > > that HW requires from SW to compose format of configuration mechanism #1 > > (or #1-ext). Also it would make pci controller drivers more readable as > > from the macro/function it would be easy to understand what it is doing. > > So does this effectively boil down to instead of having a boolean enable > parameter for bit #31, I should create two functions with different names > (well, reusing the existing ones perhaps for one of them but the same > idea)? Couldn't I just name the function and that enable parameter > differently and/or document things better without having two functions? > It's not like these these two things came to be mostly the same by chance, > they're very much related. What I'm trying to say. If your HW requires from SW to generate RAW commands, then driver has to implement logic for generating _both_ type0 and type1 commands and also needs to have logic to correctly choose which command to generate. If you are developer of the driver for such HW then the worst what can happen is that you forgot to add logic for generating type 0 commands. And finding such bug can be really hard for somebody who does not know difference between mech #1 and type 1. mech #1 is something which generates _both_ type 0 and type 1. When working with mech #1 I do not see any reason why driver would want to disable access (by setting enable bit to 0). So I think that helper kernel function should always set enable bit to 1. If there is a reason to disable access then there could be another function (without arguments) which will do it. It is wrong to call function with enable=0 and expecting that it would _enable_ access to HW. So I'm suggesting to: 1) let macros for mech #1 and #1-ext there as is 2) introduce new macros (functions, abstractions, ...) for generating configuration commands type 0 and type 1. It can be either one macro which takes an argument type of the command (0 or 1, ideally asserted that no other value is valid). Or it can be two commands, one for type 0 and one for type 1. Type of the command is defined by BIT(1) and BIT(0). Personally I'm for having two macros, one for type 0 and one for type 1. And that is because type 0 command has only function number and register number. Whether type 1 has additionally bus number and device number. So it would be easier to look if correct parameters are passed for particular format. So if author of driver figure out that has to write a code which needs to issue type 1 command, would be also forced to write a code for issuing type 0 command. And for other people it would be easy to review if driver has code for both type 0 and type 1. For example having macros: PCI_CONF1_ADDRESS(bus, dev, func, reg) PCI_CONF1_EXT_ADDRESS(bus, dev, func, reg) PCI_CONF_COMMAND_TYPE0_ADDRESS(func, reg) PCI_CONF_COMMAND_TYPE1_ADDRESS(bus, dev, func, reg) (and maybe PCI_CONF_COMMAND_TYPE1_EXT_ADDRESS if there is such vendor ext) I'm not sure if the PCI_CONF_COMMAND_TYPEX_ADDRESS is the best name, but I wrote something as an example. 3) switch drivers to use new macros. And figure out if the drivers are not buggy and are trying to sets wire-0 bits... > > Also important is that if you are in pci controller driver composing > > command for type 0 then with very high probability the HW is designed in > > a way that it requires from SW to also compose command type 1. And it > > would be an mistake if the driver compose only type 0 or type 1. I > > remember from the past an issue: why endpoint PCIe card is working, but > > it is not working if it is connected behind PCIe switch. (mistake was: > > HW was always sending type 0 command due to SW issue) > > Do you think this would warrant something that actually combines the > type 0/1 calculations inside? Just asking your opinion, I'm not sure how > easy the arch code would be to adapt such a bigger change because of the > variations. I do not know how hard is to refactor all arch/ code. I guess there are lot of code from history, which may be wrong/buggy and for which nobody is doing periodic testing. Nowadays new pci controller drivers are in the drivers/pci/ subdir. So maybe the best option is to let arch/ pci code as is? > -- > i. > > > > Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> > > > --- > > > drivers/pci/pci.h | 43 ++--------------------- > > > include/linux/pci.h | 85 +++++++++++++++++++++++++++++++++++++++++++++ > > > 2 files changed, 88 insertions(+), 40 deletions(-) > > > > > > diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h > > > index 17fed1846847..cf0530a60105 100644 > > > --- a/drivers/pci/pci.h > > > +++ b/drivers/pci/pci.h > > > @@ -833,49 +833,12 @@ struct pci_devres { > > > > > > struct pci_devres *find_pci_dr(struct pci_dev *pdev); > > > > > > -/* > > > - * Config Address for PCI Configuration Mechanism #1 > > > - * > > > - * See PCI Local Bus Specification, Revision 3.0, > > > - * Section 3.2.2.3.2, Figure 3-2, p. 50. > > > - */ > > > - > > > -#define PCI_CONF1_BUS_SHIFT 16 /* Bus number */ > > > -#define PCI_CONF1_DEV_SHIFT 11 /* Device number */ > > > -#define PCI_CONF1_FUNC_SHIFT 8 /* Function number */ > > > - > > > -#define PCI_CONF1_BUS_MASK 0xff > > > -#define PCI_CONF1_DEV_MASK 0x1f > > > -#define PCI_CONF1_FUNC_MASK 0x7 > > > -#define PCI_CONF1_REG_MASK 0xfc /* Limit aligned offset to a maximum of 256B */ > > > - > > > -#define PCI_CONF1_ENABLE BIT(31) > > > -#define PCI_CONF1_BUS(x) (((x) & PCI_CONF1_BUS_MASK) << PCI_CONF1_BUS_SHIFT) > > > -#define PCI_CONF1_DEV(x) (((x) & PCI_CONF1_DEV_MASK) << PCI_CONF1_DEV_SHIFT) > > > -#define PCI_CONF1_FUNC(x) (((x) & PCI_CONF1_FUNC_MASK) << PCI_CONF1_FUNC_SHIFT) > > > -#define PCI_CONF1_REG(x) ((x) & PCI_CONF1_REG_MASK) > > > - > > > #define PCI_CONF1_ADDRESS(bus, dev, func, reg) \ > > > (PCI_CONF1_ENABLE | \ > > > - PCI_CONF1_BUS(bus) | \ > > > - PCI_CONF1_DEV(dev) | \ > > > - PCI_CONF1_FUNC(func) | \ > > > - PCI_CONF1_REG(reg)) > > > - > > > -/* > > > - * Extension of PCI Config Address for accessing extended PCIe registers > > > - * > > > - * No standardized specification, but used on lot of non-ECAM-compliant ARM SoCs > > > - * or on AMD Barcelona and new CPUs. Reserved bits [27:24] of PCI Config Address > > > - * are used for specifying additional 4 high bits of PCI Express register. > > > - */ > > > - > > > -#define PCI_CONF1_EXT_REG_SHIFT 16 > > > -#define PCI_CONF1_EXT_REG_MASK 0xf00 > > > -#define PCI_CONF1_EXT_REG(x) (((x) & PCI_CONF1_EXT_REG_MASK) << PCI_CONF1_EXT_REG_SHIFT) > > > + pci_conf1_addr(bus, PCI_DEVFN(dev, func), reg & ~0x3U)) > > > > > > #define PCI_CONF1_EXT_ADDRESS(bus, dev, func, reg) \ > > > - (PCI_CONF1_ADDRESS(bus, dev, func, reg) | \ > > > - PCI_CONF1_EXT_REG(reg)) > > > + (PCI_CONF1_ENABLE | \ > > > + pci_conf1_ext_addr(bus, PCI_DEVFN(dev, func), reg & ~0x3U)) > > > > > > #endif /* DRIVERS_PCI_H */ > > > diff --git a/include/linux/pci.h b/include/linux/pci.h > > > index 16493426a04f..4c4e3bb52a0a 100644 > > > --- a/include/linux/pci.h > > > +++ b/include/linux/pci.h > > > @@ -26,6 +26,8 @@ > > > #include <linux/args.h> > > > #include <linux/mod_devicetable.h> > > > > > > +#include <linux/bits.h> > > > +#include <linux/bitfield.h> > > > #include <linux/types.h> > > > #include <linux/init.h> > > > #include <linux/ioport.h> > > > @@ -1183,6 +1185,89 @@ void pci_sort_breadthfirst(void); > > > #define dev_is_pci(d) ((d)->bus == &pci_bus_type) > > > #define dev_is_pf(d) ((dev_is_pci(d) ? to_pci_dev(d)->is_physfn : false)) > > > > > > +/* > > > + * Config Address for PCI Configuration Mechanism #0/1 > > > + * > > > + * See PCI Local Bus Specification, Revision 3.0, > > > + * Section 3.2.2.3.2, Figure 3-1 and 3-2, p. 48-50. > > > + */ > > > +#define PCI_CONF_REG 0x000000ffU /* common for Type 0/1 */ > > > +#define PCI_CONF_FUNC 0x00000700U /* common for Type 0/1 */ > > > +#define PCI_CONF1_DEV 0x0000f800U > > > +#define PCI_CONF1_BUS 0x00ff0000U > > > +#define PCI_CONF1_ENABLE BIT(31) > > > + > > > +/** > > > + * pci_conf0_addr - PCI Base Configuration Space address for Type 0 access > > > + * @devfn: Device and function numbers (device number will be ignored) > > > + * @reg: Base configuration space offset > > > + * > > > + * Calculates the PCI Configuration Space address for Type 0 accesses. > > > + * > > > + * Note: the caller is responsible for adding the bits outside of [10:0]. > > > + * > > > + * Return: Base Configuration Space address. > > > + */ > > > +static inline u32 pci_conf0_addr(u8 devfn, u8 reg) > > > +{ > > > + return FIELD_PREP(PCI_CONF_FUNC, PCI_FUNC(devfn)) | > > > + FIELD_PREP(PCI_CONF_REG, reg & ~3); > > > +} > > > + > > > +/** > > > + * pci_conf1_addr - PCI Base Configuration Space address for Type 1 access > > > + * @bus: Bus number of the device > > > + * @devfn: Device and function numbers > > > + * @reg: Base configuration space offset > > > + * @enable: Assert enable bit (bit 31) > > > + * > > > + * Calculates the PCI Base Configuration Space (first 256 bytes) address for > > > + * Type 1 accesses. > > > + * > > > + * Note: the caller is responsible for adding the bits outside of [24:2] > > > + * and enable bit. > > > + * > > > + * Return: PCI Base Configuration Space address. > > > + */ > > > +static inline u32 pci_conf1_addr(u8 bus, u8 devfn, u8 reg, bool enable) > > > +{ > > > + return (enable ? PCI_CONF1_ENABLE : 0) | > > > + FIELD_PREP(PCI_CONF1_BUS, bus) | > > > + FIELD_PREP(PCI_CONF1_DEV | PCI_CONF_FUNC, devfn) | > > > + FIELD_PREP(PCI_CONF_REG, reg & ~3); > > > +} > > > + > > > +/* > > > + * Extension of PCI Config Address for accessing extended PCIe registers > > > + * > > > + * No standardized specification, but used on lot of non-ECAM-compliant ARM SoCs > > > + * or on AMD Barcelona and new CPUs. Reserved bits [27:24] of PCI Config Address > > > + * are used for specifying additional 4 high bits of PCI Express register. > > > + */ > > > +#define PCI_CONF1_EXT_REG 0x0f000000UL > > > + > > > +/** > > > + * pci_conf1_ext_addr - PCI Configuration Space address for Type 1 access > > > + * @bus: Bus number of the device > > > + * @devfn: Device and function numbers > > > + * @reg: Base or Extended Configuration space offset > > > + * @enable: Assert enable bit (bit 31) > > > + * > > > + * Calculates the PCI Base and Extended (4096 bytes per PCI function) > > > + * Configuration Space address for Type 1 accesses. This function assumes > > > + * the Extended Conguration Space is using the reserved bits [27:24]. > > > + * > > > + * Note: the caller is responsible for adding the bits outside of [27:2] and > > > + * enable bit. > > > + * > > > + * Return: PCI Configuration Space address. > > > + */ > > > +static inline u32 pci_conf1_ext_addr(u8 bus, u8 devfn, u16 reg, bool enable) > > > +{ > > > + return FIELD_PREP(PCI_CONF1_EXT_REG, (reg & 0xf00) >> 8) | > > > + pci_conf1_addr(bus, devfn, reg & 0xff, enable); > > > +} > > > + > > > /* Generic PCI functions exported to card drivers */ > > > > > > u8 pci_bus_find_capability(struct pci_bus *bus, unsigned int devfn, int cap); > > > -- > > > 2.39.2 > > > > >
diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 17fed1846847..cf0530a60105 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -833,49 +833,12 @@ struct pci_devres { struct pci_devres *find_pci_dr(struct pci_dev *pdev); -/* - * Config Address for PCI Configuration Mechanism #1 - * - * See PCI Local Bus Specification, Revision 3.0, - * Section 3.2.2.3.2, Figure 3-2, p. 50. - */ - -#define PCI_CONF1_BUS_SHIFT 16 /* Bus number */ -#define PCI_CONF1_DEV_SHIFT 11 /* Device number */ -#define PCI_CONF1_FUNC_SHIFT 8 /* Function number */ - -#define PCI_CONF1_BUS_MASK 0xff -#define PCI_CONF1_DEV_MASK 0x1f -#define PCI_CONF1_FUNC_MASK 0x7 -#define PCI_CONF1_REG_MASK 0xfc /* Limit aligned offset to a maximum of 256B */ - -#define PCI_CONF1_ENABLE BIT(31) -#define PCI_CONF1_BUS(x) (((x) & PCI_CONF1_BUS_MASK) << PCI_CONF1_BUS_SHIFT) -#define PCI_CONF1_DEV(x) (((x) & PCI_CONF1_DEV_MASK) << PCI_CONF1_DEV_SHIFT) -#define PCI_CONF1_FUNC(x) (((x) & PCI_CONF1_FUNC_MASK) << PCI_CONF1_FUNC_SHIFT) -#define PCI_CONF1_REG(x) ((x) & PCI_CONF1_REG_MASK) - #define PCI_CONF1_ADDRESS(bus, dev, func, reg) \ (PCI_CONF1_ENABLE | \ - PCI_CONF1_BUS(bus) | \ - PCI_CONF1_DEV(dev) | \ - PCI_CONF1_FUNC(func) | \ - PCI_CONF1_REG(reg)) - -/* - * Extension of PCI Config Address for accessing extended PCIe registers - * - * No standardized specification, but used on lot of non-ECAM-compliant ARM SoCs - * or on AMD Barcelona and new CPUs. Reserved bits [27:24] of PCI Config Address - * are used for specifying additional 4 high bits of PCI Express register. - */ - -#define PCI_CONF1_EXT_REG_SHIFT 16 -#define PCI_CONF1_EXT_REG_MASK 0xf00 -#define PCI_CONF1_EXT_REG(x) (((x) & PCI_CONF1_EXT_REG_MASK) << PCI_CONF1_EXT_REG_SHIFT) + pci_conf1_addr(bus, PCI_DEVFN(dev, func), reg & ~0x3U)) #define PCI_CONF1_EXT_ADDRESS(bus, dev, func, reg) \ - (PCI_CONF1_ADDRESS(bus, dev, func, reg) | \ - PCI_CONF1_EXT_REG(reg)) + (PCI_CONF1_ENABLE | \ + pci_conf1_ext_addr(bus, PCI_DEVFN(dev, func), reg & ~0x3U)) #endif /* DRIVERS_PCI_H */ diff --git a/include/linux/pci.h b/include/linux/pci.h index 16493426a04f..4c4e3bb52a0a 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -26,6 +26,8 @@ #include <linux/args.h> #include <linux/mod_devicetable.h> +#include <linux/bits.h> +#include <linux/bitfield.h> #include <linux/types.h> #include <linux/init.h> #include <linux/ioport.h> @@ -1183,6 +1185,89 @@ void pci_sort_breadthfirst(void); #define dev_is_pci(d) ((d)->bus == &pci_bus_type) #define dev_is_pf(d) ((dev_is_pci(d) ? to_pci_dev(d)->is_physfn : false)) +/* + * Config Address for PCI Configuration Mechanism #0/1 + * + * See PCI Local Bus Specification, Revision 3.0, + * Section 3.2.2.3.2, Figure 3-1 and 3-2, p. 48-50. + */ +#define PCI_CONF_REG 0x000000ffU /* common for Type 0/1 */ +#define PCI_CONF_FUNC 0x00000700U /* common for Type 0/1 */ +#define PCI_CONF1_DEV 0x0000f800U +#define PCI_CONF1_BUS 0x00ff0000U +#define PCI_CONF1_ENABLE BIT(31) + +/** + * pci_conf0_addr - PCI Base Configuration Space address for Type 0 access + * @devfn: Device and function numbers (device number will be ignored) + * @reg: Base configuration space offset + * + * Calculates the PCI Configuration Space address for Type 0 accesses. + * + * Note: the caller is responsible for adding the bits outside of [10:0]. + * + * Return: Base Configuration Space address. + */ +static inline u32 pci_conf0_addr(u8 devfn, u8 reg) +{ + return FIELD_PREP(PCI_CONF_FUNC, PCI_FUNC(devfn)) | + FIELD_PREP(PCI_CONF_REG, reg & ~3); +} + +/** + * pci_conf1_addr - PCI Base Configuration Space address for Type 1 access + * @bus: Bus number of the device + * @devfn: Device and function numbers + * @reg: Base configuration space offset + * @enable: Assert enable bit (bit 31) + * + * Calculates the PCI Base Configuration Space (first 256 bytes) address for + * Type 1 accesses. + * + * Note: the caller is responsible for adding the bits outside of [24:2] + * and enable bit. + * + * Return: PCI Base Configuration Space address. + */ +static inline u32 pci_conf1_addr(u8 bus, u8 devfn, u8 reg, bool enable) +{ + return (enable ? PCI_CONF1_ENABLE : 0) | + FIELD_PREP(PCI_CONF1_BUS, bus) | + FIELD_PREP(PCI_CONF1_DEV | PCI_CONF_FUNC, devfn) | + FIELD_PREP(PCI_CONF_REG, reg & ~3); +} + +/* + * Extension of PCI Config Address for accessing extended PCIe registers + * + * No standardized specification, but used on lot of non-ECAM-compliant ARM SoCs + * or on AMD Barcelona and new CPUs. Reserved bits [27:24] of PCI Config Address + * are used for specifying additional 4 high bits of PCI Express register. + */ +#define PCI_CONF1_EXT_REG 0x0f000000UL + +/** + * pci_conf1_ext_addr - PCI Configuration Space address for Type 1 access + * @bus: Bus number of the device + * @devfn: Device and function numbers + * @reg: Base or Extended Configuration space offset + * @enable: Assert enable bit (bit 31) + * + * Calculates the PCI Base and Extended (4096 bytes per PCI function) + * Configuration Space address for Type 1 accesses. This function assumes + * the Extended Conguration Space is using the reserved bits [27:24]. + * + * Note: the caller is responsible for adding the bits outside of [27:2] and + * enable bit. + * + * Return: PCI Configuration Space address. + */ +static inline u32 pci_conf1_ext_addr(u8 bus, u8 devfn, u16 reg, bool enable) +{ + return FIELD_PREP(PCI_CONF1_EXT_REG, (reg & 0xf00) >> 8) | + pci_conf1_addr(bus, devfn, reg & 0xff, enable); +} + /* Generic PCI functions exported to card drivers */ u8 pci_bus_find_capability(struct pci_bus *bus, unsigned int devfn, int cap);
Many places in arch and PCI controller code need to calculate PCI Configuration Space Addresses for Type 0/1 accesses. There are small variations between archs when it comes to bits outside of [10:2] (Type 0) and [24:2] (Type 1) but the basic calculation can still be generalized. drivers/pci/pci.h has PCI_CONF1{,_EXT}_ADDRESS() but due to their location the use is limited to PCI subsys and the also always enable PCI_CONF1_ENABLE which is not what all the callers want. Add generic pci_conf{0,1}_addr() and pci_conf1_ext_addr() helpers into include/linux/pci.h which can be reused by various parts of the kernel that have to calculate PCI Conf Type 0/1 addresses. The PCI_CONF* defines are needed by the new helpers so move also them to include/linux/pci.h. The new helpers use true bitmasks and FIELD_PREP() instead of open coded masking and shifting so adjust PCI_CONF* definitions to match that. Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> --- drivers/pci/pci.h | 43 ++--------------------- include/linux/pci.h | 85 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 88 insertions(+), 40 deletions(-)