| Message ID | 20200506151429.12255-8-kishon@ti.com (mailing list archive) |
|---|---|
| State | Superseded, archived |
| Delegated to: | Lorenzo Pieralisi |
| Headers | show |
| Series | Add PCIe support to TI's J721E SoC | expand |
On Wed, May 06, 2020 at 08:44:22PM +0530, Kishon Vijay Abraham I wrote: > Cadence driver uses "mem" memory resource to obtain the offset of > configuration space address region, memory space address region and > message space address region. The obtained offset is used to program > the Address Translation Unit (ATU). However certain platforms like TI's > J721E SoC require the absolute address to be programmed in the ATU and not > just the offset. Once again, Cadence host binding is broken (or at least the example is). The 'mem' region shouldn't even exist. It is overlapping the config space and 'ranges': reg = <0x0 0xfb000000 0x0 0x01000000>, <0x0 0x41000000 0x0 0x00001000>, <0x0 0x40000000 0x0 0x04000000>; reg-names = "reg", "cfg", "mem"; ranges = <0x02000000 0x0 0x42000000 0x0 0x42000000 0x0 0x1000000>, <0x01000000 0x0 0x43000000 0x0 0x43000000 0x0 0x0010000>; 16M of registers looks a bit odd. I guess it doesn't matter unless you have a 32-bit platform and care about your virtual space. Probably should have been 3 regions for LM, RP, and AT looking at the driver. Whatever outbound address translation you need should be based on 'ranges'. Rob
Hi Rob, On 5/21/2020 3:04 AM, Rob Herring wrote: > On Wed, May 06, 2020 at 08:44:22PM +0530, Kishon Vijay Abraham I wrote: >> Cadence driver uses "mem" memory resource to obtain the offset of >> configuration space address region, memory space address region and >> message space address region. The obtained offset is used to program >> the Address Translation Unit (ATU). However certain platforms like TI's >> J721E SoC require the absolute address to be programmed in the ATU and not >> just the offset. > > Once again, Cadence host binding is broken (or at least the example is). > The 'mem' region shouldn't even exist. It is overlapping the config > space and 'ranges': > > reg = <0x0 0xfb000000 0x0 0x01000000>, > <0x0 0x41000000 0x0 0x00001000>, > <0x0 0x40000000 0x0 0x04000000>; > reg-names = "reg", "cfg", "mem"; > > ranges = <0x02000000 0x0 0x42000000 0x0 0x42000000 0x0 0x1000000>, > <0x01000000 0x0 0x43000000 0x0 0x43000000 0x0 0x0010000>; > > > 16M of registers looks a bit odd. I guess it doesn't matter > unless you have a 32-bit platform and care about your virtual > space. Probably should have been 3 regions for LM, RP, and AT looking > at the driver. The "mem" region in never ioremapped. However $patch removes requiring to add "mem" memory resource. > > Whatever outbound address translation you need should be based on > 'ranges'. You mean we don't need to add "new *ops* for CPU addr fixup"?. The issue is ranges provides CPU address and PCI address. The CPU will access whatever is populated in ranges to access the PCI bus. However while programming the ATU, we cannot use the CPU address provided in ranges directly (in some platforms) because the controller does not see the full address and only the lower 28bits. This similar restriction was there with Designware (mostly an integration issue) and we used *ops* to fixup the address that has to be programmed in ATU. The Designware initially used a wrapper so that ranges property can be directly used [1]. However this approach was later removed in [2] [1] -> https://lore.kernel.org/patchwork/patch/468523/ [2] -> https://lkml.org/lkml/2015/10/16/232 Thanks Kishon
On Thu, May 21, 2020 at 5:35 AM Kishon Vijay Abraham I <kishon@ti.com> wrote: > > Hi Rob, > > On 5/21/2020 3:04 AM, Rob Herring wrote: > > On Wed, May 06, 2020 at 08:44:22PM +0530, Kishon Vijay Abraham I wrote: > >> Cadence driver uses "mem" memory resource to obtain the offset of > >> configuration space address region, memory space address region and > >> message space address region. The obtained offset is used to program > >> the Address Translation Unit (ATU). However certain platforms like TI's > >> J721E SoC require the absolute address to be programmed in the ATU and not > >> just the offset. > > > > Once again, Cadence host binding is broken (or at least the example is). > > The 'mem' region shouldn't even exist. It is overlapping the config > > space and 'ranges': > > > > reg = <0x0 0xfb000000 0x0 0x01000000>, > > <0x0 0x41000000 0x0 0x00001000>, > > <0x0 0x40000000 0x0 0x04000000>; > > reg-names = "reg", "cfg", "mem"; > > > > ranges = <0x02000000 0x0 0x42000000 0x0 0x42000000 0x0 0x1000000>, > > <0x01000000 0x0 0x43000000 0x0 0x43000000 0x0 0x0010000>; > > > > > > 16M of registers looks a bit odd. I guess it doesn't matter > > unless you have a 32-bit platform and care about your virtual > > space. Probably should have been 3 regions for LM, RP, and AT looking > > at the driver. > > The "mem" region in never ioremapped. However $patch removes requiring to add > "mem" memory resource. I was referring to ioremapping 'reg' region. > > > > Whatever outbound address translation you need should be based on > > 'ranges'. > > You mean we don't need to add "new *ops* for CPU addr fixup"?. The issue is > ranges provides CPU address and PCI address. The CPU will access whatever is > populated in ranges to access the PCI bus. However while programming the ATU, > we cannot use the CPU address provided in ranges directly (in some platforms) > because the controller does not see the full address and only the lower 28bits. Okay, that is clearer as to what the difference is. I think this should be 2 patches. One dropping 'mem' usage and using a mask and the 2nd making the mask per platform. Really, the parent node of the PCI controller should probably have 'ranges' and you could extract a mask from that. Looks like that is what you had for DRA7... I'm not sure if ABI stability is important for the Cadence platform. I'd assume that's just some IP eval system and probably not? Why do you need an ops here? All you need is a mask value. > This similar restriction was there with Designware (mostly an integration > issue) and we used *ops* to fixup the address that has to be programmed in ATU. > The Designware initially used a wrapper so that ranges property can be directly > used [1]. However this approach was later removed in [2] > > [1] -> https://lore.kernel.org/patchwork/patch/468523/ > [2] -> https://lkml.org/lkml/2015/10/16/232 So while you had the data for a mask in DT, the driver now hardcodes it? Rob
Hi Rob, On 5/22/2020 10:15 PM, Rob Herring wrote: > On Thu, May 21, 2020 at 5:35 AM Kishon Vijay Abraham I <kishon@ti.com> wrote: >> >> Hi Rob, >> >> On 5/21/2020 3:04 AM, Rob Herring wrote: >>> On Wed, May 06, 2020 at 08:44:22PM +0530, Kishon Vijay Abraham I wrote: >>>> Cadence driver uses "mem" memory resource to obtain the offset of >>>> configuration space address region, memory space address region and >>>> message space address region. The obtained offset is used to program >>>> the Address Translation Unit (ATU). However certain platforms like TI's >>>> J721E SoC require the absolute address to be programmed in the ATU and not >>>> just the offset. >>> >>> Once again, Cadence host binding is broken (or at least the example is). >>> The 'mem' region shouldn't even exist. It is overlapping the config >>> space and 'ranges': >>> >>> reg = <0x0 0xfb000000 0x0 0x01000000>, >>> <0x0 0x41000000 0x0 0x00001000>, >>> <0x0 0x40000000 0x0 0x04000000>; >>> reg-names = "reg", "cfg", "mem"; >>> >>> ranges = <0x02000000 0x0 0x42000000 0x0 0x42000000 0x0 0x1000000>, >>> <0x01000000 0x0 0x43000000 0x0 0x43000000 0x0 0x0010000>; >>> >>> >>> 16M of registers looks a bit odd. I guess it doesn't matter >>> unless you have a 32-bit platform and care about your virtual >>> space. Probably should have been 3 regions for LM, RP, and AT looking >>> at the driver. >> >> The "mem" region in never ioremapped. However $patch removes requiring to add >> "mem" memory resource. > > I was referring to ioremapping 'reg' region. > >>> >>> Whatever outbound address translation you need should be based on >>> 'ranges'. >> >> You mean we don't need to add "new *ops* for CPU addr fixup"?. The issue is >> ranges provides CPU address and PCI address. The CPU will access whatever is >> populated in ranges to access the PCI bus. However while programming the ATU, >> we cannot use the CPU address provided in ranges directly (in some platforms) >> because the controller does not see the full address and only the lower 28bits. > > Okay, that is clearer as to what the difference is. I think this > should be 2 patches. One dropping 'mem' usage and using a mask and the > 2nd making the mask per platform. hmm okay. > > Really, the parent node of the PCI controller should probably have > 'ranges' and you could extract a mask from that. Looks like that is > what you had for DRA7... I'm not sure if ABI stability is important > for the Cadence platform. I'd assume that's just some IP eval system > and probably not? Right TI's J721E should be the first HW platform to use Cadence in mainline. > > Why do you need an ops here? All you need is a mask value. So how do we get platform specific mask? Use a different binding to specify the mask value? > >> This similar restriction was there with Designware (mostly an integration >> issue) and we used *ops* to fixup the address that has to be programmed in ATU. >> The Designware initially used a wrapper so that ranges property can be directly >> used [1]. However this approach was later removed in [2] >> >> [1] -> https://lore.kernel.org/patchwork/patch/468523/ >> [2] -> https://lkml.org/lkml/2015/10/16/232 > > So while you had the data for a mask in DT, the driver now hardcodes it? Yes, that's correct. Which approach should we use for Cadence? Thanks Kishon
diff --git a/drivers/pci/controller/cadence/pcie-cadence-host.c b/drivers/pci/controller/cadence/pcie-cadence-host.c index 6c84520318a7..7c220671e66f 100644 --- a/drivers/pci/controller/cadence/pcie-cadence-host.c +++ b/drivers/pci/controller/cadence/pcie-cadence-host.c @@ -104,15 +104,14 @@ static int cdns_pcie_host_init_root_port(struct cdns_pcie_rc *rc) static int cdns_pcie_host_init_address_translation(struct cdns_pcie_rc *rc) { struct cdns_pcie *pcie = &rc->pcie; - struct resource *mem_res = pcie->mem_res; struct resource *bus_range = rc->bus_range; struct resource *cfg_res = rc->cfg_res; struct device *dev = pcie->dev; struct device_node *np = dev->of_node; struct of_pci_range_parser parser; + u64 cpu_addr = cfg_res->start; struct of_pci_range range; u32 addr0, addr1, desc1; - u64 cpu_addr; int r, err; /* @@ -125,7 +124,9 @@ static int cdns_pcie_host_init_address_translation(struct cdns_pcie_rc *rc) cdns_pcie_writel(pcie, CDNS_PCIE_AT_OB_REGION_PCI_ADDR1(0), addr1); cdns_pcie_writel(pcie, CDNS_PCIE_AT_OB_REGION_DESC1(0), desc1); - cpu_addr = cfg_res->start - mem_res->start; + if (pcie->ops->cpu_addr_fixup) + cpu_addr = pcie->ops->cpu_addr_fixup(pcie, cpu_addr); + addr0 = CDNS_PCIE_AT_OB_REGION_CPU_ADDR0_NBITS(12) | (lower_32_bits(cpu_addr) & GENMASK(31, 8)); addr1 = upper_32_bits(cpu_addr); @@ -266,14 +267,6 @@ int cdns_pcie_host_setup(struct cdns_pcie_rc *rc) } rc->cfg_res = res; - res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "mem"); - if (!res) { - dev_err(dev, "missing \"mem\"\n"); - return -EINVAL; - } - - pcie->mem_res = res; - ret = cdns_pcie_start_link(pcie); if (ret) { dev_err(dev, "Failed to start link\n"); diff --git a/drivers/pci/controller/cadence/pcie-cadence-plat.c b/drivers/pci/controller/cadence/pcie-cadence-plat.c index f5c6bf6dfcb8..6f5f07b3eed1 100644 --- a/drivers/pci/controller/cadence/pcie-cadence-plat.c +++ b/drivers/pci/controller/cadence/pcie-cadence-plat.c @@ -13,6 +13,8 @@ #include <linux/of_device.h> #include "pcie-cadence.h" +#define CDNS_PLAT_CPU_TO_BUS_ADDR 0x0FFFFFFF + /** * struct cdns_plat_pcie - private data for this PCIe platform driver * @pcie: Cadence PCIe controller @@ -30,6 +32,15 @@ struct cdns_plat_pcie_of_data { static const struct of_device_id cdns_plat_pcie_of_match[]; +static u64 cdns_plat_cpu_addr_fixup(struct cdns_pcie *pcie, u64 cpu_addr) +{ + return cpu_addr & CDNS_PLAT_CPU_TO_BUS_ADDR; +} + +static const struct cdns_pcie_ops cdns_plat_ops = { + .cpu_addr_fixup = cdns_plat_cpu_addr_fixup, +}; + static int cdns_plat_pcie_probe(struct platform_device *pdev) { const struct cdns_plat_pcie_of_data *data; @@ -66,6 +77,7 @@ static int cdns_plat_pcie_probe(struct platform_device *pdev) rc = pci_host_bridge_priv(bridge); rc->pcie.dev = dev; + rc->pcie.ops = &cdns_plat_ops; cdns_plat_pcie->pcie = &rc->pcie; cdns_plat_pcie->is_rc = is_rc; @@ -93,6 +105,7 @@ static int cdns_plat_pcie_probe(struct platform_device *pdev) return -ENOMEM; ep->pcie.dev = dev; + ep->pcie.ops = &cdns_plat_ops; cdns_plat_pcie->pcie = &ep->pcie; cdns_plat_pcie->is_rc = is_rc; diff --git a/drivers/pci/controller/cadence/pcie-cadence.c b/drivers/pci/controller/cadence/pcie-cadence.c index 874b7fa93577..29d32019145c 100644 --- a/drivers/pci/controller/cadence/pcie-cadence.c +++ b/drivers/pci/controller/cadence/pcie-cadence.c @@ -113,7 +113,9 @@ void cdns_pcie_set_outbound_region(struct cdns_pcie *pcie, u8 fn, cdns_pcie_writel(pcie, CDNS_PCIE_AT_OB_REGION_DESC1(r), desc1); /* Set the CPU address */ - cpu_addr -= pcie->mem_res->start; + if (pcie->ops->cpu_addr_fixup) + cpu_addr = pcie->ops->cpu_addr_fixup(pcie, cpu_addr); + addr0 = CDNS_PCIE_AT_OB_REGION_CPU_ADDR0_NBITS(nbits) | (lower_32_bits(cpu_addr) & GENMASK(31, 8)); addr1 = upper_32_bits(cpu_addr); @@ -140,7 +142,9 @@ void cdns_pcie_set_outbound_region_for_normal_msg(struct cdns_pcie *pcie, u8 fn, } /* Set the CPU address */ - cpu_addr -= pcie->mem_res->start; + if (pcie->ops->cpu_addr_fixup) + cpu_addr = pcie->ops->cpu_addr_fixup(pcie, cpu_addr); + addr0 = CDNS_PCIE_AT_OB_REGION_CPU_ADDR0_NBITS(17) | (lower_32_bits(cpu_addr) & GENMASK(31, 8)); addr1 = upper_32_bits(cpu_addr); diff --git a/drivers/pci/controller/cadence/pcie-cadence.h b/drivers/pci/controller/cadence/pcie-cadence.h index 03c21ce4c9e9..16a6b0ee547c 100644 --- a/drivers/pci/controller/cadence/pcie-cadence.h +++ b/drivers/pci/controller/cadence/pcie-cadence.h @@ -234,6 +234,7 @@ struct cdns_pcie_ops { int (*start_link)(struct cdns_pcie *pcie); void (*stop_link)(struct cdns_pcie *pcie); bool (*link_up)(struct cdns_pcie *pcie); + u64 (*cpu_addr_fixup)(struct cdns_pcie *pcie, u64 cpu_addr); }; /**
Cadence driver uses "mem" memory resource to obtain the offset of configuration space address region, memory space address region and message space address region. The obtained offset is used to program the Address Translation Unit (ATU). However certain platforms like TI's J721E SoC require the absolute address to be programmed in the ATU and not just the offset. Signed-off-by: Kishon Vijay Abraham I <kishon@ti.com> --- .../pci/controller/cadence/pcie-cadence-host.c | 15 ++++----------- .../pci/controller/cadence/pcie-cadence-plat.c | 13 +++++++++++++ drivers/pci/controller/cadence/pcie-cadence.c | 8 ++++++-- drivers/pci/controller/cadence/pcie-cadence.h | 1 + 4 files changed, 24 insertions(+), 13 deletions(-)