[v1,1/4] dt-bindings: PCI: microchip: add fabric address translation properties

Commit Message

Daire McNamara Sept. 2, 2022, 2:21 p.m. UTC

From: Conor Dooley <[e-mail]>

On PolarFire SoC both in- & out-bound address translations occur in two
stages. The specific translations are tightly coupled to the FPGA
designs and supplement the {dma-,}ranges properties. The first stage of
the translation is done by the FPGA fabric & the second by the root
port.
Add two properties so that the translation tables in the root port's
bridge layer can be configured to account for the translation done by
the FPGA fabric.

Signed-off-by: Conor Dooley <[e-mail]>
Signed-off-by: Daire McNamara <[e-mail]>
---
 .../bindings/pci/microchip,pcie-host.yaml     | 107 ++++++++++++++++++
 1 file changed, 107 insertions(+)

Comments

Rob Herring Sept. 2, 2022, 4:28 p.m. UTC | #1

On Fri, Sep 2, 2022 at 9:22 AM <[e-mail]> wrote:
>
> From: Conor Dooley <[e-mail]>
>
> On PolarFire SoC both in- & out-bound address translations occur in two
> stages. The specific translations are tightly coupled to the FPGA
> designs and supplement the {dma-,}ranges properties. The first stage of
> the translation is done by the FPGA fabric & the second by the root
> port.
> Add two properties so that the translation tables in the root port's
> bridge layer can be configured to account for the translation done by
> the FPGA fabric.

I'm skeptical that ranges/dma-ranges can't handle what you need.
Anything in this area is going to need justification 'ranges doesn't
work because x, y, z...'.

>
> Signed-off-by: Conor Dooley <[e-mail]>
> Signed-off-by: Daire McNamara <[e-mail]>
> ---
>  .../bindings/pci/microchip,pcie-host.yaml     | 107 ++++++++++++++++++
>  1 file changed, 107 insertions(+)
>
> diff --git a/Documentation/devicetree/bindings/pci/microchip,pcie-host.yaml b/Documentation/devicetree/bindings/pci/microchip,pcie-host.yaml
> index 23d95c65acff..29bb1fe99a2e 100644
> --- a/Documentation/devicetree/bindings/pci/microchip,pcie-host.yaml
> +++ b/Documentation/devicetree/bindings/pci/microchip,pcie-host.yaml
> @@ -71,6 +71,113 @@ properties:
>      minItems: 1
>      maxItems: 6
>
> +  microchip,outbound-fabric-translation-ranges:
> +    $ref: /schemas/types.yaml#/definitions/uint32-matrix
> +    minItems: 1
> +    maxItems: 32
> +    description: |
> +      The CPU-to-PCIe (outbound) address translation takes place in two stages.
> +      Depending on the FPGA bitstream, the outbound address translation tables
> +      in the PCIe root port's bridge layer will need to be configured to account
> +      for only its part of the overall outbound address translation.
> +
> +      The first stage of outbound address translation occurs between the CPU address
> +      and an intermediate "FPGA address". The second stage of outbound address
> +      translation occurs between this FPGA address and the PCIe address. Use this
> +      property, in conjunction with the ranges property, to divide the overall
> +      address translation into these two stages so that the PCIe address
> +      translation tables can be correctly configured.

Sounds like you need 2 levels of ranges/dma-ranges.

/ {
    fpga-bus {
        ranges = ...
        dma-ranges = ...
        pcie@... {
            ranges = ...
            dma-ranges = ...
        };
    };
};

> +
> +      If this property is present, one entry is required per range. This is so
> +      FPGA designers can choose to route different address ranges through different
> +      Fabric Interface Controllers and other logic as they see fit.
> +
> +      If this property is not present, the entire address translation
> +      in any ranges property is attempted by the root port driver via its outbound
> +      address translation tables.
> +
> +      Each element in this property has three components. The first is a
> +      PCIe address, the second is an FPGA address, and the third is a size.
> +      These properties may be 32 or 64 bit values.
> +
> +      In operation, the driver will expect a one-to-one correspondance between
> +      range properties and this property.  For each pair of range and
> +      outbound-fabric-translation-range properties, the root port driver will
> +      subtract the FPGA address in this property from the CPU address in the
> +      corresponding range property and use the remainder to program its
> +      outbound address translation tables.
> +
> +      For each range, take its PCIe address and size - these are the PCIe
> +      address & size for the element. The FPGA address is derived from a given
> +      FPGA fabric design and is the address delivered by that FPGA fabric
> +      design to the Core Complex. For a trivial configuration, it is likely to be the
> +      lower 32 bits of the PCIe address in the range property and the upper
> +      bits of the base address of the Fabric Interface Controller the design uses.
> +      Otherwise, it is tightly coupled with the data path configured in the
> +      FPGA fabric between the root port and the Core Complex.
> +
> +      For more information on the tables, see Section 1.3.3,
> +      "PCIe/AXI4 Address Translation" of the PolarFire SoC PCIe User Guide:
> +      https://www.microsemi.com/document-portal/doc_download/1245812-polarfire-fpga-and-polarfire-soc-fpga-pci-express-user-guide
> +
> +    items:
> +      minItems: 3
> +      maxItems: 6
> +
> +  microchip,inbound-fabric-translation-ranges:
> +    $ref: /schemas/types.yaml#/definitions/uint32-matrix
> +    minItems: 1
> +    maxItems: 32
> +    description: |
> +      The PCIe-to-CPU (inbound) address translation takes place in two stages.
> +      Depending on the FPGA bitstream, the inbound address translation tables
> +      in the PCIe root port's bridge layer will need to be configured to account
> +      for only its part of the overall inbound address translation.
> +
> +      The first stage of address translation occurs between the PCIe address and
> +      an intermediate FPGA address. The second stage of address translation
> +      occurs between the FPGA address and the CPU address. Use this property
> +      in conjunction with the dma-ranges property to divide the address
> +      translation into these two stages.
> +
> +      If this property is present, one entry is required per dma-range. This is so
> +      FPGA designers can choose to route different address ranges through different
> +      Fabric Interface Controllers and other logic as they see fit.
> +
> +      If this property is not present, the entire address translation
> +      in any dma-ranges property is attempted by the root port driver via its
> +      inbound address translation tables.
> +
> +      Each element in this property has three components. The first is a
> +      PCIe address, the second is an FPGA address, and the third is a size.
> +      These properties may be 32 or 64 bit values.
> +
> +      In operation, the driver will expect a one-to-one correspondance between
> +      dma-range properties and this property.  For each pair of dma-range and
> +      inbound-fabric-translation-range properties, the root port driver will
> +      subtract the FPGA address in this property from the CPU address in the
> +      corresponding dma-range property and use the remainder to program its
> +      inbound address translation tables.
> +
> +      From each dma-range, take its PCIe address and size - these are the PCIe
> +      address & size for the element. The FPGA address is derived from a given
> +      FPGA fabric design and is the address delivered by that FPGA fabric
> +      design to the Core Complex. For a trivial configuration, this property
> +      is unlikely to be required (i.e. no fabric translation on the inbound
> +      interface).  Otherwise, it is tightly coupled with the inbound data path
> +      configured in the FPGA fabric between the root port and the Core Complex.
> +      It is expected that more than one translation range may be added to
> +      an FPGA fabric design, e.g. to deliver data to cached or non-cached
> +      DDR.
> +
> +      For more information on the tables, see Section 1.3.3,
> +      "PCIe/AXI4 Address Translation" of the PolarFire SoC PCIe User Guide:
> +      https://www.microsemi.com/document-portal/doc_download/1245812-polarfire-fpga-and-polarfire-soc-fpga-pci-express-user-guide
> +
> +    items:
> +      minItems: 4
> +      maxItems: 7
> +
>    msi-controller:
>      description: Identifies the node as an MSI controller.
>
> --
> 2.25.1
>

Daire McNamara Sept. 2, 2022, 4:51 p.m. UTC | #2

Thanks Rob for taking the time to get back to me.  Yes, I thought 2
levels of translation might be the way to go. If you could point me to
a gadget that has 2-level translation working, that'd be great

On Fri, 2022-09-02 at 11:28 -0500, Rob Herring wrote:
> EXTERNAL EMAIL: Do not click links or open attachments unless you
> know the content is safe
> 
> On Fri, Sep 2, 2022 at 9:22 AM <[e-mail]> wrote:
> > From: Conor Dooley <[e-mail]>
> > 
> > On PolarFire SoC both in- & out-bound address translations occur in
> > two
> > stages. The specific translations are tightly coupled to the FPGA
> > designs and supplement the {dma-,}ranges properties. The first
> > stage of
> > the translation is done by the FPGA fabric & the second by the root
> > port.
> > Add two properties so that the translation tables in the root
> > port's
> > bridge layer can be configured to account for the translation done
> > by
> > the FPGA fabric.
> 
> I'm skeptical that ranges/dma-ranges can't handle what you need.
> Anything in this area is going to need justification 'ranges doesn't
> work because x, y, z...'.
Cool.
> 
> > Signed-off-by: Conor Dooley <[e-mail]>
> > Signed-off-by: Daire McNamara <[e-mail]>
> > ---
> >  .../bindings/pci/microchip,pcie-host.yaml     | 107
> > ++++++++++++++++++
> >  1 file changed, 107 insertions(+)
> > 
> > diff --git a/Documentation/devicetree/bindings/pci/microchip,pcie-
> > host.yaml b/Documentation/devicetree/bindings/pci/microchip,pcie-
> > host.yaml
> > index 23d95c65acff..29bb1fe99a2e 100644
> > --- a/Documentation/devicetree/bindings/pci/microchip,pcie-
> > host.yaml
> > +++ b/Documentation/devicetree/bindings/pci/microchip,pcie-
> > host.yaml
> > @@ -71,6 +71,113 @@ properties:
> >      minItems: 1
> >      maxItems: 6
> > 
> > +  microchip,outbound-fabric-translation-ranges:
> > +    $ref: /schemas/types.yaml#/definitions/uint32-matrix
> > +    minItems: 1
> > +    maxItems: 32
> > +    description: |
> > +      The CPU-to-PCIe (outbound) address translation takes place
> > in two stages.
> > +      Depending on the FPGA bitstream, the outbound address
> > translation tables
> > +      in the PCIe root port's bridge layer will need to be
> > configured to account
> > +      for only its part of the overall outbound address
> > translation.
> > +
> > +      The first stage of outbound address translation occurs
> > between the CPU address
> > +      and an intermediate "FPGA address". The second stage of
> > outbound address
> > +      translation occurs between this FPGA address and the PCIe
> > address. Use this
> > +      property, in conjunction with the ranges property, to divide
> > the overall
> > +      address translation into these two stages so that the PCIe
> > address
> > +      translation tables can be correctly configured.
> 
> Sounds like you need 2 levels of ranges/dma-ranges.
> 
> / {
>     fpga-bus {
>         ranges = ...
>         dma-ranges = ...
>         pcie@... {
>             ranges = ...
>             dma-ranges = ...
>         };
>     };
> };
> 
Very possibly! I'd prefer that approach, tbh, if I can get it working. 
I'll go back and try 2 levels again.


> > +      If this property is present, one entry is required per
> > range. This is so
> > +      FPGA designers can choose to route different address ranges
> > through different
> > +      Fabric Interface Controllers and other logic as they see
> > fit.
> > +
> > +      If this property is not present, the entire address
> > translation
> > +      in any ranges property is attempted by the root port driver
> > via its outbound
> > +      address translation tables.
> > +
> > +      Each element in this property has three components. The
> > first is a
> > +      PCIe address, the second is an FPGA address, and the third
> > is a size.
> > +      These properties may be 32 or 64 bit values.
> > +
> > +      In operation, the driver will expect a one-to-one
> > correspondance between
> > +      range properties and this property.  For each pair of range
> > and
> > +      outbound-fabric-translation-range properties, the root port
> > driver will
> > +      subtract the FPGA address in this property from the CPU
> > address in the
> > +      corresponding range property and use the remainder to
> > program its
> > +      outbound address translation tables.
> > +
> > +      For each range, take its PCIe address and size - these are
> > the PCIe
> > +      address & size for the element. The FPGA address is derived
> > from a given
> > +      FPGA fabric design and is the address delivered by that FPGA
> > fabric
> > +      design to the Core Complex. For a trivial configuration, it
> > is likely to be the
> > +      lower 32 bits of the PCIe address in the range property and
> > the upper
> > +      bits of the base address of the Fabric Interface Controller
> > the design uses.
> > +      Otherwise, it is tightly coupled with the data path
> > configured in the
> > +      FPGA fabric between the root port and the Core Complex.
> > +
> > +      For more information on the tables, see Section 1.3.3,
> > +      "PCIe/AXI4 Address Translation" of the PolarFire SoC PCIe
> > User Guide:
> > +      
> > https://www.microsemi.com/document-portal/doc_download/1245812-polarfire-fpga-and-polarfire-soc-fpga-pci-express-user-guide
> > +
> > +    items:
> > +      minItems: 3
> > +      maxItems: 6
> > +
> > +  microchip,inbound-fabric-translation-ranges:
> > +    $ref: /schemas/types.yaml#/definitions/uint32-matrix
> > +    minItems: 1
> > +    maxItems: 32
> > +    description: |
> > +      The PCIe-to-CPU (inbound) address translation takes place in
> > two stages.
> > +      Depending on the FPGA bitstream, the inbound address
> > translation tables
> > +      in the PCIe root port's bridge layer will need to be
> > configured to account
> > +      for only its part of the overall inbound address
> > translation.
> > +
> > +      The first stage of address translation occurs between the
> > PCIe address and
> > +      an intermediate FPGA address. The second stage of address
> > translation
> > +      occurs between the FPGA address and the CPU address. Use
> > this property
> > +      in conjunction with the dma-ranges property to divide the
> > address
> > +      translation into these two stages.
> > +
> > +      If this property is present, one entry is required per dma-
> > range. This is so
> > +      FPGA designers can choose to route different address ranges
> > through different
> > +      Fabric Interface Controllers and other logic as they see
> > fit.
> > +
> > +      If this property is not present, the entire address
> > translation
> > +      in any dma-ranges property is attempted by the root port
> > driver via its
> > +      inbound address translation tables.
> > +
> > +      Each element in this property has three components. The
> > first is a
> > +      PCIe address, the second is an FPGA address, and the third
> > is a size.
> > +      These properties may be 32 or 64 bit values.
> > +
> > +      In operation, the driver will expect a one-to-one
> > correspondance between
> > +      dma-range properties and this property.  For each pair of
> > dma-range and
> > +      inbound-fabric-translation-range properties, the root port
> > driver will
> > +      subtract the FPGA address in this property from the CPU
> > address in the
> > +      corresponding dma-range property and use the remainder to
> > program its
> > +      inbound address translation tables.
> > +
> > +      From each dma-range, take its PCIe address and size - these
> > are the PCIe
> > +      address & size for the element. The FPGA address is derived
> > from a given
> > +      FPGA fabric design and is the address delivered by that FPGA
> > fabric
> > +      design to the Core Complex. For a trivial configuration,
> > this property
> > +      is unlikely to be required (i.e. no fabric translation on
> > the inbound
> > +      interface).  Otherwise, it is tightly coupled with the
> > inbound data path
> > +      configured in the FPGA fabric between the root port and the
> > Core Complex.
> > +      It is expected that more than one translation range may be
> > added to
> > +      an FPGA fabric design, e.g. to deliver data to cached or
> > non-cached
> > +      DDR.
> > +
> > +      For more information on the tables, see Section 1.3.3,
> > +      "PCIe/AXI4 Address Translation" of the PolarFire SoC PCIe
> > User Guide:
> > +      
> > https://www.microsemi.com/document-portal/doc_download/1245812-polarfire-fpga-and-polarfire-soc-fpga-pci-express-user-guide
> > +
> > +    items:
> > +      minItems: 4
> > +      maxItems: 7
> > +
> >    msi-controller:
> >      description: Identifies the node as an MSI controller.
> > 
> > --
> > 2.25.1
> >

Daire McNamara Sept. 5, 2022, 2:54 p.m. UTC | #3

On Fri, 2022-09-02 at 11:28 -0500, Rob Herring wrote:
> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
> 
> On Fri, Sep 2, 2022 at 9:22 AM <[e-mail]> wrote:
> > From: Conor Dooley <[e-mail]>
> > 
> > On PolarFire SoC both in- & out-bound address translations occur in two
> > stages. The specific translations are tightly coupled to the FPGA
> > designs and supplement the {dma-,}ranges properties. The first stage of
> > the translation is done by the FPGA fabric & the second by the root
> > port.
> > Add two properties so that the translation tables in the root port's
> > bridge layer can be configured to account for the translation done by
> > the FPGA fabric.
> 
> I'm skeptical that ranges/dma-ranges can't handle what you need.
> Anything in this area is going to need justification 'ranges doesn't
> work because x, y, z...'.
> 
> > Signed-off-by: Conor Dooley <[e-mail]>
> > Signed-off-by: Daire McNamara <[e-mail]>
> > ---
> >  .../bindings/pci/microchip,pcie-host.yaml     | 107 ++++++++++++++++++
> >  1 file changed, 107 insertions(+)
> > 
> > diff --git a/Documentation/devicetree/bindings/pci/microchip,pcie-host.yaml b/Documentation/devicetree/bindings/pci/microchip,pcie-host.yaml
> > index 23d95c65acff..29bb1fe99a2e 100644
> > --- a/Documentation/devicetree/bindings/pci/microchip,pcie-host.yaml
> > +++ b/Documentation/devicetree/bindings/pci/microchip,pcie-host.yaml
> > @@ -71,6 +71,113 @@ properties:
> >      minItems: 1
> >      maxItems: 6
> > 
> > +  microchip,outbound-fabric-translation-ranges:
> > +    $ref: /schemas/types.yaml#/definitions/uint32-matrix
> > +    minItems: 1
> > +    maxItems: 32
> > +    description: |
> > +      The CPU-to-PCIe (outbound) address translation takes place in two stages.
> > +      Depending on the FPGA bitstream, the outbound address translation tables
> > +      in the PCIe root port's bridge layer will need to be configured to account
> > +      for only its part of the overall outbound address translation.
> > +
> > +      The first stage of outbound address translation occurs between the CPU address
> > +      and an intermediate "FPGA address". The second stage of outbound address
> > +      translation occurs between this FPGA address and the PCIe address. Use this
> > +      property, in conjunction with the ranges property, to divide the overall
> > +      address translation into these two stages so that the PCIe address
> > +      translation tables can be correctly configured.
> 
> Sounds like you need 2 levels of ranges/dma-ranges.
> 
> / {
>     fpga-bus {
>         ranges = ...
>         dma-ranges = ...
>         pcie@... {
>             ranges = ...
>             dma-ranges = ...
>         };
>     };
> };
Thanks a million for looking at this! Very much appreciated.

So, this is what I tried.  I've cut down the dts I used to what I think is the minimum
fragment to discuss the issue I'm facing.

So, I replaced this stanza:

pcie: pcie@3000000000 {
    ...
    reg = <0x30 0x0 0x0 0x8000000>, <0x0 0x43000000 0x0 0x10000>;
    reg-names = "cfg", "apb";
    ranges = <0x0000000 0x0 0x0000000 0x30 0x0000000 0x0 0x8000000>,
             <0x3000000 0x0 0x8000000 0x30 0x8000000 0x0 0x80000000>;
    ...
};

with this two-level stanza:

fpga_bus: fpga-bus {
    #address-cells = <2>;
    #size-cells = <2>;
    ranges = <0 0 0x30 0 0x40 0>;
    compatible = "simple-bus";
    ...

    pcie: pcie@0 {
        reg = <0x0 0x0 0x0 0x8000000>, <0x0 0x43000000 0x0 0x10000>;
        reg-names = "cfg", "apb";
        ranges = <0x0000000 0x0 0x0000000 0 0x0000000 0x0 0x8000000>,
                 <0x3000000 0x0 0x8000000 0 0x8000000 0x0 0x80000000>;
        ...
    };
};

and I ran into two problems:
1) the ranges presented to the driver via  resource_list_for_each_entry(entry, &bridge->windows) 
   were unchanged. The start and end of both resources were still in 0x30'0000'0000 space, 
   not 0x0000'0000 as I'd hoped. The two levels of range had been amalgamated before 
   presentation to the rootport driver, so my initial problem was unchanged ...

2) a new issue cropped up. While the 'cfg' register property is in 0x30'0000'0000 space, 
   the 'abp' interface is actually delivered over a separate FIC and is in a 0x4000'0000 
   memory space. In the two-level stanza, it was now being provided to the rootport 
   driver at a base of 0x30'4000'0000 which is incorrect. This is very typical for 
   designers to route abp over a different FIC to axi. 

I hope I'm making the issues clear.

Any suggestions welcome!

> 
> > +
> > +      If this property is present, one entry is required per range. This is so
> > +      FPGA designers can choose to route different address ranges through different
> > +      Fabric Interface Controllers and other logic as they see fit.
> > +
> > +      If this property is not present, the entire address translation
> > +      in any ranges property is attempted by the root port driver via its outbound
> > +      address translation tables.
> > +
> > +      Each element in this property has three components. The first is a
> > +      PCIe address, the second is an FPGA address, and the third is a size.
> > +      These properties may be 32 or 64 bit values.
> > +
> > +      In operation, the driver will expect a one-to-one correspondance between
> > +      range properties and this property.  For each pair of range and
> > +      outbound-fabric-translation-range properties, the root port driver will
> > +      subtract the FPGA address in this property from the CPU address in the
> > +      corresponding range property and use the remainder to program its
> > +      outbound address translation tables.
> > +
> > +      For each range, take its PCIe address and size - these are the PCIe
> > +      address & size for the element. The FPGA address is derived from a given
> > +      FPGA fabric design and is the address delivered by that FPGA fabric
> > +      design to the Core Complex. For a trivial configuration, it is likely to be the
> > +      lower 32 bits of the PCIe address in the range property and the upper
> > +      bits of the base address of the Fabric Interface Controller the design uses.
> > +      Otherwise, it is tightly coupled with the data path configured in the
> > +      FPGA fabric between the root port and the Core Complex.
> > +
> > +      For more information on the tables, see Section 1.3.3,
> > +      "PCIe/AXI4 Address Translation" of the PolarFire SoC PCIe User Guide:
> > +      https://www.microsemi.com/document-portal/doc_download/1245812-polarfire-fpga-and-polarfire-soc-fpga-pci-express-user-guide
> > +
> > +    items:
> > +      minItems: 3
> > +      maxItems: 6
> > +
> > +  microchip,inbound-fabric-translation-ranges:
> > +    $ref: /schemas/types.yaml#/definitions/uint32-matrix
> > +    minItems: 1
> > +    maxItems: 32
> > +    description: |
> > +      The PCIe-to-CPU (inbound) address translation takes place in two stages.
> > +      Depending on the FPGA bitstream, the inbound address translation tables
> > +      in the PCIe root port's bridge layer will need to be configured to account
> > +      for only its part of the overall inbound address translation.
> > +
> > +      The first stage of address translation occurs between the PCIe address and
> > +      an intermediate FPGA address. The second stage of address translation
> > +      occurs between the FPGA address and the CPU address. Use this property
> > +      in conjunction with the dma-ranges property to divide the address
> > +      translation into these two stages.
> > +
> > +      If this property is present, one entry is required per dma-range. This is so
> > +      FPGA designers can choose to route different address ranges through different
> > +      Fabric Interface Controllers and other logic as they see fit.
> > +
> > +      If this property is not present, the entire address translation
> > +      in any dma-ranges property is attempted by the root port driver via its
> > +      inbound address translation tables.
> > +
> > +      Each element in this property has three components. The first is a
> > +      PCIe address, the second is an FPGA address, and the third is a size.
> > +      These properties may be 32 or 64 bit values.
> > +
> > +      In operation, the driver will expect a one-to-one correspondance between
> > +      dma-range properties and this property.  For each pair of dma-range and
> > +      inbound-fabric-translation-range properties, the root port driver will
> > +      subtract the FPGA address in this property from the CPU address in the
> > +      corresponding dma-range property and use the remainder to program its
> > +      inbound address translation tables.
> > +
> > +      From each dma-range, take its PCIe address and size - these are the PCIe
> > +      address & size for the element. The FPGA address is derived from a given
> > +      FPGA fabric design and is the address delivered by that FPGA fabric
> > +      design to the Core Complex. For a trivial configuration, this property
> > +      is unlikely to be required (i.e. no fabric translation on the inbound
> > +      interface).  Otherwise, it is tightly coupled with the inbound data path
> > +      configured in the FPGA fabric between the root port and the Core Complex.
> > +      It is expected that more than one translation range may be added to
> > +      an FPGA fabric design, e.g. to deliver data to cached or non-cached
> > +      DDR.
> > +
> > +      For more information on the tables, see Section 1.3.3,
> > +      "PCIe/AXI4 Address Translation" of the PolarFire SoC PCIe User Guide:
> > +      https://www.microsemi.com/document-portal/doc_download/1245812-polarfire-fpga-and-polarfire-soc-fpga-pci-express-user-guide
> > +
> > +    items:
> > +      minItems: 4
> > +      maxItems: 7
> > +
> >    msi-controller:
> >      description: Identifies the node as an MSI controller.
> > 
> > --
> > 2.25.1
> >

Rob Herring (Arm) Sept. 8, 2022, 8:57 p.m. UTC | #4

On Mon, Sep 05, 2022 at 02:54:07PM +0000, [e-mail] wrote:
> On Fri, 2022-09-02 at 11:28 -0500, Rob Herring wrote:
> > EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
> > 
> > On Fri, Sep 2, 2022 at 9:22 AM <[e-mail]> wrote:
> > > From: Conor Dooley <[e-mail]>
> > > 
> > > On PolarFire SoC both in- & out-bound address translations occur in two
> > > stages. The specific translations are tightly coupled to the FPGA
> > > designs and supplement the {dma-,}ranges properties. The first stage of
> > > the translation is done by the FPGA fabric & the second by the root
> > > port.
> > > Add two properties so that the translation tables in the root port's
> > > bridge layer can be configured to account for the translation done by
> > > the FPGA fabric.
> > 
> > I'm skeptical that ranges/dma-ranges can't handle what you need.
> > Anything in this area is going to need justification 'ranges doesn't
> > work because x, y, z...'.
> > 
> > > Signed-off-by: Conor Dooley <[e-mail]>
> > > Signed-off-by: Daire McNamara <[e-mail]>
> > > ---
> > >  .../bindings/pci/microchip,pcie-host.yaml     | 107 ++++++++++++++++++
> > >  1 file changed, 107 insertions(+)
> > > 
> > > diff --git a/Documentation/devicetree/bindings/pci/microchip,pcie-host.yaml b/Documentation/devicetree/bindings/pci/microchip,pcie-host.yaml
> > > index 23d95c65acff..29bb1fe99a2e 100644
> > > --- a/Documentation/devicetree/bindings/pci/microchip,pcie-host.yaml
> > > +++ b/Documentation/devicetree/bindings/pci/microchip,pcie-host.yaml
> > > @@ -71,6 +71,113 @@ properties:
> > >      minItems: 1
> > >      maxItems: 6
> > > 
> > > +  microchip,outbound-fabric-translation-ranges:
> > > +    $ref: /schemas/types.yaml#/definitions/uint32-matrix
> > > +    minItems: 1
> > > +    maxItems: 32
> > > +    description: |
> > > +      The CPU-to-PCIe (outbound) address translation takes place in two stages.
> > > +      Depending on the FPGA bitstream, the outbound address translation tables
> > > +      in the PCIe root port's bridge layer will need to be configured to account
> > > +      for only its part of the overall outbound address translation.
> > > +
> > > +      The first stage of outbound address translation occurs between the CPU address
> > > +      and an intermediate "FPGA address". The second stage of outbound address
> > > +      translation occurs between this FPGA address and the PCIe address. Use this
> > > +      property, in conjunction with the ranges property, to divide the overall
> > > +      address translation into these two stages so that the PCIe address
> > > +      translation tables can be correctly configured.
> > 
> > Sounds like you need 2 levels of ranges/dma-ranges.
> > 
> > / {
> >     fpga-bus {
> >         ranges = ...
> >         dma-ranges = ...
> >         pcie@... {
> >             ranges = ...
> >             dma-ranges = ...
> >         };
> >     };
> > };
> Thanks a million for looking at this! Very much appreciated.
> 
> So, this is what I tried.  I've cut down the dts I used to what I think is the minimum
> fragment to discuss the issue I'm facing.
> 
> So, I replaced this stanza:
> 
> pcie: pcie@3000000000 {
>     ...
>     reg = <0x30 0x0 0x0 0x8000000>, <0x0 0x43000000 0x0 0x10000>;
>     reg-names = "cfg", "apb";
>     ranges = <0x0000000 0x0 0x0000000 0x30 0x0000000 0x0 0x8000000>,
>              <0x3000000 0x0 0x8000000 0x30 0x8000000 0x0 0x80000000>;
>     ...
> };
> 
> with this two-level stanza:
> 
> fpga_bus: fpga-bus {
>     #address-cells = <2>;
>     #size-cells = <2>;
>     ranges = <0 0 0x30 0 0x40 0>;
>     compatible = "simple-bus";
>     ...
> 
>     pcie: pcie@0 {
>         reg = <0x0 0x0 0x0 0x8000000>, <0x0 0x43000000 0x0 0x10000>;
>         reg-names = "cfg", "apb";
>         ranges = <0x0000000 0x0 0x0000000 0 0x0000000 0x0 0x8000000>,
>                  <0x3000000 0x0 0x8000000 0 0x8000000 0x0 0x80000000>;
>         ...
>     };
> };
> 
> and I ran into two problems:
> 1) the ranges presented to the driver via  resource_list_for_each_entry(entry, &bridge->windows) 
>    were unchanged. The start and end of both resources were still in 0x30'0000'0000 space, 
>    not 0x0000'0000 as I'd hoped. The two levels of range had been amalgamated before 
>    presentation to the rootport driver, so my initial problem was unchanged ...

Yes, that's expected as the translation will walk up parents to root 
node. You will have to get the untranslated values out of 
ranges yourself. If you use the range parsing functions on the parent 
node ranges, you'll get the 0 from of_range.bus_addr.

> 
> 2) a new issue cropped up. While the 'cfg' register property is in 0x30'0000'0000 space, 
>    the 'abp' interface is actually delivered over a separate FIC and is in a 0x4000'0000 
>    memory space. In the two-level stanza, it was now being provided to the rootport 
>    driver at a base of 0x30'4000'0000 which is incorrect. This is very typical for 
>    designers to route abp over a different FIC to axi. 

If the fpga-bus ranges has a 1:1 entry for 0x43000000 child bus then it 
should get translated correctly. Worst case, you may need to define a 
child bus address outside of 0x30_00000000 range that translates back to 
0x43000000.

Rob

Patch

diff --git a/Documentation/devicetree/bindings/pci/microchip,pcie-host.yaml b/Documentation/devicetree/bindings/pci/microchip,pcie-host.yaml
index 23d95c65acff..29bb1fe99a2e 100644
--- a/Documentation/devicetree/bindings/pci/microchip,pcie-host.yaml
+++ b/Documentation/devicetree/bindings/pci/microchip,pcie-host.yaml
@@ -71,6 +71,113 @@  properties:
     minItems: 1
     maxItems: 6
 
+  microchip,outbound-fabric-translation-ranges:
+    $ref: /schemas/types.yaml#/definitions/uint32-matrix
+    minItems: 1
+    maxItems: 32
+    description: |
+      The CPU-to-PCIe (outbound) address translation takes place in two stages.
+      Depending on the FPGA bitstream, the outbound address translation tables
+      in the PCIe root port's bridge layer will need to be configured to account
+      for only its part of the overall outbound address translation.
+
+      The first stage of outbound address translation occurs between the CPU address
+      and an intermediate "FPGA address". The second stage of outbound address
+      translation occurs between this FPGA address and the PCIe address. Use this
+      property, in conjunction with the ranges property, to divide the overall
+      address translation into these two stages so that the PCIe address
+      translation tables can be correctly configured.
+
+      If this property is present, one entry is required per range. This is so
+      FPGA designers can choose to route different address ranges through different
+      Fabric Interface Controllers and other logic as they see fit.
+
+      If this property is not present, the entire address translation
+      in any ranges property is attempted by the root port driver via its outbound
+      address translation tables.
+
+      Each element in this property has three components. The first is a
+      PCIe address, the second is an FPGA address, and the third is a size.
+      These properties may be 32 or 64 bit values.
+
+      In operation, the driver will expect a one-to-one correspondance between
+      range properties and this property.  For each pair of range and
+      outbound-fabric-translation-range properties, the root port driver will
+      subtract the FPGA address in this property from the CPU address in the
+      corresponding range property and use the remainder to program its
+      outbound address translation tables.
+
+      For each range, take its PCIe address and size - these are the PCIe
+      address & size for the element. The FPGA address is derived from a given
+      FPGA fabric design and is the address delivered by that FPGA fabric
+      design to the Core Complex. For a trivial configuration, it is likely to be the
+      lower 32 bits of the PCIe address in the range property and the upper
+      bits of the base address of the Fabric Interface Controller the design uses.
+      Otherwise, it is tightly coupled with the data path configured in the
+      FPGA fabric between the root port and the Core Complex.
+
+      For more information on the tables, see Section 1.3.3,
+      "PCIe/AXI4 Address Translation" of the PolarFire SoC PCIe User Guide:
+      https://www.microsemi.com/document-portal/doc_download/1245812-polarfire-fpga-and-polarfire-soc-fpga-pci-express-user-guide
+
+    items:
+      minItems: 3
+      maxItems: 6
+
+  microchip,inbound-fabric-translation-ranges:
+    $ref: /schemas/types.yaml#/definitions/uint32-matrix
+    minItems: 1
+    maxItems: 32
+    description: |
+      The PCIe-to-CPU (inbound) address translation takes place in two stages.
+      Depending on the FPGA bitstream, the inbound address translation tables
+      in the PCIe root port's bridge layer will need to be configured to account
+      for only its part of the overall inbound address translation.
+
+      The first stage of address translation occurs between the PCIe address and
+      an intermediate FPGA address. The second stage of address translation
+      occurs between the FPGA address and the CPU address. Use this property
+      in conjunction with the dma-ranges property to divide the address
+      translation into these two stages.
+
+      If this property is present, one entry is required per dma-range. This is so
+      FPGA designers can choose to route different address ranges through different
+      Fabric Interface Controllers and other logic as they see fit.
+
+      If this property is not present, the entire address translation
+      in any dma-ranges property is attempted by the root port driver via its
+      inbound address translation tables.
+
+      Each element in this property has three components. The first is a
+      PCIe address, the second is an FPGA address, and the third is a size.
+      These properties may be 32 or 64 bit values.
+
+      In operation, the driver will expect a one-to-one correspondance between
+      dma-range properties and this property.  For each pair of dma-range and
+      inbound-fabric-translation-range properties, the root port driver will
+      subtract the FPGA address in this property from the CPU address in the
+      corresponding dma-range property and use the remainder to program its
+      inbound address translation tables.
+
+      From each dma-range, take its PCIe address and size - these are the PCIe
+      address & size for the element. The FPGA address is derived from a given
+      FPGA fabric design and is the address delivered by that FPGA fabric
+      design to the Core Complex. For a trivial configuration, this property
+      is unlikely to be required (i.e. no fabric translation on the inbound
+      interface).  Otherwise, it is tightly coupled with the inbound data path
+      configured in the FPGA fabric between the root port and the Core Complex.
+      It is expected that more than one translation range may be added to
+      an FPGA fabric design, e.g. to deliver data to cached or non-cached
+      DDR.
+
+      For more information on the tables, see Section 1.3.3,
+      "PCIe/AXI4 Address Translation" of the PolarFire SoC PCIe User Guide:
+      https://www.microsemi.com/document-portal/doc_download/1245812-polarfire-fpga-and-polarfire-soc-fpga-pci-express-user-guide
+
+    items:
+      minItems: 4
+      maxItems: 7
+
   msi-controller:
     description: Identifies the node as an MSI controller.