| Message ID | 20190625112752.83188-1-Jonathan.Cameron@huawei.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | qemu: CCIX pcie config space emulation | expand |
For reference alongside this patch set. Evaluation version of the CCIX 1.0a base specification now available, (though there is a form to complete and license agreement).. https://www.ccixconsortium.com/ccix-library/download-form/ Thanks, Jonathan On Tue, 25 Jun 2019 19:27:45 +0800 Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote: > CCIX topologies are 'layered' on top of PCIe tree topologies. > This is done primarily by allowing a single CCIX device to appear as > multiple disjoint nodes in the PCIe tree. > > This layering is described via extensive PCIe DVSEC extended > capabilities in PCIe config space across all the functions that > are present in the device (some placement rules apply). > > The extremely flexible nature of allowed topologies makes the > development of generic firmware and OS software difficult if we rely > on actual hardware setups, due to the large test set that is necessary. > > To enable the ongoing work on EDK2 and within the Linux kernel and > userspace, this series provides the bare bones of what is necessary > to be able to test 'configuration' of a CCIX topology. Note > that no actual CCIX data flow is being emulated within this patchset, > merely a substantial subset of the interface that allows the topologies > to be configured. Testing has to rely on verification based on > the result rather than true emulation of the coherency protocol. > (that is a very different form of emulation for which other tools > are perhaps better suited). > > For information on how to do the coherency protocol configuration, > see the forthcoming CCIX SW guide, in conjunction with the CCIX 1.0 > Base Specification. > > An example of a 2x2 mesh with a spur to the host can be run with: > > qemu-system-aarch64 -M virt -m 1024 -cpu cortex-a53 \ > ... > -device ioh3420,id=root_port1 \ > -device ccix-upstream-port,num_links=4,primaryport=true,rsam_entries=4,ccix_device="ccix_dev1",bus=root_port1,addr=0.0,multifunction="on",id=up0,port_id=0 \ > -device ccix-downstream-port,ccix_device="ccix_dev1",bus=up0,slot=0,chassis=2,id=bus_top,port_id=1 \ > -device ccix-downstream-port,ccix_device="ccix_dev1",bus=up0,slot=1,chassis=2,id=bus_left,port_id=2 \ > -device ccix-ep,primaryport=false,home_agents=1,request_agents=1,ccix_device="ccix_dev1",bus=root_port1,addr=0.1,multifunction="on" \ > -device ccix-upstream-port,num_links=4,primaryport=true,rsam_entries=4,ccix_device="ccix_dev2",bus=bus_top,addr=0.0,multifunction="on",id=up1,port_id=0 \ > -device ccix-downstream-port,ccix_device="ccix_dev2",bus=up1,slot=0,chassis=3,id=bus_right,port_id=1 \ > -device ccix-ep,primaryport=false,request_agents=2,ccix_device="ccix_dev2",bus=bus_top,addr=0.1,multifunction="on" \ > -device ccix-upstream-port,num_links=4,primaryport=true,rsam_entries=4,ccix_device="ccix_dev3",bus=bus_left,addr=0.0,multifunction="on",id=up2,port_id=0 \ > -device ccix-downstream-port,ccix_device="ccix_dev3",bus=up2,slot=0,chassis=4,id=bus_bottom,port_id=1,multifunciton="on" \ > -device ccix-ep,primaryport=false,request_agents=2,ccix_device="ccix_dev3",bus=bus_left,addr=0.1,multifunction="on" \ > -device ccix-ep,primaryport=true,request_agents=2,ccix_device="ccix_dev4",bus=bus_right,addr=0.0,port_id=0 \ > -device ccix-ep,primaryport=true,request_agents=2,ccix_device="ccix_dev4",bus=bus_bottom,addr=0.0,port_id=1 > > > I'm not going to try drawing all the detail (it's bad enough > trying to draw these in inkscape, but in a very much simplifed > fashion, this generates. > > ----------------- > | Host | > | | > --- root_port1-- > | > | > v > ----------------- --------------- > | ccix_dev1 | -------> | ccix_dev2 | > ----------------- --------------- > | | > V V > ----------------- --------------- > | ccix_dev3 | -------> | ccix_dev4 | > ----------------- --------------- > > $lspci -t > -[0000:00]-+-00.0 > +-01.0-[01-08]--+-00.0-[02-08]--+-00.0-[03-05]--+-00.0-[04-05]----00.0-[05]----00.0 > | | | \-00.1 > | | \-01.0-[06-08]--+-00.0-[07-08]----00.0-[08]----00.0 > | | \-00.1 > | \-00.1 > > RFC questions: > > 1. The nature of CCIX devices is that we need to extend normal > PCI devices, slots, and ports. I could not find an elegant way of > doing this without lots of code replication. The current solution > just exposes some internal functions from xio3130 port implementations. > Is there a better way to do this? > > 2. The association of the different PCIDevices to a given CCIX device is > currently somewhat of a hack. Can anyone suggest a cleaner solution > for this? I can improved the current implementation, but don't really > like that we basically search for all the parts whenever a cross > device implementation is needed. > > 3. Is emulation via a large number of PCIe devices like this a good > approach or is there a nicer way to handle it? Given we need to > be able to 'spread' the CCIX device configuration across multiple > PCIe functions anyway perhaps this is the most sensible approach. > > 4. I've cut and paste a 100+ lines of code from each of the xio3130_* > modules as we are also implemening PCIE up and downstream ports > and as this is a emulation only device, we might as well use the > same register set. There are various possible ways to avoid this: > * Add a library with the shared code in it. > * Add an xio3130_upstream.h header and similar to allow the CCIX > port modules to call those functions directly. > * Don't worry about the replication in the interests of keeping > the code structure simple. > > 5. I'm not that familiar with qemu 'style' yet, so pointers on structures > to use etc most welcome. > > Note that not all elements of CCIX are supported by the current implementation, > for example slave agents and error records are missing. These will follow > either in later revisions or as follow patches. We also have no actual > accelerator functions in the current design and hence no mapping to RAs. > > Only a subset of configuration constraints are currently implemented. > This will want tightenning up in future. > > As we don't have any actual chunks of the spec in here so I haven't > added the statement from the trademark grant that follows. > > Thanks, > > Jonathan > > This patch is being distributed by the CCIX Consortium, Inc. (CCIX) to > you and other parties that are paticipating (the "participants") in > qemu with the understanding that the participants will use CCIX's > name and trademark only when this patch is used in association with > qemu. > > CCIX is also distributing this patch to these participants with the > understanding that if any portion of the CCIX specification will be > used or referenced in qemu, the participants will not modify the cited > portion of the CCIX specification and will give CCIX propery copyright > attribution by including the following copyright notice with > the cited part of the CCIX specification: > "© 2019 CCIX CONSORTIUM, INC. ALL RIGHTS RESERVED." > > Jonathan Cameron (7): > Temp: Add the PCI_EXT_ID_DVSEC definition to the qemu pci_regs.h copy. > pci: Add Huawei vendor ID and Huawei Emulated CCIX Device IDs. > pci: CCIX config space emulation library. > pci-bridge: CCIX capable PCIE/CCIX switch upstream port. > pci-bridge: CCIX capable PCIE/CCIX switch downstream port > misc: CCIX endpoint function > Temp: Add to ARM64 makefiles for testing > > default-configs/arm-softmmu.mak | 3 +- > hw/misc/Kconfig | 5 + > hw/misc/Makefile.objs | 1 + > hw/misc/ccix-ep.c | 112 ++ > hw/pci-bridge/Kconfig | 5 + > hw/pci-bridge/Makefile.objs | 1 + > hw/pci-bridge/ccix_downstream.c | 222 ++++ > hw/pci-bridge/ccix_upstream.c | 197 ++++ > hw/pci/Kconfig | 3 + > hw/pci/Makefile.objs | 1 + > hw/pci/ccix_lib.c | 1299 +++++++++++++++++++++ > include/hw/misc/ccix.h | 28 + > include/hw/pci/pci_ids.h | 7 + > include/standard-headers/linux/pci_regs.h | 3 +- > 14 files changed, 1885 insertions(+), 2 deletions(-) > create mode 100644 hw/misc/ccix-ep.c > create mode 100644 hw/pci-bridge/ccix_downstream.c > create mode 100644 hw/pci-bridge/ccix_upstream.c > create mode 100644 hw/pci/ccix_lib.c > create mode 100644 include/hw/misc/ccix.h >
On Tue, 25 Jun 2019 at 12:28, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote: > > CCIX topologies are 'layered' on top of PCIe tree topologies. > This is done primarily by allowing a single CCIX device to appear as > multiple disjoint nodes in the PCIe tree. > This patch is being distributed by the CCIX Consortium, Inc. (CCIX) to > you and other parties that are paticipating (the "participants") in > qemu with the understanding that the participants will use CCIX's > name and trademark only when this patch is used in association with > qemu. > > CCIX is also distributing this patch to these participants with the > understanding that if any portion of the CCIX specification will be > used or referenced in qemu, the participants will not modify the cited > portion of the CCIX specification and will give CCIX propery copyright > attribution by including the following copyright notice with > the cited part of the CCIX specification: > "© 2019 CCIX CONSORTIUM, INC. ALL RIGHTS RESERVED." (Apologies for replying to this now quite old email, but your more recent followup email drew it to my attention.) I think that as a project, QEMU can't take patches which come with this kind of additional constraint on top of the GPL. Could you drop these extra legal clauses, please? thanks -- PMM
On Fri, 16 Aug 2019 13:59:02 +0100 Peter Maydell <peter.maydell@linaro.org> wrote: > On Tue, 25 Jun 2019 at 12:28, Jonathan Cameron > <Jonathan.Cameron@huawei.com> wrote: > > > > CCIX topologies are 'layered' on top of PCIe tree topologies. > > This is done primarily by allowing a single CCIX device to appear as > > multiple disjoint nodes in the PCIe tree. > > > This patch is being distributed by the CCIX Consortium, Inc. (CCIX) to > > you and other parties that are paticipating (the "participants") in > > qemu with the understanding that the participants will use CCIX's > > name and trademark only when this patch is used in association with > > qemu. > > > > CCIX is also distributing this patch to these participants with the > > understanding that if any portion of the CCIX specification will be > > used or referenced in qemu, the participants will not modify the cited > > portion of the CCIX specification and will give CCIX propery copyright > > attribution by including the following copyright notice with > > the cited part of the CCIX specification: > > "© 2019 CCIX CONSORTIUM, INC. ALL RIGHTS RESERVED." > > (Apologies for replying to this now quite old email, but your > more recent followup email drew it to my attention.) > > I think that as a project, QEMU can't take patches which come > with this kind of additional constraint on top of the GPL. Could > you drop these extra legal clauses, please? Jon, if you could raise this with CCIX legal that would be great. Thanks, Jonathan > > thanks > -- PMM
CCIX topologies are 'layered' on top of PCIe tree topologies. This is done primarily by allowing a single CCIX device to appear as multiple disjoint nodes in the PCIe tree. This layering is described via extensive PCIe DVSEC extended capabilities in PCIe config space across all the functions that are present in the device (some placement rules apply). The extremely flexible nature of allowed topologies makes the development of generic firmware and OS software difficult if we rely on actual hardware setups, due to the large test set that is necessary. To enable the ongoing work on EDK2 and within the Linux kernel and userspace, this series provides the bare bones of what is necessary to be able to test 'configuration' of a CCIX topology. Note that no actual CCIX data flow is being emulated within this patchset, merely a substantial subset of the interface that allows the topologies to be configured. Testing has to rely on verification based on the result rather than true emulation of the coherency protocol. (that is a very different form of emulation for which other tools are perhaps better suited). For information on how to do the coherency protocol configuration, see the forthcoming CCIX SW guide, in conjunction with the CCIX 1.0 Base Specification. An example of a 2x2 mesh with a spur to the host can be run with: qemu-system-aarch64 -M virt -m 1024 -cpu cortex-a53 \ ... -device ioh3420,id=root_port1 \ -device ccix-upstream-port,num_links=4,primaryport=true,rsam_entries=4,ccix_device="ccix_dev1",bus=root_port1,addr=0.0,multifunction="on",id=up0,port_id=0 \ -device ccix-downstream-port,ccix_device="ccix_dev1",bus=up0,slot=0,chassis=2,id=bus_top,port_id=1 \ -device ccix-downstream-port,ccix_device="ccix_dev1",bus=up0,slot=1,chassis=2,id=bus_left,port_id=2 \ -device ccix-ep,primaryport=false,home_agents=1,request_agents=1,ccix_device="ccix_dev1",bus=root_port1,addr=0.1,multifunction="on" \ -device ccix-upstream-port,num_links=4,primaryport=true,rsam_entries=4,ccix_device="ccix_dev2",bus=bus_top,addr=0.0,multifunction="on",id=up1,port_id=0 \ -device ccix-downstream-port,ccix_device="ccix_dev2",bus=up1,slot=0,chassis=3,id=bus_right,port_id=1 \ -device ccix-ep,primaryport=false,request_agents=2,ccix_device="ccix_dev2",bus=bus_top,addr=0.1,multifunction="on" \ -device ccix-upstream-port,num_links=4,primaryport=true,rsam_entries=4,ccix_device="ccix_dev3",bus=bus_left,addr=0.0,multifunction="on",id=up2,port_id=0 \ -device ccix-downstream-port,ccix_device="ccix_dev3",bus=up2,slot=0,chassis=4,id=bus_bottom,port_id=1,multifunciton="on" \ -device ccix-ep,primaryport=false,request_agents=2,ccix_device="ccix_dev3",bus=bus_left,addr=0.1,multifunction="on" \ -device ccix-ep,primaryport=true,request_agents=2,ccix_device="ccix_dev4",bus=bus_right,addr=0.0,port_id=0 \ -device ccix-ep,primaryport=true,request_agents=2,ccix_device="ccix_dev4",bus=bus_bottom,addr=0.0,port_id=1 I'm not going to try drawing all the detail (it's bad enough trying to draw these in inkscape, but in a very much simplifed fashion, this generates. ----------------- | Host | | | --- root_port1-- | | v ----------------- --------------- | ccix_dev1 | -------> | ccix_dev2 | ----------------- --------------- | | V V ----------------- --------------- | ccix_dev3 | -------> | ccix_dev4 | ----------------- --------------- $lspci -t -[0000:00]-+-00.0 +-01.0-[01-08]--+-00.0-[02-08]--+-00.0-[03-05]--+-00.0-[04-05]----00.0-[05]----00.0 | | | \-00.1 | | \-01.0-[06-08]--+-00.0-[07-08]----00.0-[08]----00.0 | | \-00.1 | \-00.1 RFC questions: 1. The nature of CCIX devices is that we need to extend normal PCI devices, slots, and ports. I could not find an elegant way of doing this without lots of code replication. The current solution just exposes some internal functions from xio3130 port implementations. Is there a better way to do this? 2. The association of the different PCIDevices to a given CCIX device is currently somewhat of a hack. Can anyone suggest a cleaner solution for this? I can improved the current implementation, but don't really like that we basically search for all the parts whenever a cross device implementation is needed. 3. Is emulation via a large number of PCIe devices like this a good approach or is there a nicer way to handle it? Given we need to be able to 'spread' the CCIX device configuration across multiple PCIe functions anyway perhaps this is the most sensible approach. 4. I've cut and paste a 100+ lines of code from each of the xio3130_* modules as we are also implemening PCIE up and downstream ports and as this is a emulation only device, we might as well use the same register set. There are various possible ways to avoid this: * Add a library with the shared code in it. * Add an xio3130_upstream.h header and similar to allow the CCIX port modules to call those functions directly. * Don't worry about the replication in the interests of keeping the code structure simple. 5. I'm not that familiar with qemu 'style' yet, so pointers on structures to use etc most welcome. Note that not all elements of CCIX are supported by the current implementation, for example slave agents and error records are missing. These will follow either in later revisions or as follow patches. We also have no actual accelerator functions in the current design and hence no mapping to RAs. Only a subset of configuration constraints are currently implemented. This will want tightenning up in future. As we don't have any actual chunks of the spec in here so I haven't added the statement from the trademark grant that follows. Thanks, Jonathan This patch is being distributed by the CCIX Consortium, Inc. (CCIX) to you and other parties that are paticipating (the "participants") in qemu with the understanding that the participants will use CCIX's name and trademark only when this patch is used in association with qemu. CCIX is also distributing this patch to these participants with the understanding that if any portion of the CCIX specification will be used or referenced in qemu, the participants will not modify the cited portion of the CCIX specification and will give CCIX propery copyright attribution by including the following copyright notice with the cited part of the CCIX specification: "© 2019 CCIX CONSORTIUM, INC. ALL RIGHTS RESERVED." Jonathan Cameron (7): Temp: Add the PCI_EXT_ID_DVSEC definition to the qemu pci_regs.h copy. pci: Add Huawei vendor ID and Huawei Emulated CCIX Device IDs. pci: CCIX config space emulation library. pci-bridge: CCIX capable PCIE/CCIX switch upstream port. pci-bridge: CCIX capable PCIE/CCIX switch downstream port misc: CCIX endpoint function Temp: Add to ARM64 makefiles for testing default-configs/arm-softmmu.mak | 3 +- hw/misc/Kconfig | 5 + hw/misc/Makefile.objs | 1 + hw/misc/ccix-ep.c | 112 ++ hw/pci-bridge/Kconfig | 5 + hw/pci-bridge/Makefile.objs | 1 + hw/pci-bridge/ccix_downstream.c | 222 ++++ hw/pci-bridge/ccix_upstream.c | 197 ++++ hw/pci/Kconfig | 3 + hw/pci/Makefile.objs | 1 + hw/pci/ccix_lib.c | 1299 +++++++++++++++++++++ include/hw/misc/ccix.h | 28 + include/hw/pci/pci_ids.h | 7 + include/standard-headers/linux/pci_regs.h | 3 +- 14 files changed, 1885 insertions(+), 2 deletions(-) create mode 100644 hw/misc/ccix-ep.c create mode 100644 hw/pci-bridge/ccix_downstream.c create mode 100644 hw/pci-bridge/ccix_upstream.c create mode 100644 hw/pci/ccix_lib.c create mode 100644 include/hw/misc/ccix.h