[00/19] QEMU: CXL mailbox rework and features

Message

Jonathan Cameron Sept. 25, 2023, 4:11 p.m. UTC

Based on: [PATCH] hw/cxl: Fix local variable shadowing of cap_hdrs
Based on: [PATCH v2 0/3] hw/cxl: Add dummy ACPI QTG DSM
Based on: [PATCH V2] hw/pci-bridge/cxl-upstream: Add serial number extended capability support
Based on: [PATCH v3 0/4] hw/cxl: Line length reduction and related
Based on: [PATCH v6 0/3] hw/{i2c,nvme}: mctp endpoint, nvme management interface model

I'm assuming this last dependency will go via a different tree though there
is an outstanding request for tests. That equally applies to the CXL setup,
but there are lot of moving parts. I'll experiment with basic testing
of the MCTP I2C device whilst this is being reviewed.

Based on: Message ID: [e-mail]
Based on: Message ID: [e-mail]
Based on: Message ID: [e-mail]
Based on: Message ID: [e-mail]
Based on: Message ID: [e-mail]

I've been carrying most of this series on our CXL staging tree
https://gitlab.com/jic23/qemu for some time and a lot of more recent
work around Multi Head Devices and Dynamic Capacity that we need for
Linux kernel enabling are backed up behind it. Hence I reorganized my
queue to try and land this before other less 'central' features such
as CXL PMUs and arm/virt support.

This series is focused on enabling the Fabric Management (FM) aspects of
CXL to enable orchestration and tooling development and also to enable
testing of related kernel features. That requires some substantial
reworking of existing mailbox handling (improving a few things along
the way).

Features:
1. Rework the Component Command Interface (CCI) implementation to extract
   generic parts from the existing CXL Primary Mailbox (in PCI BAR
   space).
2. Add CXL Switch Mailbox CCI function support. This CXL r3.0 feature
   enables an in band PCI path to control switch configuration.
   It is basically a CXL primary mailbox in a separate PCI function.
   On many systems this would be wired to a BMC, but as it is a normal
   PCI function we can make it an option for an emulated host providing
   us with a useful interface for testing some Fabric Mnagement
   features of CXL. In particular this will be useful for tunneling
   Fabric Management commands to Type 3 memory devices (via
   MCTP over PCI VDM)
3. Via MCTP over I2C and the two CXL MCTP bindings (FM-API and
   Memory Device) provide an alternative control interface.
   For now this can only be used with an aspeed-i2c controller.
   (There are hacks to enable this controller on CXL host systems
    but they are not suitable for upstreaming - so in upstream
    it's either MCTP access, or inband access depending on
    machine used). This is primarily implemented to enable work
   on the very different Linux kernel interface for MCTP (network
   socket based) to the IOCTL interface for the Switch Mailbox CCI.
4. A set of example commands to explore how to get access to various
   information. Additional commands will be added in future series.
   This one is big enough already!  One or two bits are made up
   for now.  Fidelity may therefore improve in future patch sets.
5. Davidlohr's work on background operations (only for the CXL
   type 3 primary mailbox for now. This wasn't directly related
   to the above series but was based on the early rework and is a
   simple feature.  Background support for MCTP interfaces and the
   Switch Mailbox CCI is left for future series.

Note that we are currently only implement a simple fixed configuration
single VH switch. If anyone is interested in working on expanding that
to deal with the high degree of configurability of topology that
vPPBs bring then let me know.

A few areas that might be of interest to reviewers:
1) The target=XXX approach to linking the 'interfaces' up with the
   underlying CXL devices.
2) The walking of various PCIe structures and similar to fill in
   the data needed for some commands.  I'm still using some
   fake data for now, but ultimately all this stuff is probably
   discoverable.
3) Tunneling fun in the final patch.  That gives an idea of
   one of the more complex corners of the Fabric Management part
   of the CXL spec and perhaps best illustrates the reason we
   have so many CCI instances inside one device a they all require
   separate state and can present different available commands.

A typical QEMU command line snippet for testing the Switch Mailbox CCI.

 -object memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/t3_cxl1.raw,size=256M,align=256M \
 -object memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/t3_lsa1.raw,size=1M \
 -object memory-backend-file,id=cxl-mem2,share=on,mem-path=/tmp/t3_cxl2.raw,size=256M,align=256M \
 -object memory-backend-file,id=cxl-lsa2,share=on,mem-path=/tmp/t3_lsa2.raw,size=1M \
 -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1,hdm_for_passthrough=true \
 -device cxl-rp,port=0,bus=cxl.1,id=cxl_rp_port0,chassis=0,slot=2 \
 -device cxl-upstream,port=2,sn=1234,bus=cxl_rp_port0,id=us0,addr=0.0,multifunction=on, \
 -device cxl-switch-mailbox-cci,bus=cxl_rp_port0,addr=0.1,target=us0 \
 -device cxl-downstream,port=0,bus=us0,id=swport0,chassis=0,slot=4 \
 -device cxl-downstream,port=1,bus=us0,id=swport1,chassis=0,slot=5 \
 -device cxl-type3,bus=swport0,memdev=cxl-mem1,id=cxl-pmem1,lsa=cxl-lsa1,sn=3 \
 -device cxl-type3,bus=swport1,memdev=cxl-mem2,id=cxl-pmem2,lsa=cxl-lsa2,sn=4 \
 -machine cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=1k \
 -device i2c_mctp_cxl,bus=aspeed.i2c.bus.0,address=4,target=us0 \
 -device i2c_mctp_cxl,bus=aspeed.i2c.bus.0,address=5,target=cxl-pmem1 \
 -device i2c_mctp_cxl,bus=aspeed.i2c.bus.0,address=6,target=cxl-pmem2

A snippet for testing the i2c_mctp_cxl (on a suitable aspeed machine,
or with the hacks on gitlab.com/jic23/qemu to add an aspeed controller to
i386/pc).

 -object memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/t3_cxl1.raw,size=256M,align=256M \
 -object memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/t3_lsa1.raw,size=1M \
 -object memory-backend-file,id=cxl-mem2,share=on,mem-path=/tmp/t3_cxl2.raw,size=256M,align=256M \
 -object memory-backend-file,id=cxl-lsa2,share=on,mem-path=/tmp/t3_lsa2.raw,size=1M \
 -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1,hdm_for_passthrough=true \
 -device cxl-rp,port=0,bus=cxl.1,id=cxl_rp_port0,chassis=0,slot=2 \
 -device cxl-upstream,port=2,sn=1234,bus=cxl_rp_port0,id=us0,addr=0.0,multifunction=on, \
 -device cxl-downstream,port=0,bus=us0,id=swport0,chassis=0,slot=4 \
 -device cxl-downstream,port=1,bus=us0,id=swport1,chassis=0,slot=5 \
 -device cxl-type3,bus=swport0,memdev=cxl-mem1,id=cxl-pmem1,lsa=cxl-lsa1,sn=3 \
 -device cxl-type3,bus=swport1,memdev=cxl-mem2,id=cxl-pmem2,lsa=cxl-lsa2,sn=4 \
 -device i2c_mctp_cxl,bus=aspeed.i2c.bus.0,address=4,target=us0 \
 -device i2c_mctp_cxl,bus=aspeed.i2c.bus.0,address=5,target=cxl-pmem1 \
 -device i2c_mctp_cxl,bus=aspeed.i2c.bus.0,address=6,target=cxl-pmem2

Rather than ending up with a very long cover letter, I'll post
where to find test userspace code to talk to the switch-cci and the
i2c_mctp_cxl device + tunnel to the devices below them separately
along with pointers to the kernel series needed (for the switch
mailbox CCI).

I'll shortly push out a fresh tree including dependencies for this set
and everything we currently have on top of it at

gitlab.com/jic23/qemu cxl-2023-09-26

Davidlohr Bueso (3):
  hw/cxl/mbox: Add support for background operations
  hw/cxl/mbox: Wire up interrupts for background completion
  hw/cxl: Add support for device sanitation

Gregory Price (1):
  hw/cxl/type3: Cleanup multiple CXL_TYPE3() calls in read/write
    functions

Jonathan Cameron (15):
  hw/cxl/mbox: Pull the payload out of struct cxl_cmd and make instances
    constant
  hw/cxl/mbox: Split mailbox command payload into separate input and
    output
  hw/cxl/mbox: Pull the CCI definition out of the CXLDeviceState
  hw/cxl/mbox: Generalize the CCI command processing
  hw/pci-bridge/cxl_upstream: Move defintion of device to header.
  hw/cxl/i2c_mctp_cxl: Initial device emulation
  hw/cxl/mbox: Add Information and Status / Identify command
  docs: cxl: Add example commandline for MCTP CXL CCIs
  hw/cxl/mbox: Add Physical Switch Identify command.
  hw/cxl: Add a switch mailbox CCI function
  hw/pci-bridge/cxl_downstream: Set default link width and link speed
  hw/cxl: Implement Physical Ports status retrieval
  hw/cxl/mbox: Add Get Background Operation Status Command
  hw/cxl: Add dummy security state get
  hw/cxl: Add tunneled command support to mailbox for switch cci/mctp.

 docs/system/devices/cxl.rst               |   27 +
 include/hw/cxl/cxl.h                      |    6 +
 include/hw/cxl/cxl_component.h            |    3 +-
 include/hw/cxl/cxl_device.h               |  110 ++-
 include/hw/pci-bridge/cxl_upstream_port.h |   20 +
 hw/cxl/cxl-device-utils.c                 |  120 ++-
 hw/cxl/cxl-events.c                       |    2 +-
 hw/cxl/cxl-mailbox-utils.c                | 1038 ++++++++++++++++++---
 hw/cxl/i2c_mctp_cxl.c                     |  290 ++++++
 hw/cxl/switch-mailbox-cci.c               |  111 +++
 hw/mem/cxl_type3.c                        |   32 +-
 hw/pci-bridge/cxl_downstream.c            |   18 +-
 hw/pci-bridge/cxl_upstream.c              |   11 +-
 hw/cxl/Kconfig                            |    4 +
 hw/cxl/meson.build                        |    2 +
 15 files changed, 1609 insertions(+), 185 deletions(-)
 create mode 100644 include/hw/pci-bridge/cxl_upstream_port.h
 create mode 100644 hw/cxl/i2c_mctp_cxl.c
 create mode 100644 hw/cxl/switch-mailbox-cci.c

Comments

Jonathan Cameron Sept. 25, 2023, 4:50 p.m. UTC | #1

On Mon, 25 Sep 2023 17:11:05 +0100
Jonathan Cameron <[e-mail]> wrote:

> Based on: [PATCH] hw/cxl: Fix local variable shadowing of cap_hdrs
> Based on: [PATCH v2 0/3] hw/cxl: Add dummy ACPI QTG DSM
Missed one. 
Based on: [PATCH v4 0/4] hw/cxl: Support emulating 4 HDM decoders throughout topology

> Based on: [PATCH V2] hw/pci-bridge/cxl-upstream: Add serial number extended capability support
> Based on: [PATCH v3 0/4] hw/cxl: Line length reduction and related
> Based on: [PATCH v6 0/3] hw/{i2c,nvme}: mctp endpoint, nvme management interface model
> 
> I'm assuming this last dependency will go via a different tree though there
> is an outstanding request for tests. That equally applies to the CXL setup,
> but there are lot of moving parts. I'll experiment with basic testing
> of the MCTP I2C device whilst this is being reviewed.
> 
> Based on: Message ID: [e-mail]
Based on: Message ID: [e-mail]
> Based on: Message ID: [e-mail]
> Based on: Message ID: [e-mail]
> Based on: Message ID: [e-mail]
And to match above, the above should have been first.

> Based on: Message ID: [e-mail]

So should be
Based on: [PATCH] hw/cxl: Fix local variable shadowing of cap_hdrs
Based on: [PATCH v2 0/3] hw/cxl: Add dummy ACPI QTG DSM
Based on: [PATCH v4 0/4] hw/cxl: Support emulating 4 HDM decoders throughout topology
Based on: [PATCH V2] hw/pci-bridge/cxl-upstream: Add serial number extended capability support
Based on: [PATCH v3 0/4] hw/cxl: Line length reduction and related
Based on: [PATCH v6 0/3] hw/{i2c,nvme}: mctp endpoint, nvme management interface model

Based on: Message ID: [e-mail]
Based on: Message ID: [e-mail]
Based on: Message ID: [e-mail]
Based on: Message ID: [e-mail]
Based on: Message ID: [e-mail]
Based on: Message ID: [e-mail]

Sorry about that.

Jonathan

Gregory Price Sept. 28, 2023, 6:12 p.m. UTC | #2

On Mon, Sep 25, 2023 at 05:11:05PM +0100, Jonathan Cameron wrote:
> I've been carrying most of this series on our CXL staging tree
> https://gitlab.com/jic23/qemu for some time and a lot of more recent
> work around Multi Head Devices and Dynamic Capacity that we need for
> Linux kernel enabling are backed up behind it. Hence I reorganized my
> queue to try and land this before other less 'central' features such
> as CXL PMUs and arm/virt support.
... 
> 
> Jonathan Cameron (15):
>   hw/cxl/mbox: Pull the payload out of struct cxl_cmd and make instances
>     constant
>   hw/cxl/mbox: Split mailbox command payload into separate input and
>     output
>   hw/cxl/mbox: Pull the CCI definition out of the CXLDeviceState
>   hw/cxl/mbox: Generalize the CCI command processing
>   hw/pci-bridge/cxl_upstream: Move defintion of device to header.

To save some list spam, I can't say i've reviewed and tested the entire
set, but this patch series to help model the Niagara work so please add
my tags as appropriate to the above.

Reviewed-by: Gregory Price <[e-mail]>
Tested-by: Gregory Price <[e-mail]>

~Gregory