mbox series

[v2,00/15] CXL Region driver

Message ID 20220112234749.1965960-1-ben.widawsky@intel.com (mailing list archive)
Headers show
Series CXL Region driver | expand

Message

Ben Widawsky Jan. 12, 2022, 11:47 p.m. UTC
Major changes since v1:
- bug fixes in certain calculations for region programming
- bug fix in for_each_cxl_decoder_target
- clarify ENIW and IG from ways and granularity
- wait_for_commit bug fix
- use devm management for region removal
- remove trace points
- add basic libnvdimm connection

Original commit message follows with minor updates for correctness.

---

This patch series introduces the CXL region driver as well as associated APIs in
CXL core. The region driver enables the creation of "regions" which is a concept
defined by the CXL 2.0 specification [1]. Region verification and programming
state are owned by the cxl_region driver (implemented in the cxl_region module).
It relies on cxl_mem to determine if devices are CXL routed, and cxl_port to
actually handle the programming of the HDM decoders. Much of the region driver
is an implementation of algorithms described in the CXL Type 3 Memory Device
Software Guide [2].

The region driver will be responsible for configuring regions found on
persistent capacities in the Label Storage Area (LSA), it will also enumerate
regions configured by BIOS, usually volatile capacities, and will allow for
dynamic region creation (which can then be stored in the LSA). It is the primary
consumer of the CXL Port [3] and CXL Mem drivers introduced previously [4]. Dan
has reworked those drivers which is a requirement for this branch. A cached copy
is included in the gitlab for this project [5]. Those patches will be posted
shortly.

The patches for the region driver could be squashed. They're broken out to aid
review and because that's the order they were implemented in. My preference is
to keep those as they are.

Some things are still missing and will be worked on while these are reviewed (in
priority order):
1. Volatile regions/LSA regions (Have a plan)
2. Switch ports (Have a plan)
3. Decoder programming restrictions (No plan). The one know restriction I've
   missed is to disallow programming HDM decoders that aren't in incremental
   system physical address ranges.
4. CXL region teardown -> nd_region teardown
5. Stress testing

Here is an example of output when programming a x2 interleave region

# ./cxlctl create-region -n -a -s $((256<<20)) /sys/bus/cxl/devices/decoder0.0
[   57.564475][  T654] cxl_core:cxl_add_region:478: cxl region0.0:0: Added region0.0:0 to decoder0.0
[   57.608949][  T655] cxl_region:allocate_address_space:170: cxl_region region0.0:0: resource [mem 0x4c00000000-0x4c1fffffff]
[   57.610056][  T655] cxl_core:cxl_commit_decoder:394: cxl_port port1: decoder1.0
[   57.610056][  T655]  Base 0x0000004c00000000
[   57.610056][  T655]  Size 512M
[   57.610056][  T655]  IG 0 (256b)
[   57.610056][  T655]  ENIW 1 (x2)
[   57.610056][  T655]  TargetList: 0 1 -1 -1 -1 -1 -1 -1
[   57.613584][  T655] cxl_core:cxl_commit_decoder:394: cxl_port endpoint2: decoder2.0
[   57.613584][  T655]  Base 0x0000004c00000000
[   57.613584][  T655]  Size 512M
[   57.613584][  T655]  IG 0 (256b)
[   57.613584][  T655]  ENIW 1 (x2)
[   57.613584][  T655]  TargetList: -1 -1 -1 -1 -1 -1 -1 -1
[   57.617051][  T655] cxl_core:cxl_commit_decoder:394: cxl_port endpoint3: decoder3.0
[   57.617051][  T655]  Base 0x0000004c00000000
[   57.617051][  T655]  Size 512M
[   57.617051][  T655]  IG 0 (256b)
[   57.617051][  T655]  ENIW 1 (x2)
[   57.617051][  T655]  TargetList: -1 -1 -1 -1 -1 -1 -1 -1
[   57.619433][  T655] cxl_region region0.0:0: Bound
[   57.621435][  T655] cxl_core:cxl_bus_probe:1384: cxl_region region0.0:0: probe: 0

If you're wondering how I tested this, I've baked it into my cxlctl app [6] and
lib [7]. Eventually this will get absorbed by ndctl/cxl-cli/libcxl.


Branch can be found at gitlab [8].

---

[1]: https://www.computeexpresslink.org/download-the-specification
[2]: https://cdrdv2.intel.com/v1/dl/getContent/643805?wapkw=CXL%20memory%20device%20sw%20guide
[3]: https://lore.kernel.org/linux-cxl/20211022183709.1199701-9-ben.widawsky@intel.com/
[4]: https://lore.kernel.org/linux-cxl/20211022183709.1199701-17-ben.widawsky@intel.com/
[5]: https://gitlab.com/bwidawsk/linux/-/commit/126793e22427f7975a8f2fca373764be78012e88
[6]: https://gitlab.com/bwidawsk-cxl/cxlctl
[7]: https://gitlab.com/bwidawsk-cxl/cxl_rs
[8]: https://gitlab.com/bwidawsk/linux/-/tree/cxl_region-v2

Ben Widawsky (15):
  cxl/core: Rename find_cxl_port
  cxl/core: Track port depth
  cxl/region: Add region creation ABI
  cxl/region: Introduce concept of region configuration
  cxl/mem: Cache port created by the mem dev
  cxl/region: Introduce a cxl_region driver
  cxl/acpi: Handle address space allocation
  cxl/region: Address space allocation
  cxl/region: Implement XHB verification
  cxl/region: HB port config verification
  cxl/region: Add infrastructure for decoder programming
  cxl/region: Collect host bridge decoders
  cxl: Program decoders for regions
  cxl/pmem: Convert nvdimm bridge API to use memdev
  cxl/region: Create an nd_region

 .clang-format                                 |   3 +
 Documentation/ABI/testing/sysfs-bus-cxl       |  63 ++
 .../driver-api/cxl/memory-devices.rst         |  14 +
 drivers/cxl/Makefile                          |   2 +
 drivers/cxl/acpi.c                            |  30 +
 drivers/cxl/core/Makefile                     |   1 +
 drivers/cxl/core/core.h                       |   4 +
 drivers/cxl/core/hdm.c                        | 199 +++++
 drivers/cxl/core/pmem.c                       |  19 +-
 drivers/cxl/core/port.c                       | 132 ++-
 drivers/cxl/core/region.c                     | 525 ++++++++++++
 drivers/cxl/cxl.h                             |  48 +-
 drivers/cxl/cxlmem.h                          |   9 +
 drivers/cxl/mem.c                             |  16 +-
 drivers/cxl/pmem.c                            |   2 +-
 drivers/cxl/port.c                            |  42 +-
 drivers/cxl/region.c                          | 769 ++++++++++++++++++
 drivers/cxl/region.h                          |  47 ++
 tools/testing/cxl/Kbuild                      |   1 +
 19 files changed, 1903 insertions(+), 23 deletions(-)
 create mode 100644 drivers/cxl/core/region.c
 create mode 100644 drivers/cxl/region.c
 create mode 100644 drivers/cxl/region.h

Comments

Ben Widawsky Jan. 15, 2022, 7 p.m. UTC | #1
Dang. Responded to the wrong thread...

v3 coming. Ignore this please.

Thanks.
Ben

On 22-01-12 15:47:34, Ben Widawsky wrote:
> Major changes since v1:
> - bug fixes in certain calculations for region programming
> - bug fix in for_each_cxl_decoder_target
> - clarify ENIW and IG from ways and granularity
> - wait_for_commit bug fix
> - use devm management for region removal
> - remove trace points
> - add basic libnvdimm connection
> 
> Original commit message follows with minor updates for correctness.
> 
> ---
> 
> This patch series introduces the CXL region driver as well as associated APIs in
> CXL core. The region driver enables the creation of "regions" which is a concept
> defined by the CXL 2.0 specification [1]. Region verification and programming
> state are owned by the cxl_region driver (implemented in the cxl_region module).
> It relies on cxl_mem to determine if devices are CXL routed, and cxl_port to
> actually handle the programming of the HDM decoders. Much of the region driver
> is an implementation of algorithms described in the CXL Type 3 Memory Device
> Software Guide [2].
> 
> The region driver will be responsible for configuring regions found on
> persistent capacities in the Label Storage Area (LSA), it will also enumerate
> regions configured by BIOS, usually volatile capacities, and will allow for
> dynamic region creation (which can then be stored in the LSA). It is the primary
> consumer of the CXL Port [3] and CXL Mem drivers introduced previously [4]. Dan
> has reworked those drivers which is a requirement for this branch. A cached copy
> is included in the gitlab for this project [5]. Those patches will be posted
> shortly.
> 
> The patches for the region driver could be squashed. They're broken out to aid
> review and because that's the order they were implemented in. My preference is
> to keep those as they are.
> 
> Some things are still missing and will be worked on while these are reviewed (in
> priority order):
> 1. Volatile regions/LSA regions (Have a plan)
> 2. Switch ports (Have a plan)
> 3. Decoder programming restrictions (No plan). The one know restriction I've
>    missed is to disallow programming HDM decoders that aren't in incremental
>    system physical address ranges.
> 4. CXL region teardown -> nd_region teardown
> 5. Stress testing
> 
> Here is an example of output when programming a x2 interleave region
> 
> # ./cxlctl create-region -n -a -s $((256<<20)) /sys/bus/cxl/devices/decoder0.0
> [   57.564475][  T654] cxl_core:cxl_add_region:478: cxl region0.0:0: Added region0.0:0 to decoder0.0
> [   57.608949][  T655] cxl_region:allocate_address_space:170: cxl_region region0.0:0: resource [mem 0x4c00000000-0x4c1fffffff]
> [   57.610056][  T655] cxl_core:cxl_commit_decoder:394: cxl_port port1: decoder1.0
> [   57.610056][  T655]  Base 0x0000004c00000000
> [   57.610056][  T655]  Size 512M
> [   57.610056][  T655]  IG 0 (256b)
> [   57.610056][  T655]  ENIW 1 (x2)
> [   57.610056][  T655]  TargetList: 0 1 -1 -1 -1 -1 -1 -1
> [   57.613584][  T655] cxl_core:cxl_commit_decoder:394: cxl_port endpoint2: decoder2.0
> [   57.613584][  T655]  Base 0x0000004c00000000
> [   57.613584][  T655]  Size 512M
> [   57.613584][  T655]  IG 0 (256b)
> [   57.613584][  T655]  ENIW 1 (x2)
> [   57.613584][  T655]  TargetList: -1 -1 -1 -1 -1 -1 -1 -1
> [   57.617051][  T655] cxl_core:cxl_commit_decoder:394: cxl_port endpoint3: decoder3.0
> [   57.617051][  T655]  Base 0x0000004c00000000
> [   57.617051][  T655]  Size 512M
> [   57.617051][  T655]  IG 0 (256b)
> [   57.617051][  T655]  ENIW 1 (x2)
> [   57.617051][  T655]  TargetList: -1 -1 -1 -1 -1 -1 -1 -1
> [   57.619433][  T655] cxl_region region0.0:0: Bound
> [   57.621435][  T655] cxl_core:cxl_bus_probe:1384: cxl_region region0.0:0: probe: 0
> 
> If you're wondering how I tested this, I've baked it into my cxlctl app [6] and
> lib [7]. Eventually this will get absorbed by ndctl/cxl-cli/libcxl.
> 
> 
> Branch can be found at gitlab [8].
> 
> ---
> 
> [1]: https://www.computeexpresslink.org/download-the-specification
> [2]: https://cdrdv2.intel.com/v1/dl/getContent/643805?wapkw=CXL%20memory%20device%20sw%20guide
> [3]: https://lore.kernel.org/linux-cxl/20211022183709.1199701-9-ben.widawsky@intel.com/
> [4]: https://lore.kernel.org/linux-cxl/20211022183709.1199701-17-ben.widawsky@intel.com/
> [5]: https://gitlab.com/bwidawsk/linux/-/commit/126793e22427f7975a8f2fca373764be78012e88
> [6]: https://gitlab.com/bwidawsk-cxl/cxlctl
> [7]: https://gitlab.com/bwidawsk-cxl/cxl_rs
> [8]: https://gitlab.com/bwidawsk/linux/-/tree/cxl_region-v2
> 
> Ben Widawsky (15):
>   cxl/core: Rename find_cxl_port
>   cxl/core: Track port depth
>   cxl/region: Add region creation ABI
>   cxl/region: Introduce concept of region configuration
>   cxl/mem: Cache port created by the mem dev
>   cxl/region: Introduce a cxl_region driver
>   cxl/acpi: Handle address space allocation
>   cxl/region: Address space allocation
>   cxl/region: Implement XHB verification
>   cxl/region: HB port config verification
>   cxl/region: Add infrastructure for decoder programming
>   cxl/region: Collect host bridge decoders
>   cxl: Program decoders for regions
>   cxl/pmem: Convert nvdimm bridge API to use memdev
>   cxl/region: Create an nd_region
> 
>  .clang-format                                 |   3 +
>  Documentation/ABI/testing/sysfs-bus-cxl       |  63 ++
>  .../driver-api/cxl/memory-devices.rst         |  14 +
>  drivers/cxl/Makefile                          |   2 +
>  drivers/cxl/acpi.c                            |  30 +
>  drivers/cxl/core/Makefile                     |   1 +
>  drivers/cxl/core/core.h                       |   4 +
>  drivers/cxl/core/hdm.c                        | 199 +++++
>  drivers/cxl/core/pmem.c                       |  19 +-
>  drivers/cxl/core/port.c                       | 132 ++-
>  drivers/cxl/core/region.c                     | 525 ++++++++++++
>  drivers/cxl/cxl.h                             |  48 +-
>  drivers/cxl/cxlmem.h                          |   9 +
>  drivers/cxl/mem.c                             |  16 +-
>  drivers/cxl/pmem.c                            |   2 +-
>  drivers/cxl/port.c                            |  42 +-
>  drivers/cxl/region.c                          | 769 ++++++++++++++++++
>  drivers/cxl/region.h                          |  47 ++
>  tools/testing/cxl/Kbuild                      |   1 +
>  19 files changed, 1903 insertions(+), 23 deletions(-)
>  create mode 100644 drivers/cxl/core/region.c
>  create mode 100644 drivers/cxl/region.c
>  create mode 100644 drivers/cxl/region.h
> 
> -- 
> 2.34.1
>