[0/5] cxl/dcd: Add support for Dynamic Capacity Devices (DCD)

Message

Ira Weiny June 14, 2023, 7:16 p.m. UTC

I'm submitting these on behalf of Navneet.  There was a round of
internal discussion which left a few questions but we want to get the
public discussion going.  A first public preview was posted by Dan.[1]

The series has been rebased on the type-2 work posted from Dan.[2]  As
discussed in the community call, not all of that series is required for
these patches.  This will get rebased on the subset of those patches he
is targeting for 6.5.  The series was tested using Fan Ni's Qemu DCD
series.[3]

[cover letter]

A Dynamic Capacity Device (DCD) (CXL 3.0 spec 9.13.3) is a CXL memory
device that implements dynamic capacity.  Dynamic capacity feature
allows memory capacity to change dynamically, without the need for
resetting the device.

Provide initial patches to enable DCD on non interleaving regions.
Details:

- Get the dynamic capacity region information from cxl device and add
  the advertised DC memory to driver managed resources
- Get the device dynamic capacity extent list from the device and
  maintain it in the host and add the preallocated memory to the host
- Dynamic capacity region support
- DCD region provisioning via Dax
- Dynamic capacity event records
        a. Add capacity Events
	b. Release capacity events
	c. Add the memory to the host dc region
	d. Release the memory from the host dc region
- Trace Dynamic Capacity events
- Send add capacity response to device
- Send release dynamic capacity to device

Cc: Navneet Singh <[e-mail]>
Cc: Fan Ni <[e-mail]>
Cc: Jonathan Cameron <[e-mail]>
Cc: Ira Weiny <[e-mail]>
Cc: Dan Williams <[e-mail]>
Cc: [e-mail]

[1] https://lore.kernel.org/all/[e-mail]/
[2] https://lore.kernel.org/all/[e-mail]/
[3] https://lore.kernel.org/all/[e-mail]/

---
Navneet Singh (5):
      cxl/mem : Read Dynamic capacity configuration from the device
      cxl/region: Add dynamic capacity cxl region support.
      cxl/mem : Expose dynamic capacity configuration to userspace
      cxl/mem: Add support to handle DCD add and release capacity events.
      cxl/mem: Trace Dynamic capacity Event Record

 drivers/cxl/Kconfig       |  11 +
 drivers/cxl/core/core.h   |   7 +
 drivers/cxl/core/hdm.c    | 234 ++++++++++++++++++--
 drivers/cxl/core/mbox.c   | 540 +++++++++++++++++++++++++++++++++++++++++++++-
 drivers/cxl/core/memdev.c |  72 +++++++
 drivers/cxl/core/port.c   |  18 ++
 drivers/cxl/core/region.c | 337 ++++++++++++++++++++++++++++-
 drivers/cxl/core/trace.h  |  68 +++++-
 drivers/cxl/cxl.h         |  32 ++-
 drivers/cxl/cxlmem.h      | 146 ++++++++++++-
 drivers/cxl/pci.c         |  14 +-
 drivers/dax/bus.c         |  11 +-
 drivers/dax/bus.h         |   5 +-
 drivers/dax/cxl.c         |   4 +
 14 files changed, 1453 insertions(+), 46 deletions(-)
---
base-commit: 034a16d0165be3e092d60685be7b1b05e6f3059b
change-id: 20230604-dcd-type2-upstream-0cd15f6216fd

Best regards,

Comments

Alison Schofield June 15, 2023, 12:56 a.m. UTC | #1

On Wed, Jun 14, 2023 at 12:16:27PM -0700, Ira Weiny wrote:

Is there a repo you can share?
If not, how about a recipe for applying these to cxl/next?
(Not trying to run, just want to load and view)

Thanks!

> I'm submitting these on behalf of Navneet.  There was a round of
> internal discussion which left a few questions but we want to get the
> public discussion going.  A first public preview was posted by Dan.[1]
> 
> The series has been rebased on the type-2 work posted from Dan.[2]  As
> discussed in the community call, not all of that series is required for
> these patches.  This will get rebased on the subset of those patches he
> is targeting for 6.5.  The series was tested using Fan Ni's Qemu DCD
> series.[3]
> 
> [cover letter]
> 
> A Dynamic Capacity Device (DCD) (CXL 3.0 spec 9.13.3) is a CXL memory
> device that implements dynamic capacity.  Dynamic capacity feature
> allows memory capacity to change dynamically, without the need for
> resetting the device.
> 
> Provide initial patches to enable DCD on non interleaving regions.
> Details:
> 
> - Get the dynamic capacity region information from cxl device and add
>   the advertised DC memory to driver managed resources
> - Get the device dynamic capacity extent list from the device and
>   maintain it in the host and add the preallocated memory to the host
> - Dynamic capacity region support
> - DCD region provisioning via Dax
> - Dynamic capacity event records
>         a. Add capacity Events
> 	b. Release capacity events
> 	c. Add the memory to the host dc region
> 	d. Release the memory from the host dc region
> - Trace Dynamic Capacity events
> - Send add capacity response to device
> - Send release dynamic capacity to device
> 
> Cc: Navneet Singh <[e-mail]>
> Cc: Fan Ni <[e-mail]>
> Cc: Jonathan Cameron <[e-mail]>
> Cc: Ira Weiny <[e-mail]>
> Cc: Dan Williams <[e-mail]>
> Cc: [e-mail]
> 
> [1] https://lore.kernel.org/all/[e-mail]/
> [2] https://lore.kernel.org/all/[e-mail]/
> [3] https://lore.kernel.org/all/[e-mail]/
> 
> ---
> Navneet Singh (5):
>       cxl/mem : Read Dynamic capacity configuration from the device
>       cxl/region: Add dynamic capacity cxl region support.
>       cxl/mem : Expose dynamic capacity configuration to userspace
>       cxl/mem: Add support to handle DCD add and release capacity events.
>       cxl/mem: Trace Dynamic capacity Event Record
> 
>  drivers/cxl/Kconfig       |  11 +
>  drivers/cxl/core/core.h   |   7 +
>  drivers/cxl/core/hdm.c    | 234 ++++++++++++++++++--
>  drivers/cxl/core/mbox.c   | 540 +++++++++++++++++++++++++++++++++++++++++++++-
>  drivers/cxl/core/memdev.c |  72 +++++++
>  drivers/cxl/core/port.c   |  18 ++
>  drivers/cxl/core/region.c | 337 ++++++++++++++++++++++++++++-
>  drivers/cxl/core/trace.h  |  68 +++++-
>  drivers/cxl/cxl.h         |  32 ++-
>  drivers/cxl/cxlmem.h      | 146 ++++++++++++-
>  drivers/cxl/pci.c         |  14 +-
>  drivers/dax/bus.c         |  11 +-
>  drivers/dax/bus.h         |   5 +-
>  drivers/dax/cxl.c         |   4 +
>  14 files changed, 1453 insertions(+), 46 deletions(-)
> ---
> base-commit: 034a16d0165be3e092d60685be7b1b05e6f3059b
> change-id: 20230604-dcd-type2-upstream-0cd15f6216fd
> 
> Best regards,
> -- 
> Ira Weiny <[e-mail]>
>

Ira Weiny June 15, 2023, 2:51 p.m. UTC | #2

ira.weiny@ wrote:
> I'm submitting these on behalf of Navneet.  There was a round of
> internal discussion which left a few questions but we want to get the
> public discussion going.  A first public preview was posted by Dan.[1]

Apologies for not being clear and marking these appropriately.  I
intended these to be RFC to get the discussion moving forward.  I somewhat
rushed the submission.  Depending on where the comments in this submission
go I'll try and make a better determination if the next submission is RFC
or can be a proper V1.  (Although b4 will mark them v2...  I'll have to
deal with that.)

Ira

> 
> The series has been rebased on the type-2 work posted from Dan.[2]  As
> discussed in the community call, not all of that series is required for
> these patches.  This will get rebased on the subset of those patches he
> is targeting for 6.5.  The series was tested using Fan Ni's Qemu DCD
> series.[3]
> 
> [cover letter]
> 
> A Dynamic Capacity Device (DCD) (CXL 3.0 spec 9.13.3) is a CXL memory
> device that implements dynamic capacity.  Dynamic capacity feature
> allows memory capacity to change dynamically, without the need for
> resetting the device.
> 
> Provide initial patches to enable DCD on non interleaving regions.
> Details:
> 
> - Get the dynamic capacity region information from cxl device and add
>   the advertised DC memory to driver managed resources
> - Get the device dynamic capacity extent list from the device and
>   maintain it in the host and add the preallocated memory to the host
> - Dynamic capacity region support
> - DCD region provisioning via Dax
> - Dynamic capacity event records
>         a. Add capacity Events
> 	b. Release capacity events
> 	c. Add the memory to the host dc region
> 	d. Release the memory from the host dc region
> - Trace Dynamic Capacity events
> - Send add capacity response to device
> - Send release dynamic capacity to device
> 
> Cc: Navneet Singh <[e-mail]>
> Cc: Fan Ni <[e-mail]>
> Cc: Jonathan Cameron <[e-mail]>
> Cc: Ira Weiny <[e-mail]>
> Cc: Dan Williams <[e-mail]>
> Cc: [e-mail]
> 
> [1] https://lore.kernel.org/all/[e-mail]/
> [2] https://lore.kernel.org/all/[e-mail]/
> [3] https://lore.kernel.org/all/[e-mail]/
> 
> ---
> Navneet Singh (5):
>       cxl/mem : Read Dynamic capacity configuration from the device
>       cxl/region: Add dynamic capacity cxl region support.
>       cxl/mem : Expose dynamic capacity configuration to userspace
>       cxl/mem: Add support to handle DCD add and release capacity events.
>       cxl/mem: Trace Dynamic capacity Event Record
> 
>  drivers/cxl/Kconfig       |  11 +
>  drivers/cxl/core/core.h   |   7 +
>  drivers/cxl/core/hdm.c    | 234 ++++++++++++++++++--
>  drivers/cxl/core/mbox.c   | 540 +++++++++++++++++++++++++++++++++++++++++++++-
>  drivers/cxl/core/memdev.c |  72 +++++++
>  drivers/cxl/core/port.c   |  18 ++
>  drivers/cxl/core/region.c | 337 ++++++++++++++++++++++++++++-
>  drivers/cxl/core/trace.h  |  68 +++++-
>  drivers/cxl/cxl.h         |  32 ++-
>  drivers/cxl/cxlmem.h      | 146 ++++++++++++-
>  drivers/cxl/pci.c         |  14 +-
>  drivers/dax/bus.c         |  11 +-
>  drivers/dax/bus.h         |   5 +-
>  drivers/dax/cxl.c         |   4 +
>  14 files changed, 1453 insertions(+), 46 deletions(-)
> ---
> base-commit: 034a16d0165be3e092d60685be7b1b05e6f3059b
> change-id: 20230604-dcd-type2-upstream-0cd15f6216fd
> 
> Best regards,
> -- 
> Ira Weiny <[e-mail]>
>

Ira Weiny June 16, 2023, 2:57 a.m. UTC | #3

Alison Schofield wrote:
> On Wed, Jun 14, 2023 at 12:16:27PM -0700, Ira Weiny wrote:
> 
> Is there a repo you can share?

:-/  I did not push this version anywhere.  I can recreate and push if you
like.  V2 will be based on Dan's new 12 patch clean up series.

https://lore.kernel.org/all/[e-mail]/

> If not, how about a recipe for applying these to cxl/next?

Sure.  Starting from v6.5-rc5

b4 shazam [e-mail]
b4 shazam [e-mail]

Will get you this branch.

> (Not trying to run, just want to load and view)
> 
> Thanks!

Sure.

Ira

Jonathan Cameron June 22, 2023, 3:07 p.m. UTC | #4

On Thu, 15 Jun 2023 07:51:16 -0700
Ira Weiny <[e-mail]> wrote:

> ira.weiny@ wrote:
> > I'm submitting these on behalf of Navneet.  There was a round of
> > internal discussion which left a few questions but we want to get the
> > public discussion going.  A first public preview was posted by Dan.[1]  
> 
> Apologies for not being clear and marking these appropriately.  I
> intended these to be RFC to get the discussion moving forward.  I somewhat
> rushed the submission.  Depending on where the comments in this submission
> go I'll try and make a better determination if the next submission is RFC
> or can be a proper V1.  (Although b4 will mark them v2...  I'll have to
> deal with that.)

Make sure your SoB is added after Navneet to reflect that you are
handling the posting to the mailing list even if you feel changes are insufficient
to merit a Co-developed-by tag. (no idea who is doing what :)

Jonathan


> 
> Ira
> 
> > 
> > The series has been rebased on the type-2 work posted from Dan.[2]  As
> > discussed in the community call, not all of that series is required for
> > these patches.  This will get rebased on the subset of those patches he
> > is targeting for 6.5.  The series was tested using Fan Ni's Qemu DCD
> > series.[3]
> > 
> > [cover letter]
> > 
> > A Dynamic Capacity Device (DCD) (CXL 3.0 spec 9.13.3) is a CXL memory
> > device that implements dynamic capacity.  Dynamic capacity feature
> > allows memory capacity to change dynamically, without the need for
> > resetting the device.
> > 
> > Provide initial patches to enable DCD on non interleaving regions.
> > Details:
> > 
> > - Get the dynamic capacity region information from cxl device and add
> >   the advertised DC memory to driver managed resources
> > - Get the device dynamic capacity extent list from the device and
> >   maintain it in the host and add the preallocated memory to the host
> > - Dynamic capacity region support
> > - DCD region provisioning via Dax
> > - Dynamic capacity event records
> >         a. Add capacity Events
> > 	b. Release capacity events
> > 	c. Add the memory to the host dc region
> > 	d. Release the memory from the host dc region
> > - Trace Dynamic Capacity events
> > - Send add capacity response to device
> > - Send release dynamic capacity to device
> > 
> > Cc: Navneet Singh <[e-mail]>
> > Cc: Fan Ni <[e-mail]>
> > Cc: Jonathan Cameron <[e-mail]>
> > Cc: Ira Weiny <[e-mail]>
> > Cc: Dan Williams <[e-mail]>
> > Cc: [e-mail]
> > 
> > [1] https://lore.kernel.org/all/[e-mail]/
> > [2] https://lore.kernel.org/all/[e-mail]/
> > [3] https://lore.kernel.org/all/[e-mail]/
> > 
> > ---
> > Navneet Singh (5):
> >       cxl/mem : Read Dynamic capacity configuration from the device
> >       cxl/region: Add dynamic capacity cxl region support.
> >       cxl/mem : Expose dynamic capacity configuration to userspace
> >       cxl/mem: Add support to handle DCD add and release capacity events.
> >       cxl/mem: Trace Dynamic capacity Event Record
> > 
> >  drivers/cxl/Kconfig       |  11 +
> >  drivers/cxl/core/core.h   |   7 +
> >  drivers/cxl/core/hdm.c    | 234 ++++++++++++++++++--
> >  drivers/cxl/core/mbox.c   | 540 +++++++++++++++++++++++++++++++++++++++++++++-
> >  drivers/cxl/core/memdev.c |  72 +++++++
> >  drivers/cxl/core/port.c   |  18 ++
> >  drivers/cxl/core/region.c | 337 ++++++++++++++++++++++++++++-
> >  drivers/cxl/core/trace.h  |  68 +++++-
> >  drivers/cxl/cxl.h         |  32 ++-
> >  drivers/cxl/cxlmem.h      | 146 ++++++++++++-
> >  drivers/cxl/pci.c         |  14 +-
> >  drivers/dax/bus.c         |  11 +-
> >  drivers/dax/bus.h         |   5 +-
> >  drivers/dax/cxl.c         |   4 +
> >  14 files changed, 1453 insertions(+), 46 deletions(-)
> > ---
> > base-commit: 034a16d0165be3e092d60685be7b1b05e6f3059b
> > change-id: 20230604-dcd-type2-upstream-0cd15f6216fd
> > 
> > Best regards,
> > -- 
> > Ira Weiny <[e-mail]>
> >   
> 
>

Jonathan Cameron June 22, 2023, 4:37 p.m. UTC | #5

On Thu, 22 Jun 2023 16:07:36 +0100
Jonathan Cameron <[e-mail]> wrote:

> On Thu, 15 Jun 2023 07:51:16 -0700
> Ira Weiny <[e-mail]> wrote:
> 
> > ira.weiny@ wrote:  
> > > I'm submitting these on behalf of Navneet.  There was a round of
> > > internal discussion which left a few questions but we want to get the
> > > public discussion going.  A first public preview was posted by Dan.[1]    
> > 
> > Apologies for not being clear and marking these appropriately.  I
> > intended these to be RFC to get the discussion moving forward.  I somewhat
> > rushed the submission.  Depending on where the comments in this submission
> > go I'll try and make a better determination if the next submission is RFC
> > or can be a proper V1.  (Although b4 will mark them v2...  I'll have to
> > deal with that.)  
> 
> Make sure your SoB is added after Navneet to reflect that you are
> handling the posting to the mailing list even if you feel changes are insufficient
> to merit a Co-developed-by tag. (no idea who is doing what :)
> 
Ah. I see there are comments on no sign off. Hmm. If it's legally fine
and you post it as an RFC only I don't see a strong argument for not
keeping that chain intact.

Jonathan

> Jonathan
> 
> 
> > 
> > Ira
> >   
> > > 
> > > The series has been rebased on the type-2 work posted from Dan.[2]  As
> > > discussed in the community call, not all of that series is required for
> > > these patches.  This will get rebased on the subset of those patches he
> > > is targeting for 6.5.  The series was tested using Fan Ni's Qemu DCD
> > > series.[3]
> > > 
> > > [cover letter]
> > > 
> > > A Dynamic Capacity Device (DCD) (CXL 3.0 spec 9.13.3) is a CXL memory
> > > device that implements dynamic capacity.  Dynamic capacity feature
> > > allows memory capacity to change dynamically, without the need for
> > > resetting the device.
> > > 
> > > Provide initial patches to enable DCD on non interleaving regions.
> > > Details:
> > > 
> > > - Get the dynamic capacity region information from cxl device and add
> > >   the advertised DC memory to driver managed resources
> > > - Get the device dynamic capacity extent list from the device and
> > >   maintain it in the host and add the preallocated memory to the host
> > > - Dynamic capacity region support
> > > - DCD region provisioning via Dax
> > > - Dynamic capacity event records
> > >         a. Add capacity Events
> > > 	b. Release capacity events
> > > 	c. Add the memory to the host dc region
> > > 	d. Release the memory from the host dc region
> > > - Trace Dynamic Capacity events
> > > - Send add capacity response to device
> > > - Send release dynamic capacity to device
> > > 
> > > Cc: Navneet Singh <[e-mail]>
> > > Cc: Fan Ni <[e-mail]>
> > > Cc: Jonathan Cameron <[e-mail]>
> > > Cc: Ira Weiny <[e-mail]>
> > > Cc: Dan Williams <[e-mail]>
> > > Cc: [e-mail]
> > > 
> > > [1] https://lore.kernel.org/all/[e-mail]/
> > > [2] https://lore.kernel.org/all/[e-mail]/
> > > [3] https://lore.kernel.org/all/[e-mail]/
> > > 
> > > ---
> > > Navneet Singh (5):
> > >       cxl/mem : Read Dynamic capacity configuration from the device
> > >       cxl/region: Add dynamic capacity cxl region support.
> > >       cxl/mem : Expose dynamic capacity configuration to userspace
> > >       cxl/mem: Add support to handle DCD add and release capacity events.
> > >       cxl/mem: Trace Dynamic capacity Event Record
> > > 
> > >  drivers/cxl/Kconfig       |  11 +
> > >  drivers/cxl/core/core.h   |   7 +
> > >  drivers/cxl/core/hdm.c    | 234 ++++++++++++++++++--
> > >  drivers/cxl/core/mbox.c   | 540 +++++++++++++++++++++++++++++++++++++++++++++-
> > >  drivers/cxl/core/memdev.c |  72 +++++++
> > >  drivers/cxl/core/port.c   |  18 ++
> > >  drivers/cxl/core/region.c | 337 ++++++++++++++++++++++++++++-
> > >  drivers/cxl/core/trace.h  |  68 +++++-
> > >  drivers/cxl/cxl.h         |  32 ++-
> > >  drivers/cxl/cxlmem.h      | 146 ++++++++++++-
> > >  drivers/cxl/pci.c         |  14 +-
> > >  drivers/dax/bus.c         |  11 +-
> > >  drivers/dax/bus.h         |   5 +-
> > >  drivers/dax/cxl.c         |   4 +
> > >  14 files changed, 1453 insertions(+), 46 deletions(-)
> > > ---
> > > base-commit: 034a16d0165be3e092d60685be7b1b05e6f3059b
> > > change-id: 20230604-dcd-type2-upstream-0cd15f6216fd
> > > 
> > > Best regards,
> > > -- 
> > > Ira Weiny <[e-mail]>
> > >     
> > 
> >   
> 
>

Ira Weiny June 27, 2023, 2:59 p.m. UTC | #6

Jonathan Cameron wrote:
> On Thu, 15 Jun 2023 07:51:16 -0700
> Ira Weiny <[e-mail]> wrote:
> 
> > ira.weiny@ wrote:
> > > I'm submitting these on behalf of Navneet.  There was a round of
> > > internal discussion which left a few questions but we want to get the
> > > public discussion going.  A first public preview was posted by Dan.[1]  
> > 
> > Apologies for not being clear and marking these appropriately.  I
> > intended these to be RFC to get the discussion moving forward.  I somewhat
> > rushed the submission.  Depending on where the comments in this submission
> > go I'll try and make a better determination if the next submission is RFC
> > or can be a proper V1.  (Although b4 will mark them v2...  I'll have to
> > deal with that.)
> 
> Make sure your SoB is added after Navneet to reflect that you are
> handling the posting to the mailing list even if you feel changes are insufficient
> to merit a Co-developed-by tag. (no idea who is doing what :)

Apologies.  I've never really posted for someone else whom I've not
developed the patches for.

V2 will have my tag on it.

Ira

> 
> Jonathan

Ira Weiny June 29, 2023, 3:30 p.m. UTC | #7

ira.weiny@ wrote:
> I'm submitting these on behalf of Navneet.  There was a round of
> internal discussion which left a few questions but we want to get the
> public discussion going.  A first public preview was posted by Dan.[1]

There has been a lot of review on this series so far.  Thank you!  At this
point a number of issues have been pointed out which is requiring some
extensive and careful rework of the series.

In the interest of saving folks time.  Any further review should focus on
the ABI or other big architectural issues.

The next version should be much improved, mostly due to the feedback we
have gotten thus far.  :-D

Thanks,
Ira