mbox series

[v1,0/4] ndctl: Add pcdctl tool with pcdctl list and reconfigure-region commands

Message ID 20210708183741.2952-1-james.sushanth.anandraj@intel.com (mailing list archive)
Headers show
Series ndctl: Add pcdctl tool with pcdctl list and reconfigure-region commands | expand

Message

James Anandraj July 8, 2021, 6:37 p.m. UTC
From: James Sushanth Anandraj <james.sushanth.anandraj@intel.com>

The Intel Optane Persistent Memory OS provisioning specification
describes how to support basic provisioning for Intel Optane
persistent memory 100 and 200 series for use in different
operating modes using OS software.

This patch set introduces a new utility pcdctl that implements
basic provisioning as described in the provisioning specification
document at https://cdrdv2.intel.com/v1/dl/getContent/634430 .

The pcdctl utility provides enumeration and region reconfiguration
commands for "nvdimm" subsystem devices (Non-volatile Memory). This
is implemented as a separate tool rather than as a feature of ndctl as
the steps for provisioning are specific to Intel Optane devices and 
are as follows.
1..Generate a new region configuration request using this utility.
2. Reset the platform.
3. Use this utility to list the status of operation.

James Sushanth Anandraj (4):
  Documentation/pcdctl: Add documentation for pcdctl tool and commands
  pcdctl/list: Add pcdctl-list command to enumerate 'nvdimm' devices
  pcdctl/reconfigure: Add pcdctl-reconfigure-region command
  pcdctl/reconfigure: Add support for pmem and iso-pmem modes

 Documentation/pcdctl/Makefile.am              |   59 +
 .../pcdctl/asciidoctor-extensions.rb          |   30 +
 Documentation/pcdctl/pcdctl-list.txt          |   56 +
 .../pcdctl/pcdctl-reconfigure-region.txt      |   50 +
 Documentation/pcdctl/pcdctl.txt               |   40 +
 Documentation/pcdctl/theory-of-operation.txt  |   28 +
 Makefile.am                                   |    4 +-
 configure.ac                                  |    2 +
 pcdctl/Makefile.am                            |   18 +
 pcdctl/builtin.h                              |    9 +
 pcdctl/list.c                                 |  114 ++
 pcdctl/list.h                                 |   11 +
 pcdctl/pcat.c                                 |   59 +
 pcdctl/pcat.h                                 |   13 +
 pcdctl/pcd.h                                  |  381 +++++
 pcdctl/pcdctl.c                               |   88 +
 pcdctl/reconfigure.c                          | 1458 +++++++++++++++++
 pcdctl/reconfigure.h                          |   12 +
 util/main.h                                   |    1 +
 19 files changed, 2431 insertions(+), 2 deletions(-)
 create mode 100644 Documentation/pcdctl/Makefile.am
 create mode 100644 Documentation/pcdctl/asciidoctor-extensions.rb
 create mode 100644 Documentation/pcdctl/pcdctl-list.txt
 create mode 100644 Documentation/pcdctl/pcdctl-reconfigure-region.txt
 create mode 100644 Documentation/pcdctl/pcdctl.txt
 create mode 100644 Documentation/pcdctl/theory-of-operation.txt
 create mode 100644 pcdctl/Makefile.am
 create mode 100644 pcdctl/builtin.h
 create mode 100644 pcdctl/list.c
 create mode 100644 pcdctl/list.h
 create mode 100644 pcdctl/pcat.c
 create mode 100644 pcdctl/pcat.h
 create mode 100644 pcdctl/pcd.h
 create mode 100644 pcdctl/pcdctl.c
 create mode 100644 pcdctl/reconfigure.c
 create mode 100644 pcdctl/reconfigure.h

Comments

Dan Williams July 8, 2021, 9:24 p.m. UTC | #1
[ add Jeff, Michal, and Adam ]

Hey ndctl distro maintainers,

Just wanted to highlight this new tool submission for your
consideration. The goal here is to have a Linux native provisioning
tool that covers the basics of the functionality that is outside of
the ACPI specification, and reduce the need for ipmctl outside of
exceptional device-specific debug scenarios. Recall that the ACPI NFIT
communicates the static region configuration to the OS, but changing
that configuration is a device-specific protocol plus a reboot. Until
the arrival of pcdctl, region provisioning required ipmctl.

I will note that CXL moves the region configuration into the base CXL
specification so the ndctl project will pick up a "cxl-cli" tool for
that purpose. In general, the ndctl project is open to carrying
support for persistent memory devices with open specifications. In
this case the provisioning specification for devices formerly driven
by ipmctl was opened up and provided here:

https://cdrdv2.intel.com/v1/dl/getContent/634430

Please comment on its suitability for shipping in distros alongside
the ndctl tool.

On Thu, Jul 8, 2021 at 11:38 AM James Anandraj
<james.sushanth.anandraj@intel.com> wrote:
>
> From: James Sushanth Anandraj <james.sushanth.anandraj@intel.com>
>
> The Intel Optane Persistent Memory OS provisioning specification
> describes how to support basic provisioning for Intel Optane
> persistent memory 100 and 200 series for use in different
> operating modes using OS software.
>
> This patch set introduces a new utility pcdctl that implements
> basic provisioning as described in the provisioning specification
> document at https://cdrdv2.intel.com/v1/dl/getContent/634430 .
>
> The pcdctl utility provides enumeration and region reconfiguration
> commands for "nvdimm" subsystem devices (Non-volatile Memory). This
> is implemented as a separate tool rather than as a feature of ndctl as
> the steps for provisioning are specific to Intel Optane devices and
> are as follows.
> 1..Generate a new region configuration request using this utility.
> 2. Reset the platform.
> 3. Use this utility to list the status of operation.
>
> James Sushanth Anandraj (4):
>   Documentation/pcdctl: Add documentation for pcdctl tool and commands
>   pcdctl/list: Add pcdctl-list command to enumerate 'nvdimm' devices
>   pcdctl/reconfigure: Add pcdctl-reconfigure-region command
>   pcdctl/reconfigure: Add support for pmem and iso-pmem modes
>
>  Documentation/pcdctl/Makefile.am              |   59 +
>  .../pcdctl/asciidoctor-extensions.rb          |   30 +
>  Documentation/pcdctl/pcdctl-list.txt          |   56 +
>  .../pcdctl/pcdctl-reconfigure-region.txt      |   50 +
>  Documentation/pcdctl/pcdctl.txt               |   40 +
>  Documentation/pcdctl/theory-of-operation.txt  |   28 +
>  Makefile.am                                   |    4 +-
>  configure.ac                                  |    2 +
>  pcdctl/Makefile.am                            |   18 +
>  pcdctl/builtin.h                              |    9 +
>  pcdctl/list.c                                 |  114 ++
>  pcdctl/list.h                                 |   11 +
>  pcdctl/pcat.c                                 |   59 +
>  pcdctl/pcat.h                                 |   13 +
>  pcdctl/pcd.h                                  |  381 +++++
>  pcdctl/pcdctl.c                               |   88 +
>  pcdctl/reconfigure.c                          | 1458 +++++++++++++++++
>  pcdctl/reconfigure.h                          |   12 +
>  util/main.h                                   |    1 +
>  19 files changed, 2431 insertions(+), 2 deletions(-)
>  create mode 100644 Documentation/pcdctl/Makefile.am
>  create mode 100644 Documentation/pcdctl/asciidoctor-extensions.rb
>  create mode 100644 Documentation/pcdctl/pcdctl-list.txt
>  create mode 100644 Documentation/pcdctl/pcdctl-reconfigure-region.txt
>  create mode 100644 Documentation/pcdctl/pcdctl.txt
>  create mode 100644 Documentation/pcdctl/theory-of-operation.txt
>  create mode 100644 pcdctl/Makefile.am
>  create mode 100644 pcdctl/builtin.h
>  create mode 100644 pcdctl/list.c
>  create mode 100644 pcdctl/list.h
>  create mode 100644 pcdctl/pcat.c
>  create mode 100644 pcdctl/pcat.h
>  create mode 100644 pcdctl/pcd.h
>  create mode 100644 pcdctl/pcdctl.c
>  create mode 100644 pcdctl/reconfigure.c
>  create mode 100644 pcdctl/reconfigure.h
>
> --
> 2.20.1
>
>
Adam Borowski July 9, 2021, 12:23 p.m. UTC | #2
On Thu, Jul 08, 2021 at 02:24:04PM -0700, Dan Williams wrote:
> [ add Jeff, Michal, and Adam ]
> 
> Hey ndctl distro maintainers,
> 
> Just wanted to highlight this new tool submission for your
> consideration. The goal here is to have a Linux native provisioning
> tool that covers the basics of the functionality that is outside of
> the ACPI specification, and reduce the need for ipmctl outside of
> exceptional device-specific debug scenarios. [...]

> I will note that CXL moves the region configuration into the base CXL
> specification so the ndctl project will pick up a "cxl-cli" tool for
> that purpose. [...]

> Please comment on its suitability for shipping in distros alongside
> the ndctl tool.

I see no issues with that.

You might want to suggest whether you prefer pcdctl and clx-cli to be
shipped in a separate binary package.


Meow!
Jeff Moyer July 9, 2021, 3:25 p.m. UTC | #3
Dan Williams <dan.j.williams@intel.com> writes:

> [ add Jeff, Michal, and Adam ]

[ adding Bryan Gurney, who is helping out with RHEL packaging ]

> Hey ndctl distro maintainers,
>
> Just wanted to highlight this new tool submission for your
> consideration. The goal here is to have a Linux native provisioning
> tool that covers the basics of the functionality that is outside of
> the ACPI specification, and reduce the need for ipmctl outside of
> exceptional device-specific debug scenarios. Recall that the ACPI NFIT
> communicates the static region configuration to the OS, but changing
> that configuration is a device-specific protocol plus a reboot. Until
> the arrival of pcdctl, region provisioning required ipmctl.

It's great to see progress on this, thanks!  Shipping another utility as
part of the ndctl package is fine with me, though I'm not sure why we
wouldn't just make this an ndctl sub-command.  From a user's
perspective, these are all operations on or about nvdimms.  ipmctl
didn't have separate utilities for provisioning goals and namespace
configuration, for example.

> I will note that CXL moves the region configuration into the base CXL
> specification so the ndctl project will pick up a "cxl-cli" tool for
> that purpose. In general, the ndctl project is open to carrying
> support for persistent memory devices with open specifications. In
> this case the provisioning specification for devices formerly driven
> by ipmctl was opened up and provided here:

Is there a meaningful difference to the user?  Can you show some
examples of how configuration would be different between cxl-attached
pmem and memory-bus attached pmem?

> https://cdrdv2.intel.com/v1/dl/getContent/634430
>
> Please comment on its suitability for shipping in distros alongside
> the ndctl tool.

It's completely fine to ship more tools with ndctl.  I would like a
better overall picture of configuration from the admin's perspective.
At first glance, I think we're adding unneeded complexity.

Cheers,
Jeff

p.s. I don't find the name 'pdctl' particularly endearing.  If we do
stick with a separate utility, I'd suggest coming up with a more
descriptive name.

>
> On Thu, Jul 8, 2021 at 11:38 AM James Anandraj
> <james.sushanth.anandraj@intel.com> wrote:
>>
>> From: James Sushanth Anandraj <james.sushanth.anandraj@intel.com>
>>
>> The Intel Optane Persistent Memory OS provisioning specification
>> describes how to support basic provisioning for Intel Optane
>> persistent memory 100 and 200 series for use in different
>> operating modes using OS software.
>>
>> This patch set introduces a new utility pcdctl that implements
>> basic provisioning as described in the provisioning specification
>> document at https://cdrdv2.intel.com/v1/dl/getContent/634430 .
>>
>> The pcdctl utility provides enumeration and region reconfiguration
>> commands for "nvdimm" subsystem devices (Non-volatile Memory). This
>> is implemented as a separate tool rather than as a feature of ndctl as
>> the steps for provisioning are specific to Intel Optane devices and
>> are as follows.
>> 1..Generate a new region configuration request using this utility.
>> 2. Reset the platform.
>> 3. Use this utility to list the status of operation.
>>
>> James Sushanth Anandraj (4):
>>   Documentation/pcdctl: Add documentation for pcdctl tool and commands
>>   pcdctl/list: Add pcdctl-list command to enumerate 'nvdimm' devices
>>   pcdctl/reconfigure: Add pcdctl-reconfigure-region command
>>   pcdctl/reconfigure: Add support for pmem and iso-pmem modes
>>
>>  Documentation/pcdctl/Makefile.am              |   59 +
>>  .../pcdctl/asciidoctor-extensions.rb          |   30 +
>>  Documentation/pcdctl/pcdctl-list.txt          |   56 +
>>  .../pcdctl/pcdctl-reconfigure-region.txt      |   50 +
>>  Documentation/pcdctl/pcdctl.txt               |   40 +
>>  Documentation/pcdctl/theory-of-operation.txt  |   28 +
>>  Makefile.am                                   |    4 +-
>>  configure.ac                                  |    2 +
>>  pcdctl/Makefile.am                            |   18 +
>>  pcdctl/builtin.h                              |    9 +
>>  pcdctl/list.c                                 |  114 ++
>>  pcdctl/list.h                                 |   11 +
>>  pcdctl/pcat.c                                 |   59 +
>>  pcdctl/pcat.h                                 |   13 +
>>  pcdctl/pcd.h                                  |  381 +++++
>>  pcdctl/pcdctl.c                               |   88 +
>>  pcdctl/reconfigure.c                          | 1458 +++++++++++++++++
>>  pcdctl/reconfigure.h                          |   12 +
>>  util/main.h                                   |    1 +
>>  19 files changed, 2431 insertions(+), 2 deletions(-)
>>  create mode 100644 Documentation/pcdctl/Makefile.am
>>  create mode 100644 Documentation/pcdctl/asciidoctor-extensions.rb
>>  create mode 100644 Documentation/pcdctl/pcdctl-list.txt
>>  create mode 100644 Documentation/pcdctl/pcdctl-reconfigure-region.txt
>>  create mode 100644 Documentation/pcdctl/pcdctl.txt
>>  create mode 100644 Documentation/pcdctl/theory-of-operation.txt
>>  create mode 100644 pcdctl/Makefile.am
>>  create mode 100644 pcdctl/builtin.h
>>  create mode 100644 pcdctl/list.c
>>  create mode 100644 pcdctl/list.h
>>  create mode 100644 pcdctl/pcat.c
>>  create mode 100644 pcdctl/pcat.h
>>  create mode 100644 pcdctl/pcd.h
>>  create mode 100644 pcdctl/pcdctl.c
>>  create mode 100644 pcdctl/reconfigure.c
>>  create mode 100644 pcdctl/reconfigure.h
>>
>> --
>> 2.20.1
>>
>>
Dan Williams July 9, 2021, 6:58 p.m. UTC | #4
On Fri, Jul 9, 2021 at 8:24 AM Jeff Moyer <jmoyer@redhat.com> wrote:
>
> Dan Williams <dan.j.williams@intel.com> writes:
>
> > [ add Jeff, Michal, and Adam ]
>
> [ adding Bryan Gurney, who is helping out with RHEL packaging ]
>
> > Hey ndctl distro maintainers,
> >
> > Just wanted to highlight this new tool submission for your
> > consideration. The goal here is to have a Linux native provisioning
> > tool that covers the basics of the functionality that is outside of
> > the ACPI specification, and reduce the need for ipmctl outside of
> > exceptional device-specific debug scenarios. Recall that the ACPI NFIT
> > communicates the static region configuration to the OS, but changing
> > that configuration is a device-specific protocol plus a reboot. Until
> > the arrival of pcdctl, region provisioning required ipmctl.
>
> It's great to see progress on this, thanks!  Shipping another utility as
> part of the ndctl package is fine with me, though I'm not sure why we
> wouldn't just make this an ndctl sub-command.  From a user's
> perspective, these are all operations on or about nvdimms.  ipmctl
> didn't have separate utilities for provisioning goals and namespace
> configuration, for example.

True, but ipmctl also did not make an attempt to support anything
other than Intel devices, and later versions abandoned the namespace
setup code in favor of "native OS" capabilities (ndctl on Linux).

The main rationale for splitting region provisioning to dedicated
tooling is the observation that region provisioning semantics are
platform specific. It is already the case that IBM devices have their
own provisioning tool with different semantics for the "PAPR" family.
CXL region provisioning semantics again are much different than what
is done for DDR-T devices (see below). So rather than try to abstract
all that under ndctl that wants to be vendor agnostic, offload that to
platform specific tools. My hope is that more tools like this do not
proliferate as the industry unifies on common standards for persistent
memory like CXL.

That said, the new commands could be placed under a
vendor/platform-specific name in ndctl, like:

ndctl list-ipm-region
ndctl reconfigure-ipm-region

...just not my first choice given the success to date of keeping
vendor details out of the command line interface of ndctl. The primary
blocker for ndctl to generic region provisioning would be a kernel
driver model for it, but I don't know how to reconcile "ipm-regions"
requiring a reboot and a BIOS validation step vs buses like CXL that
can reconfigure interleave sets at runtime.

> > I will note that CXL moves the region configuration into the base CXL
> > specification so the ndctl project will pick up a "cxl-cli" tool for
> > that purpose. In general, the ndctl project is open to carrying
> > support for persistent memory devices with open specifications. In
> > this case the provisioning specification for devices formerly driven
> > by ipmctl was opened up and provided here:
>
> Is there a meaningful difference to the user?  Can you show some
> examples of how configuration would be different between cxl-attached
> pmem and memory-bus attached pmem?

Yes, CXL exposes several more details and degrees of freedom to system
software. Before I list those I'll point out that to keep pcdctl
simple it only handles the simple / common configurations: all
performance-pmem (interleaved), all fault tolerant pmem
(non-interleaved), all volatile with memory-side-caching. Any
custom/expert configuration outside of those common cases is punted to
ipmctl. In comparison, the CXL tool will need to handle the full range
of configuration complexity.

The main difference to end users when provisioning regions on CXL is
the wider range of resources they need to consider. The CXL specific
resources include:

- Available PMEM capable address space as described by the ACPI CFMWS

- Device performance that matches the address space traffic class

- Decoder resources at each level of the hierarchy. I.e. a device may
be able to participate in 4 different interleave configurations, but
depending on the switch topology upstream of that device it may be
constrained to a smaller set.

- Volatile memory vs PMEM partitioning on the device. The NVDIMM
sub-system and ndctl will not have any responsibility for the volatile
memory side of CXL.

To me that looks like sufficient complexity to warrant a dedicated CXL
tool rather than try to find a lowest-common-denominator abstraction
that melds with ipm-regions for ndctl to drive generically. The CXL
tool will also handle firmware update and other CXL generic
functionality outside of PMEM.

> > https://cdrdv2.intel.com/v1/dl/getContent/634430
> >
> > Please comment on its suitability for shipping in distros alongside
> > the ndctl tool.
>
> It's completely fine to ship more tools with ndctl.  I would like a
> better overall picture of configuration from the admin's perspective.
> At first glance, I think we're adding unneeded complexity.

You mean the complexity of having to determine which platform region
provisioning tool to use before you can use ndctl to do the rest?

>
> Cheers,
> Jeff
>
> p.s. I don't find the name 'pdctl' particularly endearing.  If we do
> stick with a separate utility, I'd suggest coming up with a more
> descriptive name.

How about "ipmregion"? Where "ipm" is already in the wild as an
identifier for DDR-T configuration, and unlike ipmctl it only handles
the region provisioning subset?
Dan Williams July 9, 2021, 7:01 p.m. UTC | #5
On Fri, Jul 9, 2021 at 5:23 AM Adam Borowski <kilobyte@angband.pl> wrote:
>
> On Thu, Jul 08, 2021 at 02:24:04PM -0700, Dan Williams wrote:
> > [ add Jeff, Michal, and Adam ]
> >
> > Hey ndctl distro maintainers,
> >
> > Just wanted to highlight this new tool submission for your
> > consideration. The goal here is to have a Linux native provisioning
> > tool that covers the basics of the functionality that is outside of
> > the ACPI specification, and reduce the need for ipmctl outside of
> > exceptional device-specific debug scenarios. [...]
>
> > I will note that CXL moves the region configuration into the base CXL
> > specification so the ndctl project will pick up a "cxl-cli" tool for
> > that purpose. [...]
>
> > Please comment on its suitability for shipping in distros alongside
> > the ndctl tool.
>
> I see no issues with that.
>
> You might want to suggest whether you prefer pcdctl and clx-cli to be
> shipped in a separate binary package.

Yes, that was my expectation that ndctl, daxctl, pcdctl (ipmregion?),
and the 'cxl' tool would each be independent binary packages.
Jeff Moyer July 9, 2021, 7:21 p.m. UTC | #6
Dan Williams <dan.j.williams@intel.com> writes:

> On Fri, Jul 9, 2021 at 5:23 AM Adam Borowski <kilobyte@angband.pl> wrote:
>>
>> On Thu, Jul 08, 2021 at 02:24:04PM -0700, Dan Williams wrote:
>> > [ add Jeff, Michal, and Adam ]
>> >
>> > Hey ndctl distro maintainers,
>> >
>> > Just wanted to highlight this new tool submission for your
>> > consideration. The goal here is to have a Linux native provisioning
>> > tool that covers the basics of the functionality that is outside of
>> > the ACPI specification, and reduce the need for ipmctl outside of
>> > exceptional device-specific debug scenarios. [...]
>>
>> > I will note that CXL moves the region configuration into the base CXL
>> > specification so the ndctl project will pick up a "cxl-cli" tool for
>> > that purpose. [...]
>>
>> > Please comment on its suitability for shipping in distros alongside
>> > the ndctl tool.
>>
>> I see no issues with that.
>>
>> You might want to suggest whether you prefer pcdctl and clx-cli to be
>> shipped in a separate binary package.
>
> Yes, that was my expectation that ndctl, daxctl, pcdctl (ipmregion?),
> and the 'cxl' tool would each be independent binary packages.

Agreed.  We would only build and ship a particular tool on the platforms
that supported it.

Cheers,
Jeff
Michal Suchánek July 12, 2021, 2 p.m. UTC | #7
[ CC people who have some experience with the ACPI pmem ]

Hello,

On Fri, Jul 09, 2021 at 11:58:49AM -0700, Dan Williams wrote:
> On Fri, Jul 9, 2021 at 8:24 AM Jeff Moyer <jmoyer@redhat.com> wrote:
> >
> > Dan Williams <dan.j.williams@intel.com> writes:
> >
> > > [ add Jeff, Michal, and Adam ]
> >
> > [ adding Bryan Gurney, who is helping out with RHEL packaging ]
> >
> > > Hey ndctl distro maintainers,
> > >
> > > Just wanted to highlight this new tool submission for your
> > > consideration. The goal here is to have a Linux native provisioning
> > > tool that covers the basics of the functionality that is outside of
> > > the ACPI specification, and reduce the need for ipmctl outside of
> > > exceptional device-specific debug scenarios. Recall that the ACPI NFIT
> > > communicates the static region configuration to the OS, but changing
> > > that configuration is a device-specific protocol plus a reboot. Until
> > > the arrival of pcdctl, region provisioning required ipmctl.
> >
> > It's great to see progress on this, thanks!  Shipping another utility as
> > part of the ndctl package is fine with me, though I'm not sure why we
> > wouldn't just make this an ndctl sub-command.  From a user's
> > perspective, these are all operations on or about nvdimms.  ipmctl
> > didn't have separate utilities for provisioning goals and namespace
> > configuration, for example.
> 
> True, but ipmctl also did not make an attempt to support anything
> other than Intel devices, and later versions abandoned the namespace
> setup code in favor of "native OS" capabilities (ndctl on Linux).
> 
> The main rationale for splitting region provisioning to dedicated
> tooling is the observation that region provisioning semantics are
> platform specific. It is already the case that IBM devices have their
> own provisioning tool with different semantics for the "PAPR" family.
> CXL region provisioning semantics again are much different than what
> is done for DDR-T devices (see below). So rather than try to abstract
> all that under ndctl that wants to be vendor agnostic, offload that to
> platform specific tools. My hope is that more tools like this do not
> proliferate as the industry unifies on common standards for persistent
> memory like CXL.
> 
> That said, the new commands could be placed under a
> vendor/platform-specific name in ndctl, like:
> 
> ndctl list-ipm-region
> ndctl reconfigure-ipm-region
> 
> ...just not my first choice given the success to date of keeping
> vendor details out of the command line interface of ndctl. The primary
> blocker for ndctl to generic region provisioning would be a kernel
> driver model for it, but I don't know how to reconcile "ipm-regions"
> requiring a reboot and a BIOS validation step vs buses like CXL that
> can reconfigure interleave sets at runtime.
> 
> > > I will note that CXL moves the region configuration into the base CXL
> > > specification so the ndctl project will pick up a "cxl-cli" tool for
> > > that purpose. In general, the ndctl project is open to carrying
> > > support for persistent memory devices with open specifications. In
> > > this case the provisioning specification for devices formerly driven
> > > by ipmctl was opened up and provided here:
> >
> > Is there a meaningful difference to the user?  Can you show some
> > examples of how configuration would be different between cxl-attached
> > pmem and memory-bus attached pmem?
> 
> Yes, CXL exposes several more details and degrees of freedom to system
> software. Before I list those I'll point out that to keep pcdctl
> simple it only handles the simple / common configurations: all
> performance-pmem (interleaved), all fault tolerant pmem
> (non-interleaved), all volatile with memory-side-caching. Any
> custom/expert configuration outside of those common cases is punted to
> ipmctl.

What is the purpose of having the new tool when it cannot handle the
full configuration, and users still need to refer to the old tool for
some cases?

Then to make life or users simpler I vould completely skip the new
partial tool.

Other than that if the new tools provide some value but are
platform-specific it is slight annoyance to package different set of
tools depending on target architecture but it's not something that
cannot be managed.

Thanks

Michal

> In comparison, the CXL tool will need to handle the full range
> of configuration complexity.
> 
> The main difference to end users when provisioning regions on CXL is
> the wider range of resources they need to consider. The CXL specific
> resources include:
> 
> - Available PMEM capable address space as described by the ACPI CFMWS
> 
> - Device performance that matches the address space traffic class
> 
> - Decoder resources at each level of the hierarchy. I.e. a device may
> be able to participate in 4 different interleave configurations, but
> depending on the switch topology upstream of that device it may be
> constrained to a smaller set.
> 
> - Volatile memory vs PMEM partitioning on the device. The NVDIMM
> sub-system and ndctl will not have any responsibility for the volatile
> memory side of CXL.
> 
> To me that looks like sufficient complexity to warrant a dedicated CXL
> tool rather than try to find a lowest-common-denominator abstraction
> that melds with ipm-regions for ndctl to drive generically. The CXL
> tool will also handle firmware update and other CXL generic
> functionality outside of PMEM.
> 
> > > https://cdrdv2.intel.com/v1/dl/getContent/634430
> > >
> > > Please comment on its suitability for shipping in distros alongside
> > > the ndctl tool.
> >
> > It's completely fine to ship more tools with ndctl.  I would like a
> > better overall picture of configuration from the admin's perspective.
> > At first glance, I think we're adding unneeded complexity.
> 
> You mean the complexity of having to determine which platform region
> provisioning tool to use before you can use ndctl to do the rest?
> 
> >
> > Cheers,
> > Jeff
> >
> > p.s. I don't find the name 'pdctl' particularly endearing.  If we do
> > stick with a separate utility, I'd suggest coming up with a more
> > descriptive name.
> 
> How about "ipmregion"? Where "ipm" is already in the wild as an
> identifier for DDR-T configuration, and unlike ipmctl it only handles
> the region provisioning subset?
Dan Williams July 12, 2021, 8:19 p.m. UTC | #8
On Mon, Jul 12, 2021 at 7:03 AM Michal Suchánek <msuchanek@suse.de> wrote:
>
> [ CC people who have some experience with the ACPI pmem ]
>
> Hello,
>
> On Fri, Jul 09, 2021 at 11:58:49AM -0700, Dan Williams wrote:
> > On Fri, Jul 9, 2021 at 8:24 AM Jeff Moyer <jmoyer@redhat.com> wrote:
> > >
> > > Dan Williams <dan.j.williams@intel.com> writes:
> > >
> > > > [ add Jeff, Michal, and Adam ]
> > >
> > > [ adding Bryan Gurney, who is helping out with RHEL packaging ]
> > >
> > > > Hey ndctl distro maintainers,
> > > >
> > > > Just wanted to highlight this new tool submission for your
> > > > consideration. The goal here is to have a Linux native provisioning
> > > > tool that covers the basics of the functionality that is outside of
> > > > the ACPI specification, and reduce the need for ipmctl outside of
> > > > exceptional device-specific debug scenarios. Recall that the ACPI NFIT
> > > > communicates the static region configuration to the OS, but changing
> > > > that configuration is a device-specific protocol plus a reboot. Until
> > > > the arrival of pcdctl, region provisioning required ipmctl.
> > >
> > > It's great to see progress on this, thanks!  Shipping another utility as
> > > part of the ndctl package is fine with me, though I'm not sure why we
> > > wouldn't just make this an ndctl sub-command.  From a user's
> > > perspective, these are all operations on or about nvdimms.  ipmctl
> > > didn't have separate utilities for provisioning goals and namespace
> > > configuration, for example.
> >
> > True, but ipmctl also did not make an attempt to support anything
> > other than Intel devices, and later versions abandoned the namespace
> > setup code in favor of "native OS" capabilities (ndctl on Linux).
> >
> > The main rationale for splitting region provisioning to dedicated
> > tooling is the observation that region provisioning semantics are
> > platform specific. It is already the case that IBM devices have their
> > own provisioning tool with different semantics for the "PAPR" family.
> > CXL region provisioning semantics again are much different than what
> > is done for DDR-T devices (see below). So rather than try to abstract
> > all that under ndctl that wants to be vendor agnostic, offload that to
> > platform specific tools. My hope is that more tools like this do not
> > proliferate as the industry unifies on common standards for persistent
> > memory like CXL.
> >
> > That said, the new commands could be placed under a
> > vendor/platform-specific name in ndctl, like:
> >
> > ndctl list-ipm-region
> > ndctl reconfigure-ipm-region
> >
> > ...just not my first choice given the success to date of keeping
> > vendor details out of the command line interface of ndctl. The primary
> > blocker for ndctl to generic region provisioning would be a kernel
> > driver model for it, but I don't know how to reconcile "ipm-regions"
> > requiring a reboot and a BIOS validation step vs buses like CXL that
> > can reconfigure interleave sets at runtime.
> >
> > > > I will note that CXL moves the region configuration into the base CXL
> > > > specification so the ndctl project will pick up a "cxl-cli" tool for
> > > > that purpose. In general, the ndctl project is open to carrying
> > > > support for persistent memory devices with open specifications. In
> > > > this case the provisioning specification for devices formerly driven
> > > > by ipmctl was opened up and provided here:
> > >
> > > Is there a meaningful difference to the user?  Can you show some
> > > examples of how configuration would be different between cxl-attached
> > > pmem and memory-bus attached pmem?
> >
> > Yes, CXL exposes several more details and degrees of freedom to system
> > software. Before I list those I'll point out that to keep pcdctl
> > simple it only handles the simple / common configurations: all
> > performance-pmem (interleaved), all fault tolerant pmem
> > (non-interleaved), all volatile with memory-side-caching. Any
> > custom/expert configuration outside of those common cases is punted to
> > ipmctl.
>
> What is the purpose of having the new tool when it cannot handle the
> full configuration, and users still need to refer to the old tool for
> some cases?

A valid question, indeed I neglected to clarify the original
motivation to go this route. So a small bit of history: ndctl was
built to be Linux native and vendor agnostic, and ipmctl was built to
be OS agnostic and vendor specific. For that reason ipmctl was slower
to be picked up by distributions and some distributions still do not
ship it in their supported repos today.

One of the feedback items we, ndctl + ipmctl developers, heard is that
ndctl is easier for a distribution to support because of its
organization as a Linux-native open-source project, i.e. no OS
abstraction + typical open development on a public mailing list. We
also heard the feedback that ndctl was frustratingly deficient in
being able to handle all the major provisioning flows for DDR-T
devices except for "goal" / region management.

> Then to make life or users simpler I vould completely skip the new
> partial tool.

I think that's a reasonable position. If a distribution is already
comfortable shipping and supporting ipmctl then there's little need to
ship this new tool. Conversely if a distribution has not promoted
ipmctl to a supported package this new tool can fill a gap.

Again, this gets better for CXL where all provisioning is standardized
across vendors.

> Other than that if the new tools provide some value but are
> platform-specific it is slight annoyance to package different set of
> tools depending on target architecture but it's not something that
> cannot be managed.

Understood, I can see how this new tool requires a decision to be made
about whether to ignore it, or ship both for distributions that are
already carrying ipmctl. I don't have a strong opinion either way, I
just know that this option is something that some distributions may
want.