mbox series

[ndctl,0/7] Add partitioning support for CXL memdevs

Message ID cover.1641233076.git.alison.schofield@intel.com
Headers show
Series Add partitioning support for CXL memdevs | expand

Message

Alison Schofield Jan. 3, 2022, 8:16 p.m. UTC
From: Alison Schofield <alison.schofield@intel.com>

To support changing partitions on CXL memdevs, first provide access
to device partitioning info. The first 4 patches add accessors to all
the partition info a CXL command parser needs in order to validate
the command. This info is added to cxl list to assist the user in
creating valid partition requests.

# cxl list -MP
[
  {
    "memdev":"mem0",
    "pmem_size":0,
    "ram_size":273535729664,
    "partition":{
      "active_volatile_capacity":273535729664,
      "active_persistent_capacity":0,
      "next_volatile_capacity":268435456,
      "next_persistent_capacity":273267294208,
      "total_capacity":273535729664,
      "volatile_only_capacity":0,
      "persistent_only_capacity":0,
      "partition_alignment":268435456
    }
  }
]

Next introduce libcxl ioctl() interfaces for the SET_PARTITION_INFO
mailbox command and the new CXL command. cxl-cli does the constraints
checking. It does not offer the IMMEDIATE mode option since we do not
have driver support for that yet.

# cxl set-partition-info
 usage: cxl set-partition-info <mem0> [<mem1>..<memN>] [<options>]

    -v, --verbose         turn on debug
    -s, --volatile_size <n>
                          next volatile partition size in bytes

Guessing that a libcxl user could send the SET_PARTITION_INFO mailbox
command outside of cxl-cli tool, so a kernel patch that disables the
immediate bit, on the receiving end of the ioctl, follows.

It may be simpler to block the immediate bit in the libcxl API today,
(and as I write this cover letter I'm wondering just how far this goes
astray ;)) However, the kernel patch to peek in the payload sets us on
the path of inspecting set-partition-info mailbox commands in the future,
when immediate mode support is required.

Testing - so far I've only tested w one memdev in a Simics env. So,
next will be growing that Simics config, using cxl_test env, and 
adding a unit test.

Alison Schofield (7):
  libcxl: add GET_PARTITION_INFO mailbox command and accessors
  libcxl: add accessors for capacity fields of the IDENTIFY command
  libcxl: apply CXL_CAPACITY_MULTIPLIER to partition alignment field
  cxl: add memdev partition information to cxl-list
  libcxl: add interfaces for SET_PARTITION_INFO mailbox command
  ndctl, util: use 'unsigned long long' type in OPT_U64 define
  cxl: add command set-partition-info

 Documentation/cxl/cxl-list.txt               |  23 ++++
 Documentation/cxl/cxl-set-partition-info.txt |  27 +++++
 Documentation/cxl/partition-description.txt  |  15 +++
 Documentation/cxl/partition-options.txt      |  19 +++
 Documentation/cxl/Makefile.am                |   3 +-
 cxl/builtin.h                                |   1 +
 cxl/lib/private.h                            |  19 +++
 cxl/libcxl.h                                 |  12 ++
 util/json.h                                  |   1 +
 util/parse-options.h                         |   2 +-
 util/size.h                                  |   1 +
 cxl/cxl.c                                    |   1 +
 cxl/lib/libcxl.c                             | 117 ++++++++++++++++++-
 cxl/lib/libcxl.sym                           |  11 ++
 cxl/list.c                                   |   5 +
 cxl/memdev.c                                 |  89 ++++++++++++++
 util/json.c                                  | 112 ++++++++++++++++++
 17 files changed, 455 insertions(+), 3 deletions(-)
 create mode 100644 Documentation/cxl/cxl-set-partition-info.txt
 create mode 100644 Documentation/cxl/partition-description.txt
 create mode 100644 Documentation/cxl/partition-options.txt

Comments

Ira Weiny Jan. 6, 2022, 8:32 p.m. UTC | #1
On Mon, Jan 03, 2022 at 12:16:11PM -0800, Schofield, Alison wrote:
> From: Alison Schofield <alison.schofield@intel.com>
 
First, thanks for taking this on!  :-D

> To support changing partitions on CXL memdevs, first provide access to device
> partitioning info.
>

How about:

Users will want to configure CXL memdevs on CXL devices which support
partitioning.  CXL provides get and set partition info mailbox command to do
this.

Add support to retrieve partition info and set partition info.

...

>
>
> The first 4 patches add accessors to all
> the partition info a CXL command parser needs in order to validate
> the command. This info is added to cxl list to assist the user in
> creating valid partition requests.

Great but what is a valid partition request?

> 
> # cxl list -MP
> [
>   {
>     "memdev":"mem0",
>     "pmem_size":0,
>     "ram_size":273535729664,
>     "partition":{
>       "active_volatile_capacity":273535729664,
>       "active_persistent_capacity":0,
>       "next_volatile_capacity":268435456,
>       "next_persistent_capacity":273267294208,
>       "total_capacity":273535729664,
>       "volatile_only_capacity":0,
>       "persistent_only_capacity":0,
>       "partition_alignment":268435456
>     }
>   }
> ]
> 
> Next introduce libcxl ioctl() interfaces for the SET_PARTITION_INFO
> mailbox command and the new CXL command. cxl-cli does the constraints
> checking. It does not offer the IMMEDIATE mode option since we do not
> have driver support for that yet.

How about something like 'cxl-cli restricts the use of the IMMEDIATE flag until
such time as the driver supports it'?

But we probably should just let the command through and rely on the driver to
do what it does...

> 
> # cxl set-partition-info
>  usage: cxl set-partition-info <mem0> [<mem1>..<memN>] [<options>]
> 
>     -v, --verbose         turn on debug
>     -s, --volatile_size <n>
>                           next volatile partition size in bytes
> 
> Guessing that
  ^^^^^^^^^^^^^
  Delete

'A libcxl user can...'

>
> a libcxl user could send the SET_PARTITION_INFO mailbox
> command outside of cxl-cli tool, so a kernel patch that disables the
> immediate bit, on the receiving end of the ioctl, follows.

cool!

> 
> It may be simpler to block the immediate bit in the libcxl API today,
> (and as I write this cover letter I'm wondering just how far this goes
> astray ;)) However, the kernel patch to peek in the payload sets us on
> the path of inspecting set-partition-info mailbox commands in the future,
> when immediate mode support is required.

I'd just delete this.  I think it is best to leave the kernel to the
enforcement and not complicate the user space.

Ira

> 
> Testing - so far I've only tested w one memdev in a Simics env. So,
> next will be growing that Simics config, using cxl_test env, and 
> adding a unit test.
> 
> Alison Schofield (7):
>   libcxl: add GET_PARTITION_INFO mailbox command and accessors
>   libcxl: add accessors for capacity fields of the IDENTIFY command
>   libcxl: apply CXL_CAPACITY_MULTIPLIER to partition alignment field
>   cxl: add memdev partition information to cxl-list
>   libcxl: add interfaces for SET_PARTITION_INFO mailbox command
>   ndctl, util: use 'unsigned long long' type in OPT_U64 define
>   cxl: add command set-partition-info
> 
>  Documentation/cxl/cxl-list.txt               |  23 ++++
>  Documentation/cxl/cxl-set-partition-info.txt |  27 +++++
>  Documentation/cxl/partition-description.txt  |  15 +++
>  Documentation/cxl/partition-options.txt      |  19 +++
>  Documentation/cxl/Makefile.am                |   3 +-
>  cxl/builtin.h                                |   1 +
>  cxl/lib/private.h                            |  19 +++
>  cxl/libcxl.h                                 |  12 ++
>  util/json.h                                  |   1 +
>  util/parse-options.h                         |   2 +-
>  util/size.h                                  |   1 +
>  cxl/cxl.c                                    |   1 +
>  cxl/lib/libcxl.c                             | 117 ++++++++++++++++++-
>  cxl/lib/libcxl.sym                           |  11 ++
>  cxl/list.c                                   |   5 +
>  cxl/memdev.c                                 |  89 ++++++++++++++
>  util/json.c                                  | 112 ++++++++++++++++++
>  17 files changed, 455 insertions(+), 3 deletions(-)
>  create mode 100644 Documentation/cxl/cxl-set-partition-info.txt
>  create mode 100644 Documentation/cxl/partition-description.txt
>  create mode 100644 Documentation/cxl/partition-options.txt
> 
> -- 
> 2.31.1
>
Alison Schofield Jan. 7, 2022, 7:44 p.m. UTC | #2
Thanks for the review Ira!
There's one question back at you wrt adding MAN page to cover letter.
Otherwise, I'm absorbing all your feedback.


On Thu, Jan 06, 2022 at 12:32:46PM -0800, Ira Weiny wrote:
> On Mon, Jan 03, 2022 at 12:16:11PM -0800, Schofield, Alison wrote:
> > From: Alison Schofield <alison.schofield@intel.com>
>  
> First, thanks for taking this on!  :-D
> 
> > To support changing partitions on CXL memdevs, first provide access to device
> > partitioning info.
> >
> 
> How about:
> 
> Users will want to configure CXL memdevs on CXL devices which support
> partitioning.  CXL provides get and set partition info mailbox command to do
> this.
> 
> Add support to retrieve partition info and set partition info.
> 
> ...
> 

Will reword, thanks.

> >
> >
> > The first 4 patches add accessors to all
> > the partition info a CXL command parser needs in order to validate
> > the command. This info is added to cxl list to assist the user in
> > creating valid partition requests.
> 
> Great but what is a valid partition request?
>

I didn't regurgitate the restrictions here in the cover letter, nor
in the patch commit logs. They are only visible in the MAN page, and
I reference the MAN page in the cxl set-partition-info command patch.
I could add a pretty version of the MAN page here to give the cover
letter more details.

Would you like to see the MAN page here?

> > 
> > # cxl list -MP
> > [
> >   {
> >     "memdev":"mem0",
> >     "pmem_size":0,
> >     "ram_size":273535729664,
> >     "partition":{
> >       "active_volatile_capacity":273535729664,
> >       "active_persistent_capacity":0,
> >       "next_volatile_capacity":268435456,
> >       "next_persistent_capacity":273267294208,
> >       "total_capacity":273535729664,
> >       "volatile_only_capacity":0,
> >       "persistent_only_capacity":0,
> >       "partition_alignment":268435456
> >     }
> >   }
> > ]
> > 
> > Next introduce libcxl ioctl() interfaces for the SET_PARTITION_INFO
> > mailbox command and the new CXL command. cxl-cli does the constraints
> > checking. It does not offer the IMMEDIATE mode option since we do not
> > have driver support for that yet.
> 
> How about something like 'cxl-cli restricts the use of the IMMEDIATE flag until
> such time as the driver supports it'?

Those words are really not true. cxl-cli doesn't restrict the flags use,
it simply does not offer the option.

> 
> But we probably should just let the command through and rely on the driver to
> do what it does...
> 
Nah. I don't think I should add -i --immediate to the command yet.

(I think we covered this in chatroom)

> > 
> > # cxl set-partition-info
> >  usage: cxl set-partition-info <mem0> [<mem1>..<memN>] [<options>]
> > 
> >     -v, --verbose         turn on debug
> >     -s, --volatile_size <n>
> >                           next volatile partition size in bytes
> > 
> > Guessing that
>   ^^^^^^^^^^^^^
>   Delete
> 
> 'A libcxl user can...'
> 
> >

Will delete.


> > a libcxl user could send the SET_PARTITION_INFO mailbox
> > command outside of cxl-cli tool, so a kernel patch that disables the
> > immediate bit, on the receiving end of the ioctl, follows.
> 
> cool!
> 
> > 
> > It may be simpler to block the immediate bit in the libcxl API today,
> > (and as I write this cover letter I'm wondering just how far this goes
> > astray ;)) However, the kernel patch to peek in the payload sets us on
> > the path of inspecting set-partition-info mailbox commands in the future,
> > when immediate mode support is required.
> 
> I'd just delete this.  I think it is best to leave the kernel to the
> enforcement and not complicate the user space.

got it.

> 
> Ira
> 
> > 
> > Testing - so far I've only tested w one memdev in a Simics env. So,
> > next will be growing that Simics config, using cxl_test env, and 
> > adding a unit test.
> > 
> > Alison Schofield (7):
> >   libcxl: add GET_PARTITION_INFO mailbox command and accessors
> >   libcxl: add accessors for capacity fields of the IDENTIFY command
> >   libcxl: apply CXL_CAPACITY_MULTIPLIER to partition alignment field
> >   cxl: add memdev partition information to cxl-list
> >   libcxl: add interfaces for SET_PARTITION_INFO mailbox command
> >   ndctl, util: use 'unsigned long long' type in OPT_U64 define
> >   cxl: add command set-partition-info
> > 
> >  Documentation/cxl/cxl-list.txt               |  23 ++++
> >  Documentation/cxl/cxl-set-partition-info.txt |  27 +++++
> >  Documentation/cxl/partition-description.txt  |  15 +++
> >  Documentation/cxl/partition-options.txt      |  19 +++
> >  Documentation/cxl/Makefile.am                |   3 +-
> >  cxl/builtin.h                                |   1 +
> >  cxl/lib/private.h                            |  19 +++
> >  cxl/libcxl.h                                 |  12 ++
> >  util/json.h                                  |   1 +
> >  util/parse-options.h                         |   2 +-
> >  util/size.h                                  |   1 +
> >  cxl/cxl.c                                    |   1 +
> >  cxl/lib/libcxl.c                             | 117 ++++++++++++++++++-
> >  cxl/lib/libcxl.sym                           |  11 ++
> >  cxl/list.c                                   |   5 +
> >  cxl/memdev.c                                 |  89 ++++++++++++++
> >  util/json.c                                  | 112 ++++++++++++++++++
> >  17 files changed, 455 insertions(+), 3 deletions(-)
> >  create mode 100644 Documentation/cxl/cxl-set-partition-info.txt
> >  create mode 100644 Documentation/cxl/partition-description.txt
> >  create mode 100644 Documentation/cxl/partition-options.txt
> > 
> > -- 
> > 2.31.1
> >