mbox series

[v7,00/12] Enabling DCD emulation support in Qemu

Message ID 20240418232902.583744-1-fan.ni@samsung.com (mailing list archive)
Headers show
Series Enabling DCD emulation support in Qemu | expand

Message

fan April 18, 2024, 11:10 p.m. UTC 
A git tree of this series can be found here (with one extra commit on top
for printing out accepted/pending extent list): 
https://github.com/moking/qemu/tree/dcd-v7

v6->v7:

1. Fixed the dvsec range register issue mentioned in the the cover letter in v6.
   Only relevant bits are set to mark the device ready (Patch 6). (Jonathan)
2. Moved the if statement in cxl_setup_memory from Patch 6 to Patch 4. (Jonathan)
3. Used MIN instead of if statement to get record_count in Patch 7. (Jonathan)
4. Added "Reviewed-by" tag to Patch 7.
5. Modified cxl_dc_extent_release_dry_run so the updated extent list can be
   reused in cmd_dcd_release_dyn_cap to simplify the process in Patch 8. (Jørgen) 
6. Added comments to indicate further "TODO" items in cmd_dcd_add_dyn_cap_rsp.
    (Jonathan)
7. Avoided irrelevant code reformat in Patch 8. (Jonathan)
8. Modified QMP interfaces for adding/releasing DC extents to allow passing
   tags, selection policy, flags in the interface. (Jonathan, Gregory)
9. Redesigned the pending list so extents in the same requests are grouped
    together. A new data structure is introduced to represent "extent group"
    in pending list.  (Jonathan)
10. Added support in QMP interface for "More" flag. 
11. Check "Forced removal" flag for release request and not let it pass through.
12. Removed the dynamic capacity log type from CxlEventLog definition in cxl.json
   to avoid the side effect it may introduce to inject error to DC event log.
   (Jonathan)
13. Hard coded the event log type to dynamic capacity event log in QMP
    interfaces. (Jonathan)
14. Adding space in between "-1]". (Jonathan)
15. Some minor comment fixes.

The code is tested with similar setup and has passed similar tests as listed
in the cover letter of v5[1] and v6[2].
Also, the code is tested with the latest DCD kernel patchset[3].

[1] Qemu DCD patchset v5: https://lore.kernel.org/linux-cxl/20240304194331.1586191-1-nifan.cxl@gmail.com/T/#t
[2] Qemu DCD patchset v6: https://lore.kernel.org/linux-cxl/20240325190339.696686-1-nifan.cxl@gmail.com/T/#t
[3] DCD kernel patches: https://lore.kernel.org/linux-cxl/20240324-dcd-type2-upstream-v1-0-b7b00d623625@intel.com/T/#m11c571e21c4fe17c7d04ec5c2c7bc7cbf2cd07e3


Fan Ni (12):
  hw/cxl/cxl-mailbox-utils: Add dc_event_log_size field to output
    payload of identify memory device command
  hw/cxl/cxl-mailbox-utils: Add dynamic capacity region representative
    and mailbox command support
  include/hw/cxl/cxl_device: Rename mem_size as static_mem_size for
    type3 memory devices
  hw/mem/cxl_type3: Add support to create DC regions to type3 memory
    devices
  hw/mem/cxl-type3: Refactor ct3_build_cdat_entries_for_mr to take mr
    size instead of mr as argument
  hw/mem/cxl_type3: Add host backend and address space handling for DC
    regions
  hw/mem/cxl_type3: Add DC extent list representative and get DC extent
    list mailbox support
  hw/cxl/cxl-mailbox-utils: Add mailbox commands to support add/release
    dynamic capacity response
  hw/cxl/events: Add qmp interfaces to add/release dynamic capacity
    extents
  hw/mem/cxl_type3: Add DPA range validation for accesses to DC regions
  hw/cxl/cxl-mailbox-utils: Add superset extent release mailbox support
  hw/mem/cxl_type3: Allow to release extent superset in QMP interface

 hw/cxl/cxl-mailbox-utils.c  | 620 ++++++++++++++++++++++++++++++++++-
 hw/mem/cxl_type3.c          | 633 +++++++++++++++++++++++++++++++++---
 hw/mem/cxl_type3_stubs.c    |  20 ++
 include/hw/cxl/cxl_device.h |  81 ++++-
 include/hw/cxl/cxl_events.h |  18 +
 qapi/cxl.json               |  69 ++++
 6 files changed, 1396 insertions(+), 45 deletions(-)

Comments

Gregory Price April 19, 2024, 6:24 p.m. UTC | #1 
On Thu, Apr 18, 2024 at 04:10:51PM -0700, nifan.cxl@gmail.com wrote:
> A git tree of this series can be found here (with one extra commit on top
> for printing out accepted/pending extent list): 
> https://github.com/moking/qemu/tree/dcd-v7
> 
> v6->v7:
> 
> 1. Fixed the dvsec range register issue mentioned in the the cover letter in v6.
>    Only relevant bits are set to mark the device ready (Patch 6). (Jonathan)
> 2. Moved the if statement in cxl_setup_memory from Patch 6 to Patch 4. (Jonathan)
> 3. Used MIN instead of if statement to get record_count in Patch 7. (Jonathan)
> 4. Added "Reviewed-by" tag to Patch 7.
> 5. Modified cxl_dc_extent_release_dry_run so the updated extent list can be
>    reused in cmd_dcd_release_dyn_cap to simplify the process in Patch 8. (Jørgen) 
> 6. Added comments to indicate further "TODO" items in cmd_dcd_add_dyn_cap_rsp.
>     (Jonathan)
> 7. Avoided irrelevant code reformat in Patch 8. (Jonathan)
> 8. Modified QMP interfaces for adding/releasing DC extents to allow passing
>    tags, selection policy, flags in the interface. (Jonathan, Gregory)
> 9. Redesigned the pending list so extents in the same requests are grouped
>     together. A new data structure is introduced to represent "extent group"
>     in pending list.  (Jonathan)
> 10. Added support in QMP interface for "More" flag. 
> 11. Check "Forced removal" flag for release request and not let it pass through.
> 12. Removed the dynamic capacity log type from CxlEventLog definition in cxl.json
>    to avoid the side effect it may introduce to inject error to DC event log.
>    (Jonathan)
> 13. Hard coded the event log type to dynamic capacity event log in QMP
>     interfaces. (Jonathan)
> 14. Adding space in between "-1]". (Jonathan)
> 15. Some minor comment fixes.
> 
> The code is tested with similar setup and has passed similar tests as listed
> in the cover letter of v5[1] and v6[2].
> Also, the code is tested with the latest DCD kernel patchset[3].
> 
> [1] Qemu DCD patchset v5: https://lore.kernel.org/linux-cxl/20240304194331.1586191-1-nifan.cxl@gmail.com/T/#t
> [2] Qemu DCD patchset v6: https://lore.kernel.org/linux-cxl/20240325190339.696686-1-nifan.cxl@gmail.com/T/#t
> [3] DCD kernel patches: https://lore.kernel.org/linux-cxl/20240324-dcd-type2-upstream-v1-0-b7b00d623625@intel.com/T/#m11c571e21c4fe17c7d04ec5c2c7bc7cbf2cd07e3
>

added review to all patches, will hopefully be able to add a Tested-by
tag early next week, along with a v1 RFC for MHD bit-tracking.

We've been testing v5/v6 for a bit, so I expect as soon as we get the
MHD code ported over to v7 i'll ship a tested-by tag pretty quick.

The super-set release will complicate a few things but this doesn't
look like a blocker on our end, just a change to how we track bits in a
shared bit/bytemap.

> 
> Fan Ni (12):
>   hw/cxl/cxl-mailbox-utils: Add dc_event_log_size field to output
>     payload of identify memory device command
>   hw/cxl/cxl-mailbox-utils: Add dynamic capacity region representative
>     and mailbox command support
>   include/hw/cxl/cxl_device: Rename mem_size as static_mem_size for
>     type3 memory devices
>   hw/mem/cxl_type3: Add support to create DC regions to type3 memory
>     devices
>   hw/mem/cxl-type3: Refactor ct3_build_cdat_entries_for_mr to take mr
>     size instead of mr as argument
>   hw/mem/cxl_type3: Add host backend and address space handling for DC
>     regions
>   hw/mem/cxl_type3: Add DC extent list representative and get DC extent
>     list mailbox support
>   hw/cxl/cxl-mailbox-utils: Add mailbox commands to support add/release
>     dynamic capacity response
>   hw/cxl/events: Add qmp interfaces to add/release dynamic capacity
>     extents
>   hw/mem/cxl_type3: Add DPA range validation for accesses to DC regions
>   hw/cxl/cxl-mailbox-utils: Add superset extent release mailbox support
>   hw/mem/cxl_type3: Allow to release extent superset in QMP interface
> 
>  hw/cxl/cxl-mailbox-utils.c  | 620 ++++++++++++++++++++++++++++++++++-
>  hw/mem/cxl_type3.c          | 633 +++++++++++++++++++++++++++++++++---
>  hw/mem/cxl_type3_stubs.c    |  20 ++
>  include/hw/cxl/cxl_device.h |  81 ++++-
>  include/hw/cxl/cxl_events.h |  18 +
>  qapi/cxl.json               |  69 ++++
>  6 files changed, 1396 insertions(+), 45 deletions(-)
> 
> -- 
> 2.43.0
>
fan April 19, 2024, 6:43 p.m. UTC | #2 
On Fri, Apr 19, 2024 at 02:24:36PM -0400, Gregory Price wrote:
> On Thu, Apr 18, 2024 at 04:10:51PM -0700, nifan.cxl@gmail.com wrote:
> > A git tree of this series can be found here (with one extra commit on top
> > for printing out accepted/pending extent list): 
> > https://github.com/moking/qemu/tree/dcd-v7
> > 
> > v6->v7:
> > 
> > 1. Fixed the dvsec range register issue mentioned in the the cover letter in v6.
> >    Only relevant bits are set to mark the device ready (Patch 6). (Jonathan)
> > 2. Moved the if statement in cxl_setup_memory from Patch 6 to Patch 4. (Jonathan)
> > 3. Used MIN instead of if statement to get record_count in Patch 7. (Jonathan)
> > 4. Added "Reviewed-by" tag to Patch 7.
> > 5. Modified cxl_dc_extent_release_dry_run so the updated extent list can be
> >    reused in cmd_dcd_release_dyn_cap to simplify the process in Patch 8. (Jørgen) 
> > 6. Added comments to indicate further "TODO" items in cmd_dcd_add_dyn_cap_rsp.
> >     (Jonathan)
> > 7. Avoided irrelevant code reformat in Patch 8. (Jonathan)
> > 8. Modified QMP interfaces for adding/releasing DC extents to allow passing
> >    tags, selection policy, flags in the interface. (Jonathan, Gregory)
> > 9. Redesigned the pending list so extents in the same requests are grouped
> >     together. A new data structure is introduced to represent "extent group"
> >     in pending list.  (Jonathan)
> > 10. Added support in QMP interface for "More" flag. 
> > 11. Check "Forced removal" flag for release request and not let it pass through.
> > 12. Removed the dynamic capacity log type from CxlEventLog definition in cxl.json
> >    to avoid the side effect it may introduce to inject error to DC event log.
> >    (Jonathan)
> > 13. Hard coded the event log type to dynamic capacity event log in QMP
> >     interfaces. (Jonathan)
> > 14. Adding space in between "-1]". (Jonathan)
> > 15. Some minor comment fixes.
> > 
> > The code is tested with similar setup and has passed similar tests as listed
> > in the cover letter of v5[1] and v6[2].
> > Also, the code is tested with the latest DCD kernel patchset[3].
> > 
> > [1] Qemu DCD patchset v5: https://lore.kernel.org/linux-cxl/20240304194331.1586191-1-nifan.cxl@gmail.com/T/#t
> > [2] Qemu DCD patchset v6: https://lore.kernel.org/linux-cxl/20240325190339.696686-1-nifan.cxl@gmail.com/T/#t
> > [3] DCD kernel patches: https://lore.kernel.org/linux-cxl/20240324-dcd-type2-upstream-v1-0-b7b00d623625@intel.com/T/#m11c571e21c4fe17c7d04ec5c2c7bc7cbf2cd07e3
> >
> 
> added review to all patches, will hopefully be able to add a Tested-by
> tag early next week, along with a v1 RFC for MHD bit-tracking.
> 
> We've been testing v5/v6 for a bit, so I expect as soon as we get the
> MHD code ported over to v7 i'll ship a tested-by tag pretty quick.
> 
> The super-set release will complicate a few things but this doesn't
> look like a blocker on our end, just a change to how we track bits in a
> shared bit/bytemap.
> 

Hi Gregory,
Thanks for reviewing the patches so quickly. 

No pressure, but look forward to your MHD work. :)

Fan

> > 
> > Fan Ni (12):
> >   hw/cxl/cxl-mailbox-utils: Add dc_event_log_size field to output
> >     payload of identify memory device command
> >   hw/cxl/cxl-mailbox-utils: Add dynamic capacity region representative
> >     and mailbox command support
> >   include/hw/cxl/cxl_device: Rename mem_size as static_mem_size for
> >     type3 memory devices
> >   hw/mem/cxl_type3: Add support to create DC regions to type3 memory
> >     devices
> >   hw/mem/cxl-type3: Refactor ct3_build_cdat_entries_for_mr to take mr
> >     size instead of mr as argument
> >   hw/mem/cxl_type3: Add host backend and address space handling for DC
> >     regions
> >   hw/mem/cxl_type3: Add DC extent list representative and get DC extent
> >     list mailbox support
> >   hw/cxl/cxl-mailbox-utils: Add mailbox commands to support add/release
> >     dynamic capacity response
> >   hw/cxl/events: Add qmp interfaces to add/release dynamic capacity
> >     extents
> >   hw/mem/cxl_type3: Add DPA range validation for accesses to DC regions
> >   hw/cxl/cxl-mailbox-utils: Add superset extent release mailbox support
> >   hw/mem/cxl_type3: Allow to release extent superset in QMP interface
> > 
> >  hw/cxl/cxl-mailbox-utils.c  | 620 ++++++++++++++++++++++++++++++++++-
> >  hw/mem/cxl_type3.c          | 633 +++++++++++++++++++++++++++++++++---
> >  hw/mem/cxl_type3_stubs.c    |  20 ++
> >  include/hw/cxl/cxl_device.h |  81 ++++-
> >  include/hw/cxl/cxl_events.h |  18 +
> >  qapi/cxl.json               |  69 ++++
> >  6 files changed, 1396 insertions(+), 45 deletions(-)
> > 
> > -- 
> > 2.43.0
> >
Gregory Price April 20, 2024, 8:35 p.m. UTC | #3 
On Fri, Apr 19, 2024 at 11:43:14AM -0700, fan wrote:
> On Fri, Apr 19, 2024 at 02:24:36PM -0400, Gregory Price wrote:
> > 
> > added review to all patches, will hopefully be able to add a Tested-by
> > tag early next week, along with a v1 RFC for MHD bit-tracking.
> > 
> > We've been testing v5/v6 for a bit, so I expect as soon as we get the
> > MHD code ported over to v7 i'll ship a tested-by tag pretty quick.
> > 
> > The super-set release will complicate a few things but this doesn't
> > look like a blocker on our end, just a change to how we track bits in a
> > shared bit/bytemap.
> > 
> 
> Hi Gregory,
> Thanks for reviewing the patches so quickly. 
> 
> No pressure, but look forward to your MHD work. :)
> 
> Fan

Starting to get into versioniong hell a bit, since the Niagara work was
based off of jonathan's branch and the mhd-dcd work needs some of the
extentions from that branch - while this branch is based on master.

Probably we'll need to wait for a new cxl dated branch to try and sus
out the pain points before we push an RFC.  I would not want to have
conflicting commits for something like this for example:

https://lore.kernel.org/qemu-devel/20230901012914.226527-2-gregory.price@memverge.com/

We get merge conflicts here because this is behind that patch. So
pushing up an RFC in this state would be mostly useless to everyone.

~Gregory
Jonathan Cameron April 22, 2024, 12:04 p.m. UTC | #4 
On Sat, 20 Apr 2024 16:35:46 -0400
Gregory Price <gregory.price@memverge.com> wrote:

> On Fri, Apr 19, 2024 at 11:43:14AM -0700, fan wrote:
> > On Fri, Apr 19, 2024 at 02:24:36PM -0400, Gregory Price wrote:  
> > > 
> > > added review to all patches, will hopefully be able to add a Tested-by
> > > tag early next week, along with a v1 RFC for MHD bit-tracking.
> > > 
> > > We've been testing v5/v6 for a bit, so I expect as soon as we get the
> > > MHD code ported over to v7 i'll ship a tested-by tag pretty quick.
> > > 
> > > The super-set release will complicate a few things but this doesn't
> > > look like a blocker on our end, just a change to how we track bits in a
> > > shared bit/bytemap.
> > >   
> > 
> > Hi Gregory,
> > Thanks for reviewing the patches so quickly. 
> > 
> > No pressure, but look forward to your MHD work. :)
> > 
> > Fan  
> 
> Starting to get into versioniong hell a bit, since the Niagara work was
> based off of jonathan's branch and the mhd-dcd work needs some of the
> extentions from that branch - while this branch is based on master.
> 
> Probably we'll need to wait for a new cxl dated branch to try and sus
> out the pain points before we push an RFC.  I would not want to have
> conflicting commits for something like this for example:
> 
> https://lore.kernel.org/qemu-devel/20230901012914.226527-2-gregory.price@memverge.com/
> 
> We get merge conflicts here because this is behind that patch. So
> pushing up an RFC in this state would be mostly useless to everyone

Subtle hint noted ;) 

I'll build a fresh tree - any remaining rebases until QEMU 9.0 should be
straight forward anyway.   My ideal is that the NUMA GP series lands early
in 9.1 cycle and this can go in parallel.  I'd really like to
get this in early if possible so we can start clearing some of the other
stuff that ended up built on top of it!

Jonathan

> 
> ~Gregory
Jonathan Cameron April 22, 2024, 2:23 p.m. UTC | #5 
On Mon, 22 Apr 2024 13:04:48 +0100
Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:

> On Sat, 20 Apr 2024 16:35:46 -0400
> Gregory Price <gregory.price@memverge.com> wrote:
> 
> > On Fri, Apr 19, 2024 at 11:43:14AM -0700, fan wrote:  
> > > On Fri, Apr 19, 2024 at 02:24:36PM -0400, Gregory Price wrote:    
> > > > 
> > > > added review to all patches, will hopefully be able to add a Tested-by
> > > > tag early next week, along with a v1 RFC for MHD bit-tracking.
> > > > 
> > > > We've been testing v5/v6 for a bit, so I expect as soon as we get the
> > > > MHD code ported over to v7 i'll ship a tested-by tag pretty quick.
> > > > 
> > > > The super-set release will complicate a few things but this doesn't
> > > > look like a blocker on our end, just a change to how we track bits in a
> > > > shared bit/bytemap.
> > > >     
> > > 
> > > Hi Gregory,
> > > Thanks for reviewing the patches so quickly. 
> > > 
> > > No pressure, but look forward to your MHD work. :)
> > > 
> > > Fan    
> > 
> > Starting to get into versioniong hell a bit, since the Niagara work was
> > based off of jonathan's branch and the mhd-dcd work needs some of the
> > extentions from that branch - while this branch is based on master.
> > 
> > Probably we'll need to wait for a new cxl dated branch to try and sus
> > out the pain points before we push an RFC.  I would not want to have
> > conflicting commits for something like this for example:
> > 
> > https://lore.kernel.org/qemu-devel/20230901012914.226527-2-gregory.price@memverge.com/
> > 
> > We get merge conflicts here because this is behind that patch. So
> > pushing up an RFC in this state would be mostly useless to everyone  
> 
> Subtle hint noted ;) 
> 
> I'll build a fresh tree - any remaining rebases until QEMU 9.0 should be
> straight forward anyway.   My ideal is that the NUMA GP series lands early
> in 9.1 cycle and this can go in parallel.  I'd really like to
> get this in early if possible so we can start clearing some of the other
> stuff that ended up built on top of it!

I've pushed to gitlab.com/jic23/qemu cxl-2024-04-22-draft
Its extremely lightly tested so far.

To save time, I've temporarily dropped the fm-api DCD initiate
dynamic capacity add patch as that needs non trivial updates.

I've not yet caught up with some other outstanding series, but
I will almost certainly put them on top of DCD.

Jonathan

> 
> Jonathan
> 
> > 
> > ~Gregory  
> 
>
Jonathan Cameron April 22, 2024, 3:07 p.m. UTC | #6 
On Mon, 22 Apr 2024 15:23:16 +0100
Jonathan Cameron <Jonathan.Cameron@Huawei.com> wrote:

> On Mon, 22 Apr 2024 13:04:48 +0100
> Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
> 
> > On Sat, 20 Apr 2024 16:35:46 -0400
> > Gregory Price <gregory.price@memverge.com> wrote:
> >   
> > > On Fri, Apr 19, 2024 at 11:43:14AM -0700, fan wrote:    
> > > > On Fri, Apr 19, 2024 at 02:24:36PM -0400, Gregory Price wrote:      
> > > > > 
> > > > > added review to all patches, will hopefully be able to add a Tested-by
> > > > > tag early next week, along with a v1 RFC for MHD bit-tracking.
> > > > > 
> > > > > We've been testing v5/v6 for a bit, so I expect as soon as we get the
> > > > > MHD code ported over to v7 i'll ship a tested-by tag pretty quick.
> > > > > 
> > > > > The super-set release will complicate a few things but this doesn't
> > > > > look like a blocker on our end, just a change to how we track bits in a
> > > > > shared bit/bytemap.
> > > > >       
> > > > 
> > > > Hi Gregory,
> > > > Thanks for reviewing the patches so quickly. 
> > > > 
> > > > No pressure, but look forward to your MHD work. :)
> > > > 
> > > > Fan      
> > > 
> > > Starting to get into versioniong hell a bit, since the Niagara work was
> > > based off of jonathan's branch and the mhd-dcd work needs some of the
> > > extentions from that branch - while this branch is based on master.
> > > 
> > > Probably we'll need to wait for a new cxl dated branch to try and sus
> > > out the pain points before we push an RFC.  I would not want to have
> > > conflicting commits for something like this for example:
> > > 
> > > https://lore.kernel.org/qemu-devel/20230901012914.226527-2-gregory.price@memverge.com/
> > > 
> > > We get merge conflicts here because this is behind that patch. So
> > > pushing up an RFC in this state would be mostly useless to everyone    
> > 
> > Subtle hint noted ;) 
> > 
> > I'll build a fresh tree - any remaining rebases until QEMU 9.0 should be
> > straight forward anyway.   My ideal is that the NUMA GP series lands early
> > in 9.1 cycle and this can go in parallel.  I'd really like to
> > get this in early if possible so we can start clearing some of the other
> > stuff that ended up built on top of it!  
> 
> I've pushed to gitlab.com/jic23/qemu cxl-2024-04-22-draft
> Its extremely lightly tested so far.
> 
> To save time, I've temporarily dropped the fm-api DCD initiate
> dynamic capacity add patch as that needs non trivial updates.
> 
> I've not yet caught up with some other outstanding series, but
> I will almost certainly put them on top of DCD.

If anyone pulled in meantime... I failed to push down a fix from
my working tree on top of this.
Goes to show I shouldn't ignore patches simply named "Push down" :(

Updated on same branch.

Jonathan
> 
> Jonathan
> 
> > 
> > Jonathan
> >   
> > > 
> > > ~Gregory    
> > 
> >   
> 
>
Gregory Price April 22, 2024, 3:42 p.m. UTC | #7 
On Mon, Apr 22, 2024 at 01:04:48PM +0100, Jonathan Cameron wrote:
> On Sat, 20 Apr 2024 16:35:46 -0400
> Gregory Price <gregory.price@memverge.com> wrote:
> 
> > On Fri, Apr 19, 2024 at 11:43:14AM -0700, fan wrote:
> > > On Fri, Apr 19, 2024 at 02:24:36PM -0400, Gregory Price wrote:  
> > > > 
> > > > added review to all patches, will hopefully be able to add a Tested-by
> > > > tag early next week, along with a v1 RFC for MHD bit-tracking.
> > > > 
> > > > We've been testing v5/v6 for a bit, so I expect as soon as we get the
> > > > MHD code ported over to v7 i'll ship a tested-by tag pretty quick.
> > > > 
> > > > The super-set release will complicate a few things but this doesn't
> > > > look like a blocker on our end, just a change to how we track bits in a
> > > > shared bit/bytemap.
> > > >   
> > > 
> > > Hi Gregory,
> > > Thanks for reviewing the patches so quickly. 
> > > 
> > > No pressure, but look forward to your MHD work. :)
> > > 
> > > Fan  
> > 
> > Starting to get into versioniong hell a bit, since the Niagara work was
> > based off of jonathan's branch and the mhd-dcd work needs some of the
> > extentions from that branch - while this branch is based on master.
> > 
> > Probably we'll need to wait for a new cxl dated branch to try and sus
> > out the pain points before we push an RFC.  I would not want to have
> > conflicting commits for something like this for example:
> > 
> > https://lore.kernel.org/qemu-devel/20230901012914.226527-2-gregory.price@memverge.com/
> > 
> > We get merge conflicts here because this is behind that patch. So
> > pushing up an RFC in this state would be mostly useless to everyone
> 
> Subtle hint noted ;) 
>

Gentle nudge/poke/prod :P

Got your updates, thank you!  We should have something cleaned up today hopefully.

> I'll build a fresh tree - any remaining rebases until QEMU 9.0 should be
> straight forward anyway.   My ideal is that the NUMA GP series lands early
> in 9.1 cycle and this can go in parallel.  I'd really like to
> get this in early if possible so we can start clearing some of the other
> stuff that ended up built on top of it!
> 
> Jonathan
> 
> > 
> > ~Gregory
>
Zhijian Li (Fujitsu)" via May 14, 2024, 2:16 a.m. UTC | #8 
Hi Fan


Do you have a newer instruction to play with the DCD. It seems that
the instruction in RFC[0] doesn't work for current code.

[0] https://lore.kernel.org/all/20230511175609.2091136-1-fan.ni@samsung.com/



On 19/04/2024 07:10, nifan.cxl@gmail.com wrote:
> A git tree of this series can be found here (with one extra commit on top
> for printing out accepted/pending extent list):
> https://github.com/moking/qemu/tree/dcd-v7
> 
> v6->v7:
> 
> 1. Fixed the dvsec range register issue mentioned in the the cover letter in v6.
>     Only relevant bits are set to mark the device ready (Patch 6). (Jonathan)
> 2. Moved the if statement in cxl_setup_memory from Patch 6 to Patch 4. (Jonathan)
> 3. Used MIN instead of if statement to get record_count in Patch 7. (Jonathan)
> 4. Added "Reviewed-by" tag to Patch 7.
> 5. Modified cxl_dc_extent_release_dry_run so the updated extent list can be
>     reused in cmd_dcd_release_dyn_cap to simplify the process in Patch 8. (Jørgen)
> 6. Added comments to indicate further "TODO" items in cmd_dcd_add_dyn_cap_rsp.
>      (Jonathan)
> 7. Avoided irrelevant code reformat in Patch 8. (Jonathan)
> 8. Modified QMP interfaces for adding/releasing DC extents to allow passing
>     tags, selection policy, flags in the interface. (Jonathan, Gregory)
> 9. Redesigned the pending list so extents in the same requests are grouped
>      together. A new data structure is introduced to represent "extent group"
>      in pending list.  (Jonathan)
> 10. Added support in QMP interface for "More" flag.
> 11. Check "Forced removal" flag for release request and not let it pass through.
> 12. Removed the dynamic capacity log type from CxlEventLog definition in cxl.json
>     to avoid the side effect it may introduce to inject error to DC event log.
>     (Jonathan)
> 13. Hard coded the event log type to dynamic capacity event log in QMP
>      interfaces. (Jonathan)
> 14. Adding space in between "-1]". (Jonathan)
> 15. Some minor comment fixes.
> 
> The code is tested with similar setup and has passed similar tests as listed
> in the cover letter of v5[1] and v6[2].
> Also, the code is tested with the latest DCD kernel patchset[3].
> 
> [1] Qemu DCD patchset v5: https://lore.kernel.org/linux-cxl/20240304194331.1586191-1-nifan.cxl@gmail.com/T/#t
> [2] Qemu DCD patchset v6: https://lore.kernel.org/linux-cxl/20240325190339.696686-1-nifan.cxl@gmail.com/T/#t
> [3] DCD kernel patches: https://lore.kernel.org/linux-cxl/20240324-dcd-type2-upstream-v1-0-b7b00d623625@intel.com/T/#m11c571e21c4fe17c7d04ec5c2c7bc7cbf2cd07e3
> 
> 
> Fan Ni (12):
>    hw/cxl/cxl-mailbox-utils: Add dc_event_log_size field to output
>      payload of identify memory device command
>    hw/cxl/cxl-mailbox-utils: Add dynamic capacity region representative
>      and mailbox command support
>    include/hw/cxl/cxl_device: Rename mem_size as static_mem_size for
>      type3 memory devices
>    hw/mem/cxl_type3: Add support to create DC regions to type3 memory
>      devices
>    hw/mem/cxl-type3: Refactor ct3_build_cdat_entries_for_mr to take mr
>      size instead of mr as argument
>    hw/mem/cxl_type3: Add host backend and address space handling for DC
>      regions
>    hw/mem/cxl_type3: Add DC extent list representative and get DC extent
>      list mailbox support
>    hw/cxl/cxl-mailbox-utils: Add mailbox commands to support add/release
>      dynamic capacity response
>    hw/cxl/events: Add qmp interfaces to add/release dynamic capacity
>      extents
>    hw/mem/cxl_type3: Add DPA range validation for accesses to DC regions
>    hw/cxl/cxl-mailbox-utils: Add superset extent release mailbox support
>    hw/mem/cxl_type3: Allow to release extent superset in QMP interface
> 
>   hw/cxl/cxl-mailbox-utils.c  | 620 ++++++++++++++++++++++++++++++++++-
>   hw/mem/cxl_type3.c          | 633 +++++++++++++++++++++++++++++++++---
>   hw/mem/cxl_type3_stubs.c    |  20 ++
>   include/hw/cxl/cxl_device.h |  81 ++++-
>   include/hw/cxl/cxl_events.h |  18 +
>   qapi/cxl.json               |  69 ++++
>   6 files changed, 1396 insertions(+), 45 deletions(-)
>
fan May 16, 2024, 5:05 p.m. UTC | #9 
On Fri, Apr 19, 2024 at 02:24:36PM -0400, Gregory Price wrote:
> On Thu, Apr 18, 2024 at 04:10:51PM -0700, nifan.cxl@gmail.com wrote:
> > A git tree of this series can be found here (with one extra commit on top
> > for printing out accepted/pending extent list): 
> > https://github.com/moking/qemu/tree/dcd-v7
> > 
> > v6->v7:
> > 
> > 1. Fixed the dvsec range register issue mentioned in the the cover letter in v6.
> >    Only relevant bits are set to mark the device ready (Patch 6). (Jonathan)
> > 2. Moved the if statement in cxl_setup_memory from Patch 6 to Patch 4. (Jonathan)
> > 3. Used MIN instead of if statement to get record_count in Patch 7. (Jonathan)
> > 4. Added "Reviewed-by" tag to Patch 7.
> > 5. Modified cxl_dc_extent_release_dry_run so the updated extent list can be
> >    reused in cmd_dcd_release_dyn_cap to simplify the process in Patch 8. (Jørgen) 
> > 6. Added comments to indicate further "TODO" items in cmd_dcd_add_dyn_cap_rsp.
> >     (Jonathan)
> > 7. Avoided irrelevant code reformat in Patch 8. (Jonathan)
> > 8. Modified QMP interfaces for adding/releasing DC extents to allow passing
> >    tags, selection policy, flags in the interface. (Jonathan, Gregory)
> > 9. Redesigned the pending list so extents in the same requests are grouped
> >     together. A new data structure is introduced to represent "extent group"
> >     in pending list.  (Jonathan)
> > 10. Added support in QMP interface for "More" flag. 
> > 11. Check "Forced removal" flag for release request and not let it pass through.
> > 12. Removed the dynamic capacity log type from CxlEventLog definition in cxl.json
> >    to avoid the side effect it may introduce to inject error to DC event log.
> >    (Jonathan)
> > 13. Hard coded the event log type to dynamic capacity event log in QMP
> >     interfaces. (Jonathan)
> > 14. Adding space in between "-1]". (Jonathan)
> > 15. Some minor comment fixes.
> > 
> > The code is tested with similar setup and has passed similar tests as listed
> > in the cover letter of v5[1] and v6[2].
> > Also, the code is tested with the latest DCD kernel patchset[3].
> > 
> > [1] Qemu DCD patchset v5: https://lore.kernel.org/linux-cxl/20240304194331.1586191-1-nifan.cxl@gmail.com/T/#t
> > [2] Qemu DCD patchset v6: https://lore.kernel.org/linux-cxl/20240325190339.696686-1-nifan.cxl@gmail.com/T/#t
> > [3] DCD kernel patches: https://lore.kernel.org/linux-cxl/20240324-dcd-type2-upstream-v1-0-b7b00d623625@intel.com/T/#m11c571e21c4fe17c7d04ec5c2c7bc7cbf2cd07e3
> >
> 
> added review to all patches, will hopefully be able to add a Tested-by
> tag early next week, along with a v1 RFC for MHD bit-tracking.
> 
> We've been testing v5/v6 for a bit, so I expect as soon as we get the
> MHD code ported over to v7 i'll ship a tested-by tag pretty quick.
> 
> The super-set release will complicate a few things but this doesn't
> look like a blocker on our end, just a change to how we track bits in a
> shared bit/bytemap.
> 

Hi Gregory,
I am planning to address all the concerns in this series and send out v8
next week. Jonathan mentioned you have few related patches built on top
of this series, can you point me to the latest version so I can look
into it? Also, would you like me to carry them over to send together
with my series in next version? It could be easier for you to avoid the
potential rebase needed for your patches?

Let me know.

Thanks,
Fan

> > 
> > Fan Ni (12):
> >   hw/cxl/cxl-mailbox-utils: Add dc_event_log_size field to output
> >     payload of identify memory device command
> >   hw/cxl/cxl-mailbox-utils: Add dynamic capacity region representative
> >     and mailbox command support
> >   include/hw/cxl/cxl_device: Rename mem_size as static_mem_size for
> >     type3 memory devices
> >   hw/mem/cxl_type3: Add support to create DC regions to type3 memory
> >     devices
> >   hw/mem/cxl-type3: Refactor ct3_build_cdat_entries_for_mr to take mr
> >     size instead of mr as argument
> >   hw/mem/cxl_type3: Add host backend and address space handling for DC
> >     regions
> >   hw/mem/cxl_type3: Add DC extent list representative and get DC extent
> >     list mailbox support
> >   hw/cxl/cxl-mailbox-utils: Add mailbox commands to support add/release
> >     dynamic capacity response
> >   hw/cxl/events: Add qmp interfaces to add/release dynamic capacity
> >     extents
> >   hw/mem/cxl_type3: Add DPA range validation for accesses to DC regions
> >   hw/cxl/cxl-mailbox-utils: Add superset extent release mailbox support
> >   hw/mem/cxl_type3: Allow to release extent superset in QMP interface
> > 
> >  hw/cxl/cxl-mailbox-utils.c  | 620 ++++++++++++++++++++++++++++++++++-
> >  hw/mem/cxl_type3.c          | 633 +++++++++++++++++++++++++++++++++---
> >  hw/mem/cxl_type3_stubs.c    |  20 ++
> >  include/hw/cxl/cxl_device.h |  81 ++++-
> >  include/hw/cxl/cxl_events.h |  18 +
> >  qapi/cxl.json               |  69 ++++
> >  6 files changed, 1396 insertions(+), 45 deletions(-)
> > 
> > -- 
> > 2.43.0
> >
fan May 16, 2024, 5:12 p.m. UTC | #10 
On Tue, May 14, 2024 at 02:16:51AM +0000, Zhijian Li (Fujitsu) wrote:
> Hi Fan
> 
> 
> Do you have a newer instruction to play with the DCD. It seems that
> the instruction in RFC[0] doesn't work for current code.
> 
> [0] https://lore.kernel.org/all/20230511175609.2091136-1-fan.ni@samsung.com/
> 

For the testing, the only thing that has been changed for this series is
the QMP interface for add/release DC extents.

https://lore.kernel.org/linux-cxl/d708f7c8-2598-4a17-9cbb-935c6ae2a2be@fujitsu.com/T/#m05066f0098e976fb1c4b05db5e7ff7ca1bf27b1e

1. Add dynamic capacity extents:

For example, the command to add two continuous extents (each 128MiB long)
to region 0 (starting at DPA offset 0) looks like below:

{ "execute": "qmp_capabilities" }

{ "execute": "cxl-add-dynamic-capacity",
  "arguments": {
      "path": "/machine/peripheral/cxl-dcd0",
      "hid": 0,
      "selection-policy": 2,
      "region-id": 0,
      "tag": "",
      "extents": [
      {
          "offset": 0,
          "len": 134217728
      },
      {
          "offset": 134217728,
          "len": 134217728
      }
      ]
  }
}

2. Release dynamic capacity extents:

For example, the command to release an extent of size 128MiB from region 0
(DPA offset 128MiB) looks like below:

{ "execute": "cxl-release-dynamic-capacity",
  "arguments": {
      "path": "/machine/peripheral/cxl-dcd0",
      "hid": 0,
      "flags": 1,
      "region-id": 0,
      "tag": "",
      "extents": [
      {
          "offset": 134217728,
          "len": 134217728
      }
      ]
  }
}

btw, I have a wiki page to explain how to test CXL DCD with a tool I
wrote.
https://github.com/moking/moking.github.io/wiki/cxl%E2%80%90test%E2%80%90tool:-A-tool-to-ease-CXL-test-with-QEMU-setup%E2%80%90%E2%80%90Using-DCD-test-as-an-example

Let me know if you need more info for testing.


Fan

> 
> 
> On 19/04/2024 07:10, nifan.cxl@gmail.com wrote:
> > A git tree of this series can be found here (with one extra commit on top
> > for printing out accepted/pending extent list):
> > https://github.com/moking/qemu/tree/dcd-v7
> > 
> > v6->v7:
> > 
> > 1. Fixed the dvsec range register issue mentioned in the the cover letter in v6.
> >     Only relevant bits are set to mark the device ready (Patch 6). (Jonathan)
> > 2. Moved the if statement in cxl_setup_memory from Patch 6 to Patch 4. (Jonathan)
> > 3. Used MIN instead of if statement to get record_count in Patch 7. (Jonathan)
> > 4. Added "Reviewed-by" tag to Patch 7.
> > 5. Modified cxl_dc_extent_release_dry_run so the updated extent list can be
> >     reused in cmd_dcd_release_dyn_cap to simplify the process in Patch 8. (Jørgen)
> > 6. Added comments to indicate further "TODO" items in cmd_dcd_add_dyn_cap_rsp.
> >      (Jonathan)
> > 7. Avoided irrelevant code reformat in Patch 8. (Jonathan)
> > 8. Modified QMP interfaces for adding/releasing DC extents to allow passing
> >     tags, selection policy, flags in the interface. (Jonathan, Gregory)
> > 9. Redesigned the pending list so extents in the same requests are grouped
> >      together. A new data structure is introduced to represent "extent group"
> >      in pending list.  (Jonathan)
> > 10. Added support in QMP interface for "More" flag.
> > 11. Check "Forced removal" flag for release request and not let it pass through.
> > 12. Removed the dynamic capacity log type from CxlEventLog definition in cxl.json
> >     to avoid the side effect it may introduce to inject error to DC event log.
> >     (Jonathan)
> > 13. Hard coded the event log type to dynamic capacity event log in QMP
> >      interfaces. (Jonathan)
> > 14. Adding space in between "-1]". (Jonathan)
> > 15. Some minor comment fixes.
> > 
> > The code is tested with similar setup and has passed similar tests as listed
> > in the cover letter of v5[1] and v6[2].
> > Also, the code is tested with the latest DCD kernel patchset[3].
> > 
> > [1] Qemu DCD patchset v5: https://lore.kernel.org/linux-cxl/20240304194331.1586191-1-nifan.cxl@gmail.com/T/#t
> > [2] Qemu DCD patchset v6: https://lore.kernel.org/linux-cxl/20240325190339.696686-1-nifan.cxl@gmail.com/T/#t
> > [3] DCD kernel patches: https://lore.kernel.org/linux-cxl/20240324-dcd-type2-upstream-v1-0-b7b00d623625@intel.com/T/#m11c571e21c4fe17c7d04ec5c2c7bc7cbf2cd07e3
> > 
> > 
> > Fan Ni (12):
> >    hw/cxl/cxl-mailbox-utils: Add dc_event_log_size field to output
> >      payload of identify memory device command
> >    hw/cxl/cxl-mailbox-utils: Add dynamic capacity region representative
> >      and mailbox command support
> >    include/hw/cxl/cxl_device: Rename mem_size as static_mem_size for
> >      type3 memory devices
> >    hw/mem/cxl_type3: Add support to create DC regions to type3 memory
> >      devices
> >    hw/mem/cxl-type3: Refactor ct3_build_cdat_entries_for_mr to take mr
> >      size instead of mr as argument
> >    hw/mem/cxl_type3: Add host backend and address space handling for DC
> >      regions
> >    hw/mem/cxl_type3: Add DC extent list representative and get DC extent
> >      list mailbox support
> >    hw/cxl/cxl-mailbox-utils: Add mailbox commands to support add/release
> >      dynamic capacity response
> >    hw/cxl/events: Add qmp interfaces to add/release dynamic capacity
> >      extents
> >    hw/mem/cxl_type3: Add DPA range validation for accesses to DC regions
> >    hw/cxl/cxl-mailbox-utils: Add superset extent release mailbox support
> >    hw/mem/cxl_type3: Allow to release extent superset in QMP interface
> > 
> >   hw/cxl/cxl-mailbox-utils.c  | 620 ++++++++++++++++++++++++++++++++++-
> >   hw/mem/cxl_type3.c          | 633 +++++++++++++++++++++++++++++++++---
> >   hw/mem/cxl_type3_stubs.c    |  20 ++
> >   include/hw/cxl/cxl_device.h |  81 ++++-
> >   include/hw/cxl/cxl_events.h |  18 +
> >   qapi/cxl.json               |  69 ++++
> >   6 files changed, 1396 insertions(+), 45 deletions(-)
> >
Zhijian Li (Fujitsu)" via May 17, 2024, 2:20 a.m. UTC | #11 
Fan,

Many thanks, it helps a lot. Previous I forgot to create a new dax device(daxctl create-device region0)
Question: Why we need to create a the dax0.1, why the dax0.0 doesn't associate to the new adding DCD region.

Ira,

Let me try to report a kernel panic.

kernel: dcd-2024-04-17
qemu: dcd-2024-04-17

QEMU command line:
164     <qemu:arg value='-device'/>
165     <qemu:arg value='cxl-type3,bus=cxl-rp-hb0rp0,persistent-memdev=cxl-pmem0,lsa=cxl-pmem-lsa0,id=pmem-dcmem,volatile-dc-memdev=cxl-dcmem0,num-dc-regions=4'/>
166     <qemu:arg value='-object'/>
167     <qemu:arg value='memory-backend-file,id=cxl-dcmem0,share=on,mem-path=/home/lizhijian/images/cxldcmem0.raw,size=2048M'/>
168     <qemu:arg value='-object'/>
169     <qemu:arg value='memory-backend-file,id=cxl-pmem0,share=on,mem-path=/home/lizhijian/images/cxlpmem0.raw,size=2048M'/>
170     <qemu:arg value='-object'/>
171     <qemu:arg value='memory-backend-file,id=cxl-pmem-lsa0,share=on,mem-path=/home/lizhijian/images/cxlpmem-lsa0.raw,size=4K'/>
172     <qemu:arg value='-M'/>
173     <qemu:arg value='cxl=on,cxl-fmw.0.targets.0=pxb-cxl.0,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=8k'/>


Reproducer:
  1. guest: ./create-dc.sh
  2. host: virsh qemu-monitor-command rdma-server-cxl-persistent-dcd $(cat cxl-add-dcd.json)
  3. guest: daxctl create-device region0 # will create dax0.1
  4. daxctl reconfigure-device  --mode=system-ram --force  dax0.1 -u  # kernel panic

=====================
# cat ./create-dc.sh
#!/bin/bash
set -ex

region=$(cat /sys/bus/cxl/devices/decoder0.0/create_dc_region)
echo $region> /sys/bus/cxl/devices/decoder0.0/create_dc_region
echo 256 > /sys/bus/cxl/devices/$region/interleave_granularity
echo 1 > /sys/bus/cxl/devices/$region/interleave_ways
echo "dc0" >/sys/bus/cxl/devices/decoder2.0/mode
echo 0x10000000 >/sys/bus/cxl/devices/decoder2.0/dpa_size
echo 0x10000000 > /sys/bus/cxl/devices/$region/size
echo "decoder2.0" > /sys/bus/cxl/devices/$region/target0
echo 1 > /sys/bus/cxl/devices/$region/commit
echo $region > /sys/bus/cxl/drivers/cxl_region/bind
=========================
# cat cxl-add-dcd.json
{ "execute": "cxl-add-dynamic-capacity",
   "arguments": {
       "path": "/machine/peripheral/pmem-dcmem",
       "hid": 0,
       "selection-policy": 2,
       "region-id": 0,
       "tag": "",
       "extents": [
       {
           "offset": 0,
           "len": 268435456
       }
       ]
   }
}



[  126.909297] Demotion targets for Node 0: preferred: 1, fallback: 1
[  126.911186] Demotion targets for Node 1: null
[  126.913808] BUG: kernel NULL pointer dereference, address: 0000000000000468
[  126.915431] #PF: supervisor read access in kernel mode
[  126.917156] #PF: error_code(0x0000) - not-present page
[  126.918976] PGD 8000000006771067 P4D 8000000006771067 PUD e777067 PMD 0
[  126.920587] Oops: 0000 [#1] PREEMPT SMP PTI
[  126.921714] CPU: 0 PID: 1101 Comm: daxctl Kdump: loaded Not tainted 6.9.0-rc3-lizhijian+ #489
[  126.924914] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[  126.928620] RIP: 0010:cxl_region_perf_attrs_callback+0x25/0x110 [cxl_core]
[  126.930316] Code: 90 90 90 90 90 0f 1f 44 00 00 41 56 41 55 41 54 55 53 8b 6a 24 83 fd ff 74 20 48 83 fe 01 75 1a 48 8b 87 58 ff ff ff 48 89 fb <48> 8b b8 68 04 00 00 e8 cf a2 f4 e0 39 c5 74 13 45 31 e4 5b 44 89
[  126.934920] RSP: 0018:ffffc900007cbc58 EFLAGS: 00010246
[  126.936994] RAX: 0000000000000000 RBX: ffff888007534d60 RCX: 0000000000000020
[  126.939378] RDX: ffffc900007cbcf8 RSI: 0000000000000001 RDI: ffff888007534d60
[  126.942721] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000001
[  126.944762] R10: ffff88807fc31d80 R11: 0000000000000000 R12: 0000000000000000
[  126.946900] R13: 0000000000000001 R14: ffffc900007cbcf8 R15: ffff888007534d60
[  126.948871] FS:  00007fb2ab918880(0000) GS:ffff88807fc00000(0000) knlGS:0000000000000000
[  126.951241] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  126.952722] CR2: 0000000000000468 CR3: 000000000aaf0003 CR4: 00000000001706f0
[  126.954623] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  126.956768] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  126.958887] Call Trace:
[  126.959814]  <TASK>
[  126.960569]  ? __die+0x20/0x70
[  126.961645]  ? page_fault_oops+0x15a/0x450
[  126.962930]  ? search_module_extables+0x33/0x90
[  126.964374]  ? fixup_exception+0x22/0x310
[  126.965693]  ? exc_page_fault+0x68/0x200
[  126.967371]  ? asm_exc_page_fault+0x22/0x30
[  126.968713]  ? cxl_region_perf_attrs_callback+0x25/0x110 [cxl_core]
[  126.972508]  notifier_call_chain+0x40/0x110
[  126.974380]  blocking_notifier_call_chain+0x43/0x60
[  126.975788]  online_pages+0x24c/0x2d0
[  126.977008]  memory_subsys_online+0x233/0x290
[  126.978338]  device_online+0x64/0x90
[  126.979440]  state_store+0xae/0xc0
[  126.980510]  kernfs_fop_write_iter+0x143/0x200
[  126.981734]  vfs_write+0x3a6/0x570
[  126.982851]  ksys_write+0x65/0xf0
[  126.984006]  do_syscall_64+0x6d/0x140
[  126.985309]  entry_SYSCALL_64_after_hwframe+0x71/0x79
[  126.986927] RIP: 0033:0x7fb2abc777a7
[  126.987983] Code: 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24
[  126.992770] RSP: 002b:00007ffebec70b98 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[  126.994874] RAX: ffffffffffffffda RBX: 000000000040e1f0 RCX: 00007fb2abc777a7
[  126.996906] RDX: 000000000000000f RSI: 00007fb2abdb6434 RDI: 0000000000000004
[  126.998911] RBP: 00007ffebec70bd0 R08: 0000000000000000 R09: 00007ffebec70640
[  127.000879] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000403840
[  127.003572] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[  127.005543]  </TASK>

Thanks
Zhijian

On 17/05/2024 01:12, fan wrote:
> On Tue, May 14, 2024 at 02:16:51AM +0000, Zhijian Li (Fujitsu) wrote:
>> Hi Fan
>>
>>
>> Do you have a newer instruction to play with the DCD. It seems that
>> the instruction in RFC[0] doesn't work for current code.
>>
>> [0] https://lore.kernel.org/all/20230511175609.2091136-1-fan.ni@samsung.com/
>>
> 
> For the testing, the only thing that has been changed for this series is
> the QMP interface for add/release DC extents.
> 
> https://lore.kernel.org/linux-cxl/d708f7c8-2598-4a17-9cbb-935c6ae2a2be@fujitsu.com/T/#m05066f0098e976fb1c4b05db5e7ff7ca1bf27b1e
> 
> 1. Add dynamic capacity extents:
> 
> For example, the command to add two continuous extents (each 128MiB long)
> to region 0 (starting at DPA offset 0) looks like below:
> 
> { "execute": "qmp_capabilities" }
> 
> { "execute": "cxl-add-dynamic-capacity",
>    "arguments": {
>        "path": "/machine/peripheral/cxl-dcd0",
>        "hid": 0,
>        "selection-policy": 2,
>        "region-id": 0,
>        "tag": "",
>        "extents": [
>        {
>            "offset": 0,
>            "len": 134217728
>        },
>        {
>            "offset": 134217728,
>            "len": 134217728
>        }
>        ]
>    }
> }
> 
> 2. Release dynamic capacity extents:
> 
> For example, the command to release an extent of size 128MiB from region 0
> (DPA offset 128MiB) looks like below:
> 
> { "execute": "cxl-release-dynamic-capacity",
>    "arguments": {
>        "path": "/machine/peripheral/cxl-dcd0",
>        "hid": 0,
>        "flags": 1,
>        "region-id": 0,
>        "tag": "",
>        "extents": [
>        {
>            "offset": 134217728,
>            "len": 134217728
>        }
>        ]
>    }
> }
> 
> btw, I have a wiki page to explain how to test CXL DCD with a tool I
> wrote.
> https://github.com/moking/moking.github.io/wiki/cxl%E2%80%90test%E2%80%90tool:-A-tool-to-ease-CXL-test-with-QEMU-setup%E2%80%90%E2%80%90Using-DCD-test-as-an-example
> 


> Let me know if you need more info for testing.
> 
> 
> Fan
> 
>>
>>
>> On 19/04/2024 07:10, nifan.cxl@gmail.com wrote:
>>> A git tree of this series can be found here (with one extra commit on top
>>> for printing out accepted/pending extent list):
>>> https://github.com/moking/qemu/tree/dcd-v7
>>>
>>> v6->v7:
>>>
>>> 1. Fixed the dvsec range register issue mentioned in the the cover letter in v6.
>>>      Only relevant bits are set to mark the device ready (Patch 6). (Jonathan)
>>> 2. Moved the if statement in cxl_setup_memory from Patch 6 to Patch 4. (Jonathan)
>>> 3. Used MIN instead of if statement to get record_count in Patch 7. (Jonathan)
>>> 4. Added "Reviewed-by" tag to Patch 7.
>>> 5. Modified cxl_dc_extent_release_dry_run so the updated extent list can be
>>>      reused in cmd_dcd_release_dyn_cap to simplify the process in Patch 8. (Jørgen)
>>> 6. Added comments to indicate further "TODO" items in cmd_dcd_add_dyn_cap_rsp.
>>>       (Jonathan)
>>> 7. Avoided irrelevant code reformat in Patch 8. (Jonathan)
>>> 8. Modified QMP interfaces for adding/releasing DC extents to allow passing
>>>      tags, selection policy, flags in the interface. (Jonathan, Gregory)
>>> 9. Redesigned the pending list so extents in the same requests are grouped
>>>       together. A new data structure is introduced to represent "extent group"
>>>       in pending list.  (Jonathan)
>>> 10. Added support in QMP interface for "More" flag.
>>> 11. Check "Forced removal" flag for release request and not let it pass through.
>>> 12. Removed the dynamic capacity log type from CxlEventLog definition in cxl.json
>>>      to avoid the side effect it may introduce to inject error to DC event log.
>>>      (Jonathan)
>>> 13. Hard coded the event log type to dynamic capacity event log in QMP
>>>       interfaces. (Jonathan)
>>> 14. Adding space in between "-1]". (Jonathan)
>>> 15. Some minor comment fixes.
>>>
>>> The code is tested with similar setup and has passed similar tests as listed
>>> in the cover letter of v5[1] and v6[2].
>>> Also, the code is tested with the latest DCD kernel patchset[3].
>>>
>>> [1] Qemu DCD patchset v5: https://lore.kernel.org/linux-cxl/20240304194331.1586191-1-nifan.cxl@gmail.com/T/#t
>>> [2] Qemu DCD patchset v6: https://lore.kernel.org/linux-cxl/20240325190339.696686-1-nifan.cxl@gmail.com/T/#t
>>> [3] DCD kernel patches: https://lore.kernel.org/linux-cxl/20240324-dcd-type2-upstream-v1-0-b7b00d623625@intel.com/T/#m11c571e21c4fe17c7d04ec5c2c7bc7cbf2cd07e3
>>>
>>>
>>> Fan Ni (12):
>>>     hw/cxl/cxl-mailbox-utils: Add dc_event_log_size field to output
>>>       payload of identify memory device command
>>>     hw/cxl/cxl-mailbox-utils: Add dynamic capacity region representative
>>>       and mailbox command support
>>>     include/hw/cxl/cxl_device: Rename mem_size as static_mem_size for
>>>       type3 memory devices
>>>     hw/mem/cxl_type3: Add support to create DC regions to type3 memory
>>>       devices
>>>     hw/mem/cxl-type3: Refactor ct3_build_cdat_entries_for_mr to take mr
>>>       size instead of mr as argument
>>>     hw/mem/cxl_type3: Add host backend and address space handling for DC
>>>       regions
>>>     hw/mem/cxl_type3: Add DC extent list representative and get DC extent
>>>       list mailbox support
>>>     hw/cxl/cxl-mailbox-utils: Add mailbox commands to support add/release
>>>       dynamic capacity response
>>>     hw/cxl/events: Add qmp interfaces to add/release dynamic capacity
>>>       extents
>>>     hw/mem/cxl_type3: Add DPA range validation for accesses to DC regions
>>>     hw/cxl/cxl-mailbox-utils: Add superset extent release mailbox support
>>>     hw/mem/cxl_type3: Allow to release extent superset in QMP interface
>>>
>>>    hw/cxl/cxl-mailbox-utils.c  | 620 ++++++++++++++++++++++++++++++++++-
>>>    hw/mem/cxl_type3.c          | 633 +++++++++++++++++++++++++++++++++---
>>>    hw/mem/cxl_type3_stubs.c    |  20 ++
>>>    include/hw/cxl/cxl_device.h |  81 ++++-
>>>    include/hw/cxl/cxl_events.h |  18 +
>>>    qapi/cxl.json               |  69 ++++
>>>    6 files changed, 1396 insertions(+), 45 deletions(-)
>>>