| Message ID | 20240219173153.12114-1-Jonathan.Cameron@huawei.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | tcg/i386: Page tables in MMIO memory fixes (CXL) | expand |
On 2/19/24 07:31, Jonathan Cameron wrote: > v2: Changes documented in patch 3. > - I have not addressed Richard's comment on recursive locks as that > seems to be a more general issue not specific to this patch set. > > CXL memory is interleaved at granularities as fine as 64 bytes. > To emulate this each read and write access undergoes address translation > similar to that used in physical hardware. This is done using > cfmws_ops for a memory region per CXL Fixed Memory Window (the PA address > range in the host that is interleaved across host bridges and beyond. > The OS programs interleaved decoders in the CXL Root Bridges, switch > upstream ports and the corresponding decoders CXL type 3 devices who > have to know the Host PA to Device PA mappings). > > Unfortunately this CXL memory may be used as normal memory and anything > that can end up in RAM can be placed within it. As Linux has become > more capable of handling this memory we've started to get quite a few > bug reports for the QEMU support. However terrible the performance is > people seem to like running actual software stacks on it :( > > This doesn't work for KVM - so for now CXL emulation remains TCG only. > (unless you are very careful on how it is used!) I plan to add some > safety guards at a later date to make it slightly harder for people > to shoot themselves in the foot + a more limited set of CXL functionality > that is safe (no interleaving!) > > Previously we had some issues with TCG reading instructions from CXL > memory but that is now all working. This time the issues are around > the Page Tables being in the CXL memory + DMA buffers being placed in it. > > The test setup I've been using is simple 2 way interleave via 2 root > ports below a single CXL root complex. After configuration in Linux > these are mapped to their own Numa Node and > numactl --membind=1 ls > followed by powering down the machine is sufficient to hit all the bugs > addressed in this series. > > Thanks to Gregory, Peter and Alex for their help figuring this lot > out. Note that I've included one patch from Peter without a SoB > because so far it has only be posted in the discussion thread. > > Whilst thread started back at: > https://lore.kernel.org/all/CAAg4PaqsGZvkDk_=PH+Oz-yeEUVcVsrumncAgegRKuxe_YoFhA@mail.gmail.com/ > The QEMU part is from. > https://lore.kernel.org/all/20240201130438.00001384@Huawei.com/ > > > Gregory Price (1): > target/i386: Enable page walking from MMIO memory > > Jonathan Cameron (1): > tcg: Avoid double lock if page tables happen to be in mmio memory. > > Peter Maydell (1): > accel/tcg: Set can_do_io at at start of lookup_tb_ptr helper Thanks. Queued patches 1 and 3 to tcg-next, and adjusted patch 3 to use BQL_LOCK_GUARD as suggested by Alex. r~
On Thu, 22 Feb 2024 09:07:03 -1000 Richard Henderson <richard.henderson@linaro.org> wrote: > On 2/19/24 07:31, Jonathan Cameron wrote: > > v2: Changes documented in patch 3. > > - I have not addressed Richard's comment on recursive locks as that > > seems to be a more general issue not specific to this patch set. > > > > CXL memory is interleaved at granularities as fine as 64 bytes. > > To emulate this each read and write access undergoes address translation > > similar to that used in physical hardware. This is done using > > cfmws_ops for a memory region per CXL Fixed Memory Window (the PA address > > range in the host that is interleaved across host bridges and beyond. > > The OS programs interleaved decoders in the CXL Root Bridges, switch > > upstream ports and the corresponding decoders CXL type 3 devices who > > have to know the Host PA to Device PA mappings). > > > > Unfortunately this CXL memory may be used as normal memory and anything > > that can end up in RAM can be placed within it. As Linux has become > > more capable of handling this memory we've started to get quite a few > > bug reports for the QEMU support. However terrible the performance is > > people seem to like running actual software stacks on it :( > > > > This doesn't work for KVM - so for now CXL emulation remains TCG only. > > (unless you are very careful on how it is used!) I plan to add some > > safety guards at a later date to make it slightly harder for people > > to shoot themselves in the foot + a more limited set of CXL functionality > > that is safe (no interleaving!) > > > > Previously we had some issues with TCG reading instructions from CXL > > memory but that is now all working. This time the issues are around > > the Page Tables being in the CXL memory + DMA buffers being placed in it. > > > > The test setup I've been using is simple 2 way interleave via 2 root > > ports below a single CXL root complex. After configuration in Linux > > these are mapped to their own Numa Node and > > numactl --membind=1 ls > > followed by powering down the machine is sufficient to hit all the bugs > > addressed in this series. > > > > Thanks to Gregory, Peter and Alex for their help figuring this lot > > out. Note that I've included one patch from Peter without a SoB > > because so far it has only be posted in the discussion thread. > > > > Whilst thread started back at: > > https://lore.kernel.org/all/CAAg4PaqsGZvkDk_=PH+Oz-yeEUVcVsrumncAgegRKuxe_YoFhA@mail.gmail.com/ > > The QEMU part is from. > > https://lore.kernel.org/all/20240201130438.00001384@Huawei.com/ > > > > > > Gregory Price (1): > > target/i386: Enable page walking from MMIO memory > > > > Jonathan Cameron (1): > > tcg: Avoid double lock if page tables happen to be in mmio memory. > > > > Peter Maydell (1): > > accel/tcg: Set can_do_io at at start of lookup_tb_ptr helper > > Thanks. Queued patches 1 and 3 to tcg-next, and adjusted patch 3 to use BQL_LOCK_GUARD as > suggested by Alex. > Thanks. For patch 2, I assume this goes via an x86 specific path. Paolo? Looks like most such patches go through you. Jonathan > > r~ >
v2: Changes documented in patch 3. - I have not addressed Richard's comment on recursive locks as that seems to be a more general issue not specific to this patch set. CXL memory is interleaved at granularities as fine as 64 bytes. To emulate this each read and write access undergoes address translation similar to that used in physical hardware. This is done using cfmws_ops for a memory region per CXL Fixed Memory Window (the PA address range in the host that is interleaved across host bridges and beyond. The OS programs interleaved decoders in the CXL Root Bridges, switch upstream ports and the corresponding decoders CXL type 3 devices who have to know the Host PA to Device PA mappings). Unfortunately this CXL memory may be used as normal memory and anything that can end up in RAM can be placed within it. As Linux has become more capable of handling this memory we've started to get quite a few bug reports for the QEMU support. However terrible the performance is people seem to like running actual software stacks on it :( This doesn't work for KVM - so for now CXL emulation remains TCG only. (unless you are very careful on how it is used!) I plan to add some safety guards at a later date to make it slightly harder for people to shoot themselves in the foot + a more limited set of CXL functionality that is safe (no interleaving!) Previously we had some issues with TCG reading instructions from CXL memory but that is now all working. This time the issues are around the Page Tables being in the CXL memory + DMA buffers being placed in it. The test setup I've been using is simple 2 way interleave via 2 root ports below a single CXL root complex. After configuration in Linux these are mapped to their own Numa Node and numactl --membind=1 ls followed by powering down the machine is sufficient to hit all the bugs addressed in this series. Thanks to Gregory, Peter and Alex for their help figuring this lot out. Note that I've included one patch from Peter without a SoB because so far it has only be posted in the discussion thread. Whilst thread started back at: https://lore.kernel.org/all/CAAg4PaqsGZvkDk_=PH+Oz-yeEUVcVsrumncAgegRKuxe_YoFhA@mail.gmail.com/ The QEMU part is from. https://lore.kernel.org/all/20240201130438.00001384@Huawei.com/ Gregory Price (1): target/i386: Enable page walking from MMIO memory Jonathan Cameron (1): tcg: Avoid double lock if page tables happen to be in mmio memory. Peter Maydell (1): accel/tcg: Set can_do_io at at start of lookup_tb_ptr helper accel/tcg/cpu-exec.c | 8 ++++ accel/tcg/cputlb.c | 36 ++++++++++++++---- target/i386/tcg/sysemu/excp_helper.c | 57 +++++++++++++++------------- 3 files changed, 66 insertions(+), 35 deletions(-)