| Message ID | 167937891122.1510933.11055956062467467440.stgit@dwillia2-xfh.jf.intel.com |
|---|---|
| State | New, archived |
| Headers | show |
| Series | hw/acpi/cxl: Drop device-memory support from CFMWS entries | expand |
On Mon, 20 Mar 2023 23:08:31 -0700 Dan Williams <dan.j.williams@intel.com> wrote: > While it was a reasonable idea to specify no window restricitions at the > outset of the CXL emulation support, it turns out that in practice a > platform will never follow the QEMU example of specifying simultaneous > support for HDM-H and HDM-D[B] in a single window. > > HDM-D mandates extra bus cycles for host/device bias protocol, and HDM-DB > mandates extra bus cycles for back-invalidate protocol, so hardware must > be explicitly prepared for device-memory unlike host-only memory > (HDM-H). > > In preparation for the kernel dropping support for windows that do not > select between device and host-only memory, move QEMU exclusively to > declaring host-only windows. > > Signed-off-by: Dan Williams <dan.j.williams@intel.com> Hi Dan, Can we have some spec references? I think the Protocol tables in appendix C would work for that - but more specific examples called out from them would be good. I'm also not totally convinced it isn't a host implementation detail - key here is that host bridge's are still part of the host so can do impdef stuff as long as they look correct to CXL side and to host side. Time for some wild implementation conjecturing. Imagine a host that has host bridges of above average smartness. Those host bridges have HDM decoders (this doesn't work if not) Host is using a single HPA window for HDM-D[B] and HDM-H. The host bridge knows the target is HDM-H - it can get that form the HDM decoder Target Type bit etc. The HB can send (to the rest of the Host) whatever replies are necessary / fill in extra info to make it look like HDM-D[B] to the host interconnect protocol. (after some fun with a white board we think you can make this efficient by potentially making the Host bridge HDM decoder setup visible to other parts of the host - relatively easy give lots of time allowed for a decoder commit). Why would you do this? Limited HPA space availability on the host and wanting to be very flexible about use of the CXL windows. Obviously this is all moot if there is a constraint we can point to in a specification. BTW. I'm carrying a patch (it's on the gitlab tree) that I haven't posted yet that lets you configure this restriction at runtime as a similar potential host implementation restriction occurs for PMEM vs Volatile. That is also needed to exercise the fun corners of QTG etc. Jonathan > --- > hw/acpi/cxl.c | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/hw/acpi/cxl.c b/hw/acpi/cxl.c > index 2bf8c0799359..defb289e2fef 100644 > --- a/hw/acpi/cxl.c > +++ b/hw/acpi/cxl.c > @@ -103,8 +103,8 @@ static void cedt_build_cfmws(GArray *table_data, CXLState *cxls) > /* Host Bridge Interleave Granularity */ > build_append_int_noprefix(table_data, fw->enc_int_gran, 4); > > - /* Window Restrictions */ > - build_append_int_noprefix(table_data, 0x0f, 2); /* No restrictions */ > + /* Window Restrictions: Host-only ram and pmem */ > + build_append_int_noprefix(table_data, 0x0e, 2); > > /* QTG ID */ > build_append_int_noprefix(table_data, 0, 2); > >
Dan Williams wrote: > While it was a reasonable idea to specify no window restricitions at the > outset of the CXL emulation support, it turns out that in practice a > platform will never follow the QEMU example of specifying simultaneous > support for HDM-H and HDM-D[B] in a single window. > > HDM-D mandates extra bus cycles for host/device bias protocol, and HDM-DB > mandates extra bus cycles for back-invalidate protocol, so hardware must > be explicitly prepared for device-memory unlike host-only memory > (HDM-H). > > In preparation for the kernel dropping support for windows that do not > select between device and host-only memory, move QEMU exclusively to > declaring host-only windows. After an offline discussion determined that a sufficiently sophisticated platform might be able to support mixed HDM-H and HDM-D[B] in the same window, so the kernel is not going to drop this support.
Jonathan Cameron wrote: > On Mon, 20 Mar 2023 23:08:31 -0700 > Dan Williams <dan.j.williams@intel.com> wrote: > > > While it was a reasonable idea to specify no window restricitions at the > > outset of the CXL emulation support, it turns out that in practice a > > platform will never follow the QEMU example of specifying simultaneous > > support for HDM-H and HDM-D[B] in a single window. > > > > HDM-D mandates extra bus cycles for host/device bias protocol, and HDM-DB > > mandates extra bus cycles for back-invalidate protocol, so hardware must > > be explicitly prepared for device-memory unlike host-only memory > > (HDM-H). > > > > In preparation for the kernel dropping support for windows that do not > > select between device and host-only memory, move QEMU exclusively to > > declaring host-only windows. > > > > Signed-off-by: Dan Williams <dan.j.williams@intel.com> > Hi Dan, > > Can we have some spec references? I think the Protocol tables in > appendix C would work for that - but more specific examples called > out from them would be good. I presume our messages crossed in the ether: https://lore.kernel.org/linux-cxl/641a018ed7fb8_269929455@dwillia2-xfh.jf.intel.com.notmuch/ ...but yes, if I was still committed to this change, which I am not at this point, some references from Appendix C and "3.3.11 Forward Progress and Ordering Rules would be appropriate". > I'm also not totally convinced it isn't a host implementation detail > - key here is that host bridge's are still part of the host so can > do impdef stuff as long as they look correct to CXL side and to > host side. > > Time for some wild implementation conjecturing. > > Imagine a host that has host bridges of above average smartness. > Those host bridges have HDM decoders (this doesn't work if not) > > Host is using a single HPA window for HDM-D[B] and HDM-H. > The host bridge knows the target is HDM-H - it can get that form > the HDM decoder Target Type bit etc. The HB can send (to the > rest of the Host) whatever replies are necessary / fill in extra > info to make it look like HDM-D[B] to the host interconnect protocol. > > (after some fun with a white board we think you can make this efficient > by potentially making the Host bridge HDM decoder setup visible to > other parts of the host - relatively easy give lots of time allowed > for a decoder commit). > > Why would you do this? Limited HPA space availability on the host > and wanting to be very flexible about use of the CXL windows. Limited HPA space and fewer decode rules at the root level indeed sounds compelling. > Obviously this is all moot if there is a constraint we can point to > in a specification. At this time I don't have such a reference. > BTW. I'm carrying a patch (it's on the gitlab tree) that I haven't > posted yet that lets you configure this restriction at runtime as > a similar potential host implementation restriction occurs for > PMEM vs Volatile. That is also needed to exercise the fun corners of > QTG etc. Sounds good.
diff --git a/hw/acpi/cxl.c b/hw/acpi/cxl.c index 2bf8c0799359..defb289e2fef 100644 --- a/hw/acpi/cxl.c +++ b/hw/acpi/cxl.c @@ -103,8 +103,8 @@ static void cedt_build_cfmws(GArray *table_data, CXLState *cxls) /* Host Bridge Interleave Granularity */ build_append_int_noprefix(table_data, fw->enc_int_gran, 4); - /* Window Restrictions */ - build_append_int_noprefix(table_data, 0x0f, 2); /* No restrictions */ + /* Window Restrictions: Host-only ram and pmem */ + build_append_int_noprefix(table_data, 0x0e, 2); /* QTG ID */ build_append_int_noprefix(table_data, 0, 2);
While it was a reasonable idea to specify no window restricitions at the outset of the CXL emulation support, it turns out that in practice a platform will never follow the QEMU example of specifying simultaneous support for HDM-H and HDM-D[B] in a single window. HDM-D mandates extra bus cycles for host/device bias protocol, and HDM-DB mandates extra bus cycles for back-invalidate protocol, so hardware must be explicitly prepared for device-memory unlike host-only memory (HDM-H). In preparation for the kernel dropping support for windows that do not select between device and host-only memory, move QEMU exclusively to declaring host-only windows. Signed-off-by: Dan Williams <dan.j.williams@intel.com> --- hw/acpi/cxl.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)