| Message ID | 77d251960a557f23aa6e6e0465e0e42f1d461514.1715192606.git.alison.schofield@intel.com |
|---|---|
| State | New |
| Headers | show |
| Series | XOR Math Fixups: translation & position | expand |
alison.schofield@ wrote: > From: Alison Schofield <alison.schofield@intel.com> > > When a CXL region is created in a CXL Window (CFMWS) that uses XOR > interleave arithmetic XOR maps are applied during the HPA->DPA > translation. The XOR function changes the interleave selector > bit (aka position bit) in the HPA thereby varying which host bridge > services an HPA. The purpose is to minimize hot spots thereby > improving performance. I think it is important to either detail how "interleave hot spots" arise, or just omit that description since it is ultimately not relevant to Linux. I.e. the "why XOR" question is irrelevant compared to "what happens to a end user with a kernel that does not comprehend XOR interleave maths". > When a device reports a DPA in events such as poison, general_media, > and dram, the driver translates that DPA back to an HPA. Presently, > the CXL driver translation only considers the modulo position and > will report the wrong HPA for XOR configured CFMWS's. > > Add a helper function that restores the XOR'd bits during DPA->HPA > address translation. Plumb a root decoder callback to the new helper > when XOR interleave arithmetic is in use. For MODULO arithmetic, just > let the callback be NULL - as in no extra work required. Perhaps add a handful of sentences about the testing for this patch to make sure the maths are now correct compared to what was there before? > > Fixes: 28a3ae4ff66c ("cxl/trace: Add an HPA to cxl_poison trace events") > Signed-off-by: Alison Schofield <alison.schofield@intel.com> > --- > drivers/cxl/acpi.c | 48 ++++++++++++++++++++++++++++++++++++--- > drivers/cxl/core/port.c | 5 +++- > drivers/cxl/core/region.c | 5 ++++ > drivers/cxl/cxl.h | 6 ++++- > 4 files changed, 59 insertions(+), 5 deletions(-) > > diff --git a/drivers/cxl/acpi.c b/drivers/cxl/acpi.c > index 571069863c62..20488e7b09ac 100644 > --- a/drivers/cxl/acpi.c > +++ b/drivers/cxl/acpi.c > @@ -74,6 +74,43 @@ static struct cxl_dport *cxl_hb_xor(struct cxl_root_decoder *cxlrd, int pos) > return cxlrd->cxlsd.target[n]; > } > > +static u64 cxl_xor_translate(struct cxl_root_decoder *cxlrd, u64 hpa) > +{ > + struct cxl_cxims_data *cximsd = cxlrd->platform_data; > + int hbiw = cxlrd->cxlsd.nr_targets; > + u64 val; > + int pos; > + > + /* No xormaps for host bridge interleave ways of 1 or 3 */ > + if (hbiw == 1 || hbiw == 3) > + return hpa; > + > + /* > + * For root decoders using xormaps (hbiw: 2,4,6,8,12,16) restore > + * the position bit to its value before the xormap was applied at > + * HPA->DPA translation. > + * > + * pos is the lowest set bit in an XORMAP > + * val is the XORALLBITS(HPA & XORMAP) > + * > + * XORALLBITS: The CXL spec (3.1 Table 9-22) defines XORALLBITS > + * as an operation that outputs a single bit by XORing all the > + * bits in the input (hpa & xormap). Implement XORALLBITS using > + * hweight64(). If the hamming weight is even the XOR of those > + * bits results in 0, if odd the XOR result is 1. > + */ > + > + for (int i = 0; i < cximsd->nr_maps; i++) { > + if (!cximsd->xormaps[i]) > + continue; > + pos = __ffs(cximsd->xormaps[i]); > + val = (hweight64(hpa & cximsd->xormaps[i]) & 1); > + hpa = (hpa & ~(1ULL << pos)) | (val << pos); > + } This looks correct to me, but so did the original broken implementation. I would feel better about a self test either integrated into tools/testing/cxl/, or documented / referenced in the CXL Device Driver Writer's Guide that gives confidence that "This is the way".
On Wed, May 29, 2024 at 08:55:30PM -0700, Dan Williams wrote: > alison.schofield@ wrote: > > From: Alison Schofield <alison.schofield@intel.com> > > > > When a CXL region is created in a CXL Window (CFMWS) that uses XOR > > interleave arithmetic XOR maps are applied during the HPA->DPA > > translation. The XOR function changes the interleave selector > > bit (aka position bit) in the HPA thereby varying which host bridge > > services an HPA. The purpose is to minimize hot spots thereby > > improving performance. > > I think it is important to either detail how "interleave hot spots" > arise, or just omit that description since it is ultimately not relevant > to Linux. I.e. the "why XOR" question is irrelevant compared to "what > happens to a end user with a kernel that does not comprehend XOR > interleave maths". I think I understand. Long ago we decided to add XOR Math support to the CXL Driver. The point now is to do it correctly not rejustify it's existence. I'll chop 'The purpose...' > > > When a device reports a DPA in events such as poison, general_media, > > and dram, the driver translates that DPA back to an HPA. Presently, > > the CXL driver translation only considers the modulo position and > > will report the wrong HPA for XOR configured CFMWS's. > > > > Add a helper function that restores the XOR'd bits during DPA->HPA > > address translation. Plumb a root decoder callback to the new helper > > when XOR interleave arithmetic is in use. For MODULO arithmetic, just > > let the callback be NULL - as in no extra work required. > > Perhaps add a handful of sentences about the testing for this patch to > make sure the maths are now correct compared to what was there before? > There was nothing there before, but I think I know what you mean. More on that below... > > > > Fixes: 28a3ae4ff66c ("cxl/trace: Add an HPA to cxl_poison trace events") > > Signed-off-by: Alison Schofield <alison.schofield@intel.com> > > --- snip > > + > > + for (int i = 0; i < cximsd->nr_maps; i++) { > > + if (!cximsd->xormaps[i]) > > + continue; > > + pos = __ffs(cximsd->xormaps[i]); > > + val = (hweight64(hpa & cximsd->xormaps[i]) & 1); > > + hpa = (hpa & ~(1ULL << pos)) | (val << pos); > > + } > > This looks correct to me, but so did the original broken implementation. > I would feel better about a self test either integrated into > tools/testing/cxl/, or documented / referenced in the CXL Device Driver > Writer's Guide that gives confidence that "This is the way". Let me see about adding a unit test to the cxl suite that exercises the translation path using data from the sample XOR translation tables that are currently being added to the CXL Driver Writer's Guide. That would allow the test to verify against a known set of DPA->HPA offsets for a given XOR region config. -- Alison
Alison Schofield wrote: > Let me see about adding a unit test to the cxl suite that exercises the > translation path using data from the sample XOR translation tables that are > currently being added to the CXL Driver Writer's Guide. That would allow the > test to verify against a known set of DPA->HPA offsets for a given XOR region > config. Yeah, that sounds good. No need to worry about integrating that into cxl_test. It could just be a sample test program in cxl-cli that shows the algorithm and then can be referenced as "see, the kernel does the same thing as this test that passes with the sample data from the Driver Writer's Guide.
On Wed, 8 May 2024 11:47:51 -0700 alison.schofield@intel.com wrote: > From: Alison Schofield <alison.schofield@intel.com> > > When a CXL region is created in a CXL Window (CFMWS) that uses XOR > interleave arithmetic XOR maps are applied during the HPA->DPA > translation. The XOR function changes the interleave selector > bit (aka position bit) in the HPA thereby varying which host bridge > services an HPA. The purpose is to minimize hot spots thereby > improving performance. > > When a device reports a DPA in events such as poison, general_media, > and dram, the driver translates that DPA back to an HPA. Presently, > the CXL driver translation only considers the modulo position and > will report the wrong HPA for XOR configured CFMWS's. > > Add a helper function that restores the XOR'd bits during DPA->HPA > address translation. Plumb a root decoder callback to the new helper > when XOR interleave arithmetic is in use. For MODULO arithmetic, just > let the callback be NULL - as in no extra work required. > > Fixes: 28a3ae4ff66c ("cxl/trace: Add an HPA to cxl_poison trace events") > Signed-off-by: Alison Schofield <alison.schofield@intel.com> Trivial comment inline. Agree entirely that some tests would be good. I ran through a few trivial cases on a bit of paper and it looks to me like it works but that hardly counts as testing :) Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com> > --- > drivers/cxl/acpi.c | 48 ++++++++++++++++++++++++++++++++++++--- > drivers/cxl/core/port.c | 5 +++- > drivers/cxl/core/region.c | 5 ++++ > drivers/cxl/cxl.h | 6 ++++- > 4 files changed, 59 insertions(+), 5 deletions(-) > > diff --git a/drivers/cxl/acpi.c b/drivers/cxl/acpi.c > index 571069863c62..20488e7b09ac 100644 > --- a/drivers/cxl/acpi.c > +++ b/drivers/cxl/acpi.c > @@ -74,6 +74,43 @@ static struct cxl_dport *cxl_hb_xor(struct cxl_root_decoder *cxlrd, int pos) > return cxlrd->cxlsd.target[n]; > } > > +static u64 cxl_xor_translate(struct cxl_root_decoder *cxlrd, u64 hpa) > +{ > + struct cxl_cxims_data *cximsd = cxlrd->platform_data; > + int hbiw = cxlrd->cxlsd.nr_targets; > + u64 val; > + int pos; > + > + /* No xormaps for host bridge interleave ways of 1 or 3 */ > + if (hbiw == 1 || hbiw == 3) > + return hpa; > + > + /* > + * For root decoders using xormaps (hbiw: 2,4,6,8,12,16) restore > + * the position bit to its value before the xormap was applied at > + * HPA->DPA translation. > + * > + * pos is the lowest set bit in an XORMAP > + * val is the XORALLBITS(HPA & XORMAP) > + * > + * XORALLBITS: The CXL spec (3.1 Table 9-22) defines XORALLBITS > + * as an operation that outputs a single bit by XORing all the > + * bits in the input (hpa & xormap). Implement XORALLBITS using > + * hweight64(). If the hamming weight is even the XOR of those > + * bits results in 0, if odd the XOR result is 1. > + */ > + > + for (int i = 0; i < cximsd->nr_maps; i++) { > + if (!cximsd->xormaps[i]) > + continue; > + pos = __ffs(cximsd->xormaps[i]); At the moment the comment on XORALLBITS isn't associated with this code very well. I'd factor it out as cxl_xorallbits() mostly so you can stick the comment next to the bit that does the work. Or maybe a #define XORALLBITS(hpa, xormap) is good enough if you move it up under the comment. > + val = (hweight64(hpa & cximsd->xormaps[i]) & 1); > + hpa = (hpa & ~(1ULL << pos)) | (val << pos); > + } > + > + return hpa; > +} > +
On Fri, Jun 07, 2024 at 04:01:14PM +0100, Jonathan Cameron wrote: > On Wed, 8 May 2024 11:47:51 -0700 > alison.schofield@intel.com wrote: > > > From: Alison Schofield <alison.schofield@intel.com> > > > > When a CXL region is created in a CXL Window (CFMWS) that uses XOR > > interleave arithmetic XOR maps are applied during the HPA->DPA > > translation. The XOR function changes the interleave selector > > bit (aka position bit) in the HPA thereby varying which host bridge > > services an HPA. The purpose is to minimize hot spots thereby > > improving performance. > > > > When a device reports a DPA in events such as poison, general_media, > > and dram, the driver translates that DPA back to an HPA. Presently, > > the CXL driver translation only considers the modulo position and > > will report the wrong HPA for XOR configured CFMWS's. > > > > Add a helper function that restores the XOR'd bits during DPA->HPA > > address translation. Plumb a root decoder callback to the new helper > > when XOR interleave arithmetic is in use. For MODULO arithmetic, just > > let the callback be NULL - as in no extra work required. > > > > Fixes: 28a3ae4ff66c ("cxl/trace: Add an HPA to cxl_poison trace events") > > Signed-off-by: Alison Schofield <alison.schofield@intel.com> > Trivial comment inline. Agree entirely that some tests would be good. > I ran through a few trivial cases on a bit of paper and it looks to > me like it works but that hardly counts as testing :) Thanks for the review and for doing some calcs. I've become very adept at working these out with paper/pencil, that hop to C implementation is the challenge ;) > > Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com> Hmm...well, let me know if I can keep that after you read below... > > > --- > > drivers/cxl/acpi.c | 48 ++++++++++++++++++++++++++++++++++++--- > > drivers/cxl/core/port.c | 5 +++- > > drivers/cxl/core/region.c | 5 ++++ > > drivers/cxl/cxl.h | 6 ++++- > > 4 files changed, 59 insertions(+), 5 deletions(-) > > > > diff --git a/drivers/cxl/acpi.c b/drivers/cxl/acpi.c > > index 571069863c62..20488e7b09ac 100644 > > --- a/drivers/cxl/acpi.c > > +++ b/drivers/cxl/acpi.c > > @@ -74,6 +74,43 @@ static struct cxl_dport *cxl_hb_xor(struct cxl_root_decoder *cxlrd, int pos) > > return cxlrd->cxlsd.target[n]; > > } > > > > +static u64 cxl_xor_translate(struct cxl_root_decoder *cxlrd, u64 hpa) > > +{ > > + struct cxl_cxims_data *cximsd = cxlrd->platform_data; > > + int hbiw = cxlrd->cxlsd.nr_targets; > > + u64 val; > > + int pos; > > + > > + /* No xormaps for host bridge interleave ways of 1 or 3 */ > > + if (hbiw == 1 || hbiw == 3) > > + return hpa; > > + > > + /* > > + * For root decoders using xormaps (hbiw: 2,4,6,8,12,16) restore > > + * the position bit to its value before the xormap was applied at > > + * HPA->DPA translation. > > + * > > + * pos is the lowest set bit in an XORMAP > > + * val is the XORALLBITS(HPA & XORMAP) > > + * > > + * XORALLBITS: The CXL spec (3.1 Table 9-22) defines XORALLBITS > > + * as an operation that outputs a single bit by XORing all the > > + * bits in the input (hpa & xormap). Implement XORALLBITS using > > + * hweight64(). If the hamming weight is even the XOR of those > > + * bits results in 0, if odd the XOR result is 1. > > + */ > > + > > + for (int i = 0; i < cximsd->nr_maps; i++) { > > + if (!cximsd->xormaps[i]) > > + continue; > > + pos = __ffs(cximsd->xormaps[i]); > > At the moment the comment on XORALLBITS isn't associated with this > code very well. I'd factor it out as cxl_xorallbits() mostly so > you can stick the comment next to the bit that does the work. > Or maybe a #define XORALLBITS(hpa, xormap) is good enough if > you move it up under the comment. > > > + val = (hweight64(hpa & cximsd->xormaps[i]) & 1); > > + hpa = (hpa & ~(1ULL << pos)) | (val << pos); > > + } > > + > > + return hpa; > > +} > > + You haven't convinced me that readers will not be able to associate the block comment directly above the for-loop with the work inside the for-loop. Especially since this is a 25 line function with a single focus. I intentionally didn't insert line-by-line commentary in the for loop, but rather told the story in the comment and then just did it. Maybe repeating 'val' here, wraps up the comment better: - * bits results in 0, if odd the XOR result is 1. + * bits results in val==0, if odd the XOR results in val==1. -- Alison > >
diff --git a/drivers/cxl/acpi.c b/drivers/cxl/acpi.c index 571069863c62..20488e7b09ac 100644 --- a/drivers/cxl/acpi.c +++ b/drivers/cxl/acpi.c @@ -74,6 +74,43 @@ static struct cxl_dport *cxl_hb_xor(struct cxl_root_decoder *cxlrd, int pos) return cxlrd->cxlsd.target[n]; } +static u64 cxl_xor_translate(struct cxl_root_decoder *cxlrd, u64 hpa) +{ + struct cxl_cxims_data *cximsd = cxlrd->platform_data; + int hbiw = cxlrd->cxlsd.nr_targets; + u64 val; + int pos; + + /* No xormaps for host bridge interleave ways of 1 or 3 */ + if (hbiw == 1 || hbiw == 3) + return hpa; + + /* + * For root decoders using xormaps (hbiw: 2,4,6,8,12,16) restore + * the position bit to its value before the xormap was applied at + * HPA->DPA translation. + * + * pos is the lowest set bit in an XORMAP + * val is the XORALLBITS(HPA & XORMAP) + * + * XORALLBITS: The CXL spec (3.1 Table 9-22) defines XORALLBITS + * as an operation that outputs a single bit by XORing all the + * bits in the input (hpa & xormap). Implement XORALLBITS using + * hweight64(). If the hamming weight is even the XOR of those + * bits results in 0, if odd the XOR result is 1. + */ + + for (int i = 0; i < cximsd->nr_maps; i++) { + if (!cximsd->xormaps[i]) + continue; + pos = __ffs(cximsd->xormaps[i]); + val = (hweight64(hpa & cximsd->xormaps[i]) & 1); + hpa = (hpa & ~(1ULL << pos)) | (val << pos); + } + + return hpa; +} + struct cxl_cxims_context { struct device *dev; struct cxl_root_decoder *cxlrd; @@ -362,6 +399,7 @@ static int __cxl_parse_cfmws(struct acpi_cedt_cfmws *cfmws, struct cxl_cxims_context cxims_ctx; struct device *dev = ctx->dev; cxl_calc_hb_fn cxl_calc_hb; + cxl_translate_fn translate; struct cxl_decoder *cxld; unsigned int ways, i, ig; int rc; @@ -389,13 +427,17 @@ static int __cxl_parse_cfmws(struct acpi_cedt_cfmws *cfmws, if (rc) return rc; - if (cfmws->interleave_arithmetic == ACPI_CEDT_CFMWS_ARITHMETIC_MODULO) + if (cfmws->interleave_arithmetic == ACPI_CEDT_CFMWS_ARITHMETIC_MODULO) { cxl_calc_hb = cxl_hb_modulo; - else + translate = NULL; + + } else { cxl_calc_hb = cxl_hb_xor; + translate = cxl_xor_translate; + } struct cxl_root_decoder *cxlrd __free(put_cxlrd) = - cxl_root_decoder_alloc(root_port, ways, cxl_calc_hb); + cxl_root_decoder_alloc(root_port, ways, cxl_calc_hb, translate); if (IS_ERR(cxlrd)) return PTR_ERR(cxlrd); diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c index 762783bb091a..32346c171892 100644 --- a/drivers/cxl/core/port.c +++ b/drivers/cxl/core/port.c @@ -1808,6 +1808,7 @@ static int cxl_switch_decoder_init(struct cxl_port *port, * @port: owning CXL root of this decoder * @nr_targets: static number of downstream targets * @calc_hb: which host bridge covers the n'th position by granularity + * @translate: decoder specific address translation function * * Return: A new cxl decoder to be registered by cxl_decoder_add(). A * 'CXL root' decoder is one that decodes from a top-level / static platform @@ -1816,7 +1817,8 @@ static int cxl_switch_decoder_init(struct cxl_port *port, */ struct cxl_root_decoder *cxl_root_decoder_alloc(struct cxl_port *port, unsigned int nr_targets, - cxl_calc_hb_fn calc_hb) + cxl_calc_hb_fn calc_hb, + cxl_translate_fn translate) { struct cxl_root_decoder *cxlrd; struct cxl_switch_decoder *cxlsd; @@ -1839,6 +1841,7 @@ struct cxl_root_decoder *cxl_root_decoder_alloc(struct cxl_port *port, } cxlrd->calc_hb = calc_hb; + cxlrd->translate = translate; mutex_init(&cxlrd->range_lock); cxld = &cxlsd->cxld; diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c index 245edf748906..2fe93c5a8072 100644 --- a/drivers/cxl/core/region.c +++ b/drivers/cxl/core/region.c @@ -2752,6 +2752,7 @@ static bool cxl_is_hpa_in_range(u64 hpa, struct cxl_region *cxlr, int pos) static u64 cxl_dpa_to_hpa(u64 dpa, struct cxl_region *cxlr, struct cxl_endpoint_decoder *cxled) { + struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(cxlr->dev.parent); u64 dpa_offset, hpa_offset, bits_upper, mask_upper, hpa; struct cxl_region_params *p = &cxlr->params; int pos = cxled->pos; @@ -2791,6 +2792,10 @@ static u64 cxl_dpa_to_hpa(u64 dpa, struct cxl_region *cxlr, /* Apply the hpa_offset to the region base address */ hpa = hpa_offset + p->res->start; + /* Root decoder translation overrides typical modulo decode */ + if (cxlrd->translate) + hpa = cxlrd->translate(cxlrd, hpa); + if (!cxl_is_hpa_in_range(hpa, cxlr, cxled->pos)) return ULLONG_MAX; diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index 80f58b96dc1c..e11155002213 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -434,12 +434,14 @@ struct cxl_switch_decoder { struct cxl_root_decoder; typedef struct cxl_dport *(*cxl_calc_hb_fn)(struct cxl_root_decoder *cxlrd, int pos); +typedef u64 (*cxl_translate_fn)(struct cxl_root_decoder *cxlrd, u64 hpa); /** * struct cxl_root_decoder - Static platform CXL address decoder * @res: host / parent resource for region allocations * @region_id: region id for next region provisioning event * @calc_hb: which host bridge covers the n'th position by granularity + * @translate: decoder specific address translation function * @platform_data: platform specific configuration data * @range_lock: sync region autodiscovery by address range * @qos_class: QoS performance class cookie @@ -449,6 +451,7 @@ struct cxl_root_decoder { struct resource *res; atomic_t region_id; cxl_calc_hb_fn calc_hb; + cxl_translate_fn translate; void *platform_data; struct mutex range_lock; int qos_class; @@ -773,7 +776,8 @@ bool is_switch_decoder(struct device *dev); bool is_endpoint_decoder(struct device *dev); struct cxl_root_decoder *cxl_root_decoder_alloc(struct cxl_port *port, unsigned int nr_targets, - cxl_calc_hb_fn calc_hb); + cxl_calc_hb_fn calc_hb, + cxl_translate_fn translate); struct cxl_dport *cxl_hb_modulo(struct cxl_root_decoder *cxlrd, int pos); struct cxl_switch_decoder *cxl_switch_decoder_alloc(struct cxl_port *port, unsigned int nr_targets);