| Message ID | 20230622035126.4130151-26-terry.bowman@amd.com |
|---|---|
| State | Superseded |
| Headers | show |
| Series | cxl/pci: Add support for RCH RAS error handling | expand |
On Wed, 21 Jun 2023 22:51:24 -0500 Terry Bowman <terry.bowman@amd.com> wrote: > The RCH root port contains root command AER registers that should not be > enabled.[1] Disable these to prevent root port interrupts. I'm a little dubious about a 'because the spec says' so argument. If we can describe the path by which spurious interrupts turn up then great - if not then fair enough. One trivial spelling thing inline. With that fixed. Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com> > > [1] CXL 3.0 - 12.2.1.1 RCH Downstream Port-detected Errors > > Signed-off-by: Terry Bowman <terry.bowman@amd.com> > Signed-off-by: Robert Richter <rrichter@amd.com> > --- > drivers/cxl/core/core.h | 6 ++++++ > drivers/cxl/core/pci.c | 29 +++++++++++++++++++++++++++++ > drivers/cxl/core/port.c | 3 +++ > 3 files changed, 38 insertions(+) > > diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h > index 87467c633123..880bac9db376 100644 > --- a/drivers/cxl/core/core.h > +++ b/drivers/cxl/core/core.h > @@ -86,4 +86,10 @@ enum cxl_poison_trace_type { > CXL_POISON_TRACE_CLEAR, > }; > > +#ifdef CONFIG_PCIEAER_CXL > +void cxl_disable_rch_root_ints(struct cxl_dport *dport); > +#else > +static inline void cxl_disable_rch_root_ints(struct cxl_dport *dport) { }; > +#endif > + > #endif /* __CXL_CORE_H__ */ > diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c > index 9e0eba5ccfc4..39a2f9f4f115 100644 > --- a/drivers/cxl/core/pci.c > +++ b/drivers/cxl/core/pci.c > @@ -838,6 +838,35 @@ static void cxl_handle_rdport_errors(struct cxl_dev_state *cxlds) > cxl_handle_rdport_ras(cxlds, dport); > } > > +void cxl_disable_rch_root_ints(struct cxl_dport *dport) > +{ > + void __iomem *aer_base = dport->regs.dport_aer; > + struct pci_host_bridge *bridge; > + u32 aer_cmd_mask, aer_cmd; > + > + if (!aer_base) > + return; > + > + bridge = to_pci_host_bridge(dport->dport_dev); > + > + /* > + * Disable RCH root port command interrupts. > + * CXL 3.0 12.2.1.1 - RCH Downstream Port-detected Errors > + * > + * This sequnce may not be necessary. CXL spec states disabling Spell check. (I often forget as well :( > + * the root cmd register's interrupts is required. But, PCI spec > + * shows these are disabled by default on reset. > + */ > + if (bridge->native_cxl_error) { > + aer_cmd_mask = (PCI_ERR_ROOT_CMD_COR_EN | > + PCI_ERR_ROOT_CMD_NONFATAL_EN | > + PCI_ERR_ROOT_CMD_FATAL_EN); > + aer_cmd = readl(aer_base + PCI_ERR_ROOT_COMMAND); > + aer_cmd &= ~aer_cmd_mask; > + writel(aer_cmd, aer_base + PCI_ERR_ROOT_COMMAND); > + } > +} > + > #else > static void cxl_handle_rdport_errors(struct cxl_dev_state *cxlds) { } > #endif > diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c > index 87a12e69aa8e..2d812bbaf05f 100644 > --- a/drivers/cxl/core/port.c > +++ b/drivers/cxl/core/port.c > @@ -1035,6 +1035,9 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev, > > cxl_dport_map_regs(dport); > > + if (dport->rch) > + cxl_disable_rch_root_ints(dport); > + > cond_cxl_root_lock(port); > rc = add_dport(port, dport); > cond_cxl_root_unlock(port);
Hi Jonathan, On 6/22/23 08:12, Jonathan Cameron wrote: > On Wed, 21 Jun 2023 22:51:24 -0500 > Terry Bowman <terry.bowman@amd.com> wrote: > >> The RCH root port contains root command AER registers that should not be >> enabled.[1] Disable these to prevent root port interrupts. > > I'm a little dubious about a 'because the spec says' so argument. > If we can describe the path by which spurious interrupts turn up then > great - if not then fair enough. > This was added to follow the spec. The RCH downstream port contains root port control pci express capability for enabling and disabling root port interrupts. The interrupts are (should be) disabled by default at powerup according to the PCI spec. We know SW does not enable because the RCH downstream port is not enumerated or managed by a port driver. I cant say this patch is absolutely necessary but was not comfortable with removing it either and want to avoid undefined behavior. Regards, Terry
On Thu, 22 Jun 2023 11:33:30 -0500 Terry Bowman <Terry.Bowman@amd.com> wrote: > Hi Jonathan, > > On 6/22/23 08:12, Jonathan Cameron wrote: > > On Wed, 21 Jun 2023 22:51:24 -0500 > > Terry Bowman <terry.bowman@amd.com> wrote: > > > >> The RCH root port contains root command AER registers that should not be > >> enabled.[1] Disable these to prevent root port interrupts. > > > > I'm a little dubious about a 'because the spec says' so argument. > > If we can describe the path by which spurious interrupts turn up then > > great - if not then fair enough. > > > > This was added to follow the spec. The RCH downstream port contains > root port control pci express capability for enabling and disabling root > port interrupts. The interrupts are (should be) disabled by default at > powerup according to the PCI spec. We know SW does not enable because > the RCH downstream port is not enumerated or managed by a port driver. I > cant say this patch is absolutely necessary but was not comfortable with > removing it either and want to avoid undefined behavior. > Maybe we should just abuse firmware authors and blame them for potentially having changed the default :) Fine as is. Jonathan > Regards, > Terry
diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h index 87467c633123..880bac9db376 100644 --- a/drivers/cxl/core/core.h +++ b/drivers/cxl/core/core.h @@ -86,4 +86,10 @@ enum cxl_poison_trace_type { CXL_POISON_TRACE_CLEAR, }; +#ifdef CONFIG_PCIEAER_CXL +void cxl_disable_rch_root_ints(struct cxl_dport *dport); +#else +static inline void cxl_disable_rch_root_ints(struct cxl_dport *dport) { }; +#endif + #endif /* __CXL_CORE_H__ */ diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index 9e0eba5ccfc4..39a2f9f4f115 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -838,6 +838,35 @@ static void cxl_handle_rdport_errors(struct cxl_dev_state *cxlds) cxl_handle_rdport_ras(cxlds, dport); } +void cxl_disable_rch_root_ints(struct cxl_dport *dport) +{ + void __iomem *aer_base = dport->regs.dport_aer; + struct pci_host_bridge *bridge; + u32 aer_cmd_mask, aer_cmd; + + if (!aer_base) + return; + + bridge = to_pci_host_bridge(dport->dport_dev); + + /* + * Disable RCH root port command interrupts. + * CXL 3.0 12.2.1.1 - RCH Downstream Port-detected Errors + * + * This sequnce may not be necessary. CXL spec states disabling + * the root cmd register's interrupts is required. But, PCI spec + * shows these are disabled by default on reset. + */ + if (bridge->native_cxl_error) { + aer_cmd_mask = (PCI_ERR_ROOT_CMD_COR_EN | + PCI_ERR_ROOT_CMD_NONFATAL_EN | + PCI_ERR_ROOT_CMD_FATAL_EN); + aer_cmd = readl(aer_base + PCI_ERR_ROOT_COMMAND); + aer_cmd &= ~aer_cmd_mask; + writel(aer_cmd, aer_base + PCI_ERR_ROOT_COMMAND); + } +} + #else static void cxl_handle_rdport_errors(struct cxl_dev_state *cxlds) { } #endif diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c index 87a12e69aa8e..2d812bbaf05f 100644 --- a/drivers/cxl/core/port.c +++ b/drivers/cxl/core/port.c @@ -1035,6 +1035,9 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev, cxl_dport_map_regs(dport); + if (dport->rch) + cxl_disable_rch_root_ints(dport); + cond_cxl_root_lock(port); rc = add_dport(port, dport); cond_cxl_root_unlock(port);