From patchwork Wed Oct 18 17:17:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robert Richter X-Patchwork-Id: 13427553 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D15C13D961 for ; Wed, 18 Oct 2023 17:20:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="e8edoueB" Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2089.outbound.protection.outlook.com [40.107.93.89]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 073EB3A8F; Wed, 18 Oct 2023 10:20:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=XE+rMjUQm+kNtR8SRscZlBSp6AXvlw92x2Tq2eIk/zReVjO6NYvVXls/SiFjhACJ+pCUcEGq5DgA2/hG61FTV4eGJT/5JSY+amiFDy965pPkhK5h7bBz4WgEDfYGipuDE8wnbuComQx/ZxVgVYJ8uf1YSacxUEiJdCLmYh8bFOwEHDpPzQOWybr/O4yuVrsPLdI6kMMQZtJrD0QsYMCIQOyUQNv4l2FvhfAXOlrCE33yB8THtrZSSp2ILZr861PstlUezsGgHJ7pwoZy5k19+Uyv1CvTYV4tp4cwjoiMh39QP+ZUoMt/ptrbqY+P6VVi2P2wRAr6xBPsUoSADQbT1Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=hqT2wso/0hYRxptcVRWsFjsJ4DzNBQnDU4voR1s8jIo=; b=fMDETF/QKLs5XJrwlyQJVv3/hELXzZ5osPlpx5Dp+m1FdnFsCGe7Dt8vuaMgdUdC+aTxHKczz001v56nw3tDSGyO3oN5OiaRFq2KLpJs1aMui0Cw7MibIdaqcv/Lp0asPoOdJpgwpxXkR84OaKu0Px1hHTVfZydX5H44Ckvs0H51FiQjT8k/dgXvaFodzKKXU8qogwwoGlxyAlKD0s+LA7YepttNGQbGe3ryyuAu8dG3BfFozKzoTV35rG2IDK61EI2wpHhooBqcpdiUEN4G8TMVXGO+n+40/GoSKlLbVG+4mKEiK/qUV+UPowL7ZPLqfZJm3zhgx7ty4xRvc2CoJA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=stgolabs.net smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=hqT2wso/0hYRxptcVRWsFjsJ4DzNBQnDU4voR1s8jIo=; b=e8edoueBGCZpvuw2XWbkIsHLDwFl6Jr9SF+xIqlrz6LuDA4OY96Q/Ol5UfC7a0KWT2dDKLvtrb6tXpLWID+/v0T4Rm8PFrdtyIP/hDCnykewsibamlRjIMHwWMNExE2gf+AEmSTqj7r/o/Z/BBh2PjjZ4CD/Q40ysfWgB8Gn1SA= Received: from PR1P264CA0043.FRAP264.PROD.OUTLOOK.COM (2603:10a6:102:2cb::20) by CY5PR12MB6322.namprd12.prod.outlook.com (2603:10b6:930:21::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.23; Wed, 18 Oct 2023 17:20:35 +0000 Received: from SN1PEPF0002529E.namprd05.prod.outlook.com (2603:10a6:102:2cb:cafe::d6) by PR1P264CA0043.outlook.office365.com (2603:10a6:102:2cb::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.35 via Frontend Transport; Wed, 18 Oct 2023 17:20:35 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SN1PEPF0002529E.mail.protection.outlook.com (10.167.242.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.6907.20 via Frontend Transport; Wed, 18 Oct 2023 17:20:34 +0000 Received: from rric.localdomain (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Wed, 18 Oct 2023 12:20:04 -0500 From: Robert Richter To: Davidlohr Bueso , Jonathan Cameron , Dave Jiang , "Alison Schofield" , Vishal Verma , Ira Weiny , Ben Widawsky , Dan Williams CC: , , Bjorn Helgaas , Terry Bowman , Robert Richter , Jonathan Cameron Subject: [PATCH v12 15/20] cxl/pci: Add RCH downstream port error logging Date: Wed, 18 Oct 2023 19:17:08 +0200 Message-ID: <20231018171713.1883517-16-rrichter@amd.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20231018171713.1883517-1-rrichter@amd.com> References: <20231018171713.1883517-1-rrichter@amd.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF0002529E:EE_|CY5PR12MB6322:EE_ X-MS-Office365-Filtering-Correlation-Id: 179a2966-772e-460e-9aa1-08dbcffe89e3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: tA73G/OR/wT/cVze8ZDfYUa1NPdkFgZkL2p4EHdVTSCnq5cmLOR4m9YXbmpxgx3Z4tDeacNX7CYsugz+BblcIeT/GNXEAD6rsNC3GnAnv+M8BY9nQ0OthZdhywCUksfhGIbPazIkNyfeKbs22EabRPAPdDmjweqv5avWxwPmLLRIiVMhyHFz/cfFuQlfrSbJ6o3PVUV6FdYL7rkdb5AuQ03ZMXSiuG87rfw9JBL5EbBtToo/w6yM4WpYoLY4qqtGjZ5PLepmPO/lFwBcegcXEfzTEqfaISJg3Dg1573zT7YCPOHkElMQb6GL7g5CBAJRVr7TjC8NtsyWaBORFavg+7lSOKZ0m+mzKNEBg1QxcLVhFGnlyt45yt1xvIs9auNZf9jhnOW0KBSiHl3gfyD69Z08S3xQ7wfbO+5SBOBfj6GOvM8fnhu9eNK0skPi2/SppXSsgozw9kaRg9SLY1Dd6E+YgdPR1xRC8b9HdrSxKfME5JNQQn67ZKN8GHKpq4lz1MSGgSiqMBB+SpzARB2M86+k7Fk0qHCGMo55+eaGG2RaRtRpLM02/cAeVgcT4QeV/nPUeuYjy/povX2+mcht7Lzgw0TO3wivUAmFsorMjNAY8/lO1W00RG5zuT0stS8U45mGl6AaOugqFGSkabX4X6aRyPzI0OMP0bO88nihuKK6p6Gt21TwWX6c2Bt/ww2Ofigw2NRFBGp2ghqZvbteYZgdKQUJGJ6JUaz0zqFhBqyxWKX09EDxoEiZ6C4zRwOExNV0OAQ5vA7IR9P+52oYfQ== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230031)(4636009)(346002)(376002)(136003)(39860400002)(396003)(230922051799003)(64100799003)(1800799009)(186009)(82310400011)(451199024)(46966006)(40470700004)(36840700001)(40460700003)(40480700001)(70586007)(70206006)(54906003)(110136005)(41300700001)(316002)(478600001)(36756003)(5660300002)(8676002)(8936002)(4326008)(2906002)(36860700001)(81166007)(7416002)(356005)(336012)(47076005)(2616005)(1076003)(426003)(26005)(82740400003)(16526019)(83380400001)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2023 17:20:34.4469 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 179a2966-772e-460e-9aa1-08dbcffe89e3 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF0002529E.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB6322 X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net From: Terry Bowman RCH downstream port error logging is missing in the current CXL driver. The missing AER and RAS error logging is needed for communicating driver error details to userspace. Update the driver to include PCIe AER and CXL RAS error logging. Add RCH downstream port error handling into the existing RCiEP handler. The downstream port error handler is added to the RCiEP error handler because the downstream port is implemented in a RCRB, is not PCI enumerable, and as a result is not directly accessible to the PCI AER root port driver. The AER root port driver calls the RCiEP handler for handling RCD errors and RCH downstream port protocol errors. Update existing RCiEP correctable and uncorrectable handlers to also call the RCH handler. The RCH handler will read the RCH AER registers, check for error severity, and if an error exists will log using an existing kernel AER trace routine. The RCH handler will also log downstream port RAS errors if they exist. Co-developed-by: Robert Richter Signed-off-by: Terry Bowman Signed-off-by: Robert Richter Reviewed-by: Jonathan Cameron Reviewed-by: Dave Jiang --- drivers/cxl/core/pci.c | 96 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 96 insertions(+) diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index 3b4bb8d05035..3c85fd1ae5a9 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -777,12 +777,105 @@ void cxl_setup_parent_dport(struct device *host, struct cxl_dport *dport) } EXPORT_SYMBOL_NS_GPL(cxl_setup_parent_dport, CXL); +static void cxl_handle_rdport_cor_ras(struct cxl_dev_state *cxlds, + struct cxl_dport *dport) +{ + return __cxl_handle_cor_ras(cxlds, dport->regs.ras); +} + +static bool cxl_handle_rdport_ras(struct cxl_dev_state *cxlds, + struct cxl_dport *dport) +{ + return __cxl_handle_ras(cxlds, dport->regs.ras); +} + +/* + * Copy the AER capability registers using 32 bit read accesses. + * This is necessary because RCRB AER capability is MMIO mapped. Clear the + * status after copying. + * + * @aer_base: base address of AER capability block in RCRB + * @aer_regs: destination for copying AER capability + */ +static bool cxl_rch_get_aer_info(void __iomem *aer_base, + struct aer_capability_regs *aer_regs) +{ + int read_cnt = sizeof(struct aer_capability_regs) / sizeof(u32); + u32 *aer_regs_buf = (u32 *)aer_regs; + int n; + + if (!aer_base) + return false; + + /* Use readl() to guarantee 32-bit accesses */ + for (n = 0; n < read_cnt; n++) + aer_regs_buf[n] = readl(aer_base + n * sizeof(u32)); + + writel(aer_regs->uncor_status, aer_base + PCI_ERR_UNCOR_STATUS); + writel(aer_regs->cor_status, aer_base + PCI_ERR_COR_STATUS); + + return true; +} + +/* Get AER severity. Return false if there is no error. */ +static bool cxl_rch_get_aer_severity(struct aer_capability_regs *aer_regs, + int *severity) +{ + if (aer_regs->uncor_status & ~aer_regs->uncor_mask) { + if (aer_regs->uncor_status & PCI_ERR_ROOT_FATAL_RCV) + *severity = AER_FATAL; + else + *severity = AER_NONFATAL; + return true; + } + + if (aer_regs->cor_status & ~aer_regs->cor_mask) { + *severity = AER_CORRECTABLE; + return true; + } + + return false; +} + +static void cxl_handle_rdport_errors(struct cxl_dev_state *cxlds) +{ + struct pci_dev *pdev = to_pci_dev(cxlds->dev); + struct aer_capability_regs aer_regs; + struct cxl_dport *dport; + struct cxl_port *port; + int severity; + + port = cxl_pci_find_port(pdev, &dport); + if (!port) + return; + + put_device(&port->dev); + + if (!cxl_rch_get_aer_info(dport->regs.dport_aer, &aer_regs)) + return; + + if (!cxl_rch_get_aer_severity(&aer_regs, &severity)) + return; + + pci_print_aer(pdev, severity, &aer_regs); + + if (severity == AER_CORRECTABLE) + cxl_handle_rdport_cor_ras(cxlds, dport); + else + cxl_handle_rdport_ras(cxlds, dport); +} + +#else +static void cxl_handle_rdport_errors(struct cxl_dev_state *cxlds) { } #endif void cxl_cor_error_detected(struct pci_dev *pdev) { struct cxl_dev_state *cxlds = pci_get_drvdata(pdev); + if (cxlds->rcd) + cxl_handle_rdport_errors(cxlds); + cxl_handle_endpoint_cor_ras(cxlds); } EXPORT_SYMBOL_NS_GPL(cxl_cor_error_detected, CXL); @@ -795,6 +888,9 @@ pci_ers_result_t cxl_error_detected(struct pci_dev *pdev, struct device *dev = &cxlmd->dev; bool ue; + if (cxlds->rcd) + cxl_handle_rdport_errors(cxlds); + /* * A frozen channel indicates an impending reset which is fatal to * CXL.mem operation, and will likely crash the system. On the off