From patchwork Fri Sep 16 23:11:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 12978881 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94ECAECAAA1 for ; Fri, 16 Sep 2022 23:11:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229505AbiIPXLD (ORCPT ); Fri, 16 Sep 2022 19:11:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55662 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229769AbiIPXLC (ORCPT ); Fri, 16 Sep 2022 19:11:02 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 72ADA7AC29 for ; Fri, 16 Sep 2022 16:11:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1663369861; x=1694905861; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=rhRR/mm1cOKPCO7PS3Aui4wKh7B/XjO7DTe4et01Dow=; b=EVs7nmU6GW4sdlaPgD8uIhR9frWiHkTHC+JoxzPM1De3p/FnNcFC59yJ 8kovk9e7yEBzlqa1sNtVcu6AA77tXwUf2k5G2SEFjAcdlXgZ3n/JIOObG ZmY1jNkts5xV2CCs244y58yutmi6VcXgY8cUPHINlj/tfXBvEEU+zIj6/ ELDMD/gfS4YVbjBze7CIxQooaIJpeiUzF5A5ririi97U1MRfRUb3i4/9b 9nufOpX+BBJmw0tAH7m0+7ox6RwJ0sCGXYeQSpyl1SZCjlDgD9rYonNnY 3Yq+rp0teYnUaQsGF7ktRVuuhWjcvaAb89i5DolUmJTIdAC3/ydnueXAB g==; X-IronPort-AV: E=McAfee;i="6500,9779,10472"; a="286138525" X-IronPort-AV: E=Sophos;i="5.93,321,1654585200"; d="scan'208";a="286138525" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Sep 2022 16:11:01 -0700 X-IronPort-AV: E=Sophos;i="5.93,321,1654585200"; d="scan'208";a="686305078" Received: from djiang5-desk3.ch.intel.com ([143.182.136.137]) by fmsmga004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Sep 2022 16:11:00 -0700 Subject: [PATCH RFC v2 1/9] cxl/pci: Cleanup repeated code in cxl_probe_regs() helpers From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: alison.schofield@intel.com, vishal.l.verma@intel.com, bwidawsk@kernel.org, dan.j.williams@intel.com, jonathan.cameron@huawei.com, shiju.jose@huawei.com, rrichter@amd.com Date: Fri, 16 Sep 2022 16:11:00 -0700 Message-ID: <166336986012.3803215.4343997952762297862.stgit@djiang5-desk3.ch.intel.com> In-Reply-To: <166336972295.3803215.1047199449525031921.stgit@djiang5-desk3.ch.intel.com> References: <166336972295.3803215.1047199449525031921.stgit@djiang5-desk3.ch.intel.com> User-Agent: StGit/1.4 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org From: Dan Williams Rather then duplicating the setting of valid, length, and offset for each type, just convey a pointer to the register map to common code. Yes, the change in cxl_probe_component_regs() does not save any lines of code, but it is preparation for adding another component register type to map (RAS Capability Structure). Signed-off-by: Dan Williams Reviewed-by: Jonathan Cameron Signed-off-by: Dave Jiang --- drivers/cxl/core/regs.c | 46 ++++++++++++++++++++++++++-------------------- 1 file changed, 26 insertions(+), 20 deletions(-) diff --git a/drivers/cxl/core/regs.c b/drivers/cxl/core/regs.c index 39a129c57d40..bd6ae14b679e 100644 --- a/drivers/cxl/core/regs.c +++ b/drivers/cxl/core/regs.c @@ -59,36 +59,41 @@ void cxl_probe_component_regs(struct device *dev, void __iomem *base, for (cap = 1; cap <= cap_count; cap++) { void __iomem *register_block; - u32 hdr; - int decoder_cnt; + struct cxl_reg_map *rmap; u16 cap_id, offset; - u32 length; + u32 length, hdr; hdr = readl(base + cap * 0x4); cap_id = FIELD_GET(CXL_CM_CAP_HDR_ID_MASK, hdr); offset = FIELD_GET(CXL_CM_CAP_PTR_MASK, hdr); register_block = base + offset; + hdr = readl(register_block); + rmap = NULL; switch (cap_id) { - case CXL_CM_CAP_CAP_ID_HDM: + case CXL_CM_CAP_CAP_ID_HDM: { + int decoder_cnt; + dev_dbg(dev, "found HDM decoder capability (0x%x)\n", offset); - hdr = readl(register_block); - decoder_cnt = cxl_hdm_decoder_count(hdr); length = 0x20 * decoder_cnt + 0x10; - - map->hdm_decoder.valid = true; - map->hdm_decoder.offset = CXL_CM_OFFSET + offset; - map->hdm_decoder.size = length; + rmap = &map->hdm_decoder; break; + } default: dev_dbg(dev, "Unknown CM cap ID: %d (0x%x)\n", cap_id, offset); break; } + + if (!rmap) + continue; + rmap->valid = true; + rmap->offset = CXL_CM_OFFSET + offset; + rmap->size = length; } } EXPORT_SYMBOL_NS_GPL(cxl_probe_component_regs, CXL); @@ -117,6 +122,7 @@ void cxl_probe_device_regs(struct device *dev, void __iomem *base, cap_count = FIELD_GET(CXLDEV_CAP_ARRAY_COUNT_MASK, cap_array); for (cap = 1; cap <= cap_count; cap++) { + struct cxl_reg_map *rmap; u32 offset, length; u16 cap_id; @@ -125,28 +131,22 @@ void cxl_probe_device_regs(struct device *dev, void __iomem *base, offset = readl(base + cap * 0x10 + 0x4); length = readl(base + cap * 0x10 + 0x8); + rmap = NULL; switch (cap_id) { case CXLDEV_CAP_CAP_ID_DEVICE_STATUS: dev_dbg(dev, "found Status capability (0x%x)\n", offset); - - map->status.valid = true; - map->status.offset = offset; - map->status.size = length; + rmap = &map->status; break; case CXLDEV_CAP_CAP_ID_PRIMARY_MAILBOX: dev_dbg(dev, "found Mailbox capability (0x%x)\n", offset); - map->mbox.valid = true; - map->mbox.offset = offset; - map->mbox.size = length; + rmap = &map->mbox; break; case CXLDEV_CAP_CAP_ID_SECONDARY_MAILBOX: dev_dbg(dev, "found Secondary Mailbox capability (0x%x)\n", offset); break; case CXLDEV_CAP_CAP_ID_MEMDEV: dev_dbg(dev, "found Memory Device capability (0x%x)\n", offset); - map->memdev.valid = true; - map->memdev.offset = offset; - map->memdev.size = length; + rmap = &map->memdev; break; default: if (cap_id >= 0x8000) @@ -155,6 +155,12 @@ void cxl_probe_device_regs(struct device *dev, void __iomem *base, dev_dbg(dev, "Unknown cap ID: %#x offset: %#x\n", cap_id, offset); break; } + + if (!rmap) + continue; + rmap->valid = true; + rmap->offset = offset; + rmap->size = length; } } EXPORT_SYMBOL_NS_GPL(cxl_probe_device_regs, CXL); From patchwork Fri Sep 16 23:11:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 12978882 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D114ECAAD8 for ; Fri, 16 Sep 2022 23:11:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229769AbiIPXLJ (ORCPT ); Fri, 16 Sep 2022 19:11:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56024 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229768AbiIPXLH (ORCPT ); Fri, 16 Sep 2022 19:11:07 -0400 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4AD8E78BF3 for ; Fri, 16 Sep 2022 16:11:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1663369867; x=1694905867; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=dtCnVY9W81yc9vix7z4DELknnnqJJU5aJ9NAFNn5thk=; b=Mwa73/VaiJmN4IBuzVmRbsu4LO3mlEBparCxmNI8hK0aYddW9sKVAZBM LcKoXVswmghbUTc0Fv7WD/FQUC1dUjSPWHWgStyzLiFrD2SgX0L8ZNMPr HTiV+agl1wqqz1pIDU/TlOPhVvP48GqpZKIH41dbEOxpY4EpxFKIbVZWz SBOQ8HpqfDckXykXrQ9AiGAseY7wOR00rFvRwSzJnwvACsVCq03iVgqgS IC9xET00eBcxvb7s95e5N75BpVR+zqygMsYHBMh3RY051djcLIrcEJrWN GBIpBJYB5DVFwzXtXpmvQF9xWgrKxCpn+++hXn9p5gDdzaa2xeDHi/NqK Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10472"; a="300467066" X-IronPort-AV: E=Sophos;i="5.93,321,1654585200"; d="scan'208";a="300467066" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Sep 2022 16:11:06 -0700 X-IronPort-AV: E=Sophos;i="5.93,321,1654585200"; d="scan'208";a="686305144" Received: from djiang5-desk3.ch.intel.com ([143.182.136.137]) by fmsmga004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Sep 2022 16:11:06 -0700 Subject: [PATCH RFC v2 2/9] cxl/pci: Cleanup cxl_map_device_regs() From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: alison.schofield@intel.com, vishal.l.verma@intel.com, bwidawsk@kernel.org, dan.j.williams@intel.com, jonathan.cameron@huawei.com, shiju.jose@huawei.com, rrichter@amd.com Date: Fri, 16 Sep 2022 16:11:05 -0700 Message-ID: <166336986590.3803215.8576219076379231069.stgit@djiang5-desk3.ch.intel.com> In-Reply-To: <166336972295.3803215.1047199449525031921.stgit@djiang5-desk3.ch.intel.com> References: <166336972295.3803215.1047199449525031921.stgit@djiang5-desk3.ch.intel.com> User-Agent: StGit/1.4 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org From: Dan Williams Use a loop to reduce the duplicated code in cxl_map_device_regs(). This is in preparation for deleting cxl_map_regs(). Signed-off-by: Dan Williams Reviewed-by: Jonathan Cameron Signed-off-by: Dave Jiang --- drivers/cxl/core/regs.c | 51 ++++++++++++++++++----------------------------- 1 file changed, 20 insertions(+), 31 deletions(-) diff --git a/drivers/cxl/core/regs.c b/drivers/cxl/core/regs.c index bd6ae14b679e..f03ede86412d 100644 --- a/drivers/cxl/core/regs.c +++ b/drivers/cxl/core/regs.c @@ -211,42 +211,31 @@ int cxl_map_device_regs(struct pci_dev *pdev, struct cxl_device_regs *regs, struct cxl_register_map *map) { + resource_size_t phys_addr = + pci_resource_start(pdev, map->barno) + map->block_offset; struct device *dev = &pdev->dev; - resource_size_t phys_addr; - - phys_addr = pci_resource_start(pdev, map->barno); - phys_addr += map->block_offset; - - if (map->device_map.status.valid) { - resource_size_t addr; + struct mapinfo { + struct cxl_reg_map *rmap; + void __iomem **addr; + } mapinfo[] = { + { &map->device_map.status, ®s->status, }, + { &map->device_map.mbox, ®s->mbox, }, + { &map->device_map.memdev, ®s->memdev, }, + }; + int i; + + for (i = 0; i < ARRAY_SIZE(mapinfo); i++) { + struct mapinfo *mi = &mapinfo[i]; resource_size_t length; - - addr = phys_addr + map->device_map.status.offset; - length = map->device_map.status.size; - regs->status = devm_cxl_iomap_block(dev, addr, length); - if (!regs->status) - return -ENOMEM; - } - - if (map->device_map.mbox.valid) { resource_size_t addr; - resource_size_t length; - addr = phys_addr + map->device_map.mbox.offset; - length = map->device_map.mbox.size; - regs->mbox = devm_cxl_iomap_block(dev, addr, length); - if (!regs->mbox) - return -ENOMEM; - } - - if (map->device_map.memdev.valid) { - resource_size_t addr; - resource_size_t length; + if (!mi->rmap->valid) + continue; - addr = phys_addr + map->device_map.memdev.offset; - length = map->device_map.memdev.size; - regs->memdev = devm_cxl_iomap_block(dev, addr, length); - if (!regs->memdev) + addr = phys_addr + mi->rmap->offset; + length = mi->rmap->size; + *(mi->addr) = devm_cxl_iomap_block(dev, addr, length); + if (!*(mi->addr)) return -ENOMEM; } From patchwork Fri Sep 16 23:11:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 12978883 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7F8FECAAD8 for ; Fri, 16 Sep 2022 23:11:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229588AbiIPXLO (ORCPT ); Fri, 16 Sep 2022 19:11:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56080 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229743AbiIPXLN (ORCPT ); Fri, 16 Sep 2022 19:11:13 -0400 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E9A0B78BF3 for ; Fri, 16 Sep 2022 16:11:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1663369872; x=1694905872; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=hNrYkKwQSE+7jv9mF65FLXwsqKUJeqVGVDw1l9G3nCs=; b=IBoJ9Qwhab9+8J0Nq9rt4MHSwNksTyDlq1jIvLVN1CxEymi+Cc9TImcn cZVOrHqpW6boPooaHl8luZeC5iWM6CexmPoeh6bfgaj/gnfvZvNcPaGYg gP1L5f43bZrD6zAVsuhYSOJ5G4haiUaa6c5BpHHHKJtSsh1U82tiET1J5 hH+7iAcW1hwfRdpdwEI5CD4mPVjnqCpQwWzfEMsius0/IJJvEO5nFlsLx sHY7olFNqKGkJnNriGw5cTOdXtE1b9AJHMc5P/6YFK+BexYhdRZVdEx9x WtwnoIZ/qMZCiX5jfLcfoAUtDRLhCBiwZ6iAWoXKFNzNysTXA/8aMVC9q g==; X-IronPort-AV: E=McAfee;i="6500,9779,10472"; a="360836711" X-IronPort-AV: E=Sophos;i="5.93,321,1654585200"; d="scan'208";a="360836711" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Sep 2022 16:11:12 -0700 X-IronPort-AV: E=Sophos;i="5.93,321,1654585200"; d="scan'208";a="686305183" Received: from djiang5-desk3.ch.intel.com ([143.182.136.137]) by fmsmga004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Sep 2022 16:11:11 -0700 Subject: [PATCH RFC v2 3/9] cxl/pci: Kill cxl_map_regs() From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: alison.schofield@intel.com, vishal.l.verma@intel.com, bwidawsk@kernel.org, dan.j.williams@intel.com, jonathan.cameron@huawei.com, shiju.jose@huawei.com, rrichter@amd.com Date: Fri, 16 Sep 2022 16:11:11 -0700 Message-ID: <166336987163.3803215.16274917380401720231.stgit@djiang5-desk3.ch.intel.com> In-Reply-To: <166336972295.3803215.1047199449525031921.stgit@djiang5-desk3.ch.intel.com> References: <166336972295.3803215.1047199449525031921.stgit@djiang5-desk3.ch.intel.com> User-Agent: StGit/1.4 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org From: Dan Williams The component registers are currently unused by the cxl_pci driver. Only the physical address base of the component registers is conveyed to the cxl_mem driver. Just call cxl_map_device_registers() directly. Signed-off-by: Dan Williams Reviewed-by: Jonathan Cameron Signed-off-by: Dave Jiang --- drivers/cxl/pci.c | 23 +---------------------- 1 file changed, 1 insertion(+), 22 deletions(-) diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index faeb5d9d7a7a..82023cf0cdcf 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -347,27 +347,6 @@ static int cxl_probe_regs(struct pci_dev *pdev, struct cxl_register_map *map) return 0; } -static int cxl_map_regs(struct cxl_dev_state *cxlds, struct cxl_register_map *map) -{ - struct device *dev = cxlds->dev; - struct pci_dev *pdev = to_pci_dev(dev); - - switch (map->reg_type) { - case CXL_REGLOC_RBI_COMPONENT: - cxl_map_component_regs(pdev, &cxlds->regs.component, map); - dev_dbg(dev, "Mapping component registers...\n"); - break; - case CXL_REGLOC_RBI_MEMDEV: - cxl_map_device_regs(pdev, &cxlds->regs.device_regs, map); - dev_dbg(dev, "Probing device registers...\n"); - break; - default: - break; - } - - return 0; -} - static int cxl_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type, struct cxl_register_map *map) { @@ -461,7 +440,7 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) if (rc) return rc; - rc = cxl_map_regs(cxlds, &map); + rc = cxl_map_device_regs(pdev, &cxlds->regs.device_regs, &map); if (rc) return rc; From patchwork Fri Sep 16 23:11:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 12978884 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D5FDECAAA1 for ; Fri, 16 Sep 2022 23:11:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229773AbiIPXLU (ORCPT ); Fri, 16 Sep 2022 19:11:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56168 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229768AbiIPXLT (ORCPT ); Fri, 16 Sep 2022 19:11:19 -0400 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 25D137AC29 for ; Fri, 16 Sep 2022 16:11:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1663369878; x=1694905878; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Qx0CPoL5bUDHfVFx9WvBLWGjXfN2u8tYalmt3rEWMYg=; b=cZ+b+bNX3ppDotWouAZxVarheg3kJO83kBqFm6IU548de59rTFWl9MvM lFADlH4GN67E//RKHahesEogM5XXSwjci+BANlZWNetaJuFnXIOiq0LWb qr2DagR0KrHGKEimmmpMqOutmfzcoH12SzFbqW69oSpRsIU6Dg+bdhv8R 8O82+74dG/Z78K1rZJv9wmJ+T75XBBbSE9twP7q6LvZ3UMOU90y1Jtexg Et8oQMtb+FgP5BkmBnbs8O4/pDyNZJZ7tjiEQ0FXOrVPexgi421eTky0V h5bVBgPrHTa53o0aEGve+GwOtpzXDS+U2pJFr/aA/uajm2yY/S1+O6Xod A==; X-IronPort-AV: E=McAfee;i="6500,9779,10472"; a="278825706" X-IronPort-AV: E=Sophos;i="5.93,321,1654585200"; d="scan'208";a="278825706" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Sep 2022 16:11:17 -0700 X-IronPort-AV: E=Sophos;i="5.93,321,1654585200"; d="scan'208";a="686305217" Received: from djiang5-desk3.ch.intel.com ([143.182.136.137]) by fmsmga004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Sep 2022 16:11:17 -0700 Subject: [PATCH RFC v2 4/9] cxl/core/regs: Make cxl_map_{component, device}_regs() device generic From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: alison.schofield@intel.com, vishal.l.verma@intel.com, bwidawsk@kernel.org, dan.j.williams@intel.com, jonathan.cameron@huawei.com, shiju.jose@huawei.com, rrichter@amd.com Date: Fri, 16 Sep 2022 16:11:17 -0700 Message-ID: <166336987730.3803215.14153753062109839645.stgit@djiang5-desk3.ch.intel.com> In-Reply-To: <166336972295.3803215.1047199449525031921.stgit@djiang5-desk3.ch.intel.com> References: <166336972295.3803215.1047199449525031921.stgit@djiang5-desk3.ch.intel.com> User-Agent: StGit/1.4 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org From: Dan Williams There is no need to carry the barno and the block offset through the stack, just convert them to a resource base immediately. Signed-off-by: Dan Williams Reviewed-by: Jonathan Cameron Signed-off-by: Dave Jiang --- drivers/cxl/core/pci.c | 3 +-- drivers/cxl/core/port.c | 2 +- drivers/cxl/core/regs.c | 40 +++++++++++++++++++++++----------------- drivers/cxl/cxl.h | 14 ++++++-------- drivers/cxl/cxlpci.h | 9 --------- drivers/cxl/pci.c | 25 ++++++------------------- 6 files changed, 37 insertions(+), 56 deletions(-) diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index 9240df53ed87..2a8a88d38533 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -54,8 +54,7 @@ static int match_add_dports(struct pci_dev *pdev, void *data) dev_dbg(&port->dev, "failed to find component registers\n"); port_num = FIELD_GET(PCI_EXP_LNKCAP_PN, lnkcap); - dport = devm_cxl_add_dport(port, &pdev->dev, port_num, - cxl_regmap_to_base(pdev, &map)); + dport = devm_cxl_add_dport(port, &pdev->dev, port_num, map.resource); if (IS_ERR(dport)) { ctx->error = PTR_ERR(dport); return PTR_ERR(dport); diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c index bffde862de0b..2c78b28dbcc6 100644 --- a/drivers/cxl/core/port.c +++ b/drivers/cxl/core/port.c @@ -1236,7 +1236,7 @@ static resource_size_t find_component_registers(struct device *dev) pdev = to_pci_dev(dev); cxl_find_regblock(pdev, CXL_REGLOC_RBI_COMPONENT, &map); - return cxl_regmap_to_base(pdev, &map); + return map.resource; } static int add_port_attach_ep(struct cxl_memdev *cxlmd, diff --git a/drivers/cxl/core/regs.c b/drivers/cxl/core/regs.c index f03ede86412d..e4b0d52ac3a1 100644 --- a/drivers/cxl/core/regs.c +++ b/drivers/cxl/core/regs.c @@ -186,17 +186,13 @@ void __iomem *devm_cxl_iomap_block(struct device *dev, resource_size_t addr, return ret_val; } -int cxl_map_component_regs(struct pci_dev *pdev, - struct cxl_component_regs *regs, +int cxl_map_component_regs(struct device *dev, struct cxl_component_regs *regs, struct cxl_register_map *map) { - struct device *dev = &pdev->dev; resource_size_t phys_addr; resource_size_t length; - phys_addr = pci_resource_start(pdev, map->barno); - phys_addr += map->block_offset; - + phys_addr = map->resource; phys_addr += map->component_map.hdm_decoder.offset; length = map->component_map.hdm_decoder.size; regs->hdm_decoder = devm_cxl_iomap_block(dev, phys_addr, length); @@ -207,13 +203,11 @@ int cxl_map_component_regs(struct pci_dev *pdev, } EXPORT_SYMBOL_NS_GPL(cxl_map_component_regs, CXL); -int cxl_map_device_regs(struct pci_dev *pdev, +int cxl_map_device_regs(struct device *dev, struct cxl_device_regs *regs, struct cxl_register_map *map) { - resource_size_t phys_addr = - pci_resource_start(pdev, map->barno) + map->block_offset; - struct device *dev = &pdev->dev; + resource_size_t phys_addr = map->resource; struct mapinfo { struct cxl_reg_map *rmap; void __iomem **addr; @@ -243,13 +237,24 @@ int cxl_map_device_regs(struct pci_dev *pdev, } EXPORT_SYMBOL_NS_GPL(cxl_map_device_regs, CXL); -static void cxl_decode_regblock(u32 reg_lo, u32 reg_hi, +static bool cxl_decode_regblock(struct pci_dev *pdev, u32 reg_lo, u32 reg_hi, struct cxl_register_map *map) { - map->block_offset = ((u64)reg_hi << 32) | - (reg_lo & CXL_DVSEC_REG_LOCATOR_BLOCK_OFF_LOW_MASK); - map->barno = FIELD_GET(CXL_DVSEC_REG_LOCATOR_BIR_MASK, reg_lo); + int bar = FIELD_GET(CXL_DVSEC_REG_LOCATOR_BIR_MASK, reg_lo); + u64 offset = ((u64)reg_hi << 32) | + (reg_lo & CXL_DVSEC_REG_LOCATOR_BLOCK_OFF_LOW_MASK); + + if (offset > pci_resource_len(pdev, bar)) { + dev_warn(&pdev->dev, + "BAR%d: %pr: too small (offset: %pa, type: %d)\n", bar, + &pdev->resource[bar], &offset, map->reg_type); + return false; + } + map->reg_type = FIELD_GET(CXL_DVSEC_REG_LOCATOR_BLOCK_ID_MASK, reg_lo); + map->resource = pci_resource_start(pdev, bar) + offset; + map->max_size = pci_resource_len(pdev, bar) - offset; + return true; } /** @@ -269,7 +274,7 @@ int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type, u32 regloc_size, regblocks; int regloc, i; - map->block_offset = U64_MAX; + map->resource = CXL_RESOURCE_NONE; regloc = pci_find_dvsec_capability(pdev, PCI_DVSEC_VENDOR_ID_CXL, CXL_DVSEC_REG_LOCATOR); if (!regloc) @@ -287,13 +292,14 @@ int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type, pci_read_config_dword(pdev, regloc, ®_lo); pci_read_config_dword(pdev, regloc + 4, ®_hi); - cxl_decode_regblock(reg_lo, reg_hi, map); + if (!cxl_decode_regblock(pdev, reg_lo, reg_hi, map)) + continue; if (map->reg_type == type) return 0; } - map->block_offset = U64_MAX; + map->resource = CXL_RESOURCE_NONE; return -ENODEV; } EXPORT_SYMBOL_NS_GPL(cxl_find_regblock, CXL); diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index f680450f0b16..75506286f39c 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -187,17 +187,17 @@ struct cxl_device_reg_map { /** * struct cxl_register_map - DVSEC harvested register block mapping parameters * @base: virtual base of the register-block-BAR + @block_offset - * @block_offset: offset to start of register block in @barno + * @resource: physical resource base of the register block + * @max_size: maximum mapping size to perform register search * @reg_type: see enum cxl_regloc_type - * @barno: PCI BAR number containing the register block * @component_map: cxl_reg_map for component registers * @device_map: cxl_reg_maps for device registers */ struct cxl_register_map { void __iomem *base; - u64 block_offset; + resource_size_t resource; + resource_size_t max_size; u8 reg_type; - u8 barno; union { struct cxl_component_reg_map component_map; struct cxl_device_reg_map device_map; @@ -208,11 +208,9 @@ void cxl_probe_component_regs(struct device *dev, void __iomem *base, struct cxl_component_reg_map *map); void cxl_probe_device_regs(struct device *dev, void __iomem *base, struct cxl_device_reg_map *map); -int cxl_map_component_regs(struct pci_dev *pdev, - struct cxl_component_regs *regs, +int cxl_map_component_regs(struct device *dev, struct cxl_component_regs *regs, struct cxl_register_map *map); -int cxl_map_device_regs(struct pci_dev *pdev, - struct cxl_device_regs *regs, +int cxl_map_device_regs(struct device *dev, struct cxl_device_regs *regs, struct cxl_register_map *map); enum cxl_regloc_type; diff --git a/drivers/cxl/cxlpci.h b/drivers/cxl/cxlpci.h index eec597dbe763..920909791bb9 100644 --- a/drivers/cxl/cxlpci.h +++ b/drivers/cxl/cxlpci.h @@ -62,15 +62,6 @@ enum cxl_regloc_type { CXL_REGLOC_RBI_TYPES }; -static inline resource_size_t cxl_regmap_to_base(struct pci_dev *pdev, - struct cxl_register_map *map) -{ - if (map->block_offset == U64_MAX) - return CXL_RESOURCE_NONE; - - return pci_resource_start(pdev, map->barno) + map->block_offset; -} - int devm_cxl_port_enumerate_dports(struct cxl_port *port); struct cxl_dev_state; int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm); diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index 82023cf0cdcf..aba31c2291c4 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -276,35 +276,22 @@ static int cxl_pci_setup_mailbox(struct cxl_dev_state *cxlds) static int cxl_map_regblock(struct pci_dev *pdev, struct cxl_register_map *map) { - void __iomem *addr; - int bar = map->barno; struct device *dev = &pdev->dev; - resource_size_t offset = map->block_offset; - /* Basic sanity check that BAR is big enough */ - if (pci_resource_len(pdev, bar) < offset) { - dev_err(dev, "BAR%d: %pr: too small (offset: %pa)\n", bar, - &pdev->resource[bar], &offset); - return -ENXIO; - } - - addr = pci_iomap(pdev, bar, 0); - if (!addr) { + map->base = ioremap(map->resource, map->max_size); + if (!map->base) { dev_err(dev, "failed to map registers\n"); return -ENOMEM; } - dev_dbg(dev, "Mapped CXL Memory Device resource bar %u @ %pa\n", - bar, &offset); - - map->base = addr + map->block_offset; + dev_dbg(dev, "Mapped CXL Memory Device resource %pa\n", &map->resource); return 0; } static void cxl_unmap_regblock(struct pci_dev *pdev, struct cxl_register_map *map) { - pci_iounmap(pdev, map->base - map->block_offset); + iounmap(map->base); map->base = NULL; } @@ -440,7 +427,7 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) if (rc) return rc; - rc = cxl_map_device_regs(pdev, &cxlds->regs.device_regs, &map); + rc = cxl_map_device_regs(&pdev->dev, &cxlds->regs.device_regs, &map); if (rc) return rc; @@ -453,7 +440,7 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) if (rc) dev_warn(&pdev->dev, "No component registers (%d)\n", rc); - cxlds->component_reg_phys = cxl_regmap_to_base(pdev, &map); + cxlds->component_reg_phys = map.resource; devm_cxl_pci_create_doe(cxlds); From patchwork Fri Sep 16 23:11:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 12978885 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49DF6ECAAD8 for ; Fri, 16 Sep 2022 23:11:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229768AbiIPXL0 (ORCPT ); Fri, 16 Sep 2022 19:11:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56380 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229774AbiIPXLZ (ORCPT ); Fri, 16 Sep 2022 19:11:25 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A4A339D11A for ; Fri, 16 Sep 2022 16:11:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1663369884; x=1694905884; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=k5cHlGWY4EmcsQsYex6FDcP/zis5eJbzwrkle93UcAM=; b=inJyeKpCdu42MBRkSM9b22R3ASe/9Gx6R7uam2/jz9IQ4OAOHC5jHbu6 dtroi/D8TebpTCa7XnQxmqH/+PSYNp3i0A5+xQjJUScuE4Ho1qHJsz4LG d+g7/24s1zUFt3l82uYwDzW9WNsvWRJQ44iFcdwWpUE9+Zo+07CQYbifh na5sxlU9fJtrIHzhHtq248HPQerwbYyk2wPGqDa0drtnpornG+3ti/F4Z eSZfJNi9BMQOL1Ta2XvE9hyUcoPfH0HaWLNq+3mt1SjhYgfQSdsXB3+iz 524hB3Oqyy50Ztdp+uBZw3PYnBNrWXxoixGGuAaPQWmbdaNV6oGjep4DZ A==; X-IronPort-AV: E=McAfee;i="6500,9779,10472"; a="286138600" X-IronPort-AV: E=Sophos;i="5.93,321,1654585200"; d="scan'208";a="286138600" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Sep 2022 16:11:23 -0700 X-IronPort-AV: E=Sophos;i="5.93,321,1654585200"; d="scan'208";a="686305283" Received: from djiang5-desk3.ch.intel.com ([143.182.136.137]) by fmsmga004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Sep 2022 16:11:23 -0700 Subject: [PATCH RFC v2 5/9] cxl/port: Limit the port driver to just the HDM Decoder Capability From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: alison.schofield@intel.com, vishal.l.verma@intel.com, bwidawsk@kernel.org, dan.j.williams@intel.com, jonathan.cameron@huawei.com, shiju.jose@huawei.com, rrichter@amd.com Date: Fri, 16 Sep 2022 16:11:22 -0700 Message-ID: <166336988294.3803215.7334374806071251168.stgit@djiang5-desk3.ch.intel.com> In-Reply-To: <166336972295.3803215.1047199449525031921.stgit@djiang5-desk3.ch.intel.com> References: <166336972295.3803215.1047199449525031921.stgit@djiang5-desk3.ch.intel.com> User-Agent: StGit/1.4 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org From: Dan Williams Update the port driver to use cxl_map_component_registers() so that the component register block can be shared between the cxl_pci driver and the cxl_port driver. I.e. stop the port driver from reserving the entire component register block for itself via request_region() when it only needs the HDM Decoder Capability subset. Signed-off-by: Dan Williams Signed-off-by: Dave Jiang Reviewed-by: Jonathan Cameron --- drivers/cxl/core/hdm.c | 32 ++++++++++++++++++-------------- 1 file changed, 18 insertions(+), 14 deletions(-) diff --git a/drivers/cxl/core/hdm.c b/drivers/cxl/core/hdm.c index d1d2caea5c62..061551148cfe 100644 --- a/drivers/cxl/core/hdm.c +++ b/drivers/cxl/core/hdm.c @@ -82,18 +82,22 @@ static void parse_hdm_decoder_caps(struct cxl_hdm *cxlhdm) cxlhdm->interleave_mask |= GENMASK(14, 12); } -static void __iomem *map_hdm_decoder_regs(struct cxl_port *port, - void __iomem *crb) +static int map_hdm_decoder_regs(struct cxl_port *port, void __iomem *crb, + struct cxl_component_regs *regs) { - struct cxl_component_reg_map map; + struct cxl_register_map map = { + .resource = port->component_reg_phys, + .base = crb, + .max_size = CXL_COMPONENT_REG_BLOCK_SIZE, + }; - cxl_probe_component_regs(&port->dev, crb, &map); - if (!map.hdm_decoder.valid) { + cxl_probe_component_regs(&port->dev, crb, &map.component_map); + if (!map.component_map.hdm_decoder.valid) { dev_err(&port->dev, "HDM decoder registers invalid\n"); - return IOMEM_ERR_PTR(-ENXIO); + return -ENXIO; } - return crb + map.hdm_decoder.offset; + return cxl_map_component_regs(&port->dev, regs, &map); } /** @@ -103,25 +107,25 @@ static void __iomem *map_hdm_decoder_regs(struct cxl_port *port, struct cxl_hdm *devm_cxl_setup_hdm(struct cxl_port *port) { struct device *dev = &port->dev; - void __iomem *crb, *hdm; struct cxl_hdm *cxlhdm; + void __iomem *crb; + int rc; cxlhdm = devm_kzalloc(dev, sizeof(*cxlhdm), GFP_KERNEL); if (!cxlhdm) return ERR_PTR(-ENOMEM); cxlhdm->port = port; - crb = devm_cxl_iomap_block(dev, port->component_reg_phys, - CXL_COMPONENT_REG_BLOCK_SIZE); + crb = ioremap(port->component_reg_phys, CXL_COMPONENT_REG_BLOCK_SIZE); if (!crb) { dev_err(dev, "No component registers mapped\n"); return ERR_PTR(-ENXIO); } - hdm = map_hdm_decoder_regs(port, crb); - if (IS_ERR(hdm)) - return ERR_CAST(hdm); - cxlhdm->regs.hdm_decoder = hdm; + rc = map_hdm_decoder_regs(port, crb, &cxlhdm->regs); + iounmap(crb); + if (rc) + return ERR_PTR(rc); parse_hdm_decoder_caps(cxlhdm); if (cxlhdm->decoder_count == 0) { From patchwork Fri Sep 16 23:11:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 12978886 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3324ECAAA1 for ; Fri, 16 Sep 2022 23:11:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229771AbiIPXLc (ORCPT ); Fri, 16 Sep 2022 19:11:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56454 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229774AbiIPXLa (ORCPT ); Fri, 16 Sep 2022 19:11:30 -0400 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 868CD8275B for ; Fri, 16 Sep 2022 16:11:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1663369889; x=1694905889; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=uS1g/Sz87ibNZ66Sfid55eY8k+J8IN+yR2u95PQLPy8=; b=gwlUhZjC9OG5qQqSoHVa+f5ZSbU5WuUE7m+bCaYaytD/fL1M3QSW65kM 0bhMoqlS1KHk2h8wRW7seglrt2Z5ziftWK1TucjOXUDcitKuXztyHTD+N 5TvioCmLHo/qeZkB9hvFodHOGZLAkSwmsU22rNarUo5igy2Byuk0y0752 Juy/cPbP6sXPtY7LB68YhiBDarmSOEuS160LhBTb5/eSiWk0nqq6AmHgO AMddGsjL1Lwhr8cUDF14xC/rZrulHesxGbvmxS++Pzk02p7c5FAxiluW1 eD15yVBiyBzf9awiTlzNuxvfowe+gcOXYIEHLPHypiCUmn5D0a2k1F0DB w==; X-IronPort-AV: E=McAfee;i="6500,9779,10472"; a="279480158" X-IronPort-AV: E=Sophos;i="5.93,321,1654585200"; d="scan'208";a="279480158" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Sep 2022 16:11:29 -0700 X-IronPort-AV: E=Sophos;i="5.93,321,1654585200"; d="scan'208";a="686305341" Received: from djiang5-desk3.ch.intel.com ([143.182.136.137]) by fmsmga004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Sep 2022 16:11:28 -0700 Subject: [PATCH RFC v2 6/9] cxl/pci: Prepare for mapping RAS Capability Structure From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: alison.schofield@intel.com, vishal.l.verma@intel.com, bwidawsk@kernel.org, dan.j.williams@intel.com, jonathan.cameron@huawei.com, shiju.jose@huawei.com, rrichter@amd.com Date: Fri, 16 Sep 2022 16:11:28 -0700 Message-ID: <166336988849.3803215.9304329275216442016.stgit@djiang5-desk3.ch.intel.com> In-Reply-To: <166336972295.3803215.1047199449525031921.stgit@djiang5-desk3.ch.intel.com> References: <166336972295.3803215.1047199449525031921.stgit@djiang5-desk3.ch.intel.com> User-Agent: StGit/1.4 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org From: Dan Williams The RAS Capabilitiy Structure is a CXL Component register capability block. Unlike the HDM Decoder Capability, it will be referenced by the cxl_pci driver in response to PCIe AER events. Due to this it is no longer the case that cxl_map_component_regs() can assume that it should map all component registers. Plumb a bitmask of capability ids to map through cxl_map_component_regs(). For symmetry cxl_probe_device_regs() is updated to populate @id in 'struct cxl_reg_map' even though cxl_map_device_regs() does not have a need to map a subset of the device registers per caller. Signed-off-by: Dan Williams Reviewed-by: Jonathan Cameron Signed-off-by: Dave Jiang --- drivers/cxl/core/hdm.c | 3 ++- drivers/cxl/core/regs.c | 36 ++++++++++++++++++++++++++---------- drivers/cxl/cxl.h | 4 +++- 3 files changed, 31 insertions(+), 12 deletions(-) diff --git a/drivers/cxl/core/hdm.c b/drivers/cxl/core/hdm.c index 061551148cfe..100d0881bde4 100644 --- a/drivers/cxl/core/hdm.c +++ b/drivers/cxl/core/hdm.c @@ -97,7 +97,8 @@ static int map_hdm_decoder_regs(struct cxl_port *port, void __iomem *crb, return -ENXIO; } - return cxl_map_component_regs(&port->dev, regs, &map); + return cxl_map_component_regs(&port->dev, regs, &map, + BIT(CXL_CM_CAP_CAP_ID_HDM)); } /** diff --git a/drivers/cxl/core/regs.c b/drivers/cxl/core/regs.c index e4b0d52ac3a1..97e8f4201493 100644 --- a/drivers/cxl/core/regs.c +++ b/drivers/cxl/core/regs.c @@ -92,6 +92,7 @@ void cxl_probe_component_regs(struct device *dev, void __iomem *base, if (!rmap) continue; rmap->valid = true; + rmap->id = cap_id; rmap->offset = CXL_CM_OFFSET + offset; rmap->size = length; } @@ -159,6 +160,7 @@ void cxl_probe_device_regs(struct device *dev, void __iomem *base, if (!rmap) continue; rmap->valid = true; + rmap->id = cap_id; rmap->offset = offset; rmap->size = length; } @@ -187,17 +189,31 @@ void __iomem *devm_cxl_iomap_block(struct device *dev, resource_size_t addr, } int cxl_map_component_regs(struct device *dev, struct cxl_component_regs *regs, - struct cxl_register_map *map) + struct cxl_register_map *map, unsigned long map_mask) { - resource_size_t phys_addr; - resource_size_t length; - - phys_addr = map->resource; - phys_addr += map->component_map.hdm_decoder.offset; - length = map->component_map.hdm_decoder.size; - regs->hdm_decoder = devm_cxl_iomap_block(dev, phys_addr, length); - if (!regs->hdm_decoder) - return -ENOMEM; + struct mapinfo { + struct cxl_reg_map *rmap; + void __iomem **addr; + } mapinfo[] = { + { &map->component_map.hdm_decoder, ®s->hdm_decoder }, + }; + int i; + + for (i = 0; i < ARRAY_SIZE(mapinfo); i++) { + struct mapinfo *mi = &mapinfo[i]; + resource_size_t phys_addr; + resource_size_t length; + + if (!mi->rmap->valid) + continue; + if (!test_bit(mi->rmap->id, &map_mask)) + continue; + phys_addr = map->resource + mi->rmap->offset; + length = mi->rmap->size; + *(mi->addr) = devm_cxl_iomap_block(dev, phys_addr, length); + if (!*(mi->addr)) + return -ENOMEM; + } return 0; } diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index 75506286f39c..c2bf1759d55c 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -170,6 +170,7 @@ struct cxl_regs { struct cxl_reg_map { bool valid; + int id; unsigned long offset; unsigned long size; }; @@ -209,7 +210,8 @@ void cxl_probe_component_regs(struct device *dev, void __iomem *base, void cxl_probe_device_regs(struct device *dev, void __iomem *base, struct cxl_device_reg_map *map); int cxl_map_component_regs(struct device *dev, struct cxl_component_regs *regs, - struct cxl_register_map *map); + struct cxl_register_map *map, + unsigned long map_mask); int cxl_map_device_regs(struct device *dev, struct cxl_device_regs *regs, struct cxl_register_map *map); From patchwork Fri Sep 16 23:11:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 12978887 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5BBFECAAD8 for ; Fri, 16 Sep 2022 23:11:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229774AbiIPXLh (ORCPT ); Fri, 16 Sep 2022 19:11:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56554 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229786AbiIPXLg (ORCPT ); Fri, 16 Sep 2022 19:11:36 -0400 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3CD5FA9C21 for ; Fri, 16 Sep 2022 16:11:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1663369895; x=1694905895; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=gg+3iSAVq1tvnupycmQNEBwDWW4jqnVdykYo+TLrFlk=; b=jwA2nU8IIR2h+NCZu4KJPUKMXa79uk0mPmsn6XQm3+cKbpBstVNB8Vdv TB1FEAGQJfYA66+LT1gTXLe/7LL4go0YdiHnWoplUMPS1Shr64seZ67R0 rLC6Oiliy8VBEnqC+/5PJBmxfWi8AYTYAZqugSqx3qZq9iYejK3fVR+4F y1yPBEIn/NyOEd3E42Cn+JU+JQc+2xa2Fy5IV0FRAP9Q8Uz05HUrzDqV9 +DuzydVnLex/082alCeVdTUWL0Oi52j2f9in2w8yPYFLH2llCmBlNTpSu araytP6WvVGaoCbsJ7vvVeANWUmGmCb2tHa2Mn3QIEduDAFsmuYQo+Nxb Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10472"; a="325374920" X-IronPort-AV: E=Sophos;i="5.93,321,1654585200"; d="scan'208";a="325374920" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Sep 2022 16:11:34 -0700 X-IronPort-AV: E=Sophos;i="5.93,321,1654585200"; d="scan'208";a="686305358" Received: from djiang5-desk3.ch.intel.com ([143.182.136.137]) by fmsmga004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Sep 2022 16:11:34 -0700 Subject: [PATCH RFC v2 7/9] cxl/pci: Find and map the RAS Capability Structure From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: alison.schofield@intel.com, vishal.l.verma@intel.com, bwidawsk@kernel.org, dan.j.williams@intel.com, jonathan.cameron@huawei.com, shiju.jose@huawei.com, rrichter@amd.com Date: Fri, 16 Sep 2022 16:11:34 -0700 Message-ID: <166336989411.3803215.14064525665389671272.stgit@djiang5-desk3.ch.intel.com> In-Reply-To: <166336972295.3803215.1047199449525031921.stgit@djiang5-desk3.ch.intel.com> References: <166336972295.3803215.1047199449525031921.stgit@djiang5-desk3.ch.intel.com> User-Agent: StGit/1.4 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org From: Dan Williams The RAS Capability Structure has some ancillary information that may be relevant with respect to AER events, link and protcol error status registers. Map the RAS Capability Registers in support of defining a 'struct pci_error_handlers' instance for the cxl_pci driver. Signed-off-by: Dan Williams Reviewed-by: Jonathan Cameron Signed-off-by: Dave Jiang --- drivers/cxl/core/regs.c | 7 +++++++ drivers/cxl/cxl.h | 19 +++++++++++++++++++ drivers/cxl/pci.c | 8 ++++++++ 3 files changed, 34 insertions(+) diff --git a/drivers/cxl/core/regs.c b/drivers/cxl/core/regs.c index 97e8f4201493..b10f8b79ec40 100644 --- a/drivers/cxl/core/regs.c +++ b/drivers/cxl/core/regs.c @@ -83,6 +83,12 @@ void cxl_probe_component_regs(struct device *dev, void __iomem *base, rmap = &map->hdm_decoder; break; } + case CXL_CM_CAP_CAP_ID_RAS: + dev_dbg(dev, "found RAS capability (0x%x)\n", + offset); + length = CXL_RAS_CAPABILITY_LENGTH; + rmap = &map->ras; + break; default: dev_dbg(dev, "Unknown CM cap ID: %d (0x%x)\n", cap_id, offset); @@ -196,6 +202,7 @@ int cxl_map_component_regs(struct device *dev, struct cxl_component_regs *regs, void __iomem **addr; } mapinfo[] = { { &map->component_map.hdm_decoder, ®s->hdm_decoder }, + { &map->component_map.ras, ®s->ras }, }; int i; diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index c2bf1759d55c..ce17ccd8b125 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -33,6 +33,7 @@ #define CXL_CM_CAP_HDR_ARRAY_SIZE_MASK GENMASK(31, 24) #define CXL_CM_CAP_PTR_MASK GENMASK(31, 20) +#define CXL_CM_CAP_CAP_ID_RAS 0x2 #define CXL_CM_CAP_CAP_ID_HDM 0x5 #define CXL_CM_CAP_CAP_HDM_VERSION 1 @@ -119,6 +120,21 @@ static inline int ways_to_cxl(unsigned int ways, u8 *iw) return 0; } +/* RAS Registers CXL 2.0 8.2.5.9 CXL RAS Capability Structure */ +#define CXL_RAS_UNCORRECTABLE_STATUS_OFFSET 0x0 +#define CXL_RAS_UNCORRECTABLE_STATUS_MASK (GENMASK(16, 14) | GENMASK(11, 0)) +#define CXL_RAS_UNCORRECTABLE_MASK_OFFSET 0x4 +#define CXL_RAS_UNCORRECTABLE_MASK_MASK (GENMASK(16, 14) | GENMASK(11, 0)) +#define CXL_RAS_UNCORRECTABLE_SEVERITY_OFFSET 0x8 +#define CXL_RAS_UNCORRECTABLE_SEVERITY_MASK (GENMASK(16, 14) | GENMASK(11, 0)) +#define CXL_RAS_CORRECTABLE_STATUS_OFFSET 0xC +#define CXL_RAS_CORRECTABLE_STATUS_MASK GENMASK(6, 0) +#define CXL_RAS_CORRECTABLE_MASK_OFFSET 0x10 +#define CXL_RAS_CORRECTABLE_MASK_MASK GENMASK(6, 0) +#define CXL_RAS_CAP_CONTROL_OFFSET 0x14 +#define CXL_RAS_HEADER_LOG_OFFSET 0x18 +#define CXL_RAS_CAPABILITY_LENGTH 0x58 + /* CXL 2.0 8.2.8.1 Device Capabilities Array Register */ #define CXLDEV_CAP_ARRAY_OFFSET 0x0 #define CXLDEV_CAP_ARRAY_CAP_ID 0 @@ -153,9 +169,11 @@ struct cxl_regs { /* * Common set of CXL Component register block base pointers * @hdm_decoder: CXL 2.0 8.2.5.12 CXL HDM Decoder Capability Structure + * @ras: CXL 2.0 8.2.5.9 CXL RAS Capability Structure */ struct_group_tagged(cxl_component_regs, component, void __iomem *hdm_decoder; + void __iomem *ras; ); /* * Common set of CXL Device register block base pointers @@ -177,6 +195,7 @@ struct cxl_reg_map { struct cxl_component_reg_map { struct cxl_reg_map hdm_decoder; + struct cxl_reg_map ras; }; struct cxl_device_reg_map { diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index aba31c2291c4..610b3a77f205 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -311,6 +311,9 @@ static int cxl_probe_regs(struct pci_dev *pdev, struct cxl_register_map *map) return -ENXIO; } + if (!comp_map->ras.valid) + dev_dbg(dev, "RAS registers not found\n"); + dev_dbg(dev, "Set up component registers\n"); break; case CXL_REGLOC_RBI_MEMDEV: @@ -444,6 +447,11 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) devm_cxl_pci_create_doe(cxlds); + rc = cxl_map_component_regs(&pdev->dev, &cxlds->regs.component, + &map, BIT(CXL_CM_CAP_CAP_ID_RAS)); + if (rc) + dev_dbg(&pdev->dev, "Failed to map RAS capability.\n"); + rc = cxl_pci_setup_mailbox(cxlds); if (rc) return rc; From patchwork Fri Sep 16 23:11:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 12978888 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5DA9ECAAA1 for ; Fri, 16 Sep 2022 23:11:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229786AbiIPXLn (ORCPT ); Fri, 16 Sep 2022 19:11:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56624 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229781AbiIPXLm (ORCPT ); Fri, 16 Sep 2022 19:11:42 -0400 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9A6357C32C for ; Fri, 16 Sep 2022 16:11:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1663369901; x=1694905901; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=e0YZmFJCpFJF3RrTonPHui8NqIYRM1Frx/pGXfoJwBs=; b=kzQPJdVnhmWq7YPfZZmj1mEOJLatFPf9hQhgEzP8Ebv92uhq9wJFNZvV lqjh/M2kO9ugmAYR2kl6EhQIkLICx0Jh4l5P0p3I5WwzRyGf57KDhS733 xkiBSxPsVTWBdeZ8rDWCA/WiCdqu3mTAH6GBWdI27X5KcHo87RlUa5RxO LA1wX7p4rwtd+X2w5RjZXdZhNU6lgqS9Y6NPjOLb/vf1GJtnCmDaV1qeo S+atKorbX1Z/deJI8sGQ5A/14QQ4LhVd2nDg4bj6gtQxVfRpV/1hE1i8V uy2Y23IWklq9ZCCYQyNTJbnRFsvGhw7pqxzBPGL94FX8tD3GgMf1T1rZY g==; X-IronPort-AV: E=McAfee;i="6500,9779,10472"; a="385389697" X-IronPort-AV: E=Sophos;i="5.93,321,1654585200"; d="scan'208";a="385389697" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Sep 2022 16:11:40 -0700 X-IronPort-AV: E=Sophos;i="5.93,321,1654585200"; d="scan'208";a="686305387" Received: from djiang5-desk3.ch.intel.com ([143.182.136.137]) by fmsmga004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Sep 2022 16:11:40 -0700 Subject: [PATCH RFC v2 8/9] cxl/pci: add tracepoint events for CXL RAS From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: alison.schofield@intel.com, vishal.l.verma@intel.com, bwidawsk@kernel.org, dan.j.williams@intel.com, jonathan.cameron@huawei.com, shiju.jose@huawei.com, rrichter@amd.com Date: Fri, 16 Sep 2022 16:11:39 -0700 Message-ID: <166336989980.3803215.5292431481210955312.stgit@djiang5-desk3.ch.intel.com> In-Reply-To: <166336972295.3803215.1047199449525031921.stgit@djiang5-desk3.ch.intel.com> References: <166336972295.3803215.1047199449525031921.stgit@djiang5-desk3.ch.intel.com> User-Agent: StGit/1.4 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org Add tracepoint events for recording the CXL uncorrectable and correctable errors. For uncorrectable errors, there is additional data up to 512B from the header log register (CXL spec rev3 8.2.4.16.7). The content of the register depends on the Uncorrectable Errors (UC) register status (CXL spec rev3 8.2.4.16.1). This implementation supports the Receiver_Overflow error where the definition is defined as first 3 bits of the Header Log data. The trace event will intake a dynamic array that will dump the Header Log data based on error. If multiple errors are set in the status register, then the 'first error' field (CXL spec rev3 v8.2.4.16.6) is read from the Error Capabilities and Control Register in order to determine the error. This implementation does not include CXL IDE Error details. Signed-off-by: Dave Jiang --- drivers/cxl/pci.c | 2 + include/trace/events/cxl_ras.h | 117 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 119 insertions(+) create mode 100644 include/trace/events/cxl_ras.h diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index 610b3a77f205..357de704e42c 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -13,6 +13,8 @@ #include "cxlmem.h" #include "cxlpci.h" #include "cxl.h" +#define CREATE_TRACE_POINTS +#include /** * DOC: cxl pci diff --git a/include/trace/events/cxl_ras.h b/include/trace/events/cxl_ras.h new file mode 100644 index 000000000000..6bb41c3b87c8 --- /dev/null +++ b/include/trace/events/cxl_ras.h @@ -0,0 +1,117 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#undef TRACE_SYSTEM +#define TRACE_SYSTEM cxl_ras + +#if !defined(_CXL_RAS_EVENTS_H) || defined(TRACE_HEADER_MULTI_READ) +#define _CXL_RAS_EVENTS_H + +#include + +#define CXL_RAS_UC_CACHE_DATA_PARITY BIT(0) +#define CXL_RAS_UC_CACHE_ADDR_PARITY BIT(1) +#define CXL_RAS_UC_CACHE_BE_PARITY BIT(2) +#define CXL_RAS_UC_CACHE_DATA_ECC BIT(3) +#define CXL_RAS_UC_MEM_DATA_PARITY BIT(4) +#define CXL_RAS_UC_MEM_ADDR_PARITY BIT(5) +#define CXL_RAS_UC_MEM_BE_PARITY BIT(6) +#define CXL_RAS_UC_MEM_DATA_ECC BIT(7) +#define CXL_RAS_UC_REINIT_THRESH BIT(8) +#define CXL_RAS_UC_RSVD_ENCODE BIT(9) +#define CXL_RAS_UC_POISON BIT(10) +#define CXL_RAS_UC_RECV_OVERFLOW BIT(11) +#define CXL_RAS_UC_INTERNAL_ERR BIT(14) +#define CXL_RAS_UC_IDE_TX_ERR BIT(15) +#define CXL_RAS_UC_IDE_RX_ERR BIT(16) + +#define show_uc_errs(status) __print_flags(status, " | ", \ + { CXL_RAS_UC_CACHE_DATA_PARITY, "Cache Data Parity Error" }, \ + { CXL_RAS_UC_CACHE_ADDR_PARITY, "Cache Address Parity Error" }, \ + { CXL_RAS_UC_CACHE_BE_PARITY, "Cache Byte Enable Parity Error" }, \ + { CXL_RAS_UC_CACHE_DATA_ECC, "Cache Data ECC Error" }, \ + { CXL_RAS_UC_MEM_DATA_PARITY, "Memory Data Parity Error" }, \ + { CXL_RAS_UC_MEM_ADDR_PARITY, "Memory Address Parity Error" }, \ + { CXL_RAS_UC_MEM_BE_PARITY, "Memory Byte Enable Parity Error" }, \ + { CXL_RAS_UC_MEM_DATA_ECC, "Memory Data ECC Error" }, \ + { CXL_RAS_UC_REINIT_THRESH, "REINIT Threshold Hit" }, \ + { CXL_RAS_UC_RSVD_ENCODE, "Received Unrecognized Encoding" }, \ + { CXL_RAS_UC_POISON, "Received Poison From Peer" }, \ + { CXL_RAS_UC_RECV_OVERFLOW, "Receiver Overflow" }, \ + { CXL_RAS_UC_INTERNAL_ERR, "Component Specific Error" }, \ + { CXL_RAS_UC_IDE_TX_ERR, "IDE Tx Error" }, \ + { CXL_RAS_UC_IDE_RX_ERR, "IDE Rx Error" } \ +) + +#define CXL_RAS_UC_OVFL_D2H_REQ BIT(0) +#define CXL_RAS_UC_OVFL_D2H_RSP BIT(1) +#define CXL_RAS_UC_OVFL_D2H_DATA BIT(2) +#define CXL_RAS_UC_OVFL_S2M_NDR BIT(3) +#define CXL_RAS_UC_OVFL_S2M_DRS BIT(4) + +#define show_uc_ovfl(hl) __print_flags(hl, " | ", \ + { CXL_RAS_UC_OVFL_D2H_REQ, "Receiver Overflow D2H Req" }, \ + { CXL_RAS_UC_OVFL_D2H_RSP, "Receiver Overflow D2H Rsp" }, \ + { CXL_RAS_UC_OVFL_D2H_DATA, "Receiver Overflow D2H Data" }, \ + { CXL_RAS_UC_OVFL_S2M_NDR, "Receiver Overflow S2M NDR" }, \ + { CXL_RAS_UC_OVFL_S2M_DRS, "Receiver Overflow S2M DRS" } \ +) + +TRACE_EVENT(cxl_ras_uc, + TP_PROTO(const char *dev_name, u32 status, u32 fe, u8 *hl, int hl_len), + TP_ARGS(dev_name, status, fe, hl, hl_len), + TP_STRUCT__entry( + __string(dev_name, dev_name) + __field(u32, status) + __field(u32, first_error) + __dynamic_array(u8, header_log, hl_len) + __field(int, header_log_len) + ), + TP_fast_assign( + __assign_str(dev_name, dev_name); + __entry->status = status; + __entry->first_error = fe; + memcpy(__get_dynamic_array(header_log), hl, hl_len); + __entry->header_log_len = hl_len; + ), + TP_printk("%s: status: '%s' first_error: '%s' header log: %s", + __get_str(dev_name), show_uc_errs(__entry->status), + show_uc_errs(__entry->first_error), + __print_array(__get_dynamic_array(header_log), __entry->header_log_len, 1) + ) +); + +#define CXL_RAS_CE_CACHE_DATA_ECC BIT(0) +#define CXL_RAS_CE_MEM_DATA_ECC BIT(1) +#define CXL_RAS_CE_CRC_THRESH BIT(2) +#define CXL_RAS_CE_CACHE_POISON BIT(3) +#define CXL_RAS_CE_MEM_POISON BIT(4) +#define CXL_RAS_CE_PHYS_LAYER_ERR BIT(5) + +#define show_ce_errs(status) __print_flags(status, " | ", \ + { CXL_RAS_CE_CACHE_DATA_ECC, "Cache Data ECC Error" }, \ + { CXL_RAS_CE_MEM_DATA_ECC, "Memory Data Ecc Error" }, \ + { CXL_RAS_CE_CRC_THRESH, "CRC Threshold Hit" }, \ + { CXL_RAS_CE_CACHE_POISON, "Received Cache Poison From Peer" }, \ + { CXL_RAS_CE_MEM_POISON, "Received Memory Poison From Peer" }, \ + { CXL_RAS_CE_PHYS_LAYER_ERR, "Received Error From Physical Layer" } \ +) + +TRACE_EVENT(cxl_ras_ce, + TP_PROTO(const char *dev_name, u32 status), + TP_ARGS(dev_name, status), + TP_STRUCT__entry( + __string(dev_name, dev_name) + __field(u32, status) + ), + TP_fast_assign( + __assign_str(dev_name, dev_name); + __entry->status = status; + ), + TP_printk("%s: status: '%s'", + __get_str(dev_name), show_ce_errs(__entry->status) + ) +); + +#endif /* _CXL_RAS_EVENTS_H */ + +/* This part must be outside protection */ +#include From patchwork Fri Sep 16 23:11:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 12978889 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3634ECAAA1 for ; Fri, 16 Sep 2022 23:11:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229804AbiIPXLs (ORCPT ); Fri, 16 Sep 2022 19:11:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56666 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229781AbiIPXLr (ORCPT ); Fri, 16 Sep 2022 19:11:47 -0400 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C3CDF7C32C for ; Fri, 16 Sep 2022 16:11:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1663369906; x=1694905906; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ueGRt4thWrbvzTif0qdSUiBRjX3AujplroxNnr6zkhA=; b=OJdbZ3jRZZBUTkxSDUzEeApQQSiy0Nlx5HR8gFq/aMatt2cMbvuVuvfc 25368jhKYP2SHIDBUjhw1L37VsADD69pUr22gng3XOkBMjFcHM4XqFGry UW6I7OFGSChIj8qj4kY11VTq69MIYHA1iV/9+O0pUO4FVAZTDrZg274Eb n0O80B5IoxBOwbDZyewGDuqSDiThCgIKr3JP7KfglRnGXXapes84YKbCQ Ivm21J76eqP+xO9xBGda1Qo/Bj8v6PqyKhCjwvwl3nSwbWvUj2sH7GZlW iDq+Qt34Y6LkAH6U3l+dp6bXTAotESk3dlRftyV7dfbyOJYA/b9fOxIB0 w==; X-IronPort-AV: E=McAfee;i="6500,9779,10472"; a="278825776" X-IronPort-AV: E=Sophos;i="5.93,321,1654585200"; d="scan'208";a="278825776" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Sep 2022 16:11:46 -0700 X-IronPort-AV: E=Sophos;i="5.93,321,1654585200"; d="scan'208";a="793254566" Received: from djiang5-desk3.ch.intel.com ([143.182.136.137]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Sep 2022 16:11:45 -0700 Subject: [PATCH RFC v2 9/9] cxl/pci: Add (hopeful) error handling support From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: alison.schofield@intel.com, vishal.l.verma@intel.com, bwidawsk@kernel.org, dan.j.williams@intel.com, jonathan.cameron@huawei.com, shiju.jose@huawei.com, rrichter@amd.com Date: Fri, 16 Sep 2022 16:11:45 -0700 Message-ID: <166336990544.3803215.2332306189095144106.stgit@djiang5-desk3.ch.intel.com> In-Reply-To: <166336972295.3803215.1047199449525031921.stgit@djiang5-desk3.ch.intel.com> References: <166336972295.3803215.1047199449525031921.stgit@djiang5-desk3.ch.intel.com> User-Agent: StGit/1.4 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org From: Dan Williams Add nominal error handling that tears down CXL.mem in response to error notifications that imply a device reset. Given some CXL.mem may be operating as System RAM, there is a high likelihood that these error events are fatal. However, if the system survives the notification the expectation is that the driver behavior is equivalent to a hot-unplug and re-plug of an endpoint. Note that this does not change the mask values from the default. That awaits CXL _OSC support to determine whether platform firmware is in control of the mask registers. Signed-off-by: Dan Williams Signed-off-by: Dave Jiang --- drivers/cxl/core/memdev.c | 1 drivers/cxl/cxl.h | 2 + drivers/cxl/cxlmem.h | 2 + drivers/cxl/pci.c | 160 +++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 165 insertions(+) diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c index 20ce488a7754..a74a93310d26 100644 --- a/drivers/cxl/core/memdev.c +++ b/drivers/cxl/core/memdev.c @@ -344,6 +344,7 @@ struct cxl_memdev *devm_cxl_add_memdev(struct cxl_dev_state *cxlds) * needed as this is ordered with cdev_add() publishing the device. */ cxlmd->cxlds = cxlds; + cxlds->cxlmd = cxlmd; cdev = &cxlmd->cdev; rc = cdev_device_add(cdev, dev); diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index ce17ccd8b125..35434b110a3b 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -132,7 +132,9 @@ static inline int ways_to_cxl(unsigned int ways, u8 *iw) #define CXL_RAS_CORRECTABLE_MASK_OFFSET 0x10 #define CXL_RAS_CORRECTABLE_MASK_MASK GENMASK(6, 0) #define CXL_RAS_CAP_CONTROL_OFFSET 0x14 +#define CXL_RAS_CAP_CONTROL_FE_MASK GENMASK(5, 0) #define CXL_RAS_HEADER_LOG_OFFSET 0x18 +#define CXL_RAX_HEADER_LOG_SIZE 512 #define CXL_RAS_CAPABILITY_LENGTH 0x58 /* CXL 2.0 8.2.8.1 Device Capabilities Array Register */ diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h index 88e3a8e54b6a..b3117fd67f42 100644 --- a/drivers/cxl/cxlmem.h +++ b/drivers/cxl/cxlmem.h @@ -186,6 +186,7 @@ struct cxl_endpoint_dvsec_info { * Currently only memory devices are represented. * * @dev: The device associated with this CXL state + * @cxlmd: The device representing the CXL.mem capabilities of @dev * @regs: Parsed register blocks * @cxl_dvsec: Offset to the PCIe device DVSEC * @payload_size: Size of space for payload @@ -218,6 +219,7 @@ struct cxl_endpoint_dvsec_info { */ struct cxl_dev_state { struct device *dev; + struct cxl_memdev *cxlmd; struct cxl_regs regs; int cxl_dvsec; diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index 357de704e42c..d51e34c4c87e 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include "cxlmem.h" #include "cxlpci.h" @@ -399,6 +400,11 @@ static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds) } } +static void disable_aer(void *pdev) +{ + pci_disable_pcie_error_reporting(pdev); +} + static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) { struct cxl_register_map map; @@ -420,6 +426,7 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) cxlds = cxl_dev_state_create(&pdev->dev); if (IS_ERR(cxlds)) return PTR_ERR(cxlds); + pci_set_drvdata(pdev, cxlds); cxlds->serial = pci_get_dsn(pdev); cxlds->cxl_dvsec = pci_find_dvsec_capability( @@ -474,6 +481,14 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) if (IS_ERR(cxlmd)) return PTR_ERR(cxlmd); + if (cxlds->regs.ras) { + pci_enable_pcie_error_reporting(pdev); + rc = devm_add_action_or_reset(&pdev->dev, disable_aer, pdev); + if (rc) + return rc; + } + pci_save_state(pdev); + if (resource_size(&cxlds->pmem_res) && IS_ENABLED(CONFIG_CXL_PMEM)) rc = devm_cxl_add_nvdimm(&pdev->dev, cxlmd); @@ -487,10 +502,155 @@ static const struct pci_device_id cxl_mem_pci_tbl[] = { }; MODULE_DEVICE_TABLE(pci, cxl_mem_pci_tbl); +/* CXL spec rev3.0 8.2.4.16.1 */ +#define DATA_HEADER_SIZE 16 +#define FLIT_SIZE (64 + 2) +static int header_log_setup(struct cxl_dev_state *cxlds, u32 fe, u8 *log) +{ + void __iomem *addr; + + addr = cxlds->regs.ras + CXL_RAS_HEADER_LOG_OFFSET; + + if (fe & CXL_RAS_UC_CACHE_DATA_PARITY || fe & CXL_RAS_UC_CACHE_ADDR_PARITY || + fe & CXL_RAS_UC_CACHE_BE_PARITY || fe & CXL_RAS_UC_CACHE_DATA_ECC || + fe & CXL_RAS_UC_MEM_DATA_PARITY || fe & CXL_RAS_UC_MEM_ADDR_PARITY || + fe & CXL_RAS_UC_MEM_BE_PARITY || fe & CXL_RAS_UC_MEM_DATA_ECC) { + memcpy_fromio(log, addr, DATA_HEADER_SIZE); + return DATA_HEADER_SIZE; + } + + if (fe & CXL_RAS_UC_RSVD_ENCODE) { + memcpy_fromio(log, addr, FLIT_SIZE); + return FLIT_SIZE; + } + + if (fe & CXL_RAS_UC_RECV_OVERFLOW) { + *log = readb(addr); + return sizeof(u8); + } + + return 0; +} + +/* + * Log the state of the RAS status registers and prepare them to log the + * next error status. Return 1 if reset needed. + */ +static bool cxl_report_and_clear(struct cxl_dev_state *cxlds) +{ + struct cxl_memdev *cxlmd = cxlds->cxlmd; + struct device *dev = &cxlmd->dev; + void __iomem *addr; + u32 status; + bool ue = false; + + if (!cxlds->regs.ras) + return false; + + addr = cxlds->regs.ras + CXL_RAS_UNCORRECTABLE_STATUS_OFFSET; + status = le32_to_cpu(readl(addr)); + if (status & CXL_RAS_UNCORRECTABLE_STATUS_MASK) { + u8 hl[CXL_RAX_HEADER_LOG_SIZE]; + u32 fe; + int size; + + writel(status & CXL_RAS_UNCORRECTABLE_STATUS_MASK, addr); + ue = true; + + /* If multiple errors, log header points to first error from ctrl reg */ + if (hweight32(status) > 1) { + addr = cxlds->regs.ras + CXL_RAS_CAP_CONTROL_OFFSET; + fe = BIT(le32_to_cpu(readl(addr)) & CXL_RAS_CAP_CONTROL_FE_MASK); + } else { + fe = status; + } + + size = header_log_setup(cxlds, fe, hl); + trace_cxl_ras_uc(dev_name(dev), status, fe, hl, size); + } + + addr = cxlds->regs.ras + CXL_RAS_CORRECTABLE_STATUS_OFFSET; + status = le32_to_cpu(readl(addr)); + if (status & CXL_RAS_CORRECTABLE_STATUS_MASK) { + writel(status & CXL_RAS_CORRECTABLE_STATUS_MASK, addr); + trace_cxl_ras_ce(dev_name(dev), status); + } + + return ue; +} + +static pci_ers_result_t cxl_error_detected(struct pci_dev *pdev, + pci_channel_state_t state) +{ + struct cxl_dev_state *cxlds = pci_get_drvdata(pdev); + struct cxl_memdev *cxlmd = cxlds->cxlmd; + struct device *dev = &cxlmd->dev; + bool ue; + + /* + * A frozen channel indicates an impending reset which is fatal to + * CXL.mem operation, and will likely crash the system. On the off + * chance the situation is recoverable dump the status of the RAS + * capability registers and bounce the active state of the memdev. + */ + ue = cxl_report_and_clear(cxlds); + + switch (state) { + case pci_channel_io_normal: + if (ue) { + device_release_driver(dev); + return PCI_ERS_RESULT_NEED_RESET; + } + return PCI_ERS_RESULT_CAN_RECOVER; + case pci_channel_io_frozen: + dev_warn(&pdev->dev, + "%s: frozen state error detected, disable CXL.mem\n", + dev_name(dev)); + device_release_driver(dev); + return PCI_ERS_RESULT_NEED_RESET; + case pci_channel_io_perm_failure: + dev_warn(&pdev->dev, + "failure state error detected, request disconnect\n"); + return PCI_ERS_RESULT_DISCONNECT; + } + return PCI_ERS_RESULT_NEED_RESET; +} + +static pci_ers_result_t cxl_slot_reset(struct pci_dev *pdev) +{ + struct cxl_dev_state *cxlds = pci_get_drvdata(pdev); + struct cxl_memdev *cxlmd = cxlds->cxlmd; + struct device *dev = &cxlmd->dev; + + dev_info(&pdev->dev, "%s: restart CXL.mem after slot reset\n", + dev_name(dev)); + pci_restore_state(pdev); + if (device_attach(dev) <= 0) + return PCI_ERS_RESULT_DISCONNECT; + return PCI_ERS_RESULT_RECOVERED; +} + +static void cxl_error_resume(struct pci_dev *pdev) +{ + struct cxl_dev_state *cxlds = pci_get_drvdata(pdev); + struct cxl_memdev *cxlmd = cxlds->cxlmd; + struct device *dev = &cxlmd->dev; + + dev_info(&pdev->dev, "%s: error resume %s\n", dev_name(dev), + dev->driver ? "successful" : "failed"); +} + +static const struct pci_error_handlers cxl_error_handlers = { + .error_detected = cxl_error_detected, + .slot_reset = cxl_slot_reset, + .resume = cxl_error_resume, +}; + static struct pci_driver cxl_pci_driver = { .name = KBUILD_MODNAME, .id_table = cxl_mem_pci_tbl, .probe = cxl_pci_probe, + .err_handler = &cxl_error_handlers, .driver = { .probe_type = PROBE_PREFER_ASYNCHRONOUS, },