From patchwork Thu Dec 2 04:37:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ben Widawsky X-Patchwork-Id: 12651709 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BA233C433F5 for ; Thu, 2 Dec 2021 04:40:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1355527AbhLBEnc (ORCPT ); Wed, 1 Dec 2021 23:43:32 -0500 Received: from mga09.intel.com ([134.134.136.24]:61242 "EHLO mga09.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1355360AbhLBEnZ (ORCPT ); Wed, 1 Dec 2021 23:43:25 -0500 X-IronPort-AV: E=McAfee;i="6200,9189,10185"; a="236438362" X-IronPort-AV: E=Sophos;i="5.87,281,1631602800"; d="scan'208";a="236438362" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Dec 2021 20:40:03 -0800 X-IronPort-AV: E=Sophos;i="5.87,281,1631602800"; d="scan'208";a="745717370" Received: from liudanie-mobl1.amr.corp.intel.com (HELO bad-guy.kumite) ([10.252.143.85]) by fmsmga006-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Dec 2021 20:40:02 -0800 From: Ben Widawsky To: linux-cxl@vger.kernel.org Cc: Ben Widawsky , Alison Schofield , Dan Williams , Ira Weiny , Jonathan Cameron , Vishal Verma Subject: [PATCH v2 00/14] Add drivers for CXL ports and mem devices Date: Wed, 1 Dec 2021 20:37:36 -0800 Message-Id: <20211202043750.3501494-1-ben.widawsky@intel.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org Significant changes since v1 [1]: - Reworked decoder alloc APIs to be specific to component type - cxl_mem handles DVSEC checks instead of cxl_port - buggy switch handling in cxl_mem is #if 0'd out - cxl_pci will fail if the Device DVSEC can't be found There were a few improvements from the RFC, v1, and discussions on IRC (goto #cxl on OFTC). Dan has some patches that apply either on top of, or squashed in with these. Those are: 1. Global root port list doesn't need to exist. 2. Port enumeration doesn't rely on PCIe concepts to walk topology 3. Port driver should handle enumerating dports Here I've also dropped the prep patches which are mostly reviewed. Those are here [2]. What follows is the original cover letter from v1 [1] [1]: https://lore.kernel.org/linux-cxl/20211120000250.1663391-1-ben.widawsky@intel.com/ [2]: https://lore.kernel.org/linux-cxl/20211129214721.1668325-1-ben.widawsky@intel.com/ --- This is the first set of patches from the RFC [1] for region creation. The patches enable port enumeration for endpoint devices, and enumeration of decoder resources for ports. In the RFC [1], I felt it necessary to post the consumer of this work, the region driver, so that it was clear why these patches were necessary. Because the region patches patches are less baked, and received no review in the RFC, they are excluded here. If you find yourself unclear about why these patches are interesting, go look at the RFC [1]. Each patch contains the list of changes from RFCv2. IMHO the following are the high level most important changes: 1. Rework cxl_pci to fix mailbox handling and allow for wait media ready. 2. DVSEC range information is passed from cxl_pci and checked linux-pci is on the Cc since CXL lives in a parallel universe to PCI and some PCI mechanisms are reused here. Feedback from experts in that domain is very welcome. What was requested and not changed: 1. Dropping global list of root ports. 2. Improving find_parent_cxl_port() --- Summary ======= Two new drivers are introduced to support Compute Express Link 2.0 [2] HDM decoder enumeration. While the existing cxl_acpi and cxl_pci drivers already create some of the necessary devices, they did not do full enumeration of decoders, and they did not do port enumeration for switches. Additionally, CXL 2.0 Root Port component registers are now handled as well. cxl_port ======== The cxl_port driver is implemented within the cxl_port module. While loading of this module is optional, the other new drivers depend, and cxl_acpi depend on it for complete enumeration. The port driver is responsible for all activities around HDM decoder enumeration and programming. Introduced earlier, the concept of a port is an abstraction over CXL components with an upstream port, every host bridge, switch, and endpoint. cxl_mem ======= The cxl_mem driver's main job is to walk up the hierarchy to make the determination if it is CXL.mem routed, meaning, all components above it in the hierarchy are participating in the CXL.mem protocol. It is implemented within the cxl_mem module. As the host bridge ports are added by a platform specific driver, such as cxl_acpi, the scope of the mem driver can be reduced to scan for switches and ask cxl_core to work on enumerating them. With this done, the determination as to whether a device is CXL.mem routed can be done simply by checking if the struct device has a driver bound to it. Results ======= Running these patches should yield new devices and new drivers under /sys/bus/cxl/devices and /sys/bus/cxl/drivers. For example, in a standard QEMU run, using run_qemu [3] /sys/bus/cxl/devices (new): # The host bridge CHBS decoder lrwxrwxrwx 1 root root 0 Nov 19 15:23 decoder1.0 -> ../../../devices/platform/ACPI0017:00/root0/port1/decoder1.0 # mem0's decoder lrwxrwxrwx 1 root root 0 Nov 19 15:23 decoder2.0 -> ../../../devices/platform/ACPI0017:00/root0/port1/port2/decoder2.0 # mem1's decoder lrwxrwxrwx 1 root root 0 Nov 19 15:23 decoder3.0 -> ../../../devices/platform/ACPI0017:00/root0/port1/port3/decoder3.0 # mem0's port lrwxrwxrwx 1 root root 0 Nov 19 15:23 port2 -> ../../../devices/platform/ACPI0017:00/root0/port1/port2 # mem1's port lrwxrwxrwx 1 root root 0 Nov 19 15:23 port3 -> ../../../devices/platform/ACPI0017:00/root0/port1/port3 /sys/bus/cxl/drivers: drwxr-xr-x 2 root root 0 Nov 19 15:23 cxl_mem [1]: https://lore.kernel.org/linux-cxl/20211022183709.1199701-1-ben.widawsky@intel.com/T/#t [2]: https://www.computeexpresslink.org/download-the-specification [3]: https://github.com/pmem/run_qemu/ drwxr-xr-x 2 root root 0 Nov 19 15:23 cxl_port Ben Widawsky (14): cxl/core: Add, document, and tighten up decoder APIs cxl: Introduce endpoint decoders cxl/core: Move target population locking to caller cxl: Introduce topology host registration cxl/core: Store global list of root ports cxl/pci: Cache device DVSEC offset cxl: Cache and pass DVSEC ranges cxl/pci: Implement wait for media active cxl/pci: Store component register base in cxlds cxl: Make passthrough decoder init implicit cxl/port: Introduce a port driver cxl: Unify port enumeration for decoders cxl/port: Cleanup adding passthrough decoders cxl/mem: Introduce cxl_mem driver Documentation/ABI/testing/sysfs-bus-cxl | 9 + .../driver-api/cxl/memory-devices.rst | 14 + drivers/cxl/Kconfig | 21 + drivers/cxl/Makefile | 4 + drivers/cxl/acpi.c | 82 +-- drivers/cxl/core/Makefile | 1 + drivers/cxl/core/bus.c | 479 ++++++++++++++++-- drivers/cxl/core/core.h | 3 + drivers/cxl/core/memdev.c | 2 +- drivers/cxl/core/pci.c | 117 +++++ drivers/cxl/core/regs.c | 6 +- drivers/cxl/cxl.h | 59 ++- drivers/cxl/cxlmem.h | 23 + drivers/cxl/mem.c | 285 +++++++++++ drivers/cxl/pci.c | 200 ++++++++ drivers/cxl/pci.h | 16 + drivers/cxl/port.c | 368 ++++++++++++++ tools/testing/cxl/Kbuild | 1 + tools/testing/cxl/mock_acpi.c | 4 +- 19 files changed, 1589 insertions(+), 105 deletions(-) create mode 100644 drivers/cxl/core/pci.c create mode 100644 drivers/cxl/mem.c create mode 100644 drivers/cxl/port.c base-commit: c19a248a84b950ef53c6768751cc917ec2c6b6cf