From patchwork Mon Jan 24 17:16:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Cameron X-Patchwork-Id: 12722531 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3724FC433F5 for ; Mon, 24 Jan 2022 17:31:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244442AbiAXRb4 (ORCPT ); Mon, 24 Jan 2022 12:31:56 -0500 Received: from frasgout.his.huawei.com ([185.176.79.56]:4474 "EHLO frasgout.his.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244616AbiAXRb4 (ORCPT ); Mon, 24 Jan 2022 12:31:56 -0500 Received: from fraeml705-chm.china.huawei.com (unknown [172.18.147.201]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4JjH8j48cYz67hvg; Tue, 25 Jan 2022 01:31:33 +0800 (CST) Received: from lhreml710-chm.china.huawei.com (10.201.108.61) by fraeml705-chm.china.huawei.com (10.206.15.54) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2308.21; Mon, 24 Jan 2022 18:31:54 +0100 Received: from SecurePC-101-06.china.huawei.com (10.122.247.231) by lhreml710-chm.china.huawei.com (10.201.108.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.21; Mon, 24 Jan 2022 17:31:53 +0000 From: Jonathan Cameron To: , Marcel Apfelbaum , "Michael S . Tsirkin" , Igor Mammedov CC: , Ben Widawsky , =?utf-8?q?Alex_Benn=C3=A9e?= , Peter Maydell , , Shameerali Kolothum Thodi , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Saransh Gupta1 , Shreyas Shah , Chris Browy , Samarth Saxena , "Dan Williams" Subject: [PATCH v4 29/42] hw/cxl/host: Add support for CXL Fixed Memory Windows. Date: Mon, 24 Jan 2022 17:16:52 +0000 Message-ID: <20220124171705.10432-30-Jonathan.Cameron@huawei.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220124171705.10432-1-Jonathan.Cameron@huawei.com> References: <20220124171705.10432-1-Jonathan.Cameron@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.122.247.231] X-ClientProxiedBy: lhreml740-chm.china.huawei.com (10.201.108.190) To lhreml710-chm.china.huawei.com (10.201.108.61) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org From: Jonathan Cameron The concept of these is introduced in [1] in terms of the description the CEDT ACPI table. The principal is more general. Unlike once traffic hits the CXL root bridges, the host system memory address routing is implementation defined and effectively static once observable by standard / generic system software. Each CXL Fixed Memory Windows (CFMW) is a region of PA space which has fixed system dependent routing configured so that accesses can be routed to the CXL devices below a set of target root bridges. The accesses may be interleaved across multiple root bridges. For QEMU we could have fully specified these regions in terms of a base PA + size, but as the absolute address does not matter it is simpler to let individual platforms place the memory regions. ExampleS: -cxl-fixed-memory-window targets=cxl.0,size=128G -cxl-fixed-memory-window targets=cxl.1,size=128G -cxl-fixed-memory-window targets=cxl0,targets=cxl.1,size=256G,interleave-granularity=2k Specifies * 2x 128G regions not interleaved across root bridges, one for each of the root bridges with ids cxl.0 and cxl.1 * 256G region interleaved across root bridges with ids cxl.0 and cxl.1 with a 2k interleave granularity. When system software enumerates the devices below a given root bridge it can then decide which CFMW to use. If non interleave is desired (or possible) it can use the appropriate CFMW for the root bridge in question. If there are suitable devices to interleave across the two root bridges then it may use the 3rd CFMS. A number of other designs were considered but the following constraints made it hard to adapt existing QEMU approaches to this particular problem. 1) The size must be known before a specific architecture / board brings up it's PA memory map. We need to set up an appropriate region. 2) Using links to the host bridges provides a clean command line interface but these links cannot be established until command line devices have been added. Hence the two step process used here of first establishing the size, interleave-ways and granularity + caching the ids of the host bridges and then, once available finding the actual host bridges so they can be used later to support interleave decoding. [1] CXL 2.0 ECN: CEDT CFMWS & QTG DSM (computeexpresslink.org / specifications) Signed-off-by: Jonathan Cameron --- hw/cxl/cxl-host-stubs.c | 22 +++++++ hw/cxl/cxl-host.c | 138 ++++++++++++++++++++++++++++++++++++++++ hw/cxl/meson.build | 12 ++-- include/hw/cxl/cxl.h | 20 ++++++ qapi/machine.json | 15 +++++ qemu-options.hx | 37 +++++++++++ softmmu/vl.c | 11 ++++ 7 files changed, 251 insertions(+), 4 deletions(-) diff --git a/hw/cxl/cxl-host-stubs.c b/hw/cxl/cxl-host-stubs.c new file mode 100644 index 0000000000..f942dda41b --- /dev/null +++ b/hw/cxl/cxl-host-stubs.c @@ -0,0 +1,22 @@ +/* + * CXL host parameter parsing routine stubs + * + * Copyright (c) 2022 Huawei + */ +#include "qemu/osdep.h" +#include "qapi/error.h" +#include "qemu/option.h" +#include "hw/cxl/cxl.h" + +QemuOptsList qemu_cxl_fixed_window_opts = { + .name = "cxl-fixed-memory-window", + .implied_opt_name = "type", + .head = QTAILQ_HEAD_INITIALIZER(qemu_cxl_fixed_window_opts.head), + .desc = { { 0 } } +}; + +void parse_cxl_fixed_memory_window_opts(MachineState *ms) {}; + +void cxl_fixed_memory_window_link_targets(Error **errp) {}; + +const MemoryRegionOps cfmws_ops; diff --git a/hw/cxl/cxl-host.c b/hw/cxl/cxl-host.c new file mode 100644 index 0000000000..9f303e6d8e --- /dev/null +++ b/hw/cxl/cxl-host.c @@ -0,0 +1,138 @@ +/* + * CXL host parameter parsing routines + * + * Copyright (c) 2022 Huawei + * Modeled loosely on the NUMA options handling in hw/core/numa.c + */ + +#include "qemu/osdep.h" +#include "qemu/units.h" +#include "qemu/bitmap.h" +#include "qemu/error-report.h" +#include "qapi/error.h" +#include "sysemu/qtest.h" +#include "hw/boards.h" + +#include "qapi/opts-visitor.h" +#include "qapi/qapi-visit-machine.h" +#include "qemu/option.h" +#include "hw/cxl/cxl.h" +#include "hw/pci/pci_bus.h" +#include "hw/pci/pci_bridge.h" +#include "hw/pci/pci_host.h" +#include "hw/pci/pcie_port.h" + +QemuOptsList qemu_cxl_fixed_window_opts = { + .name = "cxl-fixed-memory-window", + .implied_opt_name = "type", + .head = QTAILQ_HEAD_INITIALIZER(qemu_cxl_fixed_window_opts.head), + .desc = { { 0 } } +}; + +static void set_cxl_fixed_memory_window_options(MachineState *ms, + CXLFixedMemoryWindowOptions *object, + Error **errp) +{ + CXLFixedWindow *fw = g_malloc0(sizeof(*fw)); + strList *target; + int i; + + for (target = object->targets; target; target = target->next) { + fw->num_targets++; + } + + fw->enc_int_ways = cxl_interleave_ways_enc(fw->num_targets, errp); + if (*errp) { + return; + } + + fw->targets = g_malloc0_n(fw->num_targets, sizeof(*fw->targets)); + for (i = 0, target = object->targets; target; i++, target = target->next) { + /* This link cannot be resolved yet, so stash the name for now */ + fw->targets[i] = g_strdup(target->value); + } + + if (object->size % (256 * MiB)) { + error_setg(errp, + "Size of a CXL fixed memory window must my a multiple of 256MiB"); + return; + } + fw->size = object->size; + + if (object->has_interleave_granularity) { + fw->enc_int_gran = + cxl_interleave_granularity_enc(object->interleave_granularity, + errp); + if (*errp) { + return; + } + } else { + /* Default to 256 byte interleave */ + fw->enc_int_gran = 0; + } + + ms->cxl_devices_state->fixed_windows = + g_list_append(ms->cxl_devices_state->fixed_windows, fw); + + return; +} + +static int parse_cxl_fixed_memory_window(void *opaque, QemuOpts *opts, + Error **errp) +{ + CXLFixedMemoryWindowOptions *object = NULL; + MachineState *ms = MACHINE(opaque); + Error *err = NULL; + Visitor *v = opts_visitor_new(opts); + + visit_type_CXLFixedMemoryWindowOptions(v, NULL, &object, errp); + visit_free(v); + if (!object) { + return -1; + } + + set_cxl_fixed_memory_window_options(ms, object, &err); + + qapi_free_CXLFixedMemoryWindowOptions(object); + if (err) { + error_propagate(errp, err); + return -1; + } + + return 0; +} + +void parse_cxl_fixed_memory_window_opts(MachineState *ms) +{ + qemu_opts_foreach(qemu_find_opts("cxl-fixed-memory-window"), + parse_cxl_fixed_memory_window, ms, &error_fatal); +} + +void cxl_fixed_memory_window_link_targets(Error **errp) +{ + MachineState *ms = MACHINE(qdev_get_machine()); + + if (ms->cxl_devices_state && ms->cxl_devices_state->fixed_windows) { + GList *it; + + for (it = ms->cxl_devices_state->fixed_windows; it; it = it->next) { + CXLFixedWindow *fw = it->data; + int i; + + for (i = 0; i < fw->num_targets; i++) { + Object *o; + bool ambig; + + o = object_resolve_path_type(fw->targets[i], + TYPE_PXB_CXL_DEVICE, + &ambig); + if (!o) { + error_setg(errp, "Could not resolve CXLFM target %s", + fw->targets[i]); + return; + } + fw->target_hbs[i] = PXB_CXL_DEV(o); + } + } + } +} diff --git a/hw/cxl/meson.build b/hw/cxl/meson.build index 0eca715d10..27dff3868b 100644 --- a/hw/cxl/meson.build +++ b/hw/cxl/meson.build @@ -1,5 +1,9 @@ -softmmu_ss.add(when: 'CONFIG_CXL', if_true: files( - 'cxl-component-utils.c', - 'cxl-device-utils.c', - 'cxl-mailbox-utils.c', +specific_ss.add(when: 'CONFIG_CXL', if_true: files( + 'cxl-component-utils.c', + 'cxl-device-utils.c', + 'cxl-mailbox-utils.c', + 'cxl-host.c', +)) +specific_ss.add(when: 'CONFIG_CXL', if_false: files( + 'cxl-host-stubs.c', )) diff --git a/include/hw/cxl/cxl.h b/include/hw/cxl/cxl.h index 6889362230..1b72c0b7b7 100644 --- a/include/hw/cxl/cxl.h +++ b/include/hw/cxl/cxl.h @@ -10,6 +10,9 @@ #ifndef CXL_H #define CXL_H +#include "qapi/qapi-types-machine.h" +#include "qemu/option.h" +#include "hw/pci/pci_bridge.h" #include "cxl_pci.h" #include "cxl_component.h" #include "cxl_device.h" @@ -20,10 +23,27 @@ #define TYPE_CXL_TYPE3_DEV "cxl-type3" #define CXL_WINDOW_MAX 10 +typedef struct CXLFixedWindow { + uint64_t size; + char **targets; + struct PXBDev *target_hbs[8]; + uint8_t num_targets; + uint8_t enc_int_ways; + uint8_t enc_int_gran; + /* Todo: XOR based interleaving */ + MemoryRegion mr; + hwaddr base; +} CXLFixedWindow; + typedef struct CXLState { bool is_enabled; MemoryRegion host_mr; unsigned int next_mr_idx; + GList *fixed_windows; } CXLState; +extern QemuOptsList qemu_cxl_fixed_window_opts; +void parse_cxl_fixed_memory_window_opts(MachineState *ms); +void cxl_fixed_memory_window_link_targets(Error **errp); + #endif diff --git a/qapi/machine.json b/qapi/machine.json index c87c81b803..87247d1d60 100644 --- a/qapi/machine.json +++ b/qapi/machine.json @@ -504,6 +504,21 @@ 'dst': 'uint16', 'val': 'uint8' }} +## +# @CXLFixedMemoryWindowOptions: +# +# Create a CXL Fixed Memory Window (for OptsVisitor) +# +# @targets: Target root bridge IDs +# +# Since X.X //fixme +## +{ 'struct': 'CXLFixedMemoryWindowOptions', + 'data': { + 'size': 'size', + '*interleave-granularity': 'size', + 'targets': ['str'] }} + ## # @X86CPURegister32: # diff --git a/qemu-options.hx b/qemu-options.hx index ba3ae6a42a..b4d2cc6f48 100644 --- a/qemu-options.hx +++ b/qemu-options.hx @@ -467,6 +467,43 @@ SRST -numa hmat-cache,node-id=1,size=10K,level=1,associativity=direct,policy=write-back,line=8 ERST +DEF("cxl-fixed-memory-window", HAS_ARG, QEMU_OPTION_cxl_fixed_memory_window, + "-cxl-fixed-memory-window targets=firsttarget,targets=secondtarget,size=size[,interleave-granularity=granularity]\n", + QEMU_ARCH_ALL) +SRST +``-cxl-fixed-memory-window targets=firsttarget,targets=secondtarget,size=size[,interleave-granularity=granularity]`` + Define a CXL Fixed Memory Window (CFMW). + + Described in the CXL 2.0 ECN: CEDT CFMWS & QTG _DSM. + + They are regions of Host Physical Addresses (HPA) on a system which + may be interleaved across one or more CXL host bridges. The system + software will assign particular devices into these windows and + configure the downstream Host-managed Device Memory (HDM) decoders + in root ports, switch ports and devices appropriately to meet the + interleave requirements before enabling the memory devices. + + ``targets=firsttarget`` provides the mapping to CXL host bridges + which may be identified by the id provied in the -device entry. + Multiple entries are needed to specify all the targets when + the fixed memory window represents interleaved memory. + + ``size=size`` sets the size of the CFMW. This must be a multiple of + 256MiB. The region will be aligned to 256MiB but the location is + platform and configuration dependent. + + ``interleave-granularity=granularity`` sets the granularity of + interleave. Default 256KiB. Only 256KiB, 512KiB, 1024KiB, 2048KiB + 4096KiB, 8192KiB and 16384KiB granularities supported. + + Example: + + :: + + -cxl-fixed-memory-window -targets=cxl.0,-targets=cxl.1,size=128G,interleave-granularity=512k + +ERST + DEF("add-fd", HAS_ARG, QEMU_OPTION_add_fd, "-add-fd fd=fd,set=set[,opaque=opaque]\n" " Add 'fd' to fd 'set'\n", QEMU_ARCH_ALL) diff --git a/softmmu/vl.c b/softmmu/vl.c index 5e1b35ba48..f83f158fff 100644 --- a/softmmu/vl.c +++ b/softmmu/vl.c @@ -91,6 +91,7 @@ #include "qemu/config-file.h" #include "qemu/qemu-options.h" #include "qemu/main-loop.h" +#include "hw/cxl/cxl.h" #ifdef CONFIG_VIRTFS #include "fsdev/qemu-fsdev.h" #endif @@ -2744,6 +2745,7 @@ void qmp_x_exit_preconfig(Error **errp) qemu_init_board(); qemu_create_cli_devices(); + cxl_fixed_memory_window_link_targets(errp); qemu_machine_creation_done(); if (loadvm) { @@ -2805,6 +2807,7 @@ void qemu_init(int argc, char **argv, char **envp) qemu_add_opts(&qemu_msg_opts); qemu_add_opts(&qemu_name_opts); qemu_add_opts(&qemu_numa_opts); + qemu_add_opts(&qemu_cxl_fixed_window_opts); qemu_add_opts(&qemu_icount_opts); qemu_add_opts(&qemu_semihosting_config_opts); qemu_add_opts(&qemu_fw_cfg_opts); @@ -2927,6 +2930,13 @@ void qemu_init(int argc, char **argv, char **envp) exit(1); } break; + case QEMU_OPTION_cxl_fixed_memory_window: + opts = qemu_opts_parse_noisily(qemu_find_opts("cxl-fixed-memory-window"), + optarg, true); + if (!opts) { + exit(1); + } + break; case QEMU_OPTION_display: parse_display(optarg); break; @@ -3764,6 +3774,7 @@ void qemu_init(int argc, char **argv, char **envp) qemu_resolve_machine_memdev(); parse_numa_opts(current_machine); + parse_cxl_fixed_memory_window_opts(current_machine); if (vmstate_dump_file) { /* dump and exit */