From patchwork Wed Feb 21 18:15:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: fan X-Patchwork-Id: 13566067 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 05DD285268 for ; Wed, 21 Feb 2024 18:21:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708539674; cv=none; b=JshtV3wYqkW+t68Xeh+u+BBwKUMxDOwYzuXy+UL0SdzpZV97ItA0AhepI+TVtJAi0NZ5prOpYcHqXQoj6thmHk14xHnhI/fm3tysSDNaUjucna6eH4ASnIhYk8TXk/Qa2iv2rwflau7LeeeGmbS1jlbZMUKumoGB3uvOj2xDbug= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708539674; c=relaxed/simple; bh=gkUBdmYA71Hv10Hl86RlZ3WRrrRHuAjCA/UFYnu1kv8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TO1/DRNZeRIo9r3nJgaS+UKVfQHTK83mO6r2vF/x8/7IW4IV6hyIxEKSViuwlJ57eSlsKsPa3SMLGZUDtkJUIFREVT46KzlL234cF6I4g5WmnyWLF1oHVMtuw2ki7sa0dqba4ueXyMqvaysjR4QMMVMgdDosl9q2udrMZkIgbcg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Y4ytpgIE; arc=none smtp.client-ip=209.85.216.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Y4ytpgIE" Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-29996cc6382so2688455a91.3 for ; Wed, 21 Feb 2024 10:21:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708539672; x=1709144472; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AkPmI8FuszQzNKc/W7C7bH++18bz1caF1vBxyxTADMI=; b=Y4ytpgIEUI+pHinWO2xyAzdKFtNkVOdq5kDp6yeAqK5FbibSY+OmM3Vkpm74PiZfqf NrwCs7kZov+EI8IavimBIYmRavuH75DIEXv9U9BCBC/0FTmD6R979p/7xpn9e3ILLRFT ZwFj8T8yldyGBPtsKvsjC6TAb6CbyZhDPrqUFIrxMlsAb2NsFvvASp3grwW+NTjb9TI3 99ikZlJpS3fV/dYyp6NJD+IuiR1lVJo2hHoyS6sWfOWYF3YHz8fzZmEWHp2Y8A7xGJ3c RM3lrr+l8woC07ZZG5F9vVlEtSmueVV2ssa8277jJTP4Qmx2nay2KweTDRQbDCet2gQR /AUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708539672; x=1709144472; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AkPmI8FuszQzNKc/W7C7bH++18bz1caF1vBxyxTADMI=; b=jZwQfKEk9mx9qYeiXjWwjUDwGbiDNfUyw6Fyfc/+ZJuZlmJTAP7T7Po1Jy+VehIpEk YkX2A7mUIoZSemkTe9eCATeE8ZHMqoDP2RDSsZ0IfMzbyxJtaWwUE+adVusIrZ4AilIo II/mqE4zItKSvMH1HB2CkyZiyxIDS5EWlBUId4HTM6Rrv9nn94hnXqm6sdUxoCf7lqlr yT6PiAoA3Hi6zr8K9fFaFwkCPp/TWol4RwpMYU3Mb6akp68eNBR2DdlIU4BEhMkqp6r1 FFbS/KpCe9tH7Bz9xS4FBhhs+MpKWw/BNp/ThI0tfjisZZK9S9BQYa8MXHlMRtm84Te1 pwbw== X-Forwarded-Encrypted: i=1; AJvYcCXo3xffJnNj/fVaeSOvbtURMLNYXrNu2MjYldbKXSVRt4EIeoxXHVS1kLC7UJ4Vgk2SfgvWyD/geQb5248B5WZs/qVBfO/LcbIy X-Gm-Message-State: AOJu0Yw7E1lGXwuIUwUn5gR9y2V6bS0NJ4Uk08TRd83N7YO/0EG5clA5 pTQNbOwvMRllXxG172QC3U0WEwj36VelyGvmxy+OoRd8siPN1vqO X-Google-Smtp-Source: AGHT+IFMcnQh5f80Wkda3K0vH+ueICO5TJY8WkNFGkF0tVy5RzTaPhNg7QC/g2xsj8usX4PLltMs+Q== X-Received: by 2002:a17:90b:1e47:b0:299:2241:820b with SMTP id pi7-20020a17090b1e4700b002992241820bmr14945559pjb.37.1708539672173; Wed, 21 Feb 2024 10:21:12 -0800 (PST) Received: from localhost.localdomain ([2601:641:300:14de:5692:7e41:13a2:69a]) by smtp.gmail.com with ESMTPSA id cs16-20020a17090af51000b002992f49922csm9979009pjb.25.2024.02.21.10.21.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 10:21:11 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Fan Ni Subject: [PATCH v4 01/10] hw/cxl/cxl-mailbox-utils: Add dc_event_log_size field to output payload of identify memory device command Date: Wed, 21 Feb 2024 10:15:54 -0800 Message-ID: <20240221182020.1086096-2-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240221182020.1086096-1-nifan.cxl@gmail.com> References: <20240221182020.1086096-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Based on CXL spec r3.1 Table 8-127 (Identify Memory Device Output Payload), dynamic capacity event log size should be part of output of the Identify command. Add dc_event_log_size to the output payload for the host to get the info. Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 4bcd727f4c..ba1d9901df 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -21,6 +21,7 @@ #include "sysemu/hostmem.h" #define CXL_CAPACITY_MULTIPLIER (256 * MiB) +#define CXL_DC_EVENT_LOG_SIZE 8 /* * How to add a new command, example. The command set FOO, with cmd BAR. @@ -780,8 +781,9 @@ static CXLRetCode cmd_identify_memory_device(const struct cxl_cmd *cmd, uint16_t inject_poison_limit; uint8_t poison_caps; uint8_t qos_telemetry_caps; + uint16_t dc_event_log_size; } QEMU_PACKED *id; - QEMU_BUILD_BUG_ON(sizeof(*id) != 0x43); + QEMU_BUILD_BUG_ON(sizeof(*id) != 0x45); CXLType3Dev *ct3d = CXL_TYPE3(cci->d); CXLType3Class *cvc = CXL_TYPE3_GET_CLASS(ct3d); CXLDeviceState *cxl_dstate = &ct3d->cxl_dstate; @@ -807,6 +809,7 @@ static CXLRetCode cmd_identify_memory_device(const struct cxl_cmd *cmd, st24_le_p(id->poison_list_max_mer, 256); /* No limit - so limited by main poison record limit */ stw_le_p(&id->inject_poison_limit, 0); + stw_le_p(&id->dc_event_log_size, CXL_DC_EVENT_LOG_SIZE); *len_out = sizeof(*id); return CXL_MBOX_SUCCESS; From patchwork Wed Feb 21 18:15:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: fan X-Patchwork-Id: 13566068 Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0E74084FD8 for ; Wed, 21 Feb 2024 18:21:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708539675; cv=none; b=jf9OBnWgVhrWjL7/a3omv+D0JJJszEMuNjFxSELEOUa4pQ2I1nim63kq0pcLAyB37ETCj2kVXt4j1GpaV9rTROVjS410O685BCI+TuMBxyU8rrRnU5v4h/7EEJeyB5x/fqUpdcCN3SlxOHMartq5nHiR9LSUQNlUvXjzpoat10E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708539675; c=relaxed/simple; bh=7Gx6/dpcKfqDRLViC/V6lC9RYGe2vAovzz0ElkU9LKo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=InoY+GPc32bqS2vsmUr15L9zaonHNnw4Q7x7ESPgZ8diEBtfZoaDkhvpR+Bt73qZdftubsLITFtwW7iQ5xsvOPr2Sq38VLJYUX3+/6CW2rx0Ja/qcV4hH1gchjPX7coiuxfyVNT+/ZXtvxhT/sWdTVPRH56VIEVNs07tnI0T8jY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Z7MvtXX0; arc=none smtp.client-ip=209.85.216.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Z7MvtXX0" Received: by mail-pj1-f45.google.com with SMTP id 98e67ed59e1d1-290ec261a61so4228011a91.0 for ; Wed, 21 Feb 2024 10:21:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708539673; x=1709144473; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=un/ALJ33UJ/4j/7Dwtf0xAxrV2NGLPOZb9LxVVKMbdE=; b=Z7MvtXX0Md9PImyrEa61bwrlu71drm024hDWxFOvup3a0LZTgGmQLi48Oa2FmwzcBy qxgTRZB3GargxpZZ+aGf+dL6MzN/ICQvzUEr6t9xo48s25oESu/a0F/0wN7aFmfgmkjv BJ+ZjDfxjffTHvKXWZR60NKk9UaMUjZ4mXuHEiPBN7DatLR3tz7gYxMa8F3AzfJGg03W gxTQ9Ldbq8P28NnvqNP8FyLoij18uy/6bRIcXOrQwHZo8yCzKw0zHW0n2tW63KXWnGwu KJcc8K5wdZg7jjESduIp7wq/Zo/tCBVExY9qLL8J4v4Ar3/4d+/0MjxzZBX6mZ16g8gg Sikw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708539673; x=1709144473; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=un/ALJ33UJ/4j/7Dwtf0xAxrV2NGLPOZb9LxVVKMbdE=; b=qtE1MD+kNPdDkmIasW6t9qhH2L/tuu186r/73DKa/inuw+AC5BUi9f1Mnu9tyMdx94 tCaXOCoYRkoTmWHKwkxDuXnSsS8uIcBZotSDrH7G0wrgRGrhBBxj3n+vDKy0CWt2IMtP b1vJQuXKHg4/undnMgIsgNnUOHEjUNjiIgH/TgB6z4PX6h+8seoQSX34ycoprMdsbgh+ QMO960UoEH+1QMin9vRHXB3jSiWiHe3+pxjPKkjnn96DPbxYx0RCVru+a1s1H81k0LXC /WTMGa3ffW1VLYcuOqFSPVbWbhsmsEuFQdS5AWbI3Utmyb6aH4xJs3yXD2kpsNxkc4iT HCpA== X-Forwarded-Encrypted: i=1; AJvYcCXCnWWkT4q/Yq+NCT3J2pX6TGaIgGfnZwLvU+P37nJjWv338Ix5EIs+91XrcM4DRKuneOIBfvv/u2+0Y0Lag6nXAeUL8e0YQVdK X-Gm-Message-State: AOJu0YweYzSjcb3aUss/LQB06RQ9yn1/XYaSwQMojr8VMSoZudApMzS6 fzZONYHC0KdDA4WqBPHTyUjCizG2HdcYb7qFVIHKuuyvGGkn7yuF X-Google-Smtp-Source: AGHT+IHZJskgqEMippermGM7haozIZGUI/e1pO3JbuOpb/ZzYlJZuIuyb9dQ1vzD1Ic/ZVpDa2Qq3w== X-Received: by 2002:a17:90a:10d0:b0:299:bade:ed0b with SMTP id b16-20020a17090a10d000b00299badeed0bmr6854885pje.45.1708539673292; Wed, 21 Feb 2024 10:21:13 -0800 (PST) Received: from localhost.localdomain ([2601:641:300:14de:5692:7e41:13a2:69a]) by smtp.gmail.com with ESMTPSA id cs16-20020a17090af51000b002992f49922csm9979009pjb.25.2024.02.21.10.21.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 10:21:13 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Fan Ni Subject: [PATCH v4 02/10] hw/cxl/cxl-mailbox-utils: Add dynamic capacity region representative and mailbox command support Date: Wed, 21 Feb 2024 10:15:55 -0800 Message-ID: <20240221182020.1086096-3-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240221182020.1086096-1-nifan.cxl@gmail.com> References: <20240221182020.1086096-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Per cxl spec r3.1, add dynamic capacity region representative based on Table 8-165 and extend the cxl type3 device definition to include dc region information. Also, based on info in 8.2.9.9.9.1, add 'Get Dynamic Capacity Configuration' mailbox support. Note: decode_len of a dc region is aligned to 256*MiB, divided by 256 * MiB before returned to the host for "Get Dynamic Capacity Configuration" mailbox command. Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 110 ++++++++++++++++++++++++++++++++++++ include/hw/cxl/cxl_device.h | 16 ++++++ 2 files changed, 126 insertions(+) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index ba1d9901df..88e3b733e3 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -22,6 +22,7 @@ #define CXL_CAPACITY_MULTIPLIER (256 * MiB) #define CXL_DC_EVENT_LOG_SIZE 8 +#define CXL_SPEC_AFTER_R30 /* * How to add a new command, example. The command set FOO, with cmd BAR. @@ -80,6 +81,8 @@ enum { #define GET_POISON_LIST 0x0 #define INJECT_POISON 0x1 #define CLEAR_POISON 0x2 + DCD_CONFIG = 0x48, + #define GET_DC_CONFIG 0x0 PHYSICAL_SWITCH = 0x51, #define IDENTIFY_SWITCH_DEVICE 0x0 #define GET_PHYSICAL_PORT_STATE 0x1 @@ -1238,6 +1241,103 @@ static CXLRetCode cmd_media_clear_poison(const struct cxl_cmd *cmd, return CXL_MBOX_SUCCESS; } +/* + * CXL r3.1 section 8.2.9.9.9.1: Get Dynamic Capacity Configuration + * (Opcode: 4800h) + */ +static CXLRetCode cmd_dcd_get_dyn_cap_config(const struct cxl_cmd *cmd, + uint8_t *payload_in, + size_t len_in, + uint8_t *payload_out, + size_t *len_out, + CXLCCI *cci) +{ + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + struct get_dyn_cap_config_in_pl { + uint8_t region_cnt; + uint8_t start_region_id; + } QEMU_PACKED; + + struct get_dyn_cap_config_out_pl { + uint8_t num_regions; + uint8_t regions_returned; + uint8_t rsvd1[6]; + struct { + uint64_t base; + uint64_t decode_len; + uint64_t region_len; + uint64_t block_size; + uint32_t dsmadhandle; + uint8_t flags; + uint8_t rsvd2[3]; + } QEMU_PACKED records[]; + /* + * if cxl spec version >= 3.1, extra output payload as defined + * in struct get_dyn_cap_config_out_pl_extra comes here. + */ + } QEMU_PACKED; + + struct get_dyn_cap_config_in_pl *in = (void *)payload_in; + struct get_dyn_cap_config_out_pl *out = (void *)payload_out; + uint16_t record_count = 0; + uint16_t i; + uint16_t out_pl_len; + uint8_t start_region_id = in->start_region_id; +#ifdef CXL_SPEC_AFTER_R30 + struct get_dyn_cap_config_out_pl_extra { + uint32_t num_extents_supported; + uint32_t num_extents_available; + uint32_t num_tags_supported; + uint32_t num_tags_available; + } QEMU_PACKED; + struct get_dyn_cap_config_out_pl_extra *extra_out; +#endif + + if (start_region_id >= ct3d->dc.num_regions) { + return CXL_MBOX_INVALID_INPUT; + } + + record_count = MIN(ct3d->dc.num_regions - in->start_region_id, + in->region_cnt); + + out_pl_len = sizeof(*out) + record_count * sizeof(out->records[0]); +#ifdef CXL_SPEC_AFTER_R30 + extra_out = (struct get_dyn_cap_config_out_pl_extra *)(payload_out + + out_pl_len); + out_pl_len += sizeof(struct get_dyn_cap_config_out_pl_extra); +#endif + assert(out_pl_len <= CXL_MAILBOX_MAX_PAYLOAD_SIZE); + + out->num_regions = ct3d->dc.num_regions; +#ifdef CXL_SPEC_AFTER_R30 + out->regions_returned = record_count; +#endif + for (i = 0; i < record_count; i++) { + stq_le_p(&out->records[i].base, + ct3d->dc.regions[start_region_id + i].base); + stq_le_p(&out->records[i].decode_len, + ct3d->dc.regions[start_region_id + i].decode_len / + CXL_CAPACITY_MULTIPLIER); + stq_le_p(&out->records[i].region_len, + ct3d->dc.regions[start_region_id + i].len); + stq_le_p(&out->records[i].block_size, + ct3d->dc.regions[start_region_id + i].block_size); + stl_le_p(&out->records[i].dsmadhandle, + ct3d->dc.regions[start_region_id + i].dsmadhandle); + out->records[i].flags = ct3d->dc.regions[start_region_id + i].flags; + } +#ifdef CXL_SPEC_AFTER_R30 + /*FIXME: need to set valid values in the future*/ + stq_le_p(&extra_out->num_extents_supported, 0); + stq_le_p(&extra_out->num_extents_available, 0); + stq_le_p(&extra_out->num_tags_supported, 0); + stq_le_p(&extra_out->num_tags_available, 0); +#endif + + *len_out = out_pl_len; + return CXL_MBOX_SUCCESS; +} + #define IMMEDIATE_CONFIG_CHANGE (1 << 1) #define IMMEDIATE_DATA_CHANGE (1 << 2) #define IMMEDIATE_POLICY_CHANGE (1 << 3) @@ -1282,6 +1382,11 @@ static const struct cxl_cmd cxl_cmd_set[256][256] = { cmd_media_clear_poison, 72, 0 }, }; +static const struct cxl_cmd cxl_cmd_set_dcd[256][256] = { + [DCD_CONFIG][GET_DC_CONFIG] = { "DCD_GET_DC_CONFIG", + cmd_dcd_get_dyn_cap_config, 2, 0 }, +}; + static const struct cxl_cmd cxl_cmd_set_sw[256][256] = { [INFOSTAT][IS_IDENTIFY] = { "IDENTIFY", cmd_infostat_identify, 0, 0 }, [INFOSTAT][BACKGROUND_OPERATION_STATUS] = { "BACKGROUND_OPERATION_STATUS", @@ -1487,7 +1592,12 @@ void cxl_initialize_mailbox_swcci(CXLCCI *cci, DeviceState *intf, void cxl_initialize_mailbox_t3(CXLCCI *cci, DeviceState *d, size_t payload_max) { + CXLType3Dev *ct3d = CXL_TYPE3(d); + cxl_copy_cci_commands(cci, cxl_cmd_set); + if (ct3d->dc.num_regions) { + cxl_copy_cci_commands(cci, cxl_cmd_set_dcd); + } cci->d = d; /* No separation for PCI MB as protocol handled in PCI device */ diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index 3cf3077afa..6df7fecdf1 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -422,6 +422,17 @@ typedef struct CXLPoison { typedef QLIST_HEAD(, CXLPoison) CXLPoisonList; #define CXL_POISON_LIST_LIMIT 256 +#define DCD_MAX_REGION_NUM 8 + +typedef struct CXLDCDRegion { + uint64_t base; + uint64_t decode_len; /* aligned to 256*MiB */ + uint64_t len; + uint64_t block_size; + uint32_t dsmadhandle; + uint8_t flags; +} CXLDCDRegion; + struct CXLType3Dev { /* Private */ PCIDevice parent_obj; @@ -454,6 +465,11 @@ struct CXLType3Dev { unsigned int poison_list_cnt; bool poison_list_overflowed; uint64_t poison_list_overflow_ts; + + struct dynamic_capacity { + uint8_t num_regions; /* 0-8 regions */ + CXLDCDRegion regions[DCD_MAX_REGION_NUM]; + } dc; }; #define TYPE_CXL_TYPE3 "cxl-type3" From patchwork Wed Feb 21 18:15:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: fan X-Patchwork-Id: 13566069 Received: from mail-pg1-f169.google.com (mail-pg1-f169.google.com [209.85.215.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1FAFC8527A for ; Wed, 21 Feb 2024 18:21:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708539676; cv=none; b=ODnE9IXvPWraKKsgGjZUznJCXCoRBn0IeDIu9fJZwq4rGp2U33c9H2JQ40H5klFr6ssSuhjOR7Q4uSOF6AsOVew+aDCQVeOqrH9mFJ/LOSPyLBGN9OoirMKEztAVpEe9FMTbLtJ18bmU5SRv9hoA/slxT0gcogj0Uz2H15vmXsc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708539676; c=relaxed/simple; bh=AZ2NQgPZiDbodG5N8nCybOdRqZa4Yfmcx3M295mbScY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SdzuafO3D7JzzjyzQIISFdfQqYeCiAvfIdOsAfw/xAqLBswDlz/qz0njwqAJGh02FzTrBCP9otFRmmiZeaRdZt+mY/32QzFJXYBy9MMuRtM9ipD+ex6S8/yIrLAFkGgVteA7JCqGN9BqbTeeWvCRxxsPCBJVhlkwB+rWe18dKYc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=iU2+xLo4; arc=none smtp.client-ip=209.85.215.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="iU2+xLo4" Received: by mail-pg1-f169.google.com with SMTP id 41be03b00d2f7-5d8b276979aso5388076a12.2 for ; Wed, 21 Feb 2024 10:21:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708539674; x=1709144474; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jnjf2LLXHCSGzRVtlUxFuz40qj2WpBzq7LFxL5kuDZQ=; b=iU2+xLo4BTlk0/mVk0tTGsrnmZOybVOJZ4HPj/Z3l9GnUy/HnsKF6entVVmIjRgMQC yJBV1aB6yDyk/YdhTaxcrvAgaTb/vxMTgZe77gkykQcxTY0Y2OqAR/C7s78+vGafrGjA wW2I4Lcd5LmoPaHKt8vttv/e6VRX2Fb4+s5WlKnI7SVFdZ4Z5+ekJ9qXBouQq1yCL/r4 K/Zd6BXfi9PyssQc0fuiqNU6st2dU6IU7qOuYM6eUh99+fYK1cPFQeDvD49sdxOcMq8p uimCzFue2+VvZAfR5bFKaqhgKaAk3tzy/rWamY66dpBr3UAnhetjh2HHdEGGT7L/TdaC Zzug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708539674; x=1709144474; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jnjf2LLXHCSGzRVtlUxFuz40qj2WpBzq7LFxL5kuDZQ=; b=qSz+YmLl/YrgIAKnWYN/YCLsAG22bjyi/+ga6q6TplY1josmJT16TugMd6Rza1jK/a iwGykk1WeL+XadZS/A2GMuSoWkieq/gyZuavayCLXlBdBl9hIQS2h5V+0rMjLByVmCxe Z2Z5rbfGoPj7Qtgt95JAGKGPcf4p65nd9s5BYtvMhkt8oqgdRzHTxc46j6UuFTFs5ZtQ 48VLpk8KMtD37guXkeV6ErgVGqTpC25LbIqmUgmjoU2dGFtlQiTRwmfYap32R1W6E92x TyqcZ8UjbRL8eqOyRNxivR1mmm3IL6jmNdUjHptTuCVRuWnCD71Ksob1TP/MbF+GIkd6 OrEQ== X-Forwarded-Encrypted: i=1; AJvYcCUJ7dvjgvt33VGySERoDkCcnhLRSGYjiZOl6BvjCg4GKawxDf7T+RzVE+QSBX15/Z3N/uoWYYYwGk+zZB5fYbahy4A9Bg2h8pzz X-Gm-Message-State: AOJu0YzfVX8pwdXNrdFeN6MUodjpkkoRVmgXGji0lrZ/NoENaF1BB3er EGI68UUm4ZovJGwzZtF8uuU7+bKgp5po4Q3Pfn1Vnf+A8XkMpfLv X-Google-Smtp-Source: AGHT+IHROlOq4gqtw0lnNWWoRjKU1S6sa8+quFg1tqY8CpOuHQhfr0DhIXnQnZ4RDbnFAreDF8H/lA== X-Received: by 2002:a17:90a:ae0d:b0:299:c344:2b5 with SMTP id t13-20020a17090aae0d00b00299c34402b5mr6298324pjq.40.1708539674434; Wed, 21 Feb 2024 10:21:14 -0800 (PST) Received: from localhost.localdomain ([2601:641:300:14de:5692:7e41:13a2:69a]) by smtp.gmail.com with ESMTPSA id cs16-20020a17090af51000b002992f49922csm9979009pjb.25.2024.02.21.10.21.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 10:21:14 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Fan Ni Subject: [PATCH v4 03/10] include/hw/cxl/cxl_device: Rename mem_size as static_mem_size for type3 memory devices Date: Wed, 21 Feb 2024 10:15:56 -0800 Message-ID: <20240221182020.1086096-4-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240221182020.1086096-1-nifan.cxl@gmail.com> References: <20240221182020.1086096-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Rename mem_size as static_mem_size for type3 memdev to cover static RAM and pmem capacity, preparing for the introduction of dynamic capacity to support dynamic capacity devices. Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 4 ++-- hw/mem/cxl_type3.c | 8 ++++---- include/hw/cxl/cxl_device.h | 2 +- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 88e3b733e3..7d2b74c9c5 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -802,7 +802,7 @@ static CXLRetCode cmd_identify_memory_device(const struct cxl_cmd *cmd, snprintf(id->fw_revision, 0x10, "BWFW VERSION %02d", 0); stq_le_p(&id->total_capacity, - cxl_dstate->mem_size / CXL_CAPACITY_MULTIPLIER); + cxl_dstate->static_mem_size / CXL_CAPACITY_MULTIPLIER); stq_le_p(&id->persistent_capacity, cxl_dstate->pmem_size / CXL_CAPACITY_MULTIPLIER); stq_le_p(&id->volatile_capacity, @@ -1178,7 +1178,7 @@ static CXLRetCode cmd_media_clear_poison(const struct cxl_cmd *cmd, struct clear_poison_pl *in = (void *)payload_in; dpa = ldq_le_p(&in->dpa); - if (dpa + CXL_CACHE_LINE_SIZE > cxl_dstate->mem_size) { + if (dpa + CXL_CACHE_LINE_SIZE > cxl_dstate->static_mem_size) { return CXL_MBOX_INVALID_PA; } diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index e8801805b9..244d2b5fd5 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -608,7 +608,7 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) } address_space_init(&ct3d->hostvmem_as, vmr, v_name); ct3d->cxl_dstate.vmem_size = memory_region_size(vmr); - ct3d->cxl_dstate.mem_size += memory_region_size(vmr); + ct3d->cxl_dstate.static_mem_size += memory_region_size(vmr); g_free(v_name); } @@ -631,7 +631,7 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) } address_space_init(&ct3d->hostpmem_as, pmr, p_name); ct3d->cxl_dstate.pmem_size = memory_region_size(pmr); - ct3d->cxl_dstate.mem_size += memory_region_size(pmr); + ct3d->cxl_dstate.static_mem_size += memory_region_size(pmr); g_free(p_name); } @@ -837,7 +837,7 @@ static int cxl_type3_hpa_to_as_and_dpa(CXLType3Dev *ct3d, return -EINVAL; } - if (*dpa_offset > ct3d->cxl_dstate.mem_size) { + if (*dpa_offset > ct3d->cxl_dstate.static_mem_size) { return -EINVAL; } @@ -1010,7 +1010,7 @@ static bool set_cacheline(CXLType3Dev *ct3d, uint64_t dpa_offset, uint8_t *data) return false; } - if (dpa_offset + CXL_CACHE_LINE_SIZE > ct3d->cxl_dstate.mem_size) { + if (dpa_offset + CXL_CACHE_LINE_SIZE > ct3d->cxl_dstate.static_mem_size) { return false; } diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index 6df7fecdf1..80188db670 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -234,7 +234,7 @@ typedef struct cxl_device_state { } timestamp; /* memory region size, HDM */ - uint64_t mem_size; + uint64_t static_mem_size; uint64_t pmem_size; uint64_t vmem_size; From patchwork Wed Feb 21 18:15:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: fan X-Patchwork-Id: 13566070 Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0BDB085299 for ; Wed, 21 Feb 2024 18:21:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708539677; cv=none; b=bG8IFU5aAIwFFhY0RP/hmO6QOrjRxZQnIqUoAfSIwYYmpeVLNgPuq11rk6oqqrhKTi7BtMDcbotBYaMd+Y69jifMALiBr4n13RlLpn1KBvj4raqy5h8YBp8SKWMhn2wVQxOBeRoYmKmWPfTGwRwvIWzWC5ieojwBr9qAnGWMKKM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708539677; c=relaxed/simple; bh=XhwQLCsT4y4NxoCTZskeCpEF4H1oP/DoZP3twHS5YYk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=T0vm8Y/x/3Bw+FefHv5iwu0vtMkRy9cZEaPtsIjRITKIRvPr7Rgrhoo92UkvioyMlzEmLUgg0hCyUJaMkFfnQnByIoGzio149T8fKKMaw091w7Dcxcoji4HcK7KvL5IDxMtb/Gb1CnCJYyvcTaRmLhZBQaf8Ojg3ByON+O8wY30= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=VXxDEHPX; arc=none smtp.client-ip=209.85.216.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="VXxDEHPX" Received: by mail-pj1-f53.google.com with SMTP id 98e67ed59e1d1-299dba8fd24so1688680a91.2 for ; Wed, 21 Feb 2024 10:21:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708539675; x=1709144475; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WJqQjnlAGwd774HUXqTHZfBD58qfYj95z3AiQp1cluE=; b=VXxDEHPXVox+Y2ULGfSJ4PIgnLT+ZkbXY00ccgzGxWjpERBk/5Vr5YxQbFe/9Lx4Rn J+7T7c5W/TbupP9DbY1SLH8sxp5HBpudG+OkXGpWEMmtBxS6eU5kXtIiu/MKUGJLAbcG KaekOGRC8DF3PAb5UTjx69Dc6R1ZFZtTcDm43ur4s+zB39JUWsf0AF6hITsiYqefEt+v 8YXVp1D8lI3HwmEfkqvFxbDksIp6ntj8OIq6MaxDLJSLyQ2rC01keeW2iR+UNTgdA9Sl WepTNBgfy7t3Bqe30c1gOASvs17Fuixs/1jrkQ2ezlSOH8sU/Von6MG6VStRvV0qLlq/ QrsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708539675; x=1709144475; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WJqQjnlAGwd774HUXqTHZfBD58qfYj95z3AiQp1cluE=; b=OgKJL+Vk6dRksiE5Z304Cxow59fVPIzRkYkltLp+TRbCspHabK0ZgxDX9Sf55pxb/x NRtDrPSJcc+tqIDaJDZ+DIZi92ucHvhbUdGRGiX4Dn5GWNe7QwlR6mxadi8ZnmjcSXNp 5AKldgbScbpxjulAobLvY2MLnWUEx2SmdukSU7qLyNyinng7S9zWG/E1cJcGcoQvQo73 iC3ReoMKTwcZFzRbqTyGesAi1+ClKnH8UvR06BdOrMmsyr+5isOx2Vx0LLtKa4ZqPfgS qNhel0EDg2k1pLbFR6Euvi28u7wmC4uIybpBn40wbmC5MsyNDYtxOZd/vxlRdxaOcD1B 936A== X-Forwarded-Encrypted: i=1; AJvYcCXv6Y70uA5muS06asD5YpoAVj5sTZnI9MBVWglUeQxFrbwV+sFnHL9yrq9r4156K59wi7nseCS8ga+BPDEbZS2rk7wvgcOyYJ5x X-Gm-Message-State: AOJu0YzbgUcxeKBSz+MfO5T50E6u7KuSL9kIdAjWwwf5WNakgG4ECWXH Lw34uu3xD5JFv4iiPMokOPfbos5DtzF8zh4DwRS1gMHUM9GROXTW X-Google-Smtp-Source: AGHT+IGIpQfawDAr7HNOdT8nJf9XYpRgVO5h7M9s1ju3JADRvCiD6h78Yzt+I16QV7f+Bfep1X3C2g== X-Received: by 2002:a17:90a:fa0c:b0:299:2ed1:d462 with SMTP id cm12-20020a17090afa0c00b002992ed1d462mr13082477pjb.6.1708539675466; Wed, 21 Feb 2024 10:21:15 -0800 (PST) Received: from localhost.localdomain ([2601:641:300:14de:5692:7e41:13a2:69a]) by smtp.gmail.com with ESMTPSA id cs16-20020a17090af51000b002992f49922csm9979009pjb.25.2024.02.21.10.21.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 10:21:15 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Fan Ni Subject: [PATCH v4 04/10] hw/mem/cxl_type3: Add support to create DC regions to type3 memory devices Date: Wed, 21 Feb 2024 10:15:57 -0800 Message-ID: <20240221182020.1086096-5-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240221182020.1086096-1-nifan.cxl@gmail.com> References: <20240221182020.1086096-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni With the change, when setting up memory for type3 memory device, we can create DC regions. A property 'num-dc-regions' is added to ct3_props to allow users to pass the number of DC regions to create. To make it easier, other region parameters like region base, length, and block size are hard coded. If needed, these parameters can be added easily. With the change, we can create DC regions with proper kernel side support like below: region=$(cat /sys/bus/cxl/devices/decoder0.0/create_dc_region) echo $region > /sys/bus/cxl/devices/decoder0.0/create_dc_region echo 256 > /sys/bus/cxl/devices/$region/interleave_granularity echo 1 > /sys/bus/cxl/devices/$region/interleave_ways echo "dc0" >/sys/bus/cxl/devices/decoder2.0/mode echo 0x40000000 >/sys/bus/cxl/devices/decoder2.0/dpa_size echo 0x40000000 > /sys/bus/cxl/devices/$region/size echo "decoder2.0" > /sys/bus/cxl/devices/$region/target0 echo 1 > /sys/bus/cxl/devices/$region/commit echo $region > /sys/bus/cxl/drivers/cxl_region/bind However, we cannot really read/write to the DC regions due to lack of 1. host backend and address space setup for DC regions; 2. mailbox command support for adding/releasing DC extents. Signed-off-by: Fan Ni --- hw/mem/cxl_type3.c | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 244d2b5fd5..c61cd2b5ac 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -567,6 +567,40 @@ static void ct3d_reg_write(void *opaque, hwaddr offset, uint64_t value, } } +/* + * TODO: dc region configuration will be updated once host backend and address + * space support is added for DCD. + */ +static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) +{ + int i; + uint64_t region_base = 0; + uint64_t region_len = 2 * GiB; + uint64_t decode_len = 2 * GiB; + uint64_t blk_size = 2 * MiB; + CXLDCDRegion *region; + + if (ct3d->hostvmem) { + region_base += ct3d->hostvmem->size; + } + if (ct3d->hostpmem) { + region_base += ct3d->hostpmem->size; + } + for (i = 0; i < ct3d->dc.num_regions; i++) { + region = &ct3d->dc.regions[i]; + region->base = region_base; + region->decode_len = decode_len; + region->len = region_len; + region->block_size = blk_size; + /* dsmad_handle is set when creating cdat table entries */ + region->flags = 0; + + region_base += region->len; + } + + return true; +} + static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) { DeviceState *ds = DEVICE(ct3d); @@ -635,6 +669,11 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) g_free(p_name); } + if (!cxl_create_dc_regions(ct3d, errp)) { + error_setg(errp, "setup DC regions failed"); + return false; + } + return true; } @@ -930,6 +969,7 @@ static Property ct3_props[] = { HostMemoryBackend *), DEFINE_PROP_UINT64("sn", CXLType3Dev, sn, UI64_NULL), DEFINE_PROP_STRING("cdat", CXLType3Dev, cxl_cstate.cdat.filename), + DEFINE_PROP_UINT8("num-dc-regions", CXLType3Dev, dc.num_regions, 0), DEFINE_PROP_END_OF_LIST(), }; From patchwork Wed Feb 21 18:15:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: fan X-Patchwork-Id: 13566071 Received: from mail-pg1-f178.google.com (mail-pg1-f178.google.com [209.85.215.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 858FE8562D for ; Wed, 21 Feb 2024 18:21:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708539679; cv=none; b=n8v+/6BXAFGrKZl8iyL2+TAJ3sEf2XVxGZts14/LthRbAc2MCDtH3QswFrhmeaB7FlcUr941jcrOIOCvW7yZ+lG+02QWD6330LpdVhD9/0oyPrwjOxb818lTb+ecLHlI5maODm5waATYVvx47AxJnt9VfcAqlLAF6KKfek5D8kI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708539679; c=relaxed/simple; bh=zTqh34z/H70305+jtLk6lO1wrIbIYiaObLrOdQdgXzg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=RC0cVmFfTZlfKLyRJ4w5rhKq+cm7dyt18aHOnHJ4Z6v2oj0jhtSFa1LjK3/zkIxDRhT42eo4PZ8uqXHBuYXCP1mDiJfxKBfO6he1Z+oa1LiHYq7oy0wf0rEV7Klykkc3S1o3feJKOyO34Edgx8BhwC/Cc7iIhvs27gL1AckbgIc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=PhdwHJ7d; arc=none smtp.client-ip=209.85.215.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="PhdwHJ7d" Received: by mail-pg1-f178.google.com with SMTP id 41be03b00d2f7-5cddc5455aeso4774670a12.1 for ; Wed, 21 Feb 2024 10:21:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708539677; x=1709144477; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eiSoVZJtjOJJ7IGlGmk8cRI2lGtWO4AhdMc312mNoV4=; b=PhdwHJ7d2neAPfJineKejxQM3yl9BznaP7zCWTOrFpDyPG+yglSG9Fe8bXzgfwwhH/ IIPRc/bcprZwjvYdADRQvkeYVRHFDzixRQACFPCkk9P+OuKjTn/6yjPBQBAgcSyuRD+A VElTvkiws8M6rs3cyQOMmKW1viPzZji1wQfKAbN0e8sZP8rxxaDOFSeT7DRwHsuO6QOH yesW8ku2MCUtmv7+Wi0ITpiN+bjUHjTkzH0HYBOsFjFM0URadTdffF75/A0xh8UmRxtB ZXgRm5kk0mpiOLkfxu4X2BRnl0TmNjv/bfbgkVBzuHA3WlMsHiKIa7ibCdVxlexloLOL IXBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708539677; x=1709144477; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eiSoVZJtjOJJ7IGlGmk8cRI2lGtWO4AhdMc312mNoV4=; b=HOXDtZ01S0WBjGyjZv9lY/b8DXHC1FKjm/QyfsMBZemeo8lsg08Ehd3wcglFtIQmhQ 88sRfL5hwcqyZm9+0a6IGt0e32jHoaxdFNtUCkoU0AJ/fToVlxeaJ4sheXdgFV3EhQAZ 2KbG1fkgemGHSpMKa05dBqG3fYi/hHvrTfTRbM/5Ip4pmtJ3Ky3uczEx0hHIPD6JTD6U kUUAiNqG5kpegcTKnAU7q1yDsp3ZyDdEsUCdwAlGI713M2jjIHxp7bkKUnZnJKpnb4Aj sTTVlyXgTnaTQu29NSQ+mf7oJSg0d7Wj3xUhk7mPvDofx1z9rUD1kixFFd+Rom5PYash EsBw== X-Forwarded-Encrypted: i=1; AJvYcCXkLXtOJfe9C+0Kci41jEyr/fgRItMEpfYOaNUYGGJ9ULWoAKqVDkZAdtXjxuC6bK+nGXdJYvOq0Xx2lNsapGqkP+l31QF4mew7 X-Gm-Message-State: AOJu0Yy9VVyg9ZEKZl9FjvQR9znpFUQrZ9YFXpjuTH5uy4LAro8cxkyM gJZbob1We1AyMrxoGMjvnk9kJ2axzK8Ldjy2x9w0Ml6X7BKxFDQj X-Google-Smtp-Source: AGHT+IGAk48olbaYcijRDumKRKOL973AAWX5mM+93KfBcG5wDx9QsekZGw40m+D/x9wD0JEaDPXDfg== X-Received: by 2002:a17:90a:cc9:b0:297:24da:887e with SMTP id 9-20020a17090a0cc900b0029724da887emr15889680pjt.18.1708539676735; Wed, 21 Feb 2024 10:21:16 -0800 (PST) Received: from localhost.localdomain ([2601:641:300:14de:5692:7e41:13a2:69a]) by smtp.gmail.com with ESMTPSA id cs16-20020a17090af51000b002992f49922csm9979009pjb.25.2024.02.21.10.21.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 10:21:16 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Fan Ni Subject: [PATCH v4 05/10] hw/mem/cxl-type3: Refactor ct3_build_cdat_entries_for_mr to take mr size insead of mr as argument Date: Wed, 21 Feb 2024 10:15:58 -0800 Message-ID: <20240221182020.1086096-6-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240221182020.1086096-1-nifan.cxl@gmail.com> References: <20240221182020.1086096-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni The function ct3_build_cdat_entries_for_mr only uses size of the passed memory region argument, refactor the function definition to make the passed arguments more specific. Signed-off-by: Fan Ni --- hw/mem/cxl_type3.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index c61cd2b5ac..6e5f908fb1 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -43,7 +43,7 @@ enum { }; static void ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, - int dsmad_handle, MemoryRegion *mr, + int dsmad_handle, uint64_t size, bool is_pmem, uint64_t dpa_base) { g_autofree CDATDsmas *dsmas = NULL; @@ -62,7 +62,7 @@ static void ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, .DSMADhandle = dsmad_handle, .flags = is_pmem ? CDAT_DSMAS_FLAG_NV : 0, .DPA_base = dpa_base, - .DPA_length = memory_region_size(mr), + .DPA_length = size, }; /* For now, no memory side cache, plausiblish numbers */ @@ -131,7 +131,7 @@ static void ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, */ .EFI_memory_type_attr = is_pmem ? 2 : 1, .DPA_offset = 0, - .DPA_length = memory_region_size(mr), + .DPA_length = size, }; /* Header always at start of structure */ @@ -148,6 +148,7 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) g_autofree CDATSubHeader **table = NULL; CXLType3Dev *ct3d = priv; MemoryRegion *volatile_mr = NULL, *nonvolatile_mr = NULL; + uint64_t vmr_size = 0, pmr_size = 0; int dsmad_handle = 0; int cur_ent = 0; int len = 0; @@ -162,6 +163,7 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) return -EINVAL; } len += CT3_CDAT_NUM_ENTRIES; + vmr_size = memory_region_size(volatile_mr); } if (ct3d->hostpmem) { @@ -170,21 +172,22 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) return -EINVAL; } len += CT3_CDAT_NUM_ENTRIES; + pmr_size = memory_region_size(nonvolatile_mr); } table = g_malloc0(len * sizeof(*table)); /* Now fill them in */ if (volatile_mr) { - ct3_build_cdat_entries_for_mr(table, dsmad_handle++, volatile_mr, + ct3_build_cdat_entries_for_mr(table, dsmad_handle++, vmr_size, false, 0); cur_ent = CT3_CDAT_NUM_ENTRIES; } if (nonvolatile_mr) { - uint64_t base = volatile_mr ? memory_region_size(volatile_mr) : 0; + uint64_t base = vmr_size; ct3_build_cdat_entries_for_mr(&(table[cur_ent]), dsmad_handle++, - nonvolatile_mr, true, base); + pmr_size, true, base); cur_ent += CT3_CDAT_NUM_ENTRIES; } assert(len == cur_ent); From patchwork Wed Feb 21 18:15:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: fan X-Patchwork-Id: 13566072 Received: from mail-pg1-f171.google.com (mail-pg1-f171.google.com [209.85.215.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DD2E484FC5 for ; Wed, 21 Feb 2024 18:21:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708539680; cv=none; b=f1WfkuYk8cLetrQQ7FPHxEK+ejgxqbgxuNsrx6EjFMVB3ZvIjKuu+XUm8YjQXmC5IVmM6Xqfwgr2i/2dDjEBgLxVlMlrx1N44M5vpt0idUwpUaauC+DPxbktDMFzi+p+QDYX/Z0d82LcUqHDfxIdPv20Vj9t1IOaBrBgtVCJYi8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708539680; c=relaxed/simple; bh=wwTzhgPfGwW02M1ZAuyEF2oDyXuSfcQBR26acC5ZY7o=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=JsgI+eKzF8KoCwUerhmvbdeO+gtMSbaLG4U+sWC2bjK9tKQyktt/By8ijC+W54wv7NuuCb0JDwgTWmKg1JvBOACEoDIY/qtbaUo3HrQGqJXjxTqSBbGloObXQYZARR+JNKwidh8ernH/s6Vdz3e124xS+O9A5gUgIbVfy45X31w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=hImn38RD; arc=none smtp.client-ip=209.85.215.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="hImn38RD" Received: by mail-pg1-f171.google.com with SMTP id 41be03b00d2f7-5ce9555d42eso686323a12.2 for ; Wed, 21 Feb 2024 10:21:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708539678; x=1709144478; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ByXkFoZstt7Taile7J7sAjWvfPjaE3AcMIHhZiOAS+I=; b=hImn38RDKtKAyU2LR0VvsKqPF0xw9l4cCP2za/9cZyTpIsdm9Uu3o5x3a7hFRBJ38S 6MK/vh6E9NnEkD+48U+sgHHF3VFYPcAaIALUhP0tS/UVgeDTtfY4Evq2xcOJsS5gryvy rFTyF/9M4aEjjm4yTsup2W6wE0UlHB7vABe3gvutn7ztk7IhJVvt0WdMv+2f7KcTbDi5 BM1FXi0vCSS1pAZk3m8id92K442+w/wjgUlC/ZKmcl7EwlAawrmpcChk7V5eDLgBHbqk WEyoqACew8Eq/UD4E3V02AY7sobP43+YykF6v/EnORlt4L+tWd4U4vtG3AG4WkHt4CjD BL9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708539678; x=1709144478; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ByXkFoZstt7Taile7J7sAjWvfPjaE3AcMIHhZiOAS+I=; b=qhMkv8Za0sxZ30mmIhOWKA3IaPrmc+9luwM4R88dJEqWO8DCGhfbaXRSsJLWDuhRsn U6EHk6G91JM7Bp5d5T8hHKroYLs+B9ALTkU+okVW5TfvR3hkHpyGsFqcHaGIKRAM9Un3 zdv2NhyzFHQvnlVGD0tyK9goWuGAZff2FTYdYXYAOK8WQj6q3cj/DSgkoYfrxszDgD0c H6+SOtxknwjLrz2eDjFcZzYmUuxYGYyW40x3eG0jf5FwtRG5o/+7R8D3QwcRD+nX+o+F IJmlCwVqsUQJ6lJWA9zH7O2oASiFHOFtP7R4n7hze/JFk359ETqQTrKHxz7/iCj3fS22 CyWQ== X-Forwarded-Encrypted: i=1; AJvYcCWfHQN7QbpS3FP3R4gtk4j9m3rrVT7ShaFgaASOPboVsW1eabtSybpdMH/3T81pclPMOWG5Gq4HcDvvRSlgnOsCrcwkC3EGZFc6 X-Gm-Message-State: AOJu0YxoTpJ1e6QvY6vqEHuBbCYZjrppL/UDpbdUu9wtGrZG/U27RVbr NlH6nowsd9+NIhIV15nmnNgUUqRcXQ8gvWoHYbl0j5KXfoJkIzW5 X-Google-Smtp-Source: AGHT+IEDrY9CBvXUVAZCTG6XPyMxCbOD4bhgwDQW7wmUAU1B3xHNd5EHMNgjtfioPjh2SMGZMZxwqw== X-Received: by 2002:a17:90a:b014:b0:299:6e88:7b6a with SMTP id x20-20020a17090ab01400b002996e887b6amr10884499pjq.36.1708539678029; Wed, 21 Feb 2024 10:21:18 -0800 (PST) Received: from localhost.localdomain ([2601:641:300:14de:5692:7e41:13a2:69a]) by smtp.gmail.com with ESMTPSA id cs16-20020a17090af51000b002992f49922csm9979009pjb.25.2024.02.21.10.21.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 10:21:17 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Fan Ni Subject: [PATCH v4 06/10] hw/mem/cxl_type3: Add host backend and address space handling for DC regions Date: Wed, 21 Feb 2024 10:15:59 -0800 Message-ID: <20240221182020.1086096-7-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240221182020.1086096-1-nifan.cxl@gmail.com> References: <20240221182020.1086096-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Add (file/memory backed) host backend, all the dynamic capacity regions will share a single, large enough host backend. Set up address space for DC regions to support read/write operations to dynamic capacity for DCD. With the change, following supports are added: 1. Add a new property to type3 device "volatile-dc-memdev" to point to host memory backend for dynamic capacity. Currently, all dc regions share one one host backend. 2. Add namespace for dynamic capacity for read/write support; 3. Create cdat entries for each dynamic capacity region; 4. Fix dvsec range registers to include DC regions. Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 16 ++- hw/mem/cxl_type3.c | 188 ++++++++++++++++++++++++++++-------- include/hw/cxl/cxl_device.h | 4 + 3 files changed, 165 insertions(+), 43 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 7d2b74c9c5..f95e417683 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -621,7 +621,8 @@ static CXLRetCode cmd_firmware_update_get_info(const struct cxl_cmd *cmd, size_t *len_out, CXLCCI *cci) { - CXLDeviceState *cxl_dstate = &CXL_TYPE3(cci->d)->cxl_dstate; + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + CXLDeviceState *cxl_dstate = &ct3d->cxl_dstate; struct { uint8_t slots_supported; uint8_t slot_info; @@ -635,7 +636,8 @@ static CXLRetCode cmd_firmware_update_get_info(const struct cxl_cmd *cmd, QEMU_BUILD_BUG_ON(sizeof(*fw_info) != 0x50); if ((cxl_dstate->vmem_size < CXL_CAPACITY_MULTIPLIER) || - (cxl_dstate->pmem_size < CXL_CAPACITY_MULTIPLIER)) { + (cxl_dstate->pmem_size < CXL_CAPACITY_MULTIPLIER) || + (ct3d->dc.total_capacity < CXL_CAPACITY_MULTIPLIER)) { return CXL_MBOX_INTERNAL_ERROR; } @@ -792,7 +794,8 @@ static CXLRetCode cmd_identify_memory_device(const struct cxl_cmd *cmd, CXLDeviceState *cxl_dstate = &ct3d->cxl_dstate; if ((!QEMU_IS_ALIGNED(cxl_dstate->vmem_size, CXL_CAPACITY_MULTIPLIER)) || - (!QEMU_IS_ALIGNED(cxl_dstate->pmem_size, CXL_CAPACITY_MULTIPLIER))) { + (!QEMU_IS_ALIGNED(cxl_dstate->pmem_size, CXL_CAPACITY_MULTIPLIER)) || + (!QEMU_IS_ALIGNED(ct3d->dc.total_capacity, CXL_CAPACITY_MULTIPLIER))) { return CXL_MBOX_INTERNAL_ERROR; } @@ -834,9 +837,11 @@ static CXLRetCode cmd_ccls_get_partition_info(const struct cxl_cmd *cmd, uint64_t next_pmem; } QEMU_PACKED *part_info = (void *)payload_out; QEMU_BUILD_BUG_ON(sizeof(*part_info) != 0x20); + CXLType3Dev *ct3d = container_of(cxl_dstate, CXLType3Dev, cxl_dstate); if ((!QEMU_IS_ALIGNED(cxl_dstate->vmem_size, CXL_CAPACITY_MULTIPLIER)) || - (!QEMU_IS_ALIGNED(cxl_dstate->pmem_size, CXL_CAPACITY_MULTIPLIER))) { + (!QEMU_IS_ALIGNED(cxl_dstate->pmem_size, CXL_CAPACITY_MULTIPLIER)) || + (!QEMU_IS_ALIGNED(ct3d->dc.total_capacity, CXL_CAPACITY_MULTIPLIER))) { return CXL_MBOX_INTERNAL_ERROR; } @@ -1178,7 +1183,8 @@ static CXLRetCode cmd_media_clear_poison(const struct cxl_cmd *cmd, struct clear_poison_pl *in = (void *)payload_in; dpa = ldq_le_p(&in->dpa); - if (dpa + CXL_CACHE_LINE_SIZE > cxl_dstate->static_mem_size) { + if (dpa + CXL_CACHE_LINE_SIZE > cxl_dstate->static_mem_size + + ct3d->dc.total_capacity) { return CXL_MBOX_INVALID_PA; } diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 6e5f908fb1..b966fa4f10 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -30,6 +30,7 @@ #include "hw/pci/msix.h" #define DWORD_BYTE 4 +#define CXL_CAPACITY_MULTIPLIER (256 * MiB) /* Default CDAT entries for a memory region */ enum { @@ -44,7 +45,8 @@ enum { static void ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, int dsmad_handle, uint64_t size, - bool is_pmem, uint64_t dpa_base) + bool is_pmem, bool is_dynamic, + uint64_t dpa_base) { g_autofree CDATDsmas *dsmas = NULL; g_autofree CDATDslbis *dslbis0 = NULL; @@ -60,7 +62,8 @@ static void ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, .length = sizeof(*dsmas), }, .DSMADhandle = dsmad_handle, - .flags = is_pmem ? CDAT_DSMAS_FLAG_NV : 0, + .flags = (is_pmem ? CDAT_DSMAS_FLAG_NV : 0) | + (is_dynamic ? CDAT_DSMAS_FLAG_DYNAMIC_CAP : 0), .DPA_base = dpa_base, .DPA_length = size, }; @@ -148,12 +151,13 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) g_autofree CDATSubHeader **table = NULL; CXLType3Dev *ct3d = priv; MemoryRegion *volatile_mr = NULL, *nonvolatile_mr = NULL; + MemoryRegion *dc_mr = NULL; uint64_t vmr_size = 0, pmr_size = 0; int dsmad_handle = 0; int cur_ent = 0; int len = 0; - if (!ct3d->hostpmem && !ct3d->hostvmem) { + if (!ct3d->hostpmem && !ct3d->hostvmem && !ct3d->dc.num_regions) { return 0; } @@ -175,21 +179,51 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) pmr_size = memory_region_size(nonvolatile_mr); } + if (ct3d->dc.num_regions) { + if (ct3d->dc.host_dc) { + dc_mr = host_memory_backend_get_memory(ct3d->dc.host_dc); + if (!dc_mr) { + return -EINVAL; + } + len += CT3_CDAT_NUM_ENTRIES * ct3d->dc.num_regions; + } else { + return -EINVAL; + } + } + table = g_malloc0(len * sizeof(*table)); /* Now fill them in */ if (volatile_mr) { ct3_build_cdat_entries_for_mr(table, dsmad_handle++, vmr_size, - false, 0); + false, false, 0); cur_ent = CT3_CDAT_NUM_ENTRIES; } if (nonvolatile_mr) { uint64_t base = vmr_size; ct3_build_cdat_entries_for_mr(&(table[cur_ent]), dsmad_handle++, - pmr_size, true, base); + pmr_size, true, false, base); cur_ent += CT3_CDAT_NUM_ENTRIES; } + + if (dc_mr) { + int i; + uint64_t region_base = vmr_size + pmr_size; + + /* FIXME: Currently we assume the dynamic capacity to be volatile. */ + for (i = 0; i < ct3d->dc.num_regions; i++) { + ct3_build_cdat_entries_for_mr(&(table[cur_ent]), + dsmad_handle++, + ct3d->dc.regions[i].len, + false, true, region_base); + ct3d->dc.regions[i].dsmadhandle = dsmad_handle - 1; + + cur_ent += CT3_CDAT_NUM_ENTRIES; + region_base += ct3d->dc.regions[i].len; + } + } + assert(len == cur_ent); *cdat_table = g_steal_pointer(&table); @@ -299,11 +333,24 @@ static void build_dvsecs(CXLType3Dev *ct3d) range2_size_hi = ct3d->hostpmem->size >> 32; range2_size_lo = (2 << 5) | (2 << 2) | 0x3 | (ct3d->hostpmem->size & 0xF0000000); + } else if (ct3d->dc.host_dc) { + range2_size_hi = ct3d->dc.host_dc->size >> 32; + range2_size_lo = (2 << 5) | (2 << 2) | 0x3 | + (ct3d->dc.host_dc->size & 0xF0000000); } - } else { + } else if (ct3d->hostpmem) { range1_size_hi = ct3d->hostpmem->size >> 32; range1_size_lo = (2 << 5) | (2 << 2) | 0x3 | (ct3d->hostpmem->size & 0xF0000000); + if (ct3d->dc.host_dc) { + range2_size_hi = ct3d->dc.host_dc->size >> 32; + range2_size_lo = (2 << 5) | (2 << 2) | 0x3 | + (ct3d->dc.host_dc->size & 0xF0000000); + } + } else { + range1_size_hi = ct3d->dc.host_dc->size >> 32; + range1_size_lo = (2 << 5) | (2 << 2) | 0x3 | + (ct3d->dc.host_dc->size & 0xF0000000); } dvsec = (uint8_t *)&(CXLDVSECDevice){ @@ -570,19 +617,32 @@ static void ct3d_reg_write(void *opaque, hwaddr offset, uint64_t value, } } -/* - * TODO: dc region configuration will be updated once host backend and address - * space support is added for DCD. - */ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) { int i; + MemoryRegion *dc_mr; + uint64_t dc_size = 0; uint64_t region_base = 0; - uint64_t region_len = 2 * GiB; - uint64_t decode_len = 2 * GiB; + uint64_t region_len; + uint64_t decode_len; uint64_t blk_size = 2 * MiB; CXLDCDRegion *region; + dc_mr = host_memory_backend_get_memory(ct3d->dc.host_dc); + dc_size = memory_region_size(dc_mr); + region_len = DIV_ROUND_UP(dc_size, ct3d->dc.num_regions); + + if (region_len * ct3d->dc.num_regions > dc_size) { + error_setg(errp, "host backend size must be multiples of region len"); + return false; + } + if (region_len % CXL_CAPACITY_MULTIPLIER != 0) { + error_setg(errp, "DC region size is unaligned to %lx", + CXL_CAPACITY_MULTIPLIER); + return false; + } + decode_len = region_len; + if (ct3d->hostvmem) { region_base += ct3d->hostvmem->size; } @@ -599,6 +659,7 @@ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) region->flags = 0; region_base += region->len; + ct3d->dc.total_capacity += region->len; } return true; @@ -608,7 +669,8 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) { DeviceState *ds = DEVICE(ct3d); - if (!ct3d->hostmem && !ct3d->hostvmem && !ct3d->hostpmem) { + if (!ct3d->hostmem && !ct3d->hostvmem && !ct3d->hostpmem + && !ct3d->dc.num_regions) { error_setg(errp, "at least one memdev property must be set"); return false; } else if (ct3d->hostmem && ct3d->hostpmem) { @@ -672,9 +734,38 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) g_free(p_name); } - if (!cxl_create_dc_regions(ct3d, errp)) { - error_setg(errp, "setup DC regions failed"); - return false; + ct3d->dc.total_capacity = 0; + if (ct3d->dc.num_regions) { + MemoryRegion *dc_mr; + char *dc_name; + + if (!ct3d->dc.host_dc) { + error_setg(errp, "dynamic capacity must have a backing device"); + return false; + } + + dc_mr = host_memory_backend_get_memory(ct3d->dc.host_dc); + if (!dc_mr) { + error_setg(errp, "dynamic capacity must have a backing device"); + return false; + } + + /* FIXME: set dc as volatile for now */ + memory_region_set_nonvolatile(dc_mr, false); + memory_region_set_enabled(dc_mr, true); + host_memory_backend_set_mapped(ct3d->dc.host_dc, true); + if (ds->id) { + dc_name = g_strdup_printf("cxl-dcd-dpa-dc-space:%s", ds->id); + } else { + dc_name = g_strdup("cxl-dcd-dpa-dc-space"); + } + address_space_init(&ct3d->dc.host_dc_as, dc_mr, dc_name); + g_free(dc_name); + + if (!cxl_create_dc_regions(ct3d, errp)) { + error_setg(errp, "setup DC regions failed"); + return false; + } } return true; @@ -766,6 +857,9 @@ err_release_cdat: err_free_special_ops: g_free(regs->special_ops); err_address_space_free: + if (ct3d->dc.host_dc) { + address_space_destroy(&ct3d->dc.host_dc_as); + } if (ct3d->hostpmem) { address_space_destroy(&ct3d->hostpmem_as); } @@ -784,6 +878,9 @@ static void ct3_exit(PCIDevice *pci_dev) pcie_aer_exit(pci_dev); cxl_doe_cdat_release(cxl_cstate); g_free(regs->special_ops); + if (ct3d->dc.host_dc) { + address_space_destroy(&ct3d->dc.host_dc_as); + } if (ct3d->hostpmem) { address_space_destroy(&ct3d->hostpmem_as); } @@ -862,16 +959,24 @@ static int cxl_type3_hpa_to_as_and_dpa(CXLType3Dev *ct3d, AddressSpace **as, uint64_t *dpa_offset) { - MemoryRegion *vmr = NULL, *pmr = NULL; + MemoryRegion *vmr = NULL, *pmr = NULL, *dc_mr = NULL; + uint64_t vmr_size = 0, pmr_size = 0, dc_size = 0; if (ct3d->hostvmem) { vmr = host_memory_backend_get_memory(ct3d->hostvmem); + vmr_size = memory_region_size(vmr); } if (ct3d->hostpmem) { pmr = host_memory_backend_get_memory(ct3d->hostpmem); + pmr_size = memory_region_size(pmr); + } + if (ct3d->dc.host_dc) { + dc_mr = host_memory_backend_get_memory(ct3d->dc.host_dc); + /* Do we want dc_size to be dc_mr->size or not?? */ + dc_size = ct3d->dc.total_capacity; } - if (!vmr && !pmr) { + if (!vmr && !pmr && !dc_mr) { return -ENODEV; } @@ -879,19 +984,18 @@ static int cxl_type3_hpa_to_as_and_dpa(CXLType3Dev *ct3d, return -EINVAL; } - if (*dpa_offset > ct3d->cxl_dstate.static_mem_size) { + if (*dpa_offset >= vmr_size + pmr_size + dc_size) { return -EINVAL; } - if (vmr) { - if (*dpa_offset < memory_region_size(vmr)) { - *as = &ct3d->hostvmem_as; - } else { - *as = &ct3d->hostpmem_as; - *dpa_offset -= memory_region_size(vmr); - } - } else { + if (*dpa_offset < vmr_size) { + *as = &ct3d->hostvmem_as; + } else if (*dpa_offset < vmr_size + pmr_size) { *as = &ct3d->hostpmem_as; + *dpa_offset -= vmr_size; + } else { + *as = &ct3d->dc.host_dc_as; + *dpa_offset -= (vmr_size + pmr_size); } return 0; @@ -973,6 +1077,8 @@ static Property ct3_props[] = { DEFINE_PROP_UINT64("sn", CXLType3Dev, sn, UI64_NULL), DEFINE_PROP_STRING("cdat", CXLType3Dev, cxl_cstate.cdat.filename), DEFINE_PROP_UINT8("num-dc-regions", CXLType3Dev, dc.num_regions, 0), + DEFINE_PROP_LINK("volatile-dc-memdev", CXLType3Dev, dc.host_dc, + TYPE_MEMORY_BACKEND, HostMemoryBackend *), DEFINE_PROP_END_OF_LIST(), }; @@ -1039,33 +1145,39 @@ static void set_lsa(CXLType3Dev *ct3d, const void *buf, uint64_t size, static bool set_cacheline(CXLType3Dev *ct3d, uint64_t dpa_offset, uint8_t *data) { - MemoryRegion *vmr = NULL, *pmr = NULL; + MemoryRegion *vmr = NULL, *pmr = NULL, *dc_mr = NULL; AddressSpace *as; + uint64_t vmr_size = 0, pmr_size = 0, dc_size = 0; if (ct3d->hostvmem) { vmr = host_memory_backend_get_memory(ct3d->hostvmem); + vmr_size = memory_region_size(vmr); } if (ct3d->hostpmem) { pmr = host_memory_backend_get_memory(ct3d->hostpmem); + pmr_size = memory_region_size(pmr); } + if (ct3d->dc.host_dc) { + dc_mr = host_memory_backend_get_memory(ct3d->dc.host_dc); + dc_size = ct3d->dc.total_capacity; + } - if (!vmr && !pmr) { + if (!vmr && !pmr && !dc_mr) { return false; } - if (dpa_offset + CXL_CACHE_LINE_SIZE > ct3d->cxl_dstate.static_mem_size) { + if (dpa_offset + CXL_CACHE_LINE_SIZE > vmr_size + pmr_size + dc_size) { return false; } - if (vmr) { - if (dpa_offset < memory_region_size(vmr)) { - as = &ct3d->hostvmem_as; - } else { - as = &ct3d->hostpmem_as; - dpa_offset -= memory_region_size(vmr); - } - } else { + if (dpa_offset < vmr_size) { + as = &ct3d->hostvmem_as; + } else if (dpa_offset < vmr_size + pmr_size) { as = &ct3d->hostpmem_as; + dpa_offset -= vmr_size; + } else { + as = &ct3d->dc.host_dc_as; + dpa_offset -= (vmr_size + pmr_size); } address_space_write(as, dpa_offset, MEMTXATTRS_UNSPECIFIED, &data, diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index 80188db670..2f244da9a1 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -467,6 +467,10 @@ struct CXLType3Dev { uint64_t poison_list_overflow_ts; struct dynamic_capacity { + HostMemoryBackend *host_dc; + AddressSpace host_dc_as; + uint64_t total_capacity; /* 256M aligned */ + uint8_t num_regions; /* 0-8 regions */ CXLDCDRegion regions[DCD_MAX_REGION_NUM]; } dc; From patchwork Wed Feb 21 18:16:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: fan X-Patchwork-Id: 13566073 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D8E4185276 for ; Wed, 21 Feb 2024 18:21:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708539681; cv=none; b=s4xWdQ5MOjPi7UGmF93uoGXWLQD9SNG+RpWXYyIr2UT0Ms2wRNBInCixIxOQn22qIdWjkY7v6pfd5yVjC3WfHvqfOHrvsQd+Xf63Ax2MNOWf9TGdOp9Ai5tv5lOxYJ6eU8l56JCpEU+ypDCDQMsLSh2UDMfHSjq9q6iIaunOvRQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708539681; c=relaxed/simple; bh=OUt0GphFQTMnW+jE+Iwc7K36cOd6hhjWqrRpfji621E=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rGvo+6GE2YAXot2hq7rbGWDAzco2IBivY2kW05sQBibfLbVrtm9WJzGZhdRzNZ/Q/eJiMAtdCvRx/1DIwMp9AI/JUXojt2XadJNgLPerrDna4n2MTmPULEAInXTWJwOr5CKvZj5Zv/iXV9cWjMoJXhkKQGltqp8ai8QyPclSGWQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=bP1gE29/; arc=none smtp.client-ip=209.85.216.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="bP1gE29/" Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-29951f5c2e7so2508527a91.1 for ; Wed, 21 Feb 2024 10:21:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708539679; x=1709144479; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=NMsgYchrz+Nvp+KunIns7+nQr9EgC+4zBQ7CzSVSfR4=; b=bP1gE29/bcJk0MCF7g2jr8lhPpscFN8DzUDYjgWApK1uSUYKbcMN8mETxk8rF+4uOH /1njRdG1qo1hzF9ToHjJpebzYpvmTjnjIANR8yysSeLy3273Kaaf9LL6hqP/BQ7l3M6J cVRsbT32R32vg1Crorzh4+b0isQ90DsXJPNuaCqe765Xik+grJX372lW42uyTD1P/ivz zZ6Hhp73oFGBUZ1L39xGZ6MD32v0CY7UBfcri1R4Iu4C4dZVkr/dKFhIE1pjXLpF80FF Mq6tZm93y2PCFeCCo4O6xPDyQq+Aido9iF6Tu0x2jOuYgFSVO6Jjyx5DIZKdM2rxziVB AVTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708539679; x=1709144479; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NMsgYchrz+Nvp+KunIns7+nQr9EgC+4zBQ7CzSVSfR4=; b=W0poPttQ2QNKMlnXmLUSM2RT60mXwND+zTxd2SgTRkEg3mWgi9AoByFSDfmx6MSwrn mnsnRPUNASTGI1c6aedW8IQ3kSeQz3hqFUhx/jwWaZ6MgEnxMwxLKkl9+mF5zulU2XB+ crZ/E4QGP/UwCMVrXZqikMUUQbvUN1HHx8PdSZMOAgjC6ou0SZ7YCpXuAUtIWB+IT2Zo ZDRiKxlKKnidW0OS+MY8Z3CxWSIp/lg8kpyimloV8y0vTbE9byqxSLZGPU4doB4gft8i 6aPSQxoLLstHdb8fEX2cpONOfk+aM1UzD3UQNJs9ElrWlEYUFUemCADByWqFI81mqc9/ LjiQ== X-Forwarded-Encrypted: i=1; AJvYcCU4UasE7FhQUZeW9jGU/9g4rHkwOMUKHNRslhnMPCh00+yc/HxLjz1cRQMprKNOLvfKF6GMMaryXhzN4Av5075/Y+XbRebkdx63 X-Gm-Message-State: AOJu0YxthI6kp5sSisSDedxRx69xaILoxNQ29K4MPTLLVhYfwKfN7jQC yyRdPJTbIRCP19Tcul4SX8lLJztRmg9tKTn1W3oMhv6ylaCRwpI/ X-Google-Smtp-Source: AGHT+IE55Txhzt5qTIZN0Q8RXv9M2omyIKe/vPoOqMifiqePhZSO9K1zIAHiHrBjclNSMl0HCCDkNQ== X-Received: by 2002:a17:90b:4b4c:b0:299:73b3:cf15 with SMTP id mi12-20020a17090b4b4c00b0029973b3cf15mr8589636pjb.12.1708539679129; Wed, 21 Feb 2024 10:21:19 -0800 (PST) Received: from localhost.localdomain ([2601:641:300:14de:5692:7e41:13a2:69a]) by smtp.gmail.com with ESMTPSA id cs16-20020a17090af51000b002992f49922csm9979009pjb.25.2024.02.21.10.21.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 10:21:18 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Fan Ni Subject: [PATCH v4 07/10] hw/mem/cxl_type3: Add DC extent list representative and get DC extent list mailbox support Date: Wed, 21 Feb 2024 10:16:00 -0800 Message-ID: <20240221182020.1086096-8-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240221182020.1086096-1-nifan.cxl@gmail.com> References: <20240221182020.1086096-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Add dynamic capacity extent list representative to the definition of CXLType3Dev and add get DC extent list mailbox command per CXL.spec.3.1:.8.2.9.9.9.2. Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 71 +++++++++++++++++++++++++++++++++++++ hw/mem/cxl_type3.c | 1 + include/hw/cxl/cxl_device.h | 23 ++++++++++++ 3 files changed, 95 insertions(+) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index f95e417683..dae7fe00ed 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -83,6 +83,7 @@ enum { #define CLEAR_POISON 0x2 DCD_CONFIG = 0x48, #define GET_DC_CONFIG 0x0 + #define GET_DYN_CAP_EXT_LIST 0x1 PHYSICAL_SWITCH = 0x51, #define IDENTIFY_SWITCH_DEVICE 0x0 #define GET_PHYSICAL_PORT_STATE 0x1 @@ -1344,6 +1345,73 @@ static CXLRetCode cmd_dcd_get_dyn_cap_config(const struct cxl_cmd *cmd, return CXL_MBOX_SUCCESS; } +/* + * CXL r3.1 section 8.2.9.9.9.2: + * Get Dynamic Capacity Extent List (Opcode 4801h) + */ +static CXLRetCode cmd_dcd_get_dyn_cap_ext_list(const struct cxl_cmd *cmd, + uint8_t *payload_in, + size_t len_in, + uint8_t *payload_out, + size_t *len_out, + CXLCCI *cci) +{ + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + struct get_dyn_cap_ext_list_in_pl { + uint32_t extent_cnt; + uint32_t start_extent_id; + } QEMU_PACKED; + + struct get_dyn_cap_ext_list_out_pl { + uint32_t count; + uint32_t total_extents; + uint32_t generation_num; + uint8_t rsvd[4]; + CXLDCExtentRaw records[]; + } QEMU_PACKED; + + struct get_dyn_cap_ext_list_in_pl *in = (void *)payload_in; + struct get_dyn_cap_ext_list_out_pl *out = (void *)payload_out; + uint16_t record_count = 0, i = 0, record_done = 0; + CXLDCExtentList *extent_list = &ct3d->dc.extents; + CXLDCExtent *ent; + uint16_t out_pl_len; + uint32_t start_extent_id = in->start_extent_id; + + if (start_extent_id > ct3d->dc.total_extent_count) { + return CXL_MBOX_INVALID_INPUT; + } + + record_count = MIN(in->extent_cnt, + ct3d->dc.total_extent_count - start_extent_id); + + out_pl_len = sizeof(*out) + record_count * sizeof(out->records[0]); + assert(out_pl_len <= CXL_MAILBOX_MAX_PAYLOAD_SIZE); + + stl_le_p(&out->count, record_count); + stl_le_p(&out->total_extents, ct3d->dc.total_extent_count); + stl_le_p(&out->generation_num, ct3d->dc.ext_list_gen_seq); + + if (record_count > 0) { + QTAILQ_FOREACH(ent, extent_list, node) { + if (i++ < start_extent_id) { + continue; + } + stq_le_p(&out->records[record_done].start_dpa, ent->start_dpa); + stq_le_p(&out->records[record_done].len, ent->len); + memcpy(&out->records[record_done].tag, ent->tag, 0x10); + stw_le_p(&out->records[record_done].shared_seq, ent->shared_seq); + record_done++; + if (record_done == record_count) { + break; + } + } + } + + *len_out = out_pl_len; + return CXL_MBOX_SUCCESS; +} + #define IMMEDIATE_CONFIG_CHANGE (1 << 1) #define IMMEDIATE_DATA_CHANGE (1 << 2) #define IMMEDIATE_POLICY_CHANGE (1 << 3) @@ -1391,6 +1459,9 @@ static const struct cxl_cmd cxl_cmd_set[256][256] = { static const struct cxl_cmd cxl_cmd_set_dcd[256][256] = { [DCD_CONFIG][GET_DC_CONFIG] = { "DCD_GET_DC_CONFIG", cmd_dcd_get_dyn_cap_config, 2, 0 }, + [DCD_CONFIG][GET_DYN_CAP_EXT_LIST] = { + "DCD_GET_DYNAMIC_CAPACITY_EXTENT_LIST", cmd_dcd_get_dyn_cap_ext_list, + 8, 0 }, }; static const struct cxl_cmd cxl_cmd_set_sw[256][256] = { diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index b966fa4f10..f4edada303 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -661,6 +661,7 @@ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) region_base += region->len; ct3d->dc.total_capacity += region->len; } + QTAILQ_INIT(&ct3d->dc.extents); return true; } diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index 2f244da9a1..12a6fb47a9 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -424,6 +424,25 @@ typedef QLIST_HEAD(, CXLPoison) CXLPoisonList; #define DCD_MAX_REGION_NUM 8 +typedef struct CXLDCExtentRaw { + uint64_t start_dpa; + uint64_t len; + uint8_t tag[0x10]; + uint16_t shared_seq; + uint8_t rsvd[0x6]; +} QEMU_PACKED CXLDCExtentRaw; + +typedef struct CXLDCExtent { + uint64_t start_dpa; + uint64_t len; + uint8_t tag[0x10]; + uint16_t shared_seq; + uint8_t rsvd[0x6]; + + QTAILQ_ENTRY(CXLDCExtent) node; +} CXLDCExtent; +typedef QTAILQ_HEAD(, CXLDCExtent) CXLDCExtentList; + typedef struct CXLDCDRegion { uint64_t base; uint64_t decode_len; /* aligned to 256*MiB */ @@ -470,6 +489,10 @@ struct CXLType3Dev { HostMemoryBackend *host_dc; AddressSpace host_dc_as; uint64_t total_capacity; /* 256M aligned */ + CXLDCExtentList extents; + + uint32_t total_extent_count; + uint32_t ext_list_gen_seq; uint8_t num_regions; /* 0-8 regions */ CXLDCDRegion regions[DCD_MAX_REGION_NUM]; From patchwork Wed Feb 21 18:16:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: fan X-Patchwork-Id: 13566074 Received: from mail-pg1-f170.google.com (mail-pg1-f170.google.com [209.85.215.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3A38585283 for ; Wed, 21 Feb 2024 18:21:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708539682; cv=none; b=L87Qz4Eif9bKQfY+amTS72Q9VfSNytDEWdeKDHQH0DVw9BvIR5Ejw8d59sLxvyIOYk+wQwLjkwFanEWdDj3tZ1XQ0kovZoJm9ko+KcHYMWIPhB9CPcGaM4+wzNKNCQyK8kvhhxgjtqRZecF9lnbpcRWaf1Uivk69YvK2VP2Fr1g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708539682; c=relaxed/simple; bh=VQTaqIV4a3sRbhlW6ABD/sK71o0UgSxK+AhEpeqAvsc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dL6NCtPpSQXVx0eV3hI5s/oR0qqNHJzonoRi/KdyKCr7qmgUq4UhcV4ey05X+CANnWDIu08twSmjbL2yMQA1ku42qzoGPaJozmNXsXU1hzxMieo/balYQrzF2aDDqMmM64KwWdioEx3LP9UD2+m+mQBv1WTNX0qTYVV26QZ+UAA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Nl8SlmJd; arc=none smtp.client-ip=209.85.215.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Nl8SlmJd" Received: by mail-pg1-f170.google.com with SMTP id 41be03b00d2f7-5dca1efad59so6417089a12.2 for ; Wed, 21 Feb 2024 10:21:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708539680; x=1709144480; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=g6VL3c1gI2rozkVMagqzxRAFNNaT8HWGIMmWFgrUxvc=; b=Nl8SlmJdGSiHCJWdNxsnJufm/hvPQpscZXbTZgPd/VcdaTinhb1a3W6gP5EZNLrmhT M3I2vO9nXJw9eDACzwKI7+ofnu1IB0U1a8rRO5VV/0P9gxFhGDrR7qtNmU2SbGDUnhuo Nipj7ns2xH+TGJZLxgMUhU+QVRpuuo1eEOuhfX2Q7u38YunQKDvcjvV3b/aCgBmT9Uht Pm2Zu8R+mdd18NXm8CVl9FzS5mbFydDS/h/zJ4s9FoYsKMcg0tqpyjrWoagKdjFcVgiW GcXLX3MuRzYXFfZoi86ALQGEtYW1g5OViQonAJ3PZqKOL3GO1pArJbFGyouZU6Nf8Bfm SEcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708539680; x=1709144480; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=g6VL3c1gI2rozkVMagqzxRAFNNaT8HWGIMmWFgrUxvc=; b=X7EWcg7y81d12MwQx0rxDM4KXjYlAgeTq503Tm/Wqzqu3sJkUr/C5l1iSeZZn+t4EI k3rNzoDpHkcudNOUDsgF5v/IjBb1mtlDC6YrwSOxZwR/0Hgk5EOwFdAzrX3ohlrcN3MD gpn4Ni8wL3+XQUDW5i5zkg4EEpIBM4bFhjX21UVhU1oHGRvW4jZRwJWR/pDogv0sIxaz XWt9gaUy7jyMoRPhXyuagnR52utkJf2R/O9x5g+GBMgnlgcHaqwy+1/T8eBdtJbnChDu TufYC8d2k+Q1eCExMOtL1ALAc3nnsL1SpHmGbaEtPg/kF6Dqu8jj01X7Urzg0iH/bgDs Fv2g== X-Forwarded-Encrypted: i=1; AJvYcCUQzJvzEJWayKFxU5W9EuXE217VMsZ6Tr/E1g8lDRyLjphZbADlo4ZtLxODYoaaBb0+DC+p04xdjB4TqnLha2pokxaUpFWwndgd X-Gm-Message-State: AOJu0YyUI9IgNqorlYG7JVb6mIOTURyjcV0WWD5GRe3DBU63w11w0end TSmTZS0d2JmtCbJQ6mPzc1zrk089aVEOnwLEft+eapsAuXsgnzgG X-Google-Smtp-Source: AGHT+IFBGjAFI6oKAXvbH/bbeOvJ8OH4HKQWlcBjiXs+PMXpY/3xEh0k13+5pB5wI3icc4F5ofUS2A== X-Received: by 2002:a17:90a:c706:b0:297:4f1:8a19 with SMTP id o6-20020a17090ac70600b0029704f18a19mr17216669pjt.23.1708539680417; Wed, 21 Feb 2024 10:21:20 -0800 (PST) Received: from localhost.localdomain ([2601:641:300:14de:5692:7e41:13a2:69a]) by smtp.gmail.com with ESMTPSA id cs16-20020a17090af51000b002992f49922csm9979009pjb.25.2024.02.21.10.21.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 10:21:20 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Fan Ni Subject: [PATCH v4 08/10] hw/cxl/cxl-mailbox-utils: Add mailbox commands to support add/release dynamic capacity response Date: Wed, 21 Feb 2024 10:16:01 -0800 Message-ID: <20240221182020.1086096-9-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240221182020.1086096-1-nifan.cxl@gmail.com> References: <20240221182020.1086096-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Per CXL spec 3.1, two mailbox commands are implemented: Add Dynamic Capacity Response (Opcode 4802h) 8.2.9.9.9.3, and Release Dynamic Capacity (Opcode 4803h) 8.2.9.9.9.4. Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 288 ++++++++++++++++++++++++++++++++++++ include/hw/cxl/cxl_device.h | 2 + 2 files changed, 290 insertions(+) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index dae7fe00ed..65ed28f700 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -84,6 +84,8 @@ enum { DCD_CONFIG = 0x48, #define GET_DC_CONFIG 0x0 #define GET_DYN_CAP_EXT_LIST 0x1 + #define ADD_DYN_CAP_RSP 0x2 + #define RELEASE_DYN_CAP 0x3 PHYSICAL_SWITCH = 0x51, #define IDENTIFY_SWITCH_DEVICE 0x0 #define GET_PHYSICAL_PORT_STATE 0x1 @@ -1412,6 +1414,286 @@ static CXLRetCode cmd_dcd_get_dyn_cap_ext_list(const struct cxl_cmd *cmd, return CXL_MBOX_SUCCESS; } +/* + * Check whether any bit between addr[nr, nr+size) is set, + * return true if any bit is set, otherwise return false + */ +static bool test_any_bits_set(const unsigned long *addr, int nr, int size) +{ + unsigned long res = find_next_bit(addr, size + nr, nr); + + return res < nr + size; +} + +CXLDCDRegion *cxl_find_dc_region(CXLType3Dev *ct3d, uint64_t dpa, uint64_t len) +{ + int i; + CXLDCDRegion *region = &ct3d->dc.regions[0]; + + if (dpa < region->base || + dpa >= region->base + ct3d->dc.total_capacity) { + return NULL; + } + + /* + * CXL r3.1 section 9.13.3: Dynamic Capacity Device (DCD) + * + * Regions are used in increasing-DPA order, with Region 0 being used for + * the lowest DPA of Dynamic Capacity and Region 7 for the highest DPA. + * So check from the last region to find where the dpa belongs. Extents that + * cross multiple regions are not allowed. + */ + for (i = ct3d->dc.num_regions - 1; i >= 0; i--) { + region = &ct3d->dc.regions[i]; + if (dpa >= region->base) { + if (dpa + len > region->base + region->len) { + return NULL; + } + return region; + } + } + + return NULL; +} + +static void cxl_insert_extent_to_extent_list(CXLDCExtentList *list, + uint64_t dpa, + uint64_t len, + uint8_t *tag, + uint16_t shared_seq) +{ + CXLDCExtent *extent; + + extent = g_new0(CXLDCExtent, 1); + extent->start_dpa = dpa; + extent->len = len; + if (tag) { + memcpy(extent->tag, tag, 0x10); + } + extent->shared_seq = shared_seq; + + QTAILQ_INSERT_TAIL(list, extent, node); +} + +static void cxl_remove_extent_from_extent_list(CXLDCExtentList *list, + CXLDCExtent *extent) +{ + QTAILQ_REMOVE(list, extent, node); + g_free(extent); +} + +/* + * CXL r3.1 Table 8-168: Add Dynamic Capacity Response Input Payload + * CXL r3.1 Table 8-170: Release Dynamic Capacity Input Payload + */ +typedef struct CXLUpdateDCExtentListInPl { + uint32_t num_entries_updated; + uint8_t flags; + uint8_t rsvd[3]; + /* CXL r3.1 Table 8-169: Updated Extent List */ + struct { + uint64_t start_dpa; + uint64_t len; + uint8_t rsvd[8]; + } QEMU_PACKED updated_entries[]; +} QEMU_PACKED CXLUpdateDCExtentListInPl; + +/* + * For the extents in the extent list to operate, check whether they are valid + * 1. The extent should be in the range of a valid DC region; + * 2. The extent should not cross multiple regions; + * 3. The start DPA and the length of the extent should align with the block + * size of the region; + * 4. The address range of multiple extents in the list should not overlap. + */ +static CXLRetCode cxl_detect_malformed_extent_list(CXLType3Dev *ct3d, + const CXLUpdateDCExtentListInPl *in) +{ + uint64_t min_block_size = UINT64_MAX; + CXLDCDRegion *region = &ct3d->dc.regions[0]; + CXLDCDRegion *lastregion = &ct3d->dc.regions[ct3d->dc.num_regions - 1]; + g_autofree unsigned long *blk_bitmap = NULL; + uint64_t dpa, len; + uint32_t i; + + for (i = 0; i < ct3d->dc.num_regions; i++) { + region = &ct3d->dc.regions[i]; + min_block_size = MIN(min_block_size, region->block_size); + } + + blk_bitmap = bitmap_new((lastregion->base + lastregion->len - + ct3d->dc.regions[0].base) / min_block_size); + + for (i = 0; i < in->num_entries_updated; i++) { + dpa = in->updated_entries[i].start_dpa; + len = in->updated_entries[i].len; + + region = cxl_find_dc_region(ct3d, dpa, len); + if (!region) { + return CXL_MBOX_INVALID_PA; + } + + dpa -= ct3d->dc.regions[0].base; + if (dpa % region->block_size || len % region->block_size) { + return CXL_MBOX_INVALID_EXTENT_LIST; + } + /* the dpa range already covered by some other extents in the list */ + if (test_any_bits_set(blk_bitmap, dpa / min_block_size, + len / min_block_size)) { + return CXL_MBOX_INVALID_EXTENT_LIST; + } + bitmap_set(blk_bitmap, dpa / min_block_size, len / min_block_size); + } + + return CXL_MBOX_SUCCESS; +} + +/* + * CXL r3.1 section 8.2.9.9.9.3: Add Dynamic Capacity Response (opcode 4802h) + * An extent is added to the extent list and becomes usable only after the + * response is processed successfully + */ +static CXLRetCode cmd_dcd_add_dyn_cap_rsp(const struct cxl_cmd *cmd, + uint8_t *payload_in, + size_t len_in, + uint8_t *payload_out, + size_t *len_out, + CXLCCI *cci) +{ + CXLUpdateDCExtentListInPl *in = (void *)payload_in; + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + CXLDCExtentList *extent_list = &ct3d->dc.extents; + CXLDCExtent *ent; + uint32_t i; + uint64_t dpa, len; + CXLRetCode ret; + + if (in->num_entries_updated == 0) { + return CXL_MBOX_SUCCESS; + } + + ret = cxl_detect_malformed_extent_list(ct3d, in); + if (ret != CXL_MBOX_SUCCESS) { + return ret; + } + + for (i = 0; i < in->num_entries_updated; i++) { + dpa = in->updated_entries[i].start_dpa; + len = in->updated_entries[i].len; + + /* + * Check if the DPA range of the to-be-added extent overlaps with + * existing extent list maintained by the device. + */ + QTAILQ_FOREACH(ent, extent_list, node) { + if (ent->start_dpa <= dpa && + dpa + len <= ent->start_dpa + ent->len) { + return CXL_MBOX_INVALID_PA; + /* Overlapping one end of the other */ + } else if ((dpa < ent->start_dpa + ent->len && + dpa + len > ent->start_dpa + ent->len) || + (dpa < ent->start_dpa && dpa + len > ent->start_dpa)) { + return CXL_MBOX_INVALID_PA; + } + } + + /* + * TODO: we will add a pending extent list based on event log record + * and verify the input response; also, the "More" flag is not + * considered at the moment. + */ + + cxl_insert_extent_to_extent_list(extent_list, dpa, len, NULL, 0); + ct3d->dc.total_extent_count += 1; + } + + return CXL_MBOX_SUCCESS; +} + +/* + * CXL r3.1 section 8.2.9.9.9.4: Release Dynamic Capacity (opcode 4803h) + */ +static CXLRetCode cmd_dcd_release_dyn_cap(const struct cxl_cmd *cmd, + uint8_t *payload_in, + size_t len_in, + uint8_t *payload_out, + size_t *len_out, + CXLCCI *cci) +{ + CXLUpdateDCExtentListInPl *in = (void *)payload_in; + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + CXLDCExtentList *extent_list = &ct3d->dc.extents; + CXLDCExtent *ent; + uint32_t i; + uint64_t dpa, len; + CXLRetCode ret; + + if (in->num_entries_updated == 0) { + return CXL_MBOX_INVALID_INPUT; + } + + ret = cxl_detect_malformed_extent_list(ct3d, in); + if (ret != CXL_MBOX_SUCCESS) { + return ret; + } + + for (i = 0; i < in->num_entries_updated; i++) { + bool found = false; + + dpa = in->updated_entries[i].start_dpa; + len = in->updated_entries[i].len; + + QTAILQ_FOREACH(ent, extent_list, node) { + if (ent->start_dpa <= dpa && + dpa + len <= ent->start_dpa + ent->len) { + /* + * If an incoming extent covers a portion of an extent + * in the device extent list, remove only the overlapping + * portion, meaning + * 1. the portions that are not covered by the incoming + * extent at both end of the original extent will become + * new extents and inserted to the extent list; and + * 2. the original extent is removed from the extent list; + * 3. dc extent count is updated accordingly. + */ + uint64_t ent_start_dpa = ent->start_dpa; + uint64_t ent_len = ent->len; + uint64_t len1 = dpa - ent_start_dpa; + uint64_t len2 = ent_start_dpa + ent_len - dpa - len; + + found = true; + cxl_remove_extent_from_extent_list(extent_list, ent); + ct3d->dc.total_extent_count -= 1; + + if (len1) { + cxl_insert_extent_to_extent_list(extent_list, + ent_start_dpa, len1, + NULL, 0); + ct3d->dc.total_extent_count += 1; + } + if (len2) { + cxl_insert_extent_to_extent_list(extent_list, dpa + len, + len2, NULL, 0); + ct3d->dc.total_extent_count += 1; + } + break; + /*Currently we reject the attempt to remove a superset*/ + } else if ((dpa < ent->start_dpa + ent->len && + dpa + len > ent->start_dpa + ent->len) || + (dpa < ent->start_dpa && dpa + len > ent->start_dpa)) { + return CXL_MBOX_INVALID_EXTENT_LIST; + } + } + + if (!found) { + /* Try to remove a non-existing extent */ + return CXL_MBOX_INVALID_PA; + } + } + + return CXL_MBOX_SUCCESS; +} + #define IMMEDIATE_CONFIG_CHANGE (1 << 1) #define IMMEDIATE_DATA_CHANGE (1 << 2) #define IMMEDIATE_POLICY_CHANGE (1 << 3) @@ -1462,6 +1744,12 @@ static const struct cxl_cmd cxl_cmd_set_dcd[256][256] = { [DCD_CONFIG][GET_DYN_CAP_EXT_LIST] = { "DCD_GET_DYNAMIC_CAPACITY_EXTENT_LIST", cmd_dcd_get_dyn_cap_ext_list, 8, 0 }, + [DCD_CONFIG][ADD_DYN_CAP_RSP] = { + "ADD_DCD_DYNAMIC_CAPACITY_RESPONSE", cmd_dcd_add_dyn_cap_rsp, + ~0, IMMEDIATE_DATA_CHANGE }, + [DCD_CONFIG][RELEASE_DYN_CAP] = { + "RELEASE_DCD_DYNAMIC_CAPACITY", cmd_dcd_release_dyn_cap, + ~0, IMMEDIATE_DATA_CHANGE }, }; static const struct cxl_cmd cxl_cmd_set_sw[256][256] = { diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index 12a6fb47a9..6178416cbb 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -548,4 +548,6 @@ void cxl_event_irq_assert(CXLType3Dev *ct3d); void cxl_set_poison_list_overflowed(CXLType3Dev *ct3d); +CXLDCDRegion *cxl_find_dc_region(CXLType3Dev *ct3d, uint64_t dpa, uint64_t len); + #endif From patchwork Wed Feb 21 18:16:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: fan X-Patchwork-Id: 13566075 Received: from mail-pg1-f180.google.com (mail-pg1-f180.google.com [209.85.215.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 521DF84FC5 for ; Wed, 21 Feb 2024 18:21:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708539684; cv=none; b=HSjxYaJ4vSUKfqe6ioiNQN+ZaaID1d/u7jqBUVF1aD8PUh2C23NaNeqeeZNhtK647kVQfQbKJk8z+GZQAKABi9dUvt13wOgW1XKqbH+NpS7xFxutPlMAuaIrMm92b/pFTnL9RNt3Chmj6PmWlMRr0OOtaxq7Z+yyHJLaOEyxHZ0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708539684; c=relaxed/simple; bh=Tgne5LFuZ4lvdJN/mPy32qQSykCqIGV3OywGEBk+nW4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TbeVU61/HNFJIS4e/GTQwnecUdlt784L0Fa4b1kM87szTlcgif/8av2770xRuLBMtxEqVHz3poDR566LPvz5hpQRNQVhh0BbryBlyPj65LV0CIBgTwkHVY1yviDmoVTyor0Sl0Mi7h9LiwqszAj0l1OwQ40h0afal+pqjo+q34A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Xc4dFbTV; arc=none smtp.client-ip=209.85.215.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Xc4dFbTV" Received: by mail-pg1-f180.google.com with SMTP id 41be03b00d2f7-5d8b70b39efso5310148a12.0 for ; Wed, 21 Feb 2024 10:21:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708539682; x=1709144482; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qYU6nCebrZao7rRGfPAJpKxrYnxAjdViLCf8sgh5PNA=; b=Xc4dFbTVzQlz4QVTkgmc4VysbxggS9CaPoHqg3iT/lzTw5djdKGLI1gf5/LLjz9HiJ tTgofdk+Mfl8mxcBzbtr5lysEstqGDjqR7VFF6r3XvzEGTAXIiKrA4pvI2POOoGCwAJd qHLEWZkVU/S+j3Cbtv8zoXJS5hVoAEoBW1fv9+MtnIwCZJ6KwjZxBe9zFwVoGm60+a+Y hS0P2/dqNa4bJchETfNUasKsX3NBQ4eemMa72uVyhNXS3chWHUPto6Msvu/FPT1bUa2l vzkzJ1g48kXE000fUPLQhy8uLvpPf/0hoLsesuUyaxn2hOaVaoTgLm4DN0CBJsWsyCqY glAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708539682; x=1709144482; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qYU6nCebrZao7rRGfPAJpKxrYnxAjdViLCf8sgh5PNA=; b=NlCLDcsZ9Uy9Ppph9GJ0k1TuvCTAgwHJNW0LizI9AeWWTLP1j/h5TzaOZmahAkanqJ s+GKHAId0DDu3COD1rXWlRGhe2VgGssEok509ku3vnGnKjUtzeN9kj7UvJBbt6i1Bxnp DndWgGEWGtJHrf7x/BYBRlDcjf+dZGQCWKgxmeQqbUp4/mCo/FuJ1tu9z72oNDY2udIK baQu8z0R+3EiKAHC4S/+99qTiSd7yJgX+VDbcaRVqQ4a/ZTYxXnIb8vw3VcDWGSXQIol dSSm7/UObRR7IHaKMhSxAefqDoQO9UFUj9yLJcoYRYMBznck1QvGJ3cjHs/3oXdN3sfP TnaQ== X-Forwarded-Encrypted: i=1; AJvYcCWf+6zot9csgsvgpS/tfvNpgpInchpH2ZdKZjxN/pAc8r6KlMuyeIeseufLhfgHX3XlCZsJLX3oAaV3/4MSOn4GiXYxFyE/xmle X-Gm-Message-State: AOJu0Yz/1cBpaUctW6ksb3koaWk1DZ/EAis1JEnk1Urgz3ARqqHXPZEB naM7e3jtByMcl8//VNWRcvZXKOGn1p9BrGEMT/tP9Zvcu59GdocA X-Google-Smtp-Source: AGHT+IHuQgqqx8QbVW6rJdY8A6OTYD9jWSvGOrewfmjtfSvKaf7JXBy3z7LPpc/SD/xl/3TLNN9UDQ== X-Received: by 2002:a17:90a:bb15:b0:299:dd2:47f with SMTP id u21-20020a17090abb1500b002990dd2047fmr14394930pjr.10.1708539681710; Wed, 21 Feb 2024 10:21:21 -0800 (PST) Received: from localhost.localdomain ([2601:641:300:14de:5692:7e41:13a2:69a]) by smtp.gmail.com with ESMTPSA id cs16-20020a17090af51000b002992f49922csm9979009pjb.25.2024.02.21.10.21.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 10:21:21 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Fan Ni Subject: [PATCH v4 09/10] hw/cxl/events: Add qmp interfaces to add/release dynamic capacity extents Date: Wed, 21 Feb 2024 10:16:02 -0800 Message-ID: <20240221182020.1086096-10-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240221182020.1086096-1-nifan.cxl@gmail.com> References: <20240221182020.1086096-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Since fabric manager emulation is not supported yet, the change implements the functions to add/release dynamic capacity extents as QMP interfaces. Note: we skips any FM issued extent release request if the exact extent does not exist in the extent list of the device. We will loose the restriction later once we have partial release support in the kernel. 1. Add dynamic capacity extents: For example, the command to add two continuous extents (each 128MiB long) to region 0 (starting at DPA offset 0) looks like below: { "execute": "qmp_capabilities" } { "execute": "cxl-add-dynamic-capacity", "arguments": { "path": "/machine/peripheral/cxl-dcd0", "region-id": 0, "extents": [ { "dpa": 0, "len": 134217728 }, { "dpa": 134217728, "len": 134217728 } ] } } 2. Release dynamic capacity extents: For example, the command to release an extent of size 128MiB from region 0 (DPA offset 128MiB) look like below: { "execute": "cxl-release-dynamic-capacity", "arguments": { "path": "/machine/peripheral/cxl-dcd0", "region-id": 0, "extents": [ { "dpa": 134217728, "len": 134217728 } ] } } Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 26 +++- hw/mem/cxl_type3.c | 249 +++++++++++++++++++++++++++++++++++- hw/mem/cxl_type3_stubs.c | 14 ++ include/hw/cxl/cxl_device.h | 7 + include/hw/cxl/cxl_events.h | 18 +++ qapi/cxl.json | 61 ++++++++- 6 files changed, 366 insertions(+), 9 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 65ed28f700..34c4ebbd12 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -1418,7 +1418,7 @@ static CXLRetCode cmd_dcd_get_dyn_cap_ext_list(const struct cxl_cmd *cmd, * Check whether any bit between addr[nr, nr+size) is set, * return true if any bit is set, otherwise return false */ -static bool test_any_bits_set(const unsigned long *addr, int nr, int size) +bool test_any_bits_set(const unsigned long *addr, int nr, int size) { unsigned long res = find_next_bit(addr, size + nr, nr); @@ -1456,7 +1456,7 @@ CXLDCDRegion *cxl_find_dc_region(CXLType3Dev *ct3d, uint64_t dpa, uint64_t len) return NULL; } -static void cxl_insert_extent_to_extent_list(CXLDCExtentList *list, +void cxl_insert_extent_to_extent_list(CXLDCExtentList *list, uint64_t dpa, uint64_t len, uint8_t *tag, @@ -1597,16 +1597,28 @@ static CXLRetCode cmd_dcd_add_dyn_cap_rsp(const struct cxl_cmd *cmd, } } - /* - * TODO: we will add a pending extent list based on event log record - * and verify the input response; also, the "More" flag is not - * considered at the moment. - */ + QTAILQ_FOREACH(ent, &ct3d->dc.extents_pending_to_add, node) { + if (ent->start_dpa <= dpa && + dpa + len <= ent->start_dpa + ent->len) { + break; + } + } + if (!ent) { + return CXL_MBOX_INVALID_PA; + } + + cxl_remove_extent_from_extent_list(&ct3d->dc.extents_pending_to_add, + ent); cxl_insert_extent_to_extent_list(extent_list, dpa, len, NULL, 0); ct3d->dc.total_extent_count += 1; } + /* + * TODO: extents_pending_to_add needs to be cleared so the extents not + * accepted can be reclaimed base on spec r3.1: 8.2.9.9.9.3 + */ + return CXL_MBOX_SUCCESS; } diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index f4edada303..b8c4273e99 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -662,6 +662,7 @@ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) ct3d->dc.total_capacity += region->len; } QTAILQ_INIT(&ct3d->dc.extents); + QTAILQ_INIT(&ct3d->dc.extents_pending_to_add); return true; } @@ -1424,7 +1425,8 @@ static int ct3d_qmp_cxl_event_log_enc(CxlEventLog log) return CXL_EVENT_TYPE_FAIL; case CXL_EVENT_LOG_FATAL: return CXL_EVENT_TYPE_FATAL; -/* DCD not yet supported */ + case CXL_EVENT_LOG_DYNCAP: + return CXL_EVENT_TYPE_DYNAMIC_CAP; default: return -EINVAL; } @@ -1675,6 +1677,251 @@ void qmp_cxl_inject_memory_module_event(const char *path, CxlEventLog log, } } +/* CXL r3.1 Table 8-50: Dynanic Capacity Event Record */ +static const QemuUUID dynamic_capacity_uuid = { + .data = UUID(0xca95afa7, 0xf183, 0x4018, 0x8c, 0x2f, + 0x95, 0x26, 0x8e, 0x10, 0x1a, 0x2a), +}; + +typedef enum CXLDCEventType { + DC_EVENT_ADD_CAPACITY = 0x0, + DC_EVENT_RELEASE_CAPACITY = 0x1, + DC_EVENT_FORCED_RELEASE_CAPACITY = 0x2, + DC_EVENT_REGION_CONFIG_UPDATED = 0x3, + DC_EVENT_ADD_CAPACITY_RSP = 0x4, + DC_EVENT_CAPACITY_RELEASED = 0x5, +} CXLDCEventType; + +/* + * Check whether the exact extent exists in the list + * Return value: the extent pointer in the list; else null + */ +static CXLDCExtent *cxl_dc_extent_exists(CXLDCExtentList *list, + CXLDCExtentRaw *ext) +{ + CXLDCExtent *ent; + + if (!ext || !list) { + return NULL; + } + + QTAILQ_FOREACH(ent, list, node) { + if (ent->start_dpa != ext->start_dpa) { + continue; + } + + /*Found exact extent*/ + return ent->len == ext->len ? ent : NULL; + } + + return NULL; +} + +/* + * The main function to process dynamic capacity event. Currently DC extents + * add/release requests are processed. + */ +static void qmp_cxl_process_dynamic_capacity(const char *path, CxlEventLog log, + CXLDCEventType type, uint16_t hid, + uint8_t rid, + CXLDCExtentRecordList *records, + Error **errp) +{ + Object *obj; + CXLEventDynamicCapacity dCap = {}; + CXLEventRecordHdr *hdr = &dCap.hdr; + CXLType3Dev *dcd; + uint8_t flags = 1 << CXL_EVENT_TYPE_INFO; + uint32_t num_extents = 0; + CXLDCExtentRecordList *list; + g_autofree CXLDCExtentRaw *extents = NULL; + uint8_t enc_log; + uint64_t offset, len, block_size; + int i; + int rc; + g_autofree unsigned long *blk_bitmap = NULL; + + obj = object_resolve_path(path, NULL); + if (!obj) { + error_setg(errp, "Unable to resolve path"); + return; + } + if (!object_dynamic_cast(obj, TYPE_CXL_TYPE3)) { + error_setg(errp, "Path not point to a valid CXL type3 device"); + return; + } + + dcd = CXL_TYPE3(obj); + if (!dcd->dc.num_regions) { + error_setg(errp, "No dynamic capacity support from the device"); + return; + } + + rc = ct3d_qmp_cxl_event_log_enc(log); + if (rc < 0) { + error_setg(errp, "Unhandled error log type"); + return; + } + enc_log = rc; + + if (rid >= dcd->dc.num_regions) { + error_setg(errp, "region id is too large"); + return; + } + block_size = dcd->dc.regions[rid].block_size; + + /* Sanity check and count the extents */ + list = records; + while (list) { + offset = list->value->offset; + len = list->value->len; + + if (len == 0) { + error_setg(errp, "extent with 0 length is not allowed"); + return; + } + + if (offset % block_size || len % block_size) { + error_setg(errp, "dpa or len is not aligned to region block size"); + return; + } + + if (offset + len > dcd->dc.regions[rid].len) { + error_setg(errp, "extent range is beyond the region end"); + return; + } + + num_extents++; + list = list->next; + } + if (num_extents == 0) { + error_setg(errp, "No extents found in the command"); + return; + } + + blk_bitmap = bitmap_new(dcd->dc.regions[rid].len / block_size); + + /* Create Extent list for event being passed to host */ + i = 0; + list = records; + extents = g_new0(CXLDCExtentRaw, num_extents); + while (list) { + CXLDCExtent *ent; + bool skip_extent = false; + + offset = list->value->offset; + len = list->value->len; + + extents[i].start_dpa = offset + dcd->dc.regions[rid].base; + extents[i].len = len; + memset(extents[i].tag, 0, 0x10); + extents[i].shared_seq = 0; + + if (type == DC_EVENT_RELEASE_CAPACITY || + type == DC_EVENT_FORCED_RELEASE_CAPACITY) { + /* + * if the extent is still pending to be added to the host, + * remove it from the pending extent list, so later when the add + * response for the extent arrives, the device can reject the + * extent as it is not in the pending list. + */ + ent = cxl_dc_extent_exists(&dcd->dc.extents_pending_to_add, + &extents[i]); + if (ent) { + QTAILQ_REMOVE(&dcd->dc.extents_pending_to_add, ent, node); + g_free(ent); + skip_extent = true; + } else if (!cxl_dc_extent_exists(&dcd->dc.extents, &extents[i])) { + /* If the exact extent is not in the accepted list, skip */ + skip_extent = true; + } + } + + /* No duplicate or overlapped extents are allowed */ + if (test_any_bits_set(blk_bitmap, offset / block_size, + len / block_size)) { + error_setg(errp, "duplicate or overlapped extents are detected"); + return; + } + bitmap_set(blk_bitmap, offset / block_size, len / block_size); + + list = list->next; + if (!skip_extent) { + i++; + } + } + num_extents = i; + + switch (type) { + case DC_EVENT_ADD_CAPACITY: + break; + default: + break; + } + /* + * CXL r3.1 section 8.2.9.2.1.6: Dynamic Capacity Event Record + * + * All Dynamic Capacity event records shall set the Event Record Severity + * field in the Common Event Record Format to Informational Event. All + * Dynamic Capacity related events shall be logged in the Dynamic Capacity + * Event Log. + */ + cxl_assign_event_header(hdr, &dynamic_capacity_uuid, flags, sizeof(dCap), + cxl_device_get_timestamp(&dcd->cxl_dstate)); + + dCap.type = type; + /* FIXME: for now, validaity flag is cleared */ + dCap.validity_flags = 0; + stw_le_p(&dCap.host_id, hid); + /* only valid for DC_REGION_CONFIG_UPDATED event */ + dCap.updated_region_id = 0; + /* + * FIXME: for now, "More" flag is cleared as there is only one extent for + * each record + */ + dCap.flags = 0; + + /* + * For current implementation, each DC event record only associates with + * one extent, so the "More" flag does not need to be set. + */ + for (i = 0; i < num_extents; i++) { + memcpy(&dCap.dynamic_capacity_extent, &extents[i], + sizeof(CXLDCExtentRaw)); + + if (type == DC_EVENT_ADD_CAPACITY) { + cxl_insert_extent_to_extent_list(&dcd->dc.extents_pending_to_add, + extents[i].start_dpa, + extents[i].len, + extents[i].tag, + extents[i].shared_seq); + } + + if (cxl_event_insert(&dcd->cxl_dstate, enc_log, + (CXLEventRecordRaw *)&dCap)) { + cxl_event_irq_assert(dcd); + } + } +} + +void qmp_cxl_add_dynamic_capacity(const char *path, uint8_t region_id, + CXLDCExtentRecordList *records, + Error **errp) +{ + qmp_cxl_process_dynamic_capacity(path, CXL_EVENT_LOG_DYNCAP, + DC_EVENT_ADD_CAPACITY, 0, + region_id, records, errp); +} + +void qmp_cxl_release_dynamic_capacity(const char *path, uint8_t region_id, + CXLDCExtentRecordList *records, + Error **errp) +{ + qmp_cxl_process_dynamic_capacity(path, CXL_EVENT_LOG_DYNCAP, + DC_EVENT_RELEASE_CAPACITY, 0, + region_id, records, errp); +} + static void ct3_class_init(ObjectClass *oc, void *data) { DeviceClass *dc = DEVICE_CLASS(oc); diff --git a/hw/mem/cxl_type3_stubs.c b/hw/mem/cxl_type3_stubs.c index 3e1851e32b..d913b11b4d 100644 --- a/hw/mem/cxl_type3_stubs.c +++ b/hw/mem/cxl_type3_stubs.c @@ -67,3 +67,17 @@ void qmp_cxl_inject_correctable_error(const char *path, CxlCorErrorType type, { error_setg(errp, "CXL Type 3 support is not compiled in"); } + +void qmp_cxl_add_dynamic_capacity(const char *path, uint8_t region_id, + CXLDCExtentRecordList *records, + Error **errp) +{ + error_setg(errp, "CXL Type 3 support is not compiled in"); +} + +void qmp_cxl_release_dynamic_capacity(const char *path, uint8_t region_id, + CXLDCExtentRecordList *records, + Error **errp) +{ + error_setg(errp, "CXL Type 3 support is not compiled in"); +} diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index 6178416cbb..1d31164bd3 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -490,6 +490,7 @@ struct CXLType3Dev { AddressSpace host_dc_as; uint64_t total_capacity; /* 256M aligned */ CXLDCExtentList extents; + CXLDCExtentList extents_pending_to_add; uint32_t total_extent_count; uint32_t ext_list_gen_seq; @@ -550,4 +551,10 @@ void cxl_set_poison_list_overflowed(CXLType3Dev *ct3d); CXLDCDRegion *cxl_find_dc_region(CXLType3Dev *ct3d, uint64_t dpa, uint64_t len); +void cxl_insert_extent_to_extent_list(CXLDCExtentList *list, + uint64_t dpa, + uint64_t len, + uint8_t *tag, + uint16_t shared_seq); +bool test_any_bits_set(const unsigned long *addr, int nr, int size); #endif diff --git a/include/hw/cxl/cxl_events.h b/include/hw/cxl/cxl_events.h index 5170b8dbf8..38cadaa0f3 100644 --- a/include/hw/cxl/cxl_events.h +++ b/include/hw/cxl/cxl_events.h @@ -166,4 +166,22 @@ typedef struct CXLEventMemoryModule { uint8_t reserved[0x3d]; } QEMU_PACKED CXLEventMemoryModule; +/* + * CXL r3.1 section Table 8-50: Dynamic Capacity Event Record + * All fields little endian. + */ +typedef struct CXLEventDynamicCapacity { + CXLEventRecordHdr hdr; + uint8_t type; + uint8_t validity_flags; + uint16_t host_id; + uint8_t updated_region_id; + uint8_t flags; + uint8_t reserved2[2]; + uint8_t dynamic_capacity_extent[0x28]; /* defined in cxl_device.h */ + uint8_t reserved[0x18]; + uint32_t extents_avail; + uint32_t tags_avail; +} QEMU_PACKED CXLEventDynamicCapacity; + #endif /* CXL_EVENTS_H */ diff --git a/qapi/cxl.json b/qapi/cxl.json index 8cc4c72fa9..19927629c7 100644 --- a/qapi/cxl.json +++ b/qapi/cxl.json @@ -19,13 +19,16 @@ # # @fatal: Fatal Event Log # +# @dyncap: Dynamic Capacity Event Log +# # Since: 8.1 ## { 'enum': 'CxlEventLog', 'data': ['informational', 'warning', 'failure', - 'fatal'] + 'fatal', + 'dyncap'] } ## @@ -361,3 +364,59 @@ ## {'command': 'cxl-inject-correctable-error', 'data': {'path': 'str', 'type': 'CxlCorErrorType'}} + +## +# @CXLDCExtentRecord: +# +# Record of a single extent to add/release +# +# @offset: offset to the start of the region where the extent to be operated +# @len: length of the extent +# +# Since: 9.0 +## +{ 'struct': 'CXLDCExtentRecord', + 'data': { + 'offset':'uint64', + 'len': 'uint64' + } +} + +## +# @cxl-add-dynamic-capacity: +# +# Command to start add dynamic capacity extents flow. The device will +# have to ackowledged the acceptance of the extents before they are usable. +# +# @path: CXL DCD canonical QOM path +# @region-id: id of the region where the extent to add/release +# @extents: Extents to add +# +# Since : 9.0 +## +{ 'command': 'cxl-add-dynamic-capacity', + 'data': { 'path': 'str', + 'region-id': 'uint8', + 'extents': [ 'CXLDCExtentRecord' ] + } +} + +## +# @cxl-release-dynamic-capacity: +# +# Command to start release dynamic capacity extents flow. The host will +# need to respond to indicate that it has released the capacity before it +# is made unavailable for read and write and can be re-added. +# +# @path: CXL DCD canonical QOM path +# @region-id: id of the region where the extent to add/release +# @extents: Extents to release +# +# Since : 9.0 +## +{ 'command': 'cxl-release-dynamic-capacity', + 'data': { 'path': 'str', + 'region-id': 'uint8', + 'extents': [ 'CXLDCExtentRecord' ] + } +} From patchwork Wed Feb 21 18:16:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: fan X-Patchwork-Id: 13566076 Received: from mail-pg1-f173.google.com (mail-pg1-f173.google.com [209.85.215.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A494085642 for ; Wed, 21 Feb 2024 18:21:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708539685; cv=none; b=XaIHglOG3+jOcJrH962StQjI2SWCd0ae1GTcAUUmQdHPMOjUx/iTifybt+fQQoTRN1W3abj+WJ/Q5QR8mobbgTVYR1IDS75fQonl9POUuGFjZiER8rn9L5IP1QCa/bnni/GqXfmpxDN6MKYTQpa4GqadV4kuKNe885K+J1f1wug= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708539685; c=relaxed/simple; bh=gi7SFXAZ79rmte8FoP+PDuMFiWsB3vEfmOINJ9CV4MQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=NHAMt+6bV+E8Q0F6b7tZhhZzJFx4SYXc6K+RLFH9sggtp/MoD887+RBTYHv9d8bKAyP5/T9LDfpVXVtrKRJ7/f8SlVppJkMYrK9oHDFAorysQiIFNBmM92qo6mDasaBAzWsdYfjTCCanpFxKZb3ceovsR0gUwGbu/2TP6MsFlNs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=DvzkMzIu; arc=none smtp.client-ip=209.85.215.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="DvzkMzIu" Received: by mail-pg1-f173.google.com with SMTP id 41be03b00d2f7-5cfd95130c6so4109171a12.1 for ; Wed, 21 Feb 2024 10:21:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708539683; x=1709144483; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lopfkfP/Wd4iBuj5Kqs/1syP004OyEXA8CH53rf8oR0=; b=DvzkMzIuCguFveLPARcZCvxNir1AstzhndI62zOJk6CId9MCClbBBKQva0e+6dNDCc 8P4w8kBtIFp2SLdJtEtmwHexyA/k3n49pIs9cFZXh6HXpNyXia/f/JdsrII2+jlAj7rL m4JiPZfKZ0T5Ge1mpCVG02YBbAyxv7q83SUnrX+6UCso9I+XqMkMFpVVtH/GgJaWT9RP 8gpCtcSyhSNWx8vC4gJq1IjyrjifKC9Y3ZeKY8vHLEdTkCA1eEcsvVJc4OXSXVyMsyXj NYJy5Yy5o5+8DtWWXI7luZrS9IylEoQnuee0KDN3wxsdRvJ0dzxTgFOtnhAqfWOrdEZY NH7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708539683; x=1709144483; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lopfkfP/Wd4iBuj5Kqs/1syP004OyEXA8CH53rf8oR0=; b=sLbBD6bdzqTBG+xE0wscf1ZmsBmhET9l/1fiuSiFTSgg3+IRU/ap+cxfcPwrQ1g/I4 73YMEZkgRinRBbZlxVQTHiRwpmd76zoNmeVlLwdhTYRHMrF5elhilwhORhDZySCuSEml ksf2bL6mzTa14BB0JVy7zBtQ9/5WJ9RlFAJ8Y+P4gxO3U73gnUG6b3XJXaHuyOJ0dfGO hvc81HrFQaIKK7VibKbLxMfgTtaP+NR2p6WO7Vt+hh+pNOAFt4Igm4cDuXWN49sicqey +o6rGgyL43CX+hJ6l8Znyy6qksvGhTr9Egemo7dPV0K50gn/uD29jaOazBe1r2Qr9esS /HAA== X-Forwarded-Encrypted: i=1; AJvYcCVzpPYVRbDHkUYhd+aotM9XXNuVdz4gIqFDr6LQQpTVoS4iX6t2Udw+Cw2x13gfrcxS3A5wsDG/Wz+HSGN3QvIm/8iJzCHdjO6E X-Gm-Message-State: AOJu0YwnfPUPYZ9Bcb8COblxDuv4PWj/2coV8e8IHbQieMwUadVwjhN/ c4k2U0O9Nv4dB3azITTl3V9Xai7WoB1lw+R4+x6ZGDBAJ3wbSt88 X-Google-Smtp-Source: AGHT+IGI8ZUHUmT+j5cQpkDxDF7pWt+w5yGxEq/geBFaBWFR/5DdTtSK+rbAurJexC4zeUi0z++sIQ== X-Received: by 2002:a17:90a:f3cc:b0:29a:3bb5:7b91 with SMTP id ha12-20020a17090af3cc00b0029a3bb57b91mr601189pjb.2.1708539682849; Wed, 21 Feb 2024 10:21:22 -0800 (PST) Received: from localhost.localdomain ([2601:641:300:14de:5692:7e41:13a2:69a]) by smtp.gmail.com with ESMTPSA id cs16-20020a17090af51000b002992f49922csm9979009pjb.25.2024.02.21.10.21.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 10:21:22 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Fan Ni Subject: [PATCH v4 10/10] hw/mem/cxl_type3: Add dpa range validation for accesses to DC regions Date: Wed, 21 Feb 2024 10:16:03 -0800 Message-ID: <20240221182020.1086096-11-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240221182020.1086096-1-nifan.cxl@gmail.com> References: <20240221182020.1086096-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Not all dpa range in the DC regions is valid to access until an extent covering the range has been added. Add a bitmap for each region to record whether a DC block in the region has been backed by DC extent. For the bitmap, a bit in the bitmap represents a DC block. When a DC extent is added, all the bits of the blocks in the extent will be set, which will be cleared when the extent is released. Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 3 ++ hw/mem/cxl_type3.c | 82 +++++++++++++++++++++++++++++++++++++ include/hw/cxl/cxl_device.h | 7 ++++ 3 files changed, 92 insertions(+) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 34c4ebbd12..fd3be2f9cf 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -1676,17 +1676,20 @@ static CXLRetCode cmd_dcd_release_dyn_cap(const struct cxl_cmd *cmd, found = true; cxl_remove_extent_from_extent_list(extent_list, ent); ct3d->dc.total_extent_count -= 1; + ct3_clear_region_block_backed(ct3d, ent_start_dpa, ent_len); if (len1) { cxl_insert_extent_to_extent_list(extent_list, ent_start_dpa, len1, NULL, 0); ct3d->dc.total_extent_count += 1; + ct3_set_region_block_backed(ct3d, ent_start_dpa, len1); } if (len2) { cxl_insert_extent_to_extent_list(extent_list, dpa + len, len2, NULL, 0); ct3d->dc.total_extent_count += 1; + ct3_set_region_block_backed(ct3d, dpa + len, len2); } break; /*Currently we reject the attempt to remove a superset*/ diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index b8c4273e99..a56906db25 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -660,6 +660,7 @@ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) region_base += region->len; ct3d->dc.total_capacity += region->len; + region->blk_bitmap = bitmap_new(region->len / region->block_size); } QTAILQ_INIT(&ct3d->dc.extents); QTAILQ_INIT(&ct3d->dc.extents_pending_to_add); @@ -667,6 +668,17 @@ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) return true; } +static void cxl_destroy_dc_regions(CXLType3Dev *ct3d) +{ + int i; + struct CXLDCDRegion *region; + + for (i = 0; i < ct3d->dc.num_regions; i++) { + region = &ct3d->dc.regions[i]; + g_free(region->blk_bitmap); + } +} + static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) { DeviceState *ds = DEVICE(ct3d); @@ -860,6 +872,7 @@ err_free_special_ops: g_free(regs->special_ops); err_address_space_free: if (ct3d->dc.host_dc) { + cxl_destroy_dc_regions(ct3d); address_space_destroy(&ct3d->dc.host_dc_as); } if (ct3d->hostpmem) { @@ -881,6 +894,7 @@ static void ct3_exit(PCIDevice *pci_dev) cxl_doe_cdat_release(cxl_cstate); g_free(regs->special_ops); if (ct3d->dc.host_dc) { + cxl_destroy_dc_regions(ct3d); address_space_destroy(&ct3d->dc.host_dc_as); } if (ct3d->hostpmem) { @@ -891,6 +905,70 @@ static void ct3_exit(PCIDevice *pci_dev) } } +/* + * Mark the DPA range [dpa, dap + len) to be backed and accessible. This + * happens when a DC extent is added and accepted by the host. + */ +void ct3_set_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len) +{ + CXLDCDRegion *region; + + region = cxl_find_dc_region(ct3d, dpa, len); + if (!region) { + return; + } + + bitmap_set(region->blk_bitmap, (dpa - region->base) / region->block_size, + len / region->block_size); +} + +/* + * Check whether the DPA range [dpa, dpa + len) is backed with DC extents. + * Used when validating read/write to dc regions + */ +bool ct3_test_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len) +{ + CXLDCDRegion *region; + uint64_t nbits; + long nr; + + region = cxl_find_dc_region(ct3d, dpa, len); + if (!region) { + return false; + } + + nr = (dpa - region->base) / region->block_size; + nbits = DIV_ROUND_UP(len, region->block_size); + /* + * if bits between [dpa, dpa + len) are all 1s, meaning the DPA range is + * backed with DC extents, return true; else return false. + */ + return find_next_zero_bit(region->blk_bitmap, nr + nbits, nr) == nr + nbits; +} + +/* + * Mark the DPA range [dpa, dap + len) to be unbacked and inaccessible. This + * happens when a dc extent is released by the host. + */ +void ct3_clear_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len) +{ + CXLDCDRegion *region; + uint64_t nbits; + long nr; + + region = cxl_find_dc_region(ct3d, dpa, len); + if (!region) { + return; + } + + nr = (dpa - region->base) / region->block_size; + nbits = len / region->block_size; + bitmap_clear(region->blk_bitmap, nr, nbits); +} + static bool cxl_type3_dpa(CXLType3Dev *ct3d, hwaddr host_addr, uint64_t *dpa) { int hdm_inc = R_CXL_HDM_DECODER1_BASE_LO - R_CXL_HDM_DECODER0_BASE_LO; @@ -996,6 +1074,10 @@ static int cxl_type3_hpa_to_as_and_dpa(CXLType3Dev *ct3d, *as = &ct3d->hostpmem_as; *dpa_offset -= vmr_size; } else { + if (!ct3_test_region_block_backed(ct3d, *dpa_offset, size)) { + return -ENODEV; + } + *as = &ct3d->dc.host_dc_as; *dpa_offset -= (vmr_size + pmr_size); } diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index 1d31164bd3..10f0389b50 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -450,6 +450,7 @@ typedef struct CXLDCDRegion { uint64_t block_size; uint32_t dsmadhandle; uint8_t flags; + unsigned long *blk_bitmap; } CXLDCDRegion; struct CXLType3Dev { @@ -557,4 +558,10 @@ void cxl_insert_extent_to_extent_list(CXLDCExtentList *list, uint8_t *tag, uint16_t shared_seq); bool test_any_bits_set(const unsigned long *addr, int nr, int size); +void ct3_set_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len); +void ct3_clear_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len); +bool ct3_test_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len); #endif