From patchwork Tue Nov 7 18:07:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13449166 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8AB863589F for ; Tue, 7 Nov 2023 18:10:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Dby5+OKq" Received: from mail-pj1-x102d.google.com (mail-pj1-x102d.google.com [IPv6:2607:f8b0:4864:20::102d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DD340C0 for ; Tue, 7 Nov 2023 10:10:17 -0800 (PST) Received: by mail-pj1-x102d.google.com with SMTP id 98e67ed59e1d1-280109daaaaso4484159a91.3 for ; Tue, 07 Nov 2023 10:10:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1699380617; x=1699985417; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ogYJbqlkzS2Yd8saws9Occt0F7m6RWm2l0bkCCgcR3U=; b=Dby5+OKqbdP4/F+fUIUKuRbqmIU7v6wykdJ/aO7ZZPijzpqGUTu/ofkOLlZBDaJk/z 2d+cGT36B9NwmxR2gAEaUz0XeLjsnUtxuF6U8FlJSTYhla+YJCgOW5RhSyGR0vfbZoqO lmX/w7bU+k85ah00eXUuPnIdtFC6DqnNuztOck+tXe77i+6u6Gba2SDDofNOM8a1iAIH t5u6DyJnMnjRkFPCTuDPIRktRMJWpDFHRJwb2dtS+jOC0jtpp9C4i29MnRJBYRy4cVpB TsXmxkFqYsmW/9btMgP0FSLgQF6d9/z4GuDNYSaPr4DUizR3S5S3ltRTgN8SRV1J8aCS XITQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699380617; x=1699985417; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ogYJbqlkzS2Yd8saws9Occt0F7m6RWm2l0bkCCgcR3U=; b=Nxm4aVD1E0KeJDBVinBqSb2hn8k5JOEbpqdU3Geyz6YGBDKkG9t9U+63hGrVgjZ1Pq dza1A/rGjnWn37yDmsO2lv5XF5BG4tkqrDwXBe/MWodTVTgKun9Y+FpDoGgWVNQcKyyB /NcMZqyBIn4yg3c5HqqMsuVE6y0BVRSSIfC2IXbnxCB5cbgvx82c3z2uw29kK9Gt/2gI pNZQ4CHlIYxjVRvuNk613QqjleRbb7rArfkwxaktZuwtGlOGrH0L0MT2FPV1GMjEVQjN NCQwBYeKeWJgEBppGbWzamJR21jiDz2voXgym+wx3mCzWS1LIaJCSb3peuv6Ix6X0RKw veyw== X-Gm-Message-State: AOJu0YzMb+Zc3SWIQqyOQ1navxwg7SkcI3xAQGRzNgsb41my2Jaj8WOp KqMaa2+v9r576RxG+XUhbLGQIIl7upA= X-Google-Smtp-Source: AGHT+IFWm9kgjVGfWor3OqA8aZVUBtUhbkrx5BtmA6zSu7sqrwDUHquMPZD7UdUq5z+wxXKrUcUVUg== X-Received: by 2002:a17:90b:4acb:b0:280:3650:382a with SMTP id mh11-20020a17090b4acb00b002803650382amr22482838pjb.16.1699380617158; Tue, 07 Nov 2023 10:10:17 -0800 (PST) Received: from DT.local (c-71-202-158-162.hsd1.ca.comcast.net. [71.202.158.162]) by smtp.gmail.com with ESMTPSA id az10-20020a170902a58a00b001cc615e6850sm131924plb.90.2023.11.07.10.10.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Nov 2023 10:10:16 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan@outlook.com, jim.harris@samsung.com, Fan Ni Subject: [PATCH v3 1/9] hw/cxl/cxl-mailbox-utils: Add dc_event_log_size field to output payload of identify memory device command Date: Tue, 7 Nov 2023 10:07:05 -0800 Message-ID: <20231107180907.553451-2-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231107180907.553451-1-nifan.cxl@gmail.com> References: <20231107180907.553451-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Based on CXL spec 3.0 Table 8-94 (Identify Memory Device Output Payload), dynamic capacity event log size should be part of output of the Identify command. Add dc_event_log_size to the output payload for the host to get the info. Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index f1145e9671..8eceedfa87 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -21,6 +21,7 @@ #include "sysemu/hostmem.h" #define CXL_CAPACITY_MULTIPLIER (256 * MiB) +#define CXL_DC_EVENT_LOG_SIZE 8 /* * How to add a new command, example. The command set FOO, with cmd BAR. @@ -753,8 +754,9 @@ static CXLRetCode cmd_identify_memory_device(const struct cxl_cmd *cmd, uint16_t inject_poison_limit; uint8_t poison_caps; uint8_t qos_telemetry_caps; + uint16_t dc_event_log_size; } QEMU_PACKED *id; - QEMU_BUILD_BUG_ON(sizeof(*id) != 0x43); + QEMU_BUILD_BUG_ON(sizeof(*id) != 0x45); CXLType3Dev *ct3d = CXL_TYPE3(cci->d); CXLType3Class *cvc = CXL_TYPE3_GET_CLASS(ct3d); CXLDeviceState *cxl_dstate = &ct3d->cxl_dstate; @@ -780,6 +782,7 @@ static CXLRetCode cmd_identify_memory_device(const struct cxl_cmd *cmd, st24_le_p(id->poison_list_max_mer, 256); /* No limit - so limited by main poison record limit */ stw_le_p(&id->inject_poison_limit, 0); + stw_le_p(&id->dc_event_log_size, CXL_DC_EVENT_LOG_SIZE); *len_out = sizeof(*id); return CXL_MBOX_SUCCESS; From patchwork Tue Nov 7 18:07:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13449167 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 278203588D for ; Tue, 7 Nov 2023 18:10:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="WcY31NN7" Received: from mail-pl1-x62f.google.com (mail-pl1-x62f.google.com [IPv6:2607:f8b0:4864:20::62f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AF57EB0 for ; Tue, 7 Nov 2023 10:10:18 -0800 (PST) Received: by mail-pl1-x62f.google.com with SMTP id d9443c01a7336-1cc3bc5df96so44122945ad.2 for ; Tue, 07 Nov 2023 10:10:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1699380618; x=1699985418; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pxa2X/M3i5M/B67pIBM08iOpkqVk5AqzieyHZGjyhtc=; b=WcY31NN7vWyO7vMWyvpdGWowIjr3ckL7+cERb+bj5Y+lY7Q/bnAe+SPefjs6K4Bmhh B56r7lZc0iqD9ytr9PAMrHJjrmqdQSuntglvUeytAqSbVk3p9qjf0vDLKA5tJYHvRa3q kpBn0Op95O6nAL7vNJ/LKoD8T3AOvFsazPkCkP6n6pias4nVCTTD/LjIQwCOmXYqI0Iu 2f1WDOqZDauo5nWi055TSh+z+jzeLZ7pqypWQXpU4WCIniJirRgfZGY4FFLYJFVkTPY4 MwD+yTDXfHKIH9atjL1357Z8G/sLrDpEJOe+V11KzvWEkwEeBej4/Bprmgplu7nQNhbf m8IA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699380618; x=1699985418; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pxa2X/M3i5M/B67pIBM08iOpkqVk5AqzieyHZGjyhtc=; b=MynyJe307OQjykSXXOH9bAhV15ZZCQmVWIS/vKYUjlH4wsD4WM+9YwCGMkiUp0ZOJL 0X/WkQY2vumNhz9fS/9UxCF8GII6+slvVEzo6zmMP847j7ZeEAHJuzK0YWUGV2dS/tzt xaZ/UfWJ1/oXM+7CYhW4jIvHFFtYDT0K5SUt6pBOt77pNa3JW2gSTXU7buYpdndqwOr9 r/QJ4DQWqSTeeoWIJpxzfYDIIOI0OPRhMNuHHFI+nH76j0AeqcdrEBc28czAz5ozamZS 98HUhC0pgGY+TVyk4eUnsqQthIbwp/0uHnuV9ynZyvtk3jdxX1PSoWczxnbNhPmppLo2 CQhg== X-Gm-Message-State: AOJu0YznwEfkmG7IbiMA8iADcnVFADc0xEg++sO5lIRQoHTZaVW8qQjY oDCjmeO7PR1ftFLMP92ZAYE= X-Google-Smtp-Source: AGHT+IEQmjibQ784vy6VRNnHQ85xLgjNWXzyE4NL1vQ8Kkz/qut4AdhAt5wqtkfkCdPsz3SW1ZwoXg== X-Received: by 2002:a17:902:fa07:b0:1cb:ffc1:23a3 with SMTP id la7-20020a170902fa0700b001cbffc123a3mr26987897plb.62.1699380618084; Tue, 07 Nov 2023 10:10:18 -0800 (PST) Received: from DT.local (c-71-202-158-162.hsd1.ca.comcast.net. [71.202.158.162]) by smtp.gmail.com with ESMTPSA id az10-20020a170902a58a00b001cc615e6850sm131924plb.90.2023.11.07.10.10.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Nov 2023 10:10:17 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan@outlook.com, jim.harris@samsung.com, Fan Ni Subject: [PATCH v3 2/9] hw/cxl/cxl-mailbox-utils: Add dynamic capacity region representative and mailbox command support Date: Tue, 7 Nov 2023 10:07:06 -0800 Message-ID: <20231107180907.553451-3-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231107180907.553451-1-nifan.cxl@gmail.com> References: <20231107180907.553451-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Per cxl spec 3.0, add dynamic capacity region representative based on Table 8-126 and extend the cxl type3 device definition to include dc region information. Also, based on info in 8.2.9.8.9.1, add 'Get Dynamic Capacity Configuration' mailbox support. Note: decode_len of a dc region is aligned to 256*MiB, need to be divided by 256 * MiB before returned to the host for "Get Dynamic Capacity Configuration" mailbox command. Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 80 +++++++++++++++++++++++++++++++++++++ hw/mem/cxl_type3.c | 6 +++ include/hw/cxl/cxl_device.h | 17 ++++++++ 3 files changed, 103 insertions(+) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 8eceedfa87..f80dd6474f 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -80,6 +80,8 @@ enum { #define GET_POISON_LIST 0x0 #define INJECT_POISON 0x1 #define CLEAR_POISON 0x2 + DCD_CONFIG = 0x48, + #define GET_DC_CONFIG 0x0 PHYSICAL_SWITCH = 0x51, #define IDENTIFY_SWITCH_DEVICE 0x0 #define GET_PHYSICAL_PORT_STATE 0x1 @@ -1210,6 +1212,74 @@ static CXLRetCode cmd_media_clear_poison(const struct cxl_cmd *cmd, return CXL_MBOX_SUCCESS; } +/* + * CXL r3.0 section 8.2.9.8.9.1: Get Dynamic Capacity Configuration + * (Opcode: 4800h) + */ +static CXLRetCode cmd_dcd_get_dyn_cap_config(const struct cxl_cmd *cmd, + uint8_t *payload_in, + size_t len_in, + uint8_t *payload_out, + size_t *len_out, + CXLCCI *cci) +{ + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + struct get_dyn_cap_config_in_pl { + uint8_t region_cnt; + uint8_t start_region_id; + } QEMU_PACKED; + + struct get_dyn_cap_config_out_pl { + uint8_t num_regions; + uint8_t rsvd1[7]; + struct { + uint64_t base; + uint64_t decode_len; + uint64_t region_len; + uint64_t block_size; + uint32_t dsmadhandle; + uint8_t flags; + uint8_t rsvd2[3]; + } QEMU_PACKED records[]; + } QEMU_PACKED; + + struct get_dyn_cap_config_in_pl *in = (void *)payload_in; + struct get_dyn_cap_config_out_pl *out = (void *)payload_out; + uint16_t record_count = 0, i; + uint16_t out_pl_len; + uint8_t start_region_id = in->start_region_id; + + if (start_region_id >= ct3d->dc.num_regions) { + return CXL_MBOX_INVALID_INPUT; + } + + record_count = MIN(ct3d->dc.num_regions - in->start_region_id, + in->region_cnt); + + out_pl_len = sizeof(*out) + record_count * sizeof(out->records[0]); + assert(out_pl_len <= CXL_MAILBOX_MAX_PAYLOAD_SIZE); + + memset(out, 0, out_pl_len); + out->num_regions = record_count; + for (i = 0; i < record_count; i++) { + stq_le_p(&out->records[i].base, + ct3d->dc.regions[start_region_id + i].base); + stq_le_p(&out->records[i].decode_len, + ct3d->dc.regions[start_region_id + i].decode_len / + CXL_CAPACITY_MULTIPLIER); + stq_le_p(&out->records[i].region_len, + ct3d->dc.regions[start_region_id + i].len); + stq_le_p(&out->records[i].block_size, + ct3d->dc.regions[start_region_id + i].block_size); + stl_le_p(&out->records[i].dsmadhandle, + ct3d->dc.regions[start_region_id + i].dsmadhandle); + out->records[i].flags = ct3d->dc.regions[start_region_id + i].flags; + } + + *len_out = out_pl_len; + return CXL_MBOX_SUCCESS; +} + #define IMMEDIATE_CONFIG_CHANGE (1 << 1) #define IMMEDIATE_DATA_CHANGE (1 << 2) #define IMMEDIATE_POLICY_CHANGE (1 << 3) @@ -1254,6 +1324,11 @@ static const struct cxl_cmd cxl_cmd_set[256][256] = { cmd_media_clear_poison, 72, 0 }, }; +static const struct cxl_cmd cxl_cmd_set_dcd[256][256] = { + [DCD_CONFIG][GET_DC_CONFIG] = { "DCD_GET_DC_CONFIG", + cmd_dcd_get_dyn_cap_config, 2, 0 }, +}; + static const struct cxl_cmd cxl_cmd_set_sw[256][256] = { [INFOSTAT][IS_IDENTIFY] = { "IDENTIFY", cmd_infostat_identify, 0, 18 }, [INFOSTAT][BACKGROUND_OPERATION_STATUS] = { "BACKGROUND_OPERATION_STATUS", @@ -1465,7 +1540,12 @@ void cxl_initialize_mailbox_swcci(CXLCCI *cci, DeviceState *intf, void cxl_initialize_mailbox_t3(CXLCCI *cci, DeviceState *d, size_t payload_max) { + CXLType3Dev *ct3d = CXL_TYPE3(d); + cxl_copy_cci_commands(cci, cxl_cmd_set); + if (ct3d->dc.num_regions) { + cxl_copy_cci_commands(cci, cxl_cmd_set_dcd); + } cci->d = d; /* No separation for PCI MB as protocol handled in PCI device */ diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 7b4d1ee774..6c1ccda159 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -1075,6 +1075,12 @@ static void ct3d_reset(DeviceState *dev) uint32_t *reg_state = ct3d->cxl_cstate.crb.cache_mem_registers; uint32_t *write_msk = ct3d->cxl_cstate.crb.cache_mem_regs_write_mask; + if (ct3d->dc.num_regions) { + ct3d->cxl_dstate.is_dcd = true; + } else { + ct3d->cxl_dstate.is_dcd = false; + } + cxl_component_register_init_common(reg_state, write_msk, CXL2_TYPE3_DEVICE); cxl_device_register_init_t3(ct3d); diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index 4f2ef0b899..334c51fddb 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -235,6 +235,7 @@ typedef struct cxl_device_state { uint64_t mem_size; uint64_t pmem_size; uint64_t vmem_size; + bool is_dcd; const struct cxl_cmd (*cxl_cmd_set)[256]; CPMUState cpmu[CXL_NUM_CPMU_INSTANCES]; @@ -417,6 +418,17 @@ typedef struct CXLPoison { typedef QLIST_HEAD(, CXLPoison) CXLPoisonList; #define CXL_POISON_LIST_LIMIT 256 +#define DCD_MAX_REGION_NUM 8 + +typedef struct CXLDCDRegion { + uint64_t base; + uint64_t decode_len; /* aligned to 256*MiB */ + uint64_t len; + uint64_t block_size; + uint32_t dsmadhandle; + uint8_t flags; +} CXLDCDRegion; + struct CXLType3Dev { /* Private */ PCIDevice parent_obj; @@ -453,6 +465,11 @@ struct CXLType3Dev { unsigned int poison_list_cnt; bool poison_list_overflowed; uint64_t poison_list_overflow_ts; + + struct dynamic_capacity { + uint8_t num_regions; /* 0-8 regions */ + CXLDCDRegion regions[DCD_MAX_REGION_NUM]; + } dc; }; #define TYPE_CXL_TYPE3 "cxl-type3" From patchwork Tue Nov 7 18:07:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13449168 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 01277358A8 for ; Tue, 7 Nov 2023 18:10:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="AiPDYEGY" Received: from mail-pl1-x634.google.com (mail-pl1-x634.google.com [IPv6:2607:f8b0:4864:20::634]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8B0D08F for ; Tue, 7 Nov 2023 10:10:19 -0800 (PST) Received: by mail-pl1-x634.google.com with SMTP id d9443c01a7336-1cc4f777ab9so42936745ad.0 for ; Tue, 07 Nov 2023 10:10:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1699380619; x=1699985419; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lerGkvWs+2MRxdzE0DMBKiSOqV9ui2+nfnjyOq/W8aE=; b=AiPDYEGYYamOo8G4T/N/l2ccB6vDkF7IOnkxx0vy1i+14lil101g+v7O4zu8m9R12o wB5xYZ7MUFx/EV7Yi7h0VBZKv3SD458z5EMglO7kcY/6a8AB5ki3V1+RiGlwCmhsRSMf 4TiMr4VkdctjN9EaIz5HSZ6gf/RCL7kcaaR0Jb+TDNBAlbXC8ldf3iGmhPvfASeJRcF9 RO22CHnEg0hlGYfnj6eajouYjiYhOS8J9V/gIvLeYSCCLClfCUBrqdIDN1O8FKHnckti ctNvWNcsM49PAmmgWeT2kEztG9uSf9mGqY4DA0ThBhvct6osYv8WAleq2xmBFf5/LGBb vEHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699380619; x=1699985419; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lerGkvWs+2MRxdzE0DMBKiSOqV9ui2+nfnjyOq/W8aE=; b=W8bJFTes9brdNjKjE1heWjSt7u1ZddNOPyU1MYk5ftHxWV9Ci9fYJXUSUl1c1k2ZGS 7zcoigf4QS0XpQf3/37BojEBepX64JKXDl3/yzvT2TIYDZ4nkaCjJseu5xOoaWQEABCx 7fqiBO3mT2S0sqJ5qxHpJjPTdzgYpLyEa7gGNQ9gYW+cLoDLRx5meDqeR7YANRBEaB/H lfEiZjN6CHCCyFktruxxIhFyffXf2j7XWSc40kNwkzgdZMEb/4tpEhSPZ81b8HAu+fdX qnVzknrm6AnGzY2PZuOPMmDLFybU/HJYp5DSItNAii/sDVKC3TSIHS/GaG39xHF5q85c DjIA== X-Gm-Message-State: AOJu0YxwaM+rn1dDn1mHmYkwZWTM2eVXVqsvAd8YkV8WFq8XLaUnPrxQ Cw1mz0IPWts+iS2mMNtru/M= X-Google-Smtp-Source: AGHT+IE9CVDa0uPYwQ3dK4HiagUY1Wab+0CtaMHuuj9jitAwMCnxS1Hx84mPVFs+ql161c+jSgZwkA== X-Received: by 2002:a17:903:245:b0:1c5:d8a3:8789 with SMTP id j5-20020a170903024500b001c5d8a38789mr32291748plh.4.1699380618992; Tue, 07 Nov 2023 10:10:18 -0800 (PST) Received: from DT.local (c-71-202-158-162.hsd1.ca.comcast.net. [71.202.158.162]) by smtp.gmail.com with ESMTPSA id az10-20020a170902a58a00b001cc615e6850sm131924plb.90.2023.11.07.10.10.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Nov 2023 10:10:18 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan@outlook.com, jim.harris@samsung.com, Fan Ni Subject: [PATCH v3 3/9] include/hw/cxl/cxl_device: Rename mem_size as static_mem_size for type3 memory devices Date: Tue, 7 Nov 2023 10:07:07 -0800 Message-ID: <20231107180907.553451-4-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231107180907.553451-1-nifan.cxl@gmail.com> References: <20231107180907.553451-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Rename mem_size as static_mem_size for type3 memdev to cover static RAM and pmem capacity, preparing for the introduction of dynamic capacity to support dynamic capacity devices. Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 4 ++-- hw/mem/cxl_type3.c | 8 ++++---- include/hw/cxl/cxl_device.h | 2 +- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index f80dd6474f..707fd9fe7f 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -774,7 +774,7 @@ static CXLRetCode cmd_identify_memory_device(const struct cxl_cmd *cmd, snprintf(id->fw_revision, 0x10, "BWFW VERSION %02d", 0); stq_le_p(&id->total_capacity, - cxl_dstate->mem_size / CXL_CAPACITY_MULTIPLIER); + cxl_dstate->static_mem_size / CXL_CAPACITY_MULTIPLIER); stq_le_p(&id->persistent_capacity, cxl_dstate->pmem_size / CXL_CAPACITY_MULTIPLIER); stq_le_p(&id->volatile_capacity, @@ -1149,7 +1149,7 @@ static CXLRetCode cmd_media_clear_poison(const struct cxl_cmd *cmd, struct clear_poison_pl *in = (void *)payload_in; dpa = ldq_le_p(&in->dpa); - if (dpa + CXL_CACHE_LINE_SIZE > cxl_dstate->mem_size) { + if (dpa + CXL_CACHE_LINE_SIZE > cxl_dstate->static_mem_size) { return CXL_MBOX_INVALID_PA; } diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 6c1ccda159..754c885cd1 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -762,7 +762,7 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) } address_space_init(&ct3d->hostvmem_as, vmr, v_name); ct3d->cxl_dstate.vmem_size = memory_region_size(vmr); - ct3d->cxl_dstate.mem_size += memory_region_size(vmr); + ct3d->cxl_dstate.static_mem_size += memory_region_size(vmr); g_free(v_name); } @@ -785,7 +785,7 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) } address_space_init(&ct3d->hostpmem_as, pmr, p_name); ct3d->cxl_dstate.pmem_size = memory_region_size(pmr); - ct3d->cxl_dstate.mem_size += memory_region_size(pmr); + ct3d->cxl_dstate.static_mem_size += memory_region_size(pmr); g_free(p_name); } @@ -1008,7 +1008,7 @@ static int cxl_type3_hpa_to_as_and_dpa(CXLType3Dev *ct3d, return -EINVAL; } - if (*dpa_offset > ct3d->cxl_dstate.mem_size) { + if (*dpa_offset > ct3d->cxl_dstate.static_mem_size) { return -EINVAL; } @@ -1188,7 +1188,7 @@ static bool set_cacheline(CXLType3Dev *ct3d, uint64_t dpa_offset, uint8_t *data) return false; } - if (dpa_offset + CXL_CACHE_LINE_SIZE > ct3d->cxl_dstate.mem_size) { + if (dpa_offset + CXL_CACHE_LINE_SIZE > ct3d->cxl_dstate.static_mem_size) { return false; } diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index 334c51fddb..de6469eef7 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -232,7 +232,7 @@ typedef struct cxl_device_state { } timestamp; /* memory region size, HDM */ - uint64_t mem_size; + uint64_t static_mem_size; uint64_t pmem_size; uint64_t vmem_size; bool is_dcd; From patchwork Tue Nov 7 18:07:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13449169 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 43A3D358AB for ; Tue, 7 Nov 2023 18:10:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="lPszRS8I" Received: from mail-pl1-x62d.google.com (mail-pl1-x62d.google.com [IPv6:2607:f8b0:4864:20::62d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7865CC0 for ; Tue, 7 Nov 2023 10:10:20 -0800 (PST) Received: by mail-pl1-x62d.google.com with SMTP id d9443c01a7336-1cc29f39e7aso40425255ad.0 for ; Tue, 07 Nov 2023 10:10:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1699380620; x=1699985420; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cIc7H0VDI7Su5JGUvQgueNQKhCP3W7elx90e94K+TnQ=; b=lPszRS8IrvQFtVD2veIUnG6P7bB4AtrQewQ4GuOPtqPhGFZ1gKqkXQ9oXhY7HGX2VP W8tnFVvZUXtWaGLKn8LjMkrkLf0kQ2jD7jyWFQpepCUTx/nCOsHjv1Z9+Vx7kKwWcqbO VQ+WbVbfchb+6VcDSviD8d9wL7jqb2tP59Jbi/jTMU96pks3mcWXYWz2mVhWnDkDeBDw BjySNH/iePi+wAa2NiiYaA1DmP2dFdLyI1WFaHvapPt2OdvqXNmCBQNQy+rqp6xPY1tN KjYd649Erpzmqkc2M34pPKsLgl2C64pm90/vuOZXIVr3XBvyzv+BqTL/iYQ0ERgBkWQ3 /BnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699380620; x=1699985420; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cIc7H0VDI7Su5JGUvQgueNQKhCP3W7elx90e94K+TnQ=; b=fba8gRBpew7pLH54BqwLktwt4VcMJDzoutudWCXNZ5dhQNvpevwi1IIhBIJ4yka07v wSGCm3gkj6xB5YpE8B7o+mbOxYTVqty1kZHshSxFeURD+M+4DT3oVTcE2M9sBfqVCyks WWn9EDA4Y/GquWj+5fZQczZbUV4lDNIqeN5ynpexqlzBGPEJl3RZSFUWiPNHR1SpIyXD VYLPUdvExqW72btGENbS9n/Z0/+PR9UVXEOEqa7TvkSCeZb5NhfHgjiDuWktWZ/E9Nyf vdmKtjxQKydxBaT7+DR7ttiMyLuYQjNxEuumbADbdYa8YJUXDMgUXbkQ6u1asOa6NGEn hrXA== X-Gm-Message-State: AOJu0Ywg1QnJjdn/+lzXI3EO7rsKwZdFEmitfj/AtFypm0BUJ7uV2507 h1XVCtDIVV+SqhJSxHawfLZBk7INwmU= X-Google-Smtp-Source: AGHT+IHGNH7ZWpcEDNR+g0tlYCx5gQyhkoR95XBku5YzVtfRDOD5hxW6GKGgVBtcG5+1BC6GyRPz2w== X-Received: by 2002:a17:902:a418:b0:1cc:2eda:bde6 with SMTP id p24-20020a170902a41800b001cc2edabde6mr26518551plq.34.1699380619857; Tue, 07 Nov 2023 10:10:19 -0800 (PST) Received: from DT.local (c-71-202-158-162.hsd1.ca.comcast.net. [71.202.158.162]) by smtp.gmail.com with ESMTPSA id az10-20020a170902a58a00b001cc615e6850sm131924plb.90.2023.11.07.10.10.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Nov 2023 10:10:19 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan@outlook.com, jim.harris@samsung.com, Fan Ni Subject: [PATCH v3 4/9] hw/mem/cxl_type3: Add support to create DC regions to type3 memory devices Date: Tue, 7 Nov 2023 10:07:08 -0800 Message-ID: <20231107180907.553451-5-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231107180907.553451-1-nifan.cxl@gmail.com> References: <20231107180907.553451-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni With the change, when setting up memory for type3 memory device, we can create DC regions. A property 'num-dc-regions' is added to ct3_props to allow users to pass the number of DC regions to create. To make it easier, other region parameters like region base, length, and block size are hard coded. If needed, these parameters can be added easily. With the change, we can create DC regions with proper kernel side support as below: region=$(cat /sys/bus/cxl/devices/decoder0.0/create_dc_region) echo $region> /sys/bus/cxl/devices/decoder0.0/create_dc_region echo 256 > /sys/bus/cxl/devices/$region/interleave_granularity echo 1 > /sys/bus/cxl/devices/$region/interleave_ways echo "dc0" >/sys/bus/cxl/devices/decoder2.0/mode echo 0x40000000 >/sys/bus/cxl/devices/decoder2.0/dpa_size echo 0x40000000 > /sys/bus/cxl/devices/$region/size echo "decoder2.0" > /sys/bus/cxl/devices/$region/target0 echo 1 > /sys/bus/cxl/devices/$region/commit echo $region > /sys/bus/cxl/drivers/cxl_region/bind Signed-off-by: Fan Ni --- hw/mem/cxl_type3.c | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 754c885cd1..2d67d2015c 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -721,6 +721,36 @@ static void ct3d_reg_write(void *opaque, hwaddr offset, uint64_t value, } } +static int cxl_create_dc_regions(CXLType3Dev *ct3d) +{ + int i; + uint64_t region_base = 0; + uint64_t region_len = 2 * GiB; + uint64_t decode_len = 8; /* 8*256MB */ + uint64_t blk_size = 2 * MiB; + CXLDCDRegion *region; + + if (ct3d->hostvmem) { + region_base += ct3d->hostvmem->size; + } + if (ct3d->hostpmem) { + region_base += ct3d->hostpmem->size; + } + for (i = 0; i < ct3d->dc.num_regions; i++) { + region = &ct3d->dc.regions[i]; + region->base = region_base; + region->decode_len = decode_len; + region->len = region_len; + region->block_size = blk_size; + /* dsmad_handle is set when creating cdat table entries */ + region->flags = 0; + + region_base += region->len; + } + + return 0; +} + static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) { DeviceState *ds = DEVICE(ct3d); @@ -789,6 +819,10 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) g_free(p_name); } + if (cxl_create_dc_regions(ct3d)) { + return false; + } + return true; } @@ -1108,6 +1142,7 @@ static Property ct3_props[] = { DEFINE_PROP_UINT64("sn", CXLType3Dev, sn, UI64_NULL), DEFINE_PROP_STRING("cdat", CXLType3Dev, cxl_cstate.cdat.filename), DEFINE_PROP_UINT16("spdm", CXLType3Dev, spdm_port, 0), + DEFINE_PROP_UINT8("num-dc-regions", CXLType3Dev, dc.num_regions, 0), DEFINE_PROP_END_OF_LIST(), }; From patchwork Tue Nov 7 18:07:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13449171 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 61D3B3589E for ; Tue, 7 Nov 2023 18:10:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="bdF2LZMn" Received: from mail-pl1-x629.google.com (mail-pl1-x629.google.com [IPv6:2607:f8b0:4864:20::629]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B6B3F8F for ; Tue, 7 Nov 2023 10:10:21 -0800 (PST) Received: by mail-pl1-x629.google.com with SMTP id d9443c01a7336-1cc3216b2a1so48538845ad.2 for ; Tue, 07 Nov 2023 10:10:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1699380621; x=1699985421; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fidJGI9tSfEUK7LuKCCNr/HhdNthEOLs0OgUT171rXg=; b=bdF2LZMn9Fg1/iM2fRX631ezfjeklzum66uQl8cvJE+X419IKlQCTjllt0nLn0XLMN ykbp6aYiBR9p8NOXFJeoXGI20KjYpGOViMUjScToipMgabNM8t+1Hwd4CgSu0S3YHmot 14UH0DNUhJFtJRPz1le9l4oa66fyZ57lYOfuQbM/2u/xAG1FikJtx4cQC+XzD1X7sup6 9hVZ8G0S+10G+2O7By495zt4HuJGWUBmQ2pPo2asr+0y0Ff29m8gD6b+Bols9JnHDFuB gY9XA3PYP5JxwT71OYuxaMG0zTXOeXgVaWaNG3OlWfXAJDgV98UKZjvS4HQ4VWi9Xlea kOow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699380621; x=1699985421; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fidJGI9tSfEUK7LuKCCNr/HhdNthEOLs0OgUT171rXg=; b=uKdJFTiqlxAMoN04l67DWJ47KrilptW6A0oj96SNu7BFP1IGemwOZTIyT6AxIjpQDW P9uLlFrbIjF4Ijrjzsc6qn0XeWgZAoK/Q3IzCxolTkXuj0YL25JFw7vtrFFQUl1WKNM9 K/GirfPb55nR4thff0zDI6FKNqgJ7ZrY52WRCkbq8iVxmQYc6LfIiIa8eyjo8zQevj0U AQIYynECZ9ze2KUPAGN8+OBmMxkp/jQPpNlbjXRqrcBSj2C2QljdwQuSTO9W7q7g20hD mwpdQi9RTqzURz9AKgVHeA9YzUV6j6JrvV+b7GTJb7yJAOuy4+6v0wTobE7/PGP2Otee E5jA== X-Gm-Message-State: AOJu0YxKeUgdUwiycSna9eOCjLEePrP2Au4MvklM+VpeeKRphJwtRZu5 9hPjeCpKdo99tQasweGeD6Q= X-Google-Smtp-Source: AGHT+IH2sOSQhzqP6upCTCtWBH3TOQDDXqGUL3eGXNC+4Wz3oTZeRcj6fBK6yv5oVwcRAzQmxyDd9g== X-Received: by 2002:a17:903:11c3:b0:1cc:5549:aabd with SMTP id q3-20020a17090311c300b001cc5549aabdmr27205343plh.8.1699380621077; Tue, 07 Nov 2023 10:10:21 -0800 (PST) Received: from DT.local (c-71-202-158-162.hsd1.ca.comcast.net. [71.202.158.162]) by smtp.gmail.com with ESMTPSA id az10-20020a170902a58a00b001cc615e6850sm131924plb.90.2023.11.07.10.10.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Nov 2023 10:10:20 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan@outlook.com, jim.harris@samsung.com, Fan Ni Subject: [PATCH v3 5/9] hw/mem/cxl_type3: Add host backend and address space handling for DC regions Date: Tue, 7 Nov 2023 10:07:09 -0800 Message-ID: <20231107180907.553451-6-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231107180907.553451-1-nifan.cxl@gmail.com> References: <20231107180907.553451-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Add (file/memory backed) host backend, all the dynamic capacity regions will share a single, large enough host backend. Set up address space for DC regions to support read/write operations to dynamic capacity for DCD. With the change, following supports are added: 1. Add a new property to type3 device "nonvolatile-dc-memdev" to point to host memory backend for dynamic capacity. Currently, all dc regions share one one host backend. 2. Add namespace for dynamic capacity for read/write support; 3. Create cdat entries for each dynamic capacity region; 4. Fix dvsec range registers to include DC regions. Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 16 ++- hw/mem/cxl_type3.c | 198 +++++++++++++++++++++++++++++------- include/hw/cxl/cxl_device.h | 4 + 3 files changed, 179 insertions(+), 39 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 707fd9fe7f..1f512b3e6b 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -596,7 +596,8 @@ static CXLRetCode cmd_firmware_update_get_info(const struct cxl_cmd *cmd, size_t *len_out, CXLCCI *cci) { - CXLDeviceState *cxl_dstate = &CXL_TYPE3(cci->d)->cxl_dstate; + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + CXLDeviceState *cxl_dstate = &ct3d->cxl_dstate; struct { uint8_t slots_supported; uint8_t slot_info; @@ -610,7 +611,8 @@ static CXLRetCode cmd_firmware_update_get_info(const struct cxl_cmd *cmd, QEMU_BUILD_BUG_ON(sizeof(*fw_info) != 0x50); if ((cxl_dstate->vmem_size < CXL_CAPACITY_MULTIPLIER) || - (cxl_dstate->pmem_size < CXL_CAPACITY_MULTIPLIER)) { + (cxl_dstate->pmem_size < CXL_CAPACITY_MULTIPLIER) || + (ct3d->dc.total_capacity < CXL_CAPACITY_MULTIPLIER)) { return CXL_MBOX_INTERNAL_ERROR; } @@ -764,7 +766,8 @@ static CXLRetCode cmd_identify_memory_device(const struct cxl_cmd *cmd, CXLDeviceState *cxl_dstate = &ct3d->cxl_dstate; if ((!QEMU_IS_ALIGNED(cxl_dstate->vmem_size, CXL_CAPACITY_MULTIPLIER)) || - (!QEMU_IS_ALIGNED(cxl_dstate->pmem_size, CXL_CAPACITY_MULTIPLIER))) { + (!QEMU_IS_ALIGNED(cxl_dstate->pmem_size, CXL_CAPACITY_MULTIPLIER)) || + (!QEMU_IS_ALIGNED(ct3d->dc.total_capacity, CXL_CAPACITY_MULTIPLIER))) { return CXL_MBOX_INTERNAL_ERROR; } @@ -805,9 +808,11 @@ static CXLRetCode cmd_ccls_get_partition_info(const struct cxl_cmd *cmd, uint64_t next_pmem; } QEMU_PACKED *part_info = (void *)payload_out; QEMU_BUILD_BUG_ON(sizeof(*part_info) != 0x20); + CXLType3Dev *ct3d = container_of(cxl_dstate, CXLType3Dev, cxl_dstate); if ((!QEMU_IS_ALIGNED(cxl_dstate->vmem_size, CXL_CAPACITY_MULTIPLIER)) || - (!QEMU_IS_ALIGNED(cxl_dstate->pmem_size, CXL_CAPACITY_MULTIPLIER))) { + (!QEMU_IS_ALIGNED(cxl_dstate->pmem_size, CXL_CAPACITY_MULTIPLIER)) || + (!QEMU_IS_ALIGNED(ct3d->dc.total_capacity, CXL_CAPACITY_MULTIPLIER))) { return CXL_MBOX_INTERNAL_ERROR; } @@ -1149,7 +1154,8 @@ static CXLRetCode cmd_media_clear_poison(const struct cxl_cmd *cmd, struct clear_poison_pl *in = (void *)payload_in; dpa = ldq_le_p(&in->dpa); - if (dpa + CXL_CACHE_LINE_SIZE > cxl_dstate->static_mem_size) { + if (dpa + CXL_CACHE_LINE_SIZE > cxl_dstate->static_mem_size + + ct3d->dc.total_capacity) { return CXL_MBOX_INVALID_PA; } diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 2d67d2015c..152a51306d 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -31,6 +31,7 @@ #include "hw/pci/spdm.h" #define DWORD_BYTE 4 +#define CXL_CAPACITY_MULTIPLIER (256 * MiB) /* Default CDAT entries for a memory region */ enum { @@ -44,8 +45,9 @@ enum { }; static int ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, - int dsmad_handle, MemoryRegion *mr, - bool is_pmem, uint64_t dpa_base) + int dsmad_handle, uint64_t size, + bool is_pmem, bool is_dynamic, + uint64_t dpa_base) { g_autofree CDATDsmas *dsmas = NULL; g_autofree CDATDslbis *dslbis0 = NULL; @@ -64,9 +66,10 @@ static int ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, .length = sizeof(*dsmas), }, .DSMADhandle = dsmad_handle, - .flags = is_pmem ? CDAT_DSMAS_FLAG_NV : 0, + .flags = (is_pmem ? CDAT_DSMAS_FLAG_NV : 0) | + (is_dynamic ? CDAT_DSMAS_FLAG_DYNAMIC_CAP : 0), .DPA_base = dpa_base, - .DPA_length = memory_region_size(mr), + .DPA_length = size, }; /* For now, no memory side cache, plausiblish numbers */ @@ -150,7 +153,7 @@ static int ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, */ .EFI_memory_type_attr = is_pmem ? 2 : 1, .DPA_offset = 0, - .DPA_length = memory_region_size(mr), + .DPA_length = size, }; /* Header always at start of structure */ @@ -169,21 +172,28 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) g_autofree CDATSubHeader **table = NULL; CXLType3Dev *ct3d = priv; MemoryRegion *volatile_mr = NULL, *nonvolatile_mr = NULL; + MemoryRegion *dc_mr = NULL; int dsmad_handle = 0; int cur_ent = 0; int len = 0; int rc, i; + uint64_t vmr_size = 0, pmr_size = 0; - if (!ct3d->hostpmem && !ct3d->hostvmem) { + if (!ct3d->hostpmem && !ct3d->hostvmem && !ct3d->dc.num_regions) { return 0; } + if (ct3d->hostpmem && ct3d->hostvmem && ct3d->dc.host_dc) { + warn_report("The device has static ram and pmem and dynamic capacity"); + } + if (ct3d->hostvmem) { volatile_mr = host_memory_backend_get_memory(ct3d->hostvmem); if (!volatile_mr) { return -EINVAL; } len += CT3_CDAT_NUM_ENTRIES; + vmr_size = memory_region_size(volatile_mr); } if (ct3d->hostpmem) { @@ -192,6 +202,19 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) return -EINVAL; } len += CT3_CDAT_NUM_ENTRIES; + pmr_size = memory_region_size(nonvolatile_mr); + } + + if (ct3d->dc.num_regions) { + if (ct3d->dc.host_dc) { + dc_mr = host_memory_backend_get_memory(ct3d->dc.host_dc); + if (!dc_mr) { + return -EINVAL; + } + len += CT3_CDAT_NUM_ENTRIES * ct3d->dc.num_regions; + } else { + return -EINVAL; + } } table = g_malloc0(len * sizeof(*table)); @@ -201,8 +224,8 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) /* Now fill them in */ if (volatile_mr) { - rc = ct3_build_cdat_entries_for_mr(table, dsmad_handle++, volatile_mr, - false, 0); + rc = ct3_build_cdat_entries_for_mr(table, dsmad_handle++, vmr_size, + false, false, 0); if (rc < 0) { return rc; } @@ -210,14 +233,38 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) } if (nonvolatile_mr) { - uint64_t base = volatile_mr ? memory_region_size(volatile_mr) : 0; rc = ct3_build_cdat_entries_for_mr(&(table[cur_ent]), dsmad_handle++, - nonvolatile_mr, true, base); + pmr_size, true, false, vmr_size); if (rc < 0) { goto error_cleanup; } cur_ent += CT3_CDAT_NUM_ENTRIES; } + + if (dc_mr) { + uint64_t region_base = vmr_size + pmr_size; + + /* + * Currently we create cdat entries for each region, should we only + * create dsmas table instead?? + * We assume all dc regions are non-volatile for now. + * + */ + for (i = 0; i < ct3d->dc.num_regions; i++) { + rc = ct3_build_cdat_entries_for_mr(&(table[cur_ent]), + dsmad_handle++, + ct3d->dc.regions[i].len, + true, true, region_base); + if (rc < 0) { + goto error_cleanup; + } + ct3d->dc.regions[i].dsmadhandle = dsmad_handle - 1; + + cur_ent += CT3_CDAT_NUM_ENTRIES; + region_base += ct3d->dc.regions[i].len; + } + } + assert(len == cur_ent); *cdat_table = g_steal_pointer(&table); @@ -445,11 +492,24 @@ static void build_dvsecs(CXLType3Dev *ct3d) range2_size_hi = ct3d->hostpmem->size >> 32; range2_size_lo = (2 << 5) | (2 << 2) | 0x3 | (ct3d->hostpmem->size & 0xF0000000); + } else if (ct3d->dc.host_dc) { + range2_size_hi = ct3d->dc.host_dc->size >> 32; + range2_size_lo = (2 << 5) | (2 << 2) | 0x3 | + (ct3d->dc.host_dc->size & 0xF0000000); } - } else { + } else if (ct3d->hostpmem) { range1_size_hi = ct3d->hostpmem->size >> 32; range1_size_lo = (2 << 5) | (2 << 2) | 0x3 | (ct3d->hostpmem->size & 0xF0000000); + if (ct3d->dc.host_dc) { + range2_size_hi = ct3d->dc.host_dc->size >> 32; + range2_size_lo = (2 << 5) | (2 << 2) | 0x3 | + (ct3d->dc.host_dc->size & 0xF0000000); + } + } else { + range1_size_hi = ct3d->dc.host_dc->size >> 32; + range1_size_lo = (2 << 5) | (2 << 2) | 0x3 | + (ct3d->dc.host_dc->size & 0xF0000000); } dvsec = (uint8_t *)&(CXLDVSECDevice){ @@ -721,6 +781,9 @@ static void ct3d_reg_write(void *opaque, hwaddr offset, uint64_t value, } } +/* + * TODO: region parameters are hard coded, may need to change in the future. + */ static int cxl_create_dc_regions(CXLType3Dev *ct3d) { int i; @@ -736,6 +799,7 @@ static int cxl_create_dc_regions(CXLType3Dev *ct3d) if (ct3d->hostpmem) { region_base += ct3d->hostpmem->size; } + for (i = 0; i < ct3d->dc.num_regions; i++) { region = &ct3d->dc.regions[i]; region->base = region_base; @@ -755,7 +819,8 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) { DeviceState *ds = DEVICE(ct3d); - if (!ct3d->hostmem && !ct3d->hostvmem && !ct3d->hostpmem) { + if (!ct3d->hostmem && !ct3d->hostvmem && !ct3d->hostpmem + && !ct3d->dc.num_regions) { error_setg(errp, "at least one memdev property must be set"); return false; } else if (ct3d->hostmem && ct3d->hostpmem) { @@ -823,6 +888,50 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) return false; } + ct3d->dc.total_capacity = 0; + if (ct3d->dc.host_dc) { + MemoryRegion *dc_mr; + char *dc_name; + uint64_t total_region_size = 0; + int i; + + dc_mr = host_memory_backend_get_memory(ct3d->dc.host_dc); + if (!dc_mr) { + error_setg(errp, "dynamic capacity must have backing device"); + return false; + } + /* FIXME: set dc as nonvolatile for now */ + memory_region_set_nonvolatile(dc_mr, true); + memory_region_set_enabled(dc_mr, true); + host_memory_backend_set_mapped(ct3d->dc.host_dc, true); + if (ds->id) { + dc_name = g_strdup_printf("cxl-dcd-dpa-dc-space:%s", ds->id); + } else { + dc_name = g_strdup("cxl-dcd-dpa-dc-space"); + } + address_space_init(&ct3d->dc.host_dc_as, dc_mr, dc_name); + g_free(dc_name); + + for (i = 0; i < ct3d->dc.num_regions; i++) { + total_region_size += ct3d->dc.regions[i].len; + } + /* Make sure the host backend is large enough to cover all dc range */ + if (total_region_size > memory_region_size(dc_mr)) { + error_setg(errp, + "too small host backend size, increase to %lu MiB or more", + total_region_size / MiB); + return false; + } + + if (dc_mr->size % CXL_CAPACITY_MULTIPLIER != 0) { + error_setg(errp, "DC region size is unaligned to %lx", + CXL_CAPACITY_MULTIPLIER); + return false; + } + + ct3d->dc.total_capacity = total_region_size; + } + return true; } @@ -933,6 +1042,9 @@ err_release_cdat: err_free_special_ops: g_free(regs->special_ops); err_address_space_free: + if (ct3d->dc.host_dc) { + address_space_destroy(&ct3d->dc.host_dc_as); + } if (ct3d->hostpmem) { address_space_destroy(&ct3d->hostpmem_as); } @@ -952,6 +1064,9 @@ static void ct3_exit(PCIDevice *pci_dev) cxl_doe_cdat_release(cxl_cstate); spdm_sock_fini(ct3d->doe_spdm.socket); g_free(regs->special_ops); + if (ct3d->dc.host_dc) { + address_space_destroy(&ct3d->dc.host_dc_as); + } if (ct3d->hostpmem) { address_space_destroy(&ct3d->hostpmem_as); } @@ -1025,16 +1140,24 @@ static int cxl_type3_hpa_to_as_and_dpa(CXLType3Dev *ct3d, AddressSpace **as, uint64_t *dpa_offset) { - MemoryRegion *vmr = NULL, *pmr = NULL; + MemoryRegion *vmr = NULL, *pmr = NULL, *dc_mr = NULL; + uint64_t vmr_size = 0, pmr_size = 0, dc_size = 0; if (ct3d->hostvmem) { vmr = host_memory_backend_get_memory(ct3d->hostvmem); + vmr_size = memory_region_size(vmr); } if (ct3d->hostpmem) { pmr = host_memory_backend_get_memory(ct3d->hostpmem); + pmr_size = memory_region_size(pmr); + } + if (ct3d->dc.host_dc) { + dc_mr = host_memory_backend_get_memory(ct3d->dc.host_dc); + /* Do we want dc_size to be dc_mr->size or not?? */ + dc_size = ct3d->dc.total_capacity; } - if (!vmr && !pmr) { + if (!vmr && !pmr && !dc_mr) { return -ENODEV; } @@ -1042,19 +1165,18 @@ static int cxl_type3_hpa_to_as_and_dpa(CXLType3Dev *ct3d, return -EINVAL; } - if (*dpa_offset > ct3d->cxl_dstate.static_mem_size) { + if (*dpa_offset >= vmr_size + pmr_size + dc_size) { return -EINVAL; } - if (vmr) { - if (*dpa_offset < memory_region_size(vmr)) { - *as = &ct3d->hostvmem_as; - } else { - *as = &ct3d->hostpmem_as; - *dpa_offset -= memory_region_size(vmr); - } - } else { + if (*dpa_offset < vmr_size) { + *as = &ct3d->hostvmem_as; + } else if (*dpa_offset < vmr_size + pmr_size) { *as = &ct3d->hostpmem_as; + *dpa_offset -= vmr_size; + } else { + *as = &ct3d->dc.host_dc_as; + *dpa_offset -= (vmr_size + pmr_size); } return 0; @@ -1143,6 +1265,8 @@ static Property ct3_props[] = { DEFINE_PROP_STRING("cdat", CXLType3Dev, cxl_cstate.cdat.filename), DEFINE_PROP_UINT16("spdm", CXLType3Dev, spdm_port, 0), DEFINE_PROP_UINT8("num-dc-regions", CXLType3Dev, dc.num_regions, 0), + DEFINE_PROP_LINK("nonvolatile-dc-memdev", CXLType3Dev, dc.host_dc, + TYPE_MEMORY_BACKEND, HostMemoryBackend *), DEFINE_PROP_END_OF_LIST(), }; @@ -1209,33 +1333,39 @@ static void set_lsa(CXLType3Dev *ct3d, const void *buf, uint64_t size, static bool set_cacheline(CXLType3Dev *ct3d, uint64_t dpa_offset, uint8_t *data) { - MemoryRegion *vmr = NULL, *pmr = NULL; + MemoryRegion *vmr = NULL, *pmr = NULL, *dc_mr = NULL; AddressSpace *as; + uint64_t vmr_size = 0, pmr_size = 0, dc_size = 0; if (ct3d->hostvmem) { vmr = host_memory_backend_get_memory(ct3d->hostvmem); + vmr_size = memory_region_size(vmr); } if (ct3d->hostpmem) { pmr = host_memory_backend_get_memory(ct3d->hostpmem); + pmr_size = memory_region_size(pmr); } + if (ct3d->dc.host_dc) { + dc_mr = host_memory_backend_get_memory(ct3d->dc.host_dc); + dc_size = ct3d->dc.total_capacity; + } - if (!vmr && !pmr) { + if (!vmr && !pmr && !dc_mr) { return false; } - if (dpa_offset + CXL_CACHE_LINE_SIZE > ct3d->cxl_dstate.static_mem_size) { + if (dpa_offset + CXL_CACHE_LINE_SIZE > vmr_size + pmr_size + dc_size) { return false; } - if (vmr) { - if (dpa_offset < memory_region_size(vmr)) { - as = &ct3d->hostvmem_as; - } else { - as = &ct3d->hostpmem_as; - dpa_offset -= memory_region_size(vmr); - } - } else { + if (dpa_offset < vmr_size) { + as = &ct3d->hostvmem_as; + } else if (dpa_offset < vmr_size + pmr_size) { as = &ct3d->hostpmem_as; + dpa_offset -= vmr_size; + } else { + as = &ct3d->dc.host_dc_as; + dpa_offset -= (vmr_size + pmr_size); } address_space_write(as, dpa_offset, MEMTXATTRS_UNSPECIFIED, &data, diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index de6469eef7..3dc6928bc5 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -467,6 +467,10 @@ struct CXLType3Dev { uint64_t poison_list_overflow_ts; struct dynamic_capacity { + HostMemoryBackend *host_dc; + AddressSpace host_dc_as; + uint64_t total_capacity; /* 256M aligned */ + uint8_t num_regions; /* 0-8 regions */ CXLDCDRegion regions[DCD_MAX_REGION_NUM]; } dc; From patchwork Tue Nov 7 18:07:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13449170 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 076E3358A0 for ; Tue, 7 Nov 2023 18:10:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="HSlWdHVf" Received: from mail-pf1-x42b.google.com (mail-pf1-x42b.google.com [IPv6:2607:f8b0:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9265D120 for ; Tue, 7 Nov 2023 10:10:22 -0800 (PST) Received: by mail-pf1-x42b.google.com with SMTP id d2e1a72fcca58-6c115026985so6102311b3a.1 for ; Tue, 07 Nov 2023 10:10:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1699380622; x=1699985422; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nUVbPa2RXh5sULa717dlwLdK6nK0VOlZVO7MX7gSRww=; b=HSlWdHVfzNkfsusr9OG+tSqD1+N4uckCxZ7l8BXFCaClFJjDsg9JWuQR9bpVola7v1 AN0D5Po/aG6reg6UPYiF0i7XtccU+sLqKAvpYMXiSULMEmrtvbY4jg7WHdoWaJ0gIZMm kyqTbF9zFOUF/gOgg6mpPF2xHb9r3hZKLt3xM7FnxGVnZKKWGC7z9xZzk1IdprHEsh7n yv13RVtvK+GykSnqbzQsXR2qbIXxVluo8xEW4x9/Eb2z6pG7AXrcm3LNwlS6ZJItJE4R hJlrDExNFYrVx/Nzz8LBzS6oiRczDwyG+nQQ69yWHUey9bCEW4wjpwHPIAfdmidQ0auh yIMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699380622; x=1699985422; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nUVbPa2RXh5sULa717dlwLdK6nK0VOlZVO7MX7gSRww=; b=hViS8kqFTSkZGyMKnKd9CIyGYiI2mtGbvh34o0HZFe+bxtlRbpD9dvH/Pv5rIUX456 CYXI4O+Uyw4EFHjRTOOQmf9txbcmWXvm+WCakNDooayS+WyuJ9OcnnrVkA3P/i3DBYbt kxHWDeOwbDd+hMJ2qkmoR8LcOaB6Xok/oa7YTpe54bMUgENehd2xLFWh0IuORvleQpAE 7UsqR9lab/thKc5GHVTAi60ZNDGr0ELvsJb+1qQnG80uEgnKM0cL1kRWdWPzlrAKZr4v lhPHwV1SnmhilxyFSMVljN4eMFxfjBiLCuy43mWpis4Nisx0yjHqiD3qBh5L2AWHaw8M cVhQ== X-Gm-Message-State: AOJu0YxX8cxfVo0eSlTpdPsHCVSYrI4OfyqR0BAbPrNpjAlQa/+czhBy EOsj1tYnSVHcl1fOYK6Y40o= X-Google-Smtp-Source: AGHT+IHbIav+SSkCelT+QQDr+/PZe04ppInU4J8uSOtgCzMCmlf2Dg5sNGz15p4xGCJORRfIXBtO/A== X-Received: by 2002:a05:6a20:3d82:b0:16b:9886:7eda with SMTP id s2-20020a056a203d8200b0016b98867edamr38574558pzi.35.1699380622013; Tue, 07 Nov 2023 10:10:22 -0800 (PST) Received: from DT.local (c-71-202-158-162.hsd1.ca.comcast.net. [71.202.158.162]) by smtp.gmail.com with ESMTPSA id az10-20020a170902a58a00b001cc615e6850sm131924plb.90.2023.11.07.10.10.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Nov 2023 10:10:21 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan@outlook.com, jim.harris@samsung.com, Fan Ni Subject: [PATCH v3 6/9] hw/mem/cxl_type3: Add DC extent list representative and get DC extent list mailbox support Date: Tue, 7 Nov 2023 10:07:10 -0800 Message-ID: <20231107180907.553451-7-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231107180907.553451-1-nifan.cxl@gmail.com> References: <20231107180907.553451-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Add dynamic capacity extent list representative to the definition of CXLType3Dev and add get DC extent list mailbox command per CXL.spec.3.0:.8.2.9.8.9.2. Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 73 +++++++++++++++++++++++++++++++++++++ hw/mem/cxl_type3.c | 1 + include/hw/cxl/cxl_device.h | 23 ++++++++++++ 3 files changed, 97 insertions(+) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 1f512b3e6b..56f4aa237a 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -82,6 +82,7 @@ enum { #define CLEAR_POISON 0x2 DCD_CONFIG = 0x48, #define GET_DC_CONFIG 0x0 + #define GET_DYN_CAP_EXT_LIST 0x1 PHYSICAL_SWITCH = 0x51, #define IDENTIFY_SWITCH_DEVICE 0x0 #define GET_PHYSICAL_PORT_STATE 0x1 @@ -1286,6 +1287,75 @@ static CXLRetCode cmd_dcd_get_dyn_cap_config(const struct cxl_cmd *cmd, return CXL_MBOX_SUCCESS; } +/* + * CXL r3.0 section 8.2.9.8.9.2: + * Get Dynamic Capacity Extent List (Opcode 4810h) + */ +static CXLRetCode cmd_dcd_get_dyn_cap_ext_list(const struct cxl_cmd *cmd, + uint8_t *payload_in, + size_t len_in, + uint8_t *payload_out, + size_t *len_out, + CXLCCI *cci) +{ + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + struct get_dyn_cap_ext_list_in_pl { + uint32_t extent_cnt; + uint32_t start_extent_id; + } QEMU_PACKED; + + struct get_dyn_cap_ext_list_out_pl { + uint32_t count; + uint32_t total_extents; + uint32_t generation_num; + uint8_t rsvd[4]; + CXLDCExtentRaw records[]; + } QEMU_PACKED; + + struct get_dyn_cap_ext_list_in_pl *in = (void *)payload_in; + struct get_dyn_cap_ext_list_out_pl *out = (void *)payload_out; + uint16_t record_count = 0, i = 0, record_done = 0; + CXLDCDExtentList *extent_list = &ct3d->dc.extents; + CXLDCDExtent *ent; + uint16_t out_pl_len; + uint32_t start_extent_id = in->start_extent_id; + + if (start_extent_id > ct3d->dc.total_extent_count) { + return CXL_MBOX_INVALID_INPUT; + } + + record_count = MIN(in->extent_cnt, + ct3d->dc.total_extent_count - start_extent_id); + + out_pl_len = sizeof(*out) + record_count * sizeof(out->records[0]); + /* May need more processing here in the future */ + assert(out_pl_len <= CXL_MAILBOX_MAX_PAYLOAD_SIZE); + + memset(out, 0, out_pl_len); + stl_le_p(&out->count, record_count); + stl_le_p(&out->total_extents, ct3d->dc.total_extent_count); + stl_le_p(&out->generation_num, ct3d->dc.ext_list_gen_seq); + + if (record_count > 0) { + QTAILQ_FOREACH(ent, extent_list, node) { + if (i++ < start_extent_id) { + continue; + } + stq_le_p(&out->records[record_done].start_dpa, ent->start_dpa); + stq_le_p(&out->records[record_done].len, ent->len); + memcpy(&out->records[record_done].tag, ent->tag, 0x10); + stw_le_p(&out->records[record_done].shared_seq, ent->shared_seq); + record_done++; + if (record_done == record_count) { + break; + } + } + } + + *len_out = out_pl_len; + return CXL_MBOX_SUCCESS; +} + #define IMMEDIATE_CONFIG_CHANGE (1 << 1) #define IMMEDIATE_DATA_CHANGE (1 << 2) #define IMMEDIATE_POLICY_CHANGE (1 << 3) @@ -1333,6 +1403,9 @@ static const struct cxl_cmd cxl_cmd_set[256][256] = { static const struct cxl_cmd cxl_cmd_set_dcd[256][256] = { [DCD_CONFIG][GET_DC_CONFIG] = { "DCD_GET_DC_CONFIG", cmd_dcd_get_dyn_cap_config, 2, 0 }, + [DCD_CONFIG][GET_DYN_CAP_EXT_LIST] = { + "DCD_GET_DYNAMIC_CAPACITY_EXTENT_LIST", cmd_dcd_get_dyn_cap_ext_list, + 8, 0 }, }; static const struct cxl_cmd cxl_cmd_set_sw[256][256] = { diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 152a51306d..c9d792a725 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -811,6 +811,7 @@ static int cxl_create_dc_regions(CXLType3Dev *ct3d) region_base += region->len; } + QTAILQ_INIT(&ct3d->dc.extents); return 0; } diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index 3dc6928bc5..5738c6f434 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -420,6 +420,25 @@ typedef QLIST_HEAD(, CXLPoison) CXLPoisonList; #define DCD_MAX_REGION_NUM 8 +typedef struct CXLDCDExtentRaw { + uint64_t start_dpa; + uint64_t len; + uint8_t tag[0x10]; + uint16_t shared_seq; + uint8_t rsvd[0x6]; +} QEMU_PACKED CXLDCExtentRaw; + +typedef struct CXLDCDExtent { + uint64_t start_dpa; + uint64_t len; + uint8_t tag[0x10]; + uint16_t shared_seq; + uint8_t rsvd[0x6]; + + QTAILQ_ENTRY(CXLDCDExtent) node; +} CXLDCDExtent; +typedef QTAILQ_HEAD(, CXLDCDExtent) CXLDCDExtentList; + typedef struct CXLDCDRegion { uint64_t base; uint64_t decode_len; /* aligned to 256*MiB */ @@ -470,6 +489,10 @@ struct CXLType3Dev { HostMemoryBackend *host_dc; AddressSpace host_dc_as; uint64_t total_capacity; /* 256M aligned */ + CXLDCDExtentList extents; + + uint32_t total_extent_count; + uint32_t ext_list_gen_seq; uint8_t num_regions; /* 0-8 regions */ CXLDCDRegion regions[DCD_MAX_REGION_NUM]; From patchwork Tue Nov 7 18:07:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13449172 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5437D358A2 for ; Tue, 7 Nov 2023 18:10:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ScBFOnEe" Received: from mail-pl1-x632.google.com (mail-pl1-x632.google.com [IPv6:2607:f8b0:4864:20::632]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7D2B0B0 for ; Tue, 7 Nov 2023 10:10:23 -0800 (PST) Received: by mail-pl1-x632.google.com with SMTP id d9443c01a7336-1cc0e78ec92so40282245ad.3 for ; Tue, 07 Nov 2023 10:10:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1699380623; x=1699985423; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qUg/SG54upIHtQz6IlNDfZW8H+6HjCMvwHk2jyDUo0c=; b=ScBFOnEegY3ntxOukrfzKdhHk8cqw2F0kzeHTAB3BiYWupsrBNYvOpP+RhkZAhEr05 jKE80hlawoEw/BgaDYl9bY0ZDXmWuqmsl7KKQojfAJVJsfGbmFovJxW/0Iho4cg2xmAf 7hKavJfKb8yZwu694pn4wvyCxwmkK+XHjwXsAZrgTdDWv6d7P/mQ8f+VriRshIEeXY/1 4F5gdbDzCyaEbGPcZlG/u7NYetkCmB5hBQSRsEuEFgV3hcRyxek+uGtOtTNa1/oDTj/J 2PMKDr+wdRTgInJBDY5sU2LzUIxHNC1pn/mWYiULw09/QDXJJlfHHZIrMp99uHPBSvoj NGDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699380623; x=1699985423; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qUg/SG54upIHtQz6IlNDfZW8H+6HjCMvwHk2jyDUo0c=; b=cMxeHyT8GhL9T3LYlgRVyoP7ncqxpcRC7zxuz6Ebp0v42NV4dI0KDRbumfV2nNadlp DIzTpwfoKoGb/SrxxEw7sWhF+mJ9agQiScAb17VLExWKdfyf81FYzGeZQnVAfKLbf9Mr +4GgPfG+TnzRq4l+3fHx5ZjRysc4QG7WBsL70V7yO+7Z5xN+hA164fnPx+ImWprsqQPS wW5nR+FUxu3EgkzcR7WgQZr146yjIqUYCv1A7G0CJH5XMy/0QPgAZUthmdE4QqwN+fiX f973Uy4dqXd/y4DWYfdpKN1xA2aESmQoPFtuqsLL56UikM5FMqHMCBA0CfYQLD+0A6DT KZBg== X-Gm-Message-State: AOJu0YzW4sNPFZkKU7yZX4TTTntwVCg6t2m/RJvm7m2vV9S+DRm5xsU3 fh5zWXkoeIRWTRPzBlo8bG8= X-Google-Smtp-Source: AGHT+IHquAmXxcjrVWrbCx9nCMe2dbsGzG8XJwXYiCLO2ckHcrK2QKzgao7HwzetjQuOJ44GGFU3pg== X-Received: by 2002:a17:903:1210:b0:1c9:e4f2:a39d with SMTP id l16-20020a170903121000b001c9e4f2a39dmr34722916plh.49.1699380622885; Tue, 07 Nov 2023 10:10:22 -0800 (PST) Received: from DT.local (c-71-202-158-162.hsd1.ca.comcast.net. [71.202.158.162]) by smtp.gmail.com with ESMTPSA id az10-20020a170902a58a00b001cc615e6850sm131924plb.90.2023.11.07.10.10.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Nov 2023 10:10:22 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan@outlook.com, jim.harris@samsung.com, Fan Ni Subject: [PATCH v3 7/9] hw/cxl/cxl-mailbox-utils: Add mailbox commands to support add/release dynamic capacity response Date: Tue, 7 Nov 2023 10:07:11 -0800 Message-ID: <20231107180907.553451-8-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231107180907.553451-1-nifan.cxl@gmail.com> References: <20231107180907.553451-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Per CXL spec 3.0, two mailbox commands are implemented: Add Dynamic Capacity Response (Opcode 4802h) 8.2.9.8.9.3, and Release Dynamic Capacity (Opcode 4803h) 8.2.9.8.9.4. Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 271 ++++++++++++++++++++++++++++++++++++ hw/mem/cxl_type3.c | 3 +- include/hw/cxl/cxl_device.h | 5 +- 3 files changed, 277 insertions(+), 2 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 56f4aa237a..9f788b03b6 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -83,6 +83,8 @@ enum { DCD_CONFIG = 0x48, #define GET_DC_CONFIG 0x0 #define GET_DYN_CAP_EXT_LIST 0x1 + #define ADD_DYN_CAP_RSP 0x2 + #define RELEASE_DYN_CAP 0x3 PHYSICAL_SWITCH = 0x51, #define IDENTIFY_SWITCH_DEVICE 0x0 #define GET_PHYSICAL_PORT_STATE 0x1 @@ -1356,6 +1358,269 @@ static CXLRetCode cmd_dcd_get_dyn_cap_ext_list(const struct cxl_cmd *cmd, return CXL_MBOX_SUCCESS; } +/* + * Check whether any bit between addr[nr, nr+size) is set, + * return true if any bit is set, otherwise return false + */ +static bool test_any_bits_set(const unsigned long *addr, int nr, int size) +{ + unsigned long res = find_next_bit(addr, size + nr, nr); + + return res < nr + size; +} + +CXLDCDRegion *cxl_find_dc_region(CXLType3Dev *ct3d, uint64_t dpa, uint64_t len) +{ + CXLDCDRegion *region = &ct3d->dc.regions[0]; + int i; + + if (dpa < region->base || + dpa >= region->base + ct3d->dc.total_capacity) { + return NULL; + } + + /* + * CXL r3.0 section 9.13.3: Dynamic Capacity Device (DCD) + * + * Regions are used in increasing-DPA order, with Region 0 being used for + * the lowest DPA of Dynamic Capacity and Region 7 for the highest DPA. + * So check from the last region to find where the dpa belongs. Extents that + * cross multiple regions are not allowed. + */ + for (i = ct3d->dc.num_regions - 1; i >= 0; i--) { + region = &ct3d->dc.regions[i]; + if (dpa >= region->base) { + /*Should we compare with decode_len or len of the region??*/ + if (dpa + len > region->base + + region->decode_len * CXL_CAPACITY_MULTIPLIER) + return NULL; + return region; + } + } + return NULL; +} + +static void cxl_insert_extent_to_extent_list(CXLDCDExtentList *list, + uint64_t dpa, + uint64_t len, + uint8_t *tag, + uint16_t shared_seq) +{ + CXLDCDExtent *extent; + + extent = g_new0(CXLDCDExtent, 1); + extent->start_dpa = dpa; + extent->len = len; + if (tag) { + memcpy(extent->tag, tag, 0x10); + } else { + memset(extent->tag, 0, 0x10); + } + extent->shared_seq = shared_seq; + + QTAILQ_INSERT_TAIL(list, extent, node); +} + +/* + * CXL r3.0 Table 8-129: Add Dynamic Capacity Response Input Payload + * CXL r3.0 Table 8-131: Release Dynamic Capacity Input Payload + */ +typedef struct updated_dc_extent_list_in_pl { + uint32_t num_entries_updated; + uint8_t rsvd[4]; + /* CXL r3.0 Table 8-130: Updated Extent List */ + struct { + uint64_t start_dpa; + uint64_t len; + uint8_t rsvd[8]; + } QEMU_PACKED updated_entries[]; +} QEMU_PACKED updated_dc_extent_list_in_pl; + +/* + * For the extents in the extent list to operate, check whether they are valid + * 1. The extent should be in the range of a valid DC region; + * 2. The extent should not cross multiple regions; + * 3. The start DPA and the length of the extent should align with the block + * size of the region; + * 4. The address range of multiple extents in the list should not overlap. + */ +static CXLRetCode cxl_detect_malformed_extent_list(CXLType3Dev *ct3d, + const updated_dc_extent_list_in_pl *in) +{ + uint64_t min_block_size = UINT64_MAX; + CXLDCDRegion *region = &ct3d->dc.regions[0]; + CXLDCDRegion *lastregion = &ct3d->dc.regions[ct3d->dc.num_regions - 1]; + g_autofree unsigned long *blk_bitmap = NULL; + uint64_t dpa, len; + uint32_t i; + + for (i = 0; i < ct3d->dc.num_regions; i++) { + region = &ct3d->dc.regions[i]; + min_block_size = MIN(min_block_size, region->block_size); + } + + blk_bitmap = bitmap_new((lastregion->len + lastregion->base - + ct3d->dc.regions[0].base) / min_block_size); + + for (i = 0; i < in->num_entries_updated; i++) { + dpa = in->updated_entries[i].start_dpa; + len = in->updated_entries[i].len; + + region = cxl_find_dc_region(ct3d, dpa, len); + if (!region) { + return CXL_MBOX_INVALID_PA; + } + + dpa -= ct3d->dc.regions[0].base; + if (dpa % region->block_size || len % region->block_size) { + return CXL_MBOX_INVALID_EXTENT_LIST; + } + /* the dpa range already covered by some other extents in the list */ + if (test_any_bits_set(blk_bitmap, dpa / min_block_size, + len / min_block_size)) { + return CXL_MBOX_INVALID_EXTENT_LIST; + } + bitmap_set(blk_bitmap, dpa / min_block_size, len / min_block_size); + } + + return CXL_MBOX_SUCCESS; +} + +/* + * CXL r3.0 section 8.2.9.8.9.3: Add Dynamic Capacity Response (opcode 4802h) + * + * Assume an extent is added only after the response is processed successfully + * TODO: for better extent list validation, a better solution would be + * maintaining a pending extent list and use it to verify the extent list in + * the response. + */ +static CXLRetCode cmd_dcd_add_dyn_cap_rsp(const struct cxl_cmd *cmd, + uint8_t *payload_in, + size_t len_in, + uint8_t *payload_out, + size_t *len_out, + CXLCCI *cci) +{ + updated_dc_extent_list_in_pl *in = (void *)payload_in; + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + CXLDCDExtentList *extent_list = &ct3d->dc.extents; + CXLDCDExtent *ent; + uint32_t i; + uint64_t dpa, len; + CXLRetCode ret; + + if (in->num_entries_updated == 0) { + return CXL_MBOX_SUCCESS; + } + + ret = cxl_detect_malformed_extent_list(ct3d, in); + if (ret != CXL_MBOX_SUCCESS) { + return ret; + } + + for (i = 0; i < in->num_entries_updated; i++) { + dpa = in->updated_entries[i].start_dpa; + len = in->updated_entries[i].len; + + /* + * Check if the DPA range of the to-be-added extent overlaps with + * existing extent list maintained by the device. + */ + QTAILQ_FOREACH(ent, extent_list, node) { + if (ent->start_dpa <= dpa && + dpa + len <= ent->start_dpa + ent->len) { + return CXL_MBOX_INVALID_PA; + /* Overlapping one end of the other */ + } else if ((dpa < ent->start_dpa + ent->len && + dpa + len > ent->start_dpa + ent->len) || + (dpa < ent->start_dpa && dpa + len > ent->start_dpa)) { + return CXL_MBOX_INVALID_PA; + } + } + + /* + * TODO: add a pending extent list based on event log record and + * verify the input response + */ + + cxl_insert_extent_to_extent_list(extent_list, dpa, len, NULL, 0); + ct3d->dc.total_extent_count += 1; + } + + return CXL_MBOX_SUCCESS; +} + +/* + * CXL r3.0 section 8.2.9.8.9.4: Release Dynamic Capacity (opcode 4803h) + */ +static CXLRetCode cmd_dcd_release_dyn_cap(const struct cxl_cmd *cmd, + uint8_t *payload_in, + size_t len_in, + uint8_t *payload_out, + size_t *len_out, + CXLCCI *cci) +{ + updated_dc_extent_list_in_pl *in = (void *)payload_in; + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + CXLDCDExtentList *extent_list = &ct3d->dc.extents; + CXLDCDExtent *ent; + uint32_t i; + uint64_t dpa, len; + CXLRetCode ret; + + if (in->num_entries_updated == 0) { + return CXL_MBOX_INVALID_INPUT; + } + + ret = cxl_detect_malformed_extent_list(ct3d, in); + if (ret != CXL_MBOX_SUCCESS) { + return ret; + } + + for (i = 0; i < in->num_entries_updated; i++) { + dpa = in->updated_entries[i].start_dpa; + len = in->updated_entries[i].len; + + QTAILQ_FOREACH(ent, extent_list, node) { + if (ent->start_dpa <= dpa && + dpa + len <= ent->start_dpa + ent->len) { + /* Remove any partial extents */ + uint64_t len1 = dpa - ent->start_dpa; + uint64_t len2 = ent->start_dpa + ent->len - dpa - len; + + if (len1) { + cxl_insert_extent_to_extent_list(extent_list, + ent->start_dpa, len1, + NULL, 0); + ct3d->dc.total_extent_count += 1; + } + if (len2) { + cxl_insert_extent_to_extent_list(extent_list, dpa + len, + len2, NULL, 0); + ct3d->dc.total_extent_count += 1; + } + break; + /*Currently we reject the attempt to remove a superset*/ + } else if ((dpa < ent->start_dpa + ent->len && + dpa + len > ent->start_dpa + ent->len) || + (dpa < ent->start_dpa && dpa + len > ent->start_dpa)) { + return CXL_MBOX_INVALID_EXTENT_LIST; + } + } + + if (ent) { + QTAILQ_REMOVE(extent_list, ent, node); + g_free(ent); + ct3d->dc.total_extent_count -= 1; + } else { + /* Try to remove a non-existing extent */ + return CXL_MBOX_INVALID_PA; + } + } + + return CXL_MBOX_SUCCESS; +} + #define IMMEDIATE_CONFIG_CHANGE (1 << 1) #define IMMEDIATE_DATA_CHANGE (1 << 2) #define IMMEDIATE_POLICY_CHANGE (1 << 3) @@ -1406,6 +1671,12 @@ static const struct cxl_cmd cxl_cmd_set_dcd[256][256] = { [DCD_CONFIG][GET_DYN_CAP_EXT_LIST] = { "DCD_GET_DYNAMIC_CAPACITY_EXTENT_LIST", cmd_dcd_get_dyn_cap_ext_list, 8, 0 }, + [DCD_CONFIG][ADD_DYN_CAP_RSP] = { + "ADD_DCD_DYNAMIC_CAPACITY_RESPONSE", cmd_dcd_add_dyn_cap_rsp, + ~0, IMMEDIATE_DATA_CHANGE }, + [DCD_CONFIG][RELEASE_DYN_CAP] = { + "RELEASE_DCD_DYNAMIC_CAPACITY", cmd_dcd_release_dyn_cap, + ~0, IMMEDIATE_DATA_CHANGE }, }; static const struct cxl_cmd cxl_cmd_set_sw[256][256] = { diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index c9d792a725..482329a499 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -789,7 +789,7 @@ static int cxl_create_dc_regions(CXLType3Dev *ct3d) int i; uint64_t region_base = 0; uint64_t region_len = 2 * GiB; - uint64_t decode_len = 8; /* 8*256MB */ + uint64_t decode_len = 2 * GiB; uint64_t blk_size = 2 * MiB; CXLDCDRegion *region; @@ -803,6 +803,7 @@ static int cxl_create_dc_regions(CXLType3Dev *ct3d) for (i = 0; i < ct3d->dc.num_regions; i++) { region = &ct3d->dc.regions[i]; region->base = region_base; + /* NOTE: Should be divided by 256 * MiB before be returned to host */ region->decode_len = decode_len; region->len = region_len; region->block_size = blk_size; diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index 5738c6f434..b3d35fe000 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -130,7 +130,8 @@ typedef enum { CXL_MBOX_INCORRECT_PASSPHRASE = 0x14, CXL_MBOX_UNSUPPORTED_MAILBOX = 0x15, CXL_MBOX_INVALID_PAYLOAD_LENGTH = 0x16, - CXL_MBOX_MAX = 0x17 + CXL_MBOX_INVALID_EXTENT_LIST = 0x1E, /* cxl r3.0: Table 8-34*/ + CXL_MBOX_MAX = 0x1F } CXLRetCode; typedef struct CXLCCI CXLCCI; @@ -548,4 +549,6 @@ void cxl_event_irq_assert(CXLType3Dev *ct3d); void cxl_set_poison_list_overflowed(CXLType3Dev *ct3d); +CXLDCDRegion *cxl_find_dc_region(CXLType3Dev *ct3d, uint64_t dpa, uint64_t len); + #endif From patchwork Tue Nov 7 18:07:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13449173 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3FA343589E for ; Tue, 7 Nov 2023 18:10:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="VwqGKmIn" Received: from mail-pl1-x631.google.com (mail-pl1-x631.google.com [IPv6:2607:f8b0:4864:20::631]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7EE03C0 for ; Tue, 7 Nov 2023 10:10:24 -0800 (PST) Received: by mail-pl1-x631.google.com with SMTP id d9443c01a7336-1cc316ccc38so48703035ad.1 for ; Tue, 07 Nov 2023 10:10:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1699380624; x=1699985424; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4lhR5l4PvKeDTkn4MSXEnhevDWdoFM18L2lwlXsuKYY=; b=VwqGKmInIqFD8A+KcYrRs2+wKGeEqN1ePaFeFNM1h5CN+rWN9nxrBlRYXew09ieCZF 4D8ta+mVd36Zu7mYikCuhYV1eSnoIgGomJusjvbTMNvwrxsjw2rvYZWpMcinq1XLahnl ouWwSuyLpnOwCTm3cP7E6YskhpiIL9zGAuQtgD36FyRLrpe35K7lNIkP+2lHfKDiyfZ1 jGdg4J8je56EvNrHztI24E+u7bg0yBJ88yNdniLNPmw6IO4FIZ0MaGzSLUNe8ArC07oj gZAKhFhu6t0f+noxJpMBNS0cFc4ZW1Us618y7Fb6RY9n+xMPZi1PX5BT993tDKKecfJT wOBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699380624; x=1699985424; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4lhR5l4PvKeDTkn4MSXEnhevDWdoFM18L2lwlXsuKYY=; b=mT4lLX4CGD2GPF2/Fef0z0v54vOAoHBNZPhYGEuX0l9cBN/kfqxjnqQatGWJyvefwQ pMIAkgWnKN+Ufb173CHw31DWpTrFq9p57xZclo6OTjrF8WJb5tHv0Y/s5lZ4kfyd3pit hx8gLSBLMq2F+0MI/PIQiAXs6oqQ4xy8+pyKLeEioy5fXrhybSSdaOIP+iCeiaIMzhXT I4CCQYgN4Ub6QVNVDZ6BLEAtF4xwhSe6GRBbLNua7hcMsmgdlB6jodHMJcU38s1XfbL2 vCU0G51G3xOZvOsyXJArmGr0b7dE/uVoaTKIPgHuG/xFbajFU5VihQnhK6OCpKeRT5OZ 8BaQ== X-Gm-Message-State: AOJu0YztJFkWcNFpzh8GnhV25m5TNRZ1iGyOJRAt0S9L+31dLQVAAVRu To4VWycT4Uh5sJO8oZhuuzI= X-Google-Smtp-Source: AGHT+IH7Yev1hBFrxPbTSy5qpYNKvsSQwQ3Ee8RWNDp/nsCivGGFhd0KNLBE/oO1T2WzjmzRvT5sJQ== X-Received: by 2002:a17:902:e84b:b0:1cc:3825:7ba2 with SMTP id t11-20020a170902e84b00b001cc38257ba2mr29811079plg.4.1699380623831; Tue, 07 Nov 2023 10:10:23 -0800 (PST) Received: from DT.local (c-71-202-158-162.hsd1.ca.comcast.net. [71.202.158.162]) by smtp.gmail.com with ESMTPSA id az10-20020a170902a58a00b001cc615e6850sm131924plb.90.2023.11.07.10.10.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Nov 2023 10:10:23 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan@outlook.com, jim.harris@samsung.com, Fan Ni Subject: [PATCH v3 8/9] hw/cxl/events: Add qmp interfaces to add/release dynamic capacity extents Date: Tue, 7 Nov 2023 10:07:12 -0800 Message-ID: <20231107180907.553451-9-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231107180907.553451-1-nifan.cxl@gmail.com> References: <20231107180907.553451-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Since fabric manager emulation is not supported yet, the change implements the functions to add/release dynamic capacity extents as QMP interfaces. Note: we block any FM issued extent release request if the exact extent does not exist in the extent list of the device. We will loose the restriction later once we have partial release support in the kernel. 1. Add dynamic capacity extents: For example, the command to add two continuous extents (each 128MiB long) to region 0 (starting at DPA offset 0) looks like below: { "execute": "qmp_capabilities" } { "execute": "cxl-add-dynamic-capacity", "arguments": { "path": "/machine/peripheral/cxl-dcd0", "region-id": 0, "extents": [ { "dpa": 0, "len": 128 }, { "dpa": 128, "len": 128 } ] } } 2. Release dynamic capacity extents: For example, the command to release an extent of size 128MiB from region 0 (DPA offset 128MiB) look like below: { "execute": "cxl-release-dynamic-capacity", "arguments": { "path": "/machine/peripheral/cxl-dcd0", "region-id": 0, "extents": [ { "dpa": 128, "len": 128 } ] } } Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 25 +++- hw/mem/cxl_type3.c | 225 +++++++++++++++++++++++++++++++++++- hw/mem/cxl_type3_stubs.c | 14 +++ include/hw/cxl/cxl_device.h | 8 +- include/hw/cxl/cxl_events.h | 15 +++ qapi/cxl.json | 60 +++++++++- 6 files changed, 338 insertions(+), 9 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 9f788b03b6..8e6a98753a 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -1362,7 +1362,7 @@ static CXLRetCode cmd_dcd_get_dyn_cap_ext_list(const struct cxl_cmd *cmd, * Check whether any bit between addr[nr, nr+size) is set, * return true if any bit is set, otherwise return false */ -static bool test_any_bits_set(const unsigned long *addr, int nr, int size) +bool test_any_bits_set(const unsigned long *addr, int nr, int size) { unsigned long res = find_next_bit(addr, size + nr, nr); @@ -1400,7 +1400,7 @@ CXLDCDRegion *cxl_find_dc_region(CXLType3Dev *ct3d, uint64_t dpa, uint64_t len) return NULL; } -static void cxl_insert_extent_to_extent_list(CXLDCDExtentList *list, +void cxl_insert_extent_to_extent_list(CXLDCDExtentList *list, uint64_t dpa, uint64_t len, uint8_t *tag, @@ -1538,15 +1538,28 @@ static CXLRetCode cmd_dcd_add_dyn_cap_rsp(const struct cxl_cmd *cmd, } } - /* - * TODO: add a pending extent list based on event log record and - * verify the input response - */ + QTAILQ_FOREACH(ent, &ct3d->dc.extents_pending_to_add, node) { + if (ent->start_dpa <= dpa && + dpa + len <= ent->start_dpa + ent->len) { + break; + } + } + if (ent) { + QTAILQ_REMOVE(&ct3d->dc.extents_pending_to_add, ent, node); + g_free(ent); + } else { + return CXL_MBOX_INVALID_PA; + } cxl_insert_extent_to_extent_list(extent_list, dpa, len, NULL, 0); ct3d->dc.total_extent_count += 1; } + /* + * TODO: extents_pending_to_add needs to be cleared so the extents not + * accepted can be reclaimed base on spec r3.0: 8.2.9.8.9.3 + */ + return CXL_MBOX_SUCCESS; } diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 482329a499..43cea3d818 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -813,6 +813,7 @@ static int cxl_create_dc_regions(CXLType3Dev *ct3d) region_base += region->len; } QTAILQ_INIT(&ct3d->dc.extents); + QTAILQ_INIT(&ct3d->dc.extents_pending_to_add); return 0; } @@ -1616,7 +1617,8 @@ static int ct3d_qmp_cxl_event_log_enc(CxlEventLog log) return CXL_EVENT_TYPE_FAIL; case CXL_EVENT_LOG_FATAL: return CXL_EVENT_TYPE_FATAL; -/* DCD not yet supported */ + case CXL_EVENT_LOG_DYNCAP: + return CXL_EVENT_TYPE_DYNAMIC_CAP; default: return -EINVAL; } @@ -1867,6 +1869,227 @@ void qmp_cxl_inject_memory_module_event(const char *path, CxlEventLog log, } } +/* CXL r3.0 Table 8-47: Dynanic Capacity Event Record */ +static const QemuUUID dynamic_capacity_uuid = { + .data = UUID(0xca95afa7, 0xf183, 0x4018, 0x8c, 0x2f, + 0x95, 0x26, 0x8e, 0x10, 0x1a, 0x2a), +}; + +typedef enum CXLDCEventType { + DC_EVENT_ADD_CAPACITY = 0x0, + DC_EVENT_RELEASE_CAPACITY = 0x1, + DC_EVENT_FORCED_RELEASE_CAPACITY = 0x2, + DC_EVENT_REGION_CONFIG_UPDATED = 0x3, + DC_EVENT_ADD_CAPACITY_RSP = 0x4, + DC_EVENT_CAPACITY_RELEASED = 0x5, + DC_EVENT_NUM +} CXLDCEventType; + +/* + * Check whether the exact extent exists in the list + * Return value: true if exists, otherwise false + */ +static bool cxl_dc_extent_exists(CXLDCDExtentList *list, CXLDCExtentRaw *ext) +{ + CXLDCDExtent *ent; + + if (!ext || !list) { + return false; + } + + QTAILQ_FOREACH(ent, list, node) { + if (ent->start_dpa != ext->start_dpa) { + continue; + } + + /*Found exact extent*/ + if (ent->len == ext->len) { + return true; + } else { + return false; + } + } + return false; +} + +static void qmp_cxl_process_dynamic_capacity(const char *path, CxlEventLog log, + CXLDCEventType type, uint16_t hid, + uint8_t rid, + CXLDCExtentRecordList *records, + Error **errp) +{ + Object *obj; + CXLEventDynamicCapacity dCap = {}; + CXLEventRecordHdr *hdr = &dCap.hdr; + CXLType3Dev *dcd; + uint8_t flags = 1 << CXL_EVENT_TYPE_INFO; + uint32_t num_extents = 0; + CXLDCExtentRecordList *list; + g_autofree CXLDCExtentRaw *extents = NULL; + CXLDCDExtentList *extent_list = NULL; + uint8_t enc_log; + uint64_t offset, len, block_size; + int i; + int rc; + g_autofree unsigned long *blk_bitmap = NULL; + + obj = object_resolve_path(path, NULL); + if (!obj) { + error_setg(errp, "Unable to resolve path"); + return; + } + if (!object_dynamic_cast(obj, TYPE_CXL_TYPE3)) { + error_setg(errp, "Path not point to a valid CXL type3 device"); + return; + } + + dcd = CXL_TYPE3(obj); + if (!dcd->dc.num_regions) { + error_setg(errp, "No dynamic capacity support from the device"); + return; + } + + rc = ct3d_qmp_cxl_event_log_enc(log); + if (rc < 0) { + error_setg(errp, "Unhandled error log type"); + return; + } + enc_log = rc; + + if (rid >= dcd->dc.num_regions) { + error_setg(errp, "region id is too large"); + return; + } + block_size = dcd->dc.regions[rid].block_size; + + /* Sanity check and count the extents */ + list = records; + while (list) { + offset = list->value->offset * MiB; + len = list->value->len * MiB; + + if (len == 0) { + error_setg(errp, "extent with 0 length is not allowed"); + return; + } + + if (offset % block_size || len % block_size) { + error_setg(errp, "dpa or len is not aligned to region block size"); + return; + } + + if (offset + len > dcd->dc.regions[rid].len) { + error_setg(errp, "extent range is beyond the region end"); + return; + } + + num_extents++; + list = list->next; + } + if (num_extents == 0) { + error_setg(errp, "No extents found in the command"); + return; + } + + blk_bitmap = bitmap_new(dcd->dc.regions[rid].len / block_size); + + /* Create Extent list for event being passed to host */ + i = 0; + list = records; + extents = g_new0(CXLDCExtentRaw, num_extents); + while (list) { + offset = list->value->offset * MiB; + len = list->value->len * MiB; + + extents[i].start_dpa = offset + dcd->dc.regions[rid].base; + extents[i].len = len; + memset(extents[i].tag, 0, 0x10); + extents[i].shared_seq = 0; + + /* + * We block the release request from FM if the exact extent has + * not been accepted by the host yet + * TODO: We can loose the restriction by skipping the check if desired + */ + if (type == DC_EVENT_RELEASE_CAPACITY || + type == DC_EVENT_FORCED_RELEASE_CAPACITY) { + if (!cxl_dc_extent_exists(&dcd->dc.extents, &extents[i])) { + error_setg(errp, "No exact extent found in the extent list"); + return; + } + } + + /* No duplicate or overlapped extents are allowed */ + if (test_any_bits_set(blk_bitmap, offset / block_size, + len / block_size)) { + error_setg(errp, "duplicate or overlapped extents are detected"); + return; + } + bitmap_set(blk_bitmap, offset / block_size, len / block_size); + + list = list->next; + i++; + } + + switch (type) { + case DC_EVENT_ADD_CAPACITY: + extent_list = &dcd->dc.extents_pending_to_add; + break; + default: + break; + } + /* + * CXL r3.0 section 8.2.9.1.5: Dynamic Capacity Event Record + * + * All Dynamic Capacity event records shall set the Event Record Severity + * field in the Common Event Record Format to Informational Event. All + * Dynamic Capacity related events shall be logged in the Dynamic Capacity + * Event Log. + */ + cxl_assign_event_header(hdr, &dynamic_capacity_uuid, flags, sizeof(dCap), + cxl_device_get_timestamp(&dcd->cxl_dstate)); + + dCap.type = type; + stw_le_p(&dCap.host_id, hid); + /* only valid for DC_REGION_CONFIG_UPDATED event */ + dCap.updated_region_id = 0; + for (i = 0; i < num_extents; i++) { + memcpy(&dCap.dynamic_capacity_extent, &extents[i], + sizeof(CXLDCExtentRaw)); + + if (extent_list) { + cxl_insert_extent_to_extent_list(extent_list, + extents[i].start_dpa, + extents[i].len, + extents[i].tag, + extents[i].shared_seq); + } + + if (cxl_event_insert(&dcd->cxl_dstate, enc_log, + (CXLEventRecordRaw *)&dCap)) { + cxl_event_irq_assert(dcd); + } + } +} + +void qmp_cxl_add_dynamic_capacity(const char *path, uint8_t region_id, + CXLDCExtentRecordList *records, + Error **errp) +{ + qmp_cxl_process_dynamic_capacity(path, CXL_EVENT_LOG_DYNCAP, + DC_EVENT_ADD_CAPACITY, 0, + region_id, records, errp); +} + +void qmp_cxl_release_dynamic_capacity(const char *path, uint8_t region_id, + CXLDCExtentRecordList *records, + Error **errp) +{ + qmp_cxl_process_dynamic_capacity(path, CXL_EVENT_LOG_DYNCAP, + DC_EVENT_RELEASE_CAPACITY, 0, + region_id, records, errp); +} + static void ct3_class_init(ObjectClass *oc, void *data) { DeviceClass *dc = DEVICE_CLASS(oc); diff --git a/hw/mem/cxl_type3_stubs.c b/hw/mem/cxl_type3_stubs.c index 3e1851e32b..d913b11b4d 100644 --- a/hw/mem/cxl_type3_stubs.c +++ b/hw/mem/cxl_type3_stubs.c @@ -67,3 +67,17 @@ void qmp_cxl_inject_correctable_error(const char *path, CxlCorErrorType type, { error_setg(errp, "CXL Type 3 support is not compiled in"); } + +void qmp_cxl_add_dynamic_capacity(const char *path, uint8_t region_id, + CXLDCExtentRecordList *records, + Error **errp) +{ + error_setg(errp, "CXL Type 3 support is not compiled in"); +} + +void qmp_cxl_release_dynamic_capacity(const char *path, uint8_t region_id, + CXLDCExtentRecordList *records, + Error **errp) +{ + error_setg(errp, "CXL Type 3 support is not compiled in"); +} diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index b3d35fe000..ca4f824b11 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -491,6 +491,7 @@ struct CXLType3Dev { AddressSpace host_dc_as; uint64_t total_capacity; /* 256M aligned */ CXLDCDExtentList extents; + CXLDCDExtentList extents_pending_to_add; uint32_t total_extent_count; uint32_t ext_list_gen_seq; @@ -550,5 +551,10 @@ void cxl_event_irq_assert(CXLType3Dev *ct3d); void cxl_set_poison_list_overflowed(CXLType3Dev *ct3d); CXLDCDRegion *cxl_find_dc_region(CXLType3Dev *ct3d, uint64_t dpa, uint64_t len); - +void cxl_insert_extent_to_extent_list(CXLDCDExtentList *list, + uint64_t dpa, + uint64_t len, + uint8_t *tag, + uint16_t shared_seq); +bool test_any_bits_set(const unsigned long *addr, int nr, int size); #endif diff --git a/include/hw/cxl/cxl_events.h b/include/hw/cxl/cxl_events.h index d778487b7e..4f8cb3215d 100644 --- a/include/hw/cxl/cxl_events.h +++ b/include/hw/cxl/cxl_events.h @@ -166,4 +166,19 @@ typedef struct CXLEventMemoryModule { uint8_t reserved[0x3d]; } QEMU_PACKED CXLEventMemoryModule; +/* + * CXL r3.0 section Table 8-47: Dynamic Capacity Event Record + * All fields little endian. + */ +typedef struct CXLEventDynamicCapacity { + CXLEventRecordHdr hdr; + uint8_t type; + uint8_t reserved1; + uint16_t host_id; + uint8_t updated_region_id; + uint8_t reserved2[3]; + uint8_t dynamic_capacity_extent[0x28]; /* defined in cxl_device.h */ + uint8_t reserved[0x20]; +} QEMU_PACKED CXLEventDynamicCapacity; + #endif /* CXL_EVENTS_H */ diff --git a/qapi/cxl.json b/qapi/cxl.json index 8cc4c72fa9..6b631f64f1 100644 --- a/qapi/cxl.json +++ b/qapi/cxl.json @@ -25,7 +25,8 @@ 'data': ['informational', 'warning', 'failure', - 'fatal'] + 'fatal', + 'dyncap'] } ## @@ -361,3 +362,60 @@ ## {'command': 'cxl-inject-correctable-error', 'data': {'path': 'str', 'type': 'CxlCorErrorType'}} + +## +# @CXLDCExtentRecord: +# +# Record of a single extent to add/release +# +# @offset: offset of the extent start related to current region base address +# @len: extent size (in MiB) +# +# Since: 8.0 +## +{ 'struct': 'CXLDCExtentRecord', + 'data': { + 'offset':'uint64', + 'len': 'uint64' + } +} + +## +# @cxl-add-dynamic-capacity: +# +# Command to start add dynamic capacity extents flow. The host will +# need to respond to indicate it accepts the capacity before it becomes +# available for read and write. +# +# @path: CXL DCD canonical QOM path +# @region-id: id of the region where the extent to add/release +# @extents: Extents to add +# +# Since : 8.2 +## +{ 'command': 'cxl-add-dynamic-capacity', + 'data': { 'path': 'str', + 'region-id': 'uint8', + 'extents': [ 'CXLDCExtentRecord' ] + } +} + +## +# @cxl-release-dynamic-capacity: +# +# Command to start release dynamic capacity extents flow. The host will +# need to respond to indicate that it has released the capacity before it +# is made unavailable for read and write and can be re-added. +# +# @path: CXL DCD canonical QOM path +# @region-id: id of the region where the extent to add/release +# @extents: Extents to release +# +# Since : 8.2 +## +{ 'command': 'cxl-release-dynamic-capacity', + 'data': { 'path': 'str', + 'region-id': 'uint8', + 'extents': [ 'CXLDCExtentRecord' ] + } +} From patchwork Tue Nov 7 18:07:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13449174 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1F0253589D for ; Tue, 7 Nov 2023 18:10:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="MXsXahkQ" Received: from mail-pg1-x531.google.com (mail-pg1-x531.google.com [IPv6:2607:f8b0:4864:20::531]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A5A06C6 for ; Tue, 7 Nov 2023 10:10:25 -0800 (PST) Received: by mail-pg1-x531.google.com with SMTP id 41be03b00d2f7-5b9a456798eso3557424a12.3 for ; Tue, 07 Nov 2023 10:10:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1699380625; x=1699985425; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Z3VD1vraDKH7t/1eP7lxzrPhMiE3bMoC5UEpP6LTwk8=; b=MXsXahkQVMb9d7VYopERZ1NdAmGj+MgDiJHkDAXPkmjsx0gy1TTO/wF5b/vYUwZq7D IcBkVPzBFNKiw3qkaoIG5zxKJRmis6YIyqUbl+/Za+gbVE6K+xbT0CNG7z5trjjoC11t kQwDqPqp5c5ZuuaMPWTzWE40YuZYYyVfdUTa861s8lSD4YW3LVbNv6rVEd0YyOmDg0x7 IHkUJ0g/Bf6NpW4evCg2RxXhJZvSim2atnKr/+3o7Z0eDAXzVBFJiclMXhoEERaGCRcr 0Cmtvot5kaSjycsAr7u5PlQOyD3aG4WJ3u1dPJxLirgm8/v1ti5TBqSeO4tXvLoTL1Qm ZibQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699380625; x=1699985425; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Z3VD1vraDKH7t/1eP7lxzrPhMiE3bMoC5UEpP6LTwk8=; b=iihwK3AKr2DmHEPbd+cRaUoQdOsm7cycO8/vku8LudrGuHXMqBjfzKzTljixOPWSyk fL78lmiJF0aZYejHYY2uxIJbVeGkw+PYv8pHv0QRNzVsiJdVYKYCGtUZX1YgHiaopZ5o frljfGvUjUfJhSP+XlYSEnGpwrWbVQVj4gXzTSzRIqgaygXuY2TFkmp/IAzLGLuaZk04 A04dwrbsuZvne6CkWMwWsaUrBbDad5s+kkgHN95y0y/8+MHUs2y/HuRFZJBNSct8KDpi BgPm37TpVs6XfWL3I2+S285X7M9pmk3ZRHyiMvvViu0tqe4Z8kEHfd+js95HAftFvd1i /nNw== X-Gm-Message-State: AOJu0YwAANQijTahHMFh4Khc1kIwysMwgWti1H3wWgL8TDHvIUZGl8wQ RSzZ9xyrJkm4xpmRotdwz8M= X-Google-Smtp-Source: AGHT+IEGtULtzP3zUAeEcHZAMhAMLm8y71Y+vs76AD4I/K8Ta3QaxjqhuJ8KDNsahiaEcfKxi25Q/A== X-Received: by 2002:a17:90a:6f84:b0:280:9074:eb3d with SMTP id e4-20020a17090a6f8400b002809074eb3dmr17621113pjk.22.1699380624853; Tue, 07 Nov 2023 10:10:24 -0800 (PST) Received: from DT.local (c-71-202-158-162.hsd1.ca.comcast.net. [71.202.158.162]) by smtp.gmail.com with ESMTPSA id az10-20020a170902a58a00b001cc615e6850sm131924plb.90.2023.11.07.10.10.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Nov 2023 10:10:24 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan@outlook.com, jim.harris@samsung.com, Fan Ni Subject: [PATCH v3 9/9] hw/mem/cxl_type3: Add dpa range validation for accesses to dc regions Date: Tue, 7 Nov 2023 10:07:13 -0800 Message-ID: <20231107180907.553451-10-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231107180907.553451-1-nifan.cxl@gmail.com> References: <20231107180907.553451-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Not all dpa range in the dc regions is valid to access until an extent covering the range has been added. Add a bitmap for each region to record whether a dc block in the region has been backed by dc extent. For the bitmap, a bit in the bitmap represents a dc block. When a dc extent is added, all the bits of the blocks in the extent will be set, which will be cleared when the extent is released. Signed-off-by: Fan Ni --- JC changes: - Rebase on what will be next gitlab.com/jic23/qemu CXL staging tree. - Drop unnecessary handling of failed bitmap allocations. In common with most QEMU allocations they fail hard anyway. - Use previously factored out cxl_find_region() helper - Minor editorial stuff in comments such as spec version references according to the standard form I'm trying to push through the code. Picked up Jørgen's fix: https://lore.kernel.org/qemu-devel/d0d7ca1d-81bc-19b3-4904-d60046ded844@wdc.com/T/#u --- hw/cxl/cxl-mailbox-utils.c | 31 +++++++++------ hw/mem/cxl_type3.c | 78 +++++++++++++++++++++++++++++++++++++ include/hw/cxl/cxl_device.h | 15 +++++-- 3 files changed, 109 insertions(+), 15 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 8e6a98753a..6be92fb5ba 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -1401,10 +1401,9 @@ CXLDCDRegion *cxl_find_dc_region(CXLType3Dev *ct3d, uint64_t dpa, uint64_t len) } void cxl_insert_extent_to_extent_list(CXLDCDExtentList *list, - uint64_t dpa, - uint64_t len, - uint8_t *tag, - uint16_t shared_seq) + uint64_t dpa, uint64_t len, + uint8_t *tag, + uint16_t shared_seq) { CXLDCDExtent *extent; @@ -1421,6 +1420,13 @@ void cxl_insert_extent_to_extent_list(CXLDCDExtentList *list, QTAILQ_INSERT_TAIL(list, extent, node); } +static void cxl_remove_extent_to_extent_list(CXLDCDExtentList *list, + CXLDCDExtent *ent) +{ + QTAILQ_REMOVE(list, ent, node); + g_free(ent); +} + /* * CXL r3.0 Table 8-129: Add Dynamic Capacity Response Input Payload * CXL r3.0 Table 8-131: Release Dynamic Capacity Input Payload @@ -1545,14 +1551,15 @@ static CXLRetCode cmd_dcd_add_dyn_cap_rsp(const struct cxl_cmd *cmd, } } if (ent) { - QTAILQ_REMOVE(&ct3d->dc.extents_pending_to_add, ent, node); - g_free(ent); + cxl_remove_extent_to_extent_list(&ct3d->dc.extents_pending_to_add, + ent); } else { return CXL_MBOX_INVALID_PA; } cxl_insert_extent_to_extent_list(extent_list, dpa, len, NULL, 0); ct3d->dc.total_extent_count += 1; + ct3_set_region_block_backed(ct3d, dpa, len); } /* @@ -1601,16 +1608,22 @@ static CXLRetCode cmd_dcd_release_dyn_cap(const struct cxl_cmd *cmd, uint64_t len1 = dpa - ent->start_dpa; uint64_t len2 = ent->start_dpa + ent->len - dpa - len; + cxl_remove_extent_to_extent_list(extent_list, ent); + ct3d->dc.total_extent_count -= 1; + ct3_clear_region_block_backed(ct3d, dpa, len); + if (len1) { cxl_insert_extent_to_extent_list(extent_list, ent->start_dpa, len1, NULL, 0); ct3d->dc.total_extent_count += 1; + ct3_set_region_block_backed(ct3d, dpa, len); } if (len2) { cxl_insert_extent_to_extent_list(extent_list, dpa + len, len2, NULL, 0); ct3d->dc.total_extent_count += 1; + ct3_set_region_block_backed(ct3d, dpa, len); } break; /*Currently we reject the attempt to remove a superset*/ @@ -1621,11 +1634,7 @@ static CXLRetCode cmd_dcd_release_dyn_cap(const struct cxl_cmd *cmd, } } - if (ent) { - QTAILQ_REMOVE(extent_list, ent, node); - g_free(ent); - ct3d->dc.total_extent_count -= 1; - } else { + if (!ent) { /* Try to remove a non-existing extent */ return CXL_MBOX_INVALID_PA; } diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 43cea3d818..4ec65a751a 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -810,6 +810,7 @@ static int cxl_create_dc_regions(CXLType3Dev *ct3d) /* dsmad_handle is set when creating cdat table entries */ region->flags = 0; + region->blk_bitmap = bitmap_new(region->len / region->block_size); region_base += region->len; } QTAILQ_INIT(&ct3d->dc.extents); @@ -818,6 +819,17 @@ static int cxl_create_dc_regions(CXLType3Dev *ct3d) return 0; } +static void cxl_destroy_dc_regions(CXLType3Dev *ct3d) +{ + int i; + struct CXLDCDRegion *region; + + for (i = 0; i < ct3d->dc.num_regions; i++) { + region = &ct3d->dc.regions[i]; + g_free(region->blk_bitmap); + } +} + static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) { DeviceState *ds = DEVICE(ct3d); @@ -1046,6 +1058,7 @@ err_free_special_ops: g_free(regs->special_ops); err_address_space_free: if (ct3d->dc.host_dc) { + cxl_destroy_dc_regions(ct3d); address_space_destroy(&ct3d->dc.host_dc_as); } if (ct3d->hostpmem) { @@ -1068,6 +1081,7 @@ static void ct3_exit(PCIDevice *pci_dev) spdm_sock_fini(ct3d->doe_spdm.socket); g_free(regs->special_ops); if (ct3d->dc.host_dc) { + cxl_destroy_dc_regions(ct3d); address_space_destroy(&ct3d->dc.host_dc_as); } if (ct3d->hostpmem) { @@ -1078,6 +1092,66 @@ static void ct3_exit(PCIDevice *pci_dev) } } +/* + * Mark the DPA range [dpa, dap + len) to be backed and accessible. This + * happens when a DC extent is added and accepted by the host. + */ +void ct3_set_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len) +{ + CXLDCDRegion *region; + + region = cxl_find_dc_region(ct3d, dpa, len); + if (!region) { + return; + } + + bitmap_set(region->blk_bitmap, (dpa - region->base) / region->block_size, + len / region->block_size); +} + +/* + * Check whether a DPA range [dpa, dpa + len) has been backed with DC extents. + * Used when validating read/write to dc regions + */ +bool ct3_test_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len) +{ + CXLDCDRegion *region; + uint64_t nbits; + long nr; + + region = cxl_find_dc_region(ct3d, dpa, len); + if (!region) { + return false; + } + + nr = (dpa - region->base) / region->block_size; + nbits = DIV_ROUND_UP(len, region->block_size); + return find_next_zero_bit(region->blk_bitmap, nr + nbits, nr) == nr + nbits; +} + +/* + * Mark the DPA range [dpa, dap + len) to be unbacked and inaccessible. This + * happens when a dc extent is return by the host. + */ +void ct3_clear_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len) +{ + CXLDCDRegion *region; + uint64_t nbits; + long nr; + + region = cxl_find_dc_region(ct3d, dpa, len); + if (!region) { + return; + } + + nr = (dpa - region->base) / region->block_size; + nbits = len / region->block_size; + bitmap_clear(region->blk_bitmap, nr, nbits); +} + static bool cxl_type3_dpa(CXLType3Dev *ct3d, hwaddr host_addr, uint64_t *dpa) { int hdm_inc = R_CXL_HDM_DECODER1_BASE_LO - R_CXL_HDM_DECODER0_BASE_LO; @@ -1178,6 +1252,10 @@ static int cxl_type3_hpa_to_as_and_dpa(CXLType3Dev *ct3d, *as = &ct3d->hostpmem_as; *dpa_offset -= vmr_size; } else { + if (!ct3_test_region_block_backed(ct3d, *dpa_offset, size)) { + return -ENODEV; + } + *as = &ct3d->dc.host_dc_as; *dpa_offset -= (vmr_size + pmr_size); } diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index ca4f824b11..b71b09700a 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -447,6 +447,7 @@ typedef struct CXLDCDRegion { uint64_t block_size; uint32_t dsmadhandle; uint8_t flags; + unsigned long *blk_bitmap; } CXLDCDRegion; struct CXLType3Dev { @@ -552,9 +553,15 @@ void cxl_set_poison_list_overflowed(CXLType3Dev *ct3d); CXLDCDRegion *cxl_find_dc_region(CXLType3Dev *ct3d, uint64_t dpa, uint64_t len); void cxl_insert_extent_to_extent_list(CXLDCDExtentList *list, - uint64_t dpa, - uint64_t len, - uint8_t *tag, - uint16_t shared_seq); + uint64_t dpa, + uint64_t len, + uint8_t *tag, + uint16_t shared_seq); bool test_any_bits_set(const unsigned long *addr, int nr, int size); +void ct3_set_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len); +void ct3_clear_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len); +bool ct3_test_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len); #endif