From patchwork Wed Sep 13 13:25:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jonathan Cameron X-Patchwork-Id: 13383008 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 583EBEDEC50 for ; Wed, 13 Sep 2023 13:26:29 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qgPsa-0000ZF-0W; Wed, 13 Sep 2023 09:26:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qgPsX-0000Xi-Hk for qemu-devel@nongnu.org; Wed, 13 Sep 2023 09:26:01 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qgPsU-0006et-71 for qemu-devel@nongnu.org; Wed, 13 Sep 2023 09:26:01 -0400 Received: from lhrpeml500005.china.huawei.com (unknown [172.18.147.207]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4Rm1R57042z6K6TF; Wed, 13 Sep 2023 21:25:21 +0800 (CST) Received: from SecurePC-101-06.china.huawei.com (10.122.247.231) by lhrpeml500005.china.huawei.com (7.191.163.240) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.31; Wed, 13 Sep 2023 14:25:54 +0100 To: , Michael Tsirkin , Fan Ni , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , CC: Subject: [PATCH v4 1/4] hw/cxl: Push cxl_decoder_count_enc() and cxl_decode_ig() into .c Date: Wed, 13 Sep 2023 14:25:20 +0100 Message-ID: <20230913132523.29780-2-Jonathan.Cameron@huawei.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230913132523.29780-1-Jonathan.Cameron@huawei.com> References: <20230913132523.29780-1-Jonathan.Cameron@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.122.247.231] X-ClientProxiedBy: lhrpeml100004.china.huawei.com (7.191.162.219) To lhrpeml500005.china.huawei.com (7.191.163.240) X-CFilter-Loop: Reflected Received-SPF: pass client-ip=185.176.79.56; envelope-from=jonathan.cameron@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Jonathan Cameron X-Patchwork-Original-From: Jonathan Cameron via From: Jonathan Cameron Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org There is no strong justification for keeping these in the header so push them down into the associated cxl-component-utils.c file. Suggested-by: Philippe Mathieu-Daudé Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Fan Ni Signed-off-by: Jonathan Cameron --- include/hw/cxl/cxl_component.h | 18 ++---------------- hw/cxl/cxl-component-utils.c | 18 ++++++++++++++++++ 2 files changed, 20 insertions(+), 16 deletions(-) diff --git a/include/hw/cxl/cxl_component.h b/include/hw/cxl/cxl_component.h index 42c7e581a7..bdb3881a6b 100644 --- a/include/hw/cxl/cxl_component.h +++ b/include/hw/cxl/cxl_component.h @@ -225,26 +225,12 @@ void cxl_component_create_dvsec(CXLComponentState *cxl_cstate, enum reg_type cxl_dev_type, uint16_t length, uint16_t type, uint8_t rev, uint8_t *body); -static inline int cxl_decoder_count_enc(int count) -{ - switch (count) { - case 1: return 0; - case 2: return 1; - case 4: return 2; - case 6: return 3; - case 8: return 4; - case 10: return 5; - } - return 0; -} +int cxl_decoder_count_enc(int count); uint8_t cxl_interleave_ways_enc(int iw, Error **errp); uint8_t cxl_interleave_granularity_enc(uint64_t gran, Error **errp); -static inline hwaddr cxl_decode_ig(int ig) -{ - return 1ULL << (ig + 8); -} +hwaddr cxl_decode_ig(int ig); CXLComponentState *cxl_get_hb_cstate(PCIHostState *hb); bool cxl_get_hb_passthrough(PCIHostState *hb); diff --git a/hw/cxl/cxl-component-utils.c b/hw/cxl/cxl-component-utils.c index 378f1082ce..ea2d4770ec 100644 --- a/hw/cxl/cxl-component-utils.c +++ b/hw/cxl/cxl-component-utils.c @@ -13,6 +13,24 @@ #include "hw/pci/pci.h" #include "hw/cxl/cxl.h" +int cxl_decoder_count_enc(int count) +{ + switch (count) { + case 1: return 0; + case 2: return 1; + case 4: return 2; + case 6: return 3; + case 8: return 4; + case 10: return 5; + } + return 0; +} + +hwaddr cxl_decode_ig(int ig) +{ + return 1ULL << (ig + 8); +} + static uint64_t cxl_cache_mem_read_reg(void *opaque, hwaddr offset, unsigned size) { From patchwork Wed Sep 13 13:25:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jonathan Cameron X-Patchwork-Id: 13383009 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 45790EDEC50 for ; Wed, 13 Sep 2023 13:26:42 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qgPt2-0001j7-49; Wed, 13 Sep 2023 09:26:32 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qgPsz-0001dj-Vr for qemu-devel@nongnu.org; Wed, 13 Sep 2023 09:26:30 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qgPsx-0006lS-GP for qemu-devel@nongnu.org; Wed, 13 Sep 2023 09:26:29 -0400 Received: from lhrpeml500005.china.huawei.com (unknown [172.18.147.201]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4Rm1Ly2SPPz6D9BV; Wed, 13 Sep 2023 21:21:46 +0800 (CST) Received: from SecurePC-101-06.china.huawei.com (10.122.247.231) by lhrpeml500005.china.huawei.com (7.191.163.240) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.31; Wed, 13 Sep 2023 14:26:25 +0100 To: , Michael Tsirkin , Fan Ni , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , CC: Subject: [PATCH v4 2/4] hw/cxl: Add utility functions decoder interleave ways and target count. Date: Wed, 13 Sep 2023 14:25:21 +0100 Message-ID: <20230913132523.29780-3-Jonathan.Cameron@huawei.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230913132523.29780-1-Jonathan.Cameron@huawei.com> References: <20230913132523.29780-1-Jonathan.Cameron@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.122.247.231] X-ClientProxiedBy: lhrpeml100004.china.huawei.com (7.191.162.219) To lhrpeml500005.china.huawei.com (7.191.163.240) X-CFilter-Loop: Reflected Received-SPF: pass client-ip=185.176.79.56; envelope-from=jonathan.cameron@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Jonathan Cameron X-Patchwork-Original-From: Jonathan Cameron via From: Jonathan Cameron Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org As an encoded version of these key configuration parameters is available in a register, provide functions to extract it again so as to avoid the need for duplicating the storage. Whilst here update the _enc() function to include additional values as defined in the CXL 3.0 specification. Whilst they are not currently used in the emulation, they may be in future and it is easier to compare with the specification if all values are covered. Add a spec reference for cxl_interleave_ways_enc() for consistency with the target count equivalent (and because it's nice to know where the magic numbers come from). Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Fan Ni Signed-off-by: Jonathan Cameron --- v4: Add specification reference for cxl_interleave_ways_enc() as suggested by Fan. --- include/hw/cxl/cxl_component.h | 2 ++ hw/cxl/cxl-component-utils.c | 60 ++++++++++++++++++++++++++++++---- 2 files changed, 56 insertions(+), 6 deletions(-) diff --git a/include/hw/cxl/cxl_component.h b/include/hw/cxl/cxl_component.h index bdb3881a6b..ef9e033919 100644 --- a/include/hw/cxl/cxl_component.h +++ b/include/hw/cxl/cxl_component.h @@ -226,8 +226,10 @@ void cxl_component_create_dvsec(CXLComponentState *cxl_cstate, uint16_t type, uint8_t rev, uint8_t *body); int cxl_decoder_count_enc(int count); +int cxl_decoder_count_dec(int enc_cnt); uint8_t cxl_interleave_ways_enc(int iw, Error **errp); +int cxl_interleave_ways_dec(uint8_t iw_enc, Error **errp); uint8_t cxl_interleave_granularity_enc(uint64_t gran, Error **errp); hwaddr cxl_decode_ig(int ig); diff --git a/hw/cxl/cxl-component-utils.c b/hw/cxl/cxl-component-utils.c index ea2d4770ec..5f38f2016f 100644 --- a/hw/cxl/cxl-component-utils.c +++ b/hw/cxl/cxl-component-utils.c @@ -13,15 +13,45 @@ #include "hw/pci/pci.h" #include "hw/cxl/cxl.h" +/* CXL r3.0 Section 8.2.4.19.1 CXL HDM Decoder Capability Register */ int cxl_decoder_count_enc(int count) { switch (count) { - case 1: return 0; - case 2: return 1; - case 4: return 2; - case 6: return 3; - case 8: return 4; - case 10: return 5; + case 1: return 0x0; + case 2: return 0x1; + case 4: return 0x2; + case 6: return 0x3; + case 8: return 0x4; + case 10: return 0x5; + /* Switches and Host Bridges may have more than 10 decoders */ + case 12: return 0x6; + case 14: return 0x7; + case 16: return 0x8; + case 20: return 0x9; + case 24: return 0xa; + case 28: return 0xb; + case 32: return 0xc; + } + return 0; +} + +int cxl_decoder_count_dec(int enc_cnt) +{ + switch (enc_cnt) { + case 0x0: return 1; + case 0x1: return 2; + case 0x2: return 4; + case 0x3: return 6; + case 0x4: return 8; + case 0x5: return 10; + /* Switches and Host Bridges may have more than 10 decoders */ + case 0x6: return 12; + case 0x7: return 14; + case 0x8: return 16; + case 0x9: return 20; + case 0xa: return 24; + case 0xb: return 28; + case 0xc: return 32; } return 0; } @@ -393,6 +423,7 @@ void cxl_component_create_dvsec(CXLComponentState *cxl, cxl->dvsec_offset += length; } +/* CXL r3.0 Section 8.2.4.19.7 CXL HDM Decoder n Control Register */ uint8_t cxl_interleave_ways_enc(int iw, Error **errp) { switch (iw) { @@ -410,6 +441,23 @@ uint8_t cxl_interleave_ways_enc(int iw, Error **errp) } } +int cxl_interleave_ways_dec(uint8_t iw_enc, Error **errp) +{ + switch (iw_enc) { + case 0x0: return 1; + case 0x1: return 2; + case 0x2: return 4; + case 0x3: return 8; + case 0x4: return 16; + case 0x8: return 3; + case 0x9: return 6; + case 0xa: return 12; + default: + error_setg(errp, "Encoded interleave ways: %d not supported", iw_enc); + return 0; + } +} + uint8_t cxl_interleave_granularity_enc(uint64_t gran, Error **errp) { switch (gran) { From patchwork Wed Sep 13 13:25:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jonathan Cameron X-Patchwork-Id: 13383010 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 553A3EDEC4D for ; Wed, 13 Sep 2023 13:27:17 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qgPtZ-0002jR-9T; Wed, 13 Sep 2023 09:27:05 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qgPtW-0002im-Aj for qemu-devel@nongnu.org; Wed, 13 Sep 2023 09:27:02 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qgPtT-0006uc-HW for qemu-devel@nongnu.org; Wed, 13 Sep 2023 09:27:01 -0400 Received: from lhrpeml500005.china.huawei.com (unknown [172.18.147.226]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4Rm1SH1j7Mz6K5sp; Wed, 13 Sep 2023 21:26:23 +0800 (CST) Received: from SecurePC-101-06.china.huawei.com (10.122.247.231) by lhrpeml500005.china.huawei.com (7.191.163.240) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.31; Wed, 13 Sep 2023 14:26:56 +0100 To: , Michael Tsirkin , Fan Ni , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , CC: Subject: [PATCH v4 3/4] hw/cxl: Fix and use same calculation for HDM decoder block size everywhere Date: Wed, 13 Sep 2023 14:25:22 +0100 Message-ID: <20230913132523.29780-4-Jonathan.Cameron@huawei.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230913132523.29780-1-Jonathan.Cameron@huawei.com> References: <20230913132523.29780-1-Jonathan.Cameron@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.122.247.231] X-ClientProxiedBy: lhrpeml100004.china.huawei.com (7.191.162.219) To lhrpeml500005.china.huawei.com (7.191.163.240) X-CFilter-Loop: Reflected Received-SPF: pass client-ip=185.176.79.56; envelope-from=jonathan.cameron@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Jonathan Cameron X-Patchwork-Original-From: Jonathan Cameron via From: Jonathan Cameron Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org In order to avoid having the size of the per HDM decoder register block repeated in lots of places, create the register definitions for HDM decoder 1 and use the offset between the first registers in HDM decoder 0 and HDM decoder 1 to establish the offset. Calculate in each function as this is more obvious and leads to shorter line lengths than a single #define which would need a long name to be specific enough. Note that the code currently only supports one decoder, so the bugs this fixes don't actually affect anything. Signed-off-by: Jonathan Cameron Reviewed-by: Fan Ni Reviewed-by: Philippe Mathieu-Daudé --- v3: New patch to separate this out from the addition of HDM decoders. --- include/hw/cxl/cxl_component.h | 2 ++ hw/cxl/cxl-component-utils.c | 19 +++++++++++-------- hw/cxl/cxl-host.c | 4 +++- hw/mem/cxl_type3.c | 24 +++++++++++++++--------- 4 files changed, 31 insertions(+), 18 deletions(-) diff --git a/include/hw/cxl/cxl_component.h b/include/hw/cxl/cxl_component.h index ef9e033919..7c864d2044 100644 --- a/include/hw/cxl/cxl_component.h +++ b/include/hw/cxl/cxl_component.h @@ -148,6 +148,8 @@ REG32(CXL_HDM_DECODER_GLOBAL_CONTROL, CXL_HDM_REGISTERS_OFFSET + 4) FIELD(CXL_HDM_DECODER_GLOBAL_CONTROL, HDM_DECODER_ENABLE, 1, 1) HDM_DECODER_INIT(0); +/* Only used for HDM decoder registers block address increment */ +HDM_DECODER_INIT(1); /* 8.2.5.13 - CXL Extended Security Capability Structure (Root complex only) */ #define EXTSEC_ENTRY_MAX 256 diff --git a/hw/cxl/cxl-component-utils.c b/hw/cxl/cxl-component-utils.c index 5f38f2016f..c0630ba5c1 100644 --- a/hw/cxl/cxl-component-utils.c +++ b/hw/cxl/cxl-component-utils.c @@ -210,6 +210,7 @@ static void hdm_init_common(uint32_t *reg_state, uint32_t *write_msk, enum reg_type type) { int decoder_count = 1; + int hdm_inc = R_CXL_HDM_DECODER1_BASE_LO - R_CXL_HDM_DECODER0_BASE_LO; int i; ARRAY_FIELD_DP32(reg_state, CXL_HDM_DECODER_CAPABILITY, DECODER_COUNT, @@ -222,19 +223,21 @@ static void hdm_init_common(uint32_t *reg_state, uint32_t *write_msk, HDM_DECODER_ENABLE, 0); write_msk[R_CXL_HDM_DECODER_GLOBAL_CONTROL] = 0x3; for (i = 0; i < decoder_count; i++) { - write_msk[R_CXL_HDM_DECODER0_BASE_LO + i * 0x20] = 0xf0000000; - write_msk[R_CXL_HDM_DECODER0_BASE_HI + i * 0x20] = 0xffffffff; - write_msk[R_CXL_HDM_DECODER0_SIZE_LO + i * 0x20] = 0xf0000000; - write_msk[R_CXL_HDM_DECODER0_SIZE_HI + i * 0x20] = 0xffffffff; - write_msk[R_CXL_HDM_DECODER0_CTRL + i * 0x20] = 0x13ff; + write_msk[R_CXL_HDM_DECODER0_BASE_LO + i * hdm_inc] = 0xf0000000; + write_msk[R_CXL_HDM_DECODER0_BASE_HI + i * hdm_inc] = 0xffffffff; + write_msk[R_CXL_HDM_DECODER0_SIZE_LO + i * hdm_inc] = 0xf0000000; + write_msk[R_CXL_HDM_DECODER0_SIZE_HI + i * hdm_inc] = 0xffffffff; + write_msk[R_CXL_HDM_DECODER0_CTRL + i * hdm_inc] = 0x13ff; if (type == CXL2_DEVICE || type == CXL2_TYPE3_DEVICE || type == CXL2_LOGICAL_DEVICE) { - write_msk[R_CXL_HDM_DECODER0_TARGET_LIST_LO + i * 0x20] = 0xf0000000; + write_msk[R_CXL_HDM_DECODER0_TARGET_LIST_LO + i * hdm_inc] = + 0xf0000000; } else { - write_msk[R_CXL_HDM_DECODER0_TARGET_LIST_LO + i * 0x20] = 0xffffffff; + write_msk[R_CXL_HDM_DECODER0_TARGET_LIST_LO + i * hdm_inc] = + 0xffffffff; } - write_msk[R_CXL_HDM_DECODER0_TARGET_LIST_HI + i * 0x20] = 0xffffffff; + write_msk[R_CXL_HDM_DECODER0_TARGET_LIST_HI + i * hdm_inc] = 0xffffffff; } } diff --git a/hw/cxl/cxl-host.c b/hw/cxl/cxl-host.c index f0920da956..73c5426476 100644 --- a/hw/cxl/cxl-host.c +++ b/hw/cxl/cxl-host.c @@ -101,12 +101,14 @@ void cxl_fmws_link_targets(CXLState *cxl_state, Error **errp) static bool cxl_hdm_find_target(uint32_t *cache_mem, hwaddr addr, uint8_t *target) { + int hdm_inc = R_CXL_HDM_DECODER1_BASE_LO - R_CXL_HDM_DECODER0_BASE_LO; uint32_t ctrl; uint32_t ig_enc; uint32_t iw_enc; uint32_t target_idx; + int i = 0; - ctrl = cache_mem[R_CXL_HDM_DECODER0_CTRL]; + ctrl = cache_mem[R_CXL_HDM_DECODER0_CTRL + i * hdm_inc]; if (!FIELD_EX32(ctrl, CXL_HDM_DECODER0_CTRL, COMMITTED)) { return false; } diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 4e314748d3..cd92813436 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -377,34 +377,36 @@ static void build_dvsecs(CXLType3Dev *ct3d) static void hdm_decoder_commit(CXLType3Dev *ct3d, int which) { + int hdm_inc = R_CXL_HDM_DECODER1_BASE_LO - R_CXL_HDM_DECODER0_BASE_LO; ComponentRegisters *cregs = &ct3d->cxl_cstate.crb; uint32_t *cache_mem = cregs->cache_mem_registers; uint32_t ctrl; assert(which == 0); - ctrl = ldl_le_p(cache_mem + R_CXL_HDM_DECODER0_CTRL); + ctrl = ldl_le_p(cache_mem + R_CXL_HDM_DECODER0_CTRL + which * hdm_inc); /* TODO: Sanity checks that the decoder is possible */ ctrl = FIELD_DP32(ctrl, CXL_HDM_DECODER0_CTRL, ERR, 0); ctrl = FIELD_DP32(ctrl, CXL_HDM_DECODER0_CTRL, COMMITTED, 1); - stl_le_p(cache_mem + R_CXL_HDM_DECODER0_CTRL, ctrl); + stl_le_p(cache_mem + R_CXL_HDM_DECODER0_CTRL + which * hdm_inc, ctrl); } static void hdm_decoder_uncommit(CXLType3Dev *ct3d, int which) { + int hdm_inc = R_CXL_HDM_DECODER1_BASE_LO - R_CXL_HDM_DECODER0_BASE_LO; ComponentRegisters *cregs = &ct3d->cxl_cstate.crb; uint32_t *cache_mem = cregs->cache_mem_registers; uint32_t ctrl; assert(which == 0); - ctrl = ldl_le_p(cache_mem + R_CXL_HDM_DECODER0_CTRL); + ctrl = ldl_le_p(cache_mem + R_CXL_HDM_DECODER0_CTRL + which * hdm_inc); ctrl = FIELD_DP32(ctrl, CXL_HDM_DECODER0_CTRL, ERR, 0); ctrl = FIELD_DP32(ctrl, CXL_HDM_DECODER0_CTRL, COMMITTED, 0); - stl_le_p(cache_mem + R_CXL_HDM_DECODER0_CTRL, ctrl); + stl_le_p(cache_mem + R_CXL_HDM_DECODER0_CTRL + which * hdm_inc, ctrl); } static int ct3d_qmp_uncor_err_to_cxl(CxlUncorErrorType qmp_err) @@ -761,26 +763,30 @@ static void ct3_exit(PCIDevice *pci_dev) /* TODO: Support multiple HDM decoders and DPA skip */ static bool cxl_type3_dpa(CXLType3Dev *ct3d, hwaddr host_addr, uint64_t *dpa) { + int hdm_inc = R_CXL_HDM_DECODER1_BASE_LO - R_CXL_HDM_DECODER0_BASE_LO; uint32_t *cache_mem = ct3d->cxl_cstate.crb.cache_mem_registers; uint64_t decoder_base, decoder_size, hpa_offset; uint32_t hdm0_ctrl; int ig, iw; + int i = 0; - decoder_base = (((uint64_t)cache_mem[R_CXL_HDM_DECODER0_BASE_HI] << 32) | - cache_mem[R_CXL_HDM_DECODER0_BASE_LO]); + decoder_base = + (((uint64_t)cache_mem[R_CXL_HDM_DECODER0_BASE_HI + i * hdm_inc] << 32) | + cache_mem[R_CXL_HDM_DECODER0_BASE_LO + i * hdm_inc]); if ((uint64_t)host_addr < decoder_base) { return false; } hpa_offset = (uint64_t)host_addr - decoder_base; - decoder_size = ((uint64_t)cache_mem[R_CXL_HDM_DECODER0_SIZE_HI] << 32) | - cache_mem[R_CXL_HDM_DECODER0_SIZE_LO]; + decoder_size = + ((uint64_t)cache_mem[R_CXL_HDM_DECODER0_SIZE_HI + i * hdm_inc] << 32) | + cache_mem[R_CXL_HDM_DECODER0_SIZE_LO + i * hdm_inc]; if (hpa_offset >= decoder_size) { return false; } - hdm0_ctrl = cache_mem[R_CXL_HDM_DECODER0_CTRL]; + hdm0_ctrl = cache_mem[R_CXL_HDM_DECODER0_CTRL + i * hdm_inc]; iw = FIELD_EX32(hdm0_ctrl, CXL_HDM_DECODER0_CTRL, IW); ig = FIELD_EX32(hdm0_ctrl, CXL_HDM_DECODER0_CTRL, IG); From patchwork Wed Sep 13 13:25:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Cameron X-Patchwork-Id: 13383011 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7BC62EDEC50 for ; Wed, 13 Sep 2023 13:27:54 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qgPu2-0003I3-57; Wed, 13 Sep 2023 09:27:34 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qgPtz-0003DC-Oa for qemu-devel@nongnu.org; Wed, 13 Sep 2023 09:27:31 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qgPtw-00079q-PH for qemu-devel@nongnu.org; Wed, 13 Sep 2023 09:27:31 -0400 Received: from lhrpeml500005.china.huawei.com (unknown [172.18.147.207]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4Rm1Ss4fP7z6K6T2; Wed, 13 Sep 2023 21:26:53 +0800 (CST) Received: from SecurePC-101-06.china.huawei.com (10.122.247.231) by lhrpeml500005.china.huawei.com (7.191.163.240) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.31; Wed, 13 Sep 2023 14:27:26 +0100 To: , Michael Tsirkin , Fan Ni , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , CC: Subject: [PATCH v4 4/4] hw/cxl: Support 4 HDM decoders at all levels of topology Date: Wed, 13 Sep 2023 14:25:23 +0100 Message-ID: <20230913132523.29780-5-Jonathan.Cameron@huawei.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230913132523.29780-1-Jonathan.Cameron@huawei.com> References: <20230913132523.29780-1-Jonathan.Cameron@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.122.247.231] X-ClientProxiedBy: lhrpeml100004.china.huawei.com (7.191.162.219) To lhrpeml500005.china.huawei.com (7.191.163.240) X-CFilter-Loop: Reflected Received-SPF: pass client-ip=185.176.79.56; envelope-from=jonathan.cameron@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Jonathan Cameron X-Patchwork-Original-From: Jonathan Cameron via From: Jonathan Cameron Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Support these decoders in CXL host bridges (pxb-cxl), CXL Switch USP and CXL Type 3 end points. Signed-off-by: Jonathan Cameron --- v3: Factor out the hdm_inc changes to previous patch. Fix use of encoded hdm count as if it were decoded in cxl-host. Minor refactoring to make that path and one in cxl_type3.c look more similar. --- include/hw/cxl/cxl_component.h | 10 +++- hw/cxl/cxl-component-utils.c | 7 ++- hw/cxl/cxl-host.c | 67 ++++++++++++++++-------- hw/mem/cxl_type3.c | 96 +++++++++++++++++++++++----------- 4 files changed, 124 insertions(+), 56 deletions(-) diff --git a/include/hw/cxl/cxl_component.h b/include/hw/cxl/cxl_component.h index 7c864d2044..3c795a6278 100644 --- a/include/hw/cxl/cxl_component.h +++ b/include/hw/cxl/cxl_component.h @@ -135,6 +135,10 @@ REG32(CXL_RAS_ERR_HEADER0, CXL_RAS_REGISTERS_OFFSET + 0x18) REG32(CXL_HDM_DECODER##n##_TARGET_LIST_LO, \ CXL_HDM_REGISTERS_OFFSET + (0x20 * n) + 0x24) \ REG32(CXL_HDM_DECODER##n##_TARGET_LIST_HI, \ + CXL_HDM_REGISTERS_OFFSET + (0x20 * n) + 0x28) \ + REG32(CXL_HDM_DECODER##n##_DPA_SKIP_LO, \ + CXL_HDM_REGISTERS_OFFSET + (0x20 * n) + 0x24) \ + REG32(CXL_HDM_DECODER##n##_DPA_SKIP_HI, \ CXL_HDM_REGISTERS_OFFSET + (0x20 * n) + 0x28) REG32(CXL_HDM_DECODER_CAPABILITY, CXL_HDM_REGISTERS_OFFSET) @@ -147,9 +151,13 @@ REG32(CXL_HDM_DECODER_GLOBAL_CONTROL, CXL_HDM_REGISTERS_OFFSET + 4) FIELD(CXL_HDM_DECODER_GLOBAL_CONTROL, POISON_ON_ERR_EN, 0, 1) FIELD(CXL_HDM_DECODER_GLOBAL_CONTROL, HDM_DECODER_ENABLE, 1, 1) +/* Support 4 decoders at all levels of topology */ +#define CXL_HDM_DECODER_COUNT 4 + HDM_DECODER_INIT(0); -/* Only used for HDM decoder registers block address increment */ HDM_DECODER_INIT(1); +HDM_DECODER_INIT(2); +HDM_DECODER_INIT(3); /* 8.2.5.13 - CXL Extended Security Capability Structure (Root complex only) */ #define EXTSEC_ENTRY_MAX 256 diff --git a/hw/cxl/cxl-component-utils.c b/hw/cxl/cxl-component-utils.c index c0630ba5c1..f3bbf0fd13 100644 --- a/hw/cxl/cxl-component-utils.c +++ b/hw/cxl/cxl-component-utils.c @@ -90,6 +90,9 @@ static void dumb_hdm_handler(CXLComponentState *cxl_cstate, hwaddr offset, switch (offset) { case A_CXL_HDM_DECODER0_CTRL: + case A_CXL_HDM_DECODER1_CTRL: + case A_CXL_HDM_DECODER2_CTRL: + case A_CXL_HDM_DECODER3_CTRL: should_commit = FIELD_EX32(value, CXL_HDM_DECODER0_CTRL, COMMIT); should_uncommit = !should_commit; break; @@ -129,7 +132,7 @@ static void cxl_cache_mem_write_reg(void *opaque, hwaddr offset, uint64_t value, } if (offset >= A_CXL_HDM_DECODER_CAPABILITY && - offset <= A_CXL_HDM_DECODER0_TARGET_LIST_HI) { + offset <= A_CXL_HDM_DECODER3_TARGET_LIST_HI) { dumb_hdm_handler(cxl_cstate, offset, value); } else { cregs->cache_mem_registers[offset / sizeof(*cregs->cache_mem_registers)] = value; @@ -209,7 +212,7 @@ static void ras_init_common(uint32_t *reg_state, uint32_t *write_msk) static void hdm_init_common(uint32_t *reg_state, uint32_t *write_msk, enum reg_type type) { - int decoder_count = 1; + int decoder_count = CXL_HDM_DECODER_COUNT; int hdm_inc = R_CXL_HDM_DECODER1_BASE_LO - R_CXL_HDM_DECODER0_BASE_LO; int i; diff --git a/hw/cxl/cxl-host.c b/hw/cxl/cxl-host.c index 73c5426476..2aa776c79c 100644 --- a/hw/cxl/cxl-host.c +++ b/hw/cxl/cxl-host.c @@ -97,35 +97,58 @@ void cxl_fmws_link_targets(CXLState *cxl_state, Error **errp) } } -/* TODO: support, multiple hdm decoders */ static bool cxl_hdm_find_target(uint32_t *cache_mem, hwaddr addr, uint8_t *target) { int hdm_inc = R_CXL_HDM_DECODER1_BASE_LO - R_CXL_HDM_DECODER0_BASE_LO; - uint32_t ctrl; - uint32_t ig_enc; - uint32_t iw_enc; - uint32_t target_idx; - int i = 0; - - ctrl = cache_mem[R_CXL_HDM_DECODER0_CTRL + i * hdm_inc]; - if (!FIELD_EX32(ctrl, CXL_HDM_DECODER0_CTRL, COMMITTED)) { - return false; - } - - ig_enc = FIELD_EX32(ctrl, CXL_HDM_DECODER0_CTRL, IG); - iw_enc = FIELD_EX32(ctrl, CXL_HDM_DECODER0_CTRL, IW); - target_idx = (addr / cxl_decode_ig(ig_enc)) % (1 << iw_enc); + unsigned int hdm_count; + bool found = false; + int i; + uint32_t cap; + + cap = ldl_le_p(cache_mem + R_CXL_HDM_DECODER_CAPABILITY); + hdm_count = cxl_decoder_count_dec(FIELD_EX32(cap, + CXL_HDM_DECODER_CAPABILITY, + DECODER_COUNT)); + for (i = 0; i < hdm_count; i++) { + uint32_t ctrl, ig_enc, iw_enc, target_idx; + uint32_t low, high; + uint64_t base, size; + + low = ldl_le_p(cache_mem + R_CXL_HDM_DECODER0_BASE_LO + i * hdm_inc); + high = ldl_le_p(cache_mem + R_CXL_HDM_DECODER0_BASE_HI + i * hdm_inc); + base = (low & 0xf0000000) | ((uint64_t)high << 32); + low = ldl_le_p(cache_mem + R_CXL_HDM_DECODER0_SIZE_LO + i * hdm_inc); + high = ldl_le_p(cache_mem + R_CXL_HDM_DECODER0_SIZE_HI + i * hdm_inc); + size = (low & 0xf0000000) | ((uint64_t)high << 32); + if (addr < base || addr >= base + size) { + continue; + } - if (target_idx < 4) { - *target = extract32(cache_mem[R_CXL_HDM_DECODER0_TARGET_LIST_LO], - target_idx * 8, 8); - } else { - *target = extract32(cache_mem[R_CXL_HDM_DECODER0_TARGET_LIST_HI], - (target_idx - 4) * 8, 8); + ctrl = ldl_le_p(cache_mem + R_CXL_HDM_DECODER0_CTRL + i * hdm_inc); + if (!FIELD_EX32(ctrl, CXL_HDM_DECODER0_CTRL, COMMITTED)) { + return false; + } + found = true; + ig_enc = FIELD_EX32(ctrl, CXL_HDM_DECODER0_CTRL, IG); + iw_enc = FIELD_EX32(ctrl, CXL_HDM_DECODER0_CTRL, IW); + target_idx = (addr / cxl_decode_ig(ig_enc)) % (1 << iw_enc); + + if (target_idx < 4) { + uint32_t val = ldl_le_p(cache_mem + + R_CXL_HDM_DECODER0_TARGET_LIST_LO + + i * hdm_inc); + *target = extract32(val, target_idx * 8, 8); + } else { + uint32_t val = ldl_le_p(cache_mem + + R_CXL_HDM_DECODER0_TARGET_LIST_HI + + i * hdm_inc); + *target = extract32(val, (target_idx - 4) * 8, 8); + } + break; } - return true; + return found; } static PCIDevice *cxl_cfmws_find_device(CXLFixedWindow *fw, hwaddr addr) diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index cd92813436..1658e0cc59 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -382,8 +382,6 @@ static void hdm_decoder_commit(CXLType3Dev *ct3d, int which) uint32_t *cache_mem = cregs->cache_mem_registers; uint32_t ctrl; - assert(which == 0); - ctrl = ldl_le_p(cache_mem + R_CXL_HDM_DECODER0_CTRL + which * hdm_inc); /* TODO: Sanity checks that the decoder is possible */ ctrl = FIELD_DP32(ctrl, CXL_HDM_DECODER0_CTRL, ERR, 0); @@ -399,8 +397,6 @@ static void hdm_decoder_uncommit(CXLType3Dev *ct3d, int which) uint32_t *cache_mem = cregs->cache_mem_registers; uint32_t ctrl; - assert(which == 0); - ctrl = ldl_le_p(cache_mem + R_CXL_HDM_DECODER0_CTRL + which * hdm_inc); ctrl = FIELD_DP32(ctrl, CXL_HDM_DECODER0_CTRL, ERR, 0); @@ -489,6 +485,21 @@ static void ct3d_reg_write(void *opaque, hwaddr offset, uint64_t value, should_uncommit = !should_commit; which_hdm = 0; break; + case A_CXL_HDM_DECODER1_CTRL: + should_commit = FIELD_EX32(value, CXL_HDM_DECODER0_CTRL, COMMIT); + should_uncommit = !should_commit; + which_hdm = 1; + break; + case A_CXL_HDM_DECODER2_CTRL: + should_commit = FIELD_EX32(value, CXL_HDM_DECODER0_CTRL, COMMIT); + should_uncommit = !should_commit; + which_hdm = 2; + break; + case A_CXL_HDM_DECODER3_CTRL: + should_commit = FIELD_EX32(value, CXL_HDM_DECODER0_CTRL, COMMIT); + should_uncommit = !should_commit; + which_hdm = 3; + break; case A_CXL_RAS_UNC_ERR_STATUS: { uint32_t capctrl = ldl_le_p(cache_mem + R_CXL_RAS_ERR_CAP_CTRL); @@ -760,40 +771,63 @@ static void ct3_exit(PCIDevice *pci_dev) } } -/* TODO: Support multiple HDM decoders and DPA skip */ static bool cxl_type3_dpa(CXLType3Dev *ct3d, hwaddr host_addr, uint64_t *dpa) { int hdm_inc = R_CXL_HDM_DECODER1_BASE_LO - R_CXL_HDM_DECODER0_BASE_LO; uint32_t *cache_mem = ct3d->cxl_cstate.crb.cache_mem_registers; - uint64_t decoder_base, decoder_size, hpa_offset; - uint32_t hdm0_ctrl; - int ig, iw; - int i = 0; - - decoder_base = - (((uint64_t)cache_mem[R_CXL_HDM_DECODER0_BASE_HI + i * hdm_inc] << 32) | - cache_mem[R_CXL_HDM_DECODER0_BASE_LO + i * hdm_inc]); - if ((uint64_t)host_addr < decoder_base) { - return false; - } - - hpa_offset = (uint64_t)host_addr - decoder_base; - - decoder_size = - ((uint64_t)cache_mem[R_CXL_HDM_DECODER0_SIZE_HI + i * hdm_inc] << 32) | - cache_mem[R_CXL_HDM_DECODER0_SIZE_LO + i * hdm_inc]; - if (hpa_offset >= decoder_size) { - return false; - } + unsigned int hdm_count; + uint32_t cap; + uint64_t dpa_base = 0; + int i; - hdm0_ctrl = cache_mem[R_CXL_HDM_DECODER0_CTRL + i * hdm_inc]; - iw = FIELD_EX32(hdm0_ctrl, CXL_HDM_DECODER0_CTRL, IW); - ig = FIELD_EX32(hdm0_ctrl, CXL_HDM_DECODER0_CTRL, IG); + cap = ldl_le_p(cache_mem + R_CXL_HDM_DECODER_CAPABILITY); + hdm_count = cxl_decoder_count_dec(FIELD_EX32(cap, + CXL_HDM_DECODER_CAPABILITY, + DECODER_COUNT)); + + for (i = 0; i < hdm_count; i++) { + uint64_t decoder_base, decoder_size, hpa_offset, skip; + uint32_t hdm_ctrl, low, high; + int ig, iw; + + low = ldl_le_p(cache_mem + R_CXL_HDM_DECODER0_BASE_LO + i * hdm_inc); + high = ldl_le_p(cache_mem + R_CXL_HDM_DECODER0_BASE_HI + i * hdm_inc); + decoder_base = ((uint64_t)high << 32) | (low & 0xf0000000); + + low = ldl_le_p(cache_mem + R_CXL_HDM_DECODER0_SIZE_LO + i * hdm_inc); + high = ldl_le_p(cache_mem + R_CXL_HDM_DECODER0_SIZE_HI + i * hdm_inc); + decoder_size = ((uint64_t)high << 32) | (low & 0xf0000000); + + low = ldl_le_p(cache_mem + R_CXL_HDM_DECODER0_DPA_SKIP_LO + + i * hdm_inc); + high = ldl_le_p(cache_mem + R_CXL_HDM_DECODER0_DPA_SKIP_HI + + i * hdm_inc); + skip = ((uint64_t)high << 32) | (low & 0xf0000000); + dpa_base += skip; + + hpa_offset = (uint64_t)host_addr - decoder_base; + + hdm_ctrl = ldl_le_p(cache_mem + R_CXL_HDM_DECODER0_CTRL + i * hdm_inc); + iw = FIELD_EX32(hdm_ctrl, CXL_HDM_DECODER0_CTRL, IW); + ig = FIELD_EX32(hdm_ctrl, CXL_HDM_DECODER0_CTRL, IG); + if (!FIELD_EX32(hdm_ctrl, CXL_HDM_DECODER0_CTRL, COMMITTED)) { + return false; + } + if (((uint64_t)host_addr < decoder_base) || + (hpa_offset >= decoder_size)) { + dpa_base += decoder_size / + cxl_interleave_ways_dec(iw, &error_fatal); + continue; + } - *dpa = (MAKE_64BIT_MASK(0, 8 + ig) & hpa_offset) | - ((MAKE_64BIT_MASK(8 + ig + iw, 64 - 8 - ig - iw) & hpa_offset) >> iw); + *dpa = dpa_base + + ((MAKE_64BIT_MASK(0, 8 + ig) & hpa_offset) | + ((MAKE_64BIT_MASK(8 + ig + iw, 64 - 8 - ig - iw) & hpa_offset) + >> iw)); - return true; + return true; + } + return false; } static int cxl_type3_hpa_to_as_and_dpa(CXLType3Dev *ct3d,