From patchwork Thu May 18 02:45:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ira Weiny X-Patchwork-Id: 13246072 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 750F2C77B7F for ; Thu, 18 May 2023 02:46:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229749AbjERCqD (ORCPT ); Wed, 17 May 2023 22:46:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36982 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229710AbjERCqC (ORCPT ); Wed, 17 May 2023 22:46:02 -0400 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 140BD3C02 for ; Wed, 17 May 2023 19:46:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1684377961; x=1715913961; h=from:date:subject:mime-version:content-transfer-encoding: message-id:references:in-reply-to:to:cc; bh=ALosJLPeK/xQ//Ur5NIceXWhZEbxkQaR0DQfbRf0RQM=; b=LN9hBJSfIeyF/bFp7tB1zgsCbUBBs3Wp0osAOUykokQA++cP1xfaa+qI oAvlixtQ4JgnrxkAJpJAmQH9i5qFXqoykFwxmYCDBzs2h5ng87fXopaMF dantWhvSGOr3UU0nhDOCXK2LY7gx8DXYv6859GxwOTIhJUR4wvVNAlOxm ta9kMuZlxDZeDAkKkf2L+rxDtTuy/r7QhBREQ8SD8Fmxrt8bHVgQfeAlR P8p2ozL+wBU7WKmVKLQG2DQWmp0QRafa5xaIOUC7AS4Ia6Z3icKxYgfc7 fh3GwiKTzf7Qen3UFUm2WMDzEWkeHyXL8gbppajnRLt+WPFh5760iZxSo g==; X-IronPort-AV: E=McAfee;i="6600,9927,10713"; a="380147095" X-IronPort-AV: E=Sophos;i="5.99,284,1677571200"; d="scan'208";a="380147095" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 May 2023 19:46:00 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10713"; a="652466710" X-IronPort-AV: E=Sophos;i="5.99,284,1677571200"; d="scan'208";a="652466710" Received: from iweiny-mobl.amr.corp.intel.com (HELO localhost) ([10.209.143.168]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 May 2023 19:45:59 -0700 From: Ira Weiny Date: Wed, 17 May 2023 19:45:54 -0700 Subject: [PATCH RFC 1/5] hw/cxl: Use define for build bug detection MIME-Version: 1.0 Message-Id: <20230517-rfc-type2-dev-v1-1-6eb2e470981b@intel.com> References: <20230517-rfc-type2-dev-v1-0-6eb2e470981b@intel.com> In-Reply-To: <20230517-rfc-type2-dev-v1-0-6eb2e470981b@intel.com> To: Jonathan Cameron Cc: qemu-devel@nongnu.org, linux-cxl@vger.kernel.org, Dave Jiang , Dan Williams , Ira Weiny X-Mailer: b4 0.13-dev-9a8cd X-Developer-Signature: v=1; a=ed25519-sha256; t=1684377956; l=1245; i=ira.weiny@intel.com; s=20221211; h=from:subject:message-id; bh=ALosJLPeK/xQ//Ur5NIceXWhZEbxkQaR0DQfbRf0RQM=; b=PTXLvgQFzxf6kFAXs/JRAWGbNfCHUPHulVnutVi+rrKIFhXkSG47HH5dDRPWhMLgGJcr8nGGR KT5/B7ZTijJDdHPdhFyfIS7WU2ze14Xt2rLktKwvF3c8GLqVY5beLQx X-Developer-Key: i=ira.weiny@intel.com; a=ed25519; pk=noldbkG+Wp1qXRrrkfY1QJpDf7QsOEthbOT7vm0PqsE= Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org Magic numbers can be confusing. Use the range size define for CXL.cachemem rather than a magic number. Update/add spec references. Signed-off-by: Ira Weiny --- include/hw/cxl/cxl_component.h | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/include/hw/cxl/cxl_component.h b/include/hw/cxl/cxl_component.h index 52b6a2d67f40..bca2b756c202 100644 --- a/include/hw/cxl/cxl_component.h +++ b/include/hw/cxl/cxl_component.h @@ -10,7 +10,7 @@ #ifndef CXL_COMPONENT_H #define CXL_COMPONENT_H -/* CXL 2.0 - 8.2.4 */ +/* CXL 3.0 - 8.2.3 */ #define CXL2_COMPONENT_IO_REGION_SIZE 0x1000 #define CXL2_COMPONENT_CM_REGION_SIZE 0x1000 #define CXL2_COMPONENT_BLOCK_SIZE 0x10000 @@ -173,7 +173,9 @@ HDM_DECODER_INIT(3); (CXL_IDE_REGISTERS_OFFSET + CXL_IDE_REGISTERS_SIZE) #define CXL_SNOOP_REGISTERS_SIZE 0x8 -QEMU_BUILD_BUG_MSG((CXL_SNOOP_REGISTERS_OFFSET + CXL_SNOOP_REGISTERS_SIZE) >= 0x1000, +/* CXL 3.0 8.2.3 Table 8-21 */ +QEMU_BUILD_BUG_MSG((CXL_SNOOP_REGISTERS_OFFSET + + CXL_SNOOP_REGISTERS_SIZE) >= CXL2_COMPONENT_CM_REGION_SIZE, "No space for registers"); typedef struct component_registers { From patchwork Thu May 18 02:45:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ira Weiny X-Patchwork-Id: 13246074 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4208AC77B7F for ; Thu, 18 May 2023 02:46:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229755AbjERCqI (ORCPT ); Wed, 17 May 2023 22:46:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37018 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229691AbjERCqE (ORCPT ); Wed, 17 May 2023 22:46:04 -0400 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2A64830FE for ; Wed, 17 May 2023 19:46:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1684377963; x=1715913963; h=from:date:subject:mime-version:content-transfer-encoding: message-id:references:in-reply-to:to:cc; bh=fgYFIBYZiC7VkXbsH0IZVBAWLK9AVSYJozqW9Xe6I5s=; b=LVppGaAOg35hEDhkeLizv7zmmBR+kzz3DL5XrdGxxX9lFwSk/y/TnzPH tIuT2NJ6lgsUpYLoMmrNUdq2F/Ab1qzwYlTm4SOnotRqJwcQm8jjJ7LFk tLyHUwTGFVyCyUjEnFyHijl4cXqnwMwEmlwKgtqNxWr+2vGnFxjDjcCzG w0FntxWUo31pkt9WaD3S5uBl7U/tDpySGkIXSdp5V7zMU8n1wD3YVyqoM ck+6gjYnb2GWC8IMh1oI1BWRIdpNe6X7eDKUYuKTAGHtRxBvDAY9EUeS4 p5d5EfeoSZzsOrGSDFI3/QAcsFcxU7sR5e9ec6eyXaKYUSHkla2fN8+G1 Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10713"; a="380147103" X-IronPort-AV: E=Sophos;i="5.99,284,1677571200"; d="scan'208";a="380147103" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 May 2023 19:46:02 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10713"; a="652466723" X-IronPort-AV: E=Sophos;i="5.99,284,1677571200"; d="scan'208";a="652466723" Received: from iweiny-mobl.amr.corp.intel.com (HELO localhost) ([10.209.143.168]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 May 2023 19:46:01 -0700 From: Ira Weiny Date: Wed, 17 May 2023 19:45:55 -0700 Subject: [PATCH RFC 2/5] hw/cxl: Refactor component register initialization MIME-Version: 1.0 Message-Id: <20230517-rfc-type2-dev-v1-2-6eb2e470981b@intel.com> References: <20230517-rfc-type2-dev-v1-0-6eb2e470981b@intel.com> In-Reply-To: <20230517-rfc-type2-dev-v1-0-6eb2e470981b@intel.com> To: Jonathan Cameron Cc: qemu-devel@nongnu.org, linux-cxl@vger.kernel.org, Dave Jiang , Dan Williams , Ira Weiny X-Mailer: b4 0.13-dev-9a8cd X-Developer-Signature: v=1; a=ed25519-sha256; t=1684377956; l=7352; i=ira.weiny@intel.com; s=20221211; h=from:subject:message-id; bh=fgYFIBYZiC7VkXbsH0IZVBAWLK9AVSYJozqW9Xe6I5s=; b=SW7kGCKggTN6hR3Zz1KBGJtDWJ6DutD7NzX+M9yuDiSMekVRqQFSqEyQRFrh93NOuHDLpOJLX 8aDLH5TsRMlCHn9+7wyp1Yo19PYUS2SHVYntsoWDtPqMIL13cQzaoBs X-Developer-Key: i=ira.weiny@intel.com; a=ed25519; pk=noldbkG+Wp1qXRrrkfY1QJpDf7QsOEthbOT7vm0PqsE= Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org CXL 3.0 8.2.4 Table 8-22 defines which capabilities are mandatory, not permitted, or optional for each type of device. cxl_component_register_init_common() uses a rather odd 'fall through' mechanism to define each component register set. This assumes that any device or capability being added builds on the previous devices capabilities. This is not true as there are mutually exclusive capabilities defined. For example, downstream ports can not have snoop but it can have Back Invalidate capable decoders. Refactor this code to make it easier to add individual capabilities as defined by a device type. Any capability which is not specified by the type is left NULL'ed out which complies with the packed nature of the register array. Update all spec references to 3.0. No functional changes should be seen with this patch. Signed-off-by: Ira Weiny --- hw/cxl/cxl-component-utils.c | 73 ++++++++++++++++-------------------------- include/hw/cxl/cxl_component.h | 36 ++++++++++++++------- 2 files changed, 52 insertions(+), 57 deletions(-) diff --git a/hw/cxl/cxl-component-utils.c b/hw/cxl/cxl-component-utils.c index 0e23e132ed3a..7949d12b7351 100644 --- a/hw/cxl/cxl-component-utils.c +++ b/hw/cxl/cxl-component-utils.c @@ -198,43 +198,19 @@ void cxl_component_register_init_common(uint32_t *reg_state, uint32_t *write_msk { int caps = 0; - /* - * In CXL 2.0 the capabilities required for each CXL component are such that, - * with the ordering chosen here, a single number can be used to define - * which capabilities should be provided. - */ - switch (type) { - case CXL2_DOWNSTREAM_PORT: - case CXL2_DEVICE: - /* RAS, Link */ - caps = 2; - break; - case CXL2_UPSTREAM_PORT: - case CXL2_TYPE3_DEVICE: - case CXL2_LOGICAL_DEVICE: - /* + HDM */ - caps = 3; - break; - case CXL2_ROOT_PORT: - /* + Extended Security, + Snoop */ - caps = 5; - break; - default: - abort(); - } - memset(reg_state, 0, CXL2_COMPONENT_CM_REGION_SIZE); /* CXL Capability Header Register */ ARRAY_FIELD_DP32(reg_state, CXL_CAPABILITY_HEADER, ID, 1); ARRAY_FIELD_DP32(reg_state, CXL_CAPABILITY_HEADER, VERSION, 1); ARRAY_FIELD_DP32(reg_state, CXL_CAPABILITY_HEADER, CACHE_MEM_VERSION, 1); - ARRAY_FIELD_DP32(reg_state, CXL_CAPABILITY_HEADER, ARRAY_SIZE, caps); #define init_cap_reg(reg, id, version) \ - QEMU_BUILD_BUG_ON(CXL_##reg##_REGISTERS_OFFSET == 0); \ + QEMU_BUILD_BUG_ON(CXL_##reg##_CAP_HDR_IDX == 0); \ do { \ - int which = R_CXL_##reg##_CAPABILITY_HEADER; \ + int which = CXL_##reg##_CAP_HDR_IDX; \ + if (CXL_##reg##_CAP_HDR_IDX > caps) \ + caps = CXL_##reg##_CAP_HDR_IDX; \ reg_state[which] = FIELD_DP32(reg_state[which], \ CXL_##reg##_CAPABILITY_HEADER, ID, id); \ reg_state[which] = \ @@ -245,25 +221,32 @@ void cxl_component_register_init_common(uint32_t *reg_state, uint32_t *write_msk CXL_##reg##_REGISTERS_OFFSET); \ } while (0) - init_cap_reg(RAS, 2, 2); - ras_init_common(reg_state, write_msk); - - init_cap_reg(LINK, 4, 2); - - if (caps < 3) { - return; - } - - init_cap_reg(HDM, 5, 1); - hdm_init_common(reg_state, write_msk, type); - - if (caps < 5) { - return; + /* CXL 3.0 8.2.4 Table 8-22 */ + switch (type) { + case CXL2_ROOT_PORT: + /* + Extended Security, + Snoop */ + init_cap_reg(EXTSEC, 6, 1); + init_cap_reg(SNOOP, 8, 1); + /* FALL THROUGH */ + case CXL2_UPSTREAM_PORT: + case CXL2_TYPE3_DEVICE: + case CXL2_LOGICAL_DEVICE: + /* + HDM */ + init_cap_reg(HDM, 5, 1); + hdm_init_common(reg_state, write_msk, type); + /* FALL THROUGH */ + case CXL2_DOWNSTREAM_PORT: + case CXL2_DEVICE: + /* RAS, Link */ + init_cap_reg(RAS, 2, 2); + ras_init_common(reg_state, write_msk); + init_cap_reg(LINK, 4, 2); + break; + default: + abort(); } - init_cap_reg(EXTSEC, 6, 1); - init_cap_reg(SNOOP, 8, 1); - + ARRAY_FIELD_DP32(reg_state, CXL_CAPABILITY_HEADER, ARRAY_SIZE, caps); #undef init_cap_reg } diff --git a/include/hw/cxl/cxl_component.h b/include/hw/cxl/cxl_component.h index bca2b756c202..7c08c02c5e9d 100644 --- a/include/hw/cxl/cxl_component.h +++ b/include/hw/cxl/cxl_component.h @@ -31,29 +31,41 @@ enum reg_type { }; /* - * Capability registers are defined at the top of the CXL.cache/mem region and - * are packed. For our purposes we will always define the caps in the same - * order. - * CXL 2.0 - 8.2.5 Table 142 for details. + * CXL 3.0 - 8.2.4 Table 8-22 and 8-23 + * + * Capability registers are defined at the top of the CXL.cache/mem region. + * They are defined to be packed and at variable offsets. However, NULL + * capabilities can be added to the packed array. To facilitate easier access + * within the QEMU code, define these at specified offsets. Then NULL out any + * capabilities for devices which don't (or can't) have a particular capability + * (see cxl_component_register_init_common). NULL capabilities are to be + * ignored by software. + * + * 'offsets' are based on index's which can then be used to report the array + * size in CXL Capability Header Register (index/offset 0). + * + * See CXL 3.0 Table 8-25 for an example of allowing a 'NULL' header. */ -/* CXL 2.0 - 8.2.5.1 */ +/* CXL 3.0 - 8.2.4.1 */ REG32(CXL_CAPABILITY_HEADER, 0) FIELD(CXL_CAPABILITY_HEADER, ID, 0, 16) FIELD(CXL_CAPABILITY_HEADER, VERSION, 16, 4) FIELD(CXL_CAPABILITY_HEADER, CACHE_MEM_VERSION, 20, 4) FIELD(CXL_CAPABILITY_HEADER, ARRAY_SIZE, 24, 8) -#define CXLx_CAPABILITY_HEADER(type, offset) \ - REG32(CXL_##type##_CAPABILITY_HEADER, offset) \ +/* CXL 3.0 - 8.2.4.2 on... */ +#define CXLx_CAPABILITY_HEADER(type, idx) \ + enum { CXL_##type##_CAP_HDR_IDX = idx }; \ + REG32(CXL_##type##_CAPABILITY_HEADER, (idx * 0x4)) \ FIELD(CXL_##type##_CAPABILITY_HEADER, ID, 0, 16) \ FIELD(CXL_##type##_CAPABILITY_HEADER, VERSION, 16, 4) \ FIELD(CXL_##type##_CAPABILITY_HEADER, PTR, 20, 12) -CXLx_CAPABILITY_HEADER(RAS, 0x4) -CXLx_CAPABILITY_HEADER(LINK, 0x8) -CXLx_CAPABILITY_HEADER(HDM, 0xc) -CXLx_CAPABILITY_HEADER(EXTSEC, 0x10) -CXLx_CAPABILITY_HEADER(SNOOP, 0x14) +CXLx_CAPABILITY_HEADER(RAS, 1) +CXLx_CAPABILITY_HEADER(LINK, 2) +CXLx_CAPABILITY_HEADER(HDM, 3) +CXLx_CAPABILITY_HEADER(EXTSEC, 4) +CXLx_CAPABILITY_HEADER(SNOOP, 5) /* * Capability structures contain the actual registers that the CXL component From patchwork Thu May 18 02:45:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ira Weiny X-Patchwork-Id: 13246075 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1460AC77B7F for ; Thu, 18 May 2023 02:46:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229779AbjERCqK (ORCPT ); Wed, 17 May 2023 22:46:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37096 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229787AbjERCqH (ORCPT ); Wed, 17 May 2023 22:46:07 -0400 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1F1A23C30 for ; Wed, 17 May 2023 19:46:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1684377965; x=1715913965; h=from:date:subject:mime-version:content-transfer-encoding: message-id:references:in-reply-to:to:cc; bh=Zvl4Us5nFwXM+K5uC8NSDV+qsek+5uIPfbxF6GEL2Jw=; b=l4nOO0ZxbLfXNUjHufWl2mVjxXiJfMgmJaq3RDafD3inKAkG8Qr8rMt0 3h8moX28Cahf2GkTSKbdioHf/qyo+xl5QLq2aLES05VL8oLAE6CvdVgOd obR0TQ0ku2CaK0wxODKkBehj7lQSEBjJWeA9OmMrH27TYfFbKeaCtRddq rD8+v/yXEW7n1MxM2FTTf6o7+bdr7WSAiKXx+n0PTm33DgPb86CleFC86 xqYqoxVIAQ9mpCaD7dlmxxAUDb+eFFs8ddxcC7r1zt+FRCMmbkLqkBah8 CzEVeNUjz80M0t5HBwOSoO8DjF20w/oOSN8mlcV26eyqaz/QVorgJ4GnY w==; X-IronPort-AV: E=McAfee;i="6600,9927,10713"; a="380147108" X-IronPort-AV: E=Sophos;i="5.99,284,1677571200"; d="scan'208";a="380147108" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 May 2023 19:46:04 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10713"; a="652466733" X-IronPort-AV: E=Sophos;i="5.99,284,1677571200"; d="scan'208";a="652466733" Received: from iweiny-mobl.amr.corp.intel.com (HELO localhost) ([10.209.143.168]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 May 2023 19:46:03 -0700 From: Ira Weiny Date: Wed, 17 May 2023 19:45:56 -0700 Subject: [PATCH RFC 3/5] hw/cxl: Derive a CXL accelerator device from Type-3 MIME-Version: 1.0 Message-Id: <20230517-rfc-type2-dev-v1-3-6eb2e470981b@intel.com> References: <20230517-rfc-type2-dev-v1-0-6eb2e470981b@intel.com> In-Reply-To: <20230517-rfc-type2-dev-v1-0-6eb2e470981b@intel.com> To: Jonathan Cameron Cc: qemu-devel@nongnu.org, linux-cxl@vger.kernel.org, Dave Jiang , Dan Williams , Ira Weiny X-Mailer: b4 0.13-dev-9a8cd X-Developer-Signature: v=1; a=ed25519-sha256; t=1684377956; l=4453; i=ira.weiny@intel.com; s=20221211; h=from:subject:message-id; bh=Zvl4Us5nFwXM+K5uC8NSDV+qsek+5uIPfbxF6GEL2Jw=; b=t6ps/6MKxcDQEG2REfSRjLaEhVncO5i5rhmh8QiXSfeH4Z7SDLmaKDpZXI5Xv19k8ydbyl7uA GyZTLf0eDebDoXj3C3zJZIv+0RLg919zQo6QcqKHakcGW/zC8UkERx6 X-Developer-Key: i=ira.weiny@intel.com; a=ed25519; pk=noldbkG+Wp1qXRrrkfY1QJpDf7QsOEthbOT7vm0PqsE= Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org It is desirable to have a test accelerator device to present various accelerator features such as Back-Invalidate (BI) registers to OS software for testing. BI accelerator devices require memory that can be easily modeled as a sub-class of Type-3 device. Derive 'cxl-accel' from cxl-type3. Add documentation for such a device. Follow on patches will add BI registers and other simulation of the accelerator device. Adding devices qemu can be done with the following example: ... -device cxl-accel,bus=sw0p0,volatile-memdev=cxl-ac-mem5,id=cxl-dev5,sn=0xCAFE0005 ... Not-Yet-Signed-off-by: Ira Weiny --- The device ID and class code are completely made up by me. As discussed in the last community call perhaps these could be declared in some more official capacity? --- docs/system/devices/cxl.rst | 11 +++++++++++ hw/mem/cxl_type3.c | 28 ++++++++++++++++++++++++++++ include/hw/cxl/cxl_device.h | 16 ++++++++++++++++ include/hw/pci/pci_ids.h | 1 + 4 files changed, 56 insertions(+) diff --git a/docs/system/devices/cxl.rst b/docs/system/devices/cxl.rst index 95900252c56a..5bc931be44b3 100644 --- a/docs/system/devices/cxl.rst +++ b/docs/system/devices/cxl.rst @@ -321,6 +321,17 @@ A very simple setup with just one directly attached CXL Type 3 Volatile Memory d -device cxl-type3,bus=root_port13,volatile-memdev=vmem0,id=cxl-vmem0 \ -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G +A very simple setup with just one directly attached CXL Type 2 Volatile Memory +Accelerator device:: + + qemu-system-x86_64 -M q35,cxl=on -m 4G,maxmem=8G,slots=8 -smp 4 \ + ... + -object memory-backend-ram,id=vmem0,share=on,size=256M \ + -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \ + -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \ + -device cxl-accel,bus=root_port13,volatile-memdev=vmem0,id=cxl-accel0 \ + -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G + The same volatile setup may optionally include an LSA region:: qemu-system-aarch64 -M virt,gic-version=3,cxl=on -m 4g,maxmem=8G,slots=8 -cpu max \ diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 3e63dbd83551..c7eafd76d1ea 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -1691,3 +1691,31 @@ static void ct3d_registers(void) } type_init(ct3d_registers); + +static void cxl_accel_class_init(ObjectClass *oc, void *data) +{ + DeviceClass *dc = DEVICE_CLASS(oc); + PCIDeviceClass *pc = PCI_DEVICE_CLASS(oc); + + pc->class_id = PCI_CLASS_CXL_QEMU_ACCEL; + pc->vendor_id = PCI_VENDOR_ID_INTEL; + pc->device_id = 0xd94; /* LVF for now */ + pc->revision = 1; + + dc->desc = "CXL Accelerator Device (Type 2)"; +} + +static const TypeInfo cxl_accel_dev_info = { + .name = TYPE_CXL_ACCEL, + .parent = TYPE_CXL_TYPE3, + .class_size = sizeof(struct CXLAccelClass), + .class_init = cxl_accel_class_init, + .instance_size = sizeof(CXLAccelDev), +}; + +static void cxl_accel_dev_registers(void) +{ + type_register_static(&cxl_accel_dev_info); +} + +type_init(cxl_accel_dev_registers); diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index cd7f28dba884..f7f6688ee6e2 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -432,6 +432,22 @@ struct CXLType3Class { bool (*set_cacheline)(CXLType3Dev *ct3d, uint64_t dpa_offset, uint8_t *data); }; +/* + * Accel devices are a type3 device but with additional functionality. + */ +struct CXLAccelDev { + /* Private: Must be first */ + CXLType3Dev parent_obj; +}; + +struct CXLAccelClass { + /* Private: Must be first */ + CXLType3Class parent_class; +}; + +#define TYPE_CXL_ACCEL "cxl-accel" +OBJECT_DECLARE_TYPE(CXLAccelDev, CXLAccelClass, CXL_ACCEL) + struct CSWMBCCIDev { PCIDevice parent_obj; CXLComponentState cxl_cstate; diff --git a/include/hw/pci/pci_ids.h b/include/hw/pci/pci_ids.h index e4386ebb2038..2dbf350ebba4 100644 --- a/include/hw/pci/pci_ids.h +++ b/include/hw/pci/pci_ids.h @@ -54,6 +54,7 @@ #define PCI_CLASS_MEMORY_RAM 0x0500 #define PCI_CLASS_MEMORY_FLASH 0x0501 #define PCI_CLASS_MEMORY_CXL 0x0502 +#define PCI_CLASS_CXL_QEMU_ACCEL 0x0503 #define PCI_CLASS_MEMORY_OTHER 0x0580 #define PCI_BASE_CLASS_BRIDGE 0x06 From patchwork Thu May 18 02:45:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ira Weiny X-Patchwork-Id: 13246076 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 388DBC77B7F for ; Thu, 18 May 2023 02:46:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229453AbjERCqS (ORCPT ); Wed, 17 May 2023 22:46:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37122 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229751AbjERCqN (ORCPT ); Wed, 17 May 2023 22:46:13 -0400 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 64AEA40DF for ; Wed, 17 May 2023 19:46:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1684377967; x=1715913967; h=from:date:subject:mime-version:content-transfer-encoding: message-id:references:in-reply-to:to:cc; bh=46y0FaBxziwuSbTAR4YOkMaktxaMBddzA2VrCi6VZS8=; b=ShLqbcxbur7kcUeKoCv/HTZsvnKM9QtqpNbScZAUJxp9+vZpZtvNtsOD vZlsVXeUUmSZZTvtTv2KZB1iImgWuiSa8kDxnDqMuBXRJYAhGbsFUHkt/ Ur1JBaqkoter3x1MQW28FGu2gVaOF4QvBawRprF7FKRHVK7k6/0Der+h2 iTosZCJ7fuoJAFIzmqMdohsMw3t7KM8Y1QuaZVM2uRhkN0tls9xLlyFlB CnFdobVobK/u0c6S/hmhI2KmsCq0A4VkzjUmHFvrQ0iIp5NLHJrUAlsqU 2GmzXGnDIo3Z5sIObABsZf5NsNpBVV8gULhlA4f7x+fph/x/yeqzktW1E g==; X-IronPort-AV: E=McAfee;i="6600,9927,10713"; a="380147111" X-IronPort-AV: E=Sophos;i="5.99,284,1677571200"; d="scan'208";a="380147111" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 May 2023 19:46:06 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10713"; a="652466741" X-IronPort-AV: E=Sophos;i="5.99,284,1677571200"; d="scan'208";a="652466741" Received: from iweiny-mobl.amr.corp.intel.com (HELO localhost) ([10.209.143.168]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 May 2023 19:46:05 -0700 From: Ira Weiny Date: Wed, 17 May 2023 19:45:57 -0700 Subject: [PATCH RFC 4/5] hw/cxl/accel: Add Back-Invalidate decoder capbility structure MIME-Version: 1.0 Message-Id: <20230517-rfc-type2-dev-v1-4-6eb2e470981b@intel.com> References: <20230517-rfc-type2-dev-v1-0-6eb2e470981b@intel.com> In-Reply-To: <20230517-rfc-type2-dev-v1-0-6eb2e470981b@intel.com> To: Jonathan Cameron Cc: qemu-devel@nongnu.org, linux-cxl@vger.kernel.org, Dave Jiang , Dan Williams , Ira Weiny X-Mailer: b4 0.13-dev-9a8cd X-Developer-Signature: v=1; a=ed25519-sha256; t=1684377956; l=3758; i=ira.weiny@intel.com; s=20221211; h=from:subject:message-id; bh=46y0FaBxziwuSbTAR4YOkMaktxaMBddzA2VrCi6VZS8=; b=Tkagk9parcBDGYn7JPISiNLaEp7zYK69R4NQPXePCpSdSOpb2z/s+4AHvAKQ/quxnhs3APqae lCSEEGALSI9D2QWgJyqJl/rTjUs087YuKIgDiLdsbb2DJ8AVIAj1DyX X-Developer-Key: i=ira.weiny@intel.com; a=ed25519; pk=noldbkG+Wp1qXRrrkfY1QJpDf7QsOEthbOT7vm0PqsE= Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org The presence of the Back-Invalidate (BI) decoder capability structure indicates a CXL downstream port, root port, or device supports the BI messages. Add the BI capability structure to the accelerator device. Not-Yet-Signed-off-by: Ira Weiny --- hw/cxl/cxl-component-utils.c | 5 +++++ hw/mem/cxl_type3.c | 11 +++++++++++ include/hw/cxl/cxl_component.h | 11 +++++++++-- 3 files changed, 25 insertions(+), 2 deletions(-) diff --git a/hw/cxl/cxl-component-utils.c b/hw/cxl/cxl-component-utils.c index 7949d12b7351..a9efa252b4ae 100644 --- a/hw/cxl/cxl-component-utils.c +++ b/hw/cxl/cxl-component-utils.c @@ -228,6 +228,7 @@ void cxl_component_register_init_common(uint32_t *reg_state, uint32_t *write_msk init_cap_reg(EXTSEC, 6, 1); init_cap_reg(SNOOP, 8, 1); /* FALL THROUGH */ + case CXL3_TYPE2_DEVICE: case CXL2_UPSTREAM_PORT: case CXL2_TYPE3_DEVICE: case CXL2_LOGICAL_DEVICE: @@ -246,6 +247,10 @@ void cxl_component_register_init_common(uint32_t *reg_state, uint32_t *write_msk abort(); } + if (type == CXL3_TYPE2_DEVICE) { + init_cap_reg(BI_DECODER, 12, 1); + } + ARRAY_FIELD_DP32(reg_state, CXL_CAPABILITY_HEADER, ARRAY_SIZE, caps); #undef init_cap_reg } diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index c7eafd76d1ea..95fdaaa18f37 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -1692,6 +1692,16 @@ static void ct3d_registers(void) type_init(ct3d_registers); +static void accel_reset(DeviceState *dev) +{ + CXLAccelDev *acceld = CXL_ACCEL(dev); + uint32_t *reg_state = acceld->parent_obj.cxl_cstate.crb.cache_mem_registers; + uint32_t *write_msk = acceld->parent_obj.cxl_cstate.crb.cache_mem_regs_write_mask; + + cxl_component_register_init_common(reg_state, write_msk, CXL3_TYPE2_DEVICE); + cxl_device_register_init_common(&acceld->parent_obj.cxl_dstate); +} + static void cxl_accel_class_init(ObjectClass *oc, void *data) { DeviceClass *dc = DEVICE_CLASS(oc); @@ -1703,6 +1713,7 @@ static void cxl_accel_class_init(ObjectClass *oc, void *data) pc->revision = 1; dc->desc = "CXL Accelerator Device (Type 2)"; + dc->reset = accel_reset; } static const TypeInfo cxl_accel_dev_info = { diff --git a/include/hw/cxl/cxl_component.h b/include/hw/cxl/cxl_component.h index 7c08c02c5e9d..a5b5512aed94 100644 --- a/include/hw/cxl/cxl_component.h +++ b/include/hw/cxl/cxl_component.h @@ -28,6 +28,7 @@ enum reg_type { CXL2_UPSTREAM_PORT, CXL2_DOWNSTREAM_PORT, CXL3_SWITCH_MAILBOX_CCI, + CXL3_TYPE2_DEVICE, }; /* @@ -66,6 +67,7 @@ CXLx_CAPABILITY_HEADER(LINK, 2) CXLx_CAPABILITY_HEADER(HDM, 3) CXLx_CAPABILITY_HEADER(EXTSEC, 4) CXLx_CAPABILITY_HEADER(SNOOP, 5) +CXLx_CAPABILITY_HEADER(BI_DECODER, 6) /* * Capability structures contain the actual registers that the CXL component @@ -185,9 +187,14 @@ HDM_DECODER_INIT(3); (CXL_IDE_REGISTERS_OFFSET + CXL_IDE_REGISTERS_SIZE) #define CXL_SNOOP_REGISTERS_SIZE 0x8 +/* CXL 3.0 8.2.4.26 - CXL BI Decoder Capability Structure */ +#define CXL_BI_DECODER_REGISTERS_OFFSET \ + (CXL_SNOOP_REGISTERS_OFFSET + CXL_SNOOP_REGISTERS_SIZE) +#define CXL_BI_DECODER_REGISTERS_SIZE 0xC + /* CXL 3.0 8.2.3 Table 8-21 */ -QEMU_BUILD_BUG_MSG((CXL_SNOOP_REGISTERS_OFFSET + - CXL_SNOOP_REGISTERS_SIZE) >= CXL2_COMPONENT_CM_REGION_SIZE, +QEMU_BUILD_BUG_MSG((CXL_BI_DECODER_REGISTERS_OFFSET + + CXL_BI_DECODER_REGISTERS_SIZE) >= CXL2_COMPONENT_CM_REGION_SIZE, "No space for registers"); typedef struct component_registers { From patchwork Thu May 18 02:45:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ira Weiny X-Patchwork-Id: 13246077 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21977C77B7D for ; Thu, 18 May 2023 02:46:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229668AbjERCqg (ORCPT ); Wed, 17 May 2023 22:46:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37686 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229691AbjERCqf (ORCPT ); Wed, 17 May 2023 22:46:35 -0400 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D8D5A44BF for ; Wed, 17 May 2023 19:46:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1684377973; x=1715913973; h=from:date:subject:mime-version:content-transfer-encoding: message-id:references:in-reply-to:to:cc; bh=+qdtcDX7aPtUFG16xTDOpRYYrPeh1b8whv2sXdCmkMg=; b=QSaNJ/94QkEepeumsBJpsWMI/ftuws0ZFB9GV/QVVXNW/Sa5SwL8k0zH Pon66k4La8rc6uLgwGg4OfKekdo2ac1fjpdCfLJ1+MEnjD0O2d8Je3eVw AQ8DxZEZr7j5jL4DvMxC+548q+Y8DTS4IkeD7POe0jXkPSg2hBYGvqdnd VXbWNmqr57wsO2QnQg6mm0bsDJQsgVBNyspxodHV3LY0Q+51UI9aLQMAt qK38v7N1lArB8pvsr2mqhNtKi2AMttZN6bdgDJSK0JrQCAhh5hVwh/zn9 RRUEnfI/2VwqjoMNp0WhoG0SLIIxHpejZTNUI20E3mXYllCBSyW2/Sx4j g==; X-IronPort-AV: E=McAfee;i="6600,9927,10713"; a="380147117" X-IronPort-AV: E=Sophos;i="5.99,284,1677571200"; d="scan'208";a="380147117" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 May 2023 19:46:12 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10713"; a="652466748" X-IronPort-AV: E=Sophos;i="5.99,284,1677571200"; d="scan'208";a="652466748" Received: from iweiny-mobl.amr.corp.intel.com (HELO localhost) ([10.209.143.168]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 May 2023 19:46:07 -0700 From: Ira Weiny Date: Wed, 17 May 2023 19:45:58 -0700 Subject: [PATCH RFC 5/5] hw/cxl: Add UIO HDM decoder register fields MIME-Version: 1.0 Message-Id: <20230517-rfc-type2-dev-v1-5-6eb2e470981b@intel.com> References: <20230517-rfc-type2-dev-v1-0-6eb2e470981b@intel.com> In-Reply-To: <20230517-rfc-type2-dev-v1-0-6eb2e470981b@intel.com> To: Jonathan Cameron Cc: qemu-devel@nongnu.org, linux-cxl@vger.kernel.org, Dave Jiang , Dan Williams , Ira Weiny X-Mailer: b4 0.13-dev-9a8cd X-Developer-Signature: v=1; a=ed25519-sha256; t=1684377956; l=2286; i=ira.weiny@intel.com; s=20221211; h=from:subject:message-id; bh=+qdtcDX7aPtUFG16xTDOpRYYrPeh1b8whv2sXdCmkMg=; b=1qJL70bed0jThaHUc+ucsX3tY2DnXquwk6PxAVz4qUOpUvEduJBAk0C6X1XNMpJZ9VieRM+dh z7G4wmZ4T7NAM6KU+UIXl7us0rSNk2ByujqzAbOiftoX55N4kSxj3EO X-Developer-Key: i=ira.weiny@intel.com; a=ed25519; pk=noldbkG+Wp1qXRrrkfY1QJpDf7QsOEthbOT7vm0PqsE= Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org HDM decoders optionally support Unordered IO (UIO) access. Devices indicate UIO support by setting the capable bit. Software can then set up to UIO decoder count HDM's as UIO enabled when configuring the HDMs on the device. Define the UIO capable bit and decoder count. Default type 2 devices to support UIO for testing. Not-Yet-Signed-off-by: Ira Weiny --- hw/cxl/cxl-component-utils.c | 6 ++++++ include/hw/cxl/cxl_component.h | 2 ++ 2 files changed, 8 insertions(+) diff --git a/hw/cxl/cxl-component-utils.c b/hw/cxl/cxl-component-utils.c index a9efa252b4ae..252b2beb2110 100644 --- a/hw/cxl/cxl-component-utils.c +++ b/hw/cxl/cxl-component-utils.c @@ -173,6 +173,12 @@ static void hdm_init_common(uint32_t *reg_state, uint32_t *write_msk, ARRAY_FIELD_DP32(reg_state, CXL_HDM_DECODER_CAPABILITY, INTERLEAVE_256B, 1); ARRAY_FIELD_DP32(reg_state, CXL_HDM_DECODER_CAPABILITY, INTERLEAVE_4K, 1); ARRAY_FIELD_DP32(reg_state, CXL_HDM_DECODER_CAPABILITY, POISON_ON_ERR_CAP, 0); + if (type == CXL3_TYPE2_DEVICE) { + ARRAY_FIELD_DP32(reg_state, CXL_HDM_DECODER_CAPABILITY, UIO_CAPABLE, 1); + ARRAY_FIELD_DP32(reg_state, CXL_HDM_DECODER_CAPABILITY, UIO_DECODER_CNT, + decoder_count); + } + ARRAY_FIELD_DP32(reg_state, CXL_HDM_DECODER_GLOBAL_CONTROL, HDM_DECODER_ENABLE, 0); write_msk[R_CXL_HDM_DECODER_GLOBAL_CONTROL] = 0x3; diff --git a/include/hw/cxl/cxl_component.h b/include/hw/cxl/cxl_component.h index a5b5512aed94..7c24e699ef80 100644 --- a/include/hw/cxl/cxl_component.h +++ b/include/hw/cxl/cxl_component.h @@ -162,6 +162,8 @@ REG32(CXL_HDM_DECODER_CAPABILITY, CXL_HDM_REGISTERS_OFFSET) FIELD(CXL_HDM_DECODER_CAPABILITY, INTERLEAVE_256B, 8, 1) FIELD(CXL_HDM_DECODER_CAPABILITY, INTERLEAVE_4K, 9, 1) FIELD(CXL_HDM_DECODER_CAPABILITY, POISON_ON_ERR_CAP, 10, 1) + FIELD(CXL_HDM_DECODER_CAPABILITY, UIO_CAPABLE, 13, 1) + FIELD(CXL_HDM_DECODER_CAPABILITY, UIO_DECODER_CNT, 16, 4) REG32(CXL_HDM_DECODER_GLOBAL_CONTROL, CXL_HDM_REGISTERS_OFFSET + 4) FIELD(CXL_HDM_DECODER_GLOBAL_CONTROL, POISON_ON_ERR_EN, 0, 1) FIELD(CXL_HDM_DECODER_GLOBAL_CONTROL, HDM_DECODER_ENABLE, 1, 1)