From patchwork Thu May 23 17:44:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13672117 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E6E0DC25B75 for ; Thu, 23 May 2024 17:51:21 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sACZ6-00069A-HU; Thu, 23 May 2024 13:49:20 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sACYh-0005wz-TQ for qemu-devel@nongnu.org; Thu, 23 May 2024 13:49:02 -0400 Received: from mail-pl1-x62c.google.com ([2607:f8b0:4864:20::62c]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sACYT-0003e1-Nh for qemu-devel@nongnu.org; Thu, 23 May 2024 13:48:47 -0400 Received: by mail-pl1-x62c.google.com with SMTP id d9443c01a7336-1f2ecea41deso118211815ad.1 for ; Thu, 23 May 2024 10:48:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716486519; x=1717091319; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sUY+tVIrLXlBWDYAZLBMIzgstxLzMinOnAd95xKbspQ=; b=lNnosEGQJm4g6YCEo9UHHvg4wxzOxIWXviyMqt8/VXfDeayg4wYPbIKzgVVvrbhg5F BSnsCOd1pVOJ5cwBRvpUC4WU0vvdSGyp9VbUj+e6LfJ4f3rkPV1GwVWs/aDUFigGMSNO pezxRGv6aSxNBb5mfrp33GORCl4+0vMI7RNgQG1ROHuPgVubmkYYBIDmuV+4uZOzMP8V J1Of04rHmmTXTnSKOZVmo9Ebw+E0Iiqdu3/gmyrVykhhhJl0OyhLe8Ew+vxdpmEf1TeS T6bv+rzDWQiyw5aMCrcN1dt+jEtw5SrK/arJ31fhqoOM5kMG3+P7LL2eQ7JH3EGiK9ac V95Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716486519; x=1717091319; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sUY+tVIrLXlBWDYAZLBMIzgstxLzMinOnAd95xKbspQ=; b=oWyzxfJTtlqBJvzRuegeTObiU9m+S54H+B+rVy2bI4n02EvkJ6TRNDR13xBNeBISus ff0hzMV+KTyYU+b2U7sxDvaKkxYYu7vVHGosrETKNV5uMRRIEu/3lmt9M8SwCT4kpGgi DUAm28XdgkijrksCJwP4/Z2yIWIbWVszpLf38lGil7ExkreYp0Rd/XiFw9/0pArj1t0t xg7ba4SQX4Pa2JWfCp+CwRbMFEMFMf2gl00OHx8uYRIWLtlv1qFjyiGT4uyJxDOAi7rE efIM+V/SyuNrOl9wTh9Ty6UZdFudRhk84rwOHOm/2gXaj9qBBhWDdwOz9Z9y9JAvYaPu n7CQ== X-Gm-Message-State: AOJu0YxxASrwZGj2/hTePV0BF8RS6Ixy1JYDT2Y/AFy0EIZGC/486nYQ Tf+udyWdOHCs6S9ADcGNSwGNQbwkMtfxKvgKG6QIuZqLl6QXJ/0stiDYCQ== X-Google-Smtp-Source: AGHT+IFf20NX3YEVB79xFwOa3YX1k2qEZb8NgiEQW9xioF3Hrc65GPKe5k6j4CXoBHvaPjkpWIscPA== X-Received: by 2002:a17:903:22c6:b0:1ec:e3c2:790e with SMTP id d9443c01a7336-1f4486fe183mr792505ad.19.1716486519218; Thu, 23 May 2024 10:48:39 -0700 (PDT) Received: from localhost.localdomain ([2601:646:8f03:9fee:4f9e:216e:ed5a:642d]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1ef0c137e7esm253436265ad.257.2024.05.23.10.48.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 May 2024 10:48:38 -0700 (PDT) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, armbru@redhat.com, mst@redhat.com, Gregory Price , Jonathan Cameron , Fan Ni Subject: [PATCH v8 01/14] hw/cxl/mailbox: change CCI cmd set structure to be a member, not a reference Date: Thu, 23 May 2024 10:44:41 -0700 Message-ID: <20240523174651.1089554-2-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240523174651.1089554-1-nifan.cxl@gmail.com> References: <20240523174651.1089554-1-nifan.cxl@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62c; envelope-from=nifan.cxl@gmail.com; helo=mail-pl1-x62c.google.com X-Spam_score_int: -16 X-Spam_score: -1.7 X-Spam_bar: - X-Spam_report: (-1.7 / 5.0 requ) BAYES_00=-1.9, DKIM_INVALID=0.1, DKIM_SIGNED=0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Gregory Price This allows devices to have fully customized CCIs, along with complex devices where wrapper devices can override or add additional CCI commands without having to replicate full command structures or pollute a base device with every command that might ever be used. Signed-off-by: Gregory Price Signed-off-by: Jonathan Cameron Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 19 +++++++++++++++---- include/hw/cxl/cxl_device.h | 2 +- 2 files changed, 16 insertions(+), 5 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index e5eb97cb91..2c9f50f0f9 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -1447,10 +1447,21 @@ void cxl_init_cci(CXLCCI *cci, size_t payload_max) bg_timercb, cci); } +static void cxl_copy_cci_commands(CXLCCI *cci, const struct cxl_cmd (*cxl_cmds)[256]) +{ + for (int set = 0; set < 256; set++) { + for (int cmd = 0; cmd < 256; cmd++) { + if (cxl_cmds[set][cmd].handler) { + cci->cxl_cmd_set[set][cmd] = cxl_cmds[set][cmd]; + } + } + } +} + void cxl_initialize_mailbox_swcci(CXLCCI *cci, DeviceState *intf, DeviceState *d, size_t payload_max) { - cci->cxl_cmd_set = cxl_cmd_set_sw; + cxl_copy_cci_commands(cci, cxl_cmd_set_sw); cci->d = d; cci->intf = intf; cxl_init_cci(cci, payload_max); @@ -1458,7 +1469,7 @@ void cxl_initialize_mailbox_swcci(CXLCCI *cci, DeviceState *intf, void cxl_initialize_mailbox_t3(CXLCCI *cci, DeviceState *d, size_t payload_max) { - cci->cxl_cmd_set = cxl_cmd_set; + cxl_copy_cci_commands(cci, cxl_cmd_set); cci->d = d; /* No separation for PCI MB as protocol handled in PCI device */ @@ -1476,7 +1487,7 @@ static const struct cxl_cmd cxl_cmd_set_t3_ld[256][256] = { void cxl_initialize_t3_ld_cci(CXLCCI *cci, DeviceState *d, DeviceState *intf, size_t payload_max) { - cci->cxl_cmd_set = cxl_cmd_set_t3_ld; + cxl_copy_cci_commands(cci, cxl_cmd_set_t3_ld); cci->d = d; cci->intf = intf; cxl_init_cci(cci, payload_max); @@ -1496,7 +1507,7 @@ void cxl_initialize_t3_fm_owned_ld_mctpcci(CXLCCI *cci, DeviceState *d, DeviceState *intf, size_t payload_max) { - cci->cxl_cmd_set = cxl_cmd_set_t3_fm_owned_ld_mctp; + cxl_copy_cci_commands(cci, cxl_cmd_set_t3_fm_owned_ld_mctp); cci->d = d; cci->intf = intf; cxl_init_cci(cci, payload_max); diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index 279b276bda..ccc4611875 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -164,7 +164,7 @@ typedef struct CXLEventLog { } CXLEventLog; typedef struct CXLCCI { - const struct cxl_cmd (*cxl_cmd_set)[256]; + struct cxl_cmd cxl_cmd_set[256][256]; struct cel_log { uint16_t opcode; uint16_t effect; From patchwork Thu May 23 17:44:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13672111 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EA682C25B7A for ; Thu, 23 May 2024 17:50:05 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sACZQ-0006fA-Ks; Thu, 23 May 2024 13:49:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sACYh-0005wW-G1 for qemu-devel@nongnu.org; Thu, 23 May 2024 13:49:02 -0400 Received: from mail-pl1-x62f.google.com ([2607:f8b0:4864:20::62f]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sACYX-0003eD-Jb for qemu-devel@nongnu.org; Thu, 23 May 2024 13:48:46 -0400 Received: by mail-pl1-x62f.google.com with SMTP id d9443c01a7336-1f332511457so6149485ad.2 for ; Thu, 23 May 2024 10:48:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716486521; x=1717091321; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OwLDZR/KWKZ72BeL4YQ5O0fvqFWZ5B2YkSZHDV4y31s=; b=fHyD3oq9KCLKeOoulFDLaWa5FKUJupCINkJvj4qugOdy+GEIDjIU/sC8c6KfVe1JLd rCm6FDKtL3h5pStULNofsKAiLxL3G8dCqJF+X7SnRRpDT4OIKo9ycuIdgwb3lyP5hBpT SU36y1GOsT/CNFArh00+eDYdjpu7AuWlZE+h8TvRdwF9U9/C76w34hR6S4wToECIs47i M2xIZsilJf1bMP7wGYwXnx6640p2NVswgc/C9EaPYQOXEAkgwI7M/cZAueJY5UeRZVSu G2YuLyM62Uue/IKMZc2l1QpT0a/1I9DXJKkgxS2emrx/iQg6mr75bsQ0L0kntdT6podO xDJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716486521; x=1717091321; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OwLDZR/KWKZ72BeL4YQ5O0fvqFWZ5B2YkSZHDV4y31s=; b=j2sd6N9yLWQv7yd2wbzHsa3PYxzsOBXmK3E8qGBt035KELQ1TVLu1pe4tKlq+leroz NDy/fWxSKwbr1a5pUeaE/QdPEWlLoGK/Xy0tP8q3tr79PO5IICM34vQYsHXe8yJw5+9n cFKCECbZ63iXdm40F0RcrOdHRB/NawkviLDkR0UEQyA+eVbOvMK/hmvUq8YnRgBfwoJq l1VwrNMFe89UHcJPDfX3mCaHhEAPFSLTqxbeiB2YyUwgjlAxI/a8vtRkKoTO4g7xiWC5 UQPHCFGYwR3dfX3H2ygjb5gW168imvOL81f6BB4qnfS9ZlDl9v7nN5OxFuRbIODWbDvc udwA== X-Gm-Message-State: AOJu0YwaXsQD0wOMl7/FnRA4OGkhjsqgJ/uWn+59QAwDgKEhE45ffuhi V/QMNZQhksfkDllWKZuhlHwF8G1ee8jxtn2zH4Q6a+UfLuySMPWn/VEfcg== X-Google-Smtp-Source: AGHT+IGMzDozcFp7jk5pIqT2r+oB433amL8mIwl7y+Q6sXnIRKqq4W15tY3TetnNKZWNSELBHrLsYQ== X-Received: by 2002:a17:903:1c8:b0:1f3:61c:30a2 with SMTP id d9443c01a7336-1f4486d4ffamr418345ad.2.1716486520814; Thu, 23 May 2024 10:48:40 -0700 (PDT) Received: from localhost.localdomain ([2601:646:8f03:9fee:4f9e:216e:ed5a:642d]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1ef0c137e7esm253436265ad.257.2024.05.23.10.48.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 May 2024 10:48:40 -0700 (PDT) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, armbru@redhat.com, mst@redhat.com, Gregory Price , Jonathan Cameron , Fan Ni Subject: [PATCH v8 02/14] hw/cxl/mailbox: interface to add CCI commands to an existing CCI Date: Thu, 23 May 2024 10:44:42 -0700 Message-ID: <20240523174651.1089554-3-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240523174651.1089554-1-nifan.cxl@gmail.com> References: <20240523174651.1089554-1-nifan.cxl@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62f; envelope-from=nifan.cxl@gmail.com; helo=mail-pl1-x62f.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Gregory Price This enables wrapper devices to customize the base device's CCI (for example, with custom commands outside the specification) without the need to change the base device. The also enabled the base device to dispatch those commands without requiring additional driver support. Signed-off-by: Gregory Price Signed-off-by: Jonathan Cameron Signed-off-by: Fan Ni --- Heavily edited by Jonathan Cameron to increase code reuse --- hw/cxl/cxl-mailbox-utils.c | 19 +++++++++++++++++-- include/hw/cxl/cxl_device.h | 2 ++ 2 files changed, 19 insertions(+), 2 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 2c9f50f0f9..4bcd727f4c 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -1424,9 +1424,9 @@ static void bg_timercb(void *opaque) } } -void cxl_init_cci(CXLCCI *cci, size_t payload_max) +static void cxl_rebuild_cel(CXLCCI *cci) { - cci->payload_max = payload_max; + cci->cel_size = 0; /* Reset for a fresh build */ for (int set = 0; set < 256; set++) { for (int cmd = 0; cmd < 256; cmd++) { if (cci->cxl_cmd_set[set][cmd].handler) { @@ -1440,6 +1440,13 @@ void cxl_init_cci(CXLCCI *cci, size_t payload_max) } } } +} + +void cxl_init_cci(CXLCCI *cci, size_t payload_max) +{ + cci->payload_max = payload_max; + cxl_rebuild_cel(cci); + cci->bg.complete_pct = 0; cci->bg.starttime = 0; cci->bg.runtime = 0; @@ -1458,6 +1465,14 @@ static void cxl_copy_cci_commands(CXLCCI *cci, const struct cxl_cmd (*cxl_cmds)[ } } +void cxl_add_cci_commands(CXLCCI *cci, const struct cxl_cmd (*cxl_cmd_set)[256], + size_t payload_max) +{ + cci->payload_max = payload_max > cci->payload_max ? payload_max : cci->payload_max; + cxl_copy_cci_commands(cci, cxl_cmd_set); + cxl_rebuild_cel(cci); +} + void cxl_initialize_mailbox_swcci(CXLCCI *cci, DeviceState *intf, DeviceState *d, size_t payload_max) { diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index ccc4611875..a5f8e25020 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -301,6 +301,8 @@ void cxl_initialize_mailbox_t3(CXLCCI *cci, DeviceState *d, size_t payload_max); void cxl_initialize_mailbox_swcci(CXLCCI *cci, DeviceState *intf, DeviceState *d, size_t payload_max); void cxl_init_cci(CXLCCI *cci, size_t payload_max); +void cxl_add_cci_commands(CXLCCI *cci, const struct cxl_cmd (*cxl_cmd_set)[256], + size_t payload_max); int cxl_process_cci_message(CXLCCI *cci, uint8_t set, uint8_t cmd, size_t len_in, uint8_t *pl_in, size_t *len_out, uint8_t *pl_out, From patchwork Thu May 23 17:44:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13672115 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B930AC25B75 for ; Thu, 23 May 2024 17:50:43 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sACZH-0006Qc-Rg; Thu, 23 May 2024 13:49:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sACYk-0005yP-KI for qemu-devel@nongnu.org; Thu, 23 May 2024 13:49:03 -0400 Received: from mail-pl1-x632.google.com ([2607:f8b0:4864:20::632]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sACYX-0003eT-Lk for qemu-devel@nongnu.org; Thu, 23 May 2024 13:48:49 -0400 Received: by mail-pl1-x632.google.com with SMTP id d9443c01a7336-1ee5235f5c9so122444335ad.2 for ; Thu, 23 May 2024 10:48:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716486522; x=1717091322; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=clywKlwl+RTvBV7mwlXItxpVpcuULIu2xU0NldTj1A8=; b=YOvwPqYaUAAKjGixJR9mhgkDRw4xrGsc28h97GsfExuCN4szIikkzkRLeHGpOA4irx GQqMtpQw5gZhXCX/tpCQs00BqXrlzFoWu/ViMiPrwo5VatPs7mFYRiHLLL8Q5umL8WLn YeEMqnVtCymD8Rhx9RGorMsNIiVp+ttoAMIvdMARmN4Zmm51l1Ysbb1dXVhEx3VzrKoS Xhptysq2Yd+fY0dKTJIwnLFZcDk4uaRDgQjkXVS4pTHL15xI/4ftQTAYlq/2byzFzDTh 2zAm5M8S7N5VmsqdBTSUY0VRAkkDPBajGQZCyoPYl2NXjJNRyl0Pz6bcUytOrkhH8ha3 oF7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716486522; x=1717091322; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=clywKlwl+RTvBV7mwlXItxpVpcuULIu2xU0NldTj1A8=; b=fHSV03XWiMYMI3JIlzREgTIyefzF5UNXBobr+NsKgOnur/qtOw1aGBLIivfyOqCeNF IlTEAQ2380JZkhYUdS7jcGB0D/h+Lu2nrEqHCJuHiqlluGgulR7j8O/kOvhqV9xUGlQO IB5juMYNGtZDg1iSiENu/hn6/7ack6Z4pH3PfIXla05pkEHKsyAG8JFsDO1FkvdudbNS dEtsyWbiwn25J/MgrRwfHNVQTUNBJQQBrBc5HS8urI5oyp7VlUJgY//RuI6qORVuLKzb 1/y0i9hAvvjcDCZuwEhw7HZU9FNTesDOVU8rEo9w9kqZ9B215djNXlXb7EyDx40G36rY yx0w== X-Gm-Message-State: AOJu0YxKeP9ZAJTxFfu6qer1Xax3EFKldL41whK6sCvFihuFO8QQZgBx ENHWetgg3cfFWWqbWjLQZdnJzkeiX3YwvklEF8eUwllkfhXftIocnWZUZw== X-Google-Smtp-Source: AGHT+IF1+uz93o85wiyhvEHCSjso2zhI95Lq8pWaJRgvaUUckzRdrRduv7BscqUJRGL8LOFeTTZY6w== X-Received: by 2002:a17:902:e80f:b0:1f3:3102:1609 with SMTP id d9443c01a7336-1f4497e588amr367845ad.50.1716486522306; Thu, 23 May 2024 10:48:42 -0700 (PDT) Received: from localhost.localdomain ([2601:646:8f03:9fee:4f9e:216e:ed5a:642d]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1ef0c137e7esm253436265ad.257.2024.05.23.10.48.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 May 2024 10:48:42 -0700 (PDT) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, armbru@redhat.com, mst@redhat.com, Fan Ni , Jonathan Cameron Subject: [PATCH v8 03/14] hw/cxl/cxl-mailbox-utils: Add dc_event_log_size field to output payload of identify memory device command Date: Thu, 23 May 2024 10:44:43 -0700 Message-ID: <20240523174651.1089554-4-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240523174651.1089554-1-nifan.cxl@gmail.com> References: <20240523174651.1089554-1-nifan.cxl@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::632; envelope-from=nifan.cxl@gmail.com; helo=mail-pl1-x632.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Fan Ni Based on CXL spec r3.1 Table 8-127 (Identify Memory Device Output Payload), dynamic capacity event log size should be part of output of the Identify command. Add dc_event_log_size to the output payload for the host to get the info. Reviewed-by: Gregory Price Reviewed-by: Jonathan Cameron Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 4bcd727f4c..ba1d9901df 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -21,6 +21,7 @@ #include "sysemu/hostmem.h" #define CXL_CAPACITY_MULTIPLIER (256 * MiB) +#define CXL_DC_EVENT_LOG_SIZE 8 /* * How to add a new command, example. The command set FOO, with cmd BAR. @@ -780,8 +781,9 @@ static CXLRetCode cmd_identify_memory_device(const struct cxl_cmd *cmd, uint16_t inject_poison_limit; uint8_t poison_caps; uint8_t qos_telemetry_caps; + uint16_t dc_event_log_size; } QEMU_PACKED *id; - QEMU_BUILD_BUG_ON(sizeof(*id) != 0x43); + QEMU_BUILD_BUG_ON(sizeof(*id) != 0x45); CXLType3Dev *ct3d = CXL_TYPE3(cci->d); CXLType3Class *cvc = CXL_TYPE3_GET_CLASS(ct3d); CXLDeviceState *cxl_dstate = &ct3d->cxl_dstate; @@ -807,6 +809,7 @@ static CXLRetCode cmd_identify_memory_device(const struct cxl_cmd *cmd, st24_le_p(id->poison_list_max_mer, 256); /* No limit - so limited by main poison record limit */ stw_le_p(&id->inject_poison_limit, 0); + stw_le_p(&id->dc_event_log_size, CXL_DC_EVENT_LOG_SIZE); *len_out = sizeof(*id); return CXL_MBOX_SUCCESS; From patchwork Thu May 23 17:44:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13672114 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BA411C25B75 for ; Thu, 23 May 2024 17:50:32 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sACZE-0006Np-G7; Thu, 23 May 2024 13:49:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sACYh-0005wy-TJ for qemu-devel@nongnu.org; Thu, 23 May 2024 13:49:02 -0400 Received: from mail-pl1-x629.google.com ([2607:f8b0:4864:20::629]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sACYX-0003ef-MT for qemu-devel@nongnu.org; Thu, 23 May 2024 13:48:48 -0400 Received: by mail-pl1-x629.google.com with SMTP id d9443c01a7336-1edc696df2bso23584405ad.0 for ; Thu, 23 May 2024 10:48:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716486524; x=1717091324; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Cp25PInGES4IUM+v+nVRXcyaxGTzommY3tewIcDpIDo=; b=IT+exrNdm4SUMOKuS+Igqrlm9vd+z9iglYIrkw+g+5uSMADk6ew8KPfNL93EFCf1qX EUNspBJafv1dN0uHSLJzHYBOxG8tHusm7nyiEPxiU6feQsYXWXvNZAv0fDwt8RLscQtw 6YMKw+FT2naAOQxIPtQ2R9BGfVko2lbAFREBy5KwpW2Y/mRJlVt1rNbkt6/vBYNFWhvH YeP47F9mdliOh4cJM/k0xW00DuFe8BAcBqGAOr0OwLOJrWDMZQ5DsHvWnXie+9ZubC44 jtkrNpu8AgLen0Jo8ADTGx7cpjl8lyemdfl+bMasx741z+s6Tcc1+CAgkY4X9e+FHw1l 25/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716486524; x=1717091324; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Cp25PInGES4IUM+v+nVRXcyaxGTzommY3tewIcDpIDo=; b=esazN94o69Ri4ypntRGv9FFH9jTwYWKyEp+iROG4+YowViIPkrwyZNfEqER9QuACxb b2dzNz4Ib8p3fFYwSQ8W8ytr024YfBpkwiuu8J0kJnES+jr1j57lsna4Z6wCpgT5o3zP sUG/CLjgj0MWR16lbECYg6iQKmuLTU0P5U6CArBgGQHtD0B6tzOTnnuJtEHpeFza3p2a fyiXjtykng66J5ByMGLPwYOvOmBR0lhhwKZSRJd7xtY1fJA5RjSzWZygnTLhCQUvkIQY /VD8qJ5lZ7fsLzeSp7U9505pKL9+KogrkfjJMH9lNvAqLgJKskJTEBlkkl7kDRPsB0wQ 5j0Q== X-Gm-Message-State: AOJu0YwpFt8B66XrMxLpOSP1w5uqIIyh1l8FgybOpAVsslHGQl4v1Cad xoDLFRlJbWAGygMyFs5WY+iUuUZ/ZXVp3kmTy2bfqtPRF1UUGPy5/FRqeA== X-Google-Smtp-Source: AGHT+IEwb7i78es7w1BShGgBFOsxchjdLVobvlyYQPez+6o4mmBdh1arixP9GXMY4IHPKtYxmiDIRg== X-Received: by 2002:a17:902:e848:b0:1f4:4631:d6aa with SMTP id d9443c01a7336-1f4486b5c83mr306735ad.14.1716486523994; Thu, 23 May 2024 10:48:43 -0700 (PDT) Received: from localhost.localdomain ([2601:646:8f03:9fee:4f9e:216e:ed5a:642d]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1ef0c137e7esm253436265ad.257.2024.05.23.10.48.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 May 2024 10:48:43 -0700 (PDT) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, armbru@redhat.com, mst@redhat.com, Fan Ni , Jonathan Cameron Subject: [PATCH v8 04/14] hw/cxl/cxl-mailbox-utils: Add dynamic capacity region representative and mailbox command support Date: Thu, 23 May 2024 10:44:44 -0700 Message-ID: <20240523174651.1089554-5-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240523174651.1089554-1-nifan.cxl@gmail.com> References: <20240523174651.1089554-1-nifan.cxl@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::629; envelope-from=nifan.cxl@gmail.com; helo=mail-pl1-x629.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Fan Ni Per cxl spec r3.1, add dynamic capacity (DC) region representative based on Table 8-165 and extend the cxl type3 device definition to include DC region information. Also, based on info in 8.2.9.9.9.1, add 'Get Dynamic Capacity Configuration' mailbox support. Note: we store region decode length as byte-wise length on the device, which should be divided by 256 * MiB before being returned to the host for "Get Dynamic Capacity Configuration" mailbox command per specification. Reviewed-by: Gregory Price Reviewed-by: Jonathan Cameron Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 96 +++++++++++++++++++++++++++++++++++++ include/hw/cxl/cxl_device.h | 16 +++++++ 2 files changed, 112 insertions(+) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index ba1d9901df..49c7944d93 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -22,6 +22,8 @@ #define CXL_CAPACITY_MULTIPLIER (256 * MiB) #define CXL_DC_EVENT_LOG_SIZE 8 +#define CXL_NUM_EXTENTS_SUPPORTED 512 +#define CXL_NUM_TAGS_SUPPORTED 0 /* * How to add a new command, example. The command set FOO, with cmd BAR. @@ -80,6 +82,8 @@ enum { #define GET_POISON_LIST 0x0 #define INJECT_POISON 0x1 #define CLEAR_POISON 0x2 + DCD_CONFIG = 0x48, + #define GET_DC_CONFIG 0x0 PHYSICAL_SWITCH = 0x51, #define IDENTIFY_SWITCH_DEVICE 0x0 #define GET_PHYSICAL_PORT_STATE 0x1 @@ -1238,6 +1242,88 @@ static CXLRetCode cmd_media_clear_poison(const struct cxl_cmd *cmd, return CXL_MBOX_SUCCESS; } +/* + * CXL r3.1 section 8.2.9.9.9.1: Get Dynamic Capacity Configuration + * (Opcode: 4800h) + */ +static CXLRetCode cmd_dcd_get_dyn_cap_config(const struct cxl_cmd *cmd, + uint8_t *payload_in, + size_t len_in, + uint8_t *payload_out, + size_t *len_out, + CXLCCI *cci) +{ + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + struct { + uint8_t region_cnt; + uint8_t start_rid; + } QEMU_PACKED *in = (void *)payload_in; + struct { + uint8_t num_regions; + uint8_t regions_returned; + uint8_t rsvd1[6]; + struct { + uint64_t base; + uint64_t decode_len; + uint64_t region_len; + uint64_t block_size; + uint32_t dsmadhandle; + uint8_t flags; + uint8_t rsvd2[3]; + } QEMU_PACKED records[]; + } QEMU_PACKED *out = (void *)payload_out; + struct { + uint32_t num_extents_supported; + uint32_t num_extents_available; + uint32_t num_tags_supported; + uint32_t num_tags_available; + } QEMU_PACKED *extra_out; + uint16_t record_count; + uint16_t i; + uint16_t out_pl_len; + uint8_t start_rid; + + start_rid = in->start_rid; + if (start_rid >= ct3d->dc.num_regions) { + return CXL_MBOX_INVALID_INPUT; + } + + record_count = MIN(ct3d->dc.num_regions - in->start_rid, in->region_cnt); + + out_pl_len = sizeof(*out) + record_count * sizeof(out->records[0]); + extra_out = (void *)(payload_out + out_pl_len); + out_pl_len += sizeof(*extra_out); + assert(out_pl_len <= CXL_MAILBOX_MAX_PAYLOAD_SIZE); + + out->num_regions = ct3d->dc.num_regions; + out->regions_returned = record_count; + for (i = 0; i < record_count; i++) { + stq_le_p(&out->records[i].base, + ct3d->dc.regions[start_rid + i].base); + stq_le_p(&out->records[i].decode_len, + ct3d->dc.regions[start_rid + i].decode_len / + CXL_CAPACITY_MULTIPLIER); + stq_le_p(&out->records[i].region_len, + ct3d->dc.regions[start_rid + i].len); + stq_le_p(&out->records[i].block_size, + ct3d->dc.regions[start_rid + i].block_size); + stl_le_p(&out->records[i].dsmadhandle, + ct3d->dc.regions[start_rid + i].dsmadhandle); + out->records[i].flags = ct3d->dc.regions[start_rid + i].flags; + } + /* + * TODO: Assign values once extents and tags are introduced + * to use. + */ + stl_le_p(&extra_out->num_extents_supported, CXL_NUM_EXTENTS_SUPPORTED); + stl_le_p(&extra_out->num_extents_available, CXL_NUM_EXTENTS_SUPPORTED); + stl_le_p(&extra_out->num_tags_supported, CXL_NUM_TAGS_SUPPORTED); + stl_le_p(&extra_out->num_tags_available, CXL_NUM_TAGS_SUPPORTED); + + *len_out = out_pl_len; + return CXL_MBOX_SUCCESS; +} + #define IMMEDIATE_CONFIG_CHANGE (1 << 1) #define IMMEDIATE_DATA_CHANGE (1 << 2) #define IMMEDIATE_POLICY_CHANGE (1 << 3) @@ -1282,6 +1368,11 @@ static const struct cxl_cmd cxl_cmd_set[256][256] = { cmd_media_clear_poison, 72, 0 }, }; +static const struct cxl_cmd cxl_cmd_set_dcd[256][256] = { + [DCD_CONFIG][GET_DC_CONFIG] = { "DCD_GET_DC_CONFIG", + cmd_dcd_get_dyn_cap_config, 2, 0 }, +}; + static const struct cxl_cmd cxl_cmd_set_sw[256][256] = { [INFOSTAT][IS_IDENTIFY] = { "IDENTIFY", cmd_infostat_identify, 0, 0 }, [INFOSTAT][BACKGROUND_OPERATION_STATUS] = { "BACKGROUND_OPERATION_STATUS", @@ -1487,7 +1578,12 @@ void cxl_initialize_mailbox_swcci(CXLCCI *cci, DeviceState *intf, void cxl_initialize_mailbox_t3(CXLCCI *cci, DeviceState *d, size_t payload_max) { + CXLType3Dev *ct3d = CXL_TYPE3(d); + cxl_copy_cci_commands(cci, cxl_cmd_set); + if (ct3d->dc.num_regions) { + cxl_copy_cci_commands(cci, cxl_cmd_set_dcd); + } cci->d = d; /* No separation for PCI MB as protocol handled in PCI device */ diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index a5f8e25020..e839370266 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -422,6 +422,17 @@ typedef struct CXLPoison { typedef QLIST_HEAD(, CXLPoison) CXLPoisonList; #define CXL_POISON_LIST_LIMIT 256 +#define DCD_MAX_NUM_REGION 8 + +typedef struct CXLDCRegion { + uint64_t base; /* aligned to 256*MiB */ + uint64_t decode_len; /* aligned to 256*MiB */ + uint64_t len; + uint64_t block_size; + uint32_t dsmadhandle; + uint8_t flags; +} CXLDCRegion; + struct CXLType3Dev { /* Private */ PCIDevice parent_obj; @@ -454,6 +465,11 @@ struct CXLType3Dev { unsigned int poison_list_cnt; bool poison_list_overflowed; uint64_t poison_list_overflow_ts; + + struct dynamic_capacity { + uint8_t num_regions; /* 0-8 regions */ + CXLDCRegion regions[DCD_MAX_NUM_REGION]; + } dc; }; #define TYPE_CXL_TYPE3 "cxl-type3" From patchwork Thu May 23 17:44:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13672119 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 88ACFC25B7A for ; Thu, 23 May 2024 17:51:23 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sACZ6-00068y-Ie; Thu, 23 May 2024 13:49:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sACYk-0005yV-Ly for qemu-devel@nongnu.org; Thu, 23 May 2024 13:49:03 -0400 Received: from mail-pl1-x62d.google.com ([2607:f8b0:4864:20::62d]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sACYa-0003ep-VW for qemu-devel@nongnu.org; Thu, 23 May 2024 13:48:57 -0400 Received: by mail-pl1-x62d.google.com with SMTP id d9443c01a7336-1ecc23e6c9dso121726485ad.2 for ; Thu, 23 May 2024 10:48:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716486526; x=1717091326; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7w7R6UVzX6+WHico1aXCCyhaIVNu2naBa1UA0NOMrYo=; b=exz7mjW7PiE5SsRgMGDhae07hwdnFntbtf05nYy+JTJkFnNmoJC/80T9Oohr5uf8ly aZThYjHhUmD+MJ75EhV3DWtl5J/l9tcna4x5xa/3BQvu3cYCF+ox2bPwdGYg4j5WKMwy IyqSiennxFvm+EXjH7qG2Liuo2yck1UnJ2OIdk4ZRGb9fU7Jh+iRji9ATk8qbo3k/Ht/ VQJHsx341Lk4of9ViKZmDo0kAGZRu7ETys4a9cvugis7yd2iMKUxPWVkjpmLFGlno6H/ I7f2uDpTXBl0h9EvMVs+xs3ho+yOJsOI7+D73YvUx8Qv0tyUIVjFxh8ri9m9b5/jcb7w VcHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716486526; x=1717091326; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7w7R6UVzX6+WHico1aXCCyhaIVNu2naBa1UA0NOMrYo=; b=mc3rvTO+DVgMTdrb64RIpOWmiGZ9BEl/3ovAQB+7xKjTizpdEniVumsvnR7cI7s5Cj tn1epq6r2PYv/UpRRTVfaCN2lJNWaLlDfDheQP6jHE3hz1ViKpWC5zsjVc5Xiy2W5qhi NGGUDwdD3xaqcdYM2I/ot0s/6WVfFRGn9B9YU+fNMXEe/SQJLhqZKMzFFWtrioG0n2Df h6aP7JaYNpL9R5K7k0M9FTBQZ4OpTkmuLYVofX5BrWxd7gSx/20NQGZf0hIJjnT+Y6vJ ZAo1XV5Ksrdf0/U5hLoM4n0hbK0fGPcDVGLOcKN69tpTmNNhZNDEw/J7OKcO49A4O4t/ H8kg== X-Gm-Message-State: AOJu0Yyq2o1mC6IBw1GDda2Ld4t+hUO9Ecx63cq8CBH5b3x4wMYdXCzn NYZ9c9wT8BPhcgU+DJkXBotzI7mmFruq4+wGN5ycpqZpqa7A7XeycDTOfw== X-Google-Smtp-Source: AGHT+IFHNTxQXFigHYpoI3dZOa8JygDT29EI1nALPQKjXb7XuMb9TVVlXj+AJkeRalDhpehak1Dwow== X-Received: by 2002:a17:903:2451:b0:1f3:50b5:65bd with SMTP id d9443c01a7336-1f44873c138mr460315ad.33.1716486525777; Thu, 23 May 2024 10:48:45 -0700 (PDT) Received: from localhost.localdomain ([2601:646:8f03:9fee:4f9e:216e:ed5a:642d]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1ef0c137e7esm253436265ad.257.2024.05.23.10.48.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 May 2024 10:48:45 -0700 (PDT) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, armbru@redhat.com, mst@redhat.com, Fan Ni , Jonathan Cameron Subject: [PATCH v8 05/14] include/hw/cxl/cxl_device: Rename mem_size as static_mem_size for type3 memory devices Date: Thu, 23 May 2024 10:44:45 -0700 Message-ID: <20240523174651.1089554-6-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240523174651.1089554-1-nifan.cxl@gmail.com> References: <20240523174651.1089554-1-nifan.cxl@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62d; envelope-from=nifan.cxl@gmail.com; helo=mail-pl1-x62d.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Fan Ni Rename mem_size as static_mem_size for type3 memdev to cover static RAM and pmem capacity, preparing for the introduction of dynamic capacity to support dynamic capacity devices. Reviewed-by: Gregory Price Reviewed-by: Jonathan Cameron Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 4 ++-- hw/mem/cxl_type3.c | 8 ++++---- include/hw/cxl/cxl_device.h | 2 +- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 49c7944d93..0f2ad58a14 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -803,7 +803,7 @@ static CXLRetCode cmd_identify_memory_device(const struct cxl_cmd *cmd, snprintf(id->fw_revision, 0x10, "BWFW VERSION %02d", 0); stq_le_p(&id->total_capacity, - cxl_dstate->mem_size / CXL_CAPACITY_MULTIPLIER); + cxl_dstate->static_mem_size / CXL_CAPACITY_MULTIPLIER); stq_le_p(&id->persistent_capacity, cxl_dstate->pmem_size / CXL_CAPACITY_MULTIPLIER); stq_le_p(&id->volatile_capacity, @@ -1179,7 +1179,7 @@ static CXLRetCode cmd_media_clear_poison(const struct cxl_cmd *cmd, struct clear_poison_pl *in = (void *)payload_in; dpa = ldq_le_p(&in->dpa); - if (dpa + CXL_CACHE_LINE_SIZE > cxl_dstate->mem_size) { + if (dpa + CXL_CACHE_LINE_SIZE > cxl_dstate->static_mem_size) { return CXL_MBOX_INVALID_PA; } diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 3e42490b6c..7194c8f902 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -608,7 +608,7 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) } address_space_init(&ct3d->hostvmem_as, vmr, v_name); ct3d->cxl_dstate.vmem_size = memory_region_size(vmr); - ct3d->cxl_dstate.mem_size += memory_region_size(vmr); + ct3d->cxl_dstate.static_mem_size += memory_region_size(vmr); g_free(v_name); } @@ -631,7 +631,7 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) } address_space_init(&ct3d->hostpmem_as, pmr, p_name); ct3d->cxl_dstate.pmem_size = memory_region_size(pmr); - ct3d->cxl_dstate.mem_size += memory_region_size(pmr); + ct3d->cxl_dstate.static_mem_size += memory_region_size(pmr); g_free(p_name); } @@ -837,7 +837,7 @@ static int cxl_type3_hpa_to_as_and_dpa(CXLType3Dev *ct3d, return -EINVAL; } - if (*dpa_offset > ct3d->cxl_dstate.mem_size) { + if (*dpa_offset > ct3d->cxl_dstate.static_mem_size) { return -EINVAL; } @@ -1010,7 +1010,7 @@ static bool set_cacheline(CXLType3Dev *ct3d, uint64_t dpa_offset, uint8_t *data) return false; } - if (dpa_offset + CXL_CACHE_LINE_SIZE > ct3d->cxl_dstate.mem_size) { + if (dpa_offset + CXL_CACHE_LINE_SIZE > ct3d->cxl_dstate.static_mem_size) { return false; } diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index e839370266..f7f56b44e3 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -234,7 +234,7 @@ typedef struct cxl_device_state { } timestamp; /* memory region size, HDM */ - uint64_t mem_size; + uint64_t static_mem_size; uint64_t pmem_size; uint64_t vmem_size; From patchwork Thu May 23 17:44:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13672110 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 53CEDC25B75 for ; Thu, 23 May 2024 17:49:52 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sACZF-0006O8-OR; Thu, 23 May 2024 13:49:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sACYo-00060v-Ho for qemu-devel@nongnu.org; Thu, 23 May 2024 13:49:08 -0400 Received: from mail-pl1-x636.google.com ([2607:f8b0:4864:20::636]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sACYk-0003fC-Cj for qemu-devel@nongnu.org; Thu, 23 May 2024 13:49:02 -0400 Received: by mail-pl1-x636.google.com with SMTP id d9443c01a7336-1f3469382f2so6218515ad.0 for ; Thu, 23 May 2024 10:48:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716486527; x=1717091327; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sP+37HoAbnRWxYkSxU+Q255kuctQLJ4X/JRVMf9ilwE=; b=E/jjpNgcXvTZP6CMVXT3WBmN9IZGrqA3/bXKIvbNmCz0HwUQyn/4GA9B2iGENZIkuP KFtg9Vz7ojR4fkco162X0O+mdxhoB/rO4AbnA4A497ie7mVGYaPEf/RanxmmE7SwlIHw OaVKhPlfW/paY5uL0vNvaY+0M4szzk51YmRBE/oKn6l86XIE9eEESeVp9dAAKaJwvK4X +MeOdyj9FtD147lCaeZaJ4QEmPOywsqW+s5xipL9RIsaXkRWKunDfo68pd0P3G2ndDOR x/2mJgDtKPUQ8OxR4hxiaLHWgk5gxsjPX/EZSw49TSpzBNIYv26qvCyv690IJq+SXAHy wxcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716486527; x=1717091327; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sP+37HoAbnRWxYkSxU+Q255kuctQLJ4X/JRVMf9ilwE=; b=vfJQRF7/CuDwflE+JTSuBlB9T9bIekUjAai2W5Cgx8T2gtS219GTUKEfxtCdIgYwSA 8HlJBDaWekvXZYNzgisQt69wGi3Q7cN7XyR9M/vgro9B52TUI7LLFm3yVOrMsRzOWyoy c9BMl/uZlL8LXXkhk7szj5vUVqPl4bHmOp069KPM8Sqpo8PBa7mvA3Jj2EAhk2d9jPw/ 0RpBTNmMS3NaoLz4TEOieteS9iXozFv1ZY93yN1+Jj2rFCftXuCp42T7uWIF+LS8j00+ BH4TOThxcntXR5ieXDckdDe2yj/ZSuCVWgFOGrx6OC70cWhs0F0ogZO+bA9SbDwVyhfp yWjA== X-Gm-Message-State: AOJu0Yxaq/b8VtKLhE6tMCCnL6K28m+BiA5kpthpsJ8VL1uPzyqYxgFR 8c9OJzvHBGv/ziBf6VsNH3ogeB91iVKrI0XOMcibDhNYCdryaIkkXobaxg== X-Google-Smtp-Source: AGHT+IEe0mgnI5pXbKqlTk1fSEV++VaFM84jJggzb34ACdBzCaJNjvCav9SzhD1YF00KP+e8DzZwqw== X-Received: by 2002:a17:902:f549:b0:1f0:98f2:7cc6 with SMTP id d9443c01a7336-1f4497df612mr317095ad.45.1716486527371; Thu, 23 May 2024 10:48:47 -0700 (PDT) Received: from localhost.localdomain ([2601:646:8f03:9fee:4f9e:216e:ed5a:642d]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1ef0c137e7esm253436265ad.257.2024.05.23.10.48.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 May 2024 10:48:47 -0700 (PDT) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, armbru@redhat.com, mst@redhat.com, Fan Ni , Jonathan Cameron Subject: [PATCH v8 06/14] hw/mem/cxl_type3: Add support to create DC regions to type3 memory devices Date: Thu, 23 May 2024 10:44:46 -0700 Message-ID: <20240523174651.1089554-7-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240523174651.1089554-1-nifan.cxl@gmail.com> References: <20240523174651.1089554-1-nifan.cxl@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::636; envelope-from=nifan.cxl@gmail.com; helo=mail-pl1-x636.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Fan Ni With the change, when setting up memory for type3 memory device, we can create DC regions. A property 'num-dc-regions' is added to ct3_props to allow users to pass the number of DC regions to create. To make it easier, other region parameters like region base, length, and block size are hard coded. If needed, these parameters can be added easily. With the change, we can create DC regions with proper kernel side support like below: region=$(cat /sys/bus/cxl/devices/decoder0.0/create_dc_region) echo $region > /sys/bus/cxl/devices/decoder0.0/create_dc_region echo 256 > /sys/bus/cxl/devices/$region/interleave_granularity echo 1 > /sys/bus/cxl/devices/$region/interleave_ways echo "dc0" >/sys/bus/cxl/devices/decoder2.0/mode echo 0x40000000 >/sys/bus/cxl/devices/decoder2.0/dpa_size echo 0x40000000 > /sys/bus/cxl/devices/$region/size echo "decoder2.0" > /sys/bus/cxl/devices/$region/target0 echo 1 > /sys/bus/cxl/devices/$region/commit echo $region > /sys/bus/cxl/drivers/cxl_region/bind Reviewed-by: Gregory Price Reviewed-by: Jonathan Cameron Signed-off-by: Fan Ni Reviewed-by: Li Zhijian --- hw/mem/cxl_type3.c | 53 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 7194c8f902..06c6f9bb78 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -30,6 +30,7 @@ #include "hw/pci/msix.h" #define DWORD_BYTE 4 +#define CXL_CAPACITY_MULTIPLIER (256 * MiB) /* Default CDAT entries for a memory region */ enum { @@ -567,6 +568,50 @@ static void ct3d_reg_write(void *opaque, hwaddr offset, uint64_t value, } } +/* + * TODO: dc region configuration will be updated once host backend and address + * space support is added for DCD. + */ +static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) +{ + int i; + uint64_t region_base = 0; + uint64_t region_len = 2 * GiB; + uint64_t decode_len = 2 * GiB; + uint64_t blk_size = 2 * MiB; + CXLDCRegion *region; + MemoryRegion *mr; + + if (ct3d->hostvmem) { + mr = host_memory_backend_get_memory(ct3d->hostvmem); + region_base += memory_region_size(mr); + } + if (ct3d->hostpmem) { + mr = host_memory_backend_get_memory(ct3d->hostpmem); + region_base += memory_region_size(mr); + } + if (region_base % CXL_CAPACITY_MULTIPLIER != 0) { + error_setg(errp, "DC region base not aligned to 0x%lx", + CXL_CAPACITY_MULTIPLIER); + return false; + } + + for (i = 0, region = &ct3d->dc.regions[0]; + i < ct3d->dc.num_regions; + i++, region++, region_base += region_len) { + *region = (CXLDCRegion) { + .base = region_base, + .decode_len = decode_len, + .len = region_len, + .block_size = blk_size, + /* dsmad_handle set when creating CDAT table entries */ + .flags = 0, + }; + } + + return true; +} + static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) { DeviceState *ds = DEVICE(ct3d); @@ -635,6 +680,13 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) g_free(p_name); } + if (ct3d->dc.num_regions > 0) { + if (!cxl_create_dc_regions(ct3d, errp)) { + error_append_hint(errp, "setup DC regions failed"); + return false; + } + } + return true; } @@ -930,6 +982,7 @@ static Property ct3_props[] = { HostMemoryBackend *), DEFINE_PROP_UINT64("sn", CXLType3Dev, sn, UI64_NULL), DEFINE_PROP_STRING("cdat", CXLType3Dev, cxl_cstate.cdat.filename), + DEFINE_PROP_UINT8("num-dc-regions", CXLType3Dev, dc.num_regions, 0), DEFINE_PROP_END_OF_LIST(), }; From patchwork Thu May 23 17:44:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13672113 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B89D8C25B7C for ; Thu, 23 May 2024 17:50:20 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sACZ9-0006FF-Cr; Thu, 23 May 2024 13:49:23 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sACYx-000654-2K for qemu-devel@nongnu.org; Thu, 23 May 2024 13:49:11 -0400 Received: from mail-pl1-x629.google.com ([2607:f8b0:4864:20::629]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sACYo-0003fN-1Z for qemu-devel@nongnu.org; Thu, 23 May 2024 13:49:05 -0400 Received: by mail-pl1-x629.google.com with SMTP id d9443c01a7336-1f32a3b9491so15562035ad.0 for ; Thu, 23 May 2024 10:48:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716486529; x=1717091329; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ocPCyo8rhL+lFVIKpLa8nzkC2tFA2ADDZSvrzw2Bp6k=; b=SmrHrWrwrrU92VkZUHeHZ89TkBdRinl0htsp0s3+0Vz4WEqYPgc2Dy7fW1Tg3dzZSO +vb2qaPSvH4s4xXWs5P7UqvWGsT/kf9Ji6YPgPOD/GLXs4m9mJxNO3QeWe1ZVJK4AlEW HL5n2z8OegBvhnMDh5EtPXnbKqXVB26qhTTc+dSnxpRCj/ihgsf/2B3FJuBnOgWk+IvE 1e9ISsxoClQEHrhCcZSxyY9MoqX+PWs0mKajmXDeICq4FfTVkiRxrE4MJ1jI1Bc93gSB PWw7NFsfwW3ZLguku9uc6JuWxoQqGazS/LfGThxC2RhP2H8K19HjvSAW4h7KHTpi/nlr SwSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716486529; x=1717091329; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ocPCyo8rhL+lFVIKpLa8nzkC2tFA2ADDZSvrzw2Bp6k=; b=UB5uJScfW3Vp46VIwqQWanZ0Q6Qa9ERBt/dO2uob9CuXG5sFyEeM/AnJt75l/5BzPE uwfEDWGRkWQF9w4cz2KAXJ91cAys6rk7iAyJRom/TkrRqCObzG8rr9GAIoJDI4bkWjAH 2UF3RxqVBdyjEuadHemixrdIF02LlXWi5qK3GHuPjEvxKmgM/FA7E972O+F57hVeO2NY N53yIsdu+vAGQT582ZAwegk8cYUqzftJ4u1SKaQw30bTSC0Ha2+ZOYIeRqIWw63OZJ/0 bAk2moLecp/9M+l/FeGeKkpIzAM7MUqB/Jex/uDisYN6H4D/Q4hIbZstR4DCXWoBTsoF tYHw== X-Gm-Message-State: AOJu0YyjFcctxfSEH6M+l2GpvVVXb4y6CIeClYINFF6nEJNXutiNpo73 UcxU9+yatANYGty98dpfXiUYSs3UYLNTf2PQwFC5b1eMl9hO12J+9z5QGg== X-Google-Smtp-Source: AGHT+IFswUc2pHV2hG9gPI1WJGwqcncOm/fJ4wA/eCxhF++49CLYGicqoq3D94ExiC3uMcz2ax5Ugw== X-Received: by 2002:a17:902:f54c:b0:1e4:8eb3:26f5 with SMTP id d9443c01a7336-1f44883b0a7mr569065ad.38.1716486529000; Thu, 23 May 2024 10:48:49 -0700 (PDT) Received: from localhost.localdomain ([2601:646:8f03:9fee:4f9e:216e:ed5a:642d]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1ef0c137e7esm253436265ad.257.2024.05.23.10.48.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 May 2024 10:48:48 -0700 (PDT) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, armbru@redhat.com, mst@redhat.com, Fan Ni , Jonathan Cameron Subject: [PATCH v8 07/14] hw/mem/cxl-type3: Refactor ct3_build_cdat_entries_for_mr to take mr size instead of mr as argument Date: Thu, 23 May 2024 10:44:47 -0700 Message-ID: <20240523174651.1089554-8-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240523174651.1089554-1-nifan.cxl@gmail.com> References: <20240523174651.1089554-1-nifan.cxl@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::629; envelope-from=nifan.cxl@gmail.com; helo=mail-pl1-x629.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Fan Ni The function ct3_build_cdat_entries_for_mr only uses size of the passed memory region argument, refactor the function definition to make the passed arguments more specific. Reviewed-by: Gregory Price Reviewed-by: Jonathan Cameron Signed-off-by: Fan Ni --- hw/mem/cxl_type3.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 06c6f9bb78..51be50ce87 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -44,7 +44,7 @@ enum { }; static void ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, - int dsmad_handle, MemoryRegion *mr, + int dsmad_handle, uint64_t size, bool is_pmem, uint64_t dpa_base) { CDATDsmas *dsmas; @@ -63,7 +63,7 @@ static void ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, .DSMADhandle = dsmad_handle, .flags = is_pmem ? CDAT_DSMAS_FLAG_NV : 0, .DPA_base = dpa_base, - .DPA_length = memory_region_size(mr), + .DPA_length = size, }; /* For now, no memory side cache, plausiblish numbers */ @@ -132,7 +132,7 @@ static void ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, */ .EFI_memory_type_attr = is_pmem ? 2 : 1, .DPA_offset = 0, - .DPA_length = memory_region_size(mr), + .DPA_length = size, }; /* Header always at start of structure */ @@ -149,6 +149,7 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) g_autofree CDATSubHeader **table = NULL; CXLType3Dev *ct3d = priv; MemoryRegion *volatile_mr = NULL, *nonvolatile_mr = NULL; + uint64_t vmr_size = 0, pmr_size = 0; int dsmad_handle = 0; int cur_ent = 0; int len = 0; @@ -163,6 +164,7 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) return -EINVAL; } len += CT3_CDAT_NUM_ENTRIES; + vmr_size = memory_region_size(volatile_mr); } if (ct3d->hostpmem) { @@ -171,21 +173,22 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) return -EINVAL; } len += CT3_CDAT_NUM_ENTRIES; + pmr_size = memory_region_size(nonvolatile_mr); } table = g_malloc0(len * sizeof(*table)); /* Now fill them in */ if (volatile_mr) { - ct3_build_cdat_entries_for_mr(table, dsmad_handle++, volatile_mr, + ct3_build_cdat_entries_for_mr(table, dsmad_handle++, vmr_size, false, 0); cur_ent = CT3_CDAT_NUM_ENTRIES; } if (nonvolatile_mr) { - uint64_t base = volatile_mr ? memory_region_size(volatile_mr) : 0; + uint64_t base = vmr_size; ct3_build_cdat_entries_for_mr(&(table[cur_ent]), dsmad_handle++, - nonvolatile_mr, true, base); + pmr_size, true, base); cur_ent += CT3_CDAT_NUM_ENTRIES; } assert(len == cur_ent); From patchwork Thu May 23 17:44:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13672138 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C7DF3C25B75 for ; Thu, 23 May 2024 17:51:52 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sACZL-0006YV-Ie; Thu, 23 May 2024 13:49:35 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sACYo-00060w-Ht for qemu-devel@nongnu.org; Thu, 23 May 2024 13:49:08 -0400 Received: from mail-pl1-x633.google.com ([2607:f8b0:4864:20::633]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sACYk-0003fR-Cf for qemu-devel@nongnu.org; Thu, 23 May 2024 13:49:01 -0400 Received: by mail-pl1-x633.google.com with SMTP id d9443c01a7336-1edfc57ac0cso21417915ad.3 for ; Thu, 23 May 2024 10:48:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716486531; x=1717091331; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CvFPMLoK01mkfdgG+TlQSzE5mXYNDErR/nxyp18SFeo=; b=fOxJuq9dTpDvPEhIGgk+0z5qibRYUDqsOtxbuTt59mEBucuiEXJYtgnYcQrb9e5r7Z 7hvbKAmnxT4L76+X9wzcp34Csru/ORo5I8Fd/07u7BafXtTuck+JYdCx9CDsABheWXDz BPvjZyf4RELEQJTucuF0EQaYmxa5hfNLah3nhjc/bwp6HBi0+zJYKzBEqBfcdDn6o1JL zIahcSyYyAvSNnd5R1COA65SkHEE25XpJfmJov8ExbKVXHWbLPkNvc8lmlm7eB57a5tC /SmrkBwQeSwaj5Y12z500/TidGwfPSDG4uJBKpR0dIueBiCA7Wwi9OtaB5YhPKJGtLde 4ZPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716486531; x=1717091331; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CvFPMLoK01mkfdgG+TlQSzE5mXYNDErR/nxyp18SFeo=; b=loX6RzhRMIGQ/nJkrNrHYCovAHRWD/d40wAiTgJu1ajtjIqONUEcPbhSD8974jTFl7 qrcSjVmD4o3JK15+ED6KrOQllOzH2eiDfVtfgKQU2k6UAy79BdN6xyXa20p6BA8y2g9c BIpHGEHKxq14l85SyYkjbdkm9kSnqPUxfLwNKoR4Age4CAcryyNtce0+Vkdzxa7C6U/9 VoBaVSL1l43BUoHC06xegV4MrYGOHlyB3R/BXf+zL+WSG3Nksccmk7MhxrHcl7SK+MP8 XCqAUfMw5I8gPlaENQug4tJlzQ7GB6mpwJCBwlHlDSLfS0KEdJXHemUboatOzrCTaGQN UInQ== X-Gm-Message-State: AOJu0YzIF4hLSfFGkaEMlK3iGpHyIJsgAy+G1s5cuo/sFWchuo5xri7c 9SdncKHUPkiG/Og0PasXyC6yX3UJlgu6mJPn3rlYK+ld4mnR+50rijLg5w== X-Google-Smtp-Source: AGHT+IETzR8iasTTrODYffeR9p7FNeBvLAHH8lDyUFNCg4oA5Eiw8g9lIEKSh/CZIl1ocl9yLuGuhw== X-Received: by 2002:a17:902:d48c:b0:1e4:b1a2:b40c with SMTP id d9443c01a7336-1f4494f2e86mr464425ad.42.1716486530621; Thu, 23 May 2024 10:48:50 -0700 (PDT) Received: from localhost.localdomain ([2601:646:8f03:9fee:4f9e:216e:ed5a:642d]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1ef0c137e7esm253436265ad.257.2024.05.23.10.48.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 May 2024 10:48:50 -0700 (PDT) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, armbru@redhat.com, mst@redhat.com, Fan Ni Subject: [PATCH v8 08/14] hw/mem/cxl_type3: Add host backend and address space handling for DC regions Date: Thu, 23 May 2024 10:44:48 -0700 Message-ID: <20240523174651.1089554-9-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240523174651.1089554-1-nifan.cxl@gmail.com> References: <20240523174651.1089554-1-nifan.cxl@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::633; envelope-from=nifan.cxl@gmail.com; helo=mail-pl1-x633.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Fan Ni Add (file/memory backed) host backend for DCD. All the dynamic capacity regions will share a single, large enough host backend. Set up address space for DC regions to support read/write operations to dynamic capacity for DCD. With the change, the following support is added: 1. Add a new property to type3 device "volatile-dc-memdev" to point to host memory backend for dynamic capacity. Currently, all DC regions share one host backend; 2. Add namespace for dynamic capacity for read/write support; 3. Create cdat entries for each dynamic capacity region. Reviewed-by: Gregory Price Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 16 ++-- hw/mem/cxl_type3.c | 174 +++++++++++++++++++++++++++++------- include/hw/cxl/cxl_device.h | 8 ++ 3 files changed, 162 insertions(+), 36 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 0f2ad58a14..831cef0567 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -622,7 +622,8 @@ static CXLRetCode cmd_firmware_update_get_info(const struct cxl_cmd *cmd, size_t *len_out, CXLCCI *cci) { - CXLDeviceState *cxl_dstate = &CXL_TYPE3(cci->d)->cxl_dstate; + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + CXLDeviceState *cxl_dstate = &ct3d->cxl_dstate; struct { uint8_t slots_supported; uint8_t slot_info; @@ -636,7 +637,8 @@ static CXLRetCode cmd_firmware_update_get_info(const struct cxl_cmd *cmd, QEMU_BUILD_BUG_ON(sizeof(*fw_info) != 0x50); if ((cxl_dstate->vmem_size < CXL_CAPACITY_MULTIPLIER) || - (cxl_dstate->pmem_size < CXL_CAPACITY_MULTIPLIER)) { + (cxl_dstate->pmem_size < CXL_CAPACITY_MULTIPLIER) || + (ct3d->dc.total_capacity < CXL_CAPACITY_MULTIPLIER)) { return CXL_MBOX_INTERNAL_ERROR; } @@ -793,7 +795,8 @@ static CXLRetCode cmd_identify_memory_device(const struct cxl_cmd *cmd, CXLDeviceState *cxl_dstate = &ct3d->cxl_dstate; if ((!QEMU_IS_ALIGNED(cxl_dstate->vmem_size, CXL_CAPACITY_MULTIPLIER)) || - (!QEMU_IS_ALIGNED(cxl_dstate->pmem_size, CXL_CAPACITY_MULTIPLIER))) { + (!QEMU_IS_ALIGNED(cxl_dstate->pmem_size, CXL_CAPACITY_MULTIPLIER)) || + (!QEMU_IS_ALIGNED(ct3d->dc.total_capacity, CXL_CAPACITY_MULTIPLIER))) { return CXL_MBOX_INTERNAL_ERROR; } @@ -835,9 +838,11 @@ static CXLRetCode cmd_ccls_get_partition_info(const struct cxl_cmd *cmd, uint64_t next_pmem; } QEMU_PACKED *part_info = (void *)payload_out; QEMU_BUILD_BUG_ON(sizeof(*part_info) != 0x20); + CXLType3Dev *ct3d = container_of(cxl_dstate, CXLType3Dev, cxl_dstate); if ((!QEMU_IS_ALIGNED(cxl_dstate->vmem_size, CXL_CAPACITY_MULTIPLIER)) || - (!QEMU_IS_ALIGNED(cxl_dstate->pmem_size, CXL_CAPACITY_MULTIPLIER))) { + (!QEMU_IS_ALIGNED(cxl_dstate->pmem_size, CXL_CAPACITY_MULTIPLIER)) || + (!QEMU_IS_ALIGNED(ct3d->dc.total_capacity, CXL_CAPACITY_MULTIPLIER))) { return CXL_MBOX_INTERNAL_ERROR; } @@ -1179,7 +1184,8 @@ static CXLRetCode cmd_media_clear_poison(const struct cxl_cmd *cmd, struct clear_poison_pl *in = (void *)payload_in; dpa = ldq_le_p(&in->dpa); - if (dpa + CXL_CACHE_LINE_SIZE > cxl_dstate->static_mem_size) { + if (dpa + CXL_CACHE_LINE_SIZE > cxl_dstate->static_mem_size + + ct3d->dc.total_capacity) { return CXL_MBOX_INVALID_PA; } diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 51be50ce87..f645a3f2e9 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -45,7 +45,8 @@ enum { static void ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, int dsmad_handle, uint64_t size, - bool is_pmem, uint64_t dpa_base) + bool is_pmem, bool is_dynamic, + uint64_t dpa_base) { CDATDsmas *dsmas; CDATDslbis *dslbis0; @@ -61,7 +62,8 @@ static void ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, .length = sizeof(*dsmas), }, .DSMADhandle = dsmad_handle, - .flags = is_pmem ? CDAT_DSMAS_FLAG_NV : 0, + .flags = (is_pmem ? CDAT_DSMAS_FLAG_NV : 0) | + (is_dynamic ? CDAT_DSMAS_FLAG_DYNAMIC_CAP : 0), .DPA_base = dpa_base, .DPA_length = size, }; @@ -149,12 +151,13 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) g_autofree CDATSubHeader **table = NULL; CXLType3Dev *ct3d = priv; MemoryRegion *volatile_mr = NULL, *nonvolatile_mr = NULL; + MemoryRegion *dc_mr = NULL; uint64_t vmr_size = 0, pmr_size = 0; int dsmad_handle = 0; int cur_ent = 0; int len = 0; - if (!ct3d->hostpmem && !ct3d->hostvmem) { + if (!ct3d->hostpmem && !ct3d->hostvmem && !ct3d->dc.num_regions) { return 0; } @@ -176,21 +179,54 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) pmr_size = memory_region_size(nonvolatile_mr); } + if (ct3d->dc.num_regions) { + if (!ct3d->dc.host_dc) { + return -EINVAL; + } + dc_mr = host_memory_backend_get_memory(ct3d->dc.host_dc); + if (!dc_mr) { + return -EINVAL; + } + len += CT3_CDAT_NUM_ENTRIES * ct3d->dc.num_regions; + } + table = g_malloc0(len * sizeof(*table)); /* Now fill them in */ if (volatile_mr) { ct3_build_cdat_entries_for_mr(table, dsmad_handle++, vmr_size, - false, 0); + false, false, 0); cur_ent = CT3_CDAT_NUM_ENTRIES; } if (nonvolatile_mr) { uint64_t base = vmr_size; ct3_build_cdat_entries_for_mr(&(table[cur_ent]), dsmad_handle++, - pmr_size, true, base); + pmr_size, true, false, base); cur_ent += CT3_CDAT_NUM_ENTRIES; } + + if (dc_mr) { + int i; + uint64_t region_base = vmr_size + pmr_size; + + /* + * We assume the dynamic capacity to be volatile for now. + * Non-volatile dynamic capacity will be added if needed in the + * future. + */ + for (i = 0; i < ct3d->dc.num_regions; i++) { + ct3_build_cdat_entries_for_mr(&(table[cur_ent]), + dsmad_handle++, + ct3d->dc.regions[i].len, + false, true, region_base); + ct3d->dc.regions[i].dsmadhandle = dsmad_handle - 1; + + cur_ent += CT3_CDAT_NUM_ENTRIES; + region_base += ct3d->dc.regions[i].len; + } + } + assert(len == cur_ent); *cdat_table = g_steal_pointer(&table); @@ -301,10 +337,17 @@ static void build_dvsecs(CXLType3Dev *ct3d) range2_size_lo = (2 << 5) | (2 << 2) | 0x3 | (ct3d->hostpmem->size & 0xF0000000); } - } else { + } else if (ct3d->hostpmem) { range1_size_hi = ct3d->hostpmem->size >> 32; range1_size_lo = (2 << 5) | (2 << 2) | 0x3 | (ct3d->hostpmem->size & 0xF0000000); + } else { + /* + * For DCD with no static memory, set memory active, memory class bits. + * No range is set. + */ + range1_size_hi = 0; + range1_size_lo = (2 << 5) | (2 << 2) | 0x3; } dvsec = (uint8_t *)&(CXLDVSECDevice){ @@ -579,11 +622,28 @@ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) { int i; uint64_t region_base = 0; - uint64_t region_len = 2 * GiB; - uint64_t decode_len = 2 * GiB; + uint64_t region_len; + uint64_t decode_len; uint64_t blk_size = 2 * MiB; CXLDCRegion *region; MemoryRegion *mr; + uint64_t dc_size; + + mr = host_memory_backend_get_memory(ct3d->dc.host_dc); + dc_size = memory_region_size(mr); + region_len = DIV_ROUND_UP(dc_size, ct3d->dc.num_regions); + + if (dc_size % (ct3d->dc.num_regions * CXL_CAPACITY_MULTIPLIER) != 0) { + error_setg(errp, "backend size is not multiple of region len: 0x%lx", + region_len); + return false; + } + if (region_len % CXL_CAPACITY_MULTIPLIER != 0) { + error_setg(errp, "DC region size is unaligned to 0x%lx", + CXL_CAPACITY_MULTIPLIER); + return false; + } + decode_len = region_len; if (ct3d->hostvmem) { mr = host_memory_backend_get_memory(ct3d->hostvmem); @@ -610,6 +670,7 @@ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) /* dsmad_handle set when creating CDAT table entries */ .flags = 0, }; + ct3d->dc.total_capacity += region->len; } return true; @@ -619,7 +680,8 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) { DeviceState *ds = DEVICE(ct3d); - if (!ct3d->hostmem && !ct3d->hostvmem && !ct3d->hostpmem) { + if (!ct3d->hostmem && !ct3d->hostvmem && !ct3d->hostpmem + && !ct3d->dc.num_regions) { error_setg(errp, "at least one memdev property must be set"); return false; } else if (ct3d->hostmem && ct3d->hostpmem) { @@ -683,7 +745,37 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) g_free(p_name); } + ct3d->dc.total_capacity = 0; if (ct3d->dc.num_regions > 0) { + MemoryRegion *dc_mr; + char *dc_name; + + if (!ct3d->dc.host_dc) { + error_setg(errp, "dynamic capacity must have a backing device"); + return false; + } + + dc_mr = host_memory_backend_get_memory(ct3d->dc.host_dc); + if (!dc_mr) { + error_setg(errp, "dynamic capacity must have a backing device"); + return false; + } + + /* + * Set DC regions as volatile for now, non-volatile support can + * be added in the future if needed. + */ + memory_region_set_nonvolatile(dc_mr, false); + memory_region_set_enabled(dc_mr, true); + host_memory_backend_set_mapped(ct3d->dc.host_dc, true); + if (ds->id) { + dc_name = g_strdup_printf("cxl-dcd-dpa-dc-space:%s", ds->id); + } else { + dc_name = g_strdup("cxl-dcd-dpa-dc-space"); + } + address_space_init(&ct3d->dc.host_dc_as, dc_mr, dc_name); + g_free(dc_name); + if (!cxl_create_dc_regions(ct3d, errp)) { error_append_hint(errp, "setup DC regions failed"); return false; @@ -779,6 +871,9 @@ err_release_cdat: err_free_special_ops: g_free(regs->special_ops); err_address_space_free: + if (ct3d->dc.host_dc) { + address_space_destroy(&ct3d->dc.host_dc_as); + } if (ct3d->hostpmem) { address_space_destroy(&ct3d->hostpmem_as); } @@ -797,6 +892,9 @@ static void ct3_exit(PCIDevice *pci_dev) pcie_aer_exit(pci_dev); cxl_doe_cdat_release(cxl_cstate); g_free(regs->special_ops); + if (ct3d->dc.host_dc) { + address_space_destroy(&ct3d->dc.host_dc_as); + } if (ct3d->hostpmem) { address_space_destroy(&ct3d->hostpmem_as); } @@ -875,16 +973,23 @@ static int cxl_type3_hpa_to_as_and_dpa(CXLType3Dev *ct3d, AddressSpace **as, uint64_t *dpa_offset) { - MemoryRegion *vmr = NULL, *pmr = NULL; + MemoryRegion *vmr = NULL, *pmr = NULL, *dc_mr = NULL; + uint64_t vmr_size = 0, pmr_size = 0, dc_size = 0; if (ct3d->hostvmem) { vmr = host_memory_backend_get_memory(ct3d->hostvmem); + vmr_size = memory_region_size(vmr); } if (ct3d->hostpmem) { pmr = host_memory_backend_get_memory(ct3d->hostpmem); + pmr_size = memory_region_size(pmr); + } + if (ct3d->dc.host_dc) { + dc_mr = host_memory_backend_get_memory(ct3d->dc.host_dc); + dc_size = memory_region_size(dc_mr); } - if (!vmr && !pmr) { + if (!vmr && !pmr && !dc_mr) { return -ENODEV; } @@ -892,19 +997,18 @@ static int cxl_type3_hpa_to_as_and_dpa(CXLType3Dev *ct3d, return -EINVAL; } - if (*dpa_offset > ct3d->cxl_dstate.static_mem_size) { + if (*dpa_offset >= vmr_size + pmr_size + dc_size) { return -EINVAL; } - if (vmr) { - if (*dpa_offset < memory_region_size(vmr)) { - *as = &ct3d->hostvmem_as; - } else { - *as = &ct3d->hostpmem_as; - *dpa_offset -= memory_region_size(vmr); - } - } else { + if (*dpa_offset < vmr_size) { + *as = &ct3d->hostvmem_as; + } else if (*dpa_offset < vmr_size + pmr_size) { *as = &ct3d->hostpmem_as; + *dpa_offset -= vmr_size; + } else { + *as = &ct3d->dc.host_dc_as; + *dpa_offset -= (vmr_size + pmr_size); } return 0; @@ -986,6 +1090,8 @@ static Property ct3_props[] = { DEFINE_PROP_UINT64("sn", CXLType3Dev, sn, UI64_NULL), DEFINE_PROP_STRING("cdat", CXLType3Dev, cxl_cstate.cdat.filename), DEFINE_PROP_UINT8("num-dc-regions", CXLType3Dev, dc.num_regions, 0), + DEFINE_PROP_LINK("volatile-dc-memdev", CXLType3Dev, dc.host_dc, + TYPE_MEMORY_BACKEND, HostMemoryBackend *), DEFINE_PROP_END_OF_LIST(), }; @@ -1052,33 +1158,39 @@ static void set_lsa(CXLType3Dev *ct3d, const void *buf, uint64_t size, static bool set_cacheline(CXLType3Dev *ct3d, uint64_t dpa_offset, uint8_t *data) { - MemoryRegion *vmr = NULL, *pmr = NULL; + MemoryRegion *vmr = NULL, *pmr = NULL, *dc_mr = NULL; AddressSpace *as; + uint64_t vmr_size = 0, pmr_size = 0, dc_size = 0; if (ct3d->hostvmem) { vmr = host_memory_backend_get_memory(ct3d->hostvmem); + vmr_size = memory_region_size(vmr); } if (ct3d->hostpmem) { pmr = host_memory_backend_get_memory(ct3d->hostpmem); + pmr_size = memory_region_size(pmr); } + if (ct3d->dc.host_dc) { + dc_mr = host_memory_backend_get_memory(ct3d->dc.host_dc); + dc_size = memory_region_size(dc_mr); + } - if (!vmr && !pmr) { + if (!vmr && !pmr && !dc_mr) { return false; } - if (dpa_offset + CXL_CACHE_LINE_SIZE > ct3d->cxl_dstate.static_mem_size) { + if (dpa_offset + CXL_CACHE_LINE_SIZE > vmr_size + pmr_size + dc_size) { return false; } - if (vmr) { - if (dpa_offset < memory_region_size(vmr)) { - as = &ct3d->hostvmem_as; - } else { - as = &ct3d->hostpmem_as; - dpa_offset -= memory_region_size(vmr); - } - } else { + if (dpa_offset < vmr_size) { + as = &ct3d->hostvmem_as; + } else if (dpa_offset < vmr_size + pmr_size) { as = &ct3d->hostpmem_as; + dpa_offset -= vmr_size; + } else { + as = &ct3d->dc.host_dc_as; + dpa_offset -= (vmr_size + pmr_size); } address_space_write(as, dpa_offset, MEMTXATTRS_UNSPECIFIED, &data, diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index f7f56b44e3..c2c3df0d2a 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -467,6 +467,14 @@ struct CXLType3Dev { uint64_t poison_list_overflow_ts; struct dynamic_capacity { + HostMemoryBackend *host_dc; + AddressSpace host_dc_as; + /* + * total_capacity is equivalent to the dynamic capability + * memory region size. + */ + uint64_t total_capacity; /* 256M aligned */ + uint8_t num_regions; /* 0-8 regions */ CXLDCRegion regions[DCD_MAX_NUM_REGION]; } dc; From patchwork Thu May 23 17:44:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13672116 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B9E95C25B7A for ; Thu, 23 May 2024 17:50:51 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sACZP-0006gA-7Q; Thu, 23 May 2024 13:49:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sACYo-00060t-HT for qemu-devel@nongnu.org; Thu, 23 May 2024 13:49:08 -0400 Received: from mail-pl1-x62e.google.com ([2607:f8b0:4864:20::62e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sACYk-0003fY-D0 for qemu-devel@nongnu.org; Thu, 23 May 2024 13:49:01 -0400 Received: by mail-pl1-x62e.google.com with SMTP id d9443c01a7336-1ee5235f5c9so122445865ad.2 for ; Thu, 23 May 2024 10:48:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716486532; x=1717091332; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=V3Dkdm32lp5f6RMYNOcQ/PPZ87Y6R6yDh0BiQEry7hs=; b=KaiPSwOc32A56JNqi2ukGOXalOkVuYBw7HF1L7KnZikOCrorhbk4cHKXwgWs5P7C4T DMTcIqmx3bTFATtbN7ezyYGpIxx5+sG94inJBqKcz6Da2fHIyKq0Ctf7ilZGJJc2sQcF 8Q49Ic5WpJSD1iQTETUAMW1RJNGP8Vyn1tLEdE5qYHNX8zqe1muAubWDNt1CiOcT8YZa sXxJv65bZKLcF0dDUqTac5r/ovanAL8KxhYq+6d6zWd1axvkf1iELQOQYwHZFyJWVU6B eXH8EkwNjGb2Fo55qKZaq+fUaqbWdCnU6/al894ZsG+SPXe6LADp5eNsJAuXgp+AdVGF eD1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716486532; x=1717091332; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=V3Dkdm32lp5f6RMYNOcQ/PPZ87Y6R6yDh0BiQEry7hs=; b=ApFTnHmKJRD7JpoIBMizFcgmqOqeuk1h4zAle3/cmkNhFYYCkF+1WxPyl5UrSs106W OQykB3D0Nf1AdQcTn5QEgZAgLXj2yc/bmfD730lCcg2KbmQOiKizFApCGfGtto0Qqi8B rnpmcirGDZRzFACpOg6zPFQpmSt0VWVFCCVTWwXgAqzFF6nwxvIy2yBhlFuJPgS7bubX pQnEl+O2q+2x0j2xsPIXSOPTz7ueUk8V4pqs7kvW+WyMLwP+ZfxAgKgQ0Rlwp1U6lgJC iNbZergssapTsC0DDE2jvZ69a6zQ5NljHNQZj++uBXCruoodfHcCJQg44+uT+mJZOKfO J/KA== X-Gm-Message-State: AOJu0Yxp7xcxwUgZ504qMnJpBcuB//FRggsWpKTz7cXohXXQ7Ekql4T1 ZGxqv+WA8LEgg9yAyvhrri6K4M8rqc4m4ycnT3URXdj+y4qs4XsewiswOg== X-Google-Smtp-Source: AGHT+IGe9DW+u5d7YhtV2coNCHdsuIuR4Bqp6Ytu8BryQ+8TGFuT5OoaEd8kf5HHcxnPPNVRtLlf1w== X-Received: by 2002:a17:902:e892:b0:1eb:d7ce:1946 with SMTP id d9443c01a7336-1f4497e5828mr430865ad.52.1716486532351; Thu, 23 May 2024 10:48:52 -0700 (PDT) Received: from localhost.localdomain ([2601:646:8f03:9fee:4f9e:216e:ed5a:642d]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1ef0c137e7esm253436265ad.257.2024.05.23.10.48.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 May 2024 10:48:52 -0700 (PDT) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, armbru@redhat.com, mst@redhat.com, Fan Ni , Svetly Todorov , Jonathan Cameron Subject: [PATCH v8 09/14] hw/mem/cxl_type3: Add DC extent list representative and get DC extent list mailbox support Date: Thu, 23 May 2024 10:44:49 -0700 Message-ID: <20240523174651.1089554-10-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240523174651.1089554-1-nifan.cxl@gmail.com> References: <20240523174651.1089554-1-nifan.cxl@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62e; envelope-from=nifan.cxl@gmail.com; helo=mail-pl1-x62e.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Fan Ni Add dynamic capacity extent list representative to the definition of CXLType3Dev and implement get DC extent list mailbox command per CXL.spec.3.1:.8.2.9.9.9.2. Tested-by: Svetly Todorov Reviewed-by: Jonathan Cameron Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 73 ++++++++++++++++++++++++++++++++++++- hw/mem/cxl_type3.c | 1 + include/hw/cxl/cxl_device.h | 22 +++++++++++ 3 files changed, 95 insertions(+), 1 deletion(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 831cef0567..1915959015 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -84,6 +84,7 @@ enum { #define CLEAR_POISON 0x2 DCD_CONFIG = 0x48, #define GET_DC_CONFIG 0x0 + #define GET_DYN_CAP_EXT_LIST 0x1 PHYSICAL_SWITCH = 0x51, #define IDENTIFY_SWITCH_DEVICE 0x0 #define GET_PHYSICAL_PORT_STATE 0x1 @@ -1322,7 +1323,8 @@ static CXLRetCode cmd_dcd_get_dyn_cap_config(const struct cxl_cmd *cmd, * to use. */ stl_le_p(&extra_out->num_extents_supported, CXL_NUM_EXTENTS_SUPPORTED); - stl_le_p(&extra_out->num_extents_available, CXL_NUM_EXTENTS_SUPPORTED); + stl_le_p(&extra_out->num_extents_available, CXL_NUM_EXTENTS_SUPPORTED - + ct3d->dc.total_extent_count); stl_le_p(&extra_out->num_tags_supported, CXL_NUM_TAGS_SUPPORTED); stl_le_p(&extra_out->num_tags_available, CXL_NUM_TAGS_SUPPORTED); @@ -1330,6 +1332,72 @@ static CXLRetCode cmd_dcd_get_dyn_cap_config(const struct cxl_cmd *cmd, return CXL_MBOX_SUCCESS; } +/* + * CXL r3.1 section 8.2.9.9.9.2: + * Get Dynamic Capacity Extent List (Opcode 4801h) + */ +static CXLRetCode cmd_dcd_get_dyn_cap_ext_list(const struct cxl_cmd *cmd, + uint8_t *payload_in, + size_t len_in, + uint8_t *payload_out, + size_t *len_out, + CXLCCI *cci) +{ + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + struct { + uint32_t extent_cnt; + uint32_t start_extent_id; + } QEMU_PACKED *in = (void *)payload_in; + struct { + uint32_t count; + uint32_t total_extents; + uint32_t generation_num; + uint8_t rsvd[4]; + CXLDCExtentRaw records[]; + } QEMU_PACKED *out = (void *)payload_out; + uint32_t start_extent_id = in->start_extent_id; + CXLDCExtentList *extent_list = &ct3d->dc.extents; + uint16_t record_count = 0, i = 0, record_done = 0; + uint16_t out_pl_len, size; + CXLDCExtent *ent; + + if (start_extent_id > ct3d->dc.total_extent_count) { + return CXL_MBOX_INVALID_INPUT; + } + + record_count = MIN(in->extent_cnt, + ct3d->dc.total_extent_count - start_extent_id); + size = CXL_MAILBOX_MAX_PAYLOAD_SIZE - sizeof(*out); + record_count = MIN(record_count, size / sizeof(out->records[0])); + out_pl_len = sizeof(*out) + record_count * sizeof(out->records[0]); + + stl_le_p(&out->count, record_count); + stl_le_p(&out->total_extents, ct3d->dc.total_extent_count); + stl_le_p(&out->generation_num, ct3d->dc.ext_list_gen_seq); + + if (record_count > 0) { + CXLDCExtentRaw *out_rec = &out->records[record_done]; + + QTAILQ_FOREACH(ent, extent_list, node) { + if (i++ < start_extent_id) { + continue; + } + stq_le_p(&out_rec->start_dpa, ent->start_dpa); + stq_le_p(&out_rec->len, ent->len); + memcpy(&out_rec->tag, ent->tag, 0x10); + stw_le_p(&out_rec->shared_seq, ent->shared_seq); + + record_done++; + if (record_done == record_count) { + break; + } + } + } + + *len_out = out_pl_len; + return CXL_MBOX_SUCCESS; +} + #define IMMEDIATE_CONFIG_CHANGE (1 << 1) #define IMMEDIATE_DATA_CHANGE (1 << 2) #define IMMEDIATE_POLICY_CHANGE (1 << 3) @@ -1377,6 +1445,9 @@ static const struct cxl_cmd cxl_cmd_set[256][256] = { static const struct cxl_cmd cxl_cmd_set_dcd[256][256] = { [DCD_CONFIG][GET_DC_CONFIG] = { "DCD_GET_DC_CONFIG", cmd_dcd_get_dyn_cap_config, 2, 0 }, + [DCD_CONFIG][GET_DYN_CAP_EXT_LIST] = { + "DCD_GET_DYNAMIC_CAPACITY_EXTENT_LIST", cmd_dcd_get_dyn_cap_ext_list, + 8, 0 }, }; static const struct cxl_cmd cxl_cmd_set_sw[256][256] = { diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index f645a3f2e9..f6ab885270 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -672,6 +672,7 @@ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) }; ct3d->dc.total_capacity += region->len; } + QTAILQ_INIT(&ct3d->dc.extents); return true; } diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index c2c3df0d2a..6aec6ac983 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -424,6 +424,25 @@ typedef QLIST_HEAD(, CXLPoison) CXLPoisonList; #define DCD_MAX_NUM_REGION 8 +typedef struct CXLDCExtentRaw { + uint64_t start_dpa; + uint64_t len; + uint8_t tag[0x10]; + uint16_t shared_seq; + uint8_t rsvd[0x6]; +} QEMU_PACKED CXLDCExtentRaw; + +typedef struct CXLDCExtent { + uint64_t start_dpa; + uint64_t len; + uint8_t tag[0x10]; + uint16_t shared_seq; + uint8_t rsvd[0x6]; + + QTAILQ_ENTRY(CXLDCExtent) node; +} CXLDCExtent; +typedef QTAILQ_HEAD(, CXLDCExtent) CXLDCExtentList; + typedef struct CXLDCRegion { uint64_t base; /* aligned to 256*MiB */ uint64_t decode_len; /* aligned to 256*MiB */ @@ -474,6 +493,9 @@ struct CXLType3Dev { * memory region size. */ uint64_t total_capacity; /* 256M aligned */ + CXLDCExtentList extents; + uint32_t total_extent_count; + uint32_t ext_list_gen_seq; uint8_t num_regions; /* 0-8 regions */ CXLDCRegion regions[DCD_MAX_NUM_REGION]; From patchwork Thu May 23 17:44:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13672142 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B00F4C25B7C for ; Thu, 23 May 2024 17:52:21 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sACZL-0006VY-CJ; Thu, 23 May 2024 13:49:35 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sACYq-000617-SV for qemu-devel@nongnu.org; Thu, 23 May 2024 13:49:08 -0400 Received: from mail-pl1-x62a.google.com ([2607:f8b0:4864:20::62a]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sACYl-0003fq-7F for qemu-devel@nongnu.org; Thu, 23 May 2024 13:49:04 -0400 Received: by mail-pl1-x62a.google.com with SMTP id d9443c01a7336-1ee12baa01cso22323975ad.0 for ; Thu, 23 May 2024 10:48:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716486534; x=1717091334; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=H+4O1LRK0w7JEk6xZ7KkbNisTzSTCmae7n6wPfH43e0=; b=biF0IpB9xbZwN1AZWmsJUlllRaAWDQShMoMx0b3ZE52FRYpZ9svlhGDWhUbu72cI7K 1kHZqJGwH7ktd45ml4+k53QKT01nz/1Y8efRP0BtgMrTpme/1tG2RpAUBB0p5KbzJbUU Ku+3Tg77UBqrOT42CP6A3URgCSZ0uIcmMtUnCk8ayBJFDb5/0fMV4TJFESsC3EPAr6VF JbjwaJAw1Z0C6U5eBTZjMCwXkIkm111nYstX7andtqXRVk0MNSq8ZSbq+1yA7ZZe/tEb 5DIWhsvmi5x+vWi1gXdSeKBjFbHc1NXjYSNlQ2f96zXfqRIAJvqPJef4erVur3C2epkW vESA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716486534; x=1717091334; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=H+4O1LRK0w7JEk6xZ7KkbNisTzSTCmae7n6wPfH43e0=; b=v3Z1d1x279NTOA0c3JBKpL3MN8uK+VdlNZ/Dogmb66gZ4KWtf4u6pTJAKhkYeqQq13 NrxHVEF4Jnlnv1FmIHJIP4WaPOfSnHy8ftUC3DVlio7FCYIK1woBcEEUJAoA3ih0C+rf +LcVI0nalmWCxnJFsz35QxbRBDfw0aLkSn11VF0fZ/WrjJGrZQV6ut3Z0PuE1f9iJNfK WxJQdX8AGjIcmrDFqZnuLF7uOpu/Ag2+V0mmKZc6mbUVwdQ0BQDcYp9S1JY6hk9DLDu6 SgNqX4IBsjUMkRxhTZjc4P51PWLm9Hpk1lEHS5ukSmVBV3wZ76wb+IWAQdKMCKNSAc7t c/oA== X-Gm-Message-State: AOJu0YwWj7rAZNAXRmgS1w95cisfsNJeBxh8qHaOSzsJklhT22jfJNj5 IlNQXDD+8IUBqbjCpYQA20oMqGJf/KVxTGIfM9CBmDMTpDCx2e7QDSKl4w== X-Google-Smtp-Source: AGHT+IEYspVOvGlpOoOl8kIMU1N/66STiITd90cw+dE9S7GNgdR+9/11FKNb0K1MorrvNvQRfgAUpg== X-Received: by 2002:a17:902:f544:b0:1e8:ef23:80f6 with SMTP id d9443c01a7336-1f44893f1d1mr550575ad.37.1716486533910; Thu, 23 May 2024 10:48:53 -0700 (PDT) Received: from localhost.localdomain ([2601:646:8f03:9fee:4f9e:216e:ed5a:642d]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1ef0c137e7esm253436265ad.257.2024.05.23.10.48.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 May 2024 10:48:53 -0700 (PDT) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, armbru@redhat.com, mst@redhat.com, Fan Ni , Svetly Todorov Subject: [PATCH v8 10/14] hw/cxl/cxl-mailbox-utils: Add mailbox commands to support add/release dynamic capacity response Date: Thu, 23 May 2024 10:44:50 -0700 Message-ID: <20240523174651.1089554-11-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240523174651.1089554-1-nifan.cxl@gmail.com> References: <20240523174651.1089554-1-nifan.cxl@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62a; envelope-from=nifan.cxl@gmail.com; helo=mail-pl1-x62a.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Fan Ni Per CXL spec 3.1, two mailbox commands are implemented: Add Dynamic Capacity Response (Opcode 4802h) 8.2.9.9.9.3, and Release Dynamic Capacity (Opcode 4803h) 8.2.9.9.9.4. For the process of the above two commands, we use two-pass approach. Pass 1: Check whether the input payload is valid or not; if not, skip Pass 2 and return mailbox process error. Pass 2: Do the real work--add or release extents, respectively. Tested-by: Svetly Todorov Reviewed-by: Gregory Price Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 394 ++++++++++++++++++++++++++++++++++++ hw/mem/cxl_type3.c | 11 + include/hw/cxl/cxl_device.h | 4 + 3 files changed, 409 insertions(+) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 1915959015..9d54e10cd4 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -19,6 +19,7 @@ #include "qemu/units.h" #include "qemu/uuid.h" #include "sysemu/hostmem.h" +#include "qemu/range.h" #define CXL_CAPACITY_MULTIPLIER (256 * MiB) #define CXL_DC_EVENT_LOG_SIZE 8 @@ -85,6 +86,8 @@ enum { DCD_CONFIG = 0x48, #define GET_DC_CONFIG 0x0 #define GET_DYN_CAP_EXT_LIST 0x1 + #define ADD_DYN_CAP_RSP 0x2 + #define RELEASE_DYN_CAP 0x3 PHYSICAL_SWITCH = 0x51, #define IDENTIFY_SWITCH_DEVICE 0x0 #define GET_PHYSICAL_PORT_STATE 0x1 @@ -1398,6 +1401,391 @@ static CXLRetCode cmd_dcd_get_dyn_cap_ext_list(const struct cxl_cmd *cmd, return CXL_MBOX_SUCCESS; } +/* + * Check whether any bit between addr[nr, nr+size) is set, + * return true if any bit is set, otherwise return false + */ +static bool test_any_bits_set(const unsigned long *addr, unsigned long nr, + unsigned long size) +{ + unsigned long res = find_next_bit(addr, size + nr, nr); + + return res < nr + size; +} + +CXLDCRegion *cxl_find_dc_region(CXLType3Dev *ct3d, uint64_t dpa, uint64_t len) +{ + int i; + CXLDCRegion *region = &ct3d->dc.regions[0]; + + if (dpa < region->base || + dpa >= region->base + ct3d->dc.total_capacity) { + return NULL; + } + + /* + * CXL r3.1 section 9.13.3: Dynamic Capacity Device (DCD) + * + * Regions are used in increasing-DPA order, with Region 0 being used for + * the lowest DPA of Dynamic Capacity and Region 7 for the highest DPA. + * So check from the last region to find where the dpa belongs. Extents that + * cross multiple regions are not allowed. + */ + for (i = ct3d->dc.num_regions - 1; i >= 0; i--) { + region = &ct3d->dc.regions[i]; + if (dpa >= region->base) { + if (dpa + len > region->base + region->len) { + return NULL; + } + return region; + } + } + + return NULL; +} + +static void cxl_insert_extent_to_extent_list(CXLDCExtentList *list, + uint64_t dpa, + uint64_t len, + uint8_t *tag, + uint16_t shared_seq) +{ + CXLDCExtent *extent; + + extent = g_new0(CXLDCExtent, 1); + extent->start_dpa = dpa; + extent->len = len; + if (tag) { + memcpy(extent->tag, tag, 0x10); + } + extent->shared_seq = shared_seq; + + QTAILQ_INSERT_TAIL(list, extent, node); +} + +void cxl_remove_extent_from_extent_list(CXLDCExtentList *list, + CXLDCExtent *extent) +{ + QTAILQ_REMOVE(list, extent, node); + g_free(extent); +} + +/* + * CXL r3.1 Table 8-168: Add Dynamic Capacity Response Input Payload + * CXL r3.1 Table 8-170: Release Dynamic Capacity Input Payload + */ +typedef struct CXLUpdateDCExtentListInPl { + uint32_t num_entries_updated; + uint8_t flags; + uint8_t rsvd[3]; + /* CXL r3.1 Table 8-169: Updated Extent */ + struct { + uint64_t start_dpa; + uint64_t len; + uint8_t rsvd[8]; + } QEMU_PACKED updated_entries[]; +} QEMU_PACKED CXLUpdateDCExtentListInPl; + +/* + * For the extents in the extent list to operate, check whether they are valid + * 1. The extent should be in the range of a valid DC region; + * 2. The extent should not cross multiple regions; + * 3. The start DPA and the length of the extent should align with the block + * size of the region; + * 4. The address range of multiple extents in the list should not overlap. + */ +static CXLRetCode cxl_detect_malformed_extent_list(CXLType3Dev *ct3d, + const CXLUpdateDCExtentListInPl *in) +{ + uint64_t min_block_size = UINT64_MAX; + CXLDCRegion *region; + CXLDCRegion *lastregion = &ct3d->dc.regions[ct3d->dc.num_regions - 1]; + g_autofree unsigned long *blk_bitmap = NULL; + uint64_t dpa, len; + uint32_t i; + + for (i = 0; i < ct3d->dc.num_regions; i++) { + region = &ct3d->dc.regions[i]; + min_block_size = MIN(min_block_size, region->block_size); + } + + blk_bitmap = bitmap_new((lastregion->base + lastregion->len - + ct3d->dc.regions[0].base) / min_block_size); + + for (i = 0; i < in->num_entries_updated; i++) { + dpa = in->updated_entries[i].start_dpa; + len = in->updated_entries[i].len; + + region = cxl_find_dc_region(ct3d, dpa, len); + if (!region) { + return CXL_MBOX_INVALID_PA; + } + + dpa -= ct3d->dc.regions[0].base; + if (dpa % region->block_size || len % region->block_size) { + return CXL_MBOX_INVALID_EXTENT_LIST; + } + /* the dpa range already covered by some other extents in the list */ + if (test_any_bits_set(blk_bitmap, dpa / min_block_size, + len / min_block_size)) { + return CXL_MBOX_INVALID_EXTENT_LIST; + } + bitmap_set(blk_bitmap, dpa / min_block_size, len / min_block_size); + } + + return CXL_MBOX_SUCCESS; +} + +static CXLRetCode cxl_dcd_add_dyn_cap_rsp_dry_run(CXLType3Dev *ct3d, + const CXLUpdateDCExtentListInPl *in) +{ + uint32_t i; + CXLDCExtent *ent; + uint64_t dpa, len; + Range range1, range2; + + for (i = 0; i < in->num_entries_updated; i++) { + dpa = in->updated_entries[i].start_dpa; + len = in->updated_entries[i].len; + + range_init_nofail(&range1, dpa, len); + + /* + * TODO: once the pending extent list is added, check against + * the list will be added here. + */ + + /* to-be-added range should not overlap with range already accepted */ + QTAILQ_FOREACH(ent, &ct3d->dc.extents, node) { + range_init_nofail(&range2, ent->start_dpa, ent->len); + if (range_overlaps_range(&range1, &range2)) { + return CXL_MBOX_INVALID_PA; + } + } + } + return CXL_MBOX_SUCCESS; +} + +/* + * CXL r3.1 section 8.2.9.9.9.3: Add Dynamic Capacity Response (Opcode 4802h) + * An extent is added to the extent list and becomes usable only after the + * response is processed successfully. + */ +static CXLRetCode cmd_dcd_add_dyn_cap_rsp(const struct cxl_cmd *cmd, + uint8_t *payload_in, + size_t len_in, + uint8_t *payload_out, + size_t *len_out, + CXLCCI *cci) +{ + CXLUpdateDCExtentListInPl *in = (void *)payload_in; + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + CXLDCExtentList *extent_list = &ct3d->dc.extents; + uint32_t i; + uint64_t dpa, len; + CXLRetCode ret; + + if (in->num_entries_updated == 0) { + /* + * TODO: once the pending list is introduced, extents in the beginning + * will get wiped out. + */ + return CXL_MBOX_SUCCESS; + } + + /* Adding extents causes exceeding device's extent tracking ability. */ + if (in->num_entries_updated + ct3d->dc.total_extent_count > + CXL_NUM_EXTENTS_SUPPORTED) { + return CXL_MBOX_RESOURCES_EXHAUSTED; + } + + ret = cxl_detect_malformed_extent_list(ct3d, in); + if (ret != CXL_MBOX_SUCCESS) { + return ret; + } + + ret = cxl_dcd_add_dyn_cap_rsp_dry_run(ct3d, in); + if (ret != CXL_MBOX_SUCCESS) { + return ret; + } + + for (i = 0; i < in->num_entries_updated; i++) { + dpa = in->updated_entries[i].start_dpa; + len = in->updated_entries[i].len; + + cxl_insert_extent_to_extent_list(extent_list, dpa, len, NULL, 0); + ct3d->dc.total_extent_count += 1; + /* + * TODO: we will add a pending extent list based on event log record + * and process the list accordingly here. + */ + } + + return CXL_MBOX_SUCCESS; +} + +/* + * Copy extent list from src to dst + * Return value: number of extents copied + */ +static uint32_t copy_extent_list(CXLDCExtentList *dst, + const CXLDCExtentList *src) +{ + uint32_t cnt = 0; + CXLDCExtent *ent; + + if (!dst || !src) { + return 0; + } + + QTAILQ_FOREACH(ent, src, node) { + cxl_insert_extent_to_extent_list(dst, ent->start_dpa, ent->len, + ent->tag, ent->shared_seq); + cnt++; + } + return cnt; +} + +static CXLRetCode cxl_dc_extent_release_dry_run(CXLType3Dev *ct3d, + const CXLUpdateDCExtentListInPl *in, CXLDCExtentList *updated_list, + uint32_t *updated_list_size) +{ + CXLDCExtent *ent, *ent_next; + uint64_t dpa, len; + uint32_t i; + int cnt_delta = 0; + CXLRetCode ret = CXL_MBOX_SUCCESS; + + QTAILQ_INIT(updated_list); + copy_extent_list(updated_list, &ct3d->dc.extents); + + for (i = 0; i < in->num_entries_updated; i++) { + Range range; + + dpa = in->updated_entries[i].start_dpa; + len = in->updated_entries[i].len; + + while (len > 0) { + QTAILQ_FOREACH(ent, updated_list, node) { + range_init_nofail(&range, ent->start_dpa, ent->len); + + if (range_contains(&range, dpa)) { + uint64_t len1, len2 = 0, len_done = 0; + uint64_t ent_start_dpa = ent->start_dpa; + uint64_t ent_len = ent->len; + + len1 = dpa - ent->start_dpa; + /* Found the extent or the subset of an existing extent */ + if (range_contains(&range, dpa + len - 1)) { + len2 = ent_start_dpa + ent_len - dpa - len; + } else { + /* + * TODO: we reject the attempt to remove an extent + * that overlaps with multiple extents in the device + * for now. We will allow it once superset release + * support is added. + */ + ret = CXL_MBOX_INVALID_PA; + goto free_and_exit; + } + len_done = ent_len - len1 - len2; + + cxl_remove_extent_from_extent_list(updated_list, ent); + cnt_delta--; + + if (len1) { + cxl_insert_extent_to_extent_list(updated_list, + ent_start_dpa, + len1, NULL, 0); + cnt_delta++; + } + if (len2) { + cxl_insert_extent_to_extent_list(updated_list, + dpa + len, + len2, NULL, 0); + cnt_delta++; + } + + if (cnt_delta + ct3d->dc.total_extent_count > + CXL_NUM_EXTENTS_SUPPORTED) { + ret = CXL_MBOX_RESOURCES_EXHAUSTED; + goto free_and_exit; + } + + len -= len_done; + /* len == 0 here until superset release is added */ + break; + } + } + if (len) { + ret = CXL_MBOX_INVALID_PA; + goto free_and_exit; + } + } + } +free_and_exit: + if (ret != CXL_MBOX_SUCCESS) { + QTAILQ_FOREACH_SAFE(ent, updated_list, node, ent_next) { + cxl_remove_extent_from_extent_list(updated_list, ent); + } + *updated_list_size = 0; + } else { + *updated_list_size = ct3d->dc.total_extent_count + cnt_delta; + } + + return ret; +} + +/* + * CXL r3.1 section 8.2.9.9.9.4: Release Dynamic Capacity (Opcode 4803h) + */ +static CXLRetCode cmd_dcd_release_dyn_cap(const struct cxl_cmd *cmd, + uint8_t *payload_in, + size_t len_in, + uint8_t *payload_out, + size_t *len_out, + CXLCCI *cci) +{ + CXLUpdateDCExtentListInPl *in = (void *)payload_in; + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + CXLDCExtentList updated_list; + CXLDCExtent *ent, *ent_next; + uint32_t updated_list_size; + CXLRetCode ret; + + if (in->num_entries_updated == 0) { + return CXL_MBOX_INVALID_INPUT; + } + + ret = cxl_detect_malformed_extent_list(ct3d, in); + if (ret != CXL_MBOX_SUCCESS) { + return ret; + } + + ret = cxl_dc_extent_release_dry_run(ct3d, in, &updated_list, + &updated_list_size); + if (ret != CXL_MBOX_SUCCESS) { + return ret; + } + + /* + * If the dry run release passes, the returned updated_list will + * be the updated extent list and we just need to clear the extents + * in the accepted list and copy extents in the updated_list to accepted + * list and update the extent count; + */ + QTAILQ_FOREACH_SAFE(ent, &ct3d->dc.extents, node, ent_next) { + cxl_remove_extent_from_extent_list(&ct3d->dc.extents, ent); + } + copy_extent_list(&ct3d->dc.extents, &updated_list); + QTAILQ_FOREACH_SAFE(ent, &updated_list, node, ent_next) { + cxl_remove_extent_from_extent_list(&updated_list, ent); + } + ct3d->dc.total_extent_count = updated_list_size; + + return CXL_MBOX_SUCCESS; +} + #define IMMEDIATE_CONFIG_CHANGE (1 << 1) #define IMMEDIATE_DATA_CHANGE (1 << 2) #define IMMEDIATE_POLICY_CHANGE (1 << 3) @@ -1448,6 +1836,12 @@ static const struct cxl_cmd cxl_cmd_set_dcd[256][256] = { [DCD_CONFIG][GET_DYN_CAP_EXT_LIST] = { "DCD_GET_DYNAMIC_CAPACITY_EXTENT_LIST", cmd_dcd_get_dyn_cap_ext_list, 8, 0 }, + [DCD_CONFIG][ADD_DYN_CAP_RSP] = { + "DCD_ADD_DYNAMIC_CAPACITY_RESPONSE", cmd_dcd_add_dyn_cap_rsp, + ~0, IMMEDIATE_DATA_CHANGE }, + [DCD_CONFIG][RELEASE_DYN_CAP] = { + "DCD_RELEASE_DYNAMIC_CAPACITY", cmd_dcd_release_dyn_cap, + ~0, IMMEDIATE_DATA_CHANGE }, }; static const struct cxl_cmd cxl_cmd_set_sw[256][256] = { diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index f6ab885270..7c9038938f 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -677,6 +677,15 @@ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) return true; } +static void cxl_destroy_dc_regions(CXLType3Dev *ct3d) +{ + CXLDCExtent *ent, *ent_next; + + QTAILQ_FOREACH_SAFE(ent, &ct3d->dc.extents, node, ent_next) { + cxl_remove_extent_from_extent_list(&ct3d->dc.extents, ent); + } +} + static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) { DeviceState *ds = DEVICE(ct3d); @@ -873,6 +882,7 @@ err_free_special_ops: g_free(regs->special_ops); err_address_space_free: if (ct3d->dc.host_dc) { + cxl_destroy_dc_regions(ct3d); address_space_destroy(&ct3d->dc.host_dc_as); } if (ct3d->hostpmem) { @@ -894,6 +904,7 @@ static void ct3_exit(PCIDevice *pci_dev) cxl_doe_cdat_release(cxl_cstate); g_free(regs->special_ops); if (ct3d->dc.host_dc) { + cxl_destroy_dc_regions(ct3d); address_space_destroy(&ct3d->dc.host_dc_as); } if (ct3d->hostpmem) { diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index 6aec6ac983..df3511e91b 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -551,4 +551,8 @@ void cxl_event_irq_assert(CXLType3Dev *ct3d); void cxl_set_poison_list_overflowed(CXLType3Dev *ct3d); +CXLDCRegion *cxl_find_dc_region(CXLType3Dev *ct3d, uint64_t dpa, uint64_t len); + +void cxl_remove_extent_from_extent_list(CXLDCExtentList *list, + CXLDCExtent *extent); #endif From patchwork Thu May 23 17:44:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13672118 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6EA3BC25B7C for ; Thu, 23 May 2024 17:51:22 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sACZe-0007NN-DM; Thu, 23 May 2024 13:49:54 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sACYx-000657-3n for qemu-devel@nongnu.org; Thu, 23 May 2024 13:49:12 -0400 Received: from mail-pl1-x62f.google.com ([2607:f8b0:4864:20::62f]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sACYl-0003gG-BG for qemu-devel@nongnu.org; Thu, 23 May 2024 13:49:08 -0400 Received: by mail-pl1-x62f.google.com with SMTP id d9443c01a7336-1f082d92864so124820765ad.1 for ; Thu, 23 May 2024 10:48:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716486536; x=1717091336; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1HWdSvK3rb01MwXuCorH0KLVslWq14FYs2B5Pff32z4=; b=aOHqNtk7RhoPMn93KXb0+SCjUbks5B2lodWzSJL31g1uWYY41uVK7vRGv6gvMvvmp0 shh452NiiLyhARiQqAagRoQVPUN5Qn1cgLLQtu8twHf3XNpgzazsJpbKFrZtgRZJ1Sj1 ejYJNI3jc7kDMA/a8sKum20uvHFvRVYJnSiZYJ0tDoBJd1PxeZGKvXVWZ0KU9Zs6uOmA ihNlNytEf5mgVDCugQ8pPhgpUZw8DLOXVDUmzPMQWvtUOUnOhXORIO0fEZ8BDqZtjFOq Oocjh+xTIRHJ8yTkKYOx5XVXf88upc83flnaHX9gv/gdnFOC2FKOP2bpqECamXz9pKxd lOtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716486536; x=1717091336; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1HWdSvK3rb01MwXuCorH0KLVslWq14FYs2B5Pff32z4=; b=QS6U0Rv4DcJBhcf8InQ1vbrfp7zy7FGQLGu65HtsqVcDGODhNzTkDPszpGdy2B8905 jhnqoF1tZC5TdlmyOXFSHVRrNB1b5RoeZT4Qmz9U1kJNyDzqfPFGAVXWKKTbIjeZqOod NvRb38qNYd/yu+m+ZwtFPYRBMz0AFh3mSNJu8CLBbS3Vb4bkg8l8LfdUdrCIwpQMHR1o Em+HyAYpI3FOf+gOfdsqeDBgm3EcEYdo6U6HbLDH2+M8USJqFV4OzTY1Du1x0UVUcTpv LX8cyO4tZTPkSVY886ugCAWY8Tohn6fjbTTDJVzKuis9AvqKvxD5QzKGItheruMnVn4s 7Cug== X-Gm-Message-State: AOJu0YwNkXJnQa0ur5D1A8TbcPeLFOBC5toZLTcbbEZoPoeSIXw81Mzc J5iZTPbDPgafr08DG8NhihL/MHarhyv+ofsjqRndK8fD6mcnBQ+2KuImOw== X-Google-Smtp-Source: AGHT+IEKKFANgvqvL3Rwa7MFbAtfba82Upn/BnOSPz7mqIYx8kmM9FPrz28qdm11eKHvTW1koJVmiA== X-Received: by 2002:a17:903:210c:b0:1e4:8c64:33a2 with SMTP id d9443c01a7336-1f4498ee324mr267535ad.68.1716486535554; Thu, 23 May 2024 10:48:55 -0700 (PDT) Received: from localhost.localdomain ([2601:646:8f03:9fee:4f9e:216e:ed5a:642d]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1ef0c137e7esm253436265ad.257.2024.05.23.10.48.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 May 2024 10:48:55 -0700 (PDT) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, armbru@redhat.com, mst@redhat.com, Fan Ni , Svetly Todorov Subject: [PATCH v8 11/14] hw/cxl/events: Add qmp interfaces to add/release dynamic capacity extents Date: Thu, 23 May 2024 10:44:51 -0700 Message-ID: <20240523174651.1089554-12-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240523174651.1089554-1-nifan.cxl@gmail.com> References: <20240523174651.1089554-1-nifan.cxl@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62f; envelope-from=nifan.cxl@gmail.com; helo=mail-pl1-x62f.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Fan Ni To simulate FM functionalities for initiating Dynamic Capacity Add (Opcode 5604h) and Dynamic Capacity Release (Opcode 5605h) as in CXL spec r3.1 7.6.7.6.5 and 7.6.7.6.6, we implemented two QMP interfaces to issue add/release dynamic capacity extents requests. With the change, we allow to release an extent only when its DPA range is contained by a single accepted extent in the device. That is to say, extent superset release is not supported yet. 1. Add dynamic capacity extents: For example, the command to add two continuous extents (each 128MiB long) to region 0 (starting at DPA offset 0) looks like below: { "execute": "qmp_capabilities" } { "execute": "cxl-add-dynamic-capacity", "arguments": { "path": "/machine/peripheral/cxl-dcd0", "host-id": 0, "selection-policy": "prescriptive", "region": 0, "extents": [ { "offset": 0, "len": 134217728 }, { "offset": 134217728, "len": 134217728 } ] } } 2. Release dynamic capacity extents: For example, the command to release an extent of size 128MiB from region 0 (DPA offset 128MiB) looks like below: { "execute": "cxl-release-dynamic-capacity", "arguments": { "path": "/machine/peripheral/cxl-dcd0", "host-id": 0, "removal-policy":"prescriptive", "region": 0, "extents": [ { "offset": 134217728, "len": 134217728 } ] } } Tested-by: Svetly Todorov Reviewed-by: Gregory Price Signed-off-by: Fan Ni Signed-off-by: Jonathan Cameron --- hw/cxl/cxl-mailbox-utils.c | 62 ++++++-- hw/mem/cxl_type3.c | 306 +++++++++++++++++++++++++++++++++++- hw/mem/cxl_type3_stubs.c | 25 +++ include/hw/cxl/cxl_device.h | 22 +++ include/hw/cxl/cxl_events.h | 18 +++ qapi/cxl.json | 143 +++++++++++++++++ 6 files changed, 563 insertions(+), 13 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 9d54e10cd4..ab71492697 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -1405,7 +1405,7 @@ static CXLRetCode cmd_dcd_get_dyn_cap_ext_list(const struct cxl_cmd *cmd, * Check whether any bit between addr[nr, nr+size) is set, * return true if any bit is set, otherwise return false */ -static bool test_any_bits_set(const unsigned long *addr, unsigned long nr, +bool test_any_bits_set(const unsigned long *addr, unsigned long nr, unsigned long size) { unsigned long res = find_next_bit(addr, size + nr, nr); @@ -1444,7 +1444,7 @@ CXLDCRegion *cxl_find_dc_region(CXLType3Dev *ct3d, uint64_t dpa, uint64_t len) return NULL; } -static void cxl_insert_extent_to_extent_list(CXLDCExtentList *list, +void cxl_insert_extent_to_extent_list(CXLDCExtentList *list, uint64_t dpa, uint64_t len, uint8_t *tag, @@ -1470,6 +1470,44 @@ void cxl_remove_extent_from_extent_list(CXLDCExtentList *list, g_free(extent); } +/* + * Add a new extent to the extent "group" if group exists; + * otherwise, create a new group + * Return value: the extent group where the extent is inserted. + */ +CXLDCExtentGroup *cxl_insert_extent_to_extent_group(CXLDCExtentGroup *group, + uint64_t dpa, + uint64_t len, + uint8_t *tag, + uint16_t shared_seq) +{ + if (!group) { + group = g_new0(CXLDCExtentGroup, 1); + QTAILQ_INIT(&group->list); + } + cxl_insert_extent_to_extent_list(&group->list, dpa, len, + tag, shared_seq); + return group; +} + +void cxl_extent_group_list_insert_tail(CXLDCExtentGroupList *list, + CXLDCExtentGroup *group) +{ + QTAILQ_INSERT_TAIL(list, group, node); +} + +void cxl_extent_group_list_delete_front(CXLDCExtentGroupList *list) +{ + CXLDCExtent *ent, *ent_next; + CXLDCExtentGroup *group = QTAILQ_FIRST(list); + + QTAILQ_REMOVE(list, group, node); + QTAILQ_FOREACH_SAFE(ent, &group->list, node, ent_next) { + cxl_remove_extent_from_extent_list(&group->list, ent); + } + g_free(group); +} + /* * CXL r3.1 Table 8-168: Add Dynamic Capacity Response Input Payload * CXL r3.1 Table 8-170: Release Dynamic Capacity Input Payload @@ -1541,6 +1579,7 @@ static CXLRetCode cxl_dcd_add_dyn_cap_rsp_dry_run(CXLType3Dev *ct3d, { uint32_t i; CXLDCExtent *ent; + CXLDCExtentGroup *ext_group; uint64_t dpa, len; Range range1, range2; @@ -1551,9 +1590,13 @@ static CXLRetCode cxl_dcd_add_dyn_cap_rsp_dry_run(CXLType3Dev *ct3d, range_init_nofail(&range1, dpa, len); /* - * TODO: once the pending extent list is added, check against - * the list will be added here. + * The host-accepted DPA range must be contained by the first extent + * group in the pending list */ + ext_group = QTAILQ_FIRST(&ct3d->dc.extents_pending); + if (!cxl_extents_contains_dpa_range(&ext_group->list, dpa, len)) { + return CXL_MBOX_INVALID_PA; + } /* to-be-added range should not overlap with range already accepted */ QTAILQ_FOREACH(ent, &ct3d->dc.extents, node) { @@ -1586,10 +1629,7 @@ static CXLRetCode cmd_dcd_add_dyn_cap_rsp(const struct cxl_cmd *cmd, CXLRetCode ret; if (in->num_entries_updated == 0) { - /* - * TODO: once the pending list is introduced, extents in the beginning - * will get wiped out. - */ + cxl_extent_group_list_delete_front(&ct3d->dc.extents_pending); return CXL_MBOX_SUCCESS; } @@ -1615,11 +1655,9 @@ static CXLRetCode cmd_dcd_add_dyn_cap_rsp(const struct cxl_cmd *cmd, cxl_insert_extent_to_extent_list(extent_list, dpa, len, NULL, 0); ct3d->dc.total_extent_count += 1; - /* - * TODO: we will add a pending extent list based on event log record - * and process the list accordingly here. - */ } + /* Remove the first extent group in the pending list */ + cxl_extent_group_list_delete_front(&ct3d->dc.extents_pending); return CXL_MBOX_SUCCESS; } diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 7c9038938f..2161766b14 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -673,6 +673,7 @@ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) ct3d->dc.total_capacity += region->len; } QTAILQ_INIT(&ct3d->dc.extents); + QTAILQ_INIT(&ct3d->dc.extents_pending); return true; } @@ -680,10 +681,19 @@ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) static void cxl_destroy_dc_regions(CXLType3Dev *ct3d) { CXLDCExtent *ent, *ent_next; + CXLDCExtentGroup *group, *group_next; QTAILQ_FOREACH_SAFE(ent, &ct3d->dc.extents, node, ent_next) { cxl_remove_extent_from_extent_list(&ct3d->dc.extents, ent); } + + QTAILQ_FOREACH_SAFE(group, &ct3d->dc.extents_pending, node, group_next) { + QTAILQ_REMOVE(&ct3d->dc.extents_pending, group, node); + QTAILQ_FOREACH_SAFE(ent, &group->list, node, ent_next) { + cxl_remove_extent_from_extent_list(&group->list, ent); + } + g_free(group); + } } static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) @@ -1448,7 +1458,6 @@ static int ct3d_qmp_cxl_event_log_enc(CxlEventLog log) return CXL_EVENT_TYPE_FAIL; case CXL_EVENT_LOG_FATAL: return CXL_EVENT_TYPE_FATAL; -/* DCD not yet supported */ default: return -EINVAL; } @@ -1699,6 +1708,301 @@ void qmp_cxl_inject_memory_module_event(const char *path, CxlEventLog log, } } +/* CXL r3.1 Table 8-50: Dynamic Capacity Event Record */ +static const QemuUUID dynamic_capacity_uuid = { + .data = UUID(0xca95afa7, 0xf183, 0x4018, 0x8c, 0x2f, + 0x95, 0x26, 0x8e, 0x10, 0x1a, 0x2a), +}; + +typedef enum CXLDCEventType { + DC_EVENT_ADD_CAPACITY = 0x0, + DC_EVENT_RELEASE_CAPACITY = 0x1, + DC_EVENT_FORCED_RELEASE_CAPACITY = 0x2, + DC_EVENT_REGION_CONFIG_UPDATED = 0x3, + DC_EVENT_ADD_CAPACITY_RSP = 0x4, + DC_EVENT_CAPACITY_RELEASED = 0x5, +} CXLDCEventType; + +/* + * Check whether the range [dpa, dpa + len - 1] has overlaps with extents in + * the list. + * Return value: return true if has overlaps; otherwise, return false + */ +static bool cxl_extents_overlaps_dpa_range(CXLDCExtentList *list, + uint64_t dpa, uint64_t len) +{ + CXLDCExtent *ent; + Range range1, range2; + + if (!list) { + return false; + } + + range_init_nofail(&range1, dpa, len); + QTAILQ_FOREACH(ent, list, node) { + range_init_nofail(&range2, ent->start_dpa, ent->len); + if (range_overlaps_range(&range1, &range2)) { + return true; + } + } + return false; +} + +/* + * Check whether the range [dpa, dpa + len - 1] is contained by extents in + * the list. + * Will check multiple extents containment once superset release is added. + * Return value: return true if range is contained; otherwise, return false + */ +bool cxl_extents_contains_dpa_range(CXLDCExtentList *list, + uint64_t dpa, uint64_t len) +{ + CXLDCExtent *ent; + Range range1, range2; + + if (!list) { + return false; + } + + range_init_nofail(&range1, dpa, len); + QTAILQ_FOREACH(ent, list, node) { + range_init_nofail(&range2, ent->start_dpa, ent->len); + if (range_contains_range(&range2, &range1)) { + return true; + } + } + return false; +} + +static bool cxl_extent_groups_overlaps_dpa_range(CXLDCExtentGroupList *list, + uint64_t dpa, uint64_t len) +{ + CXLDCExtentGroup *group; + + if (!list) { + return false; + } + + QTAILQ_FOREACH(group, list, node) { + if (cxl_extents_overlaps_dpa_range(&group->list, dpa, len)) { + return true; + } + } + return false; +} + +/* + * The main function to process dynamic capacity event with extent list. + * Currently DC extents add/release requests are processed. + */ +static void qmp_cxl_process_dynamic_capacity_prescriptive(const char *path, + uint16_t hid, CXLDCEventType type, uint8_t rid, + CXLDynamicCapacityExtentList *records, Error **errp) +{ + Object *obj; + CXLEventDynamicCapacity dCap = {}; + CXLEventRecordHdr *hdr = &dCap.hdr; + CXLType3Dev *dcd; + uint8_t flags = 1 << CXL_EVENT_TYPE_INFO; + uint32_t num_extents = 0; + CXLDynamicCapacityExtentList *list; + CXLDCExtentGroup *group = NULL; + g_autofree CXLDCExtentRaw *extents = NULL; + uint8_t enc_log = CXL_EVENT_TYPE_DYNAMIC_CAP; + uint64_t dpa, offset, len, block_size; + g_autofree unsigned long *blk_bitmap = NULL; + int i; + + obj = object_resolve_path_type(path, TYPE_CXL_TYPE3, NULL); + if (!obj) { + error_setg(errp, "Unable to resolve CXL type 3 device"); + return; + } + + dcd = CXL_TYPE3(obj); + if (!dcd->dc.num_regions) { + error_setg(errp, "No dynamic capacity support from the device"); + return; + } + + + if (rid >= dcd->dc.num_regions) { + error_setg(errp, "region id is too large"); + return; + } + block_size = dcd->dc.regions[rid].block_size; + blk_bitmap = bitmap_new(dcd->dc.regions[rid].len / block_size); + + /* Sanity check and count the extents */ + list = records; + while (list) { + offset = list->value->offset; + len = list->value->len; + dpa = offset + dcd->dc.regions[rid].base; + + if (len == 0) { + error_setg(errp, "extent with 0 length is not allowed"); + return; + } + + if (offset % block_size || len % block_size) { + error_setg(errp, "dpa or len is not aligned to region block size"); + return; + } + + if (offset + len > dcd->dc.regions[rid].len) { + error_setg(errp, "extent range is beyond the region end"); + return; + } + + /* No duplicate or overlapped extents are allowed */ + if (test_any_bits_set(blk_bitmap, offset / block_size, + len / block_size)) { + error_setg(errp, "duplicate or overlapped extents are detected"); + return; + } + bitmap_set(blk_bitmap, offset / block_size, len / block_size); + + if (type == DC_EVENT_RELEASE_CAPACITY) { + if (cxl_extent_groups_overlaps_dpa_range(&dcd->dc.extents_pending, + dpa, len)) { + error_setg(errp, + "cannot release extent with pending DPA range"); + return; + } + if (!cxl_extents_contains_dpa_range(&dcd->dc.extents, dpa, len)) { + error_setg(errp, + "cannot release extent with non-existing DPA range"); + return; + } + } else if (type == DC_EVENT_ADD_CAPACITY) { + if (cxl_extents_overlaps_dpa_range(&dcd->dc.extents, dpa, len)) { + error_setg(errp, + "cannot add DPA already accessible to the same LD"); + return; + } + if (cxl_extent_groups_overlaps_dpa_range(&dcd->dc.extents_pending, + dpa, len)) { + error_setg(errp, + "cannot add DPA again while still pending"); + return; + } + } + list = list->next; + num_extents++; + } + + /* Create extent list for event being passed to host */ + i = 0; + list = records; + extents = g_new0(CXLDCExtentRaw, num_extents); + while (list) { + offset = list->value->offset; + len = list->value->len; + dpa = dcd->dc.regions[rid].base + offset; + + extents[i].start_dpa = dpa; + extents[i].len = len; + memset(extents[i].tag, 0, 0x10); + extents[i].shared_seq = 0; + if (type == DC_EVENT_ADD_CAPACITY) { + group = cxl_insert_extent_to_extent_group(group, + extents[i].start_dpa, + extents[i].len, + extents[i].tag, + extents[i].shared_seq); + } + + list = list->next; + i++; + } + if (group) { + cxl_extent_group_list_insert_tail(&dcd->dc.extents_pending, group); + } + + /* + * CXL r3.1 section 8.2.9.2.1.6: Dynamic Capacity Event Record + * + * All Dynamic Capacity event records shall set the Event Record Severity + * field in the Common Event Record Format to Informational Event. All + * Dynamic Capacity related events shall be logged in the Dynamic Capacity + * Event Log. + */ + cxl_assign_event_header(hdr, &dynamic_capacity_uuid, flags, sizeof(dCap), + cxl_device_get_timestamp(&dcd->cxl_dstate)); + + dCap.type = type; + /* FIXME: for now, validity flag is cleared */ + dCap.validity_flags = 0; + stw_le_p(&dCap.host_id, hid); + /* only valid for DC_REGION_CONFIG_UPDATED event */ + dCap.updated_region_id = 0; + dCap.flags = 0; + for (i = 0; i < num_extents; i++) { + memcpy(&dCap.dynamic_capacity_extent, &extents[i], + sizeof(CXLDCExtentRaw)); + + if (i < num_extents - 1) { + /* Set "More" flag */ + dCap.flags |= BIT(0); + } + + if (cxl_event_insert(&dcd->cxl_dstate, enc_log, + (CXLEventRecordRaw *)&dCap)) { + cxl_event_irq_assert(dcd); + } + } +} + +void qmp_cxl_add_dynamic_capacity(const char *path, uint16_t host_id, + CXLExtSelPolicy sel_policy, uint8_t region, + const char *tag, + CXLDynamicCapacityExtentList *extents, + Error **errp) +{ + switch (sel_policy) { + case CXL_EXT_SEL_POLICY_PRESCRIPTIVE: + qmp_cxl_process_dynamic_capacity_prescriptive(path, host_id, + DC_EVENT_ADD_CAPACITY, + region, extents, errp); + return; + default: + error_setg(errp, "Selection policy not supported"); + return; + } +} + +void qmp_cxl_release_dynamic_capacity(const char *path, uint16_t host_id, + CXLExtRemovalPolicy removal_policy, + bool has_forced_removal, + bool forced_removal, + bool has_sanitize_on_release, + bool sanitize_on_release, + uint8_t region, + const char *tag, + CXLDynamicCapacityExtentList *extents, + Error **errp) +{ + CXLDCEventType type = DC_EVENT_RELEASE_CAPACITY; + + if (has_forced_removal && forced_removal) { + /* TODO: enable forced removal in the future */ + type = DC_EVENT_FORCED_RELEASE_CAPACITY; + error_setg(errp, "Forced removal not supported yet"); + return; + } + + switch (removal_policy) { + case CXL_EXT_REMOVAL_POLICY_PRESCRIPTIVE: + qmp_cxl_process_dynamic_capacity_prescriptive(path, host_id, type, + region, extents, errp); + return; + default: + error_setg(errp, "Removal policy not supported"); + return; + } +} + static void ct3_class_init(ObjectClass *oc, void *data) { DeviceClass *dc = DEVICE_CLASS(oc); diff --git a/hw/mem/cxl_type3_stubs.c b/hw/mem/cxl_type3_stubs.c index 3e1851e32b..45419bbefe 100644 --- a/hw/mem/cxl_type3_stubs.c +++ b/hw/mem/cxl_type3_stubs.c @@ -67,3 +67,28 @@ void qmp_cxl_inject_correctable_error(const char *path, CxlCorErrorType type, { error_setg(errp, "CXL Type 3 support is not compiled in"); } + +void qmp_cxl_add_dynamic_capacity(const char *path, + uint16_t host_id, + CXLExtSelPolicy sel_policy, + uint8_t region, + const char *tag, + CXLDynamicCapacityExtentList *extents, + Error **errp) +{ + error_setg(errp, "CXL Type 3 support is not compiled in"); +} + +void qmp_cxl_release_dynamic_capacity(const char *path, uint16_t host_id, + CXLExtRemovalPolicy removal_policy, + bool has_forced_removal, + bool forced_removal, + bool has_sanitize_on_release, + bool sanitize_on_release, + uint8_t region, + const char *tag, + CXLDynamicCapacityExtentList *extents, + Error **errp) +{ + error_setg(errp, "CXL Type 3 support is not compiled in"); +} diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index df3511e91b..c69ff6b5de 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -443,6 +443,12 @@ typedef struct CXLDCExtent { } CXLDCExtent; typedef QTAILQ_HEAD(, CXLDCExtent) CXLDCExtentList; +typedef struct CXLDCExtentGroup { + CXLDCExtentList list; + QTAILQ_ENTRY(CXLDCExtentGroup) node; +} CXLDCExtentGroup; +typedef QTAILQ_HEAD(, CXLDCExtentGroup) CXLDCExtentGroupList; + typedef struct CXLDCRegion { uint64_t base; /* aligned to 256*MiB */ uint64_t decode_len; /* aligned to 256*MiB */ @@ -494,6 +500,7 @@ struct CXLType3Dev { */ uint64_t total_capacity; /* 256M aligned */ CXLDCExtentList extents; + CXLDCExtentGroupList extents_pending; uint32_t total_extent_count; uint32_t ext_list_gen_seq; @@ -555,4 +562,19 @@ CXLDCRegion *cxl_find_dc_region(CXLType3Dev *ct3d, uint64_t dpa, uint64_t len); void cxl_remove_extent_from_extent_list(CXLDCExtentList *list, CXLDCExtent *extent); +void cxl_insert_extent_to_extent_list(CXLDCExtentList *list, uint64_t dpa, + uint64_t len, uint8_t *tag, + uint16_t shared_seq); +bool test_any_bits_set(const unsigned long *addr, unsigned long nr, + unsigned long size); +bool cxl_extents_contains_dpa_range(CXLDCExtentList *list, + uint64_t dpa, uint64_t len); +CXLDCExtentGroup *cxl_insert_extent_to_extent_group(CXLDCExtentGroup *group, + uint64_t dpa, + uint64_t len, + uint8_t *tag, + uint16_t shared_seq); +void cxl_extent_group_list_insert_tail(CXLDCExtentGroupList *list, + CXLDCExtentGroup *group); +void cxl_extent_group_list_delete_front(CXLDCExtentGroupList *list); #endif diff --git a/include/hw/cxl/cxl_events.h b/include/hw/cxl/cxl_events.h index 5170b8dbf8..38cadaa0f3 100644 --- a/include/hw/cxl/cxl_events.h +++ b/include/hw/cxl/cxl_events.h @@ -166,4 +166,22 @@ typedef struct CXLEventMemoryModule { uint8_t reserved[0x3d]; } QEMU_PACKED CXLEventMemoryModule; +/* + * CXL r3.1 section Table 8-50: Dynamic Capacity Event Record + * All fields little endian. + */ +typedef struct CXLEventDynamicCapacity { + CXLEventRecordHdr hdr; + uint8_t type; + uint8_t validity_flags; + uint16_t host_id; + uint8_t updated_region_id; + uint8_t flags; + uint8_t reserved2[2]; + uint8_t dynamic_capacity_extent[0x28]; /* defined in cxl_device.h */ + uint8_t reserved[0x18]; + uint32_t extents_avail; + uint32_t tags_avail; +} QEMU_PACKED CXLEventDynamicCapacity; + #endif /* CXL_EVENTS_H */ diff --git a/qapi/cxl.json b/qapi/cxl.json index 4281726dec..57d9f82014 100644 --- a/qapi/cxl.json +++ b/qapi/cxl.json @@ -361,3 +361,146 @@ ## {'command': 'cxl-inject-correctable-error', 'data': {'path': 'str', 'type': 'CxlCorErrorType'}} + +## +# @CXLDynamicCapacityExtent: +# +# A single dynamic capacity extent +# +# @offset: The offset (in bytes) to the start of the region +# where the extent belongs to. +# +# @len: The length of the extent in bytes. +# +# Since: 9.1 +## +{ 'struct': 'CXLDynamicCapacityExtent', + 'data': { + 'offset':'uint64', + 'len': 'uint64' + } +} + +## +# @CXLExtSelPolicy: +# +# The policy to use for selecting which extents comprise the added +# capacity, as defined in cxl spec r3.1 Table 7-70. +# +# @free: 0h = Free +# +# @contiguous: 1h = Continuous +# +# @prescriptive: 2h = Prescriptive +# +# @enable-shared-access: 3h = Enable Shared Access +# +# Since: 9.1 +## +{ 'enum': 'CXLExtSelPolicy', + 'data': ['free', + 'contiguous', + 'prescriptive', + 'enable-shared-access'] +} + +## +# @cxl-add-dynamic-capacity: +# +# Command to initiate to add dynamic capacity extents to a host. It +# simulates operations defined in cxl spec r3.1 7.6.7.6.5. +# +# @path: CXL DCD canonical QOM path. +# +# @host-id: The "Host ID" field as defined in cxl spec r3.1 +# Table 7-70. +# +# @selection-policy: The "Selection Policy" bits as defined in +# cxl spec r3.1 Table 7-70. It specifies the policy to use for +# selecting which extents comprise the added capacity. +# +# @region: The "Region Number" field as defined in cxl spec r3.1 +# Table 7-70. The dynamic capacity region where the capacity +# is being added. Valid range is from 0-7. +# +# @tag: The "Tag" field as defined in cxl spec r3.1 Table 7-70. +# +# @extents: The "Extent List" field as defined in cxl spec r3.1 +# Table 7-70. +# +# Since : 9.1 +## +{ 'command': 'cxl-add-dynamic-capacity', + 'data': { 'path': 'str', + 'host-id': 'uint16', + 'selection-policy': 'CXLExtSelPolicy', + 'region': 'uint8', + '*tag': 'str', + 'extents': [ 'CXLDynamicCapacityExtent' ] + } +} + +## +# @CXLExtRemovalPolicy: +# +# The policy to use for selecting which extents comprise the released +# capacity, defined in the "Flags" field in cxl spec r3.1 Table 7-71. +# +# @tag-based: value = 0h. Extents are selected by the device based +# on tag, with no requirement for contiguous extents. +# +# @prescriptive: value = 1h. Extent list of capacity to release is +# included in the request payload. +# +# Since: 9.1 +## +{ 'enum': 'CXLExtRemovalPolicy', + 'data': ['tag-based', + 'prescriptive'] +} + +## +# @cxl-release-dynamic-capacity: +# +# Command to initiate to release dynamic capacity extents from a +# host. It simulates operations defined in cxl spec r3.1 7.6.7.6.6. +# +# @path: CXL DCD canonical QOM path. +# +# @host-id: The "Host ID" field as defined in cxl spec r3.1 +# Table 7-71. +# +# @removal-policy: Bit[3:0] of the "Flags" field as defined in cxl +# spec r3.1 Table 7-71. +# +# @forced-removal: Bit[4] of the "Flags" field in cxl spec r3.1 +# Table 7-71. When set, device does not wait for a Release +# Dynamic Capacity command from the host. Host immediately +# loses access to released capacity. +# +# @sanitize-on-release: Bit[5] of the "Flags" field in cxl spec r3.1 +# Table 7-71. When set, device should sanitize all released +# capacity as a result of this request. +# +# @region: The "Region Number" field as defined in cxl spec r3.1 +# Table 7-71. The dynamic capacity region where the capacity +# is being added. Valid range is from 0-7. +# +# @tag: The "Tag" field as defined in cxl spec r3.1 Table 7-71. +# +# @extents: The "Extent List" field as defined in cxl spec r3.1 +# Table 7-71. +# +# Since : 9.1 +## +{ 'command': 'cxl-release-dynamic-capacity', + 'data': { 'path': 'str', + 'host-id': 'uint16', + 'removal-policy': 'CXLExtRemovalPolicy', + '*forced-removal': 'bool', + '*sanitize-on-release': 'bool', + 'region': 'uint8', + '*tag': 'str', + 'extents': [ 'CXLDynamicCapacityExtent' ] + } +} From patchwork Thu May 23 17:44:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13672141 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0EA25C25B75 for ; Thu, 23 May 2024 17:52:21 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sACZI-0006Rt-MY; Thu, 23 May 2024 13:49:32 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sACYx-000656-3f for qemu-devel@nongnu.org; Thu, 23 May 2024 13:49:12 -0400 Received: from mail-pl1-x636.google.com ([2607:f8b0:4864:20::636]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sACYl-0003gO-8w for qemu-devel@nongnu.org; Thu, 23 May 2024 13:49:05 -0400 Received: by mail-pl1-x636.google.com with SMTP id d9443c01a7336-1f3406f225bso7000055ad.3 for ; Thu, 23 May 2024 10:48:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716486537; x=1717091337; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FdMPl+cqllxYu/DQv6hq5NibyaBbgGeK2RHmqU1o/rQ=; b=ZU5NxjT+Uqfre8vR8AI72qJCLPgXgm1ZRAEsHk3d2YXqqGMvwCFkXd6QH5jg4RoZs0 kk2CnlUqZGxCEFjRjiZcwaJTxFsL6OAHz2KvilXLB8xLPux+3cvirkgmrZ8Z1a9UsHFu lE8lRkosrgRQ/Y5eFiJ3tJ9/UIRKYGzXw5ehYe/l5EEVuQZUQJ+2khV01fYJUJfIkEz9 Fg8NTZ8ZujtydZoGONuisCnWri9XXwp4FR2zKsYQbDOFhK6+wVqEv1VIw69cfE7dttU0 tH3IfcFExdJt9QUqP4iks1PzaBLjQDp76wqLMGEku0m2EceRVGlV7PciyTxmrSYghHHh ZOFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716486537; x=1717091337; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FdMPl+cqllxYu/DQv6hq5NibyaBbgGeK2RHmqU1o/rQ=; b=HAXSFMKa21/vMYkBDrUWJKtGoTUsnhdrxZ+C5dmoRqY0LANDb1I3fRun6xhgG3WVwe DV8WZf7NqsV09kScVZFpO/+Ypt0PIzmDlIUVteoJkDP+1HGFiANUqRRKur5i7TwanWSt JhYytr+m5XQn0SwD05Q7BzxxlRqT8djaJeRasGUq4+IcasHEUPVcCUlZid53gVePQN4C 7ORPkd2K9sVBsW3k+8p7LMbzDqspCeDdGU0mxRGOW9bnlW/75NwrzyRIzTjh/CCyGlIQ jGW4sP2Vhcw1btKN1dArUPs7dlurTigvedL7JXvTprm1GpUjR3trB+VPwkqCheqvxlvj 2mGg== X-Gm-Message-State: AOJu0YyghsgKLVWg7NVG46jxNXKsrOH3bNhW7WfO656CwbXo6qt1Sa/+ 0mpibjcTsQhMAR+LVIC7QDeCLdQF/XhCeSBTQSgGg97UQAa6PRqcv01rlA== X-Google-Smtp-Source: AGHT+IHehiabqYnVUJQxYIy6yhBc7pdUHHNaYYN4bddYFtnk788KTzylMWc10q/piC8M/olLxZvAhQ== X-Received: by 2002:a17:903:2347:b0:1f2:f9de:c9e8 with SMTP id d9443c01a7336-1f4486fd296mr777535ad.2.1716486537115; Thu, 23 May 2024 10:48:57 -0700 (PDT) Received: from localhost.localdomain ([2601:646:8f03:9fee:4f9e:216e:ed5a:642d]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1ef0c137e7esm253436265ad.257.2024.05.23.10.48.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 May 2024 10:48:56 -0700 (PDT) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, armbru@redhat.com, mst@redhat.com, Fan Ni , Svetly Todorov , Jonathan Cameron Subject: [PATCH v8 12/14] hw/mem/cxl_type3: Add DPA range validation for accesses to DC regions Date: Thu, 23 May 2024 10:44:52 -0700 Message-ID: <20240523174651.1089554-13-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240523174651.1089554-1-nifan.cxl@gmail.com> References: <20240523174651.1089554-1-nifan.cxl@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::636; envelope-from=nifan.cxl@gmail.com; helo=mail-pl1-x636.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Fan Ni All DPA ranges in the DC regions are invalid to access until an extent covering the range has been successfully accepted by the host. A bitmap is added to each region to record whether a DC block in the region has been backed by a DC extent. Each bit in the bitmap represents a DC block. When a DC extent is accepted, all the bits representing the blocks in the extent are set, which will be cleared when the extent is released. Tested-by: Svetly Todorov Reviewed-by: Gregory Price Reviewed-by: Jonathan Cameron Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 3 ++ hw/mem/cxl_type3.c | 76 +++++++++++++++++++++++++++++++++++++ include/hw/cxl/cxl_device.h | 7 ++++ 3 files changed, 86 insertions(+) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index ab71492697..045bce8f74 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -1655,6 +1655,7 @@ static CXLRetCode cmd_dcd_add_dyn_cap_rsp(const struct cxl_cmd *cmd, cxl_insert_extent_to_extent_list(extent_list, dpa, len, NULL, 0); ct3d->dc.total_extent_count += 1; + ct3_set_region_block_backed(ct3d, dpa, len); } /* Remove the first extent group in the pending list */ cxl_extent_group_list_delete_front(&ct3d->dc.extents_pending); @@ -1813,10 +1814,12 @@ static CXLRetCode cmd_dcd_release_dyn_cap(const struct cxl_cmd *cmd, * list and update the extent count; */ QTAILQ_FOREACH_SAFE(ent, &ct3d->dc.extents, node, ent_next) { + ct3_clear_region_block_backed(ct3d, ent->start_dpa, ent->len); cxl_remove_extent_from_extent_list(&ct3d->dc.extents, ent); } copy_extent_list(&ct3d->dc.extents, &updated_list); QTAILQ_FOREACH_SAFE(ent, &updated_list, node, ent_next) { + ct3_set_region_block_backed(ct3d, ent->start_dpa, ent->len); cxl_remove_extent_from_extent_list(&updated_list, ent); } ct3d->dc.total_extent_count = updated_list_size; diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 2161766b14..60cbaa9bb6 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -671,6 +671,7 @@ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) .flags = 0, }; ct3d->dc.total_capacity += region->len; + region->blk_bitmap = bitmap_new(region->len / region->block_size); } QTAILQ_INIT(&ct3d->dc.extents); QTAILQ_INIT(&ct3d->dc.extents_pending); @@ -682,6 +683,8 @@ static void cxl_destroy_dc_regions(CXLType3Dev *ct3d) { CXLDCExtent *ent, *ent_next; CXLDCExtentGroup *group, *group_next; + int i; + CXLDCRegion *region; QTAILQ_FOREACH_SAFE(ent, &ct3d->dc.extents, node, ent_next) { cxl_remove_extent_from_extent_list(&ct3d->dc.extents, ent); @@ -694,6 +697,11 @@ static void cxl_destroy_dc_regions(CXLType3Dev *ct3d) } g_free(group); } + + for (i = 0; i < ct3d->dc.num_regions; i++) { + region = &ct3d->dc.regions[i]; + g_free(region->blk_bitmap); + } } static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) @@ -925,6 +933,70 @@ static void ct3_exit(PCIDevice *pci_dev) } } +/* + * Mark the DPA range [dpa, dap + len - 1] to be backed and accessible. This + * happens when a DC extent is added and accepted by the host. + */ +void ct3_set_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len) +{ + CXLDCRegion *region; + + region = cxl_find_dc_region(ct3d, dpa, len); + if (!region) { + return; + } + + bitmap_set(region->blk_bitmap, (dpa - region->base) / region->block_size, + len / region->block_size); +} + +/* + * Check whether the DPA range [dpa, dpa + len - 1] is backed with DC extents. + * Used when validating read/write to dc regions + */ +bool ct3_test_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len) +{ + CXLDCRegion *region; + uint64_t nbits; + long nr; + + region = cxl_find_dc_region(ct3d, dpa, len); + if (!region) { + return false; + } + + nr = (dpa - region->base) / region->block_size; + nbits = DIV_ROUND_UP(len, region->block_size); + /* + * if bits between [dpa, dpa + len) are all 1s, meaning the DPA range is + * backed with DC extents, return true; else return false. + */ + return find_next_zero_bit(region->blk_bitmap, nr + nbits, nr) == nr + nbits; +} + +/* + * Mark the DPA range [dpa, dap + len - 1] to be unbacked and inaccessible. + * This happens when a dc extent is released by the host. + */ +void ct3_clear_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len) +{ + CXLDCRegion *region; + uint64_t nbits; + long nr; + + region = cxl_find_dc_region(ct3d, dpa, len); + if (!region) { + return; + } + + nr = (dpa - region->base) / region->block_size; + nbits = len / region->block_size; + bitmap_clear(region->blk_bitmap, nr, nbits); +} + static bool cxl_type3_dpa(CXLType3Dev *ct3d, hwaddr host_addr, uint64_t *dpa) { int hdm_inc = R_CXL_HDM_DECODER1_BASE_LO - R_CXL_HDM_DECODER0_BASE_LO; @@ -1029,6 +1101,10 @@ static int cxl_type3_hpa_to_as_and_dpa(CXLType3Dev *ct3d, *as = &ct3d->hostpmem_as; *dpa_offset -= vmr_size; } else { + if (!ct3_test_region_block_backed(ct3d, *dpa_offset, size)) { + return -ENODEV; + } + *as = &ct3d->dc.host_dc_as; *dpa_offset -= (vmr_size + pmr_size); } diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index c69ff6b5de..0a4fcb2800 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -456,6 +456,7 @@ typedef struct CXLDCRegion { uint64_t block_size; uint32_t dsmadhandle; uint8_t flags; + unsigned long *blk_bitmap; } CXLDCRegion; struct CXLType3Dev { @@ -577,4 +578,10 @@ CXLDCExtentGroup *cxl_insert_extent_to_extent_group(CXLDCExtentGroup *group, void cxl_extent_group_list_insert_tail(CXLDCExtentGroupList *list, CXLDCExtentGroup *group); void cxl_extent_group_list_delete_front(CXLDCExtentGroupList *list); +void ct3_set_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len); +void ct3_clear_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len); +bool ct3_test_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len); #endif From patchwork Thu May 23 17:44:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13672140 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 53F65C25B75 for ; Thu, 23 May 2024 17:52:18 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sACZW-0006sZ-BE; Thu, 23 May 2024 13:49:46 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sACYx-000655-33 for qemu-devel@nongnu.org; Thu, 23 May 2024 13:49:11 -0400 Received: from mail-pl1-x636.google.com ([2607:f8b0:4864:20::636]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sACYo-0003h0-9Z for qemu-devel@nongnu.org; Thu, 23 May 2024 13:49:05 -0400 Received: by mail-pl1-x636.google.com with SMTP id d9443c01a7336-1ee954e0aa6so22523655ad.3 for ; Thu, 23 May 2024 10:48:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716486539; x=1717091339; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=D139+frhgjL58RCnIBHCYwz5taylsc59BAUfUMuPBhQ=; b=XcWFeygSeTEpnnwFWEEUUuc1K3fnBx9IcOq7dwd4pTx4tbNATnwQRq9xNGAeKE7lJ9 COXTOhUPfjbKpAGlyxpQAK/DDUhWWd7snUvfrvV1kGonxv5rN56Cnolegf3QJ5GUSX7v cRcXjcmvuB2pqPpXvlS9MhONhw7aeekhT6ogddHhgEGAoWzvNbP3+Tz/rp4Um7UjC4+e Xb9ea0G4MuqnWXNI21n9paedfYDxz0zajIXVovr2WkiNAPmAV9GS5UE9EToh9KyHgFXu Nt/IiE++iz1lxua8JSsP+0F3SXdi8q/OHwEZKfTqNTdFezQnl5nPVKT2UbeUNingRgsF VtCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716486539; x=1717091339; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=D139+frhgjL58RCnIBHCYwz5taylsc59BAUfUMuPBhQ=; b=v6cNfHvMuemFCKnw+BrRCeaSRsD7GobKKWZD2wSw3sVtY4ZRqvXbtqk1q43Y0vc9BC eGbLkAZgGZj5oMycLRHhyVuTbuBkoLctrLyvR4+cPYs+IDYEcF7uAIfTORLyAflDj8ZY dwVxyUplrmojZQ41X0gJdBJNFJSKcJPJkOniU4V4tHoDoQQz/DfoEcsoSPMvJkU+wE5O k0sI6ILqwM/JV6etg/fRA44/qloTMYHObyLm4EZM6ccn1aaJ5PaNOdxQX/3HqyicGE0M rFpOo8kGUluYWWZmJab1wJMY078ss7UK/aP+XvURwhcg2ZsJ2Wja+pibnMJLHxi6oqgL Tvag== X-Gm-Message-State: AOJu0Ywd8FspcaPg6TyodF9oZ6C7mJfcIucVlrppyJzBwetchyF2z2XM +iGTiIoMVBDWodhpJ3bpZ1xqBLQ8NvNWfyGyy3d5NwCESqYndIR76h0tOw== X-Google-Smtp-Source: AGHT+IHiNCVNxjjampo2HbEfeyUB2Cd/CBAs9YMqeP8IBwx0S3SCvP90fhFvR7g+v85vjkr5piZi6g== X-Received: by 2002:a17:902:dac7:b0:1f3:51f3:9d74 with SMTP id d9443c01a7336-1f4486bd915mr775795ad.11.1716486538793; Thu, 23 May 2024 10:48:58 -0700 (PDT) Received: from localhost.localdomain ([2601:646:8f03:9fee:4f9e:216e:ed5a:642d]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1ef0c137e7esm253436265ad.257.2024.05.23.10.48.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 May 2024 10:48:58 -0700 (PDT) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, armbru@redhat.com, mst@redhat.com, Fan Ni , Svetly Todorov Subject: [PATCH v8 13/14] hw/cxl/cxl-mailbox-utils: Add superset extent release mailbox support Date: Thu, 23 May 2024 10:44:53 -0700 Message-ID: <20240523174651.1089554-14-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240523174651.1089554-1-nifan.cxl@gmail.com> References: <20240523174651.1089554-1-nifan.cxl@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::636; envelope-from=nifan.cxl@gmail.com; helo=mail-pl1-x636.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Fan Ni With the change, we extend the extent release mailbox command processing to allow more flexible release. As long as the DPA range of the extent to release is covered by accepted extent(s) in the device, the release can be performed. Tested-by: Svetly Todorov Reviewed-by: Gregory Price Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 21 ++++++++------------- 1 file changed, 8 insertions(+), 13 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 045bce8f74..ec8949ce7b 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -1704,6 +1704,13 @@ static CXLRetCode cxl_dc_extent_release_dry_run(CXLType3Dev *ct3d, dpa = in->updated_entries[i].start_dpa; len = in->updated_entries[i].len; + /* Check if the DPA range is not fully backed with valid extents */ + if (!ct3_test_region_block_backed(ct3d, dpa, len)) { + ret = CXL_MBOX_INVALID_PA; + goto free_and_exit; + } + + /* After this point, extent overflow is the only error can happen */ while (len > 0) { QTAILQ_FOREACH(ent, updated_list, node) { range_init_nofail(&range, ent->start_dpa, ent->len); @@ -1718,14 +1725,7 @@ static CXLRetCode cxl_dc_extent_release_dry_run(CXLType3Dev *ct3d, if (range_contains(&range, dpa + len - 1)) { len2 = ent_start_dpa + ent_len - dpa - len; } else { - /* - * TODO: we reject the attempt to remove an extent - * that overlaps with multiple extents in the device - * for now. We will allow it once superset release - * support is added. - */ - ret = CXL_MBOX_INVALID_PA; - goto free_and_exit; + dpa = ent_start_dpa + ent_len; } len_done = ent_len - len1 - len2; @@ -1752,14 +1752,9 @@ static CXLRetCode cxl_dc_extent_release_dry_run(CXLType3Dev *ct3d, } len -= len_done; - /* len == 0 here until superset release is added */ break; } } - if (len) { - ret = CXL_MBOX_INVALID_PA; - goto free_and_exit; - } } } free_and_exit: From patchwork Thu May 23 17:44:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13672112 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C26B3C25B75 for ; Thu, 23 May 2024 17:50:18 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sACZJ-0006TU-AB; Thu, 23 May 2024 13:49:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sACYx-000658-5J for qemu-devel@nongnu.org; Thu, 23 May 2024 13:49:12 -0400 Received: from mail-pl1-x629.google.com ([2607:f8b0:4864:20::629]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sACYo-0003ig-Ph for qemu-devel@nongnu.org; Thu, 23 May 2024 13:49:08 -0400 Received: by mail-pl1-x629.google.com with SMTP id d9443c01a7336-1f32a3b9491so15563855ad.0 for ; Thu, 23 May 2024 10:49:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716486540; x=1717091340; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Py+IWCEA+58Br6wfdzxyAKl2e4AQZdRZm/N5DBOTD9w=; b=eh33dUjNM5vdXPrae62VbQonypdE7oR+ZSh1/RqhunUzsyyu/QQEtWnhlnbOIEgjiE WgfzEQJLGd6FhzazsPRKuKkrb0ch9GPbd5DJBCqqmMLmj1jTWtlGee9ZL4dMq3OO98Vy h0HQN9Kdij/+lBun3Ouqhl95gAuEjGIzwwYm1Oe5ZUBZB1lnJs7MkLb0H5S6EQ8AByN5 WiQ5OfvkNaEI59aTSgx8sbtKuqG1ZaYMdIDFix9zy+osvStt6dFuUDWx1ej1WOXTRe6j nulvLje2oryP7OevYUupifoqo0S2beTKTPDO+NAHkWT9jb38f92cumQsQaT5OfteCz2Y vMEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716486540; x=1717091340; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Py+IWCEA+58Br6wfdzxyAKl2e4AQZdRZm/N5DBOTD9w=; b=mAndAxNWa/nqa9bQpaLR3dAkGfoRXQgFEPUOB0a+xQrx9iqp8X/1BZmTyVa3FXEeEw TKEw3Wx4mfd8Of0k6vz32hvUbwbKCUlzl9TqTioUJZsfhrDdKMPeK8lcokpaqAtlNLGg yTf0Vv92RtpepGQvWmY4oQqzO3jE7SuCkBJGrXrfFaOXaG74l/kWNKsGv+BiOLQGZEy3 RjlMLYRwOh887SDWDcstdq2dTgzbONbSWmt5QJYoIhG/3XmhvBKOCyJlryNFGbLMf+y4 SZGDO6Uoy+gj5unZIpHgJ3BJT1AGl9aWUbsoqLZkZVs1Ty9x51DkNT8om6/rrOs7Ghgs HZVg== X-Gm-Message-State: AOJu0YwQFCNa6FGZKummYmHUj8zJqgZ00mUMQet03/RTP0ayY79qaLfH ifZnHXz8yhFziiBMVHUh0atuCJnkWmNzhjR/ObsmvYJNPlO0pRaZqliXRA== X-Google-Smtp-Source: AGHT+IGUU65s+ThjrpW2GkGrCtHEGF5MHJn9d7jgIeQXnhn8qdEYnubOYSC43+E2eDUvC9354XVJ6A== X-Received: by 2002:a17:902:f9cb:b0:1f3:4bb7:5e16 with SMTP id d9443c01a7336-1f44873387bmr600065ad.32.1716486540365; Thu, 23 May 2024 10:49:00 -0700 (PDT) Received: from localhost.localdomain ([2601:646:8f03:9fee:4f9e:216e:ed5a:642d]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1ef0c137e7esm253436265ad.257.2024.05.23.10.48.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 May 2024 10:49:00 -0700 (PDT) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, armbru@redhat.com, mst@redhat.com, Fan Ni , Svetly Todorov , Jonathan Cameron Subject: [PATCH v8 14/14] hw/mem/cxl_type3: Allow to release extent superset in QMP interface Date: Thu, 23 May 2024 10:44:54 -0700 Message-ID: <20240523174651.1089554-15-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240523174651.1089554-1-nifan.cxl@gmail.com> References: <20240523174651.1089554-1-nifan.cxl@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::629; envelope-from=nifan.cxl@gmail.com; helo=mail-pl1-x629.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Fan Ni Before the change, the QMP interface used for add/release DC extents only allows to release an extent whose DPA range is contained by a single accepted extent in the device. With the change, we relax the constraints. As long as the DPA range of the extent is covered by accepted extents, we allow the release. Tested-by: Svetly Todorov Reviewed-by: Gregory Price Reviewed-by: Jonathan Cameron Signed-off-by: Fan Ni --- hw/mem/cxl_type3.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 60cbaa9bb6..284db94182 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -1946,7 +1946,7 @@ static void qmp_cxl_process_dynamic_capacity_prescriptive(const char *path, "cannot release extent with pending DPA range"); return; } - if (!cxl_extents_contains_dpa_range(&dcd->dc.extents, dpa, len)) { + if (!ct3_test_region_block_backed(dcd, dpa, len)) { error_setg(errp, "cannot release extent with non-existing DPA range"); return;