From patchwork Fri Feb 24 19:44:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Davidlohr Bueso X-Patchwork-Id: 13151740 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A5B7C64ED8 for ; Fri, 24 Feb 2023 19:45:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229558AbjBXTpD (ORCPT ); Fri, 24 Feb 2023 14:45:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38436 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229566AbjBXTpC (ORCPT ); Fri, 24 Feb 2023 14:45:02 -0500 Received: from bee.birch.relay.mailchannels.net (bee.birch.relay.mailchannels.net [23.83.209.14]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7408E2448B for ; Fri, 24 Feb 2023 11:44:58 -0800 (PST) X-Sender-Id: dreamhost|x-authsender|dave@stgolabs.net Received: from relay.mailchannels.net (localhost [127.0.0.1]) by relay.mailchannels.net (Postfix) with ESMTP id A9FB5501335; Fri, 24 Feb 2023 19:44:57 +0000 (UTC) Received: from pdx1-sub0-mail-a250.dreamhost.com (unknown [127.0.0.6]) (Authenticated sender: dreamhost) by relay.mailchannels.net (Postfix) with ESMTPA id 06E16500A6D; Fri, 24 Feb 2023 19:44:56 +0000 (UTC) ARC-Seal: i=1; s=arc-2022; d=mailchannels.net; t=1677267897; a=rsa-sha256; cv=none; b=ThmhuLn5FNNLFsgo+R+8Kw50wDIE5bCWZXmoOtGTpVfqs1pKfCVD6z6tVAMcN27MKVsggK Gd7Fl1UZcJE7SoAF3KK4R+mu7Kd50cJB92aX0jXu5ZqWKekBihtwkM+sJXBwjCXhh2LxOw WS+HSa76wR8PIY7sQi8LQUVobweZoFErJgE+nAGf/tENrK3HfMGjueNUuMb/IP8nn5i1u+ E4eIKT2nLBqkPeICl0Wd+Zr4iadK7/idQW1PJaJZmXMo7Ko8QVJ3y+dClFcdbPu2EyOeHL CsjdsC+BjyH8YXBJzPhoRHQbI6mSz+2x1TAiXYHEKrW61g6e7JPILXzDYJ4L9Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=mailchannels.net; s=arc-2022; t=1677267897; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=g1QDvSxx1O95soxKIdw14n3lX3fGIwHtaHOnHWi/7qo=; b=LpagVnpq7xJ5PBtowmR2zZkYS6q3p3mOFbzkH0wf0bzKirW48OwnmD4FCxvLjftYfW8Jqt xnN9tGIz0/KfX7UMNTr1ostDC6WR+TwooakDT0s5aLxVFhwHkLs8ehfgfX90E3r0iduBUY iFpeF8XvVPUFw4Pizjr8dhyOZ+gFmux0vPh/sKf21ZiM/Pj1P4EVd7iEkwhWbaVVfkVds4 kxQMBxp0czEVyVEPhbEnZFnDpe7BQdSKU6ODW3OW5MrEY4BFhqX3hSJAh0BI56z5SQmQSb 0B5MBkJzB3WJP1cz68ZaZsloexArqHBSf7v5Pz7e+lnXr2bqnV1M9GyigikQYg== ARC-Authentication-Results: i=1; rspamd-9788b98bc-b7snb; auth=pass smtp.auth=dreamhost smtp.mailfrom=dave@stgolabs.net X-Sender-Id: dreamhost|x-authsender|dave@stgolabs.net X-MC-Relay: Neutral X-MailChannels-SenderId: dreamhost|x-authsender|dave@stgolabs.net X-MailChannels-Auth-Id: dreamhost X-Invention-Lyrical: 18d8d30443a89104_1677267897568_2183879968 X-MC-Loop-Signature: 1677267897568:2392973897 X-MC-Ingress-Time: 1677267897568 Received: from pdx1-sub0-mail-a250.dreamhost.com (pop.dreamhost.com [64.90.62.162]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384) by 100.97.48.87 (trex/6.7.1); Fri, 24 Feb 2023 19:44:57 +0000 Received: from offworld.. (ip72-199-50-187.sd.sd.cox.net [72.199.50.187]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: dave@stgolabs.net) by pdx1-sub0-mail-a250.dreamhost.com (Postfix) with ESMTPSA id 4PNgMq4fD8z2d; Fri, 24 Feb 2023 11:44:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stgolabs.net; s=dreamhost; t=1677267896; bh=g1QDvSxx1O95soxKIdw14n3lX3fGIwHtaHOnHWi/7qo=; h=From:To:Cc:Subject:Date:Content-Transfer-Encoding; b=lttsjI5mHytvQhv8BDAnuhFD3zdPHvx8sJfGhMYV/hdiLAPDlL6jtViMJffh8Pbae Kt+DCtMcVNB9yakAb1ZXssOFow/Yx6Wb5TI94kof0vxhAlwSczT7tpIBhYvc0cGZBE AlZC4yO221WXi/VJBEoVyhuHO2OYCb+8bCMimGEZH0X5uv94QA2OsBmjyuDKA4VpYj cddg8PGjhG8WcbrhUwnWz80jE7T+pH/PKfx2OL3xOKHRqMeE6SdxAT1mJhSCjGLNBA a8nykVRIg5DIyKYFHcBGURMnU39FqQm8WRNX65LndVWzRUMFj35OwnP9cwGaZ6Q7CL 7ddtZxVuEuLGg== From: Davidlohr Bueso To: jonathan.cameron@huawei.com Cc: dan.j.williams@intel.com, ira.weiny@intel.com, fan.ni@samsung.com, a.manzanares@samsung.com, linux-cxl@vger.kernel.org, dave@stgolabs.net Subject: [PATCH 1/3] cxl/mbox: Add support for background operations Date: Fri, 24 Feb 2023 11:44:41 -0800 Message-Id: <20230224194443.1990440-2-dave@stgolabs.net> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230224194443.1990440-1-dave@stgolabs.net> References: <20230224194443.1990440-1-dave@stgolabs.net> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org Support background commands in the mailbox, and update cmd_infostat_bg_op_sts() accordingly. This patch does not implement mbox interrupts upon completion, so the kernel driver must rely on polling to know when the operation is done. Signed-off-by: Davidlohr Bueso --- hw/cxl/cxl-device-utils.c | 5 +- hw/cxl/cxl-mailbox-utils.c | 103 ++++++++++++++++++++++++++++++++++-- include/hw/cxl/cxl_device.h | 10 ++++ 3 files changed, 111 insertions(+), 7 deletions(-) diff --git a/hw/cxl/cxl-device-utils.c b/hw/cxl/cxl-device-utils.c index 50c76c65e755..4bb4e85dae19 100644 --- a/hw/cxl/cxl-device-utils.c +++ b/hw/cxl/cxl-device-utils.c @@ -101,8 +101,7 @@ static void mailbox_mem_writeq(uint64_t *reg_state, hwaddr offset, case A_CXL_DEV_MAILBOX_CMD: break; case A_CXL_DEV_BG_CMD_STS: - /* BG not supported */ - /* fallthrough */ + break; case A_CXL_DEV_MAILBOX_STS: /* Read only register, will get updated by the state machine */ return; @@ -273,7 +272,7 @@ static void device_reg_init_common(CXLDeviceState *cxl_dstate) static void mailbox_reg_init_common(CXLDeviceState *cxl_dstate) { - /* 2048 payload size, with no interrupt or background support */ + /* 2048 payload size, with no interrupt */ ARRAY_FIELD_DP32(cxl_dstate->mbox_reg_state32, CXL_DEV_MAILBOX_CAP, PAYLOAD_SIZE, CXL_MAILBOX_PAYLOAD_SHIFT); cxl_dstate->payload_size = CXL_MAILBOX_MAX_PAYLOAD_SIZE; diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index b02908c4a4ba..82923bb84eb0 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -350,7 +350,14 @@ static CXLRetCode cmd_infostat_bg_op_sts(struct cxl_cmd *cmd, bg_op_status = (void *)cmd->payload; memset(bg_op_status, 0, sizeof(*bg_op_status)); - /* No support yet for background operations so status all 0 */ + bg_op_status->status = ARRAY_FIELD_EX64(cxl_dstate->mbox_reg_state64, + CXL_DEV_BG_CMD_STS, PERCENTAGE_COMP) << 1; + if (cxl_dstate->bg.runtime > 0) { + bg_op_status->status |= 1U << 0; + } + bg_op_status->opcode = cxl_dstate->bg.opcode; + bg_op_status->returncode = ARRAY_FIELD_EX64(cxl_dstate->mbox_reg_state64, + CXL_DEV_BG_CMD_STS, RET_CODE); *len = sizeof(*bg_op_status); return CXL_MBOX_SUCCESS; } @@ -808,6 +815,8 @@ static CXLRetCode cmd_media_clear_poison(struct cxl_cmd *cmd, #define IMMEDIATE_DATA_CHANGE (1 << 2) #define IMMEDIATE_POLICY_CHANGE (1 << 3) #define IMMEDIATE_LOG_CHANGE (1 << 4) +#define SECURITY_STATE_CHANGE (1 << 5) +#define BACKGROUND_OPERATION (1 << 6) static struct cxl_cmd cxl_cmd_set[256][256] = { [EVENTS][GET_RECORDS] = { "EVENTS_GET_RECORDS", @@ -856,12 +865,20 @@ static struct cxl_cmd cxl_cmd_set_sw[256][256] = { cmd_identify_switch_device, 0, 0x49 }, }; +/* + * While the command is executing in the background, the device should + * update the percentage complete in the Background Command Status Register + * at least once per second. + */ +#define CXL_MBOX_BG_UPDATE_FREQ 1000UL + void cxl_process_mailbox(CXLDeviceState *cxl_dstate) { uint16_t ret = CXL_MBOX_SUCCESS; struct cxl_cmd *cxl_cmd; - uint64_t status_reg; + uint64_t status_reg = 0; opcode_handler h; + uint8_t bg_started = 0; uint64_t command_reg = cxl_dstate->mbox_reg_state64[R_CXL_DEV_MAILBOX_CMD]; uint8_t set = FIELD_EX64(command_reg, CXL_DEV_MAILBOX_CMD, COMMAND_SET); @@ -873,7 +890,17 @@ void cxl_process_mailbox(CXLDeviceState *cxl_dstate) if (len == cxl_cmd->in || cxl_cmd->in == ~0) { cxl_cmd->payload = cxl_dstate->mbox_reg_state + A_CXL_DEV_CMD_PAYLOAD; + /* Only one bg command at a time */ + if ((cxl_cmd->effect & BACKGROUND_OPERATION) && + cxl_dstate->bg.runtime > 0) { + ret = CXL_MBOX_BUSY; + goto done; + } ret = (*h)(cxl_cmd, cxl_dstate, &len); + if ((cxl_cmd->effect & BACKGROUND_OPERATION) && + ret == CXL_MBOX_BG_STARTED) { + bg_started = 1; + } assert(len <= cxl_dstate->payload_size); } else { ret = CXL_MBOX_INVALID_PAYLOAD_LENGTH; @@ -884,8 +911,12 @@ void cxl_process_mailbox(CXLDeviceState *cxl_dstate) ret = CXL_MBOX_UNSUPPORTED; } - /* Set the return code */ - status_reg = FIELD_DP64(0, CXL_DEV_MAILBOX_STS, ERRNO, ret); +done: + /* Set bg and the return code */ + if (bg_started) { + status_reg = FIELD_DP64(0, CXL_DEV_MAILBOX_STS, BG_OP, bg_started); + } + status_reg = FIELD_DP64(status_reg, CXL_DEV_MAILBOX_STS, ERRNO, ret); /* Set the return length */ command_reg = FIELD_DP64(command_reg, CXL_DEV_MAILBOX_CMD, COMMAND_SET, 0); @@ -895,11 +926,70 @@ void cxl_process_mailbox(CXLDeviceState *cxl_dstate) cxl_dstate->mbox_reg_state64[R_CXL_DEV_MAILBOX_CMD] = command_reg; cxl_dstate->mbox_reg_state64[R_CXL_DEV_MAILBOX_STS] = status_reg; + if (bg_started) { + uint64_t bg_status_reg, now; + + cxl_dstate->bg.opcode = (set << 8) | cmd; + + bg_status_reg = FIELD_DP64(0, CXL_DEV_BG_CMD_STS, OP, cxl_dstate->bg.opcode); + bg_status_reg = FIELD_DP64(bg_status_reg, CXL_DEV_BG_CMD_STS, + PERCENTAGE_COMP, 0); + cxl_dstate->mbox_reg_state64[R_CXL_DEV_BG_CMD_STS] = bg_status_reg; + + now = qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL); + cxl_dstate->bg.starttime = now; + timer_mod(cxl_dstate->bg.timer, now + CXL_MBOX_BG_UPDATE_FREQ); + } + /* Tell the host we're done */ ARRAY_FIELD_DP32(cxl_dstate->mbox_reg_state32, CXL_DEV_MAILBOX_CTRL, DOORBELL, 0); } +static void bg_timercb(void *opaque) +{ + CXLDeviceState *cxl_dstate = opaque; + uint64_t bg_status_reg = 0; + uint64_t now = qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL); + uint64_t total_time = cxl_dstate->bg.starttime + cxl_dstate->bg.runtime; + + assert(cxl_dstate->bg.runtime > 0); + bg_status_reg = FIELD_DP64(bg_status_reg, CXL_DEV_BG_CMD_STS, + OP, cxl_dstate->bg.opcode); + + if (now >= total_time) { /* we are done */ + uint64_t status_reg; + uint16_t ret = CXL_MBOX_SUCCESS; + + cxl_dstate->bg.complete_pct = 100; + /* Clear bg */ + status_reg = FIELD_DP64(0, CXL_DEV_MAILBOX_STS, BG_OP, 0); + cxl_dstate->mbox_reg_state64[R_CXL_DEV_MAILBOX_STS] = status_reg; + + bg_status_reg = FIELD_DP64(bg_status_reg, CXL_DEV_BG_CMD_STS, RET_CODE, ret); + + /* TODO add ad-hoc cmd succesful completion handling */ + + qemu_log("Background command %04xh finished: %s\n", + cxl_dstate->bg.opcode, + ret == CXL_MBOX_SUCCESS ? "success" : "aborted"); + } else { + /* estimate only */ + cxl_dstate->bg.complete_pct = 100 * now / total_time; + timer_mod(cxl_dstate->bg.timer, now + CXL_MBOX_BG_UPDATE_FREQ); + } + + bg_status_reg = FIELD_DP64(bg_status_reg, CXL_DEV_BG_CMD_STS, PERCENTAGE_COMP, + cxl_dstate->bg.complete_pct); + cxl_dstate->mbox_reg_state64[R_CXL_DEV_BG_CMD_STS] = bg_status_reg; + + if (cxl_dstate->bg.complete_pct == 100) { + cxl_dstate->bg.starttime = 0; + /* registers are updated, allow new bg-capable cmds */ + cxl_dstate->bg.runtime = 0; + } +} + void cxl_initialize_mailbox(CXLDeviceState *cxl_dstate, bool switch_cci) { if (!switch_cci) { @@ -920,4 +1010,9 @@ void cxl_initialize_mailbox(CXLDeviceState *cxl_dstate, bool switch_cci) } } } + cxl_dstate->bg.complete_pct = 0; + cxl_dstate->bg.starttime = 0; + cxl_dstate->bg.runtime = 0; + cxl_dstate->bg.timer = timer_new_ms(QEMU_CLOCK_VIRTUAL, + bg_timercb, cxl_dstate); } diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index 1fa8522d33fa..dbb8a955723b 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -216,6 +216,16 @@ typedef struct cxl_device_state { struct cxl_cmd (*cxl_cmd_set)[256]; CPMUState cpmu[CXL_NUM_CPMU_INSTANCES]; CXLEventLog event_logs[CXL_EVENT_TYPE_MAX]; + + /* background command handling (times in ms) */ + struct { + uint16_t opcode; + uint16_t complete_pct; + uint64_t starttime; + /* set by each bg cmd, cleared by the bg_timer when complete */ + uint64_t runtime; + QEMUTimer *timer; + } bg; } CXLDeviceState; /* Initialize the register block for a device */ From patchwork Fri Feb 24 19:44:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Davidlohr Bueso X-Patchwork-Id: 13151741 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DEF71C6FA8E for ; Fri, 24 Feb 2023 19:45:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229578AbjBXTpE (ORCPT ); Fri, 24 Feb 2023 14:45:04 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38444 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229524AbjBXTpD (ORCPT ); Fri, 24 Feb 2023 14:45:03 -0500 Received: from bird.elm.relay.mailchannels.net (bird.elm.relay.mailchannels.net [23.83.212.17]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D026911168 for ; Fri, 24 Feb 2023 11:44:59 -0800 (PST) X-Sender-Id: dreamhost|x-authsender|dave@stgolabs.net Received: from relay.mailchannels.net (localhost [127.0.0.1]) by relay.mailchannels.net (Postfix) with ESMTP id D7A268C0E83; Fri, 24 Feb 2023 19:44:58 +0000 (UTC) Received: from pdx1-sub0-mail-a250.dreamhost.com (unknown [127.0.0.6]) (Authenticated sender: dreamhost) by relay.mailchannels.net (Postfix) with ESMTPA id 4C1368C1127; Fri, 24 Feb 2023 19:44:58 +0000 (UTC) ARC-Seal: i=1; s=arc-2022; d=mailchannels.net; t=1677267898; a=rsa-sha256; cv=none; b=pkZTGyR24/5cbbUKPcAzR5x2iJmICgclpw42lUeNhh6idMX/iZh+uAcDMJKXsNI9E9YNHw GZzDNLsb/XXiL9azCcjttR9CRy5x48vx6VnKuyQKuIkWLCoUrbWRm146x8umJi9lShiPC2 QEo82CSMoScjyvFWKrq+MmYxXzk4vrUW1sAuLFOCIiMr6/n4iLSQJzYzQnws+C+p9CCcLM iRp8KpsMypUK9rDpvGCcIJBTF0liV7DNv4MKW/v5lHHGEkhQ178X3IMCaGB3lLK2y2rp9J MYXYypjraDV7an972EQVUWuqIVMaiGhE+0hlnA/P9OPDuKBuxl2LRMz7ounYdg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=mailchannels.net; s=arc-2022; t=1677267898; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ikC1LO3vxcDXQf+xSuIkYwrqnOjS6KA7A32KL9FtDxY=; b=1DgPm7tWWKBpo12TBI67zgFHfHHaKmKcqHFalSJIQYpVLvAs0VHkbp/OQ6RiAkcp0DX2kq MwHSv0XcmiApz1W8aNfHQ7qCDDUgxrs7YxET9Q5A/7u5sq2Kpy7lt/86YgU0/bUzwKWjrz AmT3jhoVe87SSyIyVvrvt+Txv7czOKfYoZgi28jhxoXeqwyqaA9rG+SMrAVL2UFlUwxwsO osQ0XeFcpZb1Kb17FuZ9pHR5VZQZBlkG/wxkqnKg6CqpfAG2TyYwIKQs1YRTWbzGR5nfX8 IF+S7m+ivJs6UaWTOsWFWLzyighgmyYP6i6KC41iGEDUZHLBujUHPv0c66pD6A== ARC-Authentication-Results: i=1; rspamd-9788b98bc-x62tj; auth=pass smtp.auth=dreamhost smtp.mailfrom=dave@stgolabs.net X-Sender-Id: dreamhost|x-authsender|dave@stgolabs.net X-MC-Relay: Neutral X-MailChannels-SenderId: dreamhost|x-authsender|dave@stgolabs.net X-MailChannels-Auth-Id: dreamhost X-Soft-Scare: 26b13fdb5407b346_1677267898683_3042248368 X-MC-Loop-Signature: 1677267898683:708471927 X-MC-Ingress-Time: 1677267898682 Received: from pdx1-sub0-mail-a250.dreamhost.com (pop.dreamhost.com [64.90.62.162]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384) by 100.123.200.110 (trex/6.7.1); Fri, 24 Feb 2023 19:44:58 +0000 Received: from offworld.. (ip72-199-50-187.sd.sd.cox.net [72.199.50.187]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: dave@stgolabs.net) by pdx1-sub0-mail-a250.dreamhost.com (Postfix) with ESMTPSA id 4PNgMr6nm0z2Q; Fri, 24 Feb 2023 11:44:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stgolabs.net; s=dreamhost; t=1677267897; bh=ikC1LO3vxcDXQf+xSuIkYwrqnOjS6KA7A32KL9FtDxY=; h=From:To:Cc:Subject:Date:Content-Transfer-Encoding; b=cnLysW/8FM1AwIdUiuJxq2CAcT+V/c2OFbqS8fgKrPZy35FuD7ACLC4Oz48PZBJbD 9QmoFaliOim86lyhKFV65r9RKkbJ1sqar/6V7O2PIjAVBLWVHwW3PBdo+zxOzLIrpe mnUKSupPoVex65n62XRYS9pdcA+8zHQlDKWKKvY1Lu9sNTvp0a0CTIcfAGdY+igy8W XFdk4d4dv7VZfXyX8aS/H4aA1SElnZ12nERnHgfnwPPg8KuxNzaZPVhRk8VxtHlppb 7H5XG96EKBgQZ9uKhNwGPo4ooUyeXj5GTp/qAxeIALPDyTcCXv427jJMhXBPl/osJm ERziYakY/7bXw== From: Davidlohr Bueso To: jonathan.cameron@huawei.com Cc: dan.j.williams@intel.com, ira.weiny@intel.com, fan.ni@samsung.com, a.manzanares@samsung.com, linux-cxl@vger.kernel.org, dave@stgolabs.net Subject: [PATCH 2/3] cxl/mbox: Wire up interrupts for background completion Date: Fri, 24 Feb 2023 11:44:42 -0800 Message-Id: <20230224194443.1990440-3-dave@stgolabs.net> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230224194443.1990440-1-dave@stgolabs.net> References: <20230224194443.1990440-1-dave@stgolabs.net> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org Notify when the background operation is done. Signed-off-by: Davidlohr Bueso --- hw/cxl/cxl-device-utils.c | 10 +++++++++- hw/cxl/cxl-mailbox-utils.c | 11 +++++++++++ include/hw/cxl/cxl_device.h | 1 + 3 files changed, 21 insertions(+), 1 deletion(-) diff --git a/hw/cxl/cxl-device-utils.c b/hw/cxl/cxl-device-utils.c index 4bb4e85dae19..a4a2c6a80004 100644 --- a/hw/cxl/cxl-device-utils.c +++ b/hw/cxl/cxl-device-utils.c @@ -272,10 +272,18 @@ static void device_reg_init_common(CXLDeviceState *cxl_dstate) static void mailbox_reg_init_common(CXLDeviceState *cxl_dstate) { - /* 2048 payload size, with no interrupt */ + const uint8_t msi_n = 9; + + /* 2048 payload size */ ARRAY_FIELD_DP32(cxl_dstate->mbox_reg_state32, CXL_DEV_MAILBOX_CAP, PAYLOAD_SIZE, CXL_MAILBOX_PAYLOAD_SHIFT); cxl_dstate->payload_size = CXL_MAILBOX_MAX_PAYLOAD_SIZE; + /* irq support */ + ARRAY_FIELD_DP32(cxl_dstate->mbox_reg_state32, CXL_DEV_MAILBOX_CAP, + BG_INT_CAP, 1); + ARRAY_FIELD_DP32(cxl_dstate->mbox_reg_state32, CXL_DEV_MAILBOX_CAP, + MSI_N, msi_n); + cxl_dstate->mbox_msi_n = msi_n; } static void memdev_reg_init_common(CXLDeviceState *cxl_dstate) { } diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 82923bb84eb0..61f0b8d675bc 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -8,6 +8,8 @@ */ #include "qemu/osdep.h" +#include "hw/pci/msi.h" +#include "hw/pci/msix.h" #include "hw/cxl/cxl.h" #include "hw/cxl/cxl_events.h" #include "hw/pci/pci.h" @@ -984,9 +986,18 @@ static void bg_timercb(void *opaque) cxl_dstate->mbox_reg_state64[R_CXL_DEV_BG_CMD_STS] = bg_status_reg; if (cxl_dstate->bg.complete_pct == 100) { + CXLType3Dev *ct3d = container_of(cxl_dstate, CXLType3Dev, cxl_dstate); + PCIDevice *pdev = &ct3d->parent_obj; + cxl_dstate->bg.starttime = 0; /* registers are updated, allow new bg-capable cmds */ cxl_dstate->bg.runtime = 0; + + if (msix_enabled(pdev)) { + msix_notify(pdev, cxl_dstate->mbox_msi_n); + } else if (msi_enabled(pdev)) { + msi_notify(pdev, cxl_dstate->mbox_msi_n); + } } } diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index dbb8a955723b..f986651b6ead 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -189,6 +189,7 @@ typedef struct cxl_device_state { struct { MemoryRegion mailbox; uint16_t payload_size; + uint8_t mbox_msi_n; union { uint8_t mbox_reg_state[CXL_MAILBOX_REGISTERS_LENGTH]; uint16_t mbox_reg_state16[CXL_MAILBOX_REGISTERS_LENGTH / 2]; From patchwork Fri Feb 24 19:44:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Davidlohr Bueso X-Patchwork-Id: 13151742 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC8AAC7EE23 for ; Fri, 24 Feb 2023 19:45:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229503AbjBXTpE (ORCPT ); Fri, 24 Feb 2023 14:45:04 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38450 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229566AbjBXTpD (ORCPT ); Fri, 24 Feb 2023 14:45:03 -0500 Received: from bird.elm.relay.mailchannels.net (bird.elm.relay.mailchannels.net [23.83.212.17]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CB3691EBF1 for ; Fri, 24 Feb 2023 11:45:00 -0800 (PST) X-Sender-Id: dreamhost|x-authsender|dave@stgolabs.net Received: from relay.mailchannels.net (localhost [127.0.0.1]) by relay.mailchannels.net (Postfix) with ESMTP id 0ECB7922134; Fri, 24 Feb 2023 19:45:00 +0000 (UTC) Received: from pdx1-sub0-mail-a250.dreamhost.com (unknown [127.0.0.6]) (Authenticated sender: dreamhost) by relay.mailchannels.net (Postfix) with ESMTPA id 80271921A4A; Fri, 24 Feb 2023 19:44:59 +0000 (UTC) ARC-Seal: i=1; s=arc-2022; d=mailchannels.net; t=1677267899; a=rsa-sha256; cv=none; b=hQONZwvkUxJtdjQodN+KnM4smVkxumNOSrXf5ktgYOd0BPuFh4gaf4wN54Y2A4guLY+WCa p9Qg7umf7Pnj2rTAdTS0ZnRgAzWcL+YKVpJuWCrJaRRzNkteIGAcwGGvD5DlaIGQpQ9CMM KK1QD9+DgBMtWw4bxck+TVvrcdu4k7tAvzySv+MoclrQyNfAAuBX9eq/poZauabUnNQgTZ VVYGieKh5au3QmhXn8LpooktMzWJJTabg3T+ISfrnqvGnp56wcIrxG+nasAcXxOhszkE7A 6Rlp8NF20z6pF352vU8tUwrH51WgMoCuFyi/Ap1+xFdm/374D6RQfK7qKnZ9iw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=mailchannels.net; s=arc-2022; t=1677267899; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4jYSbd7KX24tHHYK5VIK7MYvqt3YDHpwXDgcJjgDjg8=; b=e87832d3qAMmhYgMziNP2+Jy4Uv4V/yUSeoYeTdsLmAdntaQqiBAByQVOVoqNVBM2V8Mbw 3dxBRsIKkWNw/mLrpj6gT0WWr2fulkm0Q6XkfAdPqaIcnMwwvtof6ZYF4oJyavk7tdHQ4a AFFZc1JjpPPea0+ixJEM7bLQgzpEIwejVgV7TjMuGS4RshDRu+C8POvi8wTsXblYLm3XyR gbUyhHRrfrYdZ3+NxqKXMFReAw1dqcoVvyejVgPC8+UAwif1XdwMnVio6UbWHet7poP/tF uTGGVQMhWU7BJzTgok1V31LrNmvkI7sJq8NGW+OL0Ipd4qGZVqu9XAzEfYsAFA== ARC-Authentication-Results: i=1; rspamd-9788b98bc-l9rxh; auth=pass smtp.auth=dreamhost smtp.mailfrom=dave@stgolabs.net X-Sender-Id: dreamhost|x-authsender|dave@stgolabs.net X-MC-Relay: Neutral X-MailChannels-SenderId: dreamhost|x-authsender|dave@stgolabs.net X-MailChannels-Auth-Id: dreamhost X-Ski-Interest: 328ddc1a593d27b2_1677267899893_1478116902 X-MC-Loop-Signature: 1677267899893:856376867 X-MC-Ingress-Time: 1677267899893 Received: from pdx1-sub0-mail-a250.dreamhost.com (pop.dreamhost.com [64.90.62.162]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384) by 100.116.179.67 (trex/6.7.1); Fri, 24 Feb 2023 19:44:59 +0000 Received: from offworld.. (ip72-199-50-187.sd.sd.cox.net [72.199.50.187]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: dave@stgolabs.net) by pdx1-sub0-mail-a250.dreamhost.com (Postfix) with ESMTPSA id 4PNgMt225MzMP; Fri, 24 Feb 2023 11:44:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stgolabs.net; s=dreamhost; t=1677267899; bh=4jYSbd7KX24tHHYK5VIK7MYvqt3YDHpwXDgcJjgDjg8=; h=From:To:Cc:Subject:Date:Content-Type:Content-Transfer-Encoding; b=nkKIT0zb/PNTuyw/TzpfurjFKr4vG0vIakzrCDUULGI1QxH1fjcfaAOMUqdzJyzMb tBQgWi7Gzf7Hn69YZqe5De4qiONO/ZFcIRc/ogVLsWTuvmfXcPKLaIl2xotAvLwUaY ChywSSlXTv++T/87cQAcb/z6CJ9uTOxkXalYJvcZDeIMxRlmG92ciSYPHN++uNICa/ pPA1juLmpd8xK9NzAuZv5nRAv1Slp48FQtQtz63NH/GB9oPgOyXh93OXM6G+fUdhTb qZx3cOjHDaWdLDyCxHhEAHgoKdxN10Eric0NEtojLspvgoFDhTPTQyru2ddE4z8QGP ICZfrO/v/bCEg== From: Davidlohr Bueso To: jonathan.cameron@huawei.com Cc: dan.j.williams@intel.com, ira.weiny@intel.com, fan.ni@samsung.com, a.manzanares@samsung.com, linux-cxl@vger.kernel.org, dave@stgolabs.net Subject: [PATCH 3/3] cxl: Add support for device sanitation Date: Fri, 24 Feb 2023 11:44:43 -0800 Message-Id: <20230224194443.1990440-4-dave@stgolabs.net> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230224194443.1990440-1-dave@stgolabs.net> References: <20230224194443.1990440-1-dave@stgolabs.net> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org Make use of the background operations through the sanitize command, per CXL 3.0 specs. Traditionally run times can be rather long, depending on the size of the media. Estimate times based on: https://pmem.io/documents/NVDIMM_DSM_Interface-V1.8.pdf Signed-off-by: Davidlohr Bueso --- hw/cxl/cxl-mailbox-utils.c | 139 +++++++++++++++++++++++++++++++++++- hw/mem/cxl_type3.c | 9 ++- include/hw/cxl/cxl_device.h | 17 +++++ 3 files changed, 162 insertions(+), 3 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 61f0b8d675bc..aa0641f786e2 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -18,6 +18,7 @@ #include "qemu/log.h" #include "qemu/units.h" #include "qemu/uuid.h" +#include "sysemu/hostmem.h" #define CXL_CAPACITY_MULTIPLIER (256 * MiB) @@ -71,6 +72,9 @@ enum { #define GET_PARTITION_INFO 0x0 #define GET_LSA 0x2 #define SET_LSA 0x3 + SANITIZE = 0x44, + #define OVERWRITE 0x0 + #define SECURE_ERASE 0x1 MEDIA_AND_POISON = 0x43, #define GET_POISON_LIST 0x0 #define INJECT_POISON 0x1 @@ -626,6 +630,109 @@ static CXLRetCode cmd_ccls_set_lsa(struct cxl_cmd *cmd, return CXL_MBOX_SUCCESS; } +/* Perform the actual device zeroing */ +static void __do_sanitization(CXLDeviceState *cxl_dstate) +{ + MemoryRegion *mr; + CXLType3Dev *ct3d = container_of(cxl_dstate, CXLType3Dev, cxl_dstate); + + if (ct3d->hostvmem) { + mr = host_memory_backend_get_memory(ct3d->hostvmem); + if (mr) { + void *hostmem = memory_region_get_ram_ptr(mr); + memset(hostmem, 0, memory_region_size(mr)); + } + } + + if (ct3d->hostpmem) { + mr = host_memory_backend_get_memory(ct3d->hostpmem); + if (mr) { + void *hostmem = memory_region_get_ram_ptr(mr); + memset(hostmem, 0, memory_region_size(mr)); + } + } + if (ct3d->lsa) { + mr = host_memory_backend_get_memory(ct3d->lsa); + if (mr) { + void *lsa = memory_region_get_ram_ptr(mr); + memset(lsa, 0, memory_region_size(mr)); + } + } +} + +/* + * CXL 3.0 spec section 8.2.9.8.5.1 - Sanitize. + * + * Once the Sanitize command has started successfully, the device shall be + * placed in the media disabled state. If the command fails or is interrupted + * by a reset or power failure, it shall remain in the media disabled state + * until a successful Sanitize command has been completed. During this state: + * + * 1. Memory writes to the device will have no effect, and all memory reads + * will return random values (no user data returned, even for locations that + * the failed Sanitize operation didn’t sanitize yet). + * + * 2. Mailbox commands shall still be processed in the disabled state, except + * that commands that access Sanitized areas shall fail with the Media Disabled + * error code. + */ +static CXLRetCode cmd_sanitize_overwrite(struct cxl_cmd *cmd, + CXLDeviceState *cxl_dstate, + uint16_t *len) +{ + uint64_t total_mem; /* in Mb */ + int secs; + + total_mem = (cxl_dstate->vmem_size + cxl_dstate->pmem_size) >> 20; + if (total_mem <= 512) { + secs = 4; + } else if (total_mem <= 1024) { + secs = 8; + } else if (total_mem <= 2 * 1024) { + secs = 15; + } else if (total_mem <= 4 * 1024) { + secs = 30; + } else if (total_mem <= 8 * 1024) { + secs = 60; + } else if (total_mem <= 16 * 1024) { + secs = 2 * 60; + } else if (total_mem <= 32 * 1024) { + secs = 4 * 60; + } else if (total_mem <= 64 * 1024) { + secs = 8 * 60; + } else if (total_mem <= 128 * 1024) { + secs = 15 * 60; + } else if (total_mem <= 256 * 1024) { + secs = 30 * 60; + } else if (total_mem <= 512 * 1024) { + secs = 60 * 60; + } else if (total_mem <= 1024 * 1024) { + secs = 120 * 60; + } else { + secs = 240 * 60; /* max 4 hrs */ + } + + /* EBUSY other bg cmds as of now */ + cxl_dstate->bg.runtime = secs * 1000UL; + *len = 0; + + qemu_log_mask(LOG_UNIMP, + "Sanitize/overwrite command runtime for %ldMb media: %d seconds\n", + total_mem, secs); + + cxl_dev_disable_media(cxl_dstate); + + if (secs > 2) { + /* sanitize when done */ + return CXL_MBOX_BG_STARTED; + } else { + __do_sanitization(cxl_dstate); + cxl_dev_enable_media(cxl_dstate); + + return CXL_MBOX_SUCCESS; + } +} + /* * This is very inefficient, but good enough for now! * Also the payload will always fit, so no need to handle the MORE flag and @@ -843,6 +950,8 @@ static struct cxl_cmd cxl_cmd_set[256][256] = { [CCLS][GET_LSA] = { "CCLS_GET_LSA", cmd_ccls_get_lsa, 8, 0 }, [CCLS][SET_LSA] = { "CCLS_SET_LSA", cmd_ccls_set_lsa, ~0, IMMEDIATE_CONFIG_CHANGE | IMMEDIATE_DATA_CHANGE }, + [SANITIZE][OVERWRITE] = { "SANITIZE_OVERWRITE", cmd_sanitize_overwrite, + 0, IMMEDIATE_DATA_CHANGE | SECURITY_STATE_CHANGE | BACKGROUND_OPERATION }, [MEDIA_AND_POISON][GET_POISON_LIST] = { "MEDIA_AND_POISON_GET_POISON_LIST", cmd_media_get_poison_list, 16, 0 }, [MEDIA_AND_POISON][INJECT_POISON] = { "MEDIA_AND_POISON_INJECT_POISON", @@ -898,6 +1007,21 @@ void cxl_process_mailbox(CXLDeviceState *cxl_dstate) ret = CXL_MBOX_BUSY; goto done; } + /* forbid any selected commands while overwriting */ + if (sanitize_running(cxl_dstate)) { + if (h == cmd_events_get_records || + h == cmd_ccls_get_partition_info || + h == cmd_ccls_set_lsa || + h == cmd_ccls_get_lsa || + h == cmd_logs_get_log || + h == cmd_media_get_poison_list || + h == cmd_media_inject_poison || + h == cmd_media_clear_poison || + h == cmd_sanitize_overwrite) { + ret = CXL_MBOX_MEDIA_DISABLED; + goto done; + } + } ret = (*h)(cxl_cmd, cxl_dstate, &len); if ((cxl_cmd->effect & BACKGROUND_OPERATION) && ret == CXL_MBOX_BG_STARTED) { @@ -970,8 +1094,19 @@ static void bg_timercb(void *opaque) bg_status_reg = FIELD_DP64(bg_status_reg, CXL_DEV_BG_CMD_STS, RET_CODE, ret); - /* TODO add ad-hoc cmd succesful completion handling */ - + if (ret == CXL_MBOX_SUCCESS) { + switch (cxl_dstate->bg.opcode) { + case 0x4400: /* sanitize */ + __do_sanitization(cxl_dstate); + cxl_dev_enable_media(cxl_dstate); + break; + case 0x4304: /* TODO: scan media */ + break; + default: + __builtin_unreachable(); + break; + } + } qemu_log("Background command %04xh finished: %s\n", cxl_dstate->bg.opcode, ret == CXL_MBOX_SUCCESS ? "success" : "aborted"); diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 334ce92f5e0f..2b483d3d8ea9 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -12,6 +12,7 @@ #include "qemu/pmem.h" #include "qemu/range.h" #include "qemu/rcu.h" +#include "qemu/guest-random.h" #include "sysemu/hostmem.h" #include "sysemu/numa.h" #include "hw/cxl/cxl.h" @@ -988,6 +989,10 @@ MemTxResult cxl_type3_read(PCIDevice *d, hwaddr host_addr, uint64_t *data, return MEMTX_ERROR; } + if (sanitize_running(&CXL_TYPE3(d)->cxl_dstate)) { + qemu_guest_getrandom_nofail(data, size); + return MEMTX_OK; + } return address_space_read(as, dpa_offset, attrs, data, size); } @@ -1003,7 +1008,9 @@ MemTxResult cxl_type3_write(PCIDevice *d, hwaddr host_addr, uint64_t data, if (res) { return MEMTX_ERROR; } - + if (sanitize_running(&CXL_TYPE3(d)->cxl_dstate)) { + return MEMTX_OK; + } return address_space_write(as, dpa_offset, attrs, &data, size); } diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index f986651b6ead..e28536969397 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -346,6 +346,23 @@ REG64(CXL_MEM_DEV_STS, 0) FIELD(CXL_MEM_DEV_STS, MBOX_READY, 4, 1) FIELD(CXL_MEM_DEV_STS, RESET_NEEDED, 5, 3) +static inline void __toggle_media(CXLDeviceState *cxl_dstate, int val) +{ + uint64_t dev_status_reg; + + dev_status_reg = FIELD_DP64(0, CXL_MEM_DEV_STS, MEDIA_STATUS, val); + cxl_dstate->mbox_reg_state64[R_CXL_MEM_DEV_STS] = dev_status_reg; +} +#define cxl_dev_disable_media(cxlds) \ + do { __toggle_media((cxlds), 0x3); } while (0) +#define cxl_dev_enable_media(cxlds) \ + do { __toggle_media((cxlds), 0x1); } while (0) + +static inline bool sanitize_running(CXLDeviceState *cxl_dstate) +{ + return !!cxl_dstate->bg.runtime && cxl_dstate->bg.opcode == 0x4400; +} + typedef struct CXLError { QTAILQ_ENTRY(CXLError) node; int type; /* Error code as per FE definition */