From patchwork Thu Jun 23 21:34:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Klaus Jensen X-Patchwork-Id: 12893216 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D41A8C433EF for ; Thu, 23 Jun 2022 21:47:43 +0000 (UTC) Received: from localhost ([::1]:50512 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1o4Ufu-00040N-RR for qemu-devel@archiver.kernel.org; Thu, 23 Jun 2022 17:47:42 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54960) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o4UTp-0004R7-SE; Thu, 23 Jun 2022 17:35:13 -0400 Received: from wout5-smtp.messagingengine.com ([64.147.123.21]:47733) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o4UTl-0006cz-6r; Thu, 23 Jun 2022 17:35:13 -0400 Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.west.internal (Postfix) with ESMTP id F1C0D3200978; Thu, 23 Jun 2022 17:35:05 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Thu, 23 Jun 2022 17:35:07 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=irrelevant.dk; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to; s=fm1; t=1656020105; x= 1656106505; bh=3h62N1IrLjIp0ND2Kjg+L7gZ21qshRstG5g1XZMVC4Q=; b=k kr+IUBGeSnEOdPi3vMHciHqdlFR2C3vxGKDvr5+6qOjLB39TXwuOFClu7aO4CCVf yVjmQGFj45y7742VcF35YAhnsRyJ1G+45BVQCKmFp9Tr8NcXfPjXAcrmXmH4aVcu XooGavDeryajQgSZ7hG7eheaRu4OJyeVU6SC4ikCMd4jkGpQAecranzhrwbapFem 4Kw0ADeio642OZqqRQYxPz6HQ3//RyOZQ/aDvno/NG4YlkVluLfJm17MJ/H77Hps MsNdBY2gor/McGfPKcLm/rGIJjc58hgWyjQof6kQqinpfmGPG+Khcg4QO7qm2XM6 T9P36jlwsWqbAsbMK09gQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t=1656020105; x= 1656106505; bh=3h62N1IrLjIp0ND2Kjg+L7gZ21qshRstG5g1XZMVC4Q=; b=J cawU9HhkIQRFe2qRCxjS+hDEo+x5Q/Twm4F/G+ckqd1FJY3YsQ1jSYqcSoaNaVxI LW4CfzlyFTQ0CT5CMs3cplBSIm/AiOahEIqde06nogy0abAD81BJcBXidttBlBCF lfX47M1wjfGV0EtypMIVF6nYi7cUBOHruIjonqtvDnW/HL9+GiNGmPbxkPGwSTWO u1MYQBkan2PWVxAV1rn4oJb4GhHnQd50TjhGBZ121LC4EMG/dtZjttaXPch1hJfU wsiGH9ZDPa1eeTOTAG5p+2hvdBjsiABaHRIWfbWsr4kwsnbFSoYBANP0tsZBqXi5 bviRuBqErBr7wJOYSmMog== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedrudefjedgudeifecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefhvfevufffkffojghfgggtgfesthekredtredtjeenucfhrhhomhepmfhl rghushculfgvnhhsvghnuceoihhtshesihhrrhgvlhgvvhgrnhhtrdgukheqnecuggftrf grthhtvghrnhepfeevtdeuteeuudffvefggfdtfedtueelfffhieegffekgeefjeefffet jeeihfdvnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomh epihhtshesihhrrhgvlhgvvhgrnhhtrdgukh X-ME-Proxy: Feedback-ID: idc91472f:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 23 Jun 2022 17:35:03 -0400 (EDT) From: Klaus Jensen To: Peter Maydell , qemu-devel@nongnu.org Cc: Stefan Hajnoczi , Igor Mammedov , Ani Sinha , Hanna Reitz , Kevin Wolf , "Michael S. Tsirkin" , Klaus Jensen , qemu-block@nongnu.org, Keith Busch , Fam Zheng , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Marcel Apfelbaum , =?utf-8?q?=C5=81ukasz_Gieryk?= , Klaus Jensen Subject: [PULL 04/15] hw/nvme: Implement the Function Level Reset Date: Thu, 23 Jun 2022 23:34:31 +0200 Message-Id: <20220623213442.67789-5-its@irrelevant.dk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220623213442.67789-1-its@irrelevant.dk> References: <20220623213442.67789-1-its@irrelevant.dk> MIME-Version: 1.0 Received-SPF: pass client-ip=64.147.123.21; envelope-from=its@irrelevant.dk; helo=wout5-smtp.messagingengine.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Łukasz Gieryk This patch implements the Function Level Reset, a feature currently not implemented for the Nvme device, while listed as a mandatory ("shall") in the 1.4 spec. The implementation reuses FLR-related building blocks defined for the pci-bridge module, and follows the same logic: - FLR capability is advertised in the PCIE config, - custom pci_write_config callback detects a write to the trigger register and performs the PCI reset, - which, eventually, calls the custom dc->reset handler. Depending on reset type, parts of the state should (or should not) be cleared. To distinguish the type of reset, an additional parameter is passed to the reset function. This patch also enables advertisement of the Power Management PCI capability. The main reason behind it is to announce the no_soft_reset=1 bit, to signal SR-IOV support where each VF can be reset individually. The implementation purposedly ignores writes to the PMCS.PS register, as even such naïve behavior is enough to correctly handle the D3->D0 transition. It’s worth to note, that the power state transition back to to D3, with all the corresponding side effects, wasn't and stil isn't handled properly. Signed-off-by: Łukasz Gieryk Reviewed-by: Klaus Jensen Acked-by: Michael S. Tsirkin Signed-off-by: Klaus Jensen --- hw/nvme/ctrl.c | 52 ++++++++++++++++++++++++++++++++++++++++---- hw/nvme/nvme.h | 5 +++++ hw/nvme/trace-events | 1 + 3 files changed, 54 insertions(+), 4 deletions(-) diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c index b031212758cb..5ae80f114003 100644 --- a/hw/nvme/ctrl.c +++ b/hw/nvme/ctrl.c @@ -5903,7 +5903,7 @@ static void nvme_process_sq(void *opaque) } } -static void nvme_ctrl_reset(NvmeCtrl *n) +static void nvme_ctrl_reset(NvmeCtrl *n, NvmeResetType rst) { NvmeNamespace *ns; int i; @@ -5935,7 +5935,9 @@ static void nvme_ctrl_reset(NvmeCtrl *n) } if (!pci_is_vf(&n->parent_obj) && n->params.sriov_max_vfs) { - pcie_sriov_pf_disable_vfs(&n->parent_obj); + if (rst != NVME_RESET_CONTROLLER) { + pcie_sriov_pf_disable_vfs(&n->parent_obj); + } } n->aer_queued = 0; @@ -6169,7 +6171,7 @@ static void nvme_write_bar(NvmeCtrl *n, hwaddr offset, uint64_t data, } } else if (!NVME_CC_EN(data) && NVME_CC_EN(cc)) { trace_pci_nvme_mmio_stopped(); - nvme_ctrl_reset(n); + nvme_ctrl_reset(n, NVME_RESET_CONTROLLER); cc = 0; csts &= ~NVME_CSTS_READY; } @@ -6727,6 +6729,28 @@ static void nvme_init_sriov(NvmeCtrl *n, PCIDevice *pci_dev, uint16_t offset, PCI_BASE_ADDRESS_MEM_TYPE_64, bar_size); } +static int nvme_add_pm_capability(PCIDevice *pci_dev, uint8_t offset) +{ + Error *err = NULL; + int ret; + + ret = pci_add_capability(pci_dev, PCI_CAP_ID_PM, offset, + PCI_PM_SIZEOF, &err); + if (err) { + error_report_err(err); + return ret; + } + + pci_set_word(pci_dev->config + offset + PCI_PM_PMC, + PCI_PM_CAP_VER_1_2); + pci_set_word(pci_dev->config + offset + PCI_PM_CTRL, + PCI_PM_CTRL_NO_SOFT_RESET); + pci_set_word(pci_dev->wmask + offset + PCI_PM_CTRL, + PCI_PM_CTRL_STATE_MASK); + + return 0; +} + static int nvme_init_pci(NvmeCtrl *n, PCIDevice *pci_dev, Error **errp) { uint8_t *pci_conf = pci_dev->config; @@ -6748,7 +6772,9 @@ static int nvme_init_pci(NvmeCtrl *n, PCIDevice *pci_dev, Error **errp) } pci_config_set_class(pci_conf, PCI_CLASS_STORAGE_EXPRESS); + nvme_add_pm_capability(pci_dev, 0x60); pcie_endpoint_cap_init(pci_dev, 0x80); + pcie_cap_flr_init(pci_dev); if (n->params.sriov_max_vfs) { pcie_ari_init(pci_dev, 0x100, 1); } @@ -6999,7 +7025,7 @@ static void nvme_exit(PCIDevice *pci_dev) NvmeNamespace *ns; int i; - nvme_ctrl_reset(n); + nvme_ctrl_reset(n, NVME_RESET_FUNCTION); if (n->subsys) { for (i = 1; i <= NVME_MAX_NAMESPACES; i++) { @@ -7098,6 +7124,22 @@ static void nvme_set_smart_warning(Object *obj, Visitor *v, const char *name, } } +static void nvme_pci_reset(DeviceState *qdev) +{ + PCIDevice *pci_dev = PCI_DEVICE(qdev); + NvmeCtrl *n = NVME(pci_dev); + + trace_pci_nvme_pci_reset(); + nvme_ctrl_reset(n, NVME_RESET_FUNCTION); +} + +static void nvme_pci_write_config(PCIDevice *dev, uint32_t address, + uint32_t val, int len) +{ + pci_default_write_config(dev, address, val, len); + pcie_cap_flr_write_config(dev, address, val, len); +} + static const VMStateDescription nvme_vmstate = { .name = "nvme", .unmigratable = 1, @@ -7109,6 +7151,7 @@ static void nvme_class_init(ObjectClass *oc, void *data) PCIDeviceClass *pc = PCI_DEVICE_CLASS(oc); pc->realize = nvme_realize; + pc->config_write = nvme_pci_write_config; pc->exit = nvme_exit; pc->class_id = PCI_CLASS_STORAGE_EXPRESS; pc->revision = 2; @@ -7117,6 +7160,7 @@ static void nvme_class_init(ObjectClass *oc, void *data) dc->desc = "Non-Volatile Memory Express"; device_class_set_props(dc, nvme_props); dc->vmsd = &nvme_vmstate; + dc->reset = nvme_pci_reset; } static void nvme_instance_init(Object *obj) diff --git a/hw/nvme/nvme.h b/hw/nvme/nvme.h index b66421cdf9e8..7b317d3dc469 100644 --- a/hw/nvme/nvme.h +++ b/hw/nvme/nvme.h @@ -488,6 +488,11 @@ typedef struct NvmeCtrl { NvmeSecCtrlList sec_ctrl_list; } NvmeCtrl; +typedef enum NvmeResetType { + NVME_RESET_FUNCTION = 0, + NVME_RESET_CONTROLLER = 1, +} NvmeResetType; + static inline NvmeNamespace *nvme_ns(NvmeCtrl *n, uint32_t nsid) { if (!nsid || nsid > NVME_MAX_NAMESPACES) { diff --git a/hw/nvme/trace-events b/hw/nvme/trace-events index 889bbb3101e4..b07864c57322 100644 --- a/hw/nvme/trace-events +++ b/hw/nvme/trace-events @@ -110,6 +110,7 @@ pci_nvme_zd_extension_set(uint32_t zone_idx) "set descriptor extension for zone_ pci_nvme_clear_ns_close(uint32_t state, uint64_t slba) "zone state=%"PRIu32", slba=%"PRIu64" transitioned to Closed state" pci_nvme_clear_ns_reset(uint32_t state, uint64_t slba) "zone state=%"PRIu32", slba=%"PRIu64" transitioned to Empty state" pci_nvme_zoned_zrwa_implicit_flush(uint64_t zslba, uint32_t nlb) "zslba 0x%"PRIx64" nlb %"PRIu32"" +pci_nvme_pci_reset(void) "PCI Function Level Reset" # error conditions pci_nvme_err_mdts(size_t len) "len %zu"