From patchwork Tue Oct 18 03:00:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Davidlohr Bueso X-Patchwork-Id: 13009834 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 23818C433FE for ; Tue, 18 Oct 2022 03:29:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230044AbiJRD3j (ORCPT ); Mon, 17 Oct 2022 23:29:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33292 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231130AbiJRD2r (ORCPT ); Mon, 17 Oct 2022 23:28:47 -0400 Received: from beige.elm.relay.mailchannels.net (beige.elm.relay.mailchannels.net [23.83.212.16]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 04F41A4BB7; Mon, 17 Oct 2022 20:21:45 -0700 (PDT) X-Sender-Id: dreamhost|x-authsender|dave@stgolabs.net Received: from relay.mailchannels.net (localhost [127.0.0.1]) by relay.mailchannels.net (Postfix) with ESMTP id 38F24880B92; Tue, 18 Oct 2022 03:21:22 +0000 (UTC) Received: from pdx1-sub0-mail-a201 (unknown [127.0.0.6]) (Authenticated sender: dreamhost) by relay.mailchannels.net (Postfix) with ESMTPA id 9B368880B4D; Tue, 18 Oct 2022 03:21:21 +0000 (UTC) ARC-Seal: i=1; s=arc-2022; d=mailchannels.net; t=1666063281; a=rsa-sha256; cv=none; b=8zxPPS0uS1t+YDChnynjIpzPQsSC8VIH38cIxINiskvkSybdEpUW7QAY0Z0EI55KySY6sf lcj+WMUtvooV17q4RgagE1HcxuoRQLtewPuYl3eKwPu4Qo7g4pZMBRNCN5m/SpvSGnfWei 6ecEskLwdkoNR7D6UhtFj8HIdP6J9DWjpdyJZKksUzIf0SnJlIpg+fUCLoP5UEPlSxhl5W w+fl5qf6cO69wBmffNlRingtbCYXNYPWexrG9u3hrNQtTVbsnNF2Kv6xrAj3xSUNZNMEda 9+3GeTj4Pn6FWpbOS11OYtU35ovxVstK2aKo4m/uvjnn1JsoQsEPlOKFmb1BhQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=mailchannels.net; s=arc-2022; t=1666063281; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5rez8qYjf0NKGjU9X2fo6bHVw919Ec9FBSH61oJQCaA=; b=8uPSPU8vpeuvsFS2H4z9ybpNlnT5Zsljb3m6VgrDTNmzqrHhczCG2k5r98SCwin+PAz+an P5BYQpNoypxB+NMQ9OGFwiWhTDorxAFQsg73Y4qeNE5gN1l//HpgxbtEQ/NDzADESUODe5 s00oFbo5JJm0vYeUEfDIXHnwLKbKAKjwwVYJcptW4BzJf6TUSN2+IDp5XGtIBKz8TUXfMm BOs7Rjt/WcDuvC8Y0VU5LA+1LSxQO1tyuRVVbUP+OsgCaKmc9dA4G0DLdWqLHw6H9Y0ro+ 4JMwHDFu5UOkSvf1D/umrTTiwXhbmXzYkrJ4ci8ZSLOR+323jZuaCgaUJgzOxA== ARC-Authentication-Results: i=1; rspamd-7fb88f4dd5-zsbr9; auth=pass smtp.auth=dreamhost smtp.mailfrom=dave@stgolabs.net X-Sender-Id: dreamhost|x-authsender|dave@stgolabs.net X-MC-Relay: Neutral X-MailChannels-SenderId: dreamhost|x-authsender|dave@stgolabs.net X-MailChannels-Auth-Id: dreamhost X-Occur-Lettuce: 2a53bb1249e09068_1666063281979_801838855 X-MC-Loop-Signature: 1666063281979:1386239554 X-MC-Ingress-Time: 1666063281979 Received: from pdx1-sub0-mail-a201 (pop.dreamhost.com [64.90.62.162]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384) by 100.122.72.98 (trex/6.7.1); Tue, 18 Oct 2022 03:21:21 +0000 Received: from localhost.localdomain (unknown [104.36.31.104]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: dave@stgolabs.net) by pdx1-sub0-mail-a201 (Postfix) with ESMTPSA id 4MrzfS4Q3pz54; Mon, 17 Oct 2022 20:21:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stgolabs.net; s=dreamhost; t=1666063281; bh=5rez8qYjf0NKGjU9X2fo6bHVw919Ec9FBSH61oJQCaA=; h=From:To:Cc:Subject:Date:Content-Transfer-Encoding; b=nuHpR0uUQbAZRV03IPprYEp3zjTQD4IUrrcgHsLGCEvDm1A0TcD5d1BxAlnC5T8Yo stmucL3WEA6POSQailbbpJfCwnZyrbJCZpwiXcMkpui1DfJMgfexm7FnyWuLkAaBW+ 3XKUGfXEs3JD6TzMSsW7Rin+egEjN0u9Loqp11FCGUOUO9Xql00pF/LxU65XOWkLQG gh8PE9UDQtObQEHy6aWAxzuX3eoy9CYSQ1Yw9cWfwil6+9DV2EJ+EE2NZPLhAzhFK3 dUzZwxT9q7gPWeOMxVxtoh3J4ErXzy9Qb/o8nUUk79f0DRTvZO2qS1ip7UzCdgmNeT C/TuY22P4184Q== From: Davidlohr Bueso To: dan.j.williams@intel.com Cc: ira.weiny@intel.com, Jonathan.Cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, bwidawsk@kernel.org, vishal.l.verma@intel.com, a.manzanares@samsung.com, linux-cxl@vger.kernel.org, linux-kernel@vger.kernel.org, dave@stgolabs.net Subject: [PATCH 1/2] cxl/pci: Add generic MSI-X/MSI irq support Date: Mon, 17 Oct 2022 20:00:09 -0700 Message-Id: <20221018030010.20913-2-dave@stgolabs.net> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221018030010.20913-1-dave@stgolabs.net> References: <20221018030010.20913-1-dave@stgolabs.net> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org Introduce a generic irq table for CXL components/features that can have standard irq support - DOE requires dynamic vector sizing and is not considered here. For now the table is empty. Create an infrastructure to query the max vectors required for the CXL device. Upon successful allocation, users can plug in their respective isr at any point thereafter, which is supported by a new cxlds->has_irq flag, for example, if the irq setup is not done in the PCI driver, such as the case of the CXL-PMU. Reviewed-by: Dave Jiang Signed-off-by: Davidlohr Bueso Reviewed-by: Jonathan Cameron --- drivers/cxl/cxlmem.h | 3 ++ drivers/cxl/pci.c | 72 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 75 insertions(+) diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h index 88e3a8e54b6a..72b69b003302 100644 --- a/drivers/cxl/cxlmem.h +++ b/drivers/cxl/cxlmem.h @@ -211,6 +211,7 @@ struct cxl_endpoint_dvsec_info { * @info: Cached DVSEC information about the device. * @serial: PCIe Device Serial Number * @doe_mbs: PCI DOE mailbox array + * @has_irq: PCIe MSI-X/MSI support * @mbox_send: @dev specific transport for transmitting mailbox commands * * See section 8.2.9.5.2 Capacity Configuration and Label Storage for @@ -247,6 +248,8 @@ struct cxl_dev_state { struct xarray doe_mbs; + bool has_irq; + int (*mbox_send)(struct cxl_dev_state *cxlds, struct cxl_mbox_cmd *cmd); }; diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index faeb5d9d7a7a..9c3e95ebaa26 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -428,6 +428,73 @@ static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds) } } +/** + * struct cxl_irq_cap - CXL feature that is capable of receiving MSI-X/MSI irqs. + * + * @name: Name of the device/component generating this interrupt. + * @get_max_msgnum: Get the feature's largest interrupt message number. If the + * feature does not have the Interrupt Supported bit set, then + * return -1. + */ +struct cxl_irq_cap { + const char *name; + int (*get_max_msgnum)(struct cxl_dev_state *cxlds); +}; + +static const struct cxl_irq_cap cxl_irq_cap_table[] = { + NULL +}; + +static void cxl_pci_free_irq_vectors(void *data) +{ + pci_free_irq_vectors(data); +} + +/* + * Attempt to allocate the largest amount of necessary vectors. + * + * Returns 0 upon a successful allocation of *all* vectors, or a + * negative value otherwise. + */ +static int cxl_pci_alloc_irq_vectors(struct cxl_dev_state *cxlds) +{ + struct device *dev = cxlds->dev; + struct pci_dev *pdev = to_pci_dev(dev); + int rc, i, vectors = -1; + + for (i = 0; i < ARRAY_SIZE(cxl_irq_cap_table); i++) { + int irq; + + if (!cxl_irq_cap_table[i].get_max_msgnum) + continue; + + irq = cxl_irq_cap_table[i].get_max_msgnum(cxlds); + vectors = max_t(int, irq, vectors); + } + + /* + * Semantically lack of irq support is not an error, but we + * still fail to allocate, so return negative. + */ + if (vectors == -1) + return -1; + + vectors++; + rc = pci_alloc_irq_vectors(pdev, vectors, vectors, + PCI_IRQ_MSIX | PCI_IRQ_MSI); + if (rc < 0) + return rc; + + if (rc != vectors) { + dev_dbg(dev, "Not enough interrupts; use polling instead.\n"); + /* some got allocated, clean them up */ + cxl_pci_free_irq_vectors(pdev); + return -ENOSPC; + } + + return devm_add_action_or_reset(dev, cxl_pci_free_irq_vectors, pdev); +} + static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) { struct cxl_register_map map; @@ -494,6 +561,11 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) if (rc) return rc; + if (!cxl_pci_alloc_irq_vectors(cxlds)) { + cxlds->has_irq = true; + } else + cxlds->has_irq = false; + cxlmd = devm_cxl_add_memdev(cxlds); if (IS_ERR(cxlmd)) return PTR_ERR(cxlmd); From patchwork Tue Oct 18 03:00:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Davidlohr Bueso X-Patchwork-Id: 13009835 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CA2AFC4332F for ; Tue, 18 Oct 2022 03:29:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230115AbiJRD3k (ORCPT ); Mon, 17 Oct 2022 23:29:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42760 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230525AbiJRD2r (ORCPT ); Mon, 17 Oct 2022 23:28:47 -0400 Received: from bee.birch.relay.mailchannels.net (bee.birch.relay.mailchannels.net [23.83.209.14]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 763B3A027B; Mon, 17 Oct 2022 20:21:46 -0700 (PDT) X-Sender-Id: dreamhost|x-authsender|dave@stgolabs.net Received: from relay.mailchannels.net (localhost [127.0.0.1]) by relay.mailchannels.net (Postfix) with ESMTP id 16AB1341BAB; Tue, 18 Oct 2022 03:21:23 +0000 (UTC) Received: from pdx1-sub0-mail-a201 (unknown [127.0.0.6]) (Authenticated sender: dreamhost) by relay.mailchannels.net (Postfix) with ESMTPA id 81484341AE1; Tue, 18 Oct 2022 03:21:22 +0000 (UTC) ARC-Seal: i=1; s=arc-2022; d=mailchannels.net; t=1666063282; a=rsa-sha256; cv=none; b=z8Zh9yi+NbXnemOtcmJJuwW+oj7d5+3fUXFAzsU9zpwdsbQ7x8H29Iqh2HCeA/ILJYaxaL LCTeO4u7UBNP5audik4dZXWjZm2zb8+SXbebm2dqCnD9B9EXf2aT4hJdHYI4bn53mk5rB/ a6Lkz9GI2ha6lmLcTBIGO7nNN+fl7VV+bvdU+qHAy2XxX+HB7YB7y7dUJeU9WmQDFDxoy7 e80pjRngZ9NE5kqUzSzMPgdLSl4mphaocslL3h7/MAKGNGiXNsNWlZKDfyv8mITq9TpATX uzGAR//He5zNlxiJ4rW2u+rBSpu+G0YEvLCPXi8lbyaFfzhVEJ/QIeT71rxWcw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=mailchannels.net; s=arc-2022; t=1666063282; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+F77OywPESqqbC+/oPvGc0AQuP7cONHmICZM40h0KBY=; b=XSJpp8oNvxpfQl2o2RG413EojHNYKOir3pBDuZ/ZIAG9+M8qr6uYprG/wNKRf20i4QmqE2 fgo576To4iBkDI8mLWPKz7xxvmBWJKq1atzqmmazEn1YABF5njJ9tyggot2bzYIHafSjnb gBQGee0VBhMvOhepzlQ2FNL1oYlrNOXzXQjoyd+yZmWRnEFZMwDWZg4D4N3Pbbt3l0A97b XJMG475t8gm1zXViySuFTHCKSKx2iDLxekcuvonEpTf2HbLd/arwmeqbWQ/qZ5b60SM9kN bfmpeXGDu2L6EPYzmJNUjsFhvoJg35wHI2D529imn+Cd5xNdbvnEZ190cGHmeA== ARC-Authentication-Results: i=1; rspamd-7fb88f4dd5-7vxgv; auth=pass smtp.auth=dreamhost smtp.mailfrom=dave@stgolabs.net X-Sender-Id: dreamhost|x-authsender|dave@stgolabs.net X-MC-Relay: Neutral X-MailChannels-SenderId: dreamhost|x-authsender|dave@stgolabs.net X-MailChannels-Auth-Id: dreamhost X-Snatch-Cure: 60cdaa360af8dbc2_1666063282857_1569416159 X-MC-Loop-Signature: 1666063282857:1041835057 X-MC-Ingress-Time: 1666063282857 Received: from pdx1-sub0-mail-a201 (pop.dreamhost.com [64.90.62.162]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384) by 100.122.146.22 (trex/6.7.1); Tue, 18 Oct 2022 03:21:22 +0000 Received: from localhost.localdomain (unknown [104.36.31.104]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: dave@stgolabs.net) by pdx1-sub0-mail-a201 (Postfix) with ESMTPSA id 4MrzfT4PJMz4r; Mon, 17 Oct 2022 20:21:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stgolabs.net; s=dreamhost; t=1666063282; bh=+F77OywPESqqbC+/oPvGc0AQuP7cONHmICZM40h0KBY=; h=From:To:Cc:Subject:Date:Content-Transfer-Encoding; b=AU8abd9B2gT8pXCbb1HVa2wtEwJD7UPCT98DA++jcGBufcCBd3vRzosXU5rhtQPkZ ANNbuIDk82wJiei8hR++2OkTJaRk3bbr865B8SqGeL4cey05gK0f7zSkSOgi2UggS9 DJu7I8rFna9uAh5I/Nbp6VQzlVn2uP1aw1+e8tThOqOUyRUwEvJg9nfohu2UAavXhL oH3ybDdo4PwtW9icrLt/av6h09Kj78bvAzRFRxgQQxyTaUtbT+KkqFjEfC/QAjZHYE Iy5Q6jYp4IJvE0iHnfeQlX6YYEK8vMFm/MjX+7HWUw/D91iyQw2RWtc4MQowF6IkNU XxHvVCQ2QTRVg== From: Davidlohr Bueso To: dan.j.williams@intel.com Cc: ira.weiny@intel.com, Jonathan.Cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, bwidawsk@kernel.org, vishal.l.verma@intel.com, a.manzanares@samsung.com, linux-cxl@vger.kernel.org, linux-kernel@vger.kernel.org, dave@stgolabs.net Subject: [PATCH 2/2] cxl/mbox: Wire up irq support Date: Mon, 17 Oct 2022 20:00:10 -0700 Message-Id: <20221018030010.20913-3-dave@stgolabs.net> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221018030010.20913-1-dave@stgolabs.net> References: <20221018030010.20913-1-dave@stgolabs.net> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org With enough vectors properly allocated, this adds support for (the primary) mailbox interrupt, which is needed for background completion handling, beyond polling. Signed-off-by: Davidlohr Bueso --- drivers/cxl/cxl.h | 1 + drivers/cxl/pci.c | 29 ++++++++++++++++++++++++++++- 2 files changed, 29 insertions(+), 1 deletion(-) diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index f680450f0b16..13a9743b0012 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -135,6 +135,7 @@ static inline int ways_to_cxl(unsigned int ways, u8 *iw) /* CXL 2.0 8.2.8.4 Mailbox Registers */ #define CXLDEV_MBOX_CAPS_OFFSET 0x00 #define CXLDEV_MBOX_CAP_PAYLOAD_SIZE_MASK GENMASK(4, 0) +#define CXLDEV_MBOX_CAP_IRQ_MSGNUM_MASK GENMASK(10, 7) #define CXLDEV_MBOX_CTRL_OFFSET 0x04 #define CXLDEV_MBOX_CTRL_DOORBELL BIT(0) #define CXLDEV_MBOX_CMD_OFFSET 0x08 diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index 9c3e95ebaa26..c3f3ee307d7a 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -274,6 +274,32 @@ static int cxl_pci_setup_mailbox(struct cxl_dev_state *cxlds) return 0; } +static int cxl_pci_mbox_get_max_msgnum(struct cxl_dev_state *cxlds) +{ + int cap; + + cap = readl(cxlds->regs.mbox + CXLDEV_MBOX_CAPS_OFFSET); + return FIELD_GET(CXLDEV_MBOX_CAP_IRQ_MSGNUM_MASK, cap); +} + +static irqreturn_t cxl_pci_mbox_irq(int irq, void *id) +{ + /* TODO: handle completion of background commands */ + return IRQ_HANDLED; +} + +static void cxl_pci_mbox_irqsetup(struct cxl_dev_state *cxlds) +{ + struct device *dev = cxlds->dev; + struct pci_dev *pdev = to_pci_dev(dev); + int irq; + + irq = cxl_pci_mbox_get_max_msgnum(cxlds); + if (!pci_request_irq(pdev, irq, cxl_pci_mbox_irq, NULL, + cxlds, "%s-mailbox", dev_name(dev))) + dev_dbg(dev, "Mailbox irq (%d) supported", irq); +} + static int cxl_map_regblock(struct pci_dev *pdev, struct cxl_register_map *map) { void __iomem *addr; @@ -442,7 +468,7 @@ struct cxl_irq_cap { }; static const struct cxl_irq_cap cxl_irq_cap_table[] = { - NULL + { "mailbox", cxl_pci_mbox_get_max_msgnum }, }; static void cxl_pci_free_irq_vectors(void *data) @@ -562,6 +588,7 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) return rc; if (!cxl_pci_alloc_irq_vectors(cxlds)) { + cxl_pci_mbox_irqsetup(cxlds); cxlds->has_irq = true; } else cxlds->has_irq = false;