From patchwork Fri Oct 6 07:26:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 13411003 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 463D1EA1 for ; Fri, 6 Oct 2023 07:26:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="NE43Njcl" Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.126]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BB276E9 for ; Fri, 6 Oct 2023 00:26:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1696577165; x=1728113165; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=vk4tx+csN7fnrFRQFovjtKXGsT01perxiVyqKyByoUE=; b=NE43Njcl3Aq+l9XstnLy9EMLQ1lFDRFeBXAjnKQz4Zlrzw08/0pwArsV Ku3AlLpOmeYKX1UfKxJgfeb9FAK7SiwUEaj9ky/xKM8YUzHmeODlC8LIu gDzC0mSlVPQmc+bTRj5+KsdMKzgQhm3F5WlF6SLROxApYuoZI4rAFLjvt Dj19EdurRIz1oZqInphthiKfYfmsSseQgEt+fkgTg8wIymCcmUheFyNnl LS5hbKQSncWrC87GLGboNJgqE1lq9a+lh3oUsdvcRikUuf8jPI+RlB3rs jktZcZ/ETBh+CBTd2QtZ0oVgAkEHeZvC438VaNhzfnXxobSW8GELmF4rt g==; X-IronPort-AV: E=McAfee;i="6600,9927,10854"; a="368775169" X-IronPort-AV: E=Sophos;i="6.03,203,1694761200"; d="scan'208";a="368775169" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 00:26:05 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10854"; a="842735409" X-IronPort-AV: E=Sophos;i="6.03,203,1694761200"; d="scan'208";a="842735409" Received: from wbleichn-mobl.amr.corp.intel.com (HELO dwillia2-xfh.jf.intel.com) ([10.212.147.24]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 00:26:04 -0700 Subject: [PATCH v3 01/10] cxl/pci: Remove unnecessary device reference management in sanitize work From: Dan Williams To: linux-cxl@vger.kernel.org Cc: Dave Jiang , Davidlohr Bueso , Jonathan Cameron , Ira Weiny Date: Fri, 06 Oct 2023 00:26:04 -0700 Message-ID: <169657716434.1491153.11640655683077170158.stgit@dwillia2-xfh.jf.intel.com> In-Reply-To: <169657715790.1491153.3612164287133860191.stgit@dwillia2-xfh.jf.intel.com> References: <169657715790.1491153.3612164287133860191.stgit@dwillia2-xfh.jf.intel.com> User-Agent: StGit/0.18-3-g996c Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Given that any particular put_device() could be the final put of the device, the fact that there are usages of cxlds->dev after put_device(cxlds->dev) is a red flag. Drop the reference counting since the device is pinned by being registered and will not be unregistered without triggering the driver + workqueue to shutdown. Reviewed-by: Dave Jiang Reviewed-by: Davidlohr Bueso Reviewed-by: Jonathan Cameron Reviewed-by: Ira Weiny Signed-off-by: Dan Williams --- drivers/cxl/pci.c | 5 ----- 1 file changed, 5 deletions(-) diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index 44a21ab7add5..aa1b3dd9e64c 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -152,8 +152,6 @@ static void cxl_mbox_sanitize_work(struct work_struct *work) mutex_lock(&mds->mbox_mutex); if (cxl_mbox_background_complete(cxlds)) { mds->security.poll_tmo_secs = 0; - put_device(cxlds->dev); - if (mds->security.sanitize_node) sysfs_notify_dirent(mds->security.sanitize_node); @@ -296,9 +294,6 @@ static int __cxl_pci_mbox_send_cmd(struct cxl_memdev_state *mds, */ if (mbox_cmd->opcode == CXL_MBOX_OP_SANITIZE) { if (mds->security.poll) { - /* hold the device throughout */ - get_device(cxlds->dev); - /* give first timeout a second */ timeout = 1; mds->security.poll_tmo_secs = timeout; From patchwork Fri Oct 6 07:26:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 13411004 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9F5F063CC for ; Fri, 6 Oct 2023 07:26:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="U0npxT4t" Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.9]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0EF6EE4 for ; Fri, 6 Oct 2023 00:26:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1696577170; x=1728113170; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=8kvHwC+c6S61F3o2c58tYl95gq7aTQuRF2LyQEx6EBA=; b=U0npxT4tMfNgT9ngAy2MknAxHXw8fug5KM9pBT0kqrqfKmG3xCLT4WHf 2LJlFJAbblpb4g3JKINEUivc5I4AsRdW4Xn79E+eMKXtbicD5ynB4Dj5U DDEiSMEK9AzXfBlYF4vRLdra3BGFT3FJ4Uq6ki2TRHekfKgxrG6GoQ+0a DOKM+QYCy80InCM3Kp3NztK94XU5Ub/op+/46FZ3UGf/WGoTrMtdNi56V Xhf3+QIZrg16H6oUD0z9xm+7QUT7q4NfprQQssgAIkHZgUD711O8wivfT q0PoFbX/rGyE5dojhIxtiAUPcVtx33CBwoHobivZLEi7H/4YRFxLS45JU Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10854"; a="2297883" X-IronPort-AV: E=Sophos;i="6.03,203,1694761200"; d="scan'208";a="2297883" Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orvoesa101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 00:26:09 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10854"; a="755760364" X-IronPort-AV: E=Sophos;i="6.03,203,1694761200"; d="scan'208";a="755760364" Received: from wbleichn-mobl.amr.corp.intel.com (HELO dwillia2-xfh.jf.intel.com) ([10.212.147.24]) by fmsmga007-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 00:26:10 -0700 Subject: [PATCH v3 02/10] cxl/pci: Cleanup 'sanitize' to always poll From: Dan Williams To: linux-cxl@vger.kernel.org Cc: Dave Jiang , Jonathan Cameron , Ira Weiny Date: Fri, 06 Oct 2023 00:26:10 -0700 Message-ID: <169657717034.1491153.300696666588880104.stgit@dwillia2-xfh.jf.intel.com> In-Reply-To: <169657715790.1491153.3612164287133860191.stgit@dwillia2-xfh.jf.intel.com> References: <169657715790.1491153.3612164287133860191.stgit@dwillia2-xfh.jf.intel.com> User-Agent: StGit/0.18-3-g996c Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net In preparation for fixing the init/teardown of the 'sanitize' workqueue and sysfs notification mechanism, arrange for cxl_mbox_sanitize_work() to be the single location where the sysfs attribute is notified. With that change there is no distinction between polled mode and interrupt mode. All the interrupt does is accelerate the polling interval. The change to check for "mds->security.sanitize_node" under the lock is there to ensure that the interrupt, the work routine and the setup/teardown code can all have a consistent view of the registered notifier and the workqueue state. I.e. the expectation is that the interrupt is live past the point that the sanitize sysfs attribute is published, and it may race teardown, so it must be consulted under a lock. Given that new locking requirement, cxl_pci_mbox_irq() is moved from hard to thread irq context. Lastly, some opportunistic replacements of "queue_delayed_work(system_wq, ...)", which is just open coded schedule_delayed_work(), are included. Reviewed-by: Dave Jiang Reviewed-by: Jonathan Cameron Reviewed-by: Ira Weiny Signed-off-by: Dan Williams Reviewed-by: Davidlohr Bueso --- drivers/cxl/core/memdev.c | 3 +- drivers/cxl/cxlmem.h | 2 -- drivers/cxl/pci.c | 60 +++++++++++++++++++-------------------------- 3 files changed, 26 insertions(+), 39 deletions(-) diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c index 14b547c07f54..2a7a07f6d165 100644 --- a/drivers/cxl/core/memdev.c +++ b/drivers/cxl/core/memdev.c @@ -561,8 +561,7 @@ static void cxl_memdev_security_shutdown(struct device *dev) struct cxl_memdev *cxlmd = to_cxl_memdev(dev); struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlmd->cxlds); - if (mds->security.poll) - cancel_delayed_work_sync(&mds->security.poll_dwork); + cancel_delayed_work_sync(&mds->security.poll_dwork); } static void cxl_memdev_shutdown(struct device *dev) diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h index 706f8a6d1ef4..55f00ad17a77 100644 --- a/drivers/cxl/cxlmem.h +++ b/drivers/cxl/cxlmem.h @@ -360,7 +360,6 @@ struct cxl_fw_state { * * @state: state of last security operation * @enabled_cmds: All security commands enabled in the CEL - * @poll: polling for sanitization is enabled, device has no mbox irq support * @poll_tmo_secs: polling timeout * @poll_dwork: polling work item * @sanitize_node: sanitation sysfs file to notify @@ -368,7 +367,6 @@ struct cxl_fw_state { struct cxl_security_state { unsigned long state; DECLARE_BITMAP(enabled_cmds, CXL_SEC_ENABLED_MAX); - bool poll; int poll_tmo_secs; struct delayed_work poll_dwork; struct kernfs_node *sanitize_node; diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index aa1b3dd9e64c..49d9b2ef5c5c 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -128,10 +128,10 @@ static irqreturn_t cxl_pci_mbox_irq(int irq, void *id) reg = readq(cxlds->regs.mbox + CXLDEV_MBOX_BG_CMD_STATUS_OFFSET); opcode = FIELD_GET(CXLDEV_MBOX_BG_CMD_COMMAND_OPCODE_MASK, reg); if (opcode == CXL_MBOX_OP_SANITIZE) { + mutex_lock(&mds->mbox_mutex); if (mds->security.sanitize_node) - sysfs_notify_dirent(mds->security.sanitize_node); - - dev_dbg(cxlds->dev, "Sanitization operation ended\n"); + mod_delayed_work(system_wq, &mds->security.poll_dwork, 0); + mutex_unlock(&mds->mbox_mutex); } else { /* short-circuit the wait in __cxl_pci_mbox_send_cmd() */ rcuwait_wake_up(&mds->mbox_wait); @@ -160,8 +160,7 @@ static void cxl_mbox_sanitize_work(struct work_struct *work) int timeout = mds->security.poll_tmo_secs + 10; mds->security.poll_tmo_secs = min(15 * 60, timeout); - queue_delayed_work(system_wq, &mds->security.poll_dwork, - timeout * HZ); + schedule_delayed_work(&mds->security.poll_dwork, timeout * HZ); } mutex_unlock(&mds->mbox_mutex); } @@ -293,15 +292,11 @@ static int __cxl_pci_mbox_send_cmd(struct cxl_memdev_state *mds, * and allow userspace to poll(2) for completion. */ if (mbox_cmd->opcode == CXL_MBOX_OP_SANITIZE) { - if (mds->security.poll) { - /* give first timeout a second */ - timeout = 1; - mds->security.poll_tmo_secs = timeout; - queue_delayed_work(system_wq, - &mds->security.poll_dwork, - timeout * HZ); - } - + /* give first timeout a second */ + timeout = 1; + mds->security.poll_tmo_secs = timeout; + schedule_delayed_work(&mds->security.poll_dwork, + timeout * HZ); dev_dbg(dev, "Sanitization operation started\n"); goto success; } @@ -384,7 +379,9 @@ static int cxl_pci_setup_mailbox(struct cxl_memdev_state *mds) const int cap = readl(cxlds->regs.mbox + CXLDEV_MBOX_CAPS_OFFSET); struct device *dev = cxlds->dev; unsigned long timeout; + int irq, msgnum; u64 md_status; + u32 ctrl; timeout = jiffies + mbox_ready_timeout * HZ; do { @@ -432,33 +429,26 @@ static int cxl_pci_setup_mailbox(struct cxl_memdev_state *mds) dev_dbg(dev, "Mailbox payload sized %zu", mds->payload_size); rcuwait_init(&mds->mbox_wait); + INIT_DELAYED_WORK(&mds->security.poll_dwork, cxl_mbox_sanitize_work); - if (cap & CXLDEV_MBOX_CAP_BG_CMD_IRQ) { - u32 ctrl; - int irq, msgnum; - struct pci_dev *pdev = to_pci_dev(cxlds->dev); - - msgnum = FIELD_GET(CXLDEV_MBOX_CAP_IRQ_MSGNUM_MASK, cap); - irq = pci_irq_vector(pdev, msgnum); - if (irq < 0) - goto mbox_poll; - - if (cxl_request_irq(cxlds, irq, cxl_pci_mbox_irq, NULL)) - goto mbox_poll; + /* background command interrupts are optional */ + if (!(cap & CXLDEV_MBOX_CAP_BG_CMD_IRQ)) + return 0; - /* enable background command mbox irq support */ - ctrl = readl(cxlds->regs.mbox + CXLDEV_MBOX_CTRL_OFFSET); - ctrl |= CXLDEV_MBOX_CTRL_BG_CMD_IRQ; - writel(ctrl, cxlds->regs.mbox + CXLDEV_MBOX_CTRL_OFFSET); + msgnum = FIELD_GET(CXLDEV_MBOX_CAP_IRQ_MSGNUM_MASK, cap); + irq = pci_irq_vector(to_pci_dev(cxlds->dev), msgnum); + if (irq < 0) + return 0; + if (cxl_request_irq(cxlds, irq, NULL, cxl_pci_mbox_irq)) return 0; - } -mbox_poll: - mds->security.poll = true; - INIT_DELAYED_WORK(&mds->security.poll_dwork, cxl_mbox_sanitize_work); + dev_dbg(cxlds->dev, "Mailbox interrupts enabled\n"); + /* enable background command mbox irq support */ + ctrl = readl(cxlds->regs.mbox + CXLDEV_MBOX_CTRL_OFFSET); + ctrl |= CXLDEV_MBOX_CTRL_BG_CMD_IRQ; + writel(ctrl, cxlds->regs.mbox + CXLDEV_MBOX_CTRL_OFFSET); - dev_dbg(cxlds->dev, "Mailbox interrupts are unsupported"); return 0; } From patchwork Fri Oct 6 07:26:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 13411005 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5E94663DB for ; Fri, 6 Oct 2023 07:26:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="FUe8VaNN" Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AB693E9 for ; Fri, 6 Oct 2023 00:26:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1696577177; x=1728113177; h=subject:from:to:date:message-id:in-reply-to:references: mime-version:content-transfer-encoding; bh=Zr+JwxZCArcqLL0dP/lorSnIlxk60mkDmj3jKpQIRcE=; b=FUe8VaNNZhLz1Uigo030Bo7APQohGngTiSLGPfCpu2ULotGkMMtCdsQg GUTuZ+6Qjjwm+oYxyV+FYY2440ctQ82KOgUcfwkwuZATyGWegxRTEOJHi S74dKfEdc4IIPhdS9fvtPXuq0d8CAi4RRQCwXq7LeiMS4zAj9O3CuAXaY XYUOxW5f5jAblpnD+nLXje2YX37drUw0Ia1vEiaQqTGDdiJ+NFwwVcB3p jMjea+BE+JwziADknqqfMWhOYvc0FeAE+nMZ73PgzmuH7N6rOxxdC3X9d n+eZYSEAiSsFiz1fn6DU+YEZhLF8+7IWy47u+pKdDBWTSOVjwebiUihps g==; X-IronPort-AV: E=McAfee;i="6600,9927,10854"; a="450192221" X-IronPort-AV: E=Sophos;i="6.03,203,1694761200"; d="scan'208";a="450192221" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 00:26:17 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10854"; a="817923196" X-IronPort-AV: E=Sophos;i="6.03,203,1694761200"; d="scan'208";a="817923196" Received: from wbleichn-mobl.amr.corp.intel.com (HELO dwillia2-xfh.jf.intel.com) ([10.212.147.24]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 00:26:16 -0700 Subject: [PATCH v3 03/10] cxl/pci: Remove hardirq handler for cxl_request_irq() From: Dan Williams To: linux-cxl@vger.kernel.org Date: Fri, 06 Oct 2023 00:26:16 -0700 Message-ID: <169657717635.1491153.10040969889268437662.stgit@dwillia2-xfh.jf.intel.com> In-Reply-To: <169657715790.1491153.3612164287133860191.stgit@dwillia2-xfh.jf.intel.com> References: <169657715790.1491153.3612164287133860191.stgit@dwillia2-xfh.jf.intel.com> User-Agent: StGit/0.18-3-g996c Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Now that all callers of cxl_request_irq() are using threaded irqs, drop the hardirq handler option. Signed-off-by: Dan Williams Reviewed-by: Davidlohr Bueso Reviewed-by: Ira Weiny Reviewed-by: Jonathan Cameron Reviewed-by: Dave Jiang --- drivers/cxl/pci.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index 49d9b2ef5c5c..dc665b12be8f 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -90,7 +90,7 @@ struct cxl_dev_id { }; static int cxl_request_irq(struct cxl_dev_state *cxlds, int irq, - irq_handler_t handler, irq_handler_t thread_fn) + irq_handler_t thread_fn) { struct device *dev = cxlds->dev; struct cxl_dev_id *dev_id; @@ -101,9 +101,9 @@ static int cxl_request_irq(struct cxl_dev_state *cxlds, int irq, return -ENOMEM; dev_id->cxlds = cxlds; - return devm_request_threaded_irq(dev, irq, handler, thread_fn, - IRQF_SHARED | IRQF_ONESHOT, - NULL, dev_id); + return devm_request_threaded_irq(dev, irq, NULL, thread_fn, + IRQF_SHARED | IRQF_ONESHOT, NULL, + dev_id); } static bool cxl_mbox_background_complete(struct cxl_dev_state *cxlds) @@ -440,7 +440,7 @@ static int cxl_pci_setup_mailbox(struct cxl_memdev_state *mds) if (irq < 0) return 0; - if (cxl_request_irq(cxlds, irq, NULL, cxl_pci_mbox_irq)) + if (cxl_request_irq(cxlds, irq, cxl_pci_mbox_irq)) return 0; dev_dbg(cxlds->dev, "Mailbox interrupts enabled\n"); @@ -638,7 +638,7 @@ static int cxl_event_req_irq(struct cxl_dev_state *cxlds, u8 setting) if (irq < 0) return irq; - return cxl_request_irq(cxlds, irq, NULL, cxl_event_thread); + return cxl_request_irq(cxlds, irq, cxl_event_thread); } static int cxl_event_get_int_policy(struct cxl_memdev_state *mds, From patchwork Fri Oct 6 07:26:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 13411006 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2AA8763DD for ; Fri, 6 Oct 2023 07:26:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="HM8jTEPW" Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 74AC6CA for ; Fri, 6 Oct 2023 00:26:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1696577183; x=1728113183; h=subject:from:to:date:message-id:in-reply-to:references: mime-version:content-transfer-encoding; bh=tkFaQhN+VJ0Xplg8crH2zRv6PXnDOOnympUFHN3vBqs=; b=HM8jTEPWyebDasBU/Mq/4LbihVI06x0vSoKmSRedkxYvwYV9wmcoJZCK D4zQ7jeJOZJ5L83eMCiyKklkT1q91Oz0f2P1pEDD+5y6cK/wUOUt5YWRo yiGqsFwOuZ5UzY/14yI0ImWyFgwV28LZUFEA83H8HphztEVSVfNHupDpQ wprq89NIEz+qu4I4Vd/VXTa3lrbhj0FOTK8G5CL10KECwd/lHolDM2RvM MH55ubxiDNDf0NO5Vft7p7r7CEHwe6j8WAEnR7LN8zrq9IMxWSmW+gm2q PaCUqkzsG8keLPqdGhkmRZqsAxD3/k+wrJVWlpZYxnLDwPsBP6XJlXw50 A==; X-IronPort-AV: E=McAfee;i="6600,9927,10854"; a="450192229" X-IronPort-AV: E=Sophos;i="6.03,203,1694761200"; d="scan'208";a="450192229" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 00:26:23 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10854"; a="817923204" X-IronPort-AV: E=Sophos;i="6.03,203,1694761200"; d="scan'208";a="817923204" Received: from wbleichn-mobl.amr.corp.intel.com (HELO dwillia2-xfh.jf.intel.com) ([10.212.147.24]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 00:26:22 -0700 Subject: [PATCH v3 04/10] cxl/pci: Remove inconsistent usage of dev_err_probe() From: Dan Williams To: linux-cxl@vger.kernel.org Date: Fri, 06 Oct 2023 00:26:22 -0700 Message-ID: <169657718224.1491153.11083952431204965077.stgit@dwillia2-xfh.jf.intel.com> In-Reply-To: <169657715790.1491153.3612164287133860191.stgit@dwillia2-xfh.jf.intel.com> References: <169657715790.1491153.3612164287133860191.stgit@dwillia2-xfh.jf.intel.com> User-Agent: StGit/0.18-3-g996c Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net If dev_err_probe() is to be used it should at least be used consistently within the same function. It is also worth questioning whether every potential -ENOMEM needs an explicit error message. Remove the cxl_setup_fw_upload() error prints for what are rare / hardware-independent failures. Signed-off-by: Dan Williams Reviewed-by: Davidlohr Bueso Reviewed-by: Ira Weiny Reviewed-by: Jonathan Cameron Reviewed-by: Dave Jiang --- drivers/cxl/core/memdev.c | 13 ++----------- 1 file changed, 2 insertions(+), 11 deletions(-) diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c index 2a7a07f6d165..6efe4e2a2cf5 100644 --- a/drivers/cxl/core/memdev.c +++ b/drivers/cxl/core/memdev.c @@ -970,7 +970,6 @@ int cxl_memdev_setup_fw_upload(struct cxl_memdev_state *mds) struct cxl_dev_state *cxlds = &mds->cxlds; struct device *dev = &cxlds->cxlmd->dev; struct fw_upload *fwl; - int rc; if (!test_bit(CXL_MEM_COMMAND_ID_GET_FW_INFO, mds->enabled_cmds)) return 0; @@ -978,17 +977,9 @@ int cxl_memdev_setup_fw_upload(struct cxl_memdev_state *mds) fwl = firmware_upload_register(THIS_MODULE, dev, dev_name(dev), &cxl_memdev_fw_ops, mds); if (IS_ERR(fwl)) - return dev_err_probe(dev, PTR_ERR(fwl), - "Failed to register firmware loader\n"); - - rc = devm_add_action_or_reset(cxlds->dev, devm_cxl_remove_fw_upload, - fwl); - if (rc) - dev_err(dev, - "Failed to add firmware loader remove action: %d\n", - rc); + return PTR_ERR(fwl); - return rc; + return devm_add_action_or_reset(cxlds->dev, devm_cxl_remove_fw_upload, fwl); } EXPORT_SYMBOL_NS_GPL(cxl_memdev_setup_fw_upload, CXL); From patchwork Fri Oct 6 07:26:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 13411007 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BCC776AA2 for ; Fri, 6 Oct 2023 07:26:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="HVd8Bxnc" Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3F47EE4 for ; Fri, 6 Oct 2023 00:26:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1696577189; x=1728113189; h=subject:from:to:date:message-id:in-reply-to:references: mime-version:content-transfer-encoding; bh=WUVCTriV4DblB6HFbzHOc/CfOxuFpWx3xEn/9moPfQo=; b=HVd8BxncmJwJiNvfKF3tSz35RLaKFrEA/JscP4x6OC4La5FXxq+HtAzX eg3D9fkRNRKGICtbjQFTB+m6vBpZT6AGofS4yurlvkVvx52oOzix2vxCi v8wIj6EH1lAqUGAH6zlzBdIX60cqLuRPAswN3Ax1r6tiW0GPOZh42rrDX ke7np840bZ2dRsa2ZXkiYCzf5mMeLiiknIS3jlqqil13qLxCQbPHlLx5K GcoB9JhbtOe5x9F5sVS6lE/5kITGVVelub+j+oTJx3vcR9uI5lTKuOBig mmLdx24+4wJ/V9cCulKgCk57/AMOqO2NQEiGn05bA6geM+rCKt3I2I2dl g==; X-IronPort-AV: E=McAfee;i="6600,9927,10854"; a="450192237" X-IronPort-AV: E=Sophos;i="6.03,203,1694761200"; d="scan'208";a="450192237" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 00:26:29 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10854"; a="817923219" X-IronPort-AV: E=Sophos;i="6.03,203,1694761200"; d="scan'208";a="817923219" Received: from wbleichn-mobl.amr.corp.intel.com (HELO dwillia2-xfh.jf.intel.com) ([10.212.147.24]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 00:26:28 -0700 Subject: [PATCH v3 05/10] cxl/pci: Clarify devm host for memdev relative setup From: Dan Williams To: linux-cxl@vger.kernel.org Date: Fri, 06 Oct 2023 00:26:28 -0700 Message-ID: <169657718805.1491153.2457900463470349092.stgit@dwillia2-xfh.jf.intel.com> In-Reply-To: <169657715790.1491153.3612164287133860191.stgit@dwillia2-xfh.jf.intel.com> References: <169657715790.1491153.3612164287133860191.stgit@dwillia2-xfh.jf.intel.com> User-Agent: StGit/0.18-3-g996c Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net It is all too easy to get confused about @dev usage in the CXL driver stack. Before adding a new cxl_pci_probe() setup operation that has a devm lifetime dependent on @cxlds->dev binding, but also references @cxlmd->dev, and prints messages, rework the devm_cxl_add_memdev() and cxl_memdev_setup_fw_upload() function signatures to make this distinction explicit. I.e. pass in the devm context as an @host argument rather than infer it from other objects. This is in preparation for adding a devm_cxl_sanitize_setup_notifier(). Note the whitespace fixup near the change of the devm_cxl_add_memdev() signature. That uncaught typo originated in the patch that added cxl_memdev_security_init(). Signed-off-by: Dan Williams Reviewed-by: Ira Weiny Reviewed-by: Jonathan Cameron Reviewed-by: Dave Jiang --- drivers/cxl/core/memdev.c | 16 ++++++++-------- drivers/cxl/cxlmem.h | 5 +++-- drivers/cxl/pci.c | 4 ++-- tools/testing/cxl/test/mem.c | 4 ++-- 4 files changed, 15 insertions(+), 14 deletions(-) diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c index 6efe4e2a2cf5..63353d990374 100644 --- a/drivers/cxl/core/memdev.c +++ b/drivers/cxl/core/memdev.c @@ -960,12 +960,12 @@ static const struct fw_upload_ops cxl_memdev_fw_ops = { .cleanup = cxl_fw_cleanup, }; -static void devm_cxl_remove_fw_upload(void *fwl) +static void cxl_remove_fw_upload(void *fwl) { firmware_upload_unregister(fwl); } -int cxl_memdev_setup_fw_upload(struct cxl_memdev_state *mds) +int devm_cxl_setup_fw_upload(struct device *host, struct cxl_memdev_state *mds) { struct cxl_dev_state *cxlds = &mds->cxlds; struct device *dev = &cxlds->cxlmd->dev; @@ -978,10 +978,9 @@ int cxl_memdev_setup_fw_upload(struct cxl_memdev_state *mds) &cxl_memdev_fw_ops, mds); if (IS_ERR(fwl)) return PTR_ERR(fwl); - - return devm_add_action_or_reset(cxlds->dev, devm_cxl_remove_fw_upload, fwl); + return devm_add_action_or_reset(host, cxl_remove_fw_upload, fwl); } -EXPORT_SYMBOL_NS_GPL(cxl_memdev_setup_fw_upload, CXL); +EXPORT_SYMBOL_NS_GPL(devm_cxl_setup_fw_upload, CXL); static const struct file_operations cxl_memdev_fops = { .owner = THIS_MODULE, @@ -1019,9 +1018,10 @@ static int cxl_memdev_security_init(struct cxl_memdev *cxlmd) } return devm_add_action_or_reset(cxlds->dev, put_sanitize, mds); - } +} -struct cxl_memdev *devm_cxl_add_memdev(struct cxl_dev_state *cxlds) +struct cxl_memdev *devm_cxl_add_memdev(struct device *host, + struct cxl_dev_state *cxlds) { struct cxl_memdev *cxlmd; struct device *dev; @@ -1053,7 +1053,7 @@ struct cxl_memdev *devm_cxl_add_memdev(struct cxl_dev_state *cxlds) if (rc) goto err; - rc = devm_add_action_or_reset(cxlds->dev, cxl_memdev_unregister, cxlmd); + rc = devm_add_action_or_reset(host, cxl_memdev_unregister, cxlmd); if (rc) return ERR_PTR(rc); return cxlmd; diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h index 55f00ad17a77..fdb2c8dd98d0 100644 --- a/drivers/cxl/cxlmem.h +++ b/drivers/cxl/cxlmem.h @@ -84,9 +84,10 @@ static inline bool is_cxl_endpoint(struct cxl_port *port) return is_cxl_memdev(port->uport_dev); } -struct cxl_memdev *devm_cxl_add_memdev(struct cxl_dev_state *cxlds); +struct cxl_memdev *devm_cxl_add_memdev(struct device *host, + struct cxl_dev_state *cxlds); struct cxl_memdev_state; -int cxl_memdev_setup_fw_upload(struct cxl_memdev_state *mds); +int devm_cxl_setup_fw_upload(struct device *host, struct cxl_memdev_state *mds); int devm_cxl_dpa_reserve(struct cxl_endpoint_decoder *cxled, resource_size_t base, resource_size_t len, resource_size_t skipped); diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index dc665b12be8f..7c117eb62750 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -867,11 +867,11 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) if (rc) return rc; - cxlmd = devm_cxl_add_memdev(cxlds); + cxlmd = devm_cxl_add_memdev(&pdev->dev, cxlds); if (IS_ERR(cxlmd)) return PTR_ERR(cxlmd); - rc = cxl_memdev_setup_fw_upload(mds); + rc = devm_cxl_setup_fw_upload(&pdev->dev, mds); if (rc) return rc; diff --git a/tools/testing/cxl/test/mem.c b/tools/testing/cxl/test/mem.c index 464fc39ed277..68118c37f0b5 100644 --- a/tools/testing/cxl/test/mem.c +++ b/tools/testing/cxl/test/mem.c @@ -1450,11 +1450,11 @@ static int cxl_mock_mem_probe(struct platform_device *pdev) mdata->mes.mds = mds; cxl_mock_add_event_logs(&mdata->mes); - cxlmd = devm_cxl_add_memdev(cxlds); + cxlmd = devm_cxl_add_memdev(&pdev->dev, cxlds); if (IS_ERR(cxlmd)) return PTR_ERR(cxlmd); - rc = cxl_memdev_setup_fw_upload(mds); + rc = devm_cxl_setup_fw_upload(&pdev->dev, mds); if (rc) return rc; From patchwork Fri Oct 6 07:26:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 13411008 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0F5FF6D19 for ; Fri, 6 Oct 2023 07:26:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="aK9vLfO4" Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6714BE4 for ; Fri, 6 Oct 2023 00:26:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1696577195; x=1728113195; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=c5jZ2PPbXGifXbQhbdT0FI+5T+XU7dzV7zgnMFdn5q0=; b=aK9vLfO4WH/SkZA5oXgMs31q0cDiN972Y3MkiarrSmd4+O7IiSG+RsUr ewhSUxBcDIJ/Qh9HB5fwdIVY3ZGbDgC7M5qnUnVI1m+07vTWnRIzpcbzZ YvErjGH9tmzsMYXMHcjPJQuZgA9ixJSSGrgFmEMOFsCzXuoofjo9wdxJd QXzD9Eqyd4elm3dHjv5eoSErogYx9TB4BQ5jC6c5oX9wcHeJjqn5K9Aiy 1V1tWnltKj7DlJ7SwQg/6H3RynWBgTGygAAl/p/lGXFrMeFoJ418Lc7Zg UPH5THHYT03K2m/jKlsIO6f8zP9HJJC6rRgQEu991OBgzoBVir80gFrpp w==; X-IronPort-AV: E=McAfee;i="6600,9927,10854"; a="450192246" X-IronPort-AV: E=Sophos;i="6.03,203,1694761200"; d="scan'208";a="450192246" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 00:26:35 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10854"; a="817923228" X-IronPort-AV: E=Sophos;i="6.03,203,1694761200"; d="scan'208";a="817923228" Received: from wbleichn-mobl.amr.corp.intel.com (HELO dwillia2-xfh.jf.intel.com) ([10.212.147.24]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 00:26:34 -0700 Subject: [PATCH v3 06/10] cxl/pci: Fix sanitize notifier setup From: Dan Williams To: linux-cxl@vger.kernel.org Cc: Jonathan Cameron , Dave Jiang , Davidlohr Bueso , Dave Jiang , Ira Weiny Date: Fri, 06 Oct 2023 00:26:33 -0700 Message-ID: <169657719381.1491153.1571837747024067070.stgit@dwillia2-xfh.jf.intel.com> In-Reply-To: <169657715790.1491153.3612164287133860191.stgit@dwillia2-xfh.jf.intel.com> References: <169657715790.1491153.3612164287133860191.stgit@dwillia2-xfh.jf.intel.com> User-Agent: StGit/0.18-3-g996c Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Fix a race condition between the mailbox-background command interrupt firing and the security-state sysfs attribute being removed. The race is difficult to see due to the awkward placement of the sanitize-notifier setup code and the multiple places the teardown calls are made, cxl_memdev_security_init() and cxl_memdev_security_shutdown(). Unify setup in one place, cxl_sanitize_setup_notifier(). Arrange for the paired cxl_sanitize_teardown_notifier() to safely quiet the notifier and let the cxl_memdev + irq be unregistered later in the flow. Note: The special wrinkle of the sanitize notifier is that it interacts with interrupts, which are enabled early in the flow, and it interacts with memdev sysfs which is not initialized until late in the flow. Hence why this setup routine takes an @cxlmd argument, and not just @mds. This fix is also needed as a preparation fix for a memdev unregistration crash. Reported-by: Jonathan Cameron Closes: http://lore.kernel.org/r/20230929100316.00004546@Huawei.com Cc: Dave Jiang Cc: Davidlohr Bueso Fixes: 0c36b6ad436a ("cxl/mbox: Add sanitization handling machinery") Reviewed-by: Dave Jiang Reviewed-by: Ira Weiny Signed-off-by: Dan Williams Reviewed-by: Jonathan Cameron Reviewed-by: Davidlohr Bueso --- drivers/cxl/core/memdev.c | 86 +++++++++++++++++++++++---------------------- drivers/cxl/cxlmem.h | 2 + drivers/cxl/pci.c | 4 ++ 3 files changed, 50 insertions(+), 42 deletions(-) diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c index 63353d990374..4c2e24a1a89c 100644 --- a/drivers/cxl/core/memdev.c +++ b/drivers/cxl/core/memdev.c @@ -556,20 +556,11 @@ void clear_exclusive_cxl_commands(struct cxl_memdev_state *mds, } EXPORT_SYMBOL_NS_GPL(clear_exclusive_cxl_commands, CXL); -static void cxl_memdev_security_shutdown(struct device *dev) -{ - struct cxl_memdev *cxlmd = to_cxl_memdev(dev); - struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlmd->cxlds); - - cancel_delayed_work_sync(&mds->security.poll_dwork); -} - static void cxl_memdev_shutdown(struct device *dev) { struct cxl_memdev *cxlmd = to_cxl_memdev(dev); down_write(&cxl_memdev_rwsem); - cxl_memdev_security_shutdown(dev); cxlmd->cxlds = NULL; up_write(&cxl_memdev_rwsem); } @@ -991,35 +982,6 @@ static const struct file_operations cxl_memdev_fops = { .llseek = noop_llseek, }; -static void put_sanitize(void *data) -{ - struct cxl_memdev_state *mds = data; - - sysfs_put(mds->security.sanitize_node); -} - -static int cxl_memdev_security_init(struct cxl_memdev *cxlmd) -{ - struct cxl_dev_state *cxlds = cxlmd->cxlds; - struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlds); - struct device *dev = &cxlmd->dev; - struct kernfs_node *sec; - - sec = sysfs_get_dirent(dev->kobj.sd, "security"); - if (!sec) { - dev_err(dev, "sysfs_get_dirent 'security' failed\n"); - return -ENODEV; - } - mds->security.sanitize_node = sysfs_get_dirent(sec, "state"); - sysfs_put(sec); - if (!mds->security.sanitize_node) { - dev_err(dev, "sysfs_get_dirent 'state' failed\n"); - return -ENODEV; - } - - return devm_add_action_or_reset(cxlds->dev, put_sanitize, mds); -} - struct cxl_memdev *devm_cxl_add_memdev(struct device *host, struct cxl_dev_state *cxlds) { @@ -1049,10 +1011,6 @@ struct cxl_memdev *devm_cxl_add_memdev(struct device *host, if (rc) goto err; - rc = cxl_memdev_security_init(cxlmd); - if (rc) - goto err; - rc = devm_add_action_or_reset(host, cxl_memdev_unregister, cxlmd); if (rc) return ERR_PTR(rc); @@ -1069,6 +1027,50 @@ struct cxl_memdev *devm_cxl_add_memdev(struct device *host, } EXPORT_SYMBOL_NS_GPL(devm_cxl_add_memdev, CXL); +static void sanitize_teardown_notifier(void *data) +{ + struct cxl_memdev_state *mds = data; + struct kernfs_node *state; + + /* + * Prevent new irq triggered invocations of the workqueue and + * flush inflight invocations. + */ + mutex_lock(&mds->mbox_mutex); + state = mds->security.sanitize_node; + mds->security.sanitize_node = NULL; + mutex_unlock(&mds->mbox_mutex); + + cancel_delayed_work_sync(&mds->security.poll_dwork); + sysfs_put(state); +} + +int devm_cxl_sanitize_setup_notifier(struct device *host, + struct cxl_memdev *cxlmd) +{ + struct cxl_dev_state *cxlds = cxlmd->cxlds; + struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlds); + struct kernfs_node *sec; + + if (!test_bit(CXL_SEC_ENABLED_SANITIZE, mds->security.enabled_cmds)) + return 0; + + /* + * Note, the expectation is that @cxlmd would have failed to be + * created if these sysfs_get_dirent calls fail. + */ + sec = sysfs_get_dirent(cxlmd->dev.kobj.sd, "security"); + if (!sec) + return -ENOENT; + mds->security.sanitize_node = sysfs_get_dirent(sec, "state"); + sysfs_put(sec); + if (!mds->security.sanitize_node) + return -ENOENT; + + return devm_add_action_or_reset(host, sanitize_teardown_notifier, mds); +} +EXPORT_SYMBOL_NS_GPL(devm_cxl_sanitize_setup_notifier, CXL); + __init int cxl_memdev_init(void) { dev_t devt; diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h index fdb2c8dd98d0..fbdee1d63717 100644 --- a/drivers/cxl/cxlmem.h +++ b/drivers/cxl/cxlmem.h @@ -86,6 +86,8 @@ static inline bool is_cxl_endpoint(struct cxl_port *port) struct cxl_memdev *devm_cxl_add_memdev(struct device *host, struct cxl_dev_state *cxlds); +int devm_cxl_sanitize_setup_notifier(struct device *host, + struct cxl_memdev *cxlmd); struct cxl_memdev_state; int devm_cxl_setup_fw_upload(struct device *host, struct cxl_memdev_state *mds); int devm_cxl_dpa_reserve(struct cxl_endpoint_decoder *cxled, diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index 7c117eb62750..9955871e9ec1 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -875,6 +875,10 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) if (rc) return rc; + rc = devm_cxl_sanitize_setup_notifier(&pdev->dev, cxlmd); + if (rc) + return rc; + pmu_count = cxl_count_regblock(pdev, CXL_REGLOC_RBI_PMU); for (i = 0; i < pmu_count; i++) { struct cxl_pmu_regs pmu_regs; From patchwork Fri Oct 6 07:26:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 13411009 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EA4476FA8 for ; Fri, 6 Oct 2023 07:26:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="QKpYqfzU" Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5CE0AED for ; Fri, 6 Oct 2023 00:26:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1696577201; x=1728113201; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=zWyu66MtmZiYjHq9JGEEwh3Ia+iv4XGWn3EsEjLoyBs=; b=QKpYqfzUDhsyqVSk2M8AWF/ZJC3xtfobGB9z6bp8XIqshoMkeLTbc3Ly +Y0teEB1bRydLEyW87Gt0rFc2Clnh4IauZu2unPN4tZDgI08e9RM0v5Qu 8Vbo3HjpubJyNgAYVWQ8OE7OAtuRBnagZ0SifIW/cbehD2z76L7GIx2OU 6zYzJgJEhWyRY2H0uRHvlsrNxOMeORzk4r1HOaMrytS4cSYfMCLAVQxqJ QAywc03DyU4O3r0GyhYQatuwNpU0FNBcchn1MRw1NCVnISLVDia6PGp8y OQRCd77bGr4qY/NbDk4UGfgKgsSImLOrE9eb0XR/DHQyZaYl/77Praq6q Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10854"; a="450192254" X-IronPort-AV: E=Sophos;i="6.03,203,1694761200"; d="scan'208";a="450192254" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 00:26:40 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10854"; a="817923232" X-IronPort-AV: E=Sophos;i="6.03,203,1694761200"; d="scan'208";a="817923232" Received: from wbleichn-mobl.amr.corp.intel.com (HELO dwillia2-xfh.jf.intel.com) ([10.212.147.24]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 00:26:40 -0700 Subject: [PATCH v3 07/10] cxl/memdev: Fix sanitize vs decoder setup locking From: Dan Williams To: linux-cxl@vger.kernel.org Cc: Davidlohr Bueso Date: Fri, 06 Oct 2023 00:26:39 -0700 Message-ID: <169657719974.1491153.15276451196916291864.stgit@dwillia2-xfh.jf.intel.com> In-Reply-To: <169657715790.1491153.3612164287133860191.stgit@dwillia2-xfh.jf.intel.com> References: <169657715790.1491153.3612164287133860191.stgit@dwillia2-xfh.jf.intel.com> User-Agent: StGit/0.18-3-g996c Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net The sanitize operation is destructive and the expectation is that the device is unmapped while in progress. The current implementation does a lockless check for decoders being active, but then does nothing to prevent decoders from racing to be committed. Introduce state tracking to resolve this race. This incidentally cleans up unpriveleged userspace from triggering mmio read cycles by spinning on reading the 'securiry/state' attribute. Which at a minimum is a waste since the kernel state machine can cache the completion result. Lastly cxl_mem_sanitize() was mistakenly marked EXPORT_SYMBOL() in the original implementation, but an export was never required. Fixes: 0c36b6ad436a ("cxl/mbox: Add sanitization handling machinery") Cc: Davidlohr Bueso Signed-off-by: Dan Williams Reviewed-by: Davidlohr Bueso Reviewed-by: Jonathan Cameron Reviewed-by: Dave Jiang --- drivers/cxl/core/core.h | 1 + drivers/cxl/core/hdm.c | 19 ++++++++++++++++ drivers/cxl/core/mbox.c | 55 +++++++++++++++++++++++++++++++++------------ drivers/cxl/core/memdev.c | 43 +++++++++++++---------------------- drivers/cxl/core/port.c | 6 +++++ drivers/cxl/core/region.c | 6 ----- drivers/cxl/cxlmem.h | 4 ++- drivers/cxl/pci.c | 5 ++++ 8 files changed, 90 insertions(+), 49 deletions(-) diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h index 45e7e044cf4a..8e5f3d84311e 100644 --- a/drivers/cxl/core/core.h +++ b/drivers/cxl/core/core.h @@ -75,6 +75,7 @@ resource_size_t __rcrb_to_component(struct device *dev, enum cxl_rcrb which); extern struct rw_semaphore cxl_dpa_rwsem; +extern struct rw_semaphore cxl_region_rwsem; int cxl_memdev_init(void); void cxl_memdev_exit(void); diff --git a/drivers/cxl/core/hdm.c b/drivers/cxl/core/hdm.c index 4449b34a80cc..506c9e14cdf9 100644 --- a/drivers/cxl/core/hdm.c +++ b/drivers/cxl/core/hdm.c @@ -650,6 +650,25 @@ static int cxl_decoder_commit(struct cxl_decoder *cxld) return -EBUSY; } + /* + * For endpoint decoders hosted on CXL memory devices that + * support the sanitize operation, make sure sanitize is not in-flight. + */ + if (is_endpoint_decoder(&cxld->dev)) { + struct cxl_endpoint_decoder *cxled = + to_cxl_endpoint_decoder(&cxld->dev); + struct cxl_memdev *cxlmd = cxled_to_memdev(cxled); + struct cxl_memdev_state *mds = + to_cxl_memdev_state(cxlmd->cxlds); + + if (mds && mds->security.sanitize_active) { + dev_dbg(&cxlmd->dev, + "attempted to commit %s during sanitize\n", + dev_name(&cxld->dev)); + return -EBUSY; + } + } + down_read(&cxl_dpa_rwsem); /* common decoder settings */ ctrl = readl(hdm + CXL_HDM_DECODER0_CTRL_OFFSET(cxld->id)); diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c index 4df4f614f490..67aec57cc12f 100644 --- a/drivers/cxl/core/mbox.c +++ b/drivers/cxl/core/mbox.c @@ -1125,20 +1125,7 @@ int cxl_dev_state_identify(struct cxl_memdev_state *mds) } EXPORT_SYMBOL_NS_GPL(cxl_dev_state_identify, CXL); -/** - * cxl_mem_sanitize() - Send a sanitization command to the device. - * @mds: The device data for the operation - * @cmd: The specific sanitization command opcode - * - * Return: 0 if the command was executed successfully, regardless of - * whether or not the actual security operation is done in the background, - * such as for the Sanitize case. - * Error return values can be the result of the mailbox command, -EINVAL - * when security requirements are not met or invalid contexts. - * - * See CXL 3.0 @8.2.9.8.5.1 Sanitize and @8.2.9.8.5.2 Secure Erase. - */ -int cxl_mem_sanitize(struct cxl_memdev_state *mds, u16 cmd) +static int __cxl_mem_sanitize(struct cxl_memdev_state *mds, u16 cmd) { int rc; u32 sec_out = 0; @@ -1183,7 +1170,45 @@ int cxl_mem_sanitize(struct cxl_memdev_state *mds, u16 cmd) return 0; } -EXPORT_SYMBOL_NS_GPL(cxl_mem_sanitize, CXL); + + +/** + * cxl_mem_sanitize() - Send a sanitization command to the device. + * @mds: The device for the operation + * @cmd: The specific sanitization command opcode + * + * Return: 0 if the command was executed successfully, regardless of + * whether or not the actual security operation is done in the background, + * such as for the Sanitize case. + * Error return values can be the result of the mailbox command, -EINVAL + * when security requirements are not met or invalid contexts, or -EBUSY + * if the sanitize operation is already in flight. + * + * See CXL 3.0 @8.2.9.8.5.1 Sanitize and @8.2.9.8.5.2 Secure Erase. + */ +int cxl_mem_sanitize(struct cxl_memdev *cxlmd, u16 cmd) +{ + struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlmd->cxlds); + struct cxl_port *endpoint; + int rc; + + /* synchronize with cxl_mem_probe() and decoder write operations */ + device_lock(&cxlmd->dev); + endpoint = cxlmd->endpoint; + down_read(&cxl_region_rwsem); + /* + * Require an endpoint to be safe otherwise the driver can not + * be sure that the device is unmapped. + */ + if (endpoint && endpoint->commit_end == -1) + rc = __cxl_mem_sanitize(mds, cmd); + else + rc = -EBUSY; + up_read(&cxl_region_rwsem); + device_unlock(&cxlmd->dev); + + return rc; +} static int add_dpa_res(struct device *dev, struct resource *parent, struct resource *res, resource_size_t start, diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c index 4c2e24a1a89c..a02061028b71 100644 --- a/drivers/cxl/core/memdev.c +++ b/drivers/cxl/core/memdev.c @@ -125,13 +125,16 @@ static ssize_t security_state_show(struct device *dev, struct cxl_memdev *cxlmd = to_cxl_memdev(dev); struct cxl_dev_state *cxlds = cxlmd->cxlds; struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlds); - u64 reg = readq(cxlds->regs.mbox + CXLDEV_MBOX_BG_CMD_STATUS_OFFSET); - u32 pct = FIELD_GET(CXLDEV_MBOX_BG_CMD_COMMAND_PCT_MASK, reg); - u16 cmd = FIELD_GET(CXLDEV_MBOX_BG_CMD_COMMAND_OPCODE_MASK, reg); unsigned long state = mds->security.state; + int rc = 0; - if (cmd == CXL_MBOX_OP_SANITIZE && pct != 100) - return sysfs_emit(buf, "sanitize\n"); + /* sync with latest submission state */ + mutex_lock(&mds->mbox_mutex); + if (mds->security.sanitize_active) + rc = sysfs_emit(buf, "sanitize\n"); + mutex_unlock(&mds->mbox_mutex); + if (rc) + return rc; if (!(state & CXL_PMEM_SEC_STATE_USER_PASS_SET)) return sysfs_emit(buf, "disabled\n"); @@ -152,24 +155,17 @@ static ssize_t security_sanitize_store(struct device *dev, const char *buf, size_t len) { struct cxl_memdev *cxlmd = to_cxl_memdev(dev); - struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlmd->cxlds); - struct cxl_port *port = cxlmd->endpoint; bool sanitize; ssize_t rc; if (kstrtobool(buf, &sanitize) || !sanitize) return -EINVAL; - if (!port || !is_cxl_endpoint(port)) - return -EINVAL; - - /* ensure no regions are mapped to this memdev */ - if (port->commit_end != -1) - return -EBUSY; - - rc = cxl_mem_sanitize(mds, CXL_MBOX_OP_SANITIZE); + rc = cxl_mem_sanitize(cxlmd, CXL_MBOX_OP_SANITIZE); + if (rc) + return rc; - return rc ? rc : len; + return len; } static struct device_attribute dev_attr_security_sanitize = __ATTR(sanitize, 0200, NULL, security_sanitize_store); @@ -179,24 +175,17 @@ static ssize_t security_erase_store(struct device *dev, const char *buf, size_t len) { struct cxl_memdev *cxlmd = to_cxl_memdev(dev); - struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlmd->cxlds); - struct cxl_port *port = cxlmd->endpoint; ssize_t rc; bool erase; if (kstrtobool(buf, &erase) || !erase) return -EINVAL; - if (!port || !is_cxl_endpoint(port)) - return -EINVAL; - - /* ensure no regions are mapped to this memdev */ - if (port->commit_end != -1) - return -EBUSY; - - rc = cxl_mem_sanitize(mds, CXL_MBOX_OP_SECURE_ERASE); + rc = cxl_mem_sanitize(cxlmd, CXL_MBOX_OP_SECURE_ERASE); + if (rc) + return rc; - return rc ? rc : len; + return len; } static struct device_attribute dev_attr_security_erase = __ATTR(erase, 0200, NULL, security_erase_store); diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c index 7ca01a834e18..5ba606c6e03f 100644 --- a/drivers/cxl/core/port.c +++ b/drivers/cxl/core/port.c @@ -28,6 +28,12 @@ * instantiated by the core. */ +/* + * All changes to the interleave configuration occur with this lock held + * for write. + */ +DECLARE_RWSEM(cxl_region_rwsem); + static DEFINE_IDA(cxl_port_ida); static DEFINE_XARRAY(cxl_root_buses); diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c index 6d63b8798c29..d74bf1b664b6 100644 --- a/drivers/cxl/core/region.c +++ b/drivers/cxl/core/region.c @@ -28,12 +28,6 @@ * 3. Decoder targets */ -/* - * All changes to the interleave configuration occur with this lock held - * for write. - */ -static DECLARE_RWSEM(cxl_region_rwsem); - static struct cxl_region *to_cxl_region(struct device *dev); static ssize_t uuid_show(struct device *dev, struct device_attribute *attr, diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h index fbdee1d63717..6933bc20e76b 100644 --- a/drivers/cxl/cxlmem.h +++ b/drivers/cxl/cxlmem.h @@ -364,6 +364,7 @@ struct cxl_fw_state { * @state: state of last security operation * @enabled_cmds: All security commands enabled in the CEL * @poll_tmo_secs: polling timeout + * @sanitize_active: sanitize completion pending * @poll_dwork: polling work item * @sanitize_node: sanitation sysfs file to notify */ @@ -371,6 +372,7 @@ struct cxl_security_state { unsigned long state; DECLARE_BITMAP(enabled_cmds, CXL_SEC_ENABLED_MAX); int poll_tmo_secs; + bool sanitize_active; struct delayed_work poll_dwork; struct kernfs_node *sanitize_node; }; @@ -884,7 +886,7 @@ static inline void cxl_mem_active_dec(void) } #endif -int cxl_mem_sanitize(struct cxl_memdev_state *mds, u16 cmd); +int cxl_mem_sanitize(struct cxl_memdev *cxlmd, u16 cmd); struct cxl_hdm { struct cxl_component_regs regs; diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index 9955871e9ec1..06fafe59c054 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -154,6 +154,7 @@ static void cxl_mbox_sanitize_work(struct work_struct *work) mds->security.poll_tmo_secs = 0; if (mds->security.sanitize_node) sysfs_notify_dirent(mds->security.sanitize_node); + mds->security.sanitize_active = false; dev_dbg(cxlds->dev, "Sanitization operation ended\n"); } else { @@ -292,9 +293,13 @@ static int __cxl_pci_mbox_send_cmd(struct cxl_memdev_state *mds, * and allow userspace to poll(2) for completion. */ if (mbox_cmd->opcode == CXL_MBOX_OP_SANITIZE) { + if (mds->security.sanitize_active) + return -EBUSY; + /* give first timeout a second */ timeout = 1; mds->security.poll_tmo_secs = timeout; + mds->security.sanitize_active = true; schedule_delayed_work(&mds->security.poll_dwork, timeout * HZ); dev_dbg(dev, "Sanitization operation started\n"); From patchwork Fri Oct 6 07:26:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 13411010 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9EECA6FB6 for ; Fri, 6 Oct 2023 07:26:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="W7aAAKOV" Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.126]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0EA67F1 for ; Fri, 6 Oct 2023 00:26:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1696577207; x=1728113207; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=eBR2cmL2NYIOonh4VvZWdAhYxEFKkkvfvPzj5dqaF+4=; b=W7aAAKOVzqViLPsIW1H4mxp7z/sgwVJqjvkKg1FdRoT8AJbcXjIsUKRB W/uiB9pgbgv19ee2gIbDEJ7tpie7DM1a6HL7u1w1YSXnlSBGEDl75JNeF P5ysIoX0XwIckCnTMg6IRC3ejx4P5LxXle8irpJ8ps/u3wqBBDP1x6fv0 yr6WkvTWdLF4xSmrJkVvTofZstpYEqMNewZplTC2fbxFGSWfmNaTtjTXt RzLJ2ZLuZIo0RwFkIV2QuWPHvmW8Ea8G9f8BIze5gSKIRceHSTgbMKRhE zuoQnQqoQUzGK3Z/v3n7eiNR/dmUI2Fix4K/LBnjNmtbam1LMo96VcjfG g==; X-IronPort-AV: E=McAfee;i="6600,9927,10854"; a="368775202" X-IronPort-AV: E=Sophos;i="6.03,203,1694761200"; d="scan'208";a="368775202" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 00:26:46 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10854"; a="842735507" X-IronPort-AV: E=Sophos;i="6.03,203,1694761200"; d="scan'208";a="842735507" Received: from wbleichn-mobl.amr.corp.intel.com (HELO dwillia2-xfh.jf.intel.com) ([10.212.147.24]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 00:26:46 -0700 Subject: [PATCH v3 08/10] cxl/mem: Fix shutdown order From: Dan Williams To: linux-cxl@vger.kernel.org Cc: Ira Weiny , Davidlohr Bueso , Dave Jiang , Jonathan Cameron , Ira Weiny , Ira Weiny Date: Fri, 06 Oct 2023 00:26:45 -0700 Message-ID: <169657720558.1491153.15670462991242849575.stgit@dwillia2-xfh.jf.intel.com> In-Reply-To: <169657715790.1491153.3612164287133860191.stgit@dwillia2-xfh.jf.intel.com> References: <169657715790.1491153.3612164287133860191.stgit@dwillia2-xfh.jf.intel.com> User-Agent: StGit/0.18-3-g996c Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Ira reports that removing cxl_mock_mem causes a crash with the following trace: BUG: kernel NULL pointer dereference, address: 0000000000000044 [..] RIP: 0010:cxl_region_decode_reset+0x7f/0x180 [cxl_core] [..] Call Trace: cxl_region_detach+0xe8/0x210 [cxl_core] cxl_decoder_kill_region+0x27/0x40 [cxl_core] cxld_unregister+0x29/0x40 [cxl_core] devres_release_all+0xb8/0x110 device_unbind_cleanup+0xe/0x70 device_release_driver_internal+0x1d2/0x210 bus_remove_device+0xd7/0x150 device_del+0x155/0x3e0 device_unregister+0x13/0x60 devm_release_action+0x4d/0x90 ? __pfx_unregister_port+0x10/0x10 [cxl_core] delete_endpoint+0x121/0x130 [cxl_core] devres_release_all+0xb8/0x110 device_unbind_cleanup+0xe/0x70 device_release_driver_internal+0x1d2/0x210 bus_remove_device+0xd7/0x150 device_del+0x155/0x3e0 ? lock_release+0x142/0x290 cdev_device_del+0x15/0x50 cxl_memdev_unregister+0x54/0x70 [cxl_core] This crash is due to the clearing out the cxl_memdev's driver context (@cxlds) before the subsystem is done with it. This is ultimately due to the region(s), that this memdev is a member, being torn down and expecting to be able to de-reference @cxlds, like here: static int cxl_region_decode_reset(struct cxl_region *cxlr, int count) ... if (cxlds->rcd) goto endpoint_reset; ... Fix it by keeping the driver context valid until memdev-device unregistration, and subsequently the entire stack of related dependencies, unwinds. Fixes: 9cc238c7a526 ("cxl/pci: Introduce cdevm_file_operations") Reported-by: Ira Weiny Reviewed-by: Davidlohr Bueso Reviewed-by: Dave Jiang Reviewed-by: Jonathan Cameron Reviewed-by: Ira Weiny Tested-by: Ira Weiny Signed-off-by: Dan Williams --- drivers/cxl/core/memdev.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c index a02061028b71..fed9573cf355 100644 --- a/drivers/cxl/core/memdev.c +++ b/drivers/cxl/core/memdev.c @@ -559,8 +559,8 @@ static void cxl_memdev_unregister(void *_cxlmd) struct cxl_memdev *cxlmd = _cxlmd; struct device *dev = &cxlmd->dev; - cxl_memdev_shutdown(dev); cdev_device_del(&cxlmd->cdev, dev); + cxl_memdev_shutdown(dev); put_device(dev); } From patchwork Fri Oct 6 07:26:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 13411011 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DA1FF63CC for ; Fri, 6 Oct 2023 07:27:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="jy0601Vl" Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.126]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2A57FE4 for ; Fri, 6 Oct 2023 00:27:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1696577248; x=1728113248; h=subject:from:to:date:message-id:in-reply-to:references: mime-version:content-transfer-encoding; bh=izer0WWtU2ooSNIjXS5t0PkZBaz2vSUwUJBtOCcSu3c=; b=jy0601VlHUJ3/ntIN2HFzTLq1tKJlSsGNq3eqg7CCaChG5DybDpfltF5 AnPIDhoqkS39nAD82KYqqP3weD6L5E5Zt/g2ETb7wte39zH0SinGnmG5Q GljvLU1GQvTAPR1QcoZn+reYmT56LUs+UuLlZL1zwhysRyWYkJAOewPOv r5GYOS0TCCqPDIGIVUGpIC7QUnk5iAqlxUZ05m0Ge9C6bWUdOSfb6B7A8 HKNYUKHg9YR4bSv9nv31l8cZ2OXT5wsky8qoXpNA4uZ1bt0w0EuT7WyBI aSooWOYkKWKyIkfEWdASIZtf/vQt2LYWTOXFVy9oj+mh4GHO5v7+K1FOj Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10854"; a="368775246" X-IronPort-AV: E=Sophos;i="6.03,203,1694761200"; d="scan'208";a="368775246" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 00:27:27 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10854"; a="842735530" X-IronPort-AV: E=Sophos;i="6.03,203,1694761200"; d="scan'208";a="842735530" Received: from wbleichn-mobl.amr.corp.intel.com (HELO dwillia2-xfh.jf.intel.com) ([10.212.147.24]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 00:26:52 -0700 Subject: [PATCH v3 09/10] tools/testing/cxl: Make cxl_memdev_state available to other command emulation From: Dan Williams To: linux-cxl@vger.kernel.org Date: Fri, 06 Oct 2023 00:26:51 -0700 Message-ID: <169657721157.1491153.10282227094664408585.stgit@dwillia2-xfh.jf.intel.com> In-Reply-To: <169657715790.1491153.3612164287133860191.stgit@dwillia2-xfh.jf.intel.com> References: <169657715790.1491153.3612164287133860191.stgit@dwillia2-xfh.jf.intel.com> User-Agent: StGit/0.18-3-g996c Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Move @mds out of the event specific 'struct mock_event_store' and into the base 'struct cxl_mockmem_data' directly. This is in preparation for enabling cxl_test to exercise the notifier flow for 'sanitize' operation completion. Signed-off-by: Dan Williams Reviewed-by: Ira Weiny Reviewed-by: Dave Jiang --- tools/testing/cxl/test/mem.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tools/testing/cxl/test/mem.c b/tools/testing/cxl/test/mem.c index 68118c37f0b5..ab311b59899a 100644 --- a/tools/testing/cxl/test/mem.c +++ b/tools/testing/cxl/test/mem.c @@ -133,7 +133,6 @@ struct mock_event_log { }; struct mock_event_store { - struct cxl_memdev_state *mds; struct mock_event_log mock_logs[CXL_EVENT_TYPE_MAX]; u32 ev_status; }; @@ -150,6 +149,7 @@ struct cxl_mockmem_data { int user_limit; int master_limit; struct mock_event_store mes; + struct cxl_memdev_state *mds; u8 event_buf[SZ_4K]; u64 timestamp; }; @@ -326,7 +326,7 @@ static void cxl_mock_event_trigger(struct device *dev) event_reset_log(log); } - cxl_mem_get_event_records(mes->mds, mes->ev_status); + cxl_mem_get_event_records(mdata->mds, mes->ev_status); } struct cxl_event_record_raw maint_needed = { @@ -1415,6 +1415,7 @@ static int cxl_mock_mem_probe(struct platform_device *pdev) if (IS_ERR(mds)) return PTR_ERR(mds); + mdata->mds = mds; mds->mbox_send = cxl_mock_mbox_send; mds->payload_size = SZ_4K; mds->event.buf = (struct cxl_get_event_payload *) mdata->event_buf; @@ -1447,7 +1448,6 @@ static int cxl_mock_mem_probe(struct platform_device *pdev) if (rc) return rc; - mdata->mes.mds = mds; cxl_mock_add_event_logs(&mdata->mes); cxlmd = devm_cxl_add_memdev(&pdev->dev, cxlds); From patchwork Fri Oct 6 07:26:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 13411012 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DE35163DB for ; Fri, 6 Oct 2023 07:27:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="Mtfx20aY" Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.126]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8A083E9 for ; Fri, 6 Oct 2023 00:27:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1696577249; x=1728113249; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=KwmZBVREe1rJtwrTBTB2URrTzTDXNM8HysQkkcqUC1I=; b=Mtfx20aYxGWpEMuCdVTSTgRtdEMe8ohwonbng4aKeeOb007z0nkc5461 8peJubFUpmvj83xVQih9kjt0qFKYnDl+CzLwRjQk6J3Xe+jreXSjF0+7U wh4gSHwIvKCDk4rsx3rk8z4MhAKQ8v1PRHNyIs7/WSmFud3XASARwKCze ZcAWCD8k8NtFpOxNGje7QnmWqfWcvFoKVJAjnJ+yd3zYXZmUbWcelG7G/ 1aT+CwS7mFeOISnk0XHRYkFMU7G3YcwHsQf6uVSwuSCXBaXI8jxiIFdJc WcoSbwUw6mqSmFBXE4nb3XGiXYJj1ZFHvsGBJrrQEEWxZlfalSNi+zbaN g==; X-IronPort-AV: E=McAfee;i="6600,9927,10854"; a="368775247" X-IronPort-AV: E=Sophos;i="6.03,203,1694761200"; d="scan'208";a="368775247" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 00:27:28 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10854"; a="842735620" X-IronPort-AV: E=Sophos;i="6.03,203,1694761200"; d="scan'208";a="842735620" Received: from wbleichn-mobl.amr.corp.intel.com (HELO dwillia2-xfh.jf.intel.com) ([10.212.147.24]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 00:26:58 -0700 Subject: [PATCH v3 10/10] tools/testing/cxl: Add 'sanitize notifier' support From: Dan Williams To: linux-cxl@vger.kernel.org Cc: Davidlohr Bueso Date: Fri, 06 Oct 2023 00:26:57 -0700 Message-ID: <169657721756.1491153.17305814762884004365.stgit@dwillia2-xfh.jf.intel.com> In-Reply-To: <169657715790.1491153.3612164287133860191.stgit@dwillia2-xfh.jf.intel.com> References: <169657715790.1491153.3612164287133860191.stgit@dwillia2-xfh.jf.intel.com> User-Agent: StGit/0.18-3-g996c Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Allow for cxl_test regression of the sanitize notifier. Reuse the core setup infrastructure, and trigger notifications upon any sanitize submission with a programmable notification delay. Cc: Davidlohr Bueso Signed-off-by: Dan Williams Reviewed-by: Ira Weiny Reviewed-by: Dave Jiang --- tools/testing/cxl/test/mem.c | 68 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 67 insertions(+), 1 deletion(-) diff --git a/tools/testing/cxl/test/mem.c b/tools/testing/cxl/test/mem.c index ab311b59899a..76bdb1ac5816 100644 --- a/tools/testing/cxl/test/mem.c +++ b/tools/testing/cxl/test/mem.c @@ -89,6 +89,12 @@ static struct cxl_cel_entry mock_cel[] = { .effect = cpu_to_le16(EFFECT(CONF_CHANGE_COLD_RESET) | EFFECT(CONF_CHANGE_IMMEDIATE)), }, + { + .opcode = cpu_to_le16(CXL_MBOX_OP_SANITIZE), + .effect = cpu_to_le16(EFFECT(DATA_CHANGE_IMMEDIATE) | + EFFECT(SECURITY_CHANGE_IMMEDIATE) | + EFFECT(BACKGROUND_OP)), + }, }; /* See CXL 2.0 Table 181 Get Health Info Output Payload */ @@ -152,6 +158,7 @@ struct cxl_mockmem_data { struct cxl_memdev_state *mds; u8 event_buf[SZ_4K]; u64 timestamp; + unsigned long sanitize_timeout; }; static struct mock_event_log *event_find_log(struct device *dev, int log_type) @@ -567,9 +574,26 @@ static int mock_partition_info(struct cxl_mbox_cmd *cmd) return 0; } +void cxl_mockmem_sanitize_work(struct work_struct *work) +{ + struct cxl_memdev_state *mds = + container_of(work, typeof(*mds), security.poll_dwork.work); + + mutex_lock(&mds->mbox_mutex); + if (mds->security.sanitize_node) + sysfs_notify_dirent(mds->security.sanitize_node); + mds->security.sanitize_active = false; + mutex_unlock(&mds->mbox_mutex); + + dev_dbg(mds->cxlds.dev, "sanitize complete\n"); +} + static int mock_sanitize(struct cxl_mockmem_data *mdata, struct cxl_mbox_cmd *cmd) { + struct cxl_memdev_state *mds = mdata->mds; + int rc = 0; + if (cmd->size_in != 0) return -EINVAL; @@ -585,7 +609,16 @@ static int mock_sanitize(struct cxl_mockmem_data *mdata, return -ENXIO; } - return 0; /* assume less than 2 secs, no bg */ + mutex_lock(&mds->mbox_mutex); + if (schedule_delayed_work(&mds->security.poll_dwork, + msecs_to_jiffies(mdata->sanitize_timeout))) { + mds->security.sanitize_active = true; + dev_dbg(mds->cxlds.dev, "sanitize issued\n"); + } else + rc = -EBUSY; + mutex_unlock(&mds->mbox_mutex); + + return rc; } static int mock_secure_erase(struct cxl_mockmem_data *mdata, @@ -1419,6 +1452,7 @@ static int cxl_mock_mem_probe(struct platform_device *pdev) mds->mbox_send = cxl_mock_mbox_send; mds->payload_size = SZ_4K; mds->event.buf = (struct cxl_get_event_payload *) mdata->event_buf; + INIT_DELAYED_WORK(&mds->security.poll_dwork, cxl_mockmem_sanitize_work); cxlds = &mds->cxlds; cxlds->serial = pdev->id; @@ -1458,6 +1492,10 @@ static int cxl_mock_mem_probe(struct platform_device *pdev) if (rc) return rc; + rc = devm_cxl_sanitize_setup_notifier(&pdev->dev, cxlmd); + if (rc) + return rc; + cxl_mem_get_event_records(mds, CXLDEV_EVENT_STATUS_ALL); return 0; @@ -1526,10 +1564,38 @@ static ssize_t fw_buf_checksum_show(struct device *dev, static DEVICE_ATTR_RO(fw_buf_checksum); +static ssize_t sanitize_timeout_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct cxl_mockmem_data *mdata = dev_get_drvdata(dev); + + return sysfs_emit(buf, "%lu\n", mdata->sanitize_timeout); +} + +static ssize_t sanitize_timeout_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t count) +{ + struct cxl_mockmem_data *mdata = dev_get_drvdata(dev); + unsigned long val; + int rc; + + rc = kstrtoul(buf, 0, &val); + if (rc) + return rc; + + mdata->sanitize_timeout = val; + + return count; +} + +static DEVICE_ATTR_RW(sanitize_timeout); + static struct attribute *cxl_mock_mem_attrs[] = { &dev_attr_security_lock.attr, &dev_attr_event_trigger.attr, &dev_attr_fw_buf_checksum.attr, + &dev_attr_sanitize_timeout.attr, NULL }; ATTRIBUTE_GROUPS(cxl_mock_mem);