From patchwork Mon Apr 22 06:57:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aravind Iddamsetty X-Patchwork-Id: 13637702 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 62462C05023 for ; Mon, 22 Apr 2024 06:55:30 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id B9CBF11282F; Mon, 22 Apr 2024 06:55:27 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="cv4Guyd3"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.12]) by gabe.freedesktop.org (Postfix) with ESMTPS id BDACA11281F; Mon, 22 Apr 2024 06:55:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1713768925; x=1745304925; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=G4PtOt+VWc88G5WZ5DuBu5hKTpaJnZsZ86R51uNgRxQ=; b=cv4Guyd3hM+NaP+fHZaZ+djU2gFFFyP228aYYv7YZFkVl5J8S3DuLL4Z BGZNb3O3kT8+G1v7rRshrKAkXH0WlDm6icqc6qd2iSKYYcE4Oc7Y9zzqe /P9gOSaVPfem1vIKRpIBuqAr15PbaY26WQpHUCd4TF3BsbtOCM9TFn2u+ 5IKl537hwURRE6vvMcofnVWFc4fVpKVcQ7/zwagGSi1uPiSTifZkLSQvm HJRMrMYQ2ln+noQAyiR91JSGDzFG7VYw5z9ClfPJlHRBZRDG8CvHYB/fF /AlR9IfLXPmqjgOP/pDtg/ge8JsjJfSgr7Q8UAh7bDe6OAvOlXWa3dEs8 Q==; X-CSE-ConnectionGUID: 4pkt5MiOR8SxAmtOuvHopg== X-CSE-MsgGUID: +YSVFyn3SlCztHf+b0mbXg== X-IronPort-AV: E=McAfee;i="6600,9927,11051"; a="20715508" X-IronPort-AV: E=Sophos;i="6.07,220,1708416000"; d="scan'208";a="20715508" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by orvoesa104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Apr 2024 23:55:24 -0700 X-CSE-ConnectionGUID: AwZe/KWHQ4KNfu0E0QeaAQ== X-CSE-MsgGUID: qEaiOq3gR2y7aql6Zbpssw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,220,1708416000"; d="scan'208";a="28584500" Received: from aravind-dev.iind.intel.com ([10.145.162.146]) by fmviesa004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Apr 2024 23:55:21 -0700 From: Aravind Iddamsetty To: dri-devel@lists.freedesktop.org, daniel@ffwll.ch, maarten.lankhorst@linux.intel.com, airlied@gmail.com, mripard@kernel.org, tzimmermann@suse.de, rodrigo.vivi@intel.com Cc: intel-xe@lists.freedesktop.org, Thomas Hellstr_m Subject: [PATCH v3 1/4] drm: add devm release action Date: Mon, 22 Apr 2024 12:27:53 +0530 Message-Id: <20240422065756.294679-2-aravind.iddamsetty@linux.intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240422065756.294679-1-aravind.iddamsetty@linux.intel.com> References: <20240422065756.294679-1-aravind.iddamsetty@linux.intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" In scenarios where drm_dev_put is directly called by driver we want to release devm_drm_dev_init_release action associated with struct drm_device. v2: Directly expose the original function, instead of introducing a helper (Rodrigo) v3: add kernel-doc (Maxime Ripard) Cc: Maxime Ripard Cc: Thomas Hellstr_m Cc: Rodrigo Vivi Reviewed-by: Rodrigo Vivi Signed-off-by: Aravind Iddamsetty --- drivers/gpu/drm/drm_drv.c | 13 +++++++++++++ include/drm/drm_drv.h | 2 ++ 2 files changed, 15 insertions(+) diff --git a/drivers/gpu/drm/drm_drv.c b/drivers/gpu/drm/drm_drv.c index 243cacb3575c..9d0409165f1e 100644 --- a/drivers/gpu/drm/drm_drv.c +++ b/drivers/gpu/drm/drm_drv.c @@ -714,6 +714,19 @@ static int devm_drm_dev_init(struct device *parent, devm_drm_dev_init_release, dev); } +/** + * devm_drm_dev_release_action - Call the final release action of the device + * @dev: DRM device + * + * In scenarios where drm_dev_put is directly called by driver we want to release + * devm_drm_dev_init_release action associated with struct drm_device. + */ +void devm_drm_dev_release_action(struct drm_device *dev) +{ + devm_release_action(dev->dev, devm_drm_dev_init_release, dev); +} +EXPORT_SYMBOL(devm_drm_dev_release_action); + void *__devm_drm_dev_alloc(struct device *parent, const struct drm_driver *driver, size_t size, size_t offset) diff --git a/include/drm/drm_drv.h b/include/drm/drm_drv.h index 8878260d7529..fa9123684874 100644 --- a/include/drm/drm_drv.h +++ b/include/drm/drm_drv.h @@ -444,6 +444,8 @@ struct drm_driver { const struct file_operations *fops; }; +void devm_drm_dev_release_action(struct drm_device *dev); + void *__devm_drm_dev_alloc(struct device *parent, const struct drm_driver *driver, size_t size, size_t offset); From patchwork Mon Apr 22 06:57:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aravind Iddamsetty X-Patchwork-Id: 13637703 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 50FD7C4345F for ; Mon, 22 Apr 2024 06:55:33 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id CE7EF112826; Mon, 22 Apr 2024 06:55:29 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="EHbQktwt"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.12]) by gabe.freedesktop.org (Postfix) with ESMTPS id D71FD112834; Mon, 22 Apr 2024 06:55:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1713768928; x=1745304928; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=YzWiN2qaZ5OOeOp9ajc0LMSxGaArZykX00tv3fGZY2I=; b=EHbQktwtl3G4KpQtheDnFmSmoJcnc+bQrpQtJy8Elflb3CAFOkYIp28t qC/4x0C45Ed3SzYGzp8eeJaDjhbsm5YMdxTp0bjjgVL30JkNzK6N4/jjG uit6wI+cWiB0WV2EsGrAp68hBuqqR7TrIjid0me3SveD65gz1QKCP8cCs FhFebIjhwQU1ly7rXePmDzOz2Xql51j9Ykqo5viz8KOmZvDoFUHfFHKlS IdDjZDgoQbNPo9HaO/w8KgNc8R9KtDi558S6UiER/7YVmQLVA6mjXwSCD meEtbgyw4AEygFVDOme5DJKBtYtPDOLsU0mU3TAt02X/Psw55ZWR9XrCK g==; X-CSE-ConnectionGUID: xVV+cwsgStuBP3eoLAtW+A== X-CSE-MsgGUID: uJpodlBAQM6OmLskAedpTg== X-IronPort-AV: E=McAfee;i="6600,9927,11051"; a="20715514" X-IronPort-AV: E=Sophos;i="6.07,220,1708416000"; d="scan'208";a="20715514" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by orvoesa104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Apr 2024 23:55:27 -0700 X-CSE-ConnectionGUID: 49TwrkkaTLm9tzycx63yDA== X-CSE-MsgGUID: 6CDf3S2nTMSeRpmX8+Hs+w== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,220,1708416000"; d="scan'208";a="28584510" Received: from aravind-dev.iind.intel.com ([10.145.162.146]) by fmviesa004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Apr 2024 23:55:24 -0700 From: Aravind Iddamsetty To: dri-devel@lists.freedesktop.org, daniel@ffwll.ch, maarten.lankhorst@linux.intel.com, airlied@gmail.com, mripard@kernel.org, tzimmermann@suse.de, rodrigo.vivi@intel.com Cc: intel-xe@lists.freedesktop.org, Lucas De Marchi Subject: [PATCH 2/4] drm/xe: Save and restore PCI state Date: Mon, 22 Apr 2024 12:27:54 +0530 Message-Id: <20240422065756.294679-3-aravind.iddamsetty@linux.intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240422065756.294679-1-aravind.iddamsetty@linux.intel.com> References: <20240422065756.294679-1-aravind.iddamsetty@linux.intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Save and restore PCI states where ever needed. Cc: Lucas De Marchi Reviewed-by: Rodrigo Vivi Signed-off-by: Aravind Iddamsetty --- drivers/gpu/drm/xe/xe_device_types.h | 3 ++ drivers/gpu/drm/xe/xe_pci.c | 48 ++++++++++++++++++++++++++-- drivers/gpu/drm/xe/xe_pci.h | 4 ++- 3 files changed, 51 insertions(+), 4 deletions(-) diff --git a/drivers/gpu/drm/xe/xe_device_types.h b/drivers/gpu/drm/xe/xe_device_types.h index 8244b177a6a3..0a66555229e9 100644 --- a/drivers/gpu/drm/xe/xe_device_types.h +++ b/drivers/gpu/drm/xe/xe_device_types.h @@ -462,6 +462,9 @@ struct xe_device { /** @needs_flr_on_fini: requests function-reset on fini */ bool needs_flr_on_fini; + /** @pci_state: PCI state of device */ + struct pci_saved_state *pci_state; + /* private: */ #if IS_ENABLED(CONFIG_DRM_XE_DISPLAY) diff --git a/drivers/gpu/drm/xe/xe_pci.c b/drivers/gpu/drm/xe/xe_pci.c index fb20c9828563..a62300990e19 100644 --- a/drivers/gpu/drm/xe/xe_pci.c +++ b/drivers/gpu/drm/xe/xe_pci.c @@ -393,6 +393,41 @@ MODULE_DEVICE_TABLE(pci, pciidlist); #undef INTEL_VGA_DEVICE +static bool xe_save_pci_state(struct pci_dev *pdev) +{ + struct xe_device *xe = pci_get_drvdata(pdev); + + if (pci_save_state(pdev)) + return false; + + kfree(xe->pci_state); + + xe->pci_state = pci_store_saved_state(pdev); + if (!xe->pci_state) { + drm_err(&xe->drm, "Failed to store PCI saved state\n"); + return false; + } + + return true; +} + +void xe_load_pci_state(struct pci_dev *pdev) +{ + struct xe_device *xe = pci_get_drvdata(pdev); + int ret; + + if (!xe->pci_state) + return; + + ret = pci_load_saved_state(pdev, xe->pci_state); + if (ret) { + drm_warn(&xe->drm, "Failed to load PCI state err:%d\n", ret); + return; + } + + pci_restore_state(pdev); +} + /* is device_id present in comma separated list of ids */ static bool device_id_in_list(u16 device_id, const char *devices, bool negative) { @@ -698,6 +733,8 @@ static void xe_pci_remove(struct pci_dev *pdev) xe_device_remove(xe); xe_pm_runtime_fini(xe); + + kfree(xe->pci_state); pci_set_drvdata(pdev, NULL); } @@ -798,6 +835,9 @@ static int xe_pci_probe(struct pci_dev *pdev, const struct pci_device_id *ent) drm_dbg(&xe->drm, "d3cold: capable=%s\n", str_yes_no(xe->d3cold.capable)); + if (xe_save_pci_state(pdev)) + pci_restore_state(pdev); + return 0; err_driver_cleanup: @@ -849,7 +889,7 @@ static int xe_pci_suspend(struct device *dev) */ d3cold_toggle(pdev, D3COLD_ENABLE); - pci_save_state(pdev); + xe_save_pci_state(pdev); pci_disable_device(pdev); return 0; @@ -873,6 +913,8 @@ static int xe_pci_resume(struct device *dev) pci_set_master(pdev); + xe_load_pci_state(pdev); + err = xe_pm_resume(pdev_to_xe_device(pdev)); if (err) return err; @@ -890,7 +932,7 @@ static int xe_pci_runtime_suspend(struct device *dev) if (err) return err; - pci_save_state(pdev); + xe_save_pci_state(pdev); if (xe->d3cold.allowed) { d3cold_toggle(pdev, D3COLD_ENABLE); @@ -915,7 +957,7 @@ static int xe_pci_runtime_resume(struct device *dev) if (err) return err; - pci_restore_state(pdev); + xe_load_pci_state(pdev); if (xe->d3cold.allowed) { err = pci_enable_device(pdev); diff --git a/drivers/gpu/drm/xe/xe_pci.h b/drivers/gpu/drm/xe/xe_pci.h index 611c1209b14c..73b90a430d1f 100644 --- a/drivers/gpu/drm/xe/xe_pci.h +++ b/drivers/gpu/drm/xe/xe_pci.h @@ -6,7 +6,9 @@ #ifndef _XE_PCI_H_ #define _XE_PCI_H_ +struct pci_dev; + int xe_register_pci_driver(void); void xe_unregister_pci_driver(void); - +void xe_load_pci_state(struct pci_dev *pdev); #endif From patchwork Mon Apr 22 06:57:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aravind Iddamsetty X-Patchwork-Id: 13637704 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0E357C4345F for ; Mon, 22 Apr 2024 06:55:40 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 187FE11281D; Mon, 22 Apr 2024 06:55:39 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="JGXWvAsH"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.12]) by gabe.freedesktop.org (Postfix) with ESMTPS id C67F2112827; Mon, 22 Apr 2024 06:55:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1713768932; x=1745304932; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=uRxmihYD6YLqzpHDOZaQqPXSGeVHICpQAId+FuwwTHw=; b=JGXWvAsHfUisBGaydMSY/rGoD/xf2YKKVJH1xZCkZOrpNQE9ChKbrM2W bqIWbexMG/zY/Xqj1edqvqKe8YYh5poGtyrpYB74nN17j1oFuSIDuRZnt AX113vBQJN/o3dPuizJlhIi608qECkkmhze6lmi+BWarZZgRrRwHRzPFi OzEr+NOGB6LDHCUwalo743alTNZDNHH17EwuHaDy5CnOrHjiyZ7IrW+ek KcFnjfU1iXtfZKY5W72FtyLBjqs0iDb6Oz6mW4hUsAq2USSXotgBb9BI0 zvMD1XkGrLZFjgcs4gZny+0dE90GtWnCZdyzGNkWYLMUdRUYQcGbrqWzh w==; X-CSE-ConnectionGUID: mKSqojm+RVuHGJc9vzN8eQ== X-CSE-MsgGUID: QeTi3Rq+TgGDgxz0yIaY9w== X-IronPort-AV: E=McAfee;i="6600,9927,11051"; a="20715521" X-IronPort-AV: E=Sophos;i="6.07,220,1708416000"; d="scan'208";a="20715521" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by orvoesa104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Apr 2024 23:55:31 -0700 X-CSE-ConnectionGUID: ulUM35UERK670rm5dPAQPg== X-CSE-MsgGUID: RUxp+7jaTjKOiWkcQtImBw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,220,1708416000"; d="scan'208";a="28584522" Received: from aravind-dev.iind.intel.com ([10.145.162.146]) by fmviesa004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Apr 2024 23:55:27 -0700 From: Aravind Iddamsetty To: dri-devel@lists.freedesktop.org, daniel@ffwll.ch, maarten.lankhorst@linux.intel.com, airlied@gmail.com, mripard@kernel.org, tzimmermann@suse.de, rodrigo.vivi@intel.com Cc: intel-xe@lists.freedesktop.org, Lucas De Marchi , Michal Wajdeczko , Himal Prasad Ghimiray Subject: [PATCH v2 3/4] drm/xe: Extract xe_gt_idle() helper Date: Mon, 22 Apr 2024 12:27:55 +0530 Message-Id: <20240422065756.294679-4-aravind.iddamsetty@linux.intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240422065756.294679-1-aravind.iddamsetty@linux.intel.com> References: <20240422065756.294679-1-aravind.iddamsetty@linux.intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" This would be used in other places outside of gt_reset path. v2: 1. Add kernel doc for xe_gt_idle (Michal) 2. fix return as no actual error is reported by xe_uc_stop (Himal) Cc: Lucas De Marchi Cc: Michal Wajdeczko Cc: Himal Prasad Ghimiray Reviewed-by: Rodrigo Vivi Signed-off-by: Aravind Iddamsetty --- drivers/gpu/drm/xe/xe_gt.c | 28 ++++++++++++++++++---------- drivers/gpu/drm/xe/xe_gt.h | 1 + 2 files changed, 19 insertions(+), 10 deletions(-) diff --git a/drivers/gpu/drm/xe/xe_gt.c b/drivers/gpu/drm/xe/xe_gt.c index 491d0413de15..a24340219c30 100644 --- a/drivers/gpu/drm/xe/xe_gt.c +++ b/drivers/gpu/drm/xe/xe_gt.c @@ -629,6 +629,23 @@ static int do_gt_restart(struct xe_gt *gt) return 0; } +/** + * xe_gt_idle: Idle the GT. + * @gt: The gt object + * + * Typically called before initiating any resets. + * + */ +void xe_gt_idle(struct xe_gt *gt) +{ + xe_gt_sanitize(gt); + xe_uc_gucrc_disable(>->uc); + xe_uc_stop_prepare(>->uc); + xe_gt_pagefault_reset(gt); + xe_uc_stop(>->uc); + xe_gt_tlb_invalidation_reset(gt); +} + static int gt_reset(struct xe_gt *gt) { int err; @@ -645,21 +662,12 @@ static int gt_reset(struct xe_gt *gt) } xe_pm_runtime_get(gt_to_xe(gt)); - xe_gt_sanitize(gt); err = xe_force_wake_get(gt_to_fw(gt), XE_FORCEWAKE_ALL); if (err) goto err_msg; - xe_uc_gucrc_disable(>->uc); - xe_uc_stop_prepare(>->uc); - xe_gt_pagefault_reset(gt); - - err = xe_uc_stop(>->uc); - if (err) - goto err_out; - - xe_gt_tlb_invalidation_reset(gt); + xe_gt_idle(gt); err = do_gt_reset(gt); if (err) diff --git a/drivers/gpu/drm/xe/xe_gt.h b/drivers/gpu/drm/xe/xe_gt.h index ed6ea8057e35..dcc1023d20e8 100644 --- a/drivers/gpu/drm/xe/xe_gt.h +++ b/drivers/gpu/drm/xe/xe_gt.h @@ -43,6 +43,7 @@ int xe_gt_resume(struct xe_gt *gt); void xe_gt_reset_async(struct xe_gt *gt); void xe_gt_sanitize(struct xe_gt *gt); void xe_gt_remove(struct xe_gt *gt); +void xe_gt_idle(struct xe_gt *gt); /** * xe_gt_any_hw_engine_by_reset_domain - scan the list of engines and return the From patchwork Mon Apr 22 06:57:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Aravind Iddamsetty X-Patchwork-Id: 13637705 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 03ABBC4345F for ; Mon, 22 Apr 2024 06:55:43 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id F2FDA112820; Mon, 22 Apr 2024 06:55:41 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="GRSbhNEP"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.12]) by gabe.freedesktop.org (Postfix) with ESMTPS id 1B621112820; Mon, 22 Apr 2024 06:55:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1713768940; x=1745304940; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Ayul71T12aA+RBSdHxkG+LUJsbuovp4mkBVZJ1PdcKU=; b=GRSbhNEPTwYcmpvqMdx7SSXwIvfKfSKtpbjOZra4houKKN8YpZWZVVaA 9Y6Uwczg7JORjRs4kU4j9xwKMgH5psIMziuynS8Pcw1J8Eve0J89fmnpf zg3MDTVnVkHrEHgGzqkz7hl/NNP1k2elSFCtFMmgOkk2hcaGDEKZMJTw2 G4THzuEfLk2kKYNmPvC79wPpfNDajHa16pmqoqiNuS+gFDBpvc/OsNghf QogNjadV2Y0bdqXj7SBBNx7lF9ZV7BsqeSvXWFksSigGOD6PPpaPSfqGA nSIWUvgA/IdFpLbHjkn5w2xa05wBV9l8GXvxZs1EbCXk3psmo3UOydYd5 g==; X-CSE-ConnectionGUID: K4jXbf6hQDSDaaxi4Z/WWA== X-CSE-MsgGUID: OG+Z388tROuok9lPQE5AGw== X-IronPort-AV: E=McAfee;i="6600,9927,11051"; a="20715537" X-IronPort-AV: E=Sophos;i="6.07,220,1708416000"; d="scan'208";a="20715537" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by orvoesa104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Apr 2024 23:55:35 -0700 X-CSE-ConnectionGUID: m9rExLvFRhejnAT54rHR8Q== X-CSE-MsgGUID: UMTJ7iwPTKytyOdsSV9bVQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,220,1708416000"; d="scan'208";a="28584537" Received: from aravind-dev.iind.intel.com ([10.145.162.146]) by fmviesa004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Apr 2024 23:55:31 -0700 From: Aravind Iddamsetty To: dri-devel@lists.freedesktop.org, daniel@ffwll.ch, maarten.lankhorst@linux.intel.com, airlied@gmail.com, mripard@kernel.org, tzimmermann@suse.de, rodrigo.vivi@intel.com Cc: intel-xe@lists.freedesktop.org, Lucas De Marchi , Michal Wajdeczko Subject: [PATCH v3 4/4] drm/xe/FLR: Support PCIe FLR Date: Mon, 22 Apr 2024 12:27:56 +0530 Message-Id: <20240422065756.294679-5-aravind.iddamsetty@linux.intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240422065756.294679-1-aravind.iddamsetty@linux.intel.com> References: <20240422065756.294679-1-aravind.iddamsetty@linux.intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" PCI subsystem provides callbacks to inform the driver about a request to do function level reset by user, initiated by writing to sysfs entry /sys/bus/pci/devices/.../reset. This will allow the driver to handle FLR without the need to do unbind and rebind as the driver needs to reinitialize the device afresh post FLR. v2: 1. separate out gt idle and pci save/restore to a separate patch (Lucas) 2. Fixed the warnings seen around xe_guc_submit_stop, xe_guc_puc_fini v3: declare xe_pci_err_handlers as static(Michal) Cc: Rodrigo Vivi Cc: Lucas De Marchi Cc: Michal Wajdeczko Reviewed-by: Rodrigo Vivi Signed-off-by: Aravind Iddamsetty --- drivers/gpu/drm/xe/Makefile | 1 + drivers/gpu/drm/xe/xe_device_types.h | 3 + drivers/gpu/drm/xe/xe_guc_pc.c | 4 ++ drivers/gpu/drm/xe/xe_pci.c | 9 ++- drivers/gpu/drm/xe/xe_pci.h | 2 + drivers/gpu/drm/xe/xe_pci_err.c | 88 ++++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_pci_err.h | 13 ++++ 7 files changed, 119 insertions(+), 1 deletion(-) create mode 100644 drivers/gpu/drm/xe/xe_pci_err.c create mode 100644 drivers/gpu/drm/xe/xe_pci_err.h diff --git a/drivers/gpu/drm/xe/Makefile b/drivers/gpu/drm/xe/Makefile index 8bc62bfbc679..693971a1fac0 100644 --- a/drivers/gpu/drm/xe/Makefile +++ b/drivers/gpu/drm/xe/Makefile @@ -117,6 +117,7 @@ xe-y += xe_bb.o \ xe_module.o \ xe_pat.o \ xe_pci.o \ + xe_pci_err.o \ xe_pcode.o \ xe_pm.o \ xe_preempt_fence.o \ diff --git a/drivers/gpu/drm/xe/xe_device_types.h b/drivers/gpu/drm/xe/xe_device_types.h index 0a66555229e9..8c749b378a92 100644 --- a/drivers/gpu/drm/xe/xe_device_types.h +++ b/drivers/gpu/drm/xe/xe_device_types.h @@ -465,6 +465,9 @@ struct xe_device { /** @pci_state: PCI state of device */ struct pci_saved_state *pci_state; + /** @pci_device_is_reset: device went through PCIe FLR */ + bool pci_device_is_reset; + /* private: */ #if IS_ENABLED(CONFIG_DRM_XE_DISPLAY) diff --git a/drivers/gpu/drm/xe/xe_guc_pc.c b/drivers/gpu/drm/xe/xe_guc_pc.c index 509649d0e65e..efba0fbe2f5c 100644 --- a/drivers/gpu/drm/xe/xe_guc_pc.c +++ b/drivers/gpu/drm/xe/xe_guc_pc.c @@ -902,6 +902,10 @@ static void xe_guc_pc_fini(struct drm_device *drm, void *arg) return; } + /* We already have done this before going through a reset, so skip here */ + if (xe->pci_device_is_reset) + return; + XE_WARN_ON(xe_force_wake_get(gt_to_fw(pc_to_gt(pc)), XE_FORCEWAKE_ALL)); XE_WARN_ON(xe_guc_pc_gucrc_disable(pc)); XE_WARN_ON(xe_guc_pc_stop(pc)); diff --git a/drivers/gpu/drm/xe/xe_pci.c b/drivers/gpu/drm/xe/xe_pci.c index a62300990e19..b5a582afc9e7 100644 --- a/drivers/gpu/drm/xe/xe_pci.c +++ b/drivers/gpu/drm/xe/xe_pci.c @@ -23,6 +23,7 @@ #include "xe_macros.h" #include "xe_mmio.h" #include "xe_module.h" +#include "xe_pci_err.h" #include "xe_pci_types.h" #include "xe_pm.h" #include "xe_sriov.h" @@ -738,7 +739,7 @@ static void xe_pci_remove(struct pci_dev *pdev) pci_set_drvdata(pdev, NULL); } -static int xe_pci_probe(struct pci_dev *pdev, const struct pci_device_id *ent) +int xe_pci_probe(struct pci_dev *pdev, const struct pci_device_id *ent) { const struct xe_device_desc *desc = (const void *)ent->driver_data; const struct xe_subplatform_desc *subplatform_desc; @@ -986,6 +987,11 @@ static const struct dev_pm_ops xe_pm_ops = { }; #endif +static const struct pci_error_handlers xe_pci_err_handlers = { + .reset_prepare = xe_pci_reset_prepare, + .reset_done = xe_pci_reset_done, +}; + static struct pci_driver xe_pci_driver = { .name = DRIVER_NAME, .id_table = pciidlist, @@ -995,6 +1001,7 @@ static struct pci_driver xe_pci_driver = { #ifdef CONFIG_PM_SLEEP .driver.pm = &xe_pm_ops, #endif + .err_handler = &xe_pci_err_handlers, }; int xe_register_pci_driver(void) diff --git a/drivers/gpu/drm/xe/xe_pci.h b/drivers/gpu/drm/xe/xe_pci.h index 73b90a430d1f..9faf5380a09e 100644 --- a/drivers/gpu/drm/xe/xe_pci.h +++ b/drivers/gpu/drm/xe/xe_pci.h @@ -7,8 +7,10 @@ #define _XE_PCI_H_ struct pci_dev; +struct pci_device_id; int xe_register_pci_driver(void); void xe_unregister_pci_driver(void); void xe_load_pci_state(struct pci_dev *pdev); +int xe_pci_probe(struct pci_dev *pdev, const struct pci_device_id *ent); #endif diff --git a/drivers/gpu/drm/xe/xe_pci_err.c b/drivers/gpu/drm/xe/xe_pci_err.c new file mode 100644 index 000000000000..5306925ea2fa --- /dev/null +++ b/drivers/gpu/drm/xe/xe_pci_err.c @@ -0,0 +1,88 @@ +// SPDX-License-Identifier: MIT +/* + * Copyright © 2024 Intel Corporation + */ + +#include +#include + +#include "xe_device.h" +#include "xe_gt.h" +#include "xe_gt_printk.h" +#include "xe_pci.h" +#include "xe_pci_err.h" +#include "xe_pm.h" +#include "xe_uc.h" + +/** + * xe_pci_reset_prepare - Called when user issued a PCIe reset + * via /sys/bus/pci/devices/.../reset. + * @pdev: PCI device struct + */ +void xe_pci_reset_prepare(struct pci_dev *pdev) +{ + struct xe_device *xe = pci_get_drvdata(pdev); + struct xe_gt *gt; + int id, err; + + pci_warn(pdev, "preparing for PCIe reset\n"); + + drm_warn(&xe->drm, "removing device access to userspace\n"); + drm_dev_unplug(&xe->drm); + + xe_pm_runtime_get(xe); + /* idle the GTs */ + for_each_gt(gt, xe, id) { + err = xe_force_wake_get(gt_to_fw(gt), XE_FORCEWAKE_ALL); + if (err) + goto reset; + xe_uc_reset_prepare(>->uc); + xe_gt_idle(gt); + err = xe_force_wake_put(gt_to_fw(gt), XE_FORCEWAKE_ALL); + XE_WARN_ON(err); + } + xe_pm_runtime_put(xe); + +reset: + pci_disable_device(pdev); +} + +/** + * xe_pci_reset_done - Called when PCIe reset is done. + * @pdev: PCI device struct + */ +void xe_pci_reset_done(struct pci_dev *pdev) +{ + const struct pci_device_id *ent = pci_match_id(pdev->driver->id_table, pdev); + struct xe_device *xe = pci_get_drvdata(pdev); + + dev_info(&pdev->dev, + "device went through PCIe reset, reenabling the device\n"); + + if (pci_enable_device(pdev)) { + dev_err(&pdev->dev, + "Cannot re-enable PCI device after reset\n"); + return; + } + pci_set_master(pdev); + xe_load_pci_state(pdev); + + xe->pci_device_is_reset = true; + /* + * We want to completely clean the driver and even destroy + * the xe private data and reinitialize afresh similar to + * probe + */ + pdev->driver->remove(pdev); + if (pci_dev_msi_enabled(pdev)) + pci_free_irq_vectors(pdev); + + devm_drm_dev_release_action(&xe->drm); + pci_disable_device(pdev); + + /* + * if this fails the driver might be in a stale state, only option is + * to unbind and rebind + */ + xe_pci_probe(pdev, ent); +} diff --git a/drivers/gpu/drm/xe/xe_pci_err.h b/drivers/gpu/drm/xe/xe_pci_err.h new file mode 100644 index 000000000000..95a4c8ce9cf1 --- /dev/null +++ b/drivers/gpu/drm/xe/xe_pci_err.h @@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: MIT */ +/* + * Copyright © 2024 Intel Corporation + */ + +#ifndef _XE_PCI_ERR_H_ +#define _XE_PCI_ERR_H_ + +struct pci_dev; + +void xe_pci_reset_prepare(struct pci_dev *pdev); +void xe_pci_reset_done(struct pci_dev *pdev); +#endif