From patchwork Fri Oct 6 18:20:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Cavitt, Jonathan" X-Patchwork-Id: 13411827 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3C23BE92FE0 for ; Fri, 6 Oct 2023 18:31:39 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 19B2B10E556; Fri, 6 Oct 2023 18:31:37 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.65]) by gabe.freedesktop.org (Postfix) with ESMTPS id B38DE10E556 for ; Fri, 6 Oct 2023 18:31:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1696617094; x=1728153094; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=t8vNIiYkTJY6qlDw1htcI++MJKSR36uX+NulwL+Vo9I=; b=eu5JRFQ2bS/CfNn7ybPZJ15ha2X39NaXJMiVWzJjpxoKEP2Mq7Q5YcDo jX7Bb2XHk0+w2NFfQTfQ0xehWhGcZOipYg0xrKSasKgLAB4C2hL6sh39Q di5kFXKmbDb838umRPQ9s0uqfGNt1Sq/jX2DEEFbBtagNFtgV2p6Vh5lH psPigWqqpACViTmfeiKMDNXLMJjjQirCzWRvmckr4tKY8coFP6hMy6Jxq 2oBjWvKi3NkrP+Nm0UxfHMs20vEzdtfb/Tfy95aXjuYRHqJQSVzdae/ut 8tSmOs4sS/BFVerfHN4bSLUopp8h0GGwKo6lZL2lJNxJfAOzipwARbjXt w==; X-IronPort-AV: E=McAfee;i="6600,9927,10855"; a="387688164" X-IronPort-AV: E=Sophos;i="6.03,204,1694761200"; d="scan'208";a="387688164" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 11:31:33 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10855"; a="745950574" X-IronPort-AV: E=Sophos;i="6.03,204,1694761200"; d="scan'208";a="745950574" Received: from dut-internal-9dd7.jf.intel.com ([10.165.21.194]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 11:31:33 -0700 From: Jonathan Cavitt To: intel-gfx@lists.freedesktop.org Date: Fri, 6 Oct 2023 11:20:45 -0700 Message-Id: <20231006182051.1599923-2-jonathan.cavitt@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231006182051.1599923-1-jonathan.cavitt@intel.com> References: <20231006182051.1599923-1-jonathan.cavitt@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v8 1/7] drm/i915: Add GuC TLB Invalidation device info flags X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: janusz.krzysztofik@intel.com, andi.shyti@intel.com, matthew.d.roper@intel.com, jonathan.cavitt@intel.com, chris.p.wilson@linux.intel.com, nirmoy.das@intel.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" Add device info flags for if GuC TLB Invalidation is enabled. Signed-off-by: Jonathan Cavitt --- drivers/gpu/drm/i915/i915_drv.h | 3 ++- drivers/gpu/drm/i915/intel_device_info.h | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h index cb60fc9cf8737..b7933e1120aa7 100644 --- a/drivers/gpu/drm/i915/i915_drv.h +++ b/drivers/gpu/drm/i915/i915_drv.h @@ -794,11 +794,12 @@ IS_SUBPLATFORM(const struct drm_i915_private *i915, #define HAS_GUC_DEPRIVILEGE(i915) \ (INTEL_INFO(i915)->has_guc_deprivilege) +#define HAS_GUC_TLB_INVALIDATION(i915) (INTEL_INFO(i915)->has_guc_tlb_invalidation) + #define HAS_3D_PIPELINE(i915) (INTEL_INFO(i915)->has_3d_pipeline) #define HAS_ONE_EU_PER_FUSE_BIT(i915) (INTEL_INFO(i915)->has_one_eu_per_fuse_bit) #define HAS_LMEMBAR_SMEM_STOLEN(i915) (!HAS_LMEM(i915) && \ GRAPHICS_VER_FULL(i915) >= IP_VER(12, 70)) - #endif diff --git a/drivers/gpu/drm/i915/intel_device_info.h b/drivers/gpu/drm/i915/intel_device_info.h index 39817490b13fd..eba2f0b919c87 100644 --- a/drivers/gpu/drm/i915/intel_device_info.h +++ b/drivers/gpu/drm/i915/intel_device_info.h @@ -153,6 +153,7 @@ enum intel_ppgtt_type { func(has_heci_pxp); \ func(has_heci_gscfi); \ func(has_guc_deprivilege); \ + func(has_guc_tlb_invalidation); \ func(has_l3_ccs_read); \ func(has_l3_dpf); \ func(has_llc); \ From patchwork Fri Oct 6 18:20:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Cavitt, Jonathan" X-Patchwork-Id: 13411830 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D7FF7E81E1F for ; Fri, 6 Oct 2023 18:31:45 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id AA97210E559; Fri, 6 Oct 2023 18:31:39 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.65]) by gabe.freedesktop.org (Postfix) with ESMTPS id E028B10E554 for ; Fri, 6 Oct 2023 18:31:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1696617094; x=1728153094; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=M1Czg5f0AUD6s8LQvrfBEq1VZw35aoj8/qlkdkheOLU=; b=kS10nW5TL5rjV6nXHL0UJZaWoQdVOI4f4w//M5X0uEbJezvQpOUghNO4 nXjTZyrMRdcGOiXsF5V+AQh3c5kdS40o2zwUjCkTSheUYECCcxE5Uxejj 1xKeVFW8DotXNQ1HIxQ0eR5jNd6GfnKEQOLX5T6zATK8wBCvFrjWMhWXq iylUw1llVkE7hkvRh5GhSfo0ZogXiL6WZKiJx/mcPRYeYeHgtJltnM8DB mwPo13Sfi7r76OTDRwKTARLHF3ILsqqnUK7M0bDaEmd7XndV/izzxOeek dtJcwtVMF2qjrebB8jETj18BUxXn2y8IKl1OZAwqXJpViTl2Yhmu0WC4C w==; X-IronPort-AV: E=McAfee;i="6600,9927,10855"; a="387688165" X-IronPort-AV: E=Sophos;i="6.03,204,1694761200"; d="scan'208";a="387688165" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 11:31:34 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10855"; a="745950579" X-IronPort-AV: E=Sophos;i="6.03,204,1694761200"; d="scan'208";a="745950579" Received: from dut-internal-9dd7.jf.intel.com ([10.165.21.194]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 11:31:33 -0700 From: Jonathan Cavitt To: intel-gfx@lists.freedesktop.org Date: Fri, 6 Oct 2023 11:20:46 -0700 Message-Id: <20231006182051.1599923-3-jonathan.cavitt@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231006182051.1599923-1-jonathan.cavitt@intel.com> References: <20231006182051.1599923-1-jonathan.cavitt@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v8 2/7] drm/i915/guc: Add CT size delay helper X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: janusz.krzysztofik@intel.com, andi.shyti@intel.com, matthew.d.roper@intel.com, jonathan.cavitt@intel.com, chris.p.wilson@linux.intel.com, nirmoy.das@intel.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" Add a helper function to the GuC CT buffer that reports the expected time to process all outstanding requests. As of now, there is no functionality to check number of requests in the buffer, so the helper function just reports 2 seconds, or 1ms per request up to the maximum number of requests the CT buffer can store. Suggested-by: John Harrison Signed-off-by: Jonathan Cavitt --- drivers/gpu/drm/i915/gt/uc/intel_guc_ct.h | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_ct.h b/drivers/gpu/drm/i915/gt/uc/intel_guc_ct.h index 58e42901ff498..36afc1ce9fabd 100644 --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_ct.h +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_ct.h @@ -120,6 +120,19 @@ static inline bool intel_guc_ct_enabled(struct intel_guc_ct *ct) return ct->enabled; } +/* + * GuC has a timeout of 1ms for a TLB invalidation response from GAM. On a + * timeout GuC drops the request and has no mechanism to notify the host about + * the timeout. There is also no mechanism for determining the number of + * outstanding requests in the CT buffer. Ergo, keep a larger timeout that accounts + * for this individual timeout and the max number of outstanding requests that + * can be queued in CT buffer. + */ +static inline long intel_guc_ct_expected_delay(struct intel_guc_ct *ct) +{ + return HZ * 2; +} + #define INTEL_GUC_CT_SEND_NB BIT(31) #define INTEL_GUC_CT_SEND_G2H_DW_SHIFT 0 #define INTEL_GUC_CT_SEND_G2H_DW_MASK (0xff << INTEL_GUC_CT_SEND_G2H_DW_SHIFT) From patchwork Fri Oct 6 18:20:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Cavitt, Jonathan" X-Patchwork-Id: 13411833 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 95320E81E1E for ; Fri, 6 Oct 2023 18:31:48 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 49F3A10E55C; Fri, 6 Oct 2023 18:31:40 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.65]) by gabe.freedesktop.org (Postfix) with ESMTPS id 3D06410E554 for ; Fri, 6 Oct 2023 18:31:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1696617095; x=1728153095; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=73HfwV+m+Od33p/isNAXmkkDiasyhbUEtzmm2UoN1VI=; b=Vr3jsMaAlb/ZLVAybTV2thQ7W/sUr2ucQtNKrUCLaeq9xpfYooM5DXJi ZwayHfXuuoJ7fP+E8mCYALlYHQUA03Eo2pl9fRo7r9qkPmwZW1307qCkv jt7iyhGsrNpaKFtVBfQUXbKN0APBCYiBGF1AO16eb63+w/h2xsIGEg6wl Gy8RJfag8bNVKQDFTYJfFysHW3I/UvgEj6BFHxmGSkI9JrZnR0kwf5ZU1 tirx3LHo1GdnaAhzg4AEBJKYUukTN+bV1/UuIrZJGn2dcKEqJSBnqgNY+ RN2OI6bJaroNwK+fJWrm5cnUOsP2sO1hFEfzhk4N379N4Qveq4L8Fjted A==; X-IronPort-AV: E=McAfee;i="6600,9927,10855"; a="387688168" X-IronPort-AV: E=Sophos;i="6.03,204,1694761200"; d="scan'208";a="387688168" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 11:31:34 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10855"; a="745950582" X-IronPort-AV: E=Sophos;i="6.03,204,1694761200"; d="scan'208";a="745950582" Received: from dut-internal-9dd7.jf.intel.com ([10.165.21.194]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 11:31:33 -0700 From: Jonathan Cavitt To: intel-gfx@lists.freedesktop.org Date: Fri, 6 Oct 2023 11:20:47 -0700 Message-Id: <20231006182051.1599923-4-jonathan.cavitt@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231006182051.1599923-1-jonathan.cavitt@intel.com> References: <20231006182051.1599923-1-jonathan.cavitt@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v8 3/7] drm/i915: Define and use GuC and CTB TLB invalidation routines X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: janusz.krzysztofik@intel.com, andi.shyti@intel.com, matthew.d.roper@intel.com, jonathan.cavitt@intel.com, chris.p.wilson@linux.intel.com, nirmoy.das@intel.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" From: Prathap Kumar Valsan The GuC firmware had defined the interface for Translation Look-Aside Buffer (TLB) invalidation. We should use this interface when invalidating the engine and GuC TLBs. Add additional functionality to intel_gt_invalidate_tlb, invalidating the GuC TLBs and falling back to GT invalidation when the GuC is disabled. The invalidation is done by sending a request directly to the GuC tlb_lookup that invalidates the table. The invalidation is submitted as a wait request and is performed in the CT event handler. This means we cannot perform this TLB invalidation path if the CT is not enabled. If the request isn't fulfilled in two seconds, this would constitute an error in the invalidation as that would constitute either a lost request or a severe GuC overload. With this new invalidation routine, we can perform GuC-based GGTT invalidations. GuC-based GGTT invalidation is incompatible with MMIO invalidation so we should not perform MMIO invalidation when GuC-based GGTT invalidation is expected. Purpose of xarray: The tlb_lookup table is allocated as an xarray because the set of pending TLB invalidations may have no upper bound. The consequence of this is that all actions interfacing with this table need to use the xarray functions, such as xa_alloc_cyclic_irq for array insertion. Purpose of must_wait_woken: Our wait for the G2H ack for the completion of a TLB invalidation is mandatory; we must wait for the HW to confirm that the physical addresses are no longer accessible before we return those to the system. On switching to using the wait_woken() convenience routine, we introduced ourselves to an issue where wait_woken() may complete early under a kthread that is stopped. Since we send a TLB invalidation when we try to release pages from the shrinker, we can be called from any process; including kthreads. Using wait_woken() from any process context causes another issue. The use of is_kthread_should_stop() assumes that any task with PF_KTHREAD set was made by kthread_create() and has called set_kthread_struct(). This is not true for the raw kernel_thread(): BUG: kernel NULL pointer dereference, address: 0000000000000000 [ 3089.759660] Call Trace: [ 3089.762110] wait_woken+0x4f/0x80 [ 3089.765496] guc_send_invalidate_tlb+0x1fe/0x310 [i915] [ 3089.770725] ? syscall_return_via_sysret+0xf/0x7f [ 3089.775426] ? do_wait_intr_irq+0xb0/0xb0 [ 3089.779430] ? __switch_to_asm+0x40/0x70 [ 3089.783349] ? __switch_to_asm+0x34/0x70 [ 3089.787273] ? __switch_to+0x7a/0x3e0 [ 3089.790930] ? __switch_to_asm+0x34/0x70 [ 3089.794883] intel_guc_invalidate_tlb_full+0x92/0xa0 [i915] [ 3089.800487] intel_invalidate_tlb_full+0x94/0x190 [i915] [ 3089.805824] intel_invalidate_tlb_full_sync+0x1b/0x30 [i915] [ 3089.811508] __i915_gem_object_unset_pages+0x138/0x150 [i915] [ 3089.817279] __i915_gem_object_put_pages+0x25/0x90 [i915] [ 3089.822706] i915_gem_shrink+0x532/0x7e0 [i915] [ 3089.827264] i915_gem_shrinker_scan+0x3d/0xd0 [i915] [ 3089.832230] do_shrink_slab+0x12c/0x2a0 [ 3089.836065] shrink_slab+0xad/0x2b0 [ 3089.839550] shrink_node+0xcc/0x410 [ 3089.843035] do_try_to_free_pages+0xc6/0x380 [ 3089.847306] try_to_free_pages+0xec/0x1c0 [ 3089.851312] __alloc_pages_slowpath+0x3ad/0xd10 [ 3089.855845] ? update_sd_lb_stats+0x636/0x710 [ 3089.860204] __alloc_pages_nodemask+0x2d5/0x310 [ 3089.864737] new_slab+0x265/0xa80 [ 3089.868053] ___slab_alloc+0y_to_free_pages+0xec/0x1c0 [ 3089.871798] ? copy_process+0x1e5/0x1a00 [ 3089.875717] ? load_balance+0x165/0xb20 [ 3089.879555] __slab_alloc+0x1c/0x30 [ 3089.883047] kmem_cache_alloc_node+0x9f/0x240 [ 3089.887397] ? copy_process+0x1e5/0x1a00 [ 3089.891314] copy_process+0x1e5/0x1a00 [ 3089.895058] ? __switch_to_asm+0x40/0x70 [ 3089.879555] __slab_alloc+0x1c/0x30 [ 3089.883047] kmem_cache_alloc_node+0x9f/0x240 [ 3089.887397] ? copy_process+0x1e5/0x1a00 [ 3089.891314] copy_process+0x1e5/0x1a00 [ 3089.895058] ? __switch_to_asm+0x40/0x70 [ 3089.898977] ? __switch_to_asm+0x34/0x70 [ 3089.902903] ? __switch_to_asm+0x40/0x70 [ 3089.906828] ? __switch_to_asm+0x34/0x70 [ 3089.910745] _do_fork+0x83/0x350 [ 3089.913969] ? __switch_to+0x7a/0x3e0 [ 3089.917626] ? __switch_to_asm+0x34/0x70 [ 3089.921545] kernel_thread+0x58/0x80 [ 3089.925124] ? kthread_park+0x80/0x80 [ 3089.928788] kthreadd+0x162/0x1b0 [ 3089.932098] ? kthread_create_on_cpu+0xa0/0xa0 [ 3089.936538] ret_from_fork+0x1f/0x40 Signed-off-by: Prathap Kumar Valsan Signed-off-by: Bruce Chang Signed-off-by: Chris Wilson Signed-off-by: Umesh Nerlige Ramappa Signed-off-by: Jonathan Cavitt Signed-off-by: Aravind Iddamsetty Signed-off-by: Fei Yang CC: Andi Shyti --- drivers/gpu/drm/i915/gt/intel_ggtt.c | 34 ++- drivers/gpu/drm/i915/gt/intel_tlb.c | 15 +- .../gpu/drm/i915/gt/uc/abi/guc_actions_abi.h | 33 +++ drivers/gpu/drm/i915/gt/uc/intel_guc.h | 22 ++ drivers/gpu/drm/i915/gt/uc/intel_guc_ct.c | 4 + drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h | 1 + .../gpu/drm/i915/gt/uc/intel_guc_submission.c | 200 +++++++++++++++++- 7 files changed, 297 insertions(+), 12 deletions(-) diff --git a/drivers/gpu/drm/i915/gt/intel_ggtt.c b/drivers/gpu/drm/i915/gt/intel_ggtt.c index 4d7d88b92632b..a1f7bdc602996 100644 --- a/drivers/gpu/drm/i915/gt/intel_ggtt.c +++ b/drivers/gpu/drm/i915/gt/intel_ggtt.c @@ -206,22 +206,38 @@ static void gen8_ggtt_invalidate(struct i915_ggtt *ggtt) intel_uncore_write_fw(uncore, GFX_FLSH_CNTL_GEN6, GFX_FLSH_CNTL_EN); } +static void guc_ggtt_ct_invalidate(struct intel_gt *gt) +{ + struct intel_uncore *uncore = gt->uncore; + intel_wakeref_t wakeref; + + with_intel_runtime_pm_if_active(uncore->rpm, wakeref) { + struct intel_guc *guc = >->uc.guc; + + intel_guc_invalidate_tlb_guc(guc); + } +} + static void guc_ggtt_invalidate(struct i915_ggtt *ggtt) { struct drm_i915_private *i915 = ggtt->vm.i915; + struct intel_gt *gt; - gen8_ggtt_invalidate(ggtt); - - if (GRAPHICS_VER(i915) >= 12) { - struct intel_gt *gt; + if (!HAS_GUC_TLB_INVALIDATION(i915)) + gen8_ggtt_invalidate(ggtt); - list_for_each_entry(gt, &ggtt->gt_list, ggtt_link) + list_for_each_entry(gt, &ggtt->gt_list, ggtt_link) { + if (HAS_GUC_TLB_INVALIDATION(i915) && + intel_guc_is_ready(>->uc.guc)) { + guc_ggtt_ct_invalidate(gt); + } else if (GRAPHICS_VER(i915) >= 12) { intel_uncore_write_fw(gt->uncore, GEN12_GUC_TLB_INV_CR, GEN12_GUC_TLB_INV_CR_INVALIDATE); - } else { - intel_uncore_write_fw(ggtt->vm.gt->uncore, - GEN8_GTCR, GEN8_GTCR_INVALIDATE); + } else { + intel_uncore_write_fw(gt->uncore, + GEN8_GTCR, GEN8_GTCR_INVALIDATE); + } } } @@ -1243,7 +1259,7 @@ static int gen8_gmch_probe(struct i915_ggtt *ggtt) ggtt->vm.raw_insert_page = gen8_ggtt_insert_page; } - if (intel_uc_wants_guc(&ggtt->vm.gt->uc)) + if (intel_uc_wants_guc_submission(&ggtt->vm.gt->uc)) ggtt->invalidate = guc_ggtt_invalidate; else ggtt->invalidate = gen8_ggtt_invalidate; diff --git a/drivers/gpu/drm/i915/gt/intel_tlb.c b/drivers/gpu/drm/i915/gt/intel_tlb.c index 139608c30d978..1278622b9554d 100644 --- a/drivers/gpu/drm/i915/gt/intel_tlb.c +++ b/drivers/gpu/drm/i915/gt/intel_tlb.c @@ -12,6 +12,7 @@ #include "intel_gt_print.h" #include "intel_gt_regs.h" #include "intel_tlb.h" +#include "uc/intel_guc.h" /* * HW architecture suggest typical invalidation time at 40us, @@ -131,11 +132,23 @@ void intel_gt_invalidate_tlb_full(struct intel_gt *gt, u32 seqno) return; with_intel_gt_pm_if_awake(gt, wakeref) { + struct intel_guc *guc = >->uc.guc; + mutex_lock(>->tlb.invalidate_lock); if (tlb_seqno_passed(gt, seqno)) goto unlock; - mmio_invalidate_full(gt); + if (HAS_GUC_TLB_INVALIDATION(gt->i915)) { + /* + * Only perform GuC TLB invalidation if GuC is ready. + * If GuC is not ready, then there are no TLBs to + * invalidate. Ergo, skip invalidation. + */ + if (intel_guc_is_ready(guc)) + intel_guc_invalidate_tlb_engines(guc); + } else { + mmio_invalidate_full(gt); + } write_seqcount_invalidate(>->tlb.seqno); unlock: diff --git a/drivers/gpu/drm/i915/gt/uc/abi/guc_actions_abi.h b/drivers/gpu/drm/i915/gt/uc/abi/guc_actions_abi.h index f359bef046e0b..33f253410d0c8 100644 --- a/drivers/gpu/drm/i915/gt/uc/abi/guc_actions_abi.h +++ b/drivers/gpu/drm/i915/gt/uc/abi/guc_actions_abi.h @@ -138,6 +138,8 @@ enum intel_guc_action { INTEL_GUC_ACTION_REGISTER_CONTEXT_MULTI_LRC = 0x4601, INTEL_GUC_ACTION_CLIENT_SOFT_RESET = 0x5507, INTEL_GUC_ACTION_SET_ENG_UTIL_BUFF = 0x550A, + INTEL_GUC_ACTION_TLB_INVALIDATION = 0x7000, + INTEL_GUC_ACTION_TLB_INVALIDATION_DONE = 0x7001, INTEL_GUC_ACTION_STATE_CAPTURE_NOTIFICATION = 0x8002, INTEL_GUC_ACTION_NOTIFY_FLUSH_LOG_BUFFER_TO_FILE = 0x8003, INTEL_GUC_ACTION_NOTIFY_CRASH_DUMP_POSTED = 0x8004, @@ -181,4 +183,35 @@ enum intel_guc_state_capture_event_status { #define INTEL_GUC_STATE_CAPTURE_EVENT_STATUS_MASK 0x000000FF +#define INTEL_GUC_TLB_INVAL_TYPE_MASK REG_GENMASK(7, 0) +#define INTEL_GUC_TLB_INVAL_MODE_MASK REG_GENMASK(11, 8) +#define INTEL_GUC_TLB_INVAL_FLUSH_CACHE REG_BIT(31) + +enum intel_guc_tlb_invalidation_type { + INTEL_GUC_TLB_INVAL_ENGINES = 0x0, + INTEL_GUC_TLB_INVAL_GUC = 0x3, +}; + +/* + * 0: Heavy mode of Invalidation: + * The pipeline of the engine(s) for which the invalidation is targeted to is + * blocked, and all the in-flight transactions are guaranteed to be Globally + * Observed before completing the TLB invalidation + * 1: Lite mode of Invalidation: + * TLBs of the targeted engine(s) are immediately invalidated. + * In-flight transactions are NOT guaranteed to be Globally Observed before + * completing TLB invalidation. + * Light Invalidation Mode is to be used only when + * it can be guaranteed (by SW) that the address translations remain invariant + * for the in-flight transactions across the TLB invalidation. In other words, + * this mode can be used when the TLB invalidation is intended to clear out the + * stale cached translations that are no longer in use. Light Invalidation Mode + * is much faster than the Heavy Invalidation Mode, as it does not wait for the + * in-flight transactions to be GOd. + */ +enum intel_guc_tlb_inval_mode { + INTEL_GUC_TLB_INVAL_MODE_HEAVY = 0x0, + INTEL_GUC_TLB_INVAL_MODE_LITE = 0x1, +}; + #endif /* _ABI_GUC_ACTIONS_ABI_H */ diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc.h b/drivers/gpu/drm/i915/gt/uc/intel_guc.h index 6c392bad29c19..88deb43bbdc48 100644 --- a/drivers/gpu/drm/i915/gt/uc/intel_guc.h +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc.h @@ -79,6 +79,18 @@ struct intel_guc { */ atomic_t outstanding_submission_g2h; + /** @tlb_lookup: xarray to store all pending TLB invalidation requests */ + struct xarray tlb_lookup; + + /** + * @serial_slot: id to the initial waiter created in tlb_lookup, + * which is used only when failed to allocate new waiter. + */ + u32 serial_slot; + + /** @next_seqno: the next id (sequence no.) to allocate. */ + u32 next_seqno; + /** @interrupts: pointers to GuC interrupt-managing functions. */ struct { bool enabled; @@ -296,6 +308,11 @@ struct intel_guc { #define MAKE_GUC_VER_STRUCT(ver) MAKE_GUC_VER((ver).major, (ver).minor, (ver).patch) #define GUC_SUBMIT_VER(guc) MAKE_GUC_VER_STRUCT((guc)->submission_version) +struct intel_guc_tlb_wait { + struct wait_queue_head wq; + bool busy; +}; + static inline struct intel_guc *log_to_guc(struct intel_guc_log *log) { return container_of(log, struct intel_guc, log); @@ -418,6 +435,11 @@ static inline bool intel_guc_is_supported(struct intel_guc *guc) return intel_uc_fw_is_supported(&guc->fw); } +int intel_guc_invalidate_tlb_engines(struct intel_guc *guc); +int intel_guc_invalidate_tlb_guc(struct intel_guc *guc); +int intel_guc_tlb_invalidation_done(struct intel_guc *guc, u32 size, + u32 len, u32 seqno); + static inline bool intel_guc_is_wanted(struct intel_guc *guc) { return intel_uc_fw_is_enabled(&guc->fw); diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_ct.c b/drivers/gpu/drm/i915/gt/uc/intel_guc_ct.c index 6e22af31513a5..e7ac8402245b7 100644 --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_ct.c +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_ct.c @@ -1115,6 +1115,9 @@ static int ct_process_request(struct intel_guc_ct *ct, struct ct_incoming_msg *r case INTEL_GUC_ACTION_NOTIFY_EXCEPTION: ret = intel_guc_crash_process_msg(guc, action); break; + case INTEL_GUC_ACTION_TLB_INVALIDATION_DONE: + ret = intel_guc_tlb_invalidation_done(guc, request->size, len, payload[0]); + break; default: ret = -EOPNOTSUPP; break; @@ -1186,6 +1189,7 @@ static int ct_handle_event(struct intel_guc_ct *ct, struct ct_incoming_msg *requ switch (action) { case INTEL_GUC_ACTION_SCHED_CONTEXT_MODE_DONE: case INTEL_GUC_ACTION_DEREGISTER_CONTEXT_DONE: + case INTEL_GUC_ACTION_TLB_INVALIDATION_DONE: g2h_release_space(ct, request->size); } diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h b/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h index b4d56eccfb1f0..a7c9874e122a3 100644 --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h @@ -22,6 +22,7 @@ /* Payload length only i.e. don't include G2H header length */ #define G2H_LEN_DW_SCHED_CONTEXT_MODE_SET 2 #define G2H_LEN_DW_DEREGISTER_CONTEXT 1 +#define G2H_LEN_DW_INVALIDATE_TLB 1 #define GUC_CONTEXT_DISABLE 0 #define GUC_CONTEXT_ENABLE 1 diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c b/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c index 2cce5ec1ff00d..0c5ccda1b3e87 100644 --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c @@ -1798,9 +1798,11 @@ static void __guc_reset_context(struct intel_context *ce, intel_engine_mask_t st void intel_guc_submission_reset(struct intel_guc *guc, intel_engine_mask_t stalled) { + struct intel_guc_tlb_wait *wait; struct intel_context *ce; unsigned long index; unsigned long flags; + unsigned long i; if (unlikely(!guc_submission_initialized(guc))) { /* Reset called during driver load? GuC not yet initialised! */ @@ -1826,6 +1828,18 @@ void intel_guc_submission_reset(struct intel_guc *guc, intel_engine_mask_t stall /* GuC is blown away, drop all references to contexts */ xa_destroy(&guc->context_lookup); + + /* + * The full GT reset will have cleared the TLB caches and flushed the + * G2H message queue; we can release all the blocked waiters. + * + * This is safe to do unlocked because the xarray is not dependent + * on the GT reset, and there's a separate execution path for TLB + * invalidations on GT reset, and there's a large window of time + * between the GT reset and GuC becoming available. + */ + xa_for_each(&guc->tlb_lookup, i, wait) + wake_up(&wait->wq); } static void guc_cancel_context_requests(struct intel_context *ce) @@ -1948,6 +1962,50 @@ void intel_guc_submission_reset_finish(struct intel_guc *guc) static void destroyed_worker_func(struct work_struct *w); static void reset_fail_worker_func(struct work_struct *w); +static int init_tlb_lookup(struct intel_guc *guc) +{ + struct intel_guc_tlb_wait *wait; + int err; + + if (!HAS_GUC_TLB_INVALIDATION(guc_to_gt(guc)->i915)) + return 0; + + xa_init_flags(&guc->tlb_lookup, XA_FLAGS_ALLOC); + + wait = kzalloc(sizeof(*wait), GFP_KERNEL); + if (!wait) + return -ENOMEM; + + init_waitqueue_head(&wait->wq); + + /* Preallocate a shared id for use under memory pressure. */ + err = xa_alloc_cyclic_irq(&guc->tlb_lookup, &guc->serial_slot, wait, + xa_limit_32b, &guc->next_seqno, GFP_KERNEL); + /* Only error if out of memory, not when busy (list full)*/ + if (err == -ENOMEM) { + kfree(wait); + return err; + } + + return 0; +} + +static void fini_tlb_lookup(struct intel_guc *guc) +{ + struct intel_guc_tlb_wait *wait; + + if (!HAS_GUC_TLB_INVALIDATION(guc_to_gt(guc)->i915)) + return; + + wait = xa_load(&guc->tlb_lookup, guc->serial_slot); + if (wait) { + guc_dbg(guc, "fini_tlb_lookup: Unexpected item in tlb_lookup\n"); + kfree(wait); + } + + xa_destroy(&guc->tlb_lookup); +} + /* * Set up the memory resources to be shared with the GuC (via the GGTT) * at firmware loading time. @@ -1966,11 +2024,15 @@ int intel_guc_submission_init(struct intel_guc *guc) return ret; } + ret = init_tlb_lookup(guc); + if (ret) + goto destroy_pool; + guc->submission_state.guc_ids_bitmap = bitmap_zalloc(NUMBER_MULTI_LRC_GUC_ID(guc), GFP_KERNEL); if (!guc->submission_state.guc_ids_bitmap) { ret = -ENOMEM; - goto destroy_pool; + goto destroy_tlb; } guc->timestamp.ping_delay = (POLL_TIME_CLKS / gt->clock_frequency + 1) * HZ; @@ -1979,9 +2041,10 @@ int intel_guc_submission_init(struct intel_guc *guc) return 0; +destroy_tlb: + fini_tlb_lookup(guc); destroy_pool: guc_lrc_desc_pool_destroy_v69(guc); - return ret; } @@ -1994,6 +2057,7 @@ void intel_guc_submission_fini(struct intel_guc *guc) guc_lrc_desc_pool_destroy_v69(guc); i915_sched_engine_put(guc->sched_engine); bitmap_free(guc->submission_state.guc_ids_bitmap); + fini_tlb_lookup(guc); guc->submission_initialized = false; } @@ -4624,6 +4688,138 @@ g2h_context_lookup(struct intel_guc *guc, u32 ctx_id) return ce; } +static void wait_wake_outstanding_tlb_g2h(struct intel_guc *guc, u32 seqno) +{ + struct intel_guc_tlb_wait *wait; + unsigned long flags; + + xa_lock_irqsave(&guc->tlb_lookup, flags); + wait = xa_load(&guc->tlb_lookup, seqno); + + if (wait) + wake_up(&wait->wq); + else + guc_dbg(guc, + "Stale TLB invalidation response with seqno %d\n", seqno); + + xa_unlock_irqrestore(&guc->tlb_lookup, flags); +} + +int intel_guc_tlb_invalidation_done(struct intel_guc *guc, u32 size, u32 len, u32 seqno) +{ + /* Check for underflow */ + if (unlikely(len < 1 || len > size)) + return -EPROTO; + + wait_wake_outstanding_tlb_g2h(guc, seqno); + return 0; +} + +static long must_wait_woken(struct wait_queue_entry *wq_entry, long timeout) +{ + /* + * This is equivalent to wait_woken() with the exception that + * we do not wake up early if the kthread task has been completed. + * As we are called from page reclaim in any task context, + * we may be invoked from stopped kthreads, but we *must* + * complete the wait from the HW . + * + * A second problem is that since we are called under reclaim + * and wait_woken() inspected the thread state, it makes an invalid + * assumption that all PF_KTHREAD tasks have set_kthread_struct() + * called upon them, and will trigger a GPF in is_kthread_should_stop(). + */ + do { + set_current_state(TASK_UNINTERRUPTIBLE); + if (wq_entry->flags & WQ_FLAG_WOKEN) + break; + + timeout = schedule_timeout(timeout); + } while (timeout); + __set_current_state(TASK_RUNNING); + + /* See wait_woken() and woken_wake_function() */ + smp_store_mb(wq_entry->flags, wq_entry->flags & ~WQ_FLAG_WOKEN); + + return timeout; +} + +static int guc_send_invalidate_tlb(struct intel_guc *guc, enum intel_guc_tlb_inval_mode type) +{ + struct intel_guc_tlb_wait _wq, *wq = &_wq; + DEFINE_WAIT_FUNC(wait, woken_wake_function); + int err; + u32 seqno; + u32 action[] = { + INTEL_GUC_ACTION_TLB_INVALIDATION, + 0, + REG_FIELD_PREP(INTEL_GUC_TLB_INVAL_TYPE_MASK, type) | + REG_FIELD_PREP(INTEL_GUC_TLB_INVAL_MODE_MASK, + INTEL_GUC_TLB_INVAL_MODE_HEAVY) | + INTEL_GUC_TLB_INVAL_FLUSH_CACHE, + }; + u32 size = ARRAY_SIZE(action); + + init_waitqueue_head(&_wq.wq); + + if (xa_alloc_cyclic_irq(&guc->tlb_lookup, &seqno, wq, + xa_limit_32b, &guc->next_seqno, + GFP_ATOMIC | __GFP_NOWARN) < 0) { + /* Under severe memory pressure? Serialise TLB allocations */ + xa_lock_irq(&guc->tlb_lookup); + wq = xa_load(&guc->tlb_lookup, guc->serial_slot); + wait_event_lock_irq(wq->wq, + !READ_ONCE(wq->busy), + guc->tlb_lookup.xa_lock); + /* + * Update wq->busy under lock to ensure only one waiter can + * issue the TLB invalidation command using the serial slot at a + * time. The condition is set to true before releasing the lock + * so that other caller continue to wait until woken up again. + */ + wq->busy = true; + xa_unlock_irq(&guc->tlb_lookup); + + seqno = guc->serial_slot; + } + + action[1] = seqno; + + add_wait_queue(&wq->wq, &wait); + + /* + * This is a critical reclaim path and thus we must loop here: + * We cannot block for anything that is on the GPU. + */ + err = intel_guc_send_busy_loop(guc, action, size, G2H_LEN_DW_INVALIDATE_TLB, true); + if (err) + goto out; + + if (!must_wait_woken(&wait, intel_guc_ct_expected_delay(&guc->ct))) { + guc_err(guc, + "TLB invalidation response timed out for seqno %u\n", seqno); + err = -ETIME; + } +out: + remove_wait_queue(&wq->wq, &wait); + if (seqno != guc->serial_slot) + xa_erase_irq(&guc->tlb_lookup, seqno); + + return err; +} + +/* Full TLB invalidation */ +int intel_guc_invalidate_tlb_engines(struct intel_guc *guc) +{ + return guc_send_invalidate_tlb(guc, INTEL_GUC_TLB_INVAL_ENGINES); +} + +/* GuC TLB Invalidation: Invalidate the TLB's of GuC itself. */ +int intel_guc_invalidate_tlb_guc(struct intel_guc *guc) +{ + return guc_send_invalidate_tlb(guc, INTEL_GUC_TLB_INVAL_GUC); +} + int intel_guc_deregister_done_process_msg(struct intel_guc *guc, const u32 *msg, u32 len) From patchwork Fri Oct 6 18:20:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Cavitt, Jonathan" X-Patchwork-Id: 13411828 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 40344E81E1F for ; Fri, 6 Oct 2023 18:31:42 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 39FC610E558; Fri, 6 Oct 2023 18:31:39 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.65]) by gabe.freedesktop.org (Postfix) with ESMTPS id 13C7110E558 for ; Fri, 6 Oct 2023 18:31:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1696617095; x=1728153095; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=5q0KsDcxYluar3gfGlFI+QRas+MI4vT0rKvAYUnkI28=; b=Rl/OLnpLNIWtQ1x0MHs8VMcxb2r8FTpTDmNaJcJMdOvkJ+mbDYXloQVA Dm9VDgCIuvy0E21FrFFM0TzBkX16r16XzwWwXYC5Hqzr7PeCQzRjEjgvy hac8Agfbjit/CZVC3F/f54CxRiIiblFQLmfdL55g5IfOHsgkk6DTRI5bf 0WTPaR9uxrN1YQW2c+yDVdCQkb+OEa9/B3le4l38rtPx8S+rMh+1DV4i/ BVjZXCRSbeEEHB+/2xKR9yoV0ZMOaKIgMJPxj5E/2qSUtQiNEkVGEv3x1 bxcfxt7+vZ7sBbM9iclnApsZ448mgR6IrzTjhP5VJpJm2RLcUsCx0wvXK w==; X-IronPort-AV: E=McAfee;i="6600,9927,10855"; a="387688167" X-IronPort-AV: E=Sophos;i="6.03,204,1694761200"; d="scan'208";a="387688167" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 11:31:34 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10855"; a="745950585" X-IronPort-AV: E=Sophos;i="6.03,204,1694761200"; d="scan'208";a="745950585" Received: from dut-internal-9dd7.jf.intel.com ([10.165.21.194]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 11:31:34 -0700 From: Jonathan Cavitt To: intel-gfx@lists.freedesktop.org Date: Fri, 6 Oct 2023 11:20:48 -0700 Message-Id: <20231006182051.1599923-5-jonathan.cavitt@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231006182051.1599923-1-jonathan.cavitt@intel.com> References: <20231006182051.1599923-1-jonathan.cavitt@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v8 4/7] drm/i915: No TLB invalidation on suspended GT X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: janusz.krzysztofik@intel.com, andi.shyti@intel.com, matthew.d.roper@intel.com, jonathan.cavitt@intel.com, chris.p.wilson@linux.intel.com, nirmoy.das@intel.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" In case of GT is suspended, don't allow submission of new TLB invalidation request and cancel all pending requests. The TLB entries will be invalidated either during GuC reload or on system resume. Signed-off-by: Fei Yang Signed-off-by: Jonathan Cavitt CC: John Harrison --- drivers/gpu/drm/i915/gt/intel_gt.h | 1 + drivers/gpu/drm/i915/gt/uc/intel_guc.h | 1 + .../gpu/drm/i915/gt/uc/intel_guc_submission.c | 23 +++++++++++++++---- drivers/gpu/drm/i915/gt/uc/intel_uc.c | 8 +++++++ drivers/gpu/drm/i915/i915_driver.c | 2 ++ 5 files changed, 31 insertions(+), 4 deletions(-) diff --git a/drivers/gpu/drm/i915/gt/intel_gt.h b/drivers/gpu/drm/i915/gt/intel_gt.h index 970bedf6b78a7..4e3bb221d2f4d 100644 --- a/drivers/gpu/drm/i915/gt/intel_gt.h +++ b/drivers/gpu/drm/i915/gt/intel_gt.h @@ -179,4 +179,5 @@ enum i915_map_type intel_gt_coherent_map_type(struct intel_gt *gt, void intel_gt_bind_context_set_ready(struct intel_gt *gt); void intel_gt_bind_context_set_unready(struct intel_gt *gt); bool intel_gt_is_bind_context_ready(struct intel_gt *gt); + #endif /* __INTEL_GT_H__ */ diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc.h b/drivers/gpu/drm/i915/gt/uc/intel_guc.h index 88deb43bbdc48..add74000e621e 100644 --- a/drivers/gpu/drm/i915/gt/uc/intel_guc.h +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc.h @@ -536,4 +536,5 @@ void intel_guc_dump_time_info(struct intel_guc *guc, struct drm_printer *p); int intel_guc_sched_disable_gucid_threshold_max(struct intel_guc *guc); +void wake_up_all_tlb_invalidate(struct intel_guc *guc); #endif diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c b/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c index 0c5ccda1b3e87..4ed6e9e759007 100644 --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c @@ -1796,13 +1796,23 @@ static void __guc_reset_context(struct intel_context *ce, intel_engine_mask_t st intel_context_put(parent); } -void intel_guc_submission_reset(struct intel_guc *guc, intel_engine_mask_t stalled) +void wake_up_all_tlb_invalidate(struct intel_guc *guc) { struct intel_guc_tlb_wait *wait; + unsigned long i; + + if (!HAS_GUC_TLB_INVALIDATION(guc_to_gt(guc)->i915)) + return; + xa_for_each(&guc->tlb_lookup, i, wait) { + wake_up(&wait->wq); + } +} + +void intel_guc_submission_reset(struct intel_guc *guc, intel_engine_mask_t stalled) +{ struct intel_context *ce; unsigned long index; unsigned long flags; - unsigned long i; if (unlikely(!guc_submission_initialized(guc))) { /* Reset called during driver load? GuC not yet initialised! */ @@ -1838,8 +1848,7 @@ void intel_guc_submission_reset(struct intel_guc *guc, intel_engine_mask_t stall * invalidations on GT reset, and there's a large window of time * between the GT reset and GuC becoming available. */ - xa_for_each(&guc->tlb_lookup, i, wait) - wake_up(&wait->wq); + wake_up_all_tlb_invalidate(guc); } static void guc_cancel_context_requests(struct intel_context *ce) @@ -1935,6 +1944,12 @@ void intel_guc_submission_cancel_requests(struct intel_guc *guc) /* GuC is blown away, drop all references to contexts */ xa_destroy(&guc->context_lookup); + + /* + * Wedged GT won't respond to any TLB invalidation request. Simply + * release all the blocked waiters. + */ + wake_up_all_tlb_invalidate(guc); } void intel_guc_submission_reset_finish(struct intel_guc *guc) diff --git a/drivers/gpu/drm/i915/gt/uc/intel_uc.c b/drivers/gpu/drm/i915/gt/uc/intel_uc.c index 98b103375b7ab..8d6a4d8ce61bb 100644 --- a/drivers/gpu/drm/i915/gt/uc/intel_uc.c +++ b/drivers/gpu/drm/i915/gt/uc/intel_uc.c @@ -688,6 +688,9 @@ void intel_uc_suspend(struct intel_uc *uc) /* flush the GSC worker */ intel_gsc_uc_flush_work(&uc->gsc); + if (HAS_GUC_TLB_INVALIDATION(guc_to_gt(guc)->i915)) + wake_up_all_tlb_invalidate(guc); + if (!intel_guc_is_ready(guc)) { guc->interrupts.enabled = false; return; @@ -736,6 +739,11 @@ static int __uc_resume(struct intel_uc *uc, bool enable_communication) intel_gsc_uc_resume(&uc->gsc); + if (HAS_GUC_TLB_INVALIDATION(gt->i915)) { + intel_guc_invalidate_tlb_engines(guc); + intel_guc_invalidate_tlb_guc(guc); + } + return 0; } diff --git a/drivers/gpu/drm/i915/i915_driver.c b/drivers/gpu/drm/i915/i915_driver.c index ccbb2834cde07..85ac9400c2dc7 100644 --- a/drivers/gpu/drm/i915/i915_driver.c +++ b/drivers/gpu/drm/i915/i915_driver.c @@ -72,6 +72,7 @@ #include "gt/intel_gt.h" #include "gt/intel_gt_pm.h" #include "gt/intel_rc6.h" +#include "gt/intel_tlb.h" #include "pxp/intel_pxp.h" #include "pxp/intel_pxp_debugfs.h" @@ -1093,6 +1094,7 @@ static int i915_drm_suspend(struct drm_device *dev) intel_dp_mst_suspend(dev_priv); intel_runtime_pm_disable_interrupts(dev_priv); + intel_hpd_cancel_work(dev_priv); intel_suspend_encoders(dev_priv); From patchwork Fri Oct 6 18:20:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Cavitt, Jonathan" X-Patchwork-Id: 13411829 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0D392E92FE0 for ; Fri, 6 Oct 2023 18:31:44 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id ABDD810E55A; Fri, 6 Oct 2023 18:31:39 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.65]) by gabe.freedesktop.org (Postfix) with ESMTPS id 6FDEC10E556 for ; Fri, 6 Oct 2023 18:31:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1696617095; x=1728153095; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=bcSX2+D4e1bIr9aUxgrNrhX9AG1NKFMR0HjEUurwKOE=; b=g72b5GAKLWVX1zXIAcms2L5LIzO01II1JeC4qwTPgbAzFvCj4ZM1MtJg Iddt7UROqMKATRaqrblnwl6QzzqqbmhK4EgLG/Eqt24ctKO7Q+fdWlG5/ RmQ6sNNg0GABQvpxTxBUmCT0Tpn24aDNyiS1wswQqcZO0JZxXa2NEbI/3 lb3SQWutaKDx6pPpWSNGlrxrZqfDt8UWuJXSobMJCV3/r0eK9XijX8y1Z ERM6PG1LXKKD6XC4FFCGcv0aCmUopxgvJlUPw2NPLimf8njlJ8rPp5EDd pV5nR58i9lOhcSsywxGFFhyyJlVpdXCCCRNcQRX2ZQ9sU2Mfsu11rWAGV A==; X-IronPort-AV: E=McAfee;i="6600,9927,10855"; a="387688169" X-IronPort-AV: E=Sophos;i="6.03,204,1694761200"; d="scan'208";a="387688169" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 11:31:34 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10855"; a="745950588" X-IronPort-AV: E=Sophos;i="6.03,204,1694761200"; d="scan'208";a="745950588" Received: from dut-internal-9dd7.jf.intel.com ([10.165.21.194]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 11:31:34 -0700 From: Jonathan Cavitt To: intel-gfx@lists.freedesktop.org Date: Fri, 6 Oct 2023 11:20:49 -0700 Message-Id: <20231006182051.1599923-6-jonathan.cavitt@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231006182051.1599923-1-jonathan.cavitt@intel.com> References: <20231006182051.1599923-1-jonathan.cavitt@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v8 5/7] drm/i915: No TLB invalidation on wedged GT X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: janusz.krzysztofik@intel.com, andi.shyti@intel.com, matthew.d.roper@intel.com, jonathan.cavitt@intel.com, chris.p.wilson@linux.intel.com, nirmoy.das@intel.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" It is not an error for GuC TLB invalidations to fail when the GT is wedged or disabled, so do not process a wait failure as one in guc_send_invalidate_tlb. Signed-off-by: Fei Yang Signed-off-by: Jonathan Cavitt CC: John Harrison --- drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c b/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c index 4ed6e9e759007..ddb3115715e41 100644 --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c @@ -32,6 +32,7 @@ #include "i915_drv.h" #include "i915_reg.h" +#include "i915_irq.h" #include "i915_trace.h" /** @@ -4759,6 +4760,14 @@ static long must_wait_woken(struct wait_queue_entry *wq_entry, long timeout) return timeout; } +static bool intel_gt_is_enabled(const struct intel_gt *gt) +{ + /* Check if GT is wedged or suspended */ + if (intel_gt_is_wedged(gt) || !intel_irqs_enabled(gt->i915)) + return false; + return true; +} + static int guc_send_invalidate_tlb(struct intel_guc *guc, enum intel_guc_tlb_inval_mode type) { struct intel_guc_tlb_wait _wq, *wq = &_wq; @@ -4810,7 +4819,8 @@ static int guc_send_invalidate_tlb(struct intel_guc *guc, enum intel_guc_tlb_inv if (err) goto out; - if (!must_wait_woken(&wait, intel_guc_ct_expected_delay(&guc->ct))) { + if (intel_gt_is_enabled(guc_to_gt(guc)) && + !must_wait_woken(&wait, intel_guc_ct_expected_delay(&guc->ct))) { guc_err(guc, "TLB invalidation response timed out for seqno %u\n", seqno); err = -ETIME; From patchwork Fri Oct 6 18:20:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Cavitt, Jonathan" X-Patchwork-Id: 13411832 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id ABB8FE81E1F for ; Fri, 6 Oct 2023 18:31:49 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 741A110E55D; Fri, 6 Oct 2023 18:31:40 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.65]) by gabe.freedesktop.org (Postfix) with ESMTPS id 7051110E558 for ; Fri, 6 Oct 2023 18:31:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1696617095; x=1728153095; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=2U0MbsEQiPUx+pLXjNv1MGLeocvVQk6m8MBuK/HLXGw=; b=U8xpQMZUxHQdf1IWqVkczqqmn69UI2EDmfKyYC7h1GSWQvOliow5QuT0 T/cJViYbnhEv0hKeTWF9Rvtign/cYy/rb6nv/h5LNDvjIH+3qPQTuT1un NcMMLqsXvlsO+JT50ZWVWG20JkRCY5VFgrvy2LqNXB7xG7ZbpO54LctoW v6GZM1b7p67tGhccz0cTlvI6p3eNbimxLxvaM9GEUh+BCUSokdxZ7opYh eLZ18Grk9lHLKIhm/XtBuxrvivFRVpaPgo9ZlJVQsMil3Opou4MRMPKTp TrUIGNp7mdPrmPdUrt1LeU9ZJO0ZCh610gJviv1o23ddsK9WVmDwNLjat Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10855"; a="387688170" X-IronPort-AV: E=Sophos;i="6.03,204,1694761200"; d="scan'208";a="387688170" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 11:31:34 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10855"; a="745950591" X-IronPort-AV: E=Sophos;i="6.03,204,1694761200"; d="scan'208";a="745950591" Received: from dut-internal-9dd7.jf.intel.com ([10.165.21.194]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 11:31:34 -0700 From: Jonathan Cavitt To: intel-gfx@lists.freedesktop.org Date: Fri, 6 Oct 2023 11:20:50 -0700 Message-Id: <20231006182051.1599923-7-jonathan.cavitt@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231006182051.1599923-1-jonathan.cavitt@intel.com> References: <20231006182051.1599923-1-jonathan.cavitt@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v8 6/7] drm/i915/gt: Increase sleep in gt_tlb selftest sanitycheck X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: janusz.krzysztofik@intel.com, andi.shyti@intel.com, matthew.d.roper@intel.com, jonathan.cavitt@intel.com, chris.p.wilson@linux.intel.com, nirmoy.das@intel.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" For the gt_tlb live selftest, when operating on the GSC engine, increase the timeout from 10 ms to 200 ms because the GSC engine is a bit slower than the rest. Signed-off-by: Jonathan Cavitt --- drivers/gpu/drm/i915/gt/selftest_tlb.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/drm/i915/gt/selftest_tlb.c b/drivers/gpu/drm/i915/gt/selftest_tlb.c index 7e41f69fc818f..8c3bb9893dae8 100644 --- a/drivers/gpu/drm/i915/gt/selftest_tlb.c +++ b/drivers/gpu/drm/i915/gt/selftest_tlb.c @@ -136,8 +136,15 @@ pte_tlbinv(struct intel_context *ce, i915_request_get(rq); i915_request_add(rq); - /* Short sleep to sanitycheck the batch is spinning before we begin */ - msleep(10); + /* + * Short sleep to sanitycheck the batch is spinning before we begin + * FIXME: Why is GSC so slow? + */ + if (ce->engine->class == OTHER_CLASS) + msleep(200); + else + msleep(10); + if (va == vb) { if (!i915_request_completed(rq)) { pr_err("%s(%s): Semaphore sanitycheck failed %llx, with alignment %llx, using PTE size %x (phys %x, sg %x)\n", From patchwork Fri Oct 6 18:20:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Cavitt, Jonathan" X-Patchwork-Id: 13411831 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 45C3CE92FE0 for ; Fri, 6 Oct 2023 18:31:47 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 1A3F310E55B; Fri, 6 Oct 2023 18:31:40 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.65]) by gabe.freedesktop.org (Postfix) with ESMTPS id 9EF6E10E554 for ; Fri, 6 Oct 2023 18:31:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1696617095; x=1728153095; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=sCPg81uT6dezkH2hVM6SDIRyOuX6w9glL6jnh8lBjcM=; b=CfyH2EkDkFxfyGB5vqVIOncH0pkjxArOWye8mD0EB86wY7U7h5CnPvVl qHbQg2t6afaoAdu52fBA0N/HUnePHMIjNqo5JB+iF4wbnuMRtGIHVigOU EKAUSkgJxZF3jGI1rap1U5/NIwYZtprG9sHAB4TaOoX8q6wxr19OSHm5v 5Y/QiGW77891Wk6HhiHFiaonsCauQjI7w0wi4T9cVm14XA0iPDSZtr55+ hrfiHZ4N3vXjVcPb+Dripweq3FywEZkExJP4DRW3WMXsikD5kf5BKeEjH T7xkDWHogSCvWjV8MxdIB6pXq2H+jLUkqbnEsULbuN6vGsDlrfNa5+5It g==; X-IronPort-AV: E=McAfee;i="6600,9927,10855"; a="387688171" X-IronPort-AV: E=Sophos;i="6.03,204,1694761200"; d="scan'208";a="387688171" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 11:31:34 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10855"; a="745950594" X-IronPort-AV: E=Sophos;i="6.03,204,1694761200"; d="scan'208";a="745950594" Received: from dut-internal-9dd7.jf.intel.com ([10.165.21.194]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2023 11:31:34 -0700 From: Jonathan Cavitt To: intel-gfx@lists.freedesktop.org Date: Fri, 6 Oct 2023 11:20:51 -0700 Message-Id: <20231006182051.1599923-8-jonathan.cavitt@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231006182051.1599923-1-jonathan.cavitt@intel.com> References: <20231006182051.1599923-1-jonathan.cavitt@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v8 7/7] drm/i915: Enable GuC TLB invalidations for MTL X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: janusz.krzysztofik@intel.com, andi.shyti@intel.com, matthew.d.roper@intel.com, jonathan.cavitt@intel.com, chris.p.wilson@linux.intel.com, nirmoy.das@intel.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" Enable GuC TLB invalidations for MTL. Though more platforms than just MTL support GuC TLB invalidations, MTL is presently the only platform that requires it for any purpose, so only enable it there for now to minimize cross-platform impact. Signed-off-by: Jonathan Cavitt --- drivers/gpu/drm/i915/i915_pci.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/gpu/drm/i915/i915_pci.c b/drivers/gpu/drm/i915/i915_pci.c index df7c261410f79..d4b51ececbb12 100644 --- a/drivers/gpu/drm/i915/i915_pci.c +++ b/drivers/gpu/drm/i915/i915_pci.c @@ -829,6 +829,7 @@ static const struct intel_device_info mtl_info = { .has_flat_ccs = 0, .has_gmd_id = 1, .has_guc_deprivilege = 1, + .has_guc_tlb_invalidation = 1, .has_llc = 0, .has_mslice_steering = 0, .has_snoop = 1,