From patchwork Thu Feb 11 13:29:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Christian_K=C3=B6nig?= X-Patchwork-Id: 12083265 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19B7EC433E0 for ; Thu, 11 Feb 2021 13:29:47 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C19F564DE0 for ; Thu, 11 Feb 2021 13:29:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C19F564DE0 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 8E9FF6E0EB; Thu, 11 Feb 2021 13:29:45 +0000 (UTC) Received: from mail-ej1-x62a.google.com (mail-ej1-x62a.google.com [IPv6:2a00:1450:4864:20::62a]) by gabe.freedesktop.org (Postfix) with ESMTPS id 7C9E96E0EB for ; Thu, 11 Feb 2021 13:29:44 +0000 (UTC) Received: by mail-ej1-x62a.google.com with SMTP id i8so10045849ejc.7 for ; Thu, 11 Feb 2021 05:29:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=6kVXaL62eNbd5DKEqcS0gAZae+RIUEySQd7TZO396/4=; b=fp8Cww/poZLvPSoQrQb0Av793kYNMCzcd6sx8uNYrQeLvW5SNSAtk7VZht9IwRosWQ hKz4HhTbfilpVy4HkivFQGxjVopXTjMrTcvPMpqgrldLI5y1A1lrT2In5e48F5olm2wK AIQmAuND567T9xMaaKQO/iWPcSASPZyf2hbRrCsFiJK6Zbsy8IX7uKIRJlLul+BRiPmi 6zZmOSZqVeYC6NI7cI6JLoa9XOxrVpst2ga5fXzeNqgPQax/E6RfMFV3TdAT75TaLrgE jxMdh8/o9w5QNoQDn2f4tQ0ueNarYOR1sZukuqvAzeY+91I03uz/pekhpGog/JNi/DN9 Ir+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=6kVXaL62eNbd5DKEqcS0gAZae+RIUEySQd7TZO396/4=; b=HCobuhnQ14uiDmzRXcL1JSLViwn7lotTU2BOhZm9r8WQxNCVpQf4JxleFgAiWZWsXb 8Nlz7aBkWhsvfRqNWF/ND7Op1byJ0ZlV5mwFrnNpuHeXAWytBnLPpBSgjBS59sAr81PV hiF1M+MKblNdoWAeMpUpQ3nwnDk1mgq6yvTT0IPcbanewkfXECfoAtM6RK6/T3h5PY5B vGSeXr4gRaY1cybRvWNd8Bkxl+OzQRE4iZTsc9+qGWWxWrvElc1OHV43TBbHpejSlipM bTzRyK4Ut1Cc3oGThvPhJPiiLYoATRbVDqMTJmxpmwuWHdHcxsxcR4tHyFuByVhytc07 kPQg== X-Gm-Message-State: AOAM533RbsSiOk+cucPjhAjDs6fomWC79w4c8dV/YYe6i7lH1CO14SyS OjW8zFFnU5HmpzcUZqDYTDuLOfXTM9lLbw== X-Google-Smtp-Source: ABdhPJzYjErSh4w1eGF/cBMekDvjolEDcGNYJq2/8Ah0UtHujHr+6noGDVlsemx4tKoFUJZzMxOydw== X-Received: by 2002:a17:906:5390:: with SMTP id g16mr8592285ejo.19.1613050183175; Thu, 11 Feb 2021 05:29:43 -0800 (PST) Received: from abel.fritz.box ([2a02:908:1252:fb60:820:5af6:9146:2c9f]) by smtp.gmail.com with ESMTPSA id w26sm3862627edq.46.2021.02.11.05.29.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Feb 2021 05:29:42 -0800 (PST) From: " =?utf-8?q?Christian_K=C3=B6nig?= " X-Google-Original-From: =?utf-8?q?Christian_K=C3=B6nig?= To: dri-devel@lists.freedesktop.org Subject: [PATCH 1/3] drm/ttm: move swapout logic around Date: Thu, 11 Feb 2021 14:29:39 +0100 Message-Id: <20210211132941.290772-1-christian.koenig@amd.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: ray.huang@amd.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Move the iteration of the global lru into the new function ttm_global_swapout() and use that instead in drivers. Signed-off-by: Christian König --- drivers/gpu/drm/ttm/ttm_bo.c | 57 ++++++++--------------------- drivers/gpu/drm/ttm/ttm_device.c | 29 +++++++++++++++ drivers/gpu/drm/ttm/ttm_tt.c | 2 +- drivers/gpu/drm/vmwgfx/ttm_memory.c | 3 +- drivers/gpu/drm/vmwgfx/vmwgfx_drv.c | 2 +- include/drm/ttm/ttm_bo_api.h | 3 +- include/drm/ttm/ttm_device.h | 2 + 7 files changed, 53 insertions(+), 45 deletions(-) diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c index e38102282fd5..d33578a112b4 100644 --- a/drivers/gpu/drm/ttm/ttm_bo.c +++ b/drivers/gpu/drm/ttm/ttm_bo.c @@ -1184,56 +1184,35 @@ int ttm_bo_wait(struct ttm_buffer_object *bo, } EXPORT_SYMBOL(ttm_bo_wait); -/* - * A buffer object shrink method that tries to swap out the first - * buffer object on the bo_global::swap_lru list. - */ -int ttm_bo_swapout(struct ttm_operation_ctx *ctx, gfp_t gfp_flags) +int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx, + gfp_t gfp_flags) { struct ttm_global *glob = &ttm_glob; - struct ttm_buffer_object *bo; - int ret = -EBUSY; bool locked; - unsigned i; - - spin_lock(&glob->lru_lock); - for (i = 0; i < TTM_MAX_BO_PRIORITY; ++i) { - list_for_each_entry(bo, &glob->swap_lru[i], swap) { - if (!ttm_bo_evict_swapout_allowable(bo, ctx, &locked, - NULL)) - continue; - - if (!ttm_bo_get_unless_zero(bo)) { - if (locked) - dma_resv_unlock(bo->base.resv); - continue; - } + int ret; - ret = 0; - break; - } - if (!ret) - break; - } + if (!ttm_bo_evict_swapout_allowable(bo, ctx, &locked, NULL)) + return -EBUSY; - if (ret) { - spin_unlock(&glob->lru_lock); - return ret; + if (!ttm_bo_get_unless_zero(bo)) { + if (locked) + dma_resv_unlock(bo->base.resv); + return -EBUSY; } if (bo->deleted) { - ret = ttm_bo_cleanup_refs(bo, false, false, locked); + ttm_bo_cleanup_refs(bo, false, false, locked); ttm_bo_put(bo); - return ret; + return 0; } ttm_bo_del_from_lru(bo); + /* TODO: Cleanup the locking */ spin_unlock(&glob->lru_lock); - /** + /* * Move to system cached */ - if (bo->mem.mem_type != TTM_PL_SYSTEM) { struct ttm_operation_ctx ctx = { false, false }; struct ttm_resource evict_mem; @@ -1253,29 +1232,26 @@ int ttm_bo_swapout(struct ttm_operation_ctx *ctx, gfp_t gfp_flags) } } - /** + /* * Make sure BO is idle. */ - ret = ttm_bo_wait(bo, false, false); if (unlikely(ret != 0)) goto out; ttm_bo_unmap_virtual(bo); - /** + /* * Swap out. Buffer will be swapped in again as soon as * anyone tries to access a ttm page. */ - if (bo->bdev->funcs->swap_notify) bo->bdev->funcs->swap_notify(bo); ret = ttm_tt_swapout(bo->bdev, bo->ttm, gfp_flags); out: - /** - * + /* * Unreserve without putting on LRU to avoid swapping out an * already swapped buffer. */ @@ -1284,7 +1260,6 @@ int ttm_bo_swapout(struct ttm_operation_ctx *ctx, gfp_t gfp_flags) ttm_bo_put(bo); return ret; } -EXPORT_SYMBOL(ttm_bo_swapout); void ttm_bo_tt_destroy(struct ttm_buffer_object *bo) { diff --git a/drivers/gpu/drm/ttm/ttm_device.c b/drivers/gpu/drm/ttm/ttm_device.c index 95e1b7b1f2e6..dfc2a7e4e490 100644 --- a/drivers/gpu/drm/ttm/ttm_device.c +++ b/drivers/gpu/drm/ttm/ttm_device.c @@ -102,6 +102,35 @@ static int ttm_global_init(void) return ret; } +/** + * A buffer object shrink method that tries to swap out the first + * buffer object on the global::swap_lru list. + */ +long ttm_global_swapout(struct ttm_operation_ctx *ctx, gfp_t gfp_flags) +{ + struct ttm_global *glob = &ttm_glob; + struct ttm_buffer_object *bo; + unsigned i; + int ret; + + spin_lock(&glob->lru_lock); + for (i = 0; i < TTM_MAX_BO_PRIORITY; ++i) { + list_for_each_entry(bo, &glob->swap_lru[i], swap) { + uint32_t num_pages = bo->ttm->num_pages; + + ret = ttm_bo_swapout(bo, ctx, gfp_flags); + /* ttm_bo_swapout has dropped the lru_lock */ + if (!ret) + return num_pages; + if (ret != -EBUSY) + return ret; + } + } + spin_unlock(&glob->lru_lock); + return 0; +} +EXPORT_SYMBOL(ttm_global_swapout); + static void ttm_init_sysman(struct ttm_device *bdev) { struct ttm_resource_manager *man = &bdev->sysman; diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c index 2f0833c98d2c..95b5cff25f4c 100644 --- a/drivers/gpu/drm/ttm/ttm_tt.c +++ b/drivers/gpu/drm/ttm/ttm_tt.c @@ -369,7 +369,7 @@ static unsigned long ttm_tt_shrinker_scan(struct shrinker *shrink, }; int ret; - ret = ttm_bo_swapout(&ctx, GFP_NOFS); + ret = ttm_global_swapout(&ctx, GFP_NOFS); return ret < 0 ? SHRINK_EMPTY : ret; } diff --git a/drivers/gpu/drm/vmwgfx/ttm_memory.c b/drivers/gpu/drm/vmwgfx/ttm_memory.c index e972af07d029..104b95a8c7a2 100644 --- a/drivers/gpu/drm/vmwgfx/ttm_memory.c +++ b/drivers/gpu/drm/vmwgfx/ttm_memory.c @@ -38,6 +38,7 @@ #include #include +#include #include "ttm_memory.h" @@ -277,7 +278,7 @@ static void ttm_shrink(struct ttm_mem_global *glob, bool from_wq, while (ttm_zones_above_swap_target(glob, from_wq, extra)) { spin_unlock(&glob->lock); - ret = ttm_bo_swapout(ctx, GFP_KERNEL); + ret = ttm_global_swapout(ctx, GFP_KERNEL); spin_lock(&glob->lock); if (unlikely(ret < 0)) break; diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c b/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c index 4efed3bf0ef9..01da355b86f3 100644 --- a/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c @@ -1384,7 +1384,7 @@ static int vmw_pm_freeze(struct device *kdev) vmw_execbuf_release_pinned_bo(dev_priv); vmw_resource_evict_all(dev_priv); vmw_release_device_early(dev_priv); - while (ttm_bo_swapout(&ctx, GFP_KERNEL) > 0); + while (ttm_global_swapout(&ctx, GFP_KERNEL) > 0); if (dev_priv->enable_fb) vmw_fifo_resource_dec(dev_priv); if (atomic_read(&dev_priv->num_fifo_resources) != 0) { diff --git a/include/drm/ttm/ttm_bo_api.h b/include/drm/ttm/ttm_bo_api.h index 4fb523dfab32..5044ac330858 100644 --- a/include/drm/ttm/ttm_bo_api.h +++ b/include/drm/ttm/ttm_bo_api.h @@ -560,7 +560,8 @@ ssize_t ttm_bo_io(struct ttm_device *bdev, struct file *filp, const char __user *wbuf, char __user *rbuf, size_t count, loff_t *f_pos, bool write); -int ttm_bo_swapout(struct ttm_operation_ctx *ctx, gfp_t gfp_flags); +int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx, + gfp_t gfp_flags); /** * ttm_bo_uses_embedded_gem_object - check if the given bo uses the diff --git a/include/drm/ttm/ttm_device.h b/include/drm/ttm/ttm_device.h index 035bbc044a3b..6a0b267d4fe6 100644 --- a/include/drm/ttm/ttm_device.h +++ b/include/drm/ttm/ttm_device.h @@ -297,6 +297,8 @@ struct ttm_device { struct delayed_work wq; }; +long ttm_global_swapout(struct ttm_operation_ctx *ctx, gfp_t gfp_flags); + static inline struct ttm_resource_manager * ttm_manager_type(struct ttm_device *bdev, int mem_type) { From patchwork Thu Feb 11 13:29:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Christian_K=C3=B6nig?= X-Patchwork-Id: 12083269 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3BCEC433E6 for ; Thu, 11 Feb 2021 13:29:54 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7136064E8A for ; Thu, 11 Feb 2021 13:29:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7136064E8A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 102AA6E415; Thu, 11 Feb 2021 13:29:47 +0000 (UTC) Received: from mail-ej1-x632.google.com (mail-ej1-x632.google.com [IPv6:2a00:1450:4864:20::632]) by gabe.freedesktop.org (Postfix) with ESMTPS id 15F816E0EB for ; Thu, 11 Feb 2021 13:29:45 +0000 (UTC) Received: by mail-ej1-x632.google.com with SMTP id y9so10002281ejp.10 for ; Thu, 11 Feb 2021 05:29:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=G1YlUwmYfs48fdEZ9G3J8L9qwvrvrX+jwyHNUDWNnV8=; b=dEDCdhvP/x7VWouLsNCWipGbLzpqktNgxByCA4b4dBpOSwyMMJ41MpZLuXqgiWPxz1 CaiLp+n6JFBdBZSvIjZz9X1i8+FAWcg3hq8+Uuat9MGWYH7gUcp6nb4g08ztfoTi5x8V P3EmonJUOa+JbANPBqiwnpNg870R+xdpEY2NlHFkqr5m1J9+4KwCMvCU9/d7JHJOEAfN oTj11gt4OsHs0ftmrMz5aLSYz//9ER8cr5yNjzQ7tK9TGA+7WQ6im3t/2IedxT80tY36 dA/onCVNVeoJ94q9LnAVpPxppZwii2hT6AUnXladu/gpoDnjHPv/JLEhotZEn5scq5i4 lJ8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=G1YlUwmYfs48fdEZ9G3J8L9qwvrvrX+jwyHNUDWNnV8=; b=XNvfigaakeNQ6e3wfOtzhSgUM6iY/9rxfE0Le+oUJVaDldAj6fv1fFttF7vql2FuaK xtl6UM2DloSLXD4nIi3d/vtUH/bKXJIiBN13uZjPTooeyKgP9pJQvhReLMNiAI5PhWiV XerQJfAnEn7XKTxZfjePqI6HBC4sxH/rxicTSoaNMAulRljEadzNqJ1n/Rqo8AmQjSJS VvB2JhbUp2smeIFggpSQQIvnKQdXBPz/jK6E2YD9X9P3LO4J18kPi6avhcJPIIHgqGO8 uK4fMXDhO3X2a+blX0Hvi1Y4VGXFkSFaO1VHLCEjHPbCG7BGeczu6xrv/mSzXq3XanLo sVmQ== X-Gm-Message-State: AOAM530GWxQbS9iac8QfcSfhkaKOrf7Ci5oNaDQPgzVGhKDBJHZArrh7 GlbtlmeT3NAD1U4//uYbMeGEE2QsHZNnaQ== X-Google-Smtp-Source: ABdhPJwOZ9MqTFabwt00304R9Jvhb8bu8w0YodEzk7Sl/TMQWmUHvEYsFB9Tp0BizwZwoHJ1CxnL+Q== X-Received: by 2002:a17:907:9495:: with SMTP id dm21mr8491888ejc.462.1613050183766; Thu, 11 Feb 2021 05:29:43 -0800 (PST) Received: from abel.fritz.box ([2a02:908:1252:fb60:820:5af6:9146:2c9f]) by smtp.gmail.com with ESMTPSA id w26sm3862627edq.46.2021.02.11.05.29.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Feb 2021 05:29:43 -0800 (PST) From: " =?utf-8?q?Christian_K=C3=B6nig?= " X-Google-Original-From: =?utf-8?q?Christian_K=C3=B6nig?= To: dri-devel@lists.freedesktop.org Subject: [PATCH 2/3] drm/ttm: remove swap LRU v2 Date: Thu, 11 Feb 2021 14:29:40 +0100 Message-Id: <20210211132941.290772-2-christian.koenig@amd.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210211132941.290772-1-christian.koenig@amd.com> References: <20210211132941.290772-1-christian.koenig@amd.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: ray.huang@amd.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Instead evict round robin from each devices SYSTEM and TT domain. v2: reorder num_pages access reported by Dan's script Signed-off-by: Christian König --- drivers/gpu/drm/ttm/ttm_bo.c | 31 ++------------- drivers/gpu/drm/ttm/ttm_bo_util.c | 1 - drivers/gpu/drm/ttm/ttm_device.c | 60 +++++++++++++++++++++-------- drivers/gpu/drm/vmwgfx/vmwgfx_drv.c | 2 +- include/drm/ttm/ttm_bo_api.h | 1 - include/drm/ttm/ttm_bo_driver.h | 1 - include/drm/ttm/ttm_device.h | 7 +--- 7 files changed, 52 insertions(+), 51 deletions(-) diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c index d33578a112b4..a1be88be357b 100644 --- a/drivers/gpu/drm/ttm/ttm_bo.c +++ b/drivers/gpu/drm/ttm/ttm_bo.c @@ -73,7 +73,6 @@ static void ttm_bo_del_from_lru(struct ttm_buffer_object *bo) { struct ttm_device *bdev = bo->bdev; - list_del_init(&bo->swap); list_del_init(&bo->lru); if (bdev->funcs->del_from_lru_notify) @@ -104,14 +103,6 @@ void ttm_bo_move_to_lru_tail(struct ttm_buffer_object *bo, man = ttm_manager_type(bdev, mem->mem_type); list_move_tail(&bo->lru, &man->lru[bo->priority]); - if (man->use_tt && bo->ttm && - !(bo->ttm->page_flags & (TTM_PAGE_FLAG_SG | - TTM_PAGE_FLAG_SWAPPED))) { - struct list_head *swap; - - swap = &ttm_glob.swap_lru[bo->priority]; - list_move_tail(&bo->swap, swap); - } if (bdev->funcs->del_from_lru_notify) bdev->funcs->del_from_lru_notify(bo); @@ -126,9 +117,6 @@ void ttm_bo_move_to_lru_tail(struct ttm_buffer_object *bo, ttm_bo_bulk_move_set_pos(&bulk->vram[bo->priority], bo); break; } - if (bo->ttm && !(bo->ttm->page_flags & - (TTM_PAGE_FLAG_SG | TTM_PAGE_FLAG_SWAPPED))) - ttm_bo_bulk_move_set_pos(&bulk->swap[bo->priority], bo); } } EXPORT_SYMBOL(ttm_bo_move_to_lru_tail); @@ -166,20 +154,6 @@ void ttm_bo_bulk_move_lru_tail(struct ttm_lru_bulk_move *bulk) list_bulk_move_tail(&man->lru[i], &pos->first->lru, &pos->last->lru); } - - for (i = 0; i < TTM_MAX_BO_PRIORITY; ++i) { - struct ttm_lru_bulk_move_pos *pos = &bulk->swap[i]; - struct list_head *lru; - - if (!pos->first) - continue; - - dma_resv_assert_held(pos->first->base.resv); - dma_resv_assert_held(pos->last->base.resv); - - lru = &ttm_glob.swap_lru[i]; - list_bulk_move_tail(lru, &pos->first->swap, &pos->last->swap); - } } EXPORT_SYMBOL(ttm_bo_bulk_move_lru_tail); @@ -1056,7 +1030,6 @@ int ttm_bo_init_reserved(struct ttm_device *bdev, kref_init(&bo->kref); INIT_LIST_HEAD(&bo->lru); INIT_LIST_HEAD(&bo->ddestroy); - INIT_LIST_HEAD(&bo->swap); bo->bdev = bdev; bo->type = type; bo->mem.mem_type = TTM_PL_SYSTEM; @@ -1191,6 +1164,10 @@ int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx, bool locked; int ret; + if (!bo->ttm || bo->ttm->page_flags & (TTM_PAGE_FLAG_SG | + TTM_PAGE_FLAG_SWAPPED)) + return false; + if (!ttm_bo_evict_swapout_allowable(bo, ctx, &locked, NULL)) return -EBUSY; diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c index 031e5819fec4..a2a17c84ceb3 100644 --- a/drivers/gpu/drm/ttm/ttm_bo_util.c +++ b/drivers/gpu/drm/ttm/ttm_bo_util.c @@ -303,7 +303,6 @@ static int ttm_buffer_object_transfer(struct ttm_buffer_object *bo, atomic_inc(&ttm_glob.bo_count); INIT_LIST_HEAD(&fbo->base.ddestroy); INIT_LIST_HEAD(&fbo->base.lru); - INIT_LIST_HEAD(&fbo->base.swap); fbo->base.moving = NULL; drm_vma_node_reset(&fbo->base.base.vma_node); diff --git a/drivers/gpu/drm/ttm/ttm_device.c b/drivers/gpu/drm/ttm/ttm_device.c index dfc2a7e4e490..2c280fb1e992 100644 --- a/drivers/gpu/drm/ttm/ttm_device.c +++ b/drivers/gpu/drm/ttm/ttm_device.c @@ -67,7 +67,6 @@ static int ttm_global_init(void) unsigned long num_pages; struct sysinfo si; int ret = 0; - unsigned i; mutex_lock(&ttm_global_mutex); if (++ttm_glob_use_count > 1) @@ -90,8 +89,6 @@ static int ttm_global_init(void) goto out; } - for (i = 0; i < TTM_MAX_BO_PRIORITY; ++i) - INIT_LIST_HEAD(&glob->swap_lru[i]); INIT_LIST_HEAD(&glob->device_list); atomic_set(&glob->bo_count, 0); @@ -109,27 +106,60 @@ static int ttm_global_init(void) long ttm_global_swapout(struct ttm_operation_ctx *ctx, gfp_t gfp_flags) { struct ttm_global *glob = &ttm_glob; + struct ttm_device *bdev; + int ret = -EBUSY; + + mutex_lock(&ttm_global_mutex); + list_for_each_entry(bdev, &glob->device_list, device_list) { + ret = ttm_device_swapout(bdev, ctx, gfp_flags); + if (ret > 0) { + list_move_tail(&bdev->device_list, &glob->device_list); + break; + } + } + mutex_unlock(&ttm_global_mutex); + return ret; +} +EXPORT_SYMBOL(ttm_global_swapout); + +long ttm_device_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx, + gfp_t gfp_flags) +{ + struct ttm_global *glob = &ttm_glob; + struct ttm_resource_manager *man; struct ttm_buffer_object *bo; - unsigned i; + unsigned i, j; int ret; spin_lock(&glob->lru_lock); - for (i = 0; i < TTM_MAX_BO_PRIORITY; ++i) { - list_for_each_entry(bo, &glob->swap_lru[i], swap) { - uint32_t num_pages = bo->ttm->num_pages; - - ret = ttm_bo_swapout(bo, ctx, gfp_flags); - /* ttm_bo_swapout has dropped the lru_lock */ - if (!ret) - return num_pages; - if (ret != -EBUSY) - return ret; + for (i = TTM_PL_SYSTEM; i < TTM_NUM_MEM_TYPES; ++i) { + man = ttm_manager_type(bdev, i); + if (!man || !man->use_tt) + continue; + + for (j = 0; j < TTM_MAX_BO_PRIORITY; ++j) { + list_for_each_entry(bo, &man->lru[j], lru) { + long num_pages; + + if (!bo->ttm || + bo->ttm->page_flags & TTM_PAGE_FLAG_SG || + bo->ttm->page_flags & TTM_PAGE_FLAG_SWAPPED) + continue; + + num_pages = bo->ttm->num_pages; + ret = ttm_bo_swapout(bo, ctx, gfp_flags); + /* ttm_bo_swapout has dropped the lru_lock */ + if (!ret) + return num_pages; + if (ret != -EBUSY) + return ret; + } } } spin_unlock(&glob->lru_lock); return 0; } -EXPORT_SYMBOL(ttm_global_swapout); +EXPORT_SYMBOL(ttm_device_swapout); static void ttm_init_sysman(struct ttm_device *bdev) { diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c b/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c index 01da355b86f3..cef896126361 100644 --- a/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c @@ -1384,7 +1384,7 @@ static int vmw_pm_freeze(struct device *kdev) vmw_execbuf_release_pinned_bo(dev_priv); vmw_resource_evict_all(dev_priv); vmw_release_device_early(dev_priv); - while (ttm_global_swapout(&ctx, GFP_KERNEL) > 0); + while (ttm_device_swapout(&dev_priv->bdev, &ctx, GFP_KERNEL) == 0); if (dev_priv->enable_fb) vmw_fifo_resource_dec(dev_priv); if (atomic_read(&dev_priv->num_fifo_resources) != 0) { diff --git a/include/drm/ttm/ttm_bo_api.h b/include/drm/ttm/ttm_bo_api.h index 5044ac330858..3587f660e8f4 100644 --- a/include/drm/ttm/ttm_bo_api.h +++ b/include/drm/ttm/ttm_bo_api.h @@ -144,7 +144,6 @@ struct ttm_buffer_object { struct list_head lru; struct list_head ddestroy; - struct list_head swap; /** * Members protected by a bo reservation. diff --git a/include/drm/ttm/ttm_bo_driver.h b/include/drm/ttm/ttm_bo_driver.h index 8959c0075cfd..d007feef7676 100644 --- a/include/drm/ttm/ttm_bo_driver.h +++ b/include/drm/ttm/ttm_bo_driver.h @@ -69,7 +69,6 @@ struct ttm_lru_bulk_move_pos { struct ttm_lru_bulk_move { struct ttm_lru_bulk_move_pos tt[TTM_MAX_BO_PRIORITY]; struct ttm_lru_bulk_move_pos vram[TTM_MAX_BO_PRIORITY]; - struct ttm_lru_bulk_move_pos swap[TTM_MAX_BO_PRIORITY]; }; /* diff --git a/include/drm/ttm/ttm_device.h b/include/drm/ttm/ttm_device.h index 6a0b267d4fe6..cda6efb4c34b 100644 --- a/include/drm/ttm/ttm_device.h +++ b/include/drm/ttm/ttm_device.h @@ -63,11 +63,6 @@ extern struct ttm_global { */ struct list_head device_list; - /** - * Protected by the lru_lock. - */ - struct list_head swap_lru[TTM_MAX_BO_PRIORITY]; - /** * Internal protection. */ @@ -298,6 +293,8 @@ struct ttm_device { }; long ttm_global_swapout(struct ttm_operation_ctx *ctx, gfp_t gfp_flags); +long ttm_device_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx, + gfp_t gfp_flags); static inline struct ttm_resource_manager * ttm_manager_type(struct ttm_device *bdev, int mem_type) From patchwork Thu Feb 11 13:29:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Christian_K=C3=B6nig?= X-Patchwork-Id: 12083267 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 498D2C433DB for ; Thu, 11 Feb 2021 13:29:52 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id ECCB864E7E for ; Thu, 11 Feb 2021 13:29:51 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ECCB864E7E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id E34F36E0F5; Thu, 11 Feb 2021 13:29:46 +0000 (UTC) Received: from mail-ej1-x636.google.com (mail-ej1-x636.google.com [IPv6:2a00:1450:4864:20::636]) by gabe.freedesktop.org (Postfix) with ESMTPS id BA94E6E0F5 for ; Thu, 11 Feb 2021 13:29:45 +0000 (UTC) Received: by mail-ej1-x636.google.com with SMTP id jj19so10093551ejc.4 for ; Thu, 11 Feb 2021 05:29:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=rnKMD0gFWbm5rjlkLlN2rxa4obGzTJq1MtbE2gwCtoY=; b=gCeTYuDcZCBE6g9+L1Y4++w1BekKZwaVRsA//EZR8Ja71X37ubfw4NORbeMdBXKegC AywXjI9TskXuGxCbmu4MwyTWyc1jIL9LzZy/U2H8/h0xqT8TAMls7iox5vn5ReOHhfQC 8/iXMXMDtU2MxXtLH7BP4KxdHbaTflTtX4tZhDZWZlQ5AGd9eYSRE05aKkw2txwPCQ3v OYXYliVD5DqZzjlq1Op/IpDAs3wwPT5aKW4fV4qMWjSr1n4NMWtrp1iEblZiLD4G4nf7 jfB+TvKz1TD/ORJTtzXKfmGRNnARP2lDSnQs5F/KnTkPVWixYAqbHHvE5w240+p8zFiQ YF9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rnKMD0gFWbm5rjlkLlN2rxa4obGzTJq1MtbE2gwCtoY=; b=rc7BLT8YAc//cv9oQbl5YVvipVyyGlvnQtIghw4DyBHht/p7n8Ko3BD9R7PMHlxRBB NRw7p7zsq2I5lOAqlKLwqOrydjjcWFKm7HbaUwFPq43DGe0wcl5o78Y5mDtVv0Tbns5y TPdaJhGNo7MQl3g+uDZk9APCxaRxqwWBlEMUXPky/lFIZvCIfMfhkd3PZfBqdXPMeo2T eS3HI0GcFEa8MtAWioL5FTxq7oU+fV3bp3wMZMsvqg8sv49FFjI6cdWowI1e4iu0KMJv CgCtYxEeCWnEiz/vrhnKWNNTjKZRRi2ss1uHae0JQJG6G5x/q1lVhJ+DZOMs/tu3WpiO gaJQ== X-Gm-Message-State: AOAM530TNBRpXW/CFRbGRfPokm4BocZxW495rJzzaexDTePphrK+tt24 NkfNJyPcl4SiMq5g2cTbp4b2qUFDMvPzbg== X-Google-Smtp-Source: ABdhPJzc/zO0Bv26kjBcylJLNwNFXPLvgArcCW6hnWWZpTwRRw9sFLH4kghly1BVOJJ0mj8TqFmYyQ== X-Received: by 2002:a17:906:d7bc:: with SMTP id pk28mr8374263ejb.370.1613050184397; Thu, 11 Feb 2021 05:29:44 -0800 (PST) Received: from abel.fritz.box ([2a02:908:1252:fb60:820:5af6:9146:2c9f]) by smtp.gmail.com with ESMTPSA id w26sm3862627edq.46.2021.02.11.05.29.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Feb 2021 05:29:44 -0800 (PST) From: " =?utf-8?q?Christian_K=C3=B6nig?= " X-Google-Original-From: =?utf-8?q?Christian_K=C3=B6nig?= To: dri-devel@lists.freedesktop.org Subject: [PATCH 3/3] drm/ttm: switch to per device LRU lock Date: Thu, 11 Feb 2021 14:29:41 +0100 Message-Id: <20210211132941.290772-3-christian.koenig@amd.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210211132941.290772-1-christian.koenig@amd.com> References: <20210211132941.290772-1-christian.koenig@amd.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: ray.huang@amd.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Instead of having a global lock. Signed-off-by: Christian König --- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 8 ++--- drivers/gpu/drm/qxl/qxl_release.c | 5 +-- drivers/gpu/drm/ttm/ttm_bo.c | 49 ++++++++++++-------------- drivers/gpu/drm/ttm/ttm_device.c | 12 +++---- drivers/gpu/drm/ttm/ttm_execbuf_util.c | 8 ++--- drivers/gpu/drm/ttm/ttm_resource.c | 9 +++-- include/drm/ttm/ttm_bo_driver.h | 4 +-- include/drm/ttm/ttm_device.h | 4 +-- 8 files changed, 43 insertions(+), 56 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c index 9d19078246c8..ae18c0e32347 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c @@ -638,15 +638,15 @@ void amdgpu_vm_move_to_lru_tail(struct amdgpu_device *adev, struct amdgpu_vm_bo_base *bo_base; if (vm->bulk_moveable) { - spin_lock(&ttm_glob.lru_lock); + spin_lock(&adev->mman.bdev.lru_lock); ttm_bo_bulk_move_lru_tail(&vm->lru_bulk_move); - spin_unlock(&ttm_glob.lru_lock); + spin_unlock(&adev->mman.bdev.lru_lock); return; } memset(&vm->lru_bulk_move, 0, sizeof(vm->lru_bulk_move)); - spin_lock(&ttm_glob.lru_lock); + spin_lock(&adev->mman.bdev.lru_lock); list_for_each_entry(bo_base, &vm->idle, vm_status) { struct amdgpu_bo *bo = bo_base->bo; @@ -660,7 +660,7 @@ void amdgpu_vm_move_to_lru_tail(struct amdgpu_device *adev, &bo->shadow->tbo.mem, &vm->lru_bulk_move); } - spin_unlock(&ttm_glob.lru_lock); + spin_unlock(&adev->mman.bdev.lru_lock); vm->bulk_moveable = true; } diff --git a/drivers/gpu/drm/qxl/qxl_release.c b/drivers/gpu/drm/qxl/qxl_release.c index 6ed673d75f9f..29ad72cb6b54 100644 --- a/drivers/gpu/drm/qxl/qxl_release.c +++ b/drivers/gpu/drm/qxl/qxl_release.c @@ -417,16 +417,13 @@ void qxl_release_fence_buffer_objects(struct qxl_release *release) release->id | 0xf0000000, release->base.seqno); trace_dma_fence_emit(&release->base); - spin_lock(&ttm_glob.lru_lock); - list_for_each_entry(entry, &release->bos, head) { bo = entry->bo; dma_resv_add_shared_fence(bo->base.resv, &release->base); - ttm_bo_move_to_lru_tail(bo, &bo->mem, NULL); + ttm_bo_move_to_lru_tail_unlocked(bo); dma_resv_unlock(bo->base.resv); } - spin_unlock(&ttm_glob.lru_lock); ww_acquire_fini(&release->ticket); } diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c index a1be88be357b..a8103c8718a3 100644 --- a/drivers/gpu/drm/ttm/ttm_bo.c +++ b/drivers/gpu/drm/ttm/ttm_bo.c @@ -242,9 +242,9 @@ static int ttm_bo_individualize_resv(struct ttm_buffer_object *bo) * reference it any more. The only tricky case is the trylock on * the resv object while holding the lru_lock. */ - spin_lock(&ttm_glob.lru_lock); + spin_lock(&bo->bdev->lru_lock); bo->base.resv = &bo->base._resv; - spin_unlock(&ttm_glob.lru_lock); + spin_unlock(&bo->bdev->lru_lock); } return r; @@ -303,7 +303,7 @@ static int ttm_bo_cleanup_refs(struct ttm_buffer_object *bo, if (unlock_resv) dma_resv_unlock(bo->base.resv); - spin_unlock(&ttm_glob.lru_lock); + spin_unlock(&bo->bdev->lru_lock); lret = dma_resv_wait_timeout_rcu(resv, true, interruptible, 30 * HZ); @@ -313,7 +313,7 @@ static int ttm_bo_cleanup_refs(struct ttm_buffer_object *bo, else if (lret == 0) return -EBUSY; - spin_lock(&ttm_glob.lru_lock); + spin_lock(&bo->bdev->lru_lock); if (unlock_resv && !dma_resv_trylock(bo->base.resv)) { /* * We raced, and lost, someone else holds the reservation now, @@ -323,7 +323,7 @@ static int ttm_bo_cleanup_refs(struct ttm_buffer_object *bo, * delayed destruction would succeed, so just return success * here. */ - spin_unlock(&ttm_glob.lru_lock); + spin_unlock(&bo->bdev->lru_lock); return 0; } ret = 0; @@ -332,13 +332,13 @@ static int ttm_bo_cleanup_refs(struct ttm_buffer_object *bo, if (ret || unlikely(list_empty(&bo->ddestroy))) { if (unlock_resv) dma_resv_unlock(bo->base.resv); - spin_unlock(&ttm_glob.lru_lock); + spin_unlock(&bo->bdev->lru_lock); return ret; } ttm_bo_del_from_lru(bo); list_del_init(&bo->ddestroy); - spin_unlock(&ttm_glob.lru_lock); + spin_unlock(&bo->bdev->lru_lock); ttm_bo_cleanup_memtype_use(bo); if (unlock_resv) @@ -355,13 +355,12 @@ static int ttm_bo_cleanup_refs(struct ttm_buffer_object *bo, */ bool ttm_bo_delayed_delete(struct ttm_device *bdev, bool remove_all) { - struct ttm_global *glob = &ttm_glob; struct list_head removed; bool empty; INIT_LIST_HEAD(&removed); - spin_lock(&glob->lru_lock); + spin_lock(&bdev->lru_lock); while (!list_empty(&bdev->ddestroy)) { struct ttm_buffer_object *bo; @@ -372,24 +371,24 @@ bool ttm_bo_delayed_delete(struct ttm_device *bdev, bool remove_all) continue; if (remove_all || bo->base.resv != &bo->base._resv) { - spin_unlock(&glob->lru_lock); + spin_unlock(&bdev->lru_lock); dma_resv_lock(bo->base.resv, NULL); - spin_lock(&glob->lru_lock); + spin_lock(&bdev->lru_lock); ttm_bo_cleanup_refs(bo, false, !remove_all, true); } else if (dma_resv_trylock(bo->base.resv)) { ttm_bo_cleanup_refs(bo, false, !remove_all, true); } else { - spin_unlock(&glob->lru_lock); + spin_unlock(&bdev->lru_lock); } ttm_bo_put(bo); - spin_lock(&glob->lru_lock); + spin_lock(&bdev->lru_lock); } list_splice_tail(&removed, &bdev->ddestroy); empty = list_empty(&bdev->ddestroy); - spin_unlock(&glob->lru_lock); + spin_unlock(&bdev->lru_lock); return empty; } @@ -424,7 +423,7 @@ static void ttm_bo_release(struct kref *kref) ttm_bo_flush_all_fences(bo); bo->deleted = true; - spin_lock(&ttm_glob.lru_lock); + spin_lock(&bo->bdev->lru_lock); /* * Make pinned bos immediately available to @@ -438,17 +437,17 @@ static void ttm_bo_release(struct kref *kref) kref_init(&bo->kref); list_add_tail(&bo->ddestroy, &bdev->ddestroy); - spin_unlock(&ttm_glob.lru_lock); + spin_unlock(&bo->bdev->lru_lock); schedule_delayed_work(&bdev->wq, ((HZ / 100) < 1) ? 1 : HZ / 100); return; } - spin_lock(&ttm_glob.lru_lock); + spin_lock(&bo->bdev->lru_lock); ttm_bo_del_from_lru(bo); list_del(&bo->ddestroy); - spin_unlock(&ttm_glob.lru_lock); + spin_unlock(&bo->bdev->lru_lock); ttm_bo_cleanup_memtype_use(bo); dma_resv_unlock(bo->base.resv); @@ -622,7 +621,7 @@ int ttm_mem_evict_first(struct ttm_device *bdev, unsigned i; int ret; - spin_lock(&ttm_glob.lru_lock); + spin_lock(&bo->bdev->lru_lock); for (i = 0; i < TTM_MAX_BO_PRIORITY; ++i) { list_for_each_entry(bo, &man->lru[i], lru) { bool busy; @@ -659,7 +658,7 @@ int ttm_mem_evict_first(struct ttm_device *bdev, if (!bo) { if (busy_bo && !ttm_bo_get_unless_zero(busy_bo)) busy_bo = NULL; - spin_unlock(&ttm_glob.lru_lock); + spin_unlock(&bo->bdev->lru_lock); ret = ttm_mem_evict_wait_busy(busy_bo, ctx, ticket); if (busy_bo) ttm_bo_put(busy_bo); @@ -673,7 +672,7 @@ int ttm_mem_evict_first(struct ttm_device *bdev, return ret; } - spin_unlock(&ttm_glob.lru_lock); + spin_unlock(&bo->bdev->lru_lock); ret = ttm_bo_evict(bo, ctx); if (locked) @@ -773,10 +772,9 @@ static int ttm_bo_mem_placement(struct ttm_buffer_object *bo, mem->mem_type = place->mem_type; mem->placement = place->flags; - spin_lock(&ttm_glob.lru_lock); + spin_lock(&bo->bdev->lru_lock); ttm_bo_move_to_lru_tail(bo, mem, NULL); - spin_unlock(&ttm_glob.lru_lock); - + spin_unlock(&bo->bdev->lru_lock); return 0; } @@ -1160,7 +1158,6 @@ EXPORT_SYMBOL(ttm_bo_wait); int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx, gfp_t gfp_flags) { - struct ttm_global *glob = &ttm_glob; bool locked; int ret; @@ -1185,7 +1182,7 @@ int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx, ttm_bo_del_from_lru(bo); /* TODO: Cleanup the locking */ - spin_unlock(&glob->lru_lock); + spin_unlock(&bo->bdev->lru_lock); /* * Move to system cached diff --git a/drivers/gpu/drm/ttm/ttm_device.c b/drivers/gpu/drm/ttm/ttm_device.c index 2c280fb1e992..924d892109e8 100644 --- a/drivers/gpu/drm/ttm/ttm_device.c +++ b/drivers/gpu/drm/ttm/ttm_device.c @@ -81,7 +81,6 @@ static int ttm_global_init(void) ttm_pool_mgr_init(num_pages * 50 / 100); ttm_tt_mgr_init(); - spin_lock_init(&glob->lru_lock); glob->dummy_read_page = alloc_page(__GFP_ZERO | GFP_DMA32); if (unlikely(glob->dummy_read_page == NULL)) { @@ -125,13 +124,12 @@ EXPORT_SYMBOL(ttm_global_swapout); long ttm_device_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx, gfp_t gfp_flags) { - struct ttm_global *glob = &ttm_glob; struct ttm_resource_manager *man; struct ttm_buffer_object *bo; unsigned i, j; int ret; - spin_lock(&glob->lru_lock); + spin_lock(&bdev->lru_lock); for (i = TTM_PL_SYSTEM; i < TTM_NUM_MEM_TYPES; ++i) { man = ttm_manager_type(bdev, i); if (!man || !man->use_tt) @@ -156,7 +154,7 @@ long ttm_device_swapout(struct ttm_device *bdev, struct ttm_operation_ctx *ctx, } } } - spin_unlock(&glob->lru_lock); + spin_unlock(&bdev->lru_lock); return 0; } EXPORT_SYMBOL(ttm_device_swapout); @@ -223,6 +221,7 @@ int ttm_device_init(struct ttm_device *bdev, struct ttm_device_funcs *funcs, bdev->vma_manager = vma_manager; INIT_DELAYED_WORK(&bdev->wq, ttm_device_delayed_workqueue); + spin_lock_init(&bdev->lru_lock); INIT_LIST_HEAD(&bdev->ddestroy); bdev->dev_mapping = mapping; mutex_lock(&ttm_global_mutex); @@ -235,7 +234,6 @@ EXPORT_SYMBOL(ttm_device_init); void ttm_device_fini(struct ttm_device *bdev) { - struct ttm_global *glob = &ttm_glob; struct ttm_resource_manager *man; unsigned i; @@ -252,11 +250,11 @@ void ttm_device_fini(struct ttm_device *bdev) if (ttm_bo_delayed_delete(bdev, true)) pr_debug("Delayed destroy list was clean\n"); - spin_lock(&glob->lru_lock); + spin_lock(&bdev->lru_lock); for (i = 0; i < TTM_MAX_BO_PRIORITY; ++i) if (list_empty(&man->lru[0])) pr_debug("Swap list %d was clean\n", i); - spin_unlock(&glob->lru_lock); + spin_unlock(&bdev->lru_lock); ttm_pool_fini(&bdev->pool); ttm_global_release(); diff --git a/drivers/gpu/drm/ttm/ttm_execbuf_util.c b/drivers/gpu/drm/ttm/ttm_execbuf_util.c index 690ab97d52b7..071c48d672c6 100644 --- a/drivers/gpu/drm/ttm/ttm_execbuf_util.c +++ b/drivers/gpu/drm/ttm/ttm_execbuf_util.c @@ -51,14 +51,12 @@ void ttm_eu_backoff_reservation(struct ww_acquire_ctx *ticket, if (list_empty(list)) return; - spin_lock(&ttm_glob.lru_lock); list_for_each_entry(entry, list, head) { struct ttm_buffer_object *bo = entry->bo; - ttm_bo_move_to_lru_tail(bo, &bo->mem, NULL); + ttm_bo_move_to_lru_tail_unlocked(bo); dma_resv_unlock(bo->base.resv); } - spin_unlock(&ttm_glob.lru_lock); if (ticket) ww_acquire_fini(ticket); @@ -154,7 +152,6 @@ void ttm_eu_fence_buffer_objects(struct ww_acquire_ctx *ticket, if (list_empty(list)) return; - spin_lock(&ttm_glob.lru_lock); list_for_each_entry(entry, list, head) { struct ttm_buffer_object *bo = entry->bo; @@ -162,10 +159,9 @@ void ttm_eu_fence_buffer_objects(struct ww_acquire_ctx *ticket, dma_resv_add_shared_fence(bo->base.resv, fence); else dma_resv_add_excl_fence(bo->base.resv, fence); - ttm_bo_move_to_lru_tail(bo, &bo->mem, NULL); + ttm_bo_move_to_lru_tail_unlocked(bo); dma_resv_unlock(bo->base.resv); } - spin_unlock(&ttm_glob.lru_lock); if (ticket) ww_acquire_fini(ticket); } diff --git a/drivers/gpu/drm/ttm/ttm_resource.c b/drivers/gpu/drm/ttm/ttm_resource.c index ed1672a9f332..04f2eef653ab 100644 --- a/drivers/gpu/drm/ttm/ttm_resource.c +++ b/drivers/gpu/drm/ttm/ttm_resource.c @@ -91,7 +91,6 @@ int ttm_resource_manager_evict_all(struct ttm_device *bdev, .no_wait_gpu = false, .force_alloc = true }; - struct ttm_global *glob = &ttm_glob; struct dma_fence *fence; int ret; unsigned i; @@ -100,18 +99,18 @@ int ttm_resource_manager_evict_all(struct ttm_device *bdev, * Can't use standard list traversal since we're unlocking. */ - spin_lock(&glob->lru_lock); + spin_lock(&bdev->lru_lock); for (i = 0; i < TTM_MAX_BO_PRIORITY; ++i) { while (!list_empty(&man->lru[i])) { - spin_unlock(&glob->lru_lock); + spin_unlock(&bdev->lru_lock); ret = ttm_mem_evict_first(bdev, man, NULL, &ctx, NULL); if (ret) return ret; - spin_lock(&glob->lru_lock); + spin_lock(&bdev->lru_lock); } } - spin_unlock(&glob->lru_lock); + spin_unlock(&bdev->lru_lock); spin_lock(&man->move_lock); fence = dma_fence_get(man->move); diff --git a/include/drm/ttm/ttm_bo_driver.h b/include/drm/ttm/ttm_bo_driver.h index d007feef7676..dbccac957f8f 100644 --- a/include/drm/ttm/ttm_bo_driver.h +++ b/include/drm/ttm/ttm_bo_driver.h @@ -180,9 +180,9 @@ static inline int ttm_bo_reserve_slowpath(struct ttm_buffer_object *bo, static inline void ttm_bo_move_to_lru_tail_unlocked(struct ttm_buffer_object *bo) { - spin_lock(&ttm_glob.lru_lock); + spin_lock(&bo->bdev->lru_lock); ttm_bo_move_to_lru_tail(bo, &bo->mem, NULL); - spin_unlock(&ttm_glob.lru_lock); + spin_unlock(&bo->bdev->lru_lock); } static inline void ttm_bo_assign_mem(struct ttm_buffer_object *bo, diff --git a/include/drm/ttm/ttm_device.h b/include/drm/ttm/ttm_device.h index cda6efb4c34b..bae56d29e8ff 100644 --- a/include/drm/ttm/ttm_device.h +++ b/include/drm/ttm/ttm_device.h @@ -56,7 +56,6 @@ extern struct ttm_global { */ struct page *dummy_read_page; - spinlock_t lru_lock; /** * Protected by ttm_global_mutex. @@ -277,8 +276,9 @@ struct ttm_device { struct ttm_pool pool; /* - * Protected by the global:lru lock. + * Protection for the per manager LRU and ddestroy lists. */ + spinlock_t lru_lock; struct list_head ddestroy; /*