From patchwork Thu Apr 15 11:56:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Christian_K=C3=B6nig?= X-Patchwork-Id: 12205231 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D83C5C43460 for ; Thu, 15 Apr 2021 11:56:28 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 819DC610FA for ; Thu, 15 Apr 2021 11:56:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 819DC610FA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id B6F3B6EA47; Thu, 15 Apr 2021 11:56:27 +0000 (UTC) Received: from mail-ej1-x630.google.com (mail-ej1-x630.google.com [IPv6:2a00:1450:4864:20::630]) by gabe.freedesktop.org (Postfix) with ESMTPS id D8DC76EA46 for ; Thu, 15 Apr 2021 11:56:26 +0000 (UTC) Received: by mail-ej1-x630.google.com with SMTP id x12so15663519ejc.1 for ; Thu, 15 Apr 2021 04:56:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=MAgKFkUbajfTwkiBbjHCwLbY8LqpQ8PBmh7VA/cwU0E=; b=m1wDMNbKQex77GuTlU5+skb5l8cZdzF7kNI3HPNE8cQmGFBDeoAAfO3z62VHp2W3S6 g8hW0/7uNFuyEGt2cBpWlxY7pcdErauqg3dyEdRwrM0YzVCRm1joAJpAAV5NNdVsHQoF hahSKm6dCyKXf1Qc/eKCUHyZ69K7r1luODPcgcG5j0yP0ohshKObaq7m0ImYa5GWbCry 9Al1nVGzFju2O2/uydbJISxb5xBQhjAZqCnMhAz9VcnbR6pq5rT9YQjA3ZD+kQvHIDEE 6sfiAKFL4nt2J7KcscGOnWoI8wLfSRKUNDg4gXyWvv6+zlHwL+czM74h6/wzk/VJYSXm F+NA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=MAgKFkUbajfTwkiBbjHCwLbY8LqpQ8PBmh7VA/cwU0E=; b=DGg4H03XXH5psyoGlJisVrqjZ8GrdnsmKVVEf091SSc1JpbXXCnAA61QLhK5L1wN96 VIABd/Kk63teahyf2F9oWKl185cbKJoO/GUiBAFDhRh15PzHcMCELiAJ0DSKeu13ijyw Wmo4A2unXdoxrYVw+GzaGJnqNbuRe4jPnk8E9ergx2Hy32dCqHFL9rg2fy+roUSVr3jA +t6fLamvxzvTzrd2NmgXq80OkS3aeSjTPf6MpkMGJ+Ty+H4HFIiQA6CIsSZVaKEEwyhU 8v6RvqHLT3MzIrNjICEcU6h/ZfiDi773fpZsWxAlc/cE2yv5ny46hrgxIgBTm4ITV1kn zbCg== X-Gm-Message-State: AOAM533jPVRAJ+/ug2EQF9Y3SemxLWRsFczawN/NM3Pt5JUtPnx8994i kVepOLRzxjGcvCVHlUGNrRc= X-Google-Smtp-Source: ABdhPJwgvhGuyBEvsRLBj8pJVlv1MD8sM4alBQQIq3AaHw/fvXpcPyQjXZnukRoS19wBBYBX8H5fjg== X-Received: by 2002:a17:906:524f:: with SMTP id y15mr2973184ejm.65.1618487785620; Thu, 15 Apr 2021 04:56:25 -0700 (PDT) Received: from abel.fritz.box ([2a02:908:1252:fb60:3ba1:ccd9:cd93:a8ed]) by smtp.gmail.com with ESMTPSA id s9sm2182461edd.16.2021.04.15.04.56.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Apr 2021 04:56:25 -0700 (PDT) From: " =?utf-8?q?Christian_K=C3=B6nig?= " X-Google-Original-From: =?utf-8?q?Christian_K=C3=B6nig?= To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, dri-devel@lists.freedesktop.org Subject: [PATCH 1/2] mm/vmscan: add sync_shrinkers function Date: Thu, 15 Apr 2021 13:56:23 +0200 Message-Id: <20210415115624.2904-1-christian.koenig@amd.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: akpm@linux-foundation.org, ray.huang@amd.com, vbabka@suse.cz Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" To be able to switch to a spinlock and reduce lock contention in the TTM shrinker we don't want to hold a mutex while unmapping and freeing pages from the pool. But then we somehow need to prevent a race between (for example) the shrinker trying to free pages and hotplug trying to remove the device which those pages belong to. Taking and releasing the shrinker semaphore on the write side after unmapping and freeing all pages should make sure that no shrinker is running in paralell any more. Signed-off-by: Christian König Acked-by: Daniel Vetter --- include/linux/shrinker.h | 1 + mm/vmscan.c | 10 ++++++++++ 2 files changed, 11 insertions(+) diff --git a/include/linux/shrinker.h b/include/linux/shrinker.h index 0f80123650e2..6b75dc372fce 100644 --- a/include/linux/shrinker.h +++ b/include/linux/shrinker.h @@ -92,4 +92,5 @@ extern void register_shrinker_prepared(struct shrinker *shrinker); extern int register_shrinker(struct shrinker *shrinker); extern void unregister_shrinker(struct shrinker *shrinker); extern void free_prealloced_shrinker(struct shrinker *shrinker); +extern void sync_shrinkers(void); #endif diff --git a/mm/vmscan.c b/mm/vmscan.c index 562e87cbd7a1..46cd9c215d73 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -408,6 +408,16 @@ void unregister_shrinker(struct shrinker *shrinker) } EXPORT_SYMBOL(unregister_shrinker); +/** + * sync_shrinker - Wait for all running shrinkers to complete. + */ +void sync_shrinkers(void) +{ + down_write(&shrinker_rwsem); + up_write(&shrinker_rwsem); +} +EXPORT_SYMBOL(sync_shrinkers); + #define SHRINK_BATCH 128 static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, From patchwork Thu Apr 15 11:56:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Christian_K=C3=B6nig?= X-Patchwork-Id: 12205233 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.7 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, UNWANTED_LANGUAGE_BODY,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 978FEC433ED for ; Thu, 15 Apr 2021 11:56:33 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3F995611CD for ; Thu, 15 Apr 2021 11:56:33 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3F995611CD Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id DD35F6EA46; Thu, 15 Apr 2021 11:56:30 +0000 (UTC) Received: from mail-ej1-x632.google.com (mail-ej1-x632.google.com [IPv6:2a00:1450:4864:20::632]) by gabe.freedesktop.org (Postfix) with ESMTPS id 9AD446EA46 for ; Thu, 15 Apr 2021 11:56:27 +0000 (UTC) Received: by mail-ej1-x632.google.com with SMTP id r9so36433516ejj.3 for ; Thu, 15 Apr 2021 04:56:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Mmzr1BXITN8/WsQtuIlDuVTV5ipAk7FG2VM5j+Bi1So=; b=QvQMXegx4Q6FfPIpt3BYD7d/rIzGJpkhRMRMpxNI2Jc9ovzo2ea9Po/pjK9xcot10J qfrgGeGvDxK0UtoQelPc859qvv3lCo3V+I61eOMYlY8VoruRr/1VzBriz8TYVT38gXGi bKsI+0K3iURKjYw5ibQslYc04LzCB2HkYJNOWtQdeJtmREQywEIO3U6qpcFFSzp1UUSr dtudZ7kiiIBbHYxn6lznMpdvme+RuMnAyO8YBP/JdG6KqY3gxKnonMJ2xQr0SQ3k8UEY 6eAqAWXS/841WW5UHmuMlxOhssYyTS1s5DnIKzXWT6WFbvJt2mBq4t3ivxQ5Rtr6d49m ObjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Mmzr1BXITN8/WsQtuIlDuVTV5ipAk7FG2VM5j+Bi1So=; b=g4pviWQZNvYjSBrwt/UXI8eQi+MdYmXBQPEDvCx5zUMxQW12wB/tXQWdtYbANb5DXw 58SpHJKrVbZRTTg/mekW4L3ySxBeFwi4Zfyi9X/WD9A8DCCyHr9KpHEUrQxNCziwcdxx TADn3tQxWxEX93lEUTdnHlT3HHXWJRa7GqYH8I3e1/2eTCzuSBzUGnQn/o9qUUiahLYx N6gsB6XCCauKC14iAmo+RaUT6v80H3krA8QBJXnwJBuxUospVJ4Au9NJXbGWo4HWHMa8 ca1NrynG61GJnuqyDjvSXrh8NxUACWyF6hbKEc4HuUhazuejRoJNLuXXwEJz/HuMg5vh 8BBQ== X-Gm-Message-State: AOAM532IFVHfG5vT7hUjhm04NQR2Sf2wEBicrNIhoLVVIhPHW5fPFfq3 TsuufJE+Cp4al69m9Iqsxr8= X-Google-Smtp-Source: ABdhPJyRvFr8GCcyp6cP4pUhEHuqjgAxms+Re3dD8oeMsGnJsnFn3mLckPuF7TSkfuSpiaA2RZVQPA== X-Received: by 2002:a17:906:278e:: with SMTP id j14mr3012533ejc.224.1618487786343; Thu, 15 Apr 2021 04:56:26 -0700 (PDT) Received: from abel.fritz.box ([2a02:908:1252:fb60:3ba1:ccd9:cd93:a8ed]) by smtp.gmail.com with ESMTPSA id s9sm2182461edd.16.2021.04.15.04.56.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Apr 2021 04:56:26 -0700 (PDT) From: " =?utf-8?q?Christian_K=C3=B6nig?= " X-Google-Original-From: =?utf-8?q?Christian_K=C3=B6nig?= To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, dri-devel@lists.freedesktop.org Subject: [PATCH 2/2] drm/ttm: optimize the pool shrinker a bit v2 Date: Thu, 15 Apr 2021 13:56:24 +0200 Message-Id: <20210415115624.2904-2-christian.koenig@amd.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210415115624.2904-1-christian.koenig@amd.com> References: <20210415115624.2904-1-christian.koenig@amd.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: akpm@linux-foundation.org, ray.huang@amd.com, vbabka@suse.cz Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Switch back to using a spinlock again by moving the IOMMU unmap outside of the locked region. v2: Add a comment explaining why we need sync_shrinkers(). Signed-off-by: Christian König Acked-by: Huang Rui --- drivers/gpu/drm/ttm/ttm_pool.c | 44 +++++++++++++++++----------------- 1 file changed, 22 insertions(+), 22 deletions(-) diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c index cb38b1a17b09..955836d569cc 100644 --- a/drivers/gpu/drm/ttm/ttm_pool.c +++ b/drivers/gpu/drm/ttm/ttm_pool.c @@ -70,7 +70,7 @@ static struct ttm_pool_type global_uncached[MAX_ORDER]; static struct ttm_pool_type global_dma32_write_combined[MAX_ORDER]; static struct ttm_pool_type global_dma32_uncached[MAX_ORDER]; -static struct mutex shrinker_lock; +static spinlock_t shrinker_lock; static struct list_head shrinker_list; static struct shrinker mm_shrinker; @@ -263,9 +263,9 @@ static void ttm_pool_type_init(struct ttm_pool_type *pt, struct ttm_pool *pool, spin_lock_init(&pt->lock); INIT_LIST_HEAD(&pt->pages); - mutex_lock(&shrinker_lock); + spin_lock(&shrinker_lock); list_add_tail(&pt->shrinker_list, &shrinker_list); - mutex_unlock(&shrinker_lock); + spin_unlock(&shrinker_lock); } /* Remove a pool_type from the global shrinker list and free all pages */ @@ -273,9 +273,9 @@ static void ttm_pool_type_fini(struct ttm_pool_type *pt) { struct page *p; - mutex_lock(&shrinker_lock); + spin_lock(&shrinker_lock); list_del(&pt->shrinker_list); - mutex_unlock(&shrinker_lock); + spin_unlock(&shrinker_lock); while ((p = ttm_pool_type_take(pt))) ttm_pool_free_page(pt->pool, pt->caching, pt->order, p); @@ -313,24 +313,19 @@ static struct ttm_pool_type *ttm_pool_select_type(struct ttm_pool *pool, static unsigned int ttm_pool_shrink(void) { struct ttm_pool_type *pt; - unsigned int num_freed; struct page *p; - mutex_lock(&shrinker_lock); + spin_lock(&shrinker_lock); pt = list_first_entry(&shrinker_list, typeof(*pt), shrinker_list); + list_move_tail(&pt->shrinker_list, &shrinker_list); + spin_unlock(&shrinker_lock); p = ttm_pool_type_take(pt); - if (p) { - ttm_pool_free_page(pt->pool, pt->caching, pt->order, p); - num_freed = 1 << pt->order; - } else { - num_freed = 0; - } - - list_move_tail(&pt->shrinker_list, &shrinker_list); - mutex_unlock(&shrinker_lock); + if (!p) + return 0; - return num_freed; + ttm_pool_free_page(pt->pool, pt->caching, pt->order, p); + return 1 << pt->order; } /* Return the allocation order based for a page */ @@ -530,6 +525,11 @@ void ttm_pool_fini(struct ttm_pool *pool) for (j = 0; j < MAX_ORDER; ++j) ttm_pool_type_fini(&pool->caching[i].orders[j]); } + + /* We removed the pool types from the LRU, but we need to also make sure + * that no shrinker is concurrently freeing pages from the pool. + */ + sync_shrinkers(); } /* As long as pages are available make sure to release at least one */ @@ -604,7 +604,7 @@ static int ttm_pool_debugfs_globals_show(struct seq_file *m, void *data) { ttm_pool_debugfs_header(m); - mutex_lock(&shrinker_lock); + spin_lock(&shrinker_lock); seq_puts(m, "wc\t:"); ttm_pool_debugfs_orders(global_write_combined, m); seq_puts(m, "uc\t:"); @@ -613,7 +613,7 @@ static int ttm_pool_debugfs_globals_show(struct seq_file *m, void *data) ttm_pool_debugfs_orders(global_dma32_write_combined, m); seq_puts(m, "uc 32\t:"); ttm_pool_debugfs_orders(global_dma32_uncached, m); - mutex_unlock(&shrinker_lock); + spin_unlock(&shrinker_lock); ttm_pool_debugfs_footer(m); @@ -640,7 +640,7 @@ int ttm_pool_debugfs(struct ttm_pool *pool, struct seq_file *m) ttm_pool_debugfs_header(m); - mutex_lock(&shrinker_lock); + spin_lock(&shrinker_lock); for (i = 0; i < TTM_NUM_CACHING_TYPES; ++i) { seq_puts(m, "DMA "); switch (i) { @@ -656,7 +656,7 @@ int ttm_pool_debugfs(struct ttm_pool *pool, struct seq_file *m) } ttm_pool_debugfs_orders(pool->caching[i].orders, m); } - mutex_unlock(&shrinker_lock); + spin_unlock(&shrinker_lock); ttm_pool_debugfs_footer(m); return 0; @@ -693,7 +693,7 @@ int ttm_pool_mgr_init(unsigned long num_pages) if (!page_pool_size) page_pool_size = num_pages; - mutex_init(&shrinker_lock); + spin_lock_init(&shrinker_lock); INIT_LIST_HEAD(&shrinker_list); for (i = 0; i < MAX_ORDER; ++i) {