From patchwork Thu May 25 12:57:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Gleixner X-Patchwork-Id: 13255210 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ECE6DC77B7E for ; Thu, 25 May 2023 12:57:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4F64B280004; Thu, 25 May 2023 08:57:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4805A280002; Thu, 25 May 2023 08:57:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2FA03280004; Thu, 25 May 2023 08:57:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id EA9B6280002 for ; Thu, 25 May 2023 08:57:12 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id CB7F4120BA2 for ; Thu, 25 May 2023 12:57:12 +0000 (UTC) X-FDA: 80828777904.09.A14A6D3 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by imf19.hostedemail.com (Postfix) with ESMTP id 11C091A0009 for ; Thu, 25 May 2023 12:57:10 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=linutronix.de header.s=2020 header.b=q80b8+pU; dkim=pass header.d=linutronix.de header.s=2020e header.b=1uyjWFVC; dmarc=pass (policy=none) header.from=linutronix.de; spf=pass (imf19.hostedemail.com: domain of tglx@linutronix.de designates 193.142.43.55 as permitted sender) smtp.mailfrom=tglx@linutronix.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685019431; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=KNvR8V42JNc63MdTcShzQO5ErNm2SoBdzVTbIT2n5ek=; b=yuw2P6nvMcGQbj3RsuRp0Ui3hor4gvCH9I6NEzyTMdoBWIGTKzT7HPQZcqozX9mJwFkW5p ft5fzXLylWscRnxv3LxOj8CISHC+kUlQkJ4ekdI4axtSj5B1I/xAy8OEHChjj3wySqJiOe FpwItbP0WIYT3X1Pg8apklukDNIo/fA= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=linutronix.de header.s=2020 header.b=q80b8+pU; dkim=pass header.d=linutronix.de header.s=2020e header.b=1uyjWFVC; dmarc=pass (policy=none) header.from=linutronix.de; spf=pass (imf19.hostedemail.com: domain of tglx@linutronix.de designates 193.142.43.55 as permitted sender) smtp.mailfrom=tglx@linutronix.de ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685019431; a=rsa-sha256; cv=none; b=R9vxjrIrujGC4bSKMYEyIi8grt1OD4Q3SkTlohxvHO1JgkkLRjdrSw/Lkktm/ubMlZ0266 lvsf3AijKYFV7Jnz/11kr3gg3Z0yAtSqxvi6wbWRV9RZKWVfkQXGJP49AecwVaNrzKfUEn 7686WhiSQgv38dESXSKvKL9kWB0bMeQ= Message-ID: <20230525124504.864005691@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1685019429; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=KNvR8V42JNc63MdTcShzQO5ErNm2SoBdzVTbIT2n5ek=; b=q80b8+pUP/lVLX5WbNBjsSWhVDla7d4ZylW48/CGVRqJ2YSOScPr156S3SLXusnfZXZwZa biQCgD+7yvXsxxfzbQhovIzDaFwbbdIulZM8hk4BRP8Dk5zTIhSyC7hG33VQUSo7IsuUh9 X8VSM0oypRwR5HtumEWL2XfmXEJrxFCIh1ilvI9yhflABWOzGe+x4fZTLdUGqO+kPOr6I+ 5/1zkUvDO6zHkxZpC0vXyCLCfmwnczHP3qVVR4HrRXeCtXIltamtpbahGSiKzDBfx5eQow gO3fwK0hULCU6AxhzUt/Q91SqnzmpW7Jbf5HP7MsXQE3O27YCg18zChcw0tH1A== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1685019429; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=KNvR8V42JNc63MdTcShzQO5ErNm2SoBdzVTbIT2n5ek=; b=1uyjWFVCho+vjKL7w8AXzALC6IyAq9+I3kGxo7dZJz5U8V6nihh0IIzRwKQLOZDmAp/kPn CZcydAuQJPKoFkAA== From: Thomas Gleixner To: linux-mm@kvack.org Cc: Andrew Morton , Christoph Hellwig , Uladzislau Rezki , Lorenzo Stoakes , Peter Zijlstra , Baoquan He Subject: [V2 patch 6/6] mm/vmalloc: Dont purge usable blocks unnecessarily References: <20230525122342.109672430@linutronix.de> MIME-Version: 1.0 Date: Thu, 25 May 2023 14:57:09 +0200 (CEST) X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 11C091A0009 X-Stat-Signature: 8zzxd4pnra9f7nruqz56wg7z4ctk5bur X-Rspam-User: X-HE-Tag: 1685019430-578757 X-HE-Meta: U2FsdGVkX198aoC8WogbGskI2hq/UzpfrSkY8Of8C+ML3Z8Lu8nyHVaUzRWx7JAg3U7VuA06RpGE3SNOLqreEJFz7PsaCiGagSZC1VTd+7mS+UKLb8syt/jwf4+EXTBytPlmnwoKxCz7KGVfEo0/HPNQIqcDg1J+MjHS/1Nc6sVrpekqMXZpD+aQotQyM6bak36yX5t0Nnp+IWcj+IFvB3RO72tBUZTYIwbF5bV8zuLuEpTOSMrIeDfF3xyz0wA+34lCmqCfMHSl+8EtzPLSVBcSkvLXyc5JiFHltxj2YcsjPPkT2s8YST6R6YtaPan3iVUktZ4SusCVCaoF+GqlvG0NoAXFhyze1Z55seYxg9Bm4YmWirV94+SL+8URvNlA+YhNqM1EpGIBBzXS/qBAllfW+hxAtsISrg5tokG9sxHNSgmwfgmP7z4C7PEoMXOSTIo8l0rCSpzV6rujjsh5DMjaCkwGbkhQOrioQf97Rn8ge6n7d25Kh/XzbyjoyLVESci/XJuNhXuLN2TrgK7//YrWWjK7Ec7Wdw/oJIgZiNaasCNLI+pX7y4ApDCIYGteiwEVH5PiRtM7EVpUWwP2pIcxZ64qoJC3im0XEIEsBi5vb+mQ+s544X+3PkcUQsGLR6CB2uOA2qsIEmsd5QrN79vN2RskSawr2DTS3frBx+t9aN1MVGYW9wGp/yrr3XEtoAklvnEoYzoV6tXwRnTglXFUIi8nfMhaSdw4E/PH65G9opGkGTl+vNUuDF+O897nLA9c2zB9Xw7WfLIplHsF7P77U4VA2Z0m0Hx97pjSFwc+pfdug7+d1d0PZ6oqvzQEJ9p42ZASSJLdevIl3OWJ7RaNPiDeplXK+9NA2j6AmywXhn/0CFe6ag76IsPZo6BR9KZjfD3Dm2NYebHmMM2ZYCPDMPWbXAexsRcFT8MrWWj8ZwfO6PyQalQOdW2u8vgN8vDHEvpzspBpWNaIeBF AjaA6bsD n0VMRsRXP7W8UjhZWiFDqf46ZkMnCpbJJ9BAZppbDK4ESmZrdZP2zcKOS791IvNUf66uNR8NmPXg3eFIjG15/7rKbnha3EcxyfaTy X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Purging fragmented blocks is done unconditionally in several contexts: 1) From drain_vmap_area_work(), when the number of lazy to be freed vmap_areas reached the threshold 2) Reclaiming vmalloc address space from pcpu_get_vm_areas() 3) _vm_unmap_aliases() #1 There is no reason to zap fragmented vmap blocks unconditionally, simply because reclaiming all lazy areas drains at least 32MB * fls(num_online_cpus()) per invocation which is plenty. #2 Reclaiming when running out of space or due to memory pressure makes a lot of sense #3 _unmap_aliases() requires to touch everything because the caller has no clue which vmap_area used a particular page last and the vmap_area lost that information too. Except for the vfree + VM_FLUSH_RESET_PERMS case, which removes the vmap area first and then cares about the flush. That in turn requires a full walk of _all_ vmap areas including the one which was just added to the purge list. But as this has to be flushed anyway this is an opportunity to combine outstanding TLB flushes and do the housekeeping of purging freed areas, but like #1 there is no real good reason to zap usable vmap blocks unconditionally. Add a @force_purge argument to the newly split out block purge function and if not true only purge fragmented blocks which have less than 1/4 of their capacity left. Rename purge_vmap_area_lazy() to reclaim_and_purge_vmap_areas() to make it clear what the function does. Signed-off-by: Thomas Gleixner --- V2: Add the missing force_purge argument in _vm_unmap_aliases() Remove force_purge argument from the reclaim path - Baoquan --- mm/vmalloc.c | 28 ++++++++++++++++++++-------- 1 file changed, 20 insertions(+), 8 deletions(-) --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -791,7 +791,7 @@ get_subtree_max_size(struct rb_node *nod RB_DECLARE_CALLBACKS_MAX(static, free_vmap_area_rb_augment_cb, struct vmap_area, rb_node, unsigned long, subtree_max_size, va_size) -static void purge_vmap_area_lazy(void); +static void reclaim_and_purge_vmap_areas(void); static BLOCKING_NOTIFIER_HEAD(vmap_notify_list); static void drain_vmap_area_work(struct work_struct *work); static DECLARE_WORK(drain_vmap_work, drain_vmap_area_work); @@ -1649,7 +1649,7 @@ static struct vmap_area *alloc_vmap_area overflow: if (!purged) { - purge_vmap_area_lazy(); + reclaim_and_purge_vmap_areas(); purged = 1; goto retry; } @@ -1785,9 +1785,10 @@ static bool __purge_vmap_area_lazy(unsig } /* - * Kick off a purge of the outstanding lazy areas. + * Reclaim vmap areas by purging fragmented blocks and purge_vmap_area_list. */ -static void purge_vmap_area_lazy(void) +static void reclaim_and_purge_vmap_areas(void) + { mutex_lock(&vmap_purge_lock); purge_fragmented_blocks_allcpus(); @@ -1908,6 +1909,12 @@ static struct vmap_area *find_unlink_vma #define VMAP_BLOCK_SIZE (VMAP_BBMAP_BITS * PAGE_SIZE) +/* + * Purge threshold to prevent overeager purging of fragmented blocks for + * regular operations: Purge if vb->free is less than 1/4 of the capacity. + */ +#define VMAP_PURGE_THRESHOLD (VMAP_BBMAP_BITS / 4) + #define VMAP_RAM 0x1 /* indicates vm_map_ram area*/ #define VMAP_BLOCK 0x2 /* mark out the vmap_block sub-type*/ #define VMAP_FLAGS_MASK 0x3 @@ -2087,12 +2094,17 @@ static void free_vmap_block(struct vmap_ } static bool purge_fragmented_block(struct vmap_block *vb, - struct vmap_block_queue *vbq, struct list_head *purge_list) + struct vmap_block_queue *vbq, struct list_head *purge_list, + bool force_purge) { if (vb->free + vb->dirty != VMAP_BBMAP_BITS || vb->dirty == VMAP_BBMAP_BITS) return false; + /* Don't overeagerly purge usable blocks unless requested */ + if (!force_purge && vb->free < VMAP_PURGE_THRESHOLD) + return false; + /* prevent further allocs after releasing lock */ WRITE_ONCE(vb->free, 0); /* prevent purging it again */ @@ -2132,7 +2144,7 @@ static void purge_fragmented_blocks(int continue; spin_lock(&vb->lock); - purge_fragmented_block(vb, vbq, &purge); + purge_fragmented_block(vb, vbq, &purge, true); spin_unlock(&vb->lock); } rcu_read_unlock(); @@ -2269,7 +2281,7 @@ static void _vm_unmap_aliases(unsigned l * not purgeable, check whether there is dirty * space to be flushed. */ - if (!purge_fragmented_block(vb, vbq, &purge_list) && + if (!purge_fragmented_block(vb, vbq, &purge_list, false) && vb->dirty_max && vb->dirty != VMAP_BBMAP_BITS) { unsigned long va_start = vb->va->va_start; unsigned long s, e; @@ -4175,7 +4187,7 @@ struct vm_struct **pcpu_get_vm_areas(con overflow: spin_unlock(&free_vmap_area_lock); if (!purged) { - purge_vmap_area_lazy(); + reclaim_and_purge_vmap_areas(); purged = true; /* Before "retry", check if we recover. */