From patchwork Wed Mar 6 08:54:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13583547 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 45524C54E41 for ; Wed, 6 Mar 2024 08:56:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 982096B0098; Wed, 6 Mar 2024 03:55:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8DD4A6B0099; Wed, 6 Mar 2024 03:55:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6BBA06B009A; Wed, 6 Mar 2024 03:55:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 52E226B0098 for ; Wed, 6 Mar 2024 03:55:41 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id E72F480BE8 for ; Wed, 6 Mar 2024 08:55:40 +0000 (UTC) X-FDA: 81866006040.20.D18063C Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by imf24.hostedemail.com (Postfix) with ESMTP id 1EEA0180004 for ; Wed, 6 Mar 2024 08:55:38 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf24.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709715339; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=LztNmpT8tyLUiVBKSXvyTcd03jVbOPHMQmNOTNRK3pk=; b=Wi/f4B/R16QfoP85nLSrF+o1Pqew7JDQNoQHIw4/JPuknM3eblrk6tQfnWnKyS8oJFiyiW coZf0XfnahZDcts6QESplHrN29M0+0jTx8xtn8hedYAxSRUZXhdKC8rhBaffumrydtEw9Q pyHRt+7GAxrLToCASAvA4TH8gP5kaao= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf24.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709715339; a=rsa-sha256; cv=none; b=tfFSdT9kgGGxVTZOgW2Bk3+YAP+Yc6HWVTPSjoK4rp7ZO2WpLlGkilmYPGy7QsWEhKf4st RiepvHA+UAkCy7VEtGI7jsdEryMk35CpIDCVHCRprvcRI8tJWqR41gkC+s05bJkgLNpdyN CDY8jxFWYnlugla4KwoX98kpDEPtdkU= X-AuditID: a67dfc5b-d85ff70000001748-f8-65e82f7e0839 From: Byungchul Park To: linux-kernel@vger.kernel.org Cc: kernel_team@skhynix.com, torvalds@linux-foundation.org, damien.lemoal@opensource.wdc.com, linux-ide@vger.kernel.org, adilger.kernel@dilger.ca, linux-ext4@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, will@kernel.org, tglx@linutronix.de, rostedt@goodmis.org, joel@joelfernandes.org, sashal@kernel.org, daniel.vetter@ffwll.ch, duyuyang@gmail.com, johannes.berg@intel.com, tj@kernel.org, tytso@mit.edu, willy@infradead.org, david@fromorbit.com, amir73il@gmail.com, gregkh@linuxfoundation.org, kernel-team@lge.com, linux-mm@kvack.org, akpm@linux-foundation.org, mhocko@kernel.org, minchan@kernel.org, hannes@cmpxchg.org, vdavydov.dev@gmail.com, sj@kernel.org, jglisse@redhat.com, dennis@kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, vbabka@suse.cz, ngupta@vflare.org, linux-block@vger.kernel.org, josef@toxicpanda.com, linux-fsdevel@vger.kernel.org, jack@suse.cz, jlayton@kernel.org, dan.j.williams@intel.com, hch@infradead.org, djwong@kernel.org, dri-devel@lists.freedesktop.org, rodrigosiqueiramelo@gmail.com, melissa.srw@gmail.com, hamohammed.sa@gmail.com, 42.hyeyoo@gmail.com, chris.p.wilson@intel.com, gwan-gyeong.mun@intel.com, max.byungchul.park@gmail.com, boqun.feng@gmail.com, longman@redhat.com, hdanton@sina.com, her0gyugyu@gmail.com Subject: [PATCH v13 13/27] dept: Add a mechanism to refill the internal memory pools on running out Date: Wed, 6 Mar 2024 17:54:59 +0900 Message-Id: <20240306085513.41482-14-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240306085513.41482-1-byungchul@sk.com> References: <20240306085513.41482-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAAzXSfUzMcRwHcN/fczdnv53STwydYZiePOxjy+M8/Gy0jH8w08391E0PXC5C 7VLpQSlZRcW6ykmlh+v+EJUTUlo5VE5LS040V+nhbk7NuTL/fPba5+H914fBJXrSg1GEnxOU 4bJQKSUiRMNzi9bFen8XfL784uBGmg9YJ5MJKKiqoMBYWY6gQh+HwdDLvfDBZkEw1f4Gh9xs IwLN50846Jv7EDSUXqHgvXkedFpHKWjNvkZBfHEVBW9/TGPQm5OFQbnuALRlFmFgsH8jIHeI gvzceMxZvmNg15bRoFWvgIHSPBqmP/tCa183CQ09a+H23V4K6htaCWh+NIDB+8cFFPRVOEho a24hwHgjnYSHI0UU/LBpcdBaR2l4ZyjEoDrBGXR14g8Jr9INGFwtqcGg8+MTBI3J/RjoKrop eG61YFCry8bh9/2XCAauD9OQmGanIT/uOoJriTkEJPRuhKlfBdT2zfxzyyjOJ9Se5xtshQT/ uojj6/I+0XxCYw/NF+pUfG3pGr64fgjjNeNWkteVpVC8bjyL5lOHOzF+pKOD5ltuTRG8uTMX C/Q4KvKXC6GKKEHpvTVIFNLVlk+fMW678LXaRqhRnV8qcmE4dgM32GPG/7tP3Tpril3FmUz2 Wbuyy7ja9EFyxjhrEXElHXtmPJ89yY1eKZ/tE+wK7sFUP52KGEbMbuIK7vj/i1zKlVcbZmNc nO2MkQxqxhJ2I9cer3Fa5NyZZDhzkgP9O1jIPSs1EZlIXIjmlCGJIjwqTKYI3eAVEh2uuOB1 MiJMh5y/pI2ZPvYIjRsPNSGWQdK54u0u3wQJKYuKjA5rQhyDS13Fl3+bBYlYLou+KCgjTihV oUJkE1rEEFJ3sZ/tvFzCBsvOCacF4Yyg/D/FGBcPNdpWfy/ZXfXCNbNmccTPLzuTzh7ZHWBv vOwW1xidovYclA4mXarzk//xC2pJWVk8sU4asBPl68f2v0vz/mnqfhrsiF1tPX6wxBGjD3P1 7zDW3MwinmhYedCSwOUryX3Blosaz8TpLYagLtXRHf2m9Yczxra6TfTsOqaqXIu61KccC6RE ZIjMdw2ujJT9BTcn9k9HAwAA X-Brightmail-Tracker: H4sIAAAAAAAAAzWSa0hTcRyG+5+7o8VhGZ20MCYiKJmayi+LlKB2kIo+ZQWhQw860mmbl8wK c1rmNSu10mRempdZ2uxDN000L2toy1smKilmrpaStZFpFzX68vLwvvB8ehlcUkw6MQplgqBS ymOklIgQHdmj2XFx56zgfdOCoDDXG2zfswgoa2ygwPxAj6Dh0SUMLJ0yeGu3IljqfY1DSZEZ QcXkOA6PuiYQtNSmUzAwvQEGbfMUGItyKNBUNVLw5vMyBmPF1zHQGw6D6VolBm2LHwkosVBQ WqLBVmIWg0VdPQ26NDeYqr1Dw/KkDxgnhknouGskoWXUE26Xj1HwvMVIQNfjKQwGnpZRMNHw hwRTVw8B5sI8Eu7PVVLw2a7DQWebp6G/TYtBU8aK7fK33yR057VhcLn6IQaD754haM16j4Gh YZiCDpsVg2ZDEQ4/azoRTOV/oSEzd5GG0kv5CHIyiwnIGPOHpR9lVHAg32Gdx/mM5mS+xa4l +FeVHP/kzjjNZ7SO0rzWkMg313rwVc8tGF+xYCN5Q/1VijcsXKf57C+DGD/X10fzPbeWCH56 sAQ76nxStDdSiFEkCaqd+8JF0UOmUjreHHT2Q5OdSENPfLORA8OxftxEmhFfZYp150ZGFtfY kd3ONefNkKuMs1YRV913cJU3shHcfLp+rSdYN65u6T2djRhGzAZwZXf3/lO6cPqmtjWNw0pd MFdArbKE9ed6NRXUNSTSonX1yFGhTIqVK2L8vdSno1OUirNeEXGxBrTyFt2F5cLH6PuArB2x DJKuFwc7fBQkpDxJnRLbjjgGlzqKz/+cFiTiSHnKOUEVF6ZKjBHU7ciZIaSbxSGhQriEjZIn CKcFIV5Q/V8xxsEpDV2cqh3Ktx6YHmnStG4zheo6UjdZAriQ4KPF8cfNnppjlq81yeujZnbT n5R1sq0vEvb7HiuXfXU/tfXdFY6o6EuV2INQvueZsRPDbl7OUVW3F4bwwl1RCTmwpdf1QqDe 6HpjtupezyHaL/1H/69u3OOlyVs2PqwNS4z1S3LxnSGkhDpa7uOBq9Tyv2POYkEpAwAA X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: 1EEA0180004 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: qcy4tt7wjsengkgn9qfepdzuyck9amsn X-HE-Tag: 1709715338-103841 X-HE-Meta: U2FsdGVkX1/wBepOij3mlqL7iuKSGBW4Ll2u0K3y8k7wGk/NZQLBsURF5iKdn2khvjwiEJCUZ6fa5Wo/mlW3sU2E0lu+00eggToq4teQSMNHHf4pzw/uYDI+N4wiLncH821UKcy0rFZ1U4A2zHphhULnPp8vyZlqH02CXM9YzEwi76nOwuJ4AVj2gTS8EC1r0HsnriVanpooVrvwOHxsq1STtKo68x7CJI9ZdkQd5IMr4LuB9Jy5IkmCBlWZG7rXNj1VmqFuxUcEoPy5neVlOYPQkqOqxHhPOnFnXK5eBgXU1c6Mr9hzxziv3wbTTkFPMvCNdULIhpxZqoLGTuHDR6pwzbkWy2rAuWyrJ23XnJAgLfbGH+yOnjVWIy21Kz/lYjuzMNcnGvmB/Qzy5vhimrV1WIEp3uvlBmG9hjTw2ghGs0UMD7ug/38lEO8DWL/3jGCEtymPueZK0yjin/qNTD1us69g+uQbuWGDbwzlCgC6rRmvly23FrZ2DFzxAFiDkfppqeoUAHjmZx8q2vm4CdRW/e0oSAy306mTTeGukuqbQAhr6ZACifo+0DTr0TfrLcLCVSwsj/C9f1uovUt8zZz0zdoaG3TjWzk22YXz3Wqh+O0TzQ15UQTrs1CXhgIXmq2muxbg8+IJ70B+9V4quOgpubzwyct3MybWgjU6jPbh7MwtnS/zDNsRUTANaK7fFeNwm4B63nvzbxJdcmYmf/OVs6zDESLf51D6A0dGpyat4b6M50VUkqnyFF1xywI3ODXWGieTCmSrcHVPA0/ZfA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Dept engine works in a constrained environment. For example, Dept cannot make use of dynamic allocation e.g. kmalloc(). So Dept has been using static pools to keep memory chunks Dept uses. However, Dept would barely work once any of the pools gets run out. So implemented a mechanism for the refill on the lack by any chance, using irq work and workqueue that fits on the contrained environment. Signed-off-by: Byungchul Park --- include/linux/dept.h | 19 ++++-- kernel/dependency/dept.c | 104 +++++++++++++++++++++++++++----- kernel/dependency/dept_object.h | 10 +-- kernel/dependency/dept_proc.c | 8 +-- 4 files changed, 112 insertions(+), 29 deletions(-) diff --git a/include/linux/dept.h b/include/linux/dept.h index 319a5b43df89..ca1a34be4127 100644 --- a/include/linux/dept.h +++ b/include/linux/dept.h @@ -336,9 +336,19 @@ struct dept_pool { size_t obj_sz; /* - * the number of the static array + * the remaining number of the object in spool */ - atomic_t obj_nr; + int obj_nr; + + /* + * the number of the object in spool + */ + int tot_nr; + + /* + * accumulated amount of memory used by the object in byte + */ + atomic_t acc_sz; /* * offset of ->pool_node @@ -348,9 +358,10 @@ struct dept_pool { /* * pointer to the pool */ - void *spool; + void *spool; /* static pool */ + void *rpool; /* reserved pool */ struct llist_head boot_pool; - struct llist_head __percpu *lpool; + struct llist_head __percpu *lpool; /* local pool */ }; struct dept_ecxt_held { diff --git a/kernel/dependency/dept.c b/kernel/dependency/dept.c index a8e693fd590f..8ca46ad98e10 100644 --- a/kernel/dependency/dept.c +++ b/kernel/dependency/dept.c @@ -74,6 +74,9 @@ #include #include #include +#include +#include +#include #include "dept_internal.h" static int dept_stop; @@ -122,9 +125,11 @@ static int dept_per_cpu_ready; WARN(1, "DEPT_STOP: " s); \ }) -#define DEPT_INFO_ONCE(s...) pr_warn_once("DEPT_INFO_ONCE: " s) +#define DEPT_INFO_ONCE(s...) pr_warn_once("DEPT_INFO_ONCE: " s) +#define DEPT_INFO(s...) pr_warn("DEPT_INFO: " s) static arch_spinlock_t dept_spin = (arch_spinlock_t)__ARCH_SPIN_LOCK_UNLOCKED; +static arch_spinlock_t dept_pool_spin = (arch_spinlock_t)__ARCH_SPIN_LOCK_UNLOCKED; /* * DEPT internal engine should be careful in using outside functions @@ -263,6 +268,7 @@ static bool valid_key(struct dept_key *k) #define OBJECT(id, nr) \ static struct dept_##id spool_##id[nr]; \ +static struct dept_##id rpool_##id[nr]; \ static DEFINE_PER_CPU(struct llist_head, lpool_##id); #include "dept_object.h" #undef OBJECT @@ -271,14 +277,70 @@ struct dept_pool dept_pool[OBJECT_NR] = { #define OBJECT(id, nr) { \ .name = #id, \ .obj_sz = sizeof(struct dept_##id), \ - .obj_nr = ATOMIC_INIT(nr), \ + .obj_nr = nr, \ + .tot_nr = nr, \ + .acc_sz = ATOMIC_INIT(sizeof(spool_##id) + sizeof(rpool_##id)), \ .node_off = offsetof(struct dept_##id, pool_node), \ .spool = spool_##id, \ + .rpool = rpool_##id, \ .lpool = &lpool_##id, }, #include "dept_object.h" #undef OBJECT }; +static void dept_wq_work_fn(struct work_struct *work) +{ + int i; + + for (i = 0; i < OBJECT_NR; i++) { + struct dept_pool *p = dept_pool + i; + int sz = p->tot_nr * p->obj_sz; + void *rpool; + bool need; + + arch_spin_lock(&dept_pool_spin); + need = !p->rpool; + arch_spin_unlock(&dept_pool_spin); + + if (!need) + continue; + + rpool = vmalloc(sz); + + if (!rpool) { + DEPT_STOP("Failed to extend internal resources.\n"); + break; + } + + arch_spin_lock(&dept_pool_spin); + if (!p->rpool) { + p->rpool = rpool; + rpool = NULL; + atomic_add(sz, &p->acc_sz); + } + arch_spin_unlock(&dept_pool_spin); + + if (rpool) + vfree(rpool); + else + DEPT_INFO("Dept object(%s) just got refilled successfully.\n", p->name); + } +} + +static DECLARE_WORK(dept_wq_work, dept_wq_work_fn); + +static void dept_irq_work_fn(struct irq_work *w) +{ + schedule_work(&dept_wq_work); +} + +static DEFINE_IRQ_WORK(dept_irq_work, dept_irq_work_fn); + +static void request_rpool_refill(void) +{ + irq_work_queue(&dept_irq_work); +} + /* * Can use llist no matter whether CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG is * enabled or not because NMI and other contexts in the same CPU never @@ -314,19 +376,31 @@ static void *from_pool(enum object_t t) /* * Try static pool. */ - if (atomic_read(&p->obj_nr) > 0) { - int idx = atomic_dec_return(&p->obj_nr); + arch_spin_lock(&dept_pool_spin); + + if (!p->obj_nr) { + p->spool = p->rpool; + p->obj_nr = p->rpool ? p->tot_nr : 0; + p->rpool = NULL; + request_rpool_refill(); + } + + if (p->obj_nr) { + void *ret; + + p->obj_nr--; + ret = p->spool + (p->obj_nr * p->obj_sz); + arch_spin_unlock(&dept_pool_spin); - if (idx >= 0) - return p->spool + (idx * p->obj_sz); + return ret; } + arch_spin_unlock(&dept_pool_spin); - DEPT_INFO_ONCE("---------------------------------------------\n" - " Some of Dept internal resources are run out.\n" - " Dept might still work if the resources get freed.\n" - " However, the chances are Dept will suffer from\n" - " the lack from now. Needs to extend the internal\n" - " resource pools. Ask max.byungchul.park@gmail.com\n"); + DEPT_INFO("------------------------------------------\n" + " Dept object(%s) is run out.\n" + " Dept is trying to refill the object.\n" + " Nevertheless, if it fails, Dept will stop.\n", + p->name); return NULL; } @@ -2957,8 +3031,8 @@ void __init dept_init(void) pr_info("... DEPT_MAX_ECXT_HELD : %d\n", DEPT_MAX_ECXT_HELD); pr_info("... DEPT_MAX_SUBCLASSES : %d\n", DEPT_MAX_SUBCLASSES); #define OBJECT(id, nr) \ - pr_info("... memory used by %s: %zu KB\n", \ - #id, B2KB(sizeof(struct dept_##id) * nr)); + pr_info("... memory initially used by %s: %zu KB\n", \ + #id, B2KB(sizeof(spool_##id) + sizeof(rpool_##id))); #include "dept_object.h" #undef OBJECT #define HASH(id, bits) \ @@ -2966,6 +3040,6 @@ void __init dept_init(void) #id, B2KB(sizeof(struct hlist_head) * (1 << (bits)))); #include "dept_hash.h" #undef HASH - pr_info("... total memory used by objects and hashs: %zu KB\n", B2KB(mem_total)); + pr_info("... total memory initially used by objects and hashs: %zu KB\n", B2KB(mem_total)); pr_info("... per task memory footprint: %zu bytes\n", sizeof(struct dept_task)); } diff --git a/kernel/dependency/dept_object.h b/kernel/dependency/dept_object.h index 0b7eb16fe9fb..4f936adfa8ee 100644 --- a/kernel/dependency/dept_object.h +++ b/kernel/dependency/dept_object.h @@ -6,8 +6,8 @@ * nr: # of the object that should be kept in the pool. */ -OBJECT(dep, 1024 * 8) -OBJECT(class, 1024 * 8) -OBJECT(stack, 1024 * 32) -OBJECT(ecxt, 1024 * 16) -OBJECT(wait, 1024 * 32) +OBJECT(dep, 1024 * 4 * 2) +OBJECT(class, 1024 * 4) +OBJECT(stack, 1024 * 4 * 8) +OBJECT(ecxt, 1024 * 4 * 2) +OBJECT(wait, 1024 * 4 * 4) diff --git a/kernel/dependency/dept_proc.c b/kernel/dependency/dept_proc.c index 7d61dfbc5865..f07a512b203f 100644 --- a/kernel/dependency/dept_proc.c +++ b/kernel/dependency/dept_proc.c @@ -73,12 +73,10 @@ static int dept_stats_show(struct seq_file *m, void *v) { int r; - seq_puts(m, "Availability in the static pools:\n\n"); + seq_puts(m, "Accumulated amount of memory used by pools:\n\n"); #define OBJECT(id, nr) \ - r = atomic_read(&dept_pool[OBJECT_##id].obj_nr); \ - if (r < 0) \ - r = 0; \ - seq_printf(m, "%s\t%d/%d(%d%%)\n", #id, r, nr, (r * 100) / (nr)); + r = atomic_read(&dept_pool[OBJECT_##id].acc_sz); \ + seq_printf(m, "%s\t%d KB\n", #id, r / 1024); #include "dept_object.h" #undef OBJECT