From patchwork Tue Jan 31 08:39:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13122444 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F08C5C636CD for ; Tue, 31 Jan 2023 08:40:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 26B1A6B0099; Tue, 31 Jan 2023 03:40:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1A4FB6B0096; Tue, 31 Jan 2023 03:40:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E7BCB6B0098; Tue, 31 Jan 2023 03:40:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id C109F6B0093 for ; Tue, 31 Jan 2023 03:40:04 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 95B74AAFB8 for ; Tue, 31 Jan 2023 08:40:04 +0000 (UTC) X-FDA: 80414446728.06.8AAC0E2 Received: from lgeamrelo11.lge.com (lgeamrelo11.lge.com [156.147.23.51]) by imf14.hostedemail.com (Postfix) with ESMTP id 287F010001F for ; Tue, 31 Jan 2023 08:40:01 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (imf14.hostedemail.com: 156.147.23.51 is neither permitted nor denied by domain of max.byungchul.park@gmail.com) smtp.mailfrom=max.byungchul.park@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675154402; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=YrtxP3iNfNB6GMz6Yene4KD2LoXzXVa4AcHpJ/MRGX4=; b=QX1V4tgOV64OzwDop/tLiWn6spyM6ui1uSCkbGgvK9oUoV3Shpx7JUhon6FmZ4rav3b3s/ 1uFT0jxJIH2+r19juYwvCDAmSlzOCTI5S+EhvH39UJDL/2lubTmbE5ApQSxAyzECLsxi1Z 0OdoP3O7vMSwrBkakFtARTpaf480kys= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (imf14.hostedemail.com: 156.147.23.51 is neither permitted nor denied by domain of max.byungchul.park@gmail.com) smtp.mailfrom=max.byungchul.park@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675154402; a=rsa-sha256; cv=none; b=H1Jf4VGZirf5noQ3Mvat9RF2AvwalI/OY9Ztp70GpzIJYgQhAuP7aOit59CPA1eVYUs+AA tV+NPR5HYYvg9so3zikxGTNszF1jS7lg8JDHkS1pvAs5EbHH9ZJyxwxIY3ccyJ+KE0LkdS IagipdV/RI15AOypy8lF+/ldIRhuOtQ= Received: from unknown (HELO lgeamrelo04.lge.com) (156.147.1.127) by 156.147.23.51 with ESMTP; 31 Jan 2023 17:39:59 +0900 X-Original-SENDERIP: 156.147.1.127 X-Original-MAILFROM: max.byungchul.park@gmail.com Received: from unknown (HELO localhost.localdomain) (10.177.244.38) by 156.147.1.127 with ESMTP; 31 Jan 2023 17:39:59 +0900 X-Original-SENDERIP: 10.177.244.38 X-Original-MAILFROM: max.byungchul.park@gmail.com From: Byungchul Park To: linux-kernel@vger.kernel.org Cc: torvalds@linux-foundation.org, damien.lemoal@opensource.wdc.com, linux-ide@vger.kernel.org, adilger.kernel@dilger.ca, linux-ext4@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, will@kernel.org, tglx@linutronix.de, rostedt@goodmis.org, joel@joelfernandes.org, sashal@kernel.org, daniel.vetter@ffwll.ch, duyuyang@gmail.com, johannes.berg@intel.com, tj@kernel.org, tytso@mit.edu, willy@infradead.org, david@fromorbit.com, amir73il@gmail.com, gregkh@linuxfoundation.org, kernel-team@lge.com, linux-mm@kvack.org, akpm@linux-foundation.org, mhocko@kernel.org, minchan@kernel.org, hannes@cmpxchg.org, vdavydov.dev@gmail.com, sj@kernel.org, jglisse@redhat.com, dennis@kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, vbabka@suse.cz, ngupta@vflare.org, linux-block@vger.kernel.org, paolo.valente@linaro.org, josef@toxicpanda.com, linux-fsdevel@vger.kernel.org, viro@zeniv.linux.org.uk, jack@suse.cz, jlayton@kernel.org, dan.j.williams@intel.com, hch@infradead.org, djwong@kernel.org, dri-devel@lists.freedesktop.org, rodrigosiqueiramelo@gmail.com, melissa.srw@gmail.com, hamohammed.sa@gmail.com, 42.hyeyoo@gmail.com, chris.p.wilson@intel.com, gwan-gyeong.mun@intel.com, max.byungchul.park@gmail.com, boqun.feng@gmail.com, longman@redhat.com, hdanton@sina.com Subject: [PATCH v9 14/25] dept: Add a mechanism to refill the internal memory pools on running out Date: Tue, 31 Jan 2023 17:39:43 +0900 Message-Id: <1675154394-25598-15-git-send-email-max.byungchul.park@gmail.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1675154394-25598-1-git-send-email-max.byungchul.park@gmail.com> References: <1675154394-25598-1-git-send-email-max.byungchul.park@gmail.com> X-Rspamd-Queue-Id: 287F010001F X-Stat-Signature: rcaawy9auiwgjna4azjnywfnaet9ws4i X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1675154401-231745 X-HE-Meta: U2FsdGVkX1//yTzs6i3IfSgQIp5PMOi7rJUeopCrcGWgekKQuwa5jI/qljyRqExbEtLS4ZkUs5tz1D4A9mNZR5eOE3kpH1CEFbR0J3FEDIZpErf6Jkxw6KRsE6dnRSURBDa71KGTKIgr9S89Od/n5Lk9H21ltvqTKUDl0Q0+k1UiQLaQJTRduv1hO+GL5KBwf3YA+HKQ8IPd+DG7q2fwr+cIRR1VOPblk5CGWG8SQ/tF/5X7evgqw3CAOoNFsn8/I142tLMmMm/7T4hp0eJogQFToU36tb8+doDI5LLLcr5Ti2KGeLGIbn2mmaC1XLB1G6KiyZXMYZyMsj+9aCyzf6cNHoWk/J48mBEd4/NY1Q7TZkGXybr4OZNYrhtZrTbxStY/3YBRYhpvUnh76m6/v7lgRXpyZegCfFVNZvLF8h4KP5j0uh1xghc1iz7qRrMPPpkOosiXL+TpVcKs45LGyQjqxT7kTL5V2qQV2NA/JEMxz4OoeCvTvDtXB6vrpl2VdQZkSwyPo3GZaqHJVPW+JVrXY4cy5It7r00SWPnRPdIXJs9bxsbHXTCioum7CdgZAVrFCWe9nrxeqMqxlnhYHtk9cx39RVB1Twb5bBRr6+IiJinCfFU3udV0wFziqaH2sT2dVDr6BBGYbsTXESSNjX6x/ur/cFp3kwswGhOoo5zoup+0GzwToWAUpUVjr5WJfW9McoeOlx52TpEosjhVkM795rQkJ9fGO7HcPT3TMdq3F2sC+DCVYFOKpiRWtjqklLd+KwuztAudsLrvpjMI6qSA6ktrUcqPrmBN8U1wBtoofKz+wp5Sdu5UPOPWoQ3b8Ouf89c4ExeQcLnBiy2cwPLIDEIUAMOj+5ENa8wUFno2WkzfFpMP4SoaaI53k9+8Bm4QIsizFluwDK/a2I1JxjwuO82FT4xXFHGRMlCRVcZA1JyMZa7DiSjaQgiIzNGyJi2oebLIvHq/kP7POPu mgyhCcgG HQEYactbdJHgSP2SnHzmHMmjncWsI1diSuJJvmG5F1exqzzPgiPo4STGtVCnnlxZYFlLfW0YYMR0KdlTPX2huD05DBSk0MshVnp3NSyDrJ74u9UMl0HBWt7PeBBL3Wwn50xoEwUV9+XDwcfYcDFGGSrqvmYQ8Vy8SQ6497E8BdiIYDv84dsc7ru0OnzqSqhFl040AiQDO6o2MPURuQ5Mey5NI1Pa5+mOIfEa9jCiqx0ESge0pyzAP7F2YcFiKw5/N1urtpC0DSO0g/21APr7UG7Rlwr1pJg6yNMZTRRXPN7azzA+LK0CUF9cpqaAKpkzlWTio X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Dept engine works in a constrained environment. For example, Dept cannot make use of dynamic allocation e.g. kmalloc(). So Dept has been using static pools to keep memory chunks Dept uses. However, Dept would barely work once any of the pools gets run out. So implemented a mechanism for the refill on the lack by any chance, using irq work and workqueue that fits on the contrained environment. Signed-off-by: Byungchul Park --- include/linux/dept.h | 19 ++++++-- kernel/dependency/dept.c | 104 ++++++++++++++++++++++++++++++++++------ kernel/dependency/dept_object.h | 10 ++-- kernel/dependency/dept_proc.c | 8 ++-- 4 files changed, 112 insertions(+), 29 deletions(-) diff --git a/include/linux/dept.h b/include/linux/dept.h index d9ca9dd..583e8fe 100644 --- a/include/linux/dept.h +++ b/include/linux/dept.h @@ -336,9 +336,19 @@ struct dept_pool { size_t obj_sz; /* - * the number of the static array + * the remaining number of the object in spool */ - atomic_t obj_nr; + int obj_nr; + + /* + * the number of the object in spool + */ + int tot_nr; + + /* + * accumulated amount of memory used by the object in byte + */ + atomic_t acc_sz; /* * offset of ->pool_node @@ -348,9 +358,10 @@ struct dept_pool { /* * pointer to the pool */ - void *spool; + void *spool; /* static pool */ + void *rpool; /* reserved pool */ struct llist_head boot_pool; - struct llist_head __percpu *lpool; + struct llist_head __percpu *lpool; /* local pool */ }; struct dept_ecxt_held { diff --git a/kernel/dependency/dept.c b/kernel/dependency/dept.c index 1f1aca4..0ac6442 100644 --- a/kernel/dependency/dept.c +++ b/kernel/dependency/dept.c @@ -74,6 +74,9 @@ #include #include #include +#include +#include +#include #include "dept_internal.h" static int dept_stop; @@ -122,10 +125,12 @@ WARN(1, "DEPT_STOP: " s); \ }) -#define DEPT_INFO_ONCE(s...) pr_warn_once("DEPT_INFO_ONCE: " s) +#define DEPT_INFO_ONCE(s...) pr_warn_once("DEPT_INFO_ONCE: " s) +#define DEPT_INFO(s...) pr_warn("DEPT_INFO: " s) static arch_spinlock_t dept_spin = (arch_spinlock_t)__ARCH_SPIN_LOCK_UNLOCKED; static arch_spinlock_t stage_spin = (arch_spinlock_t)__ARCH_SPIN_LOCK_UNLOCKED; +static arch_spinlock_t dept_pool_spin = (arch_spinlock_t)__ARCH_SPIN_LOCK_UNLOCKED; /* * DEPT internal engine should be careful in using outside functions @@ -264,6 +269,7 @@ static inline bool valid_key(struct dept_key *k) #define OBJECT(id, nr) \ static struct dept_##id spool_##id[nr]; \ +static struct dept_##id rpool_##id[nr]; \ static DEFINE_PER_CPU(struct llist_head, lpool_##id); #include "dept_object.h" #undef OBJECT @@ -272,14 +278,70 @@ struct dept_pool dept_pool[OBJECT_NR] = { #define OBJECT(id, nr) { \ .name = #id, \ .obj_sz = sizeof(struct dept_##id), \ - .obj_nr = ATOMIC_INIT(nr), \ + .obj_nr = nr, \ + .tot_nr = nr, \ + .acc_sz = ATOMIC_INIT(sizeof(spool_##id) + sizeof(rpool_##id)), \ .node_off = offsetof(struct dept_##id, pool_node), \ .spool = spool_##id, \ + .rpool = rpool_##id, \ .lpool = &lpool_##id, }, #include "dept_object.h" #undef OBJECT }; +static void dept_wq_work_fn(struct work_struct *work) +{ + int i; + + for (i = 0; i < OBJECT_NR; i++) { + struct dept_pool *p = dept_pool + i; + int sz = p->tot_nr * p->obj_sz; + void *rpool; + bool need; + + arch_spin_lock(&dept_pool_spin); + need = !p->rpool; + arch_spin_unlock(&dept_pool_spin); + + if (!need) + continue; + + rpool = vmalloc(sz); + + if (!rpool) { + DEPT_STOP("Failed to extend internal resources.\n"); + break; + } + + arch_spin_lock(&dept_pool_spin); + if (!p->rpool) { + p->rpool = rpool; + rpool = NULL; + atomic_add(sz, &p->acc_sz); + } + arch_spin_unlock(&dept_pool_spin); + + if (rpool) + vfree(rpool); + else + DEPT_INFO("Dept object(%s) just got refilled successfully.\n", p->name); + } +} + +static DECLARE_WORK(dept_wq_work, dept_wq_work_fn); + +static void dept_irq_work_fn(struct irq_work *w) +{ + schedule_work(&dept_wq_work); +} + +static DEFINE_IRQ_WORK(dept_irq_work, dept_irq_work_fn); + +static void request_rpool_refill(void) +{ + irq_work_queue(&dept_irq_work); +} + /* * Can use llist no matter whether CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG is * enabled or not because NMI and other contexts in the same CPU never @@ -315,19 +377,31 @@ static void *from_pool(enum object_t t) /* * Try static pool. */ - if (atomic_read(&p->obj_nr) > 0) { - int idx = atomic_dec_return(&p->obj_nr); + arch_spin_lock(&dept_pool_spin); + + if (!p->obj_nr) { + p->spool = p->rpool; + p->obj_nr = p->rpool ? p->tot_nr : 0; + p->rpool = NULL; + request_rpool_refill(); + } + + if (p->obj_nr) { + void *ret; + + p->obj_nr--; + ret = p->spool + (p->obj_nr * p->obj_sz); + arch_spin_unlock(&dept_pool_spin); - if (idx >= 0) - return p->spool + (idx * p->obj_sz); + return ret; } + arch_spin_unlock(&dept_pool_spin); - DEPT_INFO_ONCE("---------------------------------------------\n" - " Some of Dept internal resources are run out.\n" - " Dept might still work if the resources get freed.\n" - " However, the chances are Dept will suffer from\n" - " the lack from now. Needs to extend the internal\n" - " resource pools. Ask max.byungchul.park@gmail.com\n"); + DEPT_INFO("------------------------------------------\n" + " Dept object(%s) is run out.\n" + " Dept is trying to refill the object.\n" + " Nevertheless, if it fails, Dept will stop.\n", + p->name); return NULL; } @@ -3000,8 +3074,8 @@ void __init dept_init(void) pr_info("... DEPT_MAX_ECXT_HELD : %d\n", DEPT_MAX_ECXT_HELD); pr_info("... DEPT_MAX_SUBCLASSES : %d\n", DEPT_MAX_SUBCLASSES); #define OBJECT(id, nr) \ - pr_info("... memory used by %s: %zu KB\n", \ - #id, B2KB(sizeof(struct dept_##id) * nr)); + pr_info("... memory initially used by %s: %zu KB\n", \ + #id, B2KB(sizeof(spool_##id) + sizeof(rpool_##id))); #include "dept_object.h" #undef OBJECT #define HASH(id, bits) \ @@ -3009,6 +3083,6 @@ void __init dept_init(void) #id, B2KB(sizeof(struct hlist_head) * (1 << (bits)))); #include "dept_hash.h" #undef HASH - pr_info("... total memory used by objects and hashs: %zu KB\n", B2KB(mem_total)); + pr_info("... total memory initially used by objects and hashs: %zu KB\n", B2KB(mem_total)); pr_info("... per task memory footprint: %zu bytes\n", sizeof(struct dept_task)); } diff --git a/kernel/dependency/dept_object.h b/kernel/dependency/dept_object.h index 0b7eb16..4f936ad 100644 --- a/kernel/dependency/dept_object.h +++ b/kernel/dependency/dept_object.h @@ -6,8 +6,8 @@ * nr: # of the object that should be kept in the pool. */ -OBJECT(dep, 1024 * 8) -OBJECT(class, 1024 * 8) -OBJECT(stack, 1024 * 32) -OBJECT(ecxt, 1024 * 16) -OBJECT(wait, 1024 * 32) +OBJECT(dep, 1024 * 4 * 2) +OBJECT(class, 1024 * 4) +OBJECT(stack, 1024 * 4 * 8) +OBJECT(ecxt, 1024 * 4 * 2) +OBJECT(wait, 1024 * 4 * 4) diff --git a/kernel/dependency/dept_proc.c b/kernel/dependency/dept_proc.c index 7d61dfb..f07a512 100644 --- a/kernel/dependency/dept_proc.c +++ b/kernel/dependency/dept_proc.c @@ -73,12 +73,10 @@ static int dept_stats_show(struct seq_file *m, void *v) { int r; - seq_puts(m, "Availability in the static pools:\n\n"); + seq_puts(m, "Accumulated amount of memory used by pools:\n\n"); #define OBJECT(id, nr) \ - r = atomic_read(&dept_pool[OBJECT_##id].obj_nr); \ - if (r < 0) \ - r = 0; \ - seq_printf(m, "%s\t%d/%d(%d%%)\n", #id, r, nr, (r * 100) / (nr)); + r = atomic_read(&dept_pool[OBJECT_##id].acc_sz); \ + seq_printf(m, "%s\t%d KB\n", #id, r / 1024); #include "dept_object.h" #undef OBJECT