From patchwork Mon Jul 3 09:47:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13299799 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 257AAC10F00 for ; Mon, 3 Jul 2023 09:50:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231289AbjGCJuE (ORCPT ); Mon, 3 Jul 2023 05:50:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50168 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231316AbjGCJtx (ORCPT ); Mon, 3 Jul 2023 05:49:53 -0400 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id D9E2D18D; Mon, 3 Jul 2023 02:49:49 -0700 (PDT) X-AuditID: a67dfc5b-d85ff70000001748-b7-64a299b31b80 From: Byungchul Park To: linux-kernel@vger.kernel.org Cc: kernel_team@skhynix.com, torvalds@linux-foundation.org, damien.lemoal@opensource.wdc.com, linux-ide@vger.kernel.org, adilger.kernel@dilger.ca, linux-ext4@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, will@kernel.org, tglx@linutronix.de, rostedt@goodmis.org, joel@joelfernandes.org, sashal@kernel.org, daniel.vetter@ffwll.ch, duyuyang@gmail.com, johannes.berg@intel.com, tj@kernel.org, tytso@mit.edu, willy@infradead.org, david@fromorbit.com, amir73il@gmail.com, gregkh@linuxfoundation.org, kernel-team@lge.com, linux-mm@kvack.org, akpm@linux-foundation.org, mhocko@kernel.org, minchan@kernel.org, hannes@cmpxchg.org, vdavydov.dev@gmail.com, sj@kernel.org, jglisse@redhat.com, dennis@kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, vbabka@suse.cz, ngupta@vflare.org, linux-block@vger.kernel.org, josef@toxicpanda.com, linux-fsdevel@vger.kernel.org, viro@zeniv.linux.org.uk, jack@suse.cz, jlayton@kernel.org, dan.j.williams@intel.com, hch@infradead.org, djwong@kernel.org, dri-devel@lists.freedesktop.org, rodrigosiqueiramelo@gmail.com, melissa.srw@gmail.com, hamohammed.sa@gmail.com, 42.hyeyoo@gmail.com, chris.p.wilson@intel.com, gwan-gyeong.mun@intel.com, max.byungchul.park@gmail.com, boqun.feng@gmail.com, longman@redhat.com, hdanton@sina.com, her0gyugyu@gmail.com Subject: [PATCH v10 rebased on v6.4 14/25] dept: Add a mechanism to refill the internal memory pools on running out Date: Mon, 3 Jul 2023 18:47:41 +0900 Message-Id: <20230703094752.79269-15-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230703094752.79269-1-byungchul@sk.com> References: <20230703094752.79269-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAAzWSf0yMcRzHfb/PzzuOxzEetWE31iZSlH2YYfOHZ8zGzObXxnHPdFOX3SWX ZiuVKDVMutPRdezcrijPmYWyk4m0ODp0dt0qhdYvOy6dCv2Yfz577f1+7/XXhyWUD6kIVqtL FfU6dZKKlpPy/hm2FS6zTRNr7QW4dCEWQj/PkWCpqqTBc7cCQeX9LAw9z7fAx6E+BCPNbwgo KfYgKO9oI+B+QwBBneMMDS1dM8EbGqShsbiAhuybVTS87R3F4L96GUOFtB2aLtowuMNfSSjp oaG0JBuPn28YwnYnA/bMpdDpuMbAaEccNAY+UFD3KRrMN/w01NY1ktBQ04mh5ZGFhkDlXwqa Gl6S4LlUSMGdARsNvUN2AuyhQQbeua0YqnPGRWd//KHgRaEbw9lb9zB4fY8RPDnXjkGq/EDD s1AfBpdUTMDv288RdBb1M5B7IcxAaVYRgoLcqyS8GXtBQY4/AUaGLfSmdcKzvkFCyHGdFOqG rKTwysYLD6+1MULOk0+MYJVOCC7HMuFmbQ8WyoMhSpCc52lBCl5mhPx+LxYGXr9mhJemEVLo 8pbgHZH75Os1YpI2TdSv3HBInmjJfYWPOzcZP/vcVCYqWJ2PWJbn4nmfIzUfyabQb2YmmOai +NbWMDHBc7nFvKvwC5WP5CzB5U3nHd+b6YliDmfgzbXdk0xyS3npvQVNsIJbw5uandSUdBFf Ue2eFMnG8+7hosmNkkvg/eYAPbXJk/HdWSumeAH/1NFKXkQKK5rmREqtLi1ZrU2Kj0lM12mN MUdSkiU0/lD206P7a1DQs6secSxSzVC0ZpRrlJQ6zZCeXI94llDNVWR3lGmUCo06/ZSoTzmo P5EkGupRJEuq5itWDZ3UKLmj6lTxmCgeF/X/W8zKIjIRu8FwID6AnIotNYq2DOzLKPM3zTuy M+7zFY81MdUkRZt2Ny1M+WXnI9r7Rz4OZy4aa+nee/C6YzPeeiX51J68sihje2x4Z3qMk1+w cdU209Mu456xB9P2LxngDp+XzfYv/zZryQ1N6ba1QXvFwgSz8YHX/jduZuhWdHVkcAB8OhVp SFTHLSP0BvU/01gjckwDAAA= X-Brightmail-Tracker: H4sIAAAAAAAAAzWSW0iTcRjG+/+/o6vV1xL6qKhYWamUSRkvdL4oPyIjgjC6qEb7cKtpsall UWmbHTyhgdp0moeYY1PLzYsOGkvJWqFZmk2ZKy0rc2YHJ5lmqdHNy4/nefhdvSwhK6IWsOq4 eFEbp9DIaQkp2bNRv9phLFOunfAuhpyMteAfvkKC6VYlDa3VNgSVtSkY+h9FwusRH4Kx5ucE 5Oe2Iijt6SagtsmLoN5ykYa297Oh3T9Egys3nQZ9+S0aXgyMY/DkXcNgs0fBs+wyDM7RjyTk 99NQmK/Hk+cThlGzlQFzchD0WgoYGO8JB5e3g4LGIhcF9V2hYCz20FBX7yKh6U4vhrZ7Jhq8 lX8oeNb0hITWnEwKqr6U0TAwYibA7B9i4KWzBMNtw6Tt0o8JCh5nOjFculmDob3zPoIHV95i sFd20NDo92Fw2HMJ+FXxCEFv1iADqRmjDBSmZCFIT80j4fnvxxQYPBEw9tNEb9soNPqGCMHg OCXUj5SQwtMyXrhb0M0IhgddjFBiTxAclhChvK4fC6Xf/ZRgt16lBfv3a4yQNtiOhS8tLYzw 5PoYKbxvz8d7Fx2UbFKKGnWiqA3bckSiMqU+xSet206/63RSySh9XRoKYHluPd/pMTJTTHMr ebd7lJjiQG4p78j8QKUhCUtwl2fylq/N9FQxj9Pxxrq+aSa5IN7+yoSmWMpt4K83W6l/0iW8 7bZzWhQwmff9zJreyLgI3mP00tlIUoJmWFGgOi4xVqHWRKzRHVclxalPrzl6ItaOJp/GfG48 5w4abotsQByL5LOk7rOlShmlSNQlxTYgniXkgVJ9zw2lTKpUJJ0RtScOaxM0oq4BLWRJ+Xzp rmjxiIyLUcSLx0XxpKj932I2YEEyCqrBUed3Hlwabutj5x7rLw+Zs/mQu7H0g63bs6kDr19W PTGDSd2qifn0Jnr5u5dOV7Fq+92qX6uy3bvPLs+qqGhVmLJr5+sLCF/G+WWva+99toUlDP8h V6gv7JjzbdaFQVWYkyncZxmo9sXLWx4eCN5Q4Uzh9s+rCZ6ZEhrsUQ13yUmdShEeQmh1ir8q 62/RMAMAAA== X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Dept engine works in a constrained environment. For example, Dept cannot make use of dynamic allocation e.g. kmalloc(). So Dept has been using static pools to keep memory chunks Dept uses. However, Dept would barely work once any of the pools gets run out. So implemented a mechanism for the refill on the lack by any chance, using irq work and workqueue that fits on the contrained environment. Signed-off-by: Byungchul Park --- include/linux/dept.h | 19 ++++-- kernel/dependency/dept.c | 104 +++++++++++++++++++++++++++----- kernel/dependency/dept_object.h | 10 +-- kernel/dependency/dept_proc.c | 8 +-- 4 files changed, 112 insertions(+), 29 deletions(-) diff --git a/include/linux/dept.h b/include/linux/dept.h index d9ca9dd50219..583e8fe2dd7b 100644 --- a/include/linux/dept.h +++ b/include/linux/dept.h @@ -336,9 +336,19 @@ struct dept_pool { size_t obj_sz; /* - * the number of the static array + * the remaining number of the object in spool */ - atomic_t obj_nr; + int obj_nr; + + /* + * the number of the object in spool + */ + int tot_nr; + + /* + * accumulated amount of memory used by the object in byte + */ + atomic_t acc_sz; /* * offset of ->pool_node @@ -348,9 +358,10 @@ struct dept_pool { /* * pointer to the pool */ - void *spool; + void *spool; /* static pool */ + void *rpool; /* reserved pool */ struct llist_head boot_pool; - struct llist_head __percpu *lpool; + struct llist_head __percpu *lpool; /* local pool */ }; struct dept_ecxt_held { diff --git a/kernel/dependency/dept.c b/kernel/dependency/dept.c index 6cf17f206b78..8454f0a14d67 100644 --- a/kernel/dependency/dept.c +++ b/kernel/dependency/dept.c @@ -74,6 +74,9 @@ #include #include #include +#include +#include +#include #include "dept_internal.h" static int dept_stop; @@ -122,10 +125,12 @@ static int dept_per_cpu_ready; WARN(1, "DEPT_STOP: " s); \ }) -#define DEPT_INFO_ONCE(s...) pr_warn_once("DEPT_INFO_ONCE: " s) +#define DEPT_INFO_ONCE(s...) pr_warn_once("DEPT_INFO_ONCE: " s) +#define DEPT_INFO(s...) pr_warn("DEPT_INFO: " s) static arch_spinlock_t dept_spin = (arch_spinlock_t)__ARCH_SPIN_LOCK_UNLOCKED; static arch_spinlock_t stage_spin = (arch_spinlock_t)__ARCH_SPIN_LOCK_UNLOCKED; +static arch_spinlock_t dept_pool_spin = (arch_spinlock_t)__ARCH_SPIN_LOCK_UNLOCKED; /* * DEPT internal engine should be careful in using outside functions @@ -264,6 +269,7 @@ static inline bool valid_key(struct dept_key *k) #define OBJECT(id, nr) \ static struct dept_##id spool_##id[nr]; \ +static struct dept_##id rpool_##id[nr]; \ static DEFINE_PER_CPU(struct llist_head, lpool_##id); #include "dept_object.h" #undef OBJECT @@ -272,14 +278,70 @@ struct dept_pool dept_pool[OBJECT_NR] = { #define OBJECT(id, nr) { \ .name = #id, \ .obj_sz = sizeof(struct dept_##id), \ - .obj_nr = ATOMIC_INIT(nr), \ + .obj_nr = nr, \ + .tot_nr = nr, \ + .acc_sz = ATOMIC_INIT(sizeof(spool_##id) + sizeof(rpool_##id)), \ .node_off = offsetof(struct dept_##id, pool_node), \ .spool = spool_##id, \ + .rpool = rpool_##id, \ .lpool = &lpool_##id, }, #include "dept_object.h" #undef OBJECT }; +static void dept_wq_work_fn(struct work_struct *work) +{ + int i; + + for (i = 0; i < OBJECT_NR; i++) { + struct dept_pool *p = dept_pool + i; + int sz = p->tot_nr * p->obj_sz; + void *rpool; + bool need; + + arch_spin_lock(&dept_pool_spin); + need = !p->rpool; + arch_spin_unlock(&dept_pool_spin); + + if (!need) + continue; + + rpool = vmalloc(sz); + + if (!rpool) { + DEPT_STOP("Failed to extend internal resources.\n"); + break; + } + + arch_spin_lock(&dept_pool_spin); + if (!p->rpool) { + p->rpool = rpool; + rpool = NULL; + atomic_add(sz, &p->acc_sz); + } + arch_spin_unlock(&dept_pool_spin); + + if (rpool) + vfree(rpool); + else + DEPT_INFO("Dept object(%s) just got refilled successfully.\n", p->name); + } +} + +static DECLARE_WORK(dept_wq_work, dept_wq_work_fn); + +static void dept_irq_work_fn(struct irq_work *w) +{ + schedule_work(&dept_wq_work); +} + +static DEFINE_IRQ_WORK(dept_irq_work, dept_irq_work_fn); + +static void request_rpool_refill(void) +{ + irq_work_queue(&dept_irq_work); +} + /* * Can use llist no matter whether CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG is * enabled or not because NMI and other contexts in the same CPU never @@ -315,19 +377,31 @@ static void *from_pool(enum object_t t) /* * Try static pool. */ - if (atomic_read(&p->obj_nr) > 0) { - int idx = atomic_dec_return(&p->obj_nr); + arch_spin_lock(&dept_pool_spin); + + if (!p->obj_nr) { + p->spool = p->rpool; + p->obj_nr = p->rpool ? p->tot_nr : 0; + p->rpool = NULL; + request_rpool_refill(); + } + + if (p->obj_nr) { + void *ret; + + p->obj_nr--; + ret = p->spool + (p->obj_nr * p->obj_sz); + arch_spin_unlock(&dept_pool_spin); - if (idx >= 0) - return p->spool + (idx * p->obj_sz); + return ret; } + arch_spin_unlock(&dept_pool_spin); - DEPT_INFO_ONCE("---------------------------------------------\n" - " Some of Dept internal resources are run out.\n" - " Dept might still work if the resources get freed.\n" - " However, the chances are Dept will suffer from\n" - " the lack from now. Needs to extend the internal\n" - " resource pools. Ask max.byungchul.park@gmail.com\n"); + DEPT_INFO("------------------------------------------\n" + " Dept object(%s) is run out.\n" + " Dept is trying to refill the object.\n" + " Nevertheless, if it fails, Dept will stop.\n", + p->name); return NULL; } @@ -3000,8 +3074,8 @@ void __init dept_init(void) pr_info("... DEPT_MAX_ECXT_HELD : %d\n", DEPT_MAX_ECXT_HELD); pr_info("... DEPT_MAX_SUBCLASSES : %d\n", DEPT_MAX_SUBCLASSES); #define OBJECT(id, nr) \ - pr_info("... memory used by %s: %zu KB\n", \ - #id, B2KB(sizeof(struct dept_##id) * nr)); + pr_info("... memory initially used by %s: %zu KB\n", \ + #id, B2KB(sizeof(spool_##id) + sizeof(rpool_##id))); #include "dept_object.h" #undef OBJECT #define HASH(id, bits) \ @@ -3009,6 +3083,6 @@ void __init dept_init(void) #id, B2KB(sizeof(struct hlist_head) * (1 << (bits)))); #include "dept_hash.h" #undef HASH - pr_info("... total memory used by objects and hashs: %zu KB\n", B2KB(mem_total)); + pr_info("... total memory initially used by objects and hashs: %zu KB\n", B2KB(mem_total)); pr_info("... per task memory footprint: %zu bytes\n", sizeof(struct dept_task)); } diff --git a/kernel/dependency/dept_object.h b/kernel/dependency/dept_object.h index 0b7eb16fe9fb..4f936adfa8ee 100644 --- a/kernel/dependency/dept_object.h +++ b/kernel/dependency/dept_object.h @@ -6,8 +6,8 @@ * nr: # of the object that should be kept in the pool. */ -OBJECT(dep, 1024 * 8) -OBJECT(class, 1024 * 8) -OBJECT(stack, 1024 * 32) -OBJECT(ecxt, 1024 * 16) -OBJECT(wait, 1024 * 32) +OBJECT(dep, 1024 * 4 * 2) +OBJECT(class, 1024 * 4) +OBJECT(stack, 1024 * 4 * 8) +OBJECT(ecxt, 1024 * 4 * 2) +OBJECT(wait, 1024 * 4 * 4) diff --git a/kernel/dependency/dept_proc.c b/kernel/dependency/dept_proc.c index 7d61dfbc5865..f07a512b203f 100644 --- a/kernel/dependency/dept_proc.c +++ b/kernel/dependency/dept_proc.c @@ -73,12 +73,10 @@ static int dept_stats_show(struct seq_file *m, void *v) { int r; - seq_puts(m, "Availability in the static pools:\n\n"); + seq_puts(m, "Accumulated amount of memory used by pools:\n\n"); #define OBJECT(id, nr) \ - r = atomic_read(&dept_pool[OBJECT_##id].obj_nr); \ - if (r < 0) \ - r = 0; \ - seq_printf(m, "%s\t%d/%d(%d%%)\n", #id, r, nr, (r * 100) / (nr)); + r = atomic_read(&dept_pool[OBJECT_##id].acc_sz); \ + seq_printf(m, "%s\t%d KB\n", #id, r / 1024); #include "dept_object.h" #undef OBJECT