From patchwork Mon Mar 30 02:32:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 11464491 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7ED2613A4 for ; Mon, 30 Mar 2020 02:34:08 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3FA2D20735 for ; Mon, 30 Mar 2020 02:34:08 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=joelfernandes.org header.i=@joelfernandes.org header.b="Hkkf7fhI" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3FA2D20735 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=joelfernandes.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id EC0F66B007D; Sun, 29 Mar 2020 22:33:46 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id DD5E26B007E; Sun, 29 Mar 2020 22:33:46 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BD7E46B0080; Sun, 29 Mar 2020 22:33:46 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0192.hostedemail.com [216.40.44.192]) by kanga.kvack.org (Postfix) with ESMTP id A2C016B007D for ; Sun, 29 Mar 2020 22:33:46 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id ACC504DBF for ; Mon, 30 Mar 2020 02:33:46 +0000 (UTC) X-FDA: 76650458052.27.sail73_5b297aa7c635c X-Spam-Summary: 2,0,0,157ff7b7abbf8bdd,d41d8cd98f00b204,joel@joelfernandes.org,,RULES_HIT:2:41:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1605:1606:1730:1747:1777:1792:2196:2199:2393:2559:2562:2907:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:4119:4250:4321:4385:5007:6119:6261:6653:6742:7514:7576:7875:7903:8660:10004:11026:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12683:12895:13148:13211:13229:13230:13894:14096:14394:21080:21324:21433:21444:21451:21627:21740:21789:21939:21987:21990:30045:30054:30070,0,RBL:209.85.222.193:@joelfernandes.org:.lbl8.mailshell.net-62.2.0.100 66.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:28,LUA_SUMMARY:none X-HE-Tag: sail73_5b297aa7c635c X-Filterd-Recvd-Size: 8774 Received: from mail-qk1-f193.google.com (mail-qk1-f193.google.com [209.85.222.193]) by imf14.hostedemail.com (Postfix) with ESMTP for ; Mon, 30 Mar 2020 02:33:46 +0000 (UTC) Received: by mail-qk1-f193.google.com with SMTP id h14so17551450qke.5 for ; Sun, 29 Mar 2020 19:33:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ehI/mevr7oGjGlbprsNt2erBqRqo03dz0gk8qK2323I=; b=Hkkf7fhI9UgKMG2bmf1X4tv05OtccwFMkvjxvsFynnH81KwsXpgysqGhe0jyrTAFx8 vtgbEk9Dq27acsmjbGZnmonsgzE0Eal+DdboJtO72c5rgwanQS6VMOhputaGm3E4H/d4 9/eWwrgjl5zhL/Jrdd1K0fyGGeiW0rdvBddNk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ehI/mevr7oGjGlbprsNt2erBqRqo03dz0gk8qK2323I=; b=KoERp73i7dY+YL7r6+vuJ/h7PzftstvrFja4b6dmxKBHaGxx9qTD/XgyMTL4OQfsl8 etllYsg56LU4FprcylJKHjHleNhw86Skb2iY/kDqWzQbuLhEkjFkKBWzhkRC2wGzOzzm fGz508B/q5fNGcDshGqlR1kz4jKh74en8PrDrCXj8n3FoMa0qaKO3kO5J/Y8FHtW918S y1bkEmUEW6U+OAp9gu6gniOhC9zL6WQzBsGre6FcPG9kdZCdNrRbaA9L5oN/Iz/YktKa 9+G1cFA0YWkUEi+MdE91tf+XhOgqQza46Gm2+EcPTk66KENTYMJxgCQXDHYk3Mo1Xqxe BEUA== X-Gm-Message-State: ANhLgQ2hugYPyCbLxWzeF5bDgorlP6AxqMqlLaTMqlMTYAliLWkw9w15 5qZc14ABgcHfUfC44URzlUeuLA== X-Google-Smtp-Source: ADFU+vsEu6bs9H+nTRyqOkzVOqwut6t53PnZdb4J1el0BLUwgt9JHzOjhj/hAdHYb48agOxgcw9Hfw== X-Received: by 2002:a37:2fc3:: with SMTP id v186mr9919220qkh.311.1585535625444; Sun, 29 Mar 2020 19:33:45 -0700 (PDT) Received: from joelaf.cam.corp.google.com ([2620:15c:6:12:9c46:e0da:efbf:69cc]) by smtp.gmail.com with ESMTPSA id q15sm10030625qtj.83.2020.03.29.19.33.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 29 Mar 2020 19:33:45 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: "Uladzislau Rezki (Sony)" , Joel Fernandes , Andrew Morton , Ingo Molnar , Josh Triplett , Lai Jiangshan , linux-mm@kvack.org, Mathieu Desnoyers , "Paul E. McKenney" , "Rafael J. Wysocki" , rcu@vger.kernel.org, Steven Rostedt Subject: [PATCH 14/18] rcu/tiny: Support reclaim for head-less object Date: Sun, 29 Mar 2020 22:32:44 -0400 Message-Id: <20200330023248.164994-15-joel@joelfernandes.org> X-Mailer: git-send-email 2.26.0.rc2.310.g2932bb562d-goog In-Reply-To: <20200330023248.164994-1-joel@joelfernandes.org> References: <20200330023248.164994-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Uladzislau Rezki (Sony)" Make a kvfree_call_rcu() function to support head-less freeing. Same as for tree-RCU, for such purpose we store pointers in array. SLAB and vmalloc ptrs. are mixed and coexist together. Under high memory pressure it can be that maintaining of arrays becomes impossible. Objects with an rcu_head are released via call_rcu(). When it comes to the head-less variant, the kvfree() call is directly inlined, i.e. we do the same as for tree-RCU: a) wait until a grace period has elapsed; b) direct inlining of the kvfree() call. Thus the current context has to follow might_sleep() annotation. Also please note that for tiny-RCU any call of synchronize_rcu() is actually a quiescent state, therefore (a) does nothing. Reviewed-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki (Sony) Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tiny.c | 157 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 156 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c index 508c82faa45c3..b1c31a935db93 100644 --- a/kernel/rcu/tiny.c +++ b/kernel/rcu/tiny.c @@ -40,6 +40,29 @@ static struct rcu_ctrlblk rcu_ctrlblk = { .curtail = &rcu_ctrlblk.rcucblist, }; +/* Can be common with tree-RCU. */ +#define KVFREE_DRAIN_JIFFIES (HZ / 50) + +/* Can be common with tree-RCU. */ +struct kvfree_rcu_bulk_data { + unsigned long nr_records; + struct kvfree_rcu_bulk_data *next; + void *records[]; +}; + +/* Can be common with tree-RCU. */ +#define KVFREE_BULK_MAX_ENTR \ + ((PAGE_SIZE - sizeof(struct kvfree_rcu_bulk_data)) / sizeof(void *)) + +static struct kvfree_rcu_bulk_data *kvhead; +static struct kvfree_rcu_bulk_data *kvhead_free; +static struct kvfree_rcu_bulk_data *kvcache; + +static DEFINE_STATIC_KEY_FALSE(rcu_init_done); +static struct delayed_work monitor_work; +static struct rcu_work rcu_work; +static bool monitor_todo; + void rcu_barrier(void) { wait_rcu_gp(call_rcu); @@ -177,9 +200,137 @@ void call_rcu(struct rcu_head *head, rcu_callback_t func) } EXPORT_SYMBOL_GPL(call_rcu); +static inline bool +kvfree_call_rcu_add_ptr_to_bulk(void *ptr) +{ + struct kvfree_rcu_bulk_data *bnode; + + if (!kvhead || kvhead->nr_records == KVFREE_BULK_MAX_ENTR) { + bnode = xchg(&kvcache, NULL); + if (!bnode) + bnode = (struct kvfree_rcu_bulk_data *) + __get_free_page(GFP_NOWAIT | __GFP_NOWARN); + + if (unlikely(!bnode)) + return false; + + /* Initialize the new block. */ + bnode->nr_records = 0; + bnode->next = kvhead; + + /* Attach it to the bvhead. */ + kvhead = bnode; + } + + /* Done. */ + kvhead->records[kvhead->nr_records++] = ptr; + return true; +} + +static void +kvfree_rcu_work(struct work_struct *work) +{ + struct kvfree_rcu_bulk_data *kvhead_tofree, *next; + unsigned long flags; + int i; + + local_irq_save(flags); + kvhead_tofree = kvhead_free; + kvhead_free = NULL; + local_irq_restore(flags); + + /* Reclaim process. */ + for (; kvhead_tofree; kvhead_tofree = next) { + next = kvhead_tofree->next; + + for (i = 0; i < kvhead_tofree->nr_records; i++) { + debug_rcu_head_unqueue((struct rcu_head *) + kvhead_tofree->records[i]); + kvfree(kvhead_tofree->records[i]); + } + + if (cmpxchg(&kvcache, NULL, kvhead_tofree)) + free_page((unsigned long) kvhead_tofree); + } +} + +static inline bool +queue_kvfree_rcu_work(void) +{ + /* Check if the free channel is available. */ + if (kvhead_free) + return false; + + kvhead_free = kvhead; + kvhead = NULL; + + /* + * Queue the job for memory reclaim after GP. + */ + queue_rcu_work(system_wq, &rcu_work); + return true; +} + +static void kvfree_rcu_monitor(struct work_struct *work) +{ + unsigned long flags; + bool queued; + + local_irq_save(flags); + queued = queue_kvfree_rcu_work(); + if (queued) + /* Success. */ + monitor_todo = false; + local_irq_restore(flags); + + /* + * If previous RCU reclaim process is still in progress, + * schedule the work one more time to try again later. + */ + if (monitor_todo) + schedule_delayed_work(&monitor_work, + KVFREE_DRAIN_JIFFIES); +} + void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) { - call_rcu(head, func); + unsigned long flags; + bool success; + void *ptr; + + if (head) { + ptr = (void *) head - (unsigned long) func; + } else { + might_sleep(); + ptr = (void *) func; + } + + if (debug_rcu_head_queue(ptr)) { + /* Probable double free, just leak. */ + WARN_ONCE(1, "%s(): Double-freed call. rcu_head %p\n", + __func__, head); + return; + } + + local_irq_save(flags); + success = kvfree_call_rcu_add_ptr_to_bulk(ptr); + if (static_branch_likely(&rcu_init_done)) { + if (success && !monitor_todo) { + monitor_todo = true; + schedule_delayed_work(&monitor_work, + KVFREE_DRAIN_JIFFIES); + } + } + local_irq_restore(flags); + + if (!success) { + if (!head) { + synchronize_rcu(); + kvfree(ptr); + } else { + call_rcu(head, func); + } + } } EXPORT_SYMBOL_GPL(kvfree_call_rcu); @@ -188,4 +339,8 @@ void __init rcu_init(void) open_softirq(RCU_SOFTIRQ, rcu_process_callbacks); rcu_early_boot_tests(); srcu_init(); + + INIT_DELAYED_WORK(&monitor_work, kvfree_rcu_monitor); + INIT_RCU_WORK(&rcu_work, kvfree_rcu_work); + static_branch_enable(&rcu_init_done); }