From patchwork Tue Apr 28 20:58:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11515551 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6770714DD for ; Tue, 28 Apr 2020 20:59:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2547C2137B for ; Tue, 28 Apr 2020 20:59:29 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="IiSiQD7I" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2547C2137B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4A6598E0006; Tue, 28 Apr 2020 16:59:28 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 456DC8E0001; Tue, 28 Apr 2020 16:59:28 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 36DA48E0006; Tue, 28 Apr 2020 16:59:28 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0120.hostedemail.com [216.40.44.120]) by kanga.kvack.org (Postfix) with ESMTP id 20CA58E0001 for ; Tue, 28 Apr 2020 16:59:28 -0400 (EDT) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id D9352180AD802 for ; Tue, 28 Apr 2020 20:59:27 +0000 (UTC) X-FDA: 76758479574.23.wall25_5297080620663 X-Spam-Summary: 2,0,0,761709f2bed6907e,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:2:41:69:355:379:541:800:960:965:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1606:1730:1747:1777:1792:2194:2196:2199:2200:2393:2559:2562:2691:2693:2898:3138:3139:3140:3141:3142:3355:3865:3866:3867:3870:3871:3874:4119:4250:4321:4385:4390:4395:5007:6117:6119:6261:6653:7514:7576:7903:9040:9413:10004:11026:11473:11658:11914:12043:12296:12297:12438:12517:12519:12555:12683:12895:13149:13230:13846:13894:14394:14687:21080:21433:21444:21451:21627:21666:21740:21990:30012:30054:30070,0,RBL:209.85.208.194:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: wall25_5297080620663 X-Filterd-Recvd-Size: 8381 Received: from mail-lj1-f194.google.com (mail-lj1-f194.google.com [209.85.208.194]) by imf41.hostedemail.com (Postfix) with ESMTP for ; Tue, 28 Apr 2020 20:59:27 +0000 (UTC) Received: by mail-lj1-f194.google.com with SMTP id j3so209000ljg.8 for ; Tue, 28 Apr 2020 13:59:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=20juLYXIkeuaSxH0ZOfDE2aYGH0+hGPgd5TgVWNgKV0=; b=IiSiQD7Iu4uhe8rSKiFsy+4MM1PYZAifb/4X08Jtk+cZrudizBa+RKHTmspM1N+54J zqJhTvKu/5RQ6kvBm1U2XnFtEFoT5WK+zwPhKmJkjxoJF7GUThJw2k4x9UKsHyN3uVGr t1obO+H+JCn5et6c+tkfu4seuB0gVaBP66MA3RnOVoiWWR6yNpPVLWRXguro8HTAimgy dWMKPS6AWlRiBSdUP8vLP6smxvt8NoPstE7Me+W9nAo+jVxPe1UXCLY3/kLTgCPzwEGN UqOgOltC5P1qNzlwU+eozjY3EfiF2YNtHyaDio7PldqVoXdf4YBXbYJs+pJcbBW0vSZH qZLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=20juLYXIkeuaSxH0ZOfDE2aYGH0+hGPgd5TgVWNgKV0=; b=Sloq4MMO2RavGsWapqMpASrZS1W4lD3dwK1TQ0f5KMF1U3c3EjVF9GzSZ79y0o7rjX Tlp3X5QEy4KaLkozCfJRuKjqss8lr9wx5D/9SUrLbLKcNiV9FEeUQXrog/oCasip8pSQ 6TKuo+nbcXIrqogrMlQBcrAECTvIOulr1VW5Pyfl0g29i9AZro0t0IzEeiGZj2WBzreV K/S9GNX0e6/XrlqmgG7Tpe5bwnEWG5iUSHjcF7CPPqm4VWRXxp9QIni2N3JqApHt8jTR AQCDBCrGUpzxbqIkWn5soLHns9b51NZKi9XjhmQaWD22iY83RdP9roWiP2RuvELk4AhB +QRQ== X-Gm-Message-State: AGi0PuaEpCNmOWlbKt++dx6PjlMkzqrdkf6oCXYRtGiilvMXHCIMASKq ATsw4gq31oV7nDtws46UT9U= X-Google-Smtp-Source: APiQypIzU/HDHdtBEfR5w4aVQIlns0HcLgmvrgZUF/xNsiisHjiPOjyV2oZEMK3gwdyuCi4Uq6EsKQ== X-Received: by 2002:a19:7008:: with SMTP id h8mr20057304lfc.43.1588107566139; Tue, 28 Apr 2020 13:59:26 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id z21sm295483ljh.42.2020.04.28.13.59.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Apr 2020 13:59:23 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko , bigeasy@linutronix.de Subject: [PATCH 01/24] rcu/tree: Keep kfree_rcu() awake during lock contention Date: Tue, 28 Apr 2020 22:58:40 +0200 Message-Id: <20200428205903.61704-2-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200428205903.61704-1-urezki@gmail.com> References: <20200428205903.61704-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Joel Fernandes (Google)" On PREEMPT_RT kernels, contending on the krcp spinlock can cause sleeping as on these kernels, the spinlock is converted to an rt-mutex. To prevent breakage of possible usage of kfree_rcu() now or in the future, make use of raw spinlocks which are not subject to such conversions. Vetting all code paths, there is no reason to believe that the raw spinlock will be held for long time so PREEMPT_RT should not suffer from lengthy acquirals of the lock. Cc: bigeasy@linutronix.de Cc: Uladzislau Rezki Reviewed-by: Uladzislau Rezki Signed-off-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki (Sony) --- kernel/rcu/tree.c | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index f288477ee1c2..cf68d3d9f5b8 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2905,7 +2905,7 @@ struct kfree_rcu_cpu { struct kfree_rcu_bulk_data *bhead; struct kfree_rcu_bulk_data *bcached; struct kfree_rcu_cpu_work krw_arr[KFREE_N_BATCHES]; - spinlock_t lock; + raw_spinlock_t lock; struct delayed_work monitor_work; bool monitor_todo; bool initialized; @@ -2939,12 +2939,12 @@ static void kfree_rcu_work(struct work_struct *work) krwp = container_of(to_rcu_work(work), struct kfree_rcu_cpu_work, rcu_work); krcp = krwp->krcp; - spin_lock_irqsave(&krcp->lock, flags); + raw_spin_lock_irqsave(&krcp->lock, flags); head = krwp->head_free; krwp->head_free = NULL; bhead = krwp->bhead_free; krwp->bhead_free = NULL; - spin_unlock_irqrestore(&krcp->lock, flags); + raw_spin_unlock_irqrestore(&krcp->lock, flags); /* "bhead" is now private, so traverse locklessly. */ for (; bhead; bhead = bnext) { @@ -3047,14 +3047,14 @@ static inline void kfree_rcu_drain_unlock(struct kfree_rcu_cpu *krcp, krcp->monitor_todo = false; if (queue_kfree_rcu_work(krcp)) { // Success! Our job is done here. - spin_unlock_irqrestore(&krcp->lock, flags); + raw_spin_unlock_irqrestore(&krcp->lock, flags); return; } // Previous RCU batch still in progress, try again later. krcp->monitor_todo = true; schedule_delayed_work(&krcp->monitor_work, KFREE_DRAIN_JIFFIES); - spin_unlock_irqrestore(&krcp->lock, flags); + raw_spin_unlock_irqrestore(&krcp->lock, flags); } /* @@ -3067,11 +3067,11 @@ static void kfree_rcu_monitor(struct work_struct *work) struct kfree_rcu_cpu *krcp = container_of(work, struct kfree_rcu_cpu, monitor_work.work); - spin_lock_irqsave(&krcp->lock, flags); + raw_spin_lock_irqsave(&krcp->lock, flags); if (krcp->monitor_todo) kfree_rcu_drain_unlock(krcp, flags); else - spin_unlock_irqrestore(&krcp->lock, flags); + raw_spin_unlock_irqrestore(&krcp->lock, flags); } static inline bool @@ -3142,7 +3142,7 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func) local_irq_save(flags); // For safely calling this_cpu_ptr(). krcp = this_cpu_ptr(&krc); if (krcp->initialized) - spin_lock(&krcp->lock); + raw_spin_lock(&krcp->lock); // Queue the object but don't yet schedule the batch. if (debug_rcu_head_queue(head)) { @@ -3173,7 +3173,7 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func) unlock_return: if (krcp->initialized) - spin_unlock(&krcp->lock); + raw_spin_unlock(&krcp->lock); local_irq_restore(flags); } EXPORT_SYMBOL_GPL(kfree_call_rcu); @@ -3205,11 +3205,11 @@ kfree_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) struct kfree_rcu_cpu *krcp = per_cpu_ptr(&krc, cpu); count = krcp->count; - spin_lock_irqsave(&krcp->lock, flags); + raw_spin_lock_irqsave(&krcp->lock, flags); if (krcp->monitor_todo) kfree_rcu_drain_unlock(krcp, flags); else - spin_unlock_irqrestore(&krcp->lock, flags); + raw_spin_unlock_irqrestore(&krcp->lock, flags); sc->nr_to_scan -= count; freed += count; @@ -3236,15 +3236,15 @@ void __init kfree_rcu_scheduler_running(void) for_each_online_cpu(cpu) { struct kfree_rcu_cpu *krcp = per_cpu_ptr(&krc, cpu); - spin_lock_irqsave(&krcp->lock, flags); + raw_spin_lock_irqsave(&krcp->lock, flags); if (!krcp->head || krcp->monitor_todo) { - spin_unlock_irqrestore(&krcp->lock, flags); + raw_spin_unlock_irqrestore(&krcp->lock, flags); continue; } krcp->monitor_todo = true; schedule_delayed_work_on(cpu, &krcp->monitor_work, KFREE_DRAIN_JIFFIES); - spin_unlock_irqrestore(&krcp->lock, flags); + raw_spin_unlock_irqrestore(&krcp->lock, flags); } } @@ -4140,7 +4140,7 @@ static void __init kfree_rcu_batch_init(void) for_each_possible_cpu(cpu) { struct kfree_rcu_cpu *krcp = per_cpu_ptr(&krc, cpu); - spin_lock_init(&krcp->lock); + raw_spin_lock_init(&krcp->lock); for (i = 0; i < KFREE_N_BATCHES; i++) { INIT_RCU_WORK(&krcp->krw_arr[i].rcu_work, kfree_rcu_work); krcp->krw_arr[i].krcp = krcp; From patchwork Tue Apr 28 20:58:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11515555 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2F09C1392 for ; Tue, 28 Apr 2020 20:59:35 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E672320B80 for ; Tue, 28 Apr 2020 20:59:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="lUSGgt+C" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E672320B80 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2AB738E0008; Tue, 28 Apr 2020 16:59:34 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 25AFD8E0007; Tue, 28 Apr 2020 16:59:34 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 170EB8E0008; Tue, 28 Apr 2020 16:59:34 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0077.hostedemail.com [216.40.44.77]) by kanga.kvack.org (Postfix) with ESMTP id F388F8E0001 for ; Tue, 28 Apr 2020 16:59:33 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 4946652CC for ; Tue, 28 Apr 2020 20:59:32 +0000 (UTC) X-FDA: 76758479784.04.steam36_53370d79a6553 X-Spam-Summary: 2,0,0,9ad538916a418998,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:41:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1541:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:3138:3139:3140:3141:3142:3353:3865:3866:3867:3868:3870:3871:3872:3874:4321:4385:5007:6261:6653:7514:7576:9413:10004:11026:11473:11658:11914:12043:12296:12297:12438:12517:12519:12555:12895:13069:13311:13357:13894:14096:14181:14384:14394:14687:14721:21080:21444:21451:21627:21666:21740:21990:30054,0,RBL:209.85.208.196:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.18.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:27,LUA_SUMMARY:none X-HE-Tag: steam36_53370d79a6553 X-Filterd-Recvd-Size: 5219 Received: from mail-lj1-f196.google.com (mail-lj1-f196.google.com [209.85.208.196]) by imf06.hostedemail.com (Postfix) with ESMTP for ; Tue, 28 Apr 2020 20:59:31 +0000 (UTC) Received: by mail-lj1-f196.google.com with SMTP id y4so215386ljn.7 for ; Tue, 28 Apr 2020 13:59:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=VbhSWD62l8JNzTfQeyAoSUlyMJCSOqg1BrQPjTRkMk4=; b=lUSGgt+CcioWbfGLfJs6ugC+Qsrh1jSAzKgYq+RqTEJ21PFv8tQASSylCGa3PsVbnY Qbu76q5eTVYVtQnCst4OOtW3BlwytMXJWb+FeiUODQpsMPCiLZgDtu394oLx3qGFh7ro CUV0dDmlU2XxAMYHQI4Iaz5+oMKXtr10rhPCmkBZRvY5UTTkNFQPY9dtPCQASLAKSgpw k79dHaNBrymsYKNqoM41Bh4gmnO22eY5dnHBGtSZ2n8G2cPbYpP+IP3MuBpoa2Vbj82P SRpq3tfsb9HM1/p3VUJqqfJs4WInmGwQPVyQhTibtyUcbpA1vCW1XkZURvjniAH8QJSV JoEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=VbhSWD62l8JNzTfQeyAoSUlyMJCSOqg1BrQPjTRkMk4=; b=QXn+S88+uZv/L+DvjqRsAD7jpSipsRcGFHoV3HfipdI4xD+F/w5UenQ2rH+TUJMhGE OQ8hfUDeBnfecri9HlhZWC1QevfOcG22sCDTUYa04IjJy/DX3b5NHpMFLacYUymkeU/S aW2e6v34JjT+Y/2mu7WkLPsEfyfitenMjWri+RiBLR+y0a7qCL+mwwD3nZVsu0xbFiOS aEfVZb54o7OW24KvQz2hKtbQkFnAt9G0f82uHFi8kJh4yIjCn3OXmvi/wYy4cQ5Si8il m8AndoZOUsJDXMVq42QHvkwNM2bKorc96KgvktKHcFBwDo3+spEA9F3LNWePe8ambqz+ MCCw== X-Gm-Message-State: AGi0PuZYX37Nbl1YpCNvwjq1UnkHSlwmW9PuSUAjH4yr6hA9cxufgKw9 Sohs0ve1ECi9mBBGjWSukcw= X-Google-Smtp-Source: APiQypKKGOVXhlmChW4BhBJ3iFPhuFW9qb1gS4bvnz1pm04t1mdMolucRbZxXE4S2RJ5qLuMNAg/SQ== X-Received: by 2002:a05:651c:28c:: with SMTP id b12mr18386730ljo.167.1588107570458; Tue, 28 Apr 2020 13:59:30 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id z21sm295483ljh.42.2020.04.28.13.59.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Apr 2020 13:59:26 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko , Sebastian Andrzej Siewior Subject: [PATCH 02/24] rcu/tree: Skip entry into the page allocator for PREEMPT_RT Date: Tue, 28 Apr 2020 22:58:41 +0200 Message-Id: <20200428205903.61704-3-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200428205903.61704-1-urezki@gmail.com> References: <20200428205903.61704-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Joel Fernandes (Google)" To keep kfree_rcu() path working on raw non-preemptible sections, prevent the optional entry into the allocator as it uses sleeping locks. In fact, even if the caller of kfree_rcu() is preemptible, this path still is not, as krcp->lock is a raw spinlock as done in previous patches. With additional page pre-allocation in the works, hitting this return is going to be much less likely soon so just prevent it for now so that PREEMPT_RT does not break. Note that page allocation here is an optimization and skipping it still makes kfree_rcu() work. Cc: Sebastian Andrzej Siewior Reviewed-by: Uladzislau Rezki Co-developed-by: Uladzislau Rezki Signed-off-by: Uladzislau Rezki Signed-off-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki (Sony) --- kernel/rcu/tree.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index cf68d3d9f5b8..cd61649e1b00 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3092,6 +3092,18 @@ kfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, if (!bnode) { WARN_ON_ONCE(sizeof(struct kfree_rcu_bulk_data) > PAGE_SIZE); + /* + * To keep this path working on raw non-preemptible + * sections, prevent the optional entry into the + * allocator as it uses sleeping locks. In fact, even + * if the caller of kfree_rcu() is preemptible, this + * path still is not, as krcp->lock is a raw spinlock. + * With additional page pre-allocation in the works, + * hitting this return is going to be much less likely. + */ + if (IS_ENABLED(CONFIG_PREEMPT_RT)) + return false; + bnode = (struct kfree_rcu_bulk_data *) __get_free_page(GFP_NOWAIT | __GFP_NOWARN); } From patchwork Tue Apr 28 20:58:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11515557 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 007801392 for ; Tue, 28 Apr 2020 20:59:37 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C1CCD21744 for ; Tue, 28 Apr 2020 20:59:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="OomEXmvE" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C1CCD21744 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4C8DC8E0001; Tue, 28 Apr 2020 16:59:34 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 371438E0009; Tue, 28 Apr 2020 16:59:34 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2155E8E0001; Tue, 28 Apr 2020 16:59:34 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0083.hostedemail.com [216.40.44.83]) by kanga.kvack.org (Postfix) with ESMTP id 038828E0007 for ; Tue, 28 Apr 2020 16:59:33 -0400 (EDT) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 81DC12493 for ; Tue, 28 Apr 2020 20:59:33 +0000 (UTC) X-FDA: 76758479826.06.train47_5367c011b3055 X-Spam-Summary: 2,0,0,1dbae68d9f68a32f,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:41:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1542:1711:1730:1747:1777:1792:2194:2196:2199:2200:2393:2559:2562:3138:3139:3140:3141:3142:3354:3865:3866:3867:3870:3871:3872:4250:4321:4385:5007:6117:6119:6261:6653:7514:7576:7903:8660:9040:9413:10004:11026:11473:11658:11914:12043:12294:12296:12297:12438:12517:12519:12555:12895:13148:13230:13894:14181:14394:14687:14721:21080:21444:21451:21627:21666:21740:21990:30012:30054,0,RBL:209.85.208.193:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.18.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: train47_5367c011b3055 X-Filterd-Recvd-Size: 5954 Received: from mail-lj1-f193.google.com (mail-lj1-f193.google.com [209.85.208.193]) by imf45.hostedemail.com (Postfix) with ESMTP for ; Tue, 28 Apr 2020 20:59:33 +0000 (UTC) Received: by mail-lj1-f193.google.com with SMTP id g4so256193ljl.2 for ; Tue, 28 Apr 2020 13:59:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tNVlKJMZgR8fQGfjNMGlZb0HEXGPqeGS+UCAapReQEQ=; b=OomEXmvEimxM1FQU4RZxUQQ6SUm/0rWP04XnM3tyK2N2EVh5hEe+B/wMbpSMFBO2a9 zfvlE9oG69hOjkwDFfbhn04b0dqQhaPQy92gtamKdDddFXpGJDjR9LDP+4SvN088uTqd NN6OYVKWiMckLURyeMjSjqHjcvw0JBhne90WJ384rFV9oE0z5/6+C/NnQNc1krfDkLA7 DJxLLSbGJOr+kvsKBrxkNkuec+OX1hgRGUPxQbLAUysbTLkm0JAI2Uyu/sWJAuIKOWgd F8mQ4JZhQa2Fqd1JAKFuF9bJC8vT4eyTlM9tV9lfACk02DBS0Ayjt9+uDFljqVo1Ne2T BJsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tNVlKJMZgR8fQGfjNMGlZb0HEXGPqeGS+UCAapReQEQ=; b=a+HfEm+DFQGJRhiF/g8R910xzCnztSUeckZeMghlwRdGiF2ObHELs1iI2QgpHwpnZD h1DJqzj+uIqF6clKCg/eW9wCq65wqc2UaZpE8YVwIej+VjDNrDS9iQZyjBGfkfJdme1K ADeemJ4J/LHNBJ2Z2wyCKcP1igEqCBmtJSUo9VabZLEhtaZRePuNgqezFISQ+8/47RV3 kbjPqpBDZdCy9JHBIgnyztvxfJnmeuBH5a3DCX5QBoIpPa//zGWxVStHav1ZHPvKaoEP aFceaACg5IMGU9bm8NuCdXyAxGBQt5qzF+oDA6KM7+ZeHzdt1vB75PPRq8VwtAMXdeET C/RA== X-Gm-Message-State: AGi0PuamoaWSRztO0lXiB2FSiDTSpboM42N5ZHGnMV5KrwP2kydqE84c CCbwfEdXH65bjRYSK0D08WU= X-Google-Smtp-Source: APiQypLM6AK8jKHmA2nJzuwDkDwYvmAhNNN+CNoOBiRU/u2OgnsIOG3G2+TrSv7A9npS7PRZThXzUA== X-Received: by 2002:a19:f614:: with SMTP id x20mr20240048lfe.84.1588107571728; Tue, 28 Apr 2020 13:59:31 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id z21sm295483ljh.42.2020.04.28.13.59.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Apr 2020 13:59:31 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 03/24] rcu/tree: Use consistent style for comments Date: Tue, 28 Apr 2020 22:58:42 +0200 Message-Id: <20200428205903.61704-4-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200428205903.61704-1-urezki@gmail.com> References: <20200428205903.61704-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Joel Fernandes (Google)" Simple clean up of comments in kfree_rcu() code to keep it consistent with majority of commenting styles. Reviewed-by: Uladzislau Rezki Signed-off-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki (Sony) --- kernel/rcu/tree.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index cd61649e1b00..1487af8e11e8 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3043,15 +3043,15 @@ static inline bool queue_kfree_rcu_work(struct kfree_rcu_cpu *krcp) static inline void kfree_rcu_drain_unlock(struct kfree_rcu_cpu *krcp, unsigned long flags) { - // Attempt to start a new batch. + /* Attempt to start a new batch. */ krcp->monitor_todo = false; if (queue_kfree_rcu_work(krcp)) { - // Success! Our job is done here. + /* Success! Our job is done here. */ raw_spin_unlock_irqrestore(&krcp->lock, flags); return; } - // Previous RCU batch still in progress, try again later. + /* Previous RCU batch still in progress, try again later. */ krcp->monitor_todo = true; schedule_delayed_work(&krcp->monitor_work, KFREE_DRAIN_JIFFIES); raw_spin_unlock_irqrestore(&krcp->lock, flags); @@ -3151,14 +3151,14 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func) unsigned long flags; struct kfree_rcu_cpu *krcp; - local_irq_save(flags); // For safely calling this_cpu_ptr(). + local_irq_save(flags); /* For safely calling this_cpu_ptr(). */ krcp = this_cpu_ptr(&krc); if (krcp->initialized) raw_spin_lock(&krcp->lock); - // Queue the object but don't yet schedule the batch. + /* Queue the object but don't yet schedule the batch. */ if (debug_rcu_head_queue(head)) { - // Probable double kfree_rcu(), just leak. + /* Probable double kfree_rcu(), just leak. */ WARN_ONCE(1, "%s(): Double-freed call. rcu_head %p\n", __func__, head); goto unlock_return; @@ -3176,7 +3176,7 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func) WRITE_ONCE(krcp->count, krcp->count + 1); - // Set timer to drain after KFREE_DRAIN_JIFFIES. + /* Set timer to drain after KFREE_DRAIN_JIFFIES. */ if (rcu_scheduler_active == RCU_SCHEDULER_RUNNING && !krcp->monitor_todo) { krcp->monitor_todo = true; @@ -3722,7 +3722,7 @@ int rcutree_offline_cpu(unsigned int cpu) rcutree_affinity_setting(cpu, cpu); - // nohz_full CPUs need the tick for stop-machine to work quickly + /* nohz_full CPUs need the tick for stop-machine to work quickly */ tick_dep_set(TICK_DEP_BIT_RCU); return 0; } From patchwork Tue Apr 28 20:58:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11515559 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2AD651392 for ; Tue, 28 Apr 2020 20:59:39 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EC93320B80 for ; Tue, 28 Apr 2020 20:59:38 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="gV/CkO5+" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EC93320B80 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3E64D8E0009; Tue, 28 Apr 2020 16:59:35 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3457E8E0007; Tue, 28 Apr 2020 16:59:35 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 199C18E0009; Tue, 28 Apr 2020 16:59:35 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0058.hostedemail.com [216.40.44.58]) by kanga.kvack.org (Postfix) with ESMTP id 003088E0007 for ; Tue, 28 Apr 2020 16:59:34 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id B5856181AC9CB for ; Tue, 28 Apr 2020 20:59:34 +0000 (UTC) X-FDA: 76758479868.04.chain75_5394c7f36bc3e X-Spam-Summary: 2,0,0,4cb808df23000f8a,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:41:355:379:541:800:960:965:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:3138:3139:3140:3141:3142:3352:3865:3866:3867:3868:3870:3871:3872:3874:4321:4385:4390:4395:5007:6261:6653:7514:7903:9413:10004:11026:11658:11914:12043:12294:12296:12297:12438:12517:12519:12555:12895:13069:13161:13229:13311:13357:13868:13894:14096:14181:14384:14394:14687:14721:21080:21433:21444:21451:21627:21666:21740:21990:30045:30054,0,RBL:209.85.208.193:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: chain75_5394c7f36bc3e X-Filterd-Recvd-Size: 4664 Received: from mail-lj1-f193.google.com (mail-lj1-f193.google.com [209.85.208.193]) by imf18.hostedemail.com (Postfix) with ESMTP for ; Tue, 28 Apr 2020 20:59:34 +0000 (UTC) Received: by mail-lj1-f193.google.com with SMTP id h4so186689ljg.12 for ; Tue, 28 Apr 2020 13:59:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=f76tELWeD+Cq+rZ2nxZRf+Lr/87HEoPSQFMjularl7s=; b=gV/CkO5+yDyR2BtTpL6acO8ksCDaXgZ/Mdr2KfSt7vXdkvrp4vOh/EADiQ+zywJynU SW+xHfk1uQ7+y0BZmXWDIBpHOcaYmayvBGUtylXj+cpbhBc7CTIYYMNu+GmNXzuoNrcK LQQL+TKgLOAdzqkzJdIXnyxJl2ESUoKsAUj7M8ZORs+qdnsrzVtbAW4m+j9OmrKINiQL kMqQ9fkjfLUv92lX/AsSVbgu57rLtx+1xZzxIXBJqv0tzjziR3KKb4pVigwpCSfoe2Sh 4ziWCwAOPAaSd2ZobdNxQWOKAXSX/P3uF0WYwAfizNleUDYBDPtSggXR9NgJBbf/lPNL 8O1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=f76tELWeD+Cq+rZ2nxZRf+Lr/87HEoPSQFMjularl7s=; b=MYNnZK6sOQg7gpklqodM16yBGiXosd47tducww67khfIWhLAmwB4wSVOaCaKuFEtas NapaoDU7AfIie/lQ/9F5T2sVd83RrtmY6gtNI95ICBK7X7V/ZNRd/fINBQ0YOCVLEvHS 4c9LrKt34kG/41DVTu/DiMxkGrLP1OnVnp7yW04DyJ5qFrYhy22ZHFOyFsrjGHmiu3OU +SjZOu7PXtjvX9SHrNa16P5jci3iHy9OT9HuLq4n3Caj44kXKZm9v7v0WcUHhR89Jnqw yjddmQSNDoJp1iCXl3xwQH5e8BIHDvNrlZEknWMjkb4QP01MxL2cclGnECnOZFvP7cn7 iirg== X-Gm-Message-State: AGi0PuY0WqVplye0QHO+meFjO3qzDkgfzbeG9lsP3DgoWGra40QH837H xgj64xdvaKwdY4hKb3lxDtg= X-Google-Smtp-Source: APiQypIyLU6bchTrGUBspJZC7W9x7kj6ckHQy61yTYUJR3oMfNtrovTxVulDU2ZLcnOnbM6O4XdWEQ== X-Received: by 2002:a2e:909a:: with SMTP id l26mr18315311ljg.262.1588107572992; Tue, 28 Apr 2020 13:59:32 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id z21sm295483ljh.42.2020.04.28.13.59.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Apr 2020 13:59:32 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 04/24] rcu/tree: Repeat the monitor if any free channel is busy Date: Tue, 28 Apr 2020 22:58:43 +0200 Message-Id: <20200428205903.61704-5-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200428205903.61704-1-urezki@gmail.com> References: <20200428205903.61704-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: It can be that one of the channels can not be detached due to the fact that its free channel and previous data has not been processed yet. From the other hand another channel can be successfully detached causing the monitor work to stop. To prevent that, if there are any channels are still in pending state after a detach attempt, just reschedule the monitor work. Signed-off-by: Uladzislau Rezki (Sony) --- kernel/rcu/tree.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 1487af8e11e8..0762ac06f0b7 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2995,7 +2995,7 @@ static void kfree_rcu_work(struct work_struct *work) static inline bool queue_kfree_rcu_work(struct kfree_rcu_cpu *krcp) { struct kfree_rcu_cpu_work *krwp; - bool queued = false; + bool repeat = false; int i; lockdep_assert_held(&krcp->lock); @@ -3033,11 +3033,14 @@ static inline bool queue_kfree_rcu_work(struct kfree_rcu_cpu *krcp) * been detached following each other, one by one. */ queue_rcu_work(system_wq, &krwp->rcu_work); - queued = true; } + + /* Repeat if any "free" corresponding channel is still busy. */ + if (krcp->bhead || krcp->head) + repeat = true; } - return queued; + return !repeat; } static inline void kfree_rcu_drain_unlock(struct kfree_rcu_cpu *krcp, From patchwork Tue Apr 28 20:58:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11515561 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 52D1414DD for ; Tue, 28 Apr 2020 20:59:41 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 10AD121707 for ; Tue, 28 Apr 2020 20:59:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="fBNRUevw" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 10AD121707 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 62ED68E000A; Tue, 28 Apr 2020 16:59:36 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5B8ED8E0007; Tue, 28 Apr 2020 16:59:36 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4D08B8E000A; Tue, 28 Apr 2020 16:59:36 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0245.hostedemail.com [216.40.44.245]) by kanga.kvack.org (Postfix) with ESMTP id 2AFBE8E0007 for ; Tue, 28 Apr 2020 16:59:36 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id D8A99181AC9C6 for ; Tue, 28 Apr 2020 20:59:35 +0000 (UTC) X-FDA: 76758479910.26.milk27_53c26643e9b51 X-Spam-Summary: 2,0,0,5cf615309182c4de,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:41:69:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1543:1711:1730:1747:1777:1792:2194:2196:2198:2199:2200:2201:2393:2559:2562:2731:2907:3138:3139:3140:3141:3142:3354:3865:3866:3867:3870:3871:3872:4117:4250:4385:5007:6117:6119:6261:6653:7514:7576:7875:8660:9413:9592:10004:11026:11473:11658:11914:12043:12296:12297:12438:12517:12519:12555:12895:13148:13149:13230:13894:14181:14394:14687:14721:21080:21444:21451:21627:21666:21740:21789:21987:21990:30034:30054,0,RBL:209.85.208.194:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.18.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: milk27_53c26643e9b51 X-Filterd-Recvd-Size: 6967 Received: from mail-lj1-f194.google.com (mail-lj1-f194.google.com [209.85.208.194]) by imf21.hostedemail.com (Postfix) with ESMTP for ; Tue, 28 Apr 2020 20:59:35 +0000 (UTC) Received: by mail-lj1-f194.google.com with SMTP id a21so202302ljb.9 for ; Tue, 28 Apr 2020 13:59:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=sSPDLCw928vWc9z2Vz2mWvGeAMzP8/jPehX8Io29R/M=; b=fBNRUevwPDJSM6q61fYhtsse5IE6E2y49tzY82VPIID1sRrr24Y4C3aaCE9B4T1ysJ mOluuoskKtW7BHBxKOgr9NHqfsm0tFoVS2yZiEKTY/jXHX2nvMQJlIcXwoKnTU43WNfI rfVhpIDwCGbdAbrwRJ9hnIG8ov2eLEn5/PxoYgnyh9/NCqjGM8OvzN5YXwHuOgpPQudb wXIItywWBp9ndAHZ5asj6ggkGlspU+FNJ1tWf1RJ3hfOwwS6RF5xvPHNaI/EBRqoATpG oLR9O8UiOtqpy4TSULHeReidBctk5DOOnqlEdLtR8zvnc2ZMQj6x1wnzz/ToBD3zaXP6 HgoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=sSPDLCw928vWc9z2Vz2mWvGeAMzP8/jPehX8Io29R/M=; b=IKLDk98opykoIHrqSp2efaAzQ+zdQNIeDZg7RnADk6zXoHgI3Df5QZUhyUvpR+sgKm +a9IK0qcIlV5mGMnBRWNsWDat7tLq9SFt2zrqiFxVvScaSnNOIFKwDzjBjjgHvUDpMHr xkbFyQ+XHQDC/pzo4EYV2K9/mr+Vqj4gY4guAa2GgZinSDxBkcjGPb/AAasXA+ptE6D0 vp9I494y8dc0VpKDmVGAmjesbTU5XEh1JmtcZPIJExd0Tr9FOJz6eIbdjOipPc8QbZbA vYrtmHM58m8YOxkVu9rdnfR1z168njJ1JHkvt195mcS47qbq21fUDVsKg35XnaKsIStb QZZw== X-Gm-Message-State: AGi0PuYbipjGFW5Zkt9lOUL2KnxCYdJ7wazbXT2OctrAiEF6mjpwF5a2 J38VMQAoSxjZIo03hDzA6PY= X-Google-Smtp-Source: APiQypJOSkeM4KBluNbofCs+yZeBUUC4FcCWL6GUCvKOg8yk0RnaTVgUFbftNLX0tMsTwccEtDwwkw== X-Received: by 2002:a2e:b5d2:: with SMTP id g18mr18864979ljn.51.1588107574209; Tue, 28 Apr 2020 13:59:34 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id z21sm295483ljh.42.2020.04.28.13.59.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Apr 2020 13:59:33 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 05/24] rcu/tree: Simplify debug_objects handling Date: Tue, 28 Apr 2020 22:58:44 +0200 Message-Id: <20200428205903.61704-6-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200428205903.61704-1-urezki@gmail.com> References: <20200428205903.61704-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Joel Fernandes (Google)" In order to prepare for future changes for headless RCU support, make the debug_objects handling in kfree_rcu use the final 'pointer' value of the object, instead of depending on the head. Signed-off-by: Uladzislau Rezki (Sony) Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 29 +++++++++++++---------------- 1 file changed, 13 insertions(+), 16 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 0762ac06f0b7..767aed49d7fd 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2860,13 +2860,11 @@ EXPORT_SYMBOL_GPL(call_rcu); * @nr_records: Number of active pointers in the array * @records: Array of the kfree_rcu() pointers * @next: Next bulk object in the block chain - * @head_free_debug: For debug, when CONFIG_DEBUG_OBJECTS_RCU_HEAD is set */ struct kfree_rcu_bulk_data { unsigned long nr_records; void *records[KFREE_BULK_MAX_ENTR]; struct kfree_rcu_bulk_data *next; - struct rcu_head *head_free_debug; }; /** @@ -2916,11 +2914,13 @@ struct kfree_rcu_cpu { static DEFINE_PER_CPU(struct kfree_rcu_cpu, krc); static __always_inline void -debug_rcu_head_unqueue_bulk(struct rcu_head *head) +debug_rcu_bhead_unqueue(struct kfree_rcu_bulk_data *bhead) { #ifdef CONFIG_DEBUG_OBJECTS_RCU_HEAD - for (; head; head = head->next) - debug_rcu_head_unqueue(head); + int i; + + for (i = 0; i < bhead->nr_records; i++) + debug_rcu_head_unqueue((struct rcu_head *)(bhead->records[i])); #endif } @@ -2950,7 +2950,7 @@ static void kfree_rcu_work(struct work_struct *work) for (; bhead; bhead = bnext) { bnext = bhead->next; - debug_rcu_head_unqueue_bulk(bhead->head_free_debug); + debug_rcu_bhead_unqueue(bhead); rcu_lock_acquire(&rcu_callback_map); trace_rcu_invoke_kfree_bulk_callback(rcu_state.name, @@ -2972,14 +2972,15 @@ static void kfree_rcu_work(struct work_struct *work) */ for (; head; head = next) { unsigned long offset = (unsigned long)head->func; + void *ptr = (void *)head - offset; next = head->next; - debug_rcu_head_unqueue(head); + debug_rcu_head_unqueue((struct rcu_head *)ptr); rcu_lock_acquire(&rcu_callback_map); trace_rcu_invoke_kfree_callback(rcu_state.name, head, offset); if (!WARN_ON_ONCE(!__is_kfree_rcu_offset(offset))) - kfree((void *)head - offset); + kfree(ptr); rcu_lock_release(&rcu_callback_map); cond_resched_tasks_rcu_qs(); @@ -3118,18 +3119,11 @@ kfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, /* Initialize the new block. */ bnode->nr_records = 0; bnode->next = krcp->bhead; - bnode->head_free_debug = NULL; /* Attach it to the head. */ krcp->bhead = bnode; } -#ifdef CONFIG_DEBUG_OBJECTS_RCU_HEAD - head->func = func; - head->next = krcp->bhead->head_free_debug; - krcp->bhead->head_free_debug = head; -#endif - /* Finally insert. */ krcp->bhead->records[krcp->bhead->nr_records++] = (void *) head - (unsigned long) func; @@ -3153,14 +3147,17 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func) { unsigned long flags; struct kfree_rcu_cpu *krcp; + void *ptr; local_irq_save(flags); /* For safely calling this_cpu_ptr(). */ krcp = this_cpu_ptr(&krc); if (krcp->initialized) raw_spin_lock(&krcp->lock); + ptr = (void *)head - (unsigned long)func; + /* Queue the object but don't yet schedule the batch. */ - if (debug_rcu_head_queue(head)) { + if (debug_rcu_head_queue(ptr)) { /* Probable double kfree_rcu(), just leak. */ WARN_ONCE(1, "%s(): Double-freed call. rcu_head %p\n", __func__, head); From patchwork Tue Apr 28 20:58:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11515565 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CD59F1392 for ; Tue, 28 Apr 2020 20:59:43 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 903C6214D8 for ; Tue, 28 Apr 2020 20:59:43 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Jpurp9Cl" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 903C6214D8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id BAE9B8E000B; Tue, 28 Apr 2020 16:59:37 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B86518E0007; Tue, 28 Apr 2020 16:59:37 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A75AC8E000B; Tue, 28 Apr 2020 16:59:37 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0138.hostedemail.com [216.40.44.138]) by kanga.kvack.org (Postfix) with ESMTP id 8416F8E0007 for ; Tue, 28 Apr 2020 16:59:37 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 4ADB0180AD802 for ; Tue, 28 Apr 2020 20:59:37 +0000 (UTC) X-FDA: 76758479994.26.cow40_53f342d92bd3d X-Spam-Summary: 10,1,0,419e0e95b1d4ef4f,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:41:355:379:404:541:800:960:965:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1541:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:2693:2907:3138:3139:3140:3141:3142:3353:3865:3866:3867:3868:3870:3871:3872:3874:4321:4385:4390:4395:5007:6261:6653:7514:7875:8603:9413:9592:10004:11026:11473:11658:11914:12043:12297:12517:12519:12555:12895:12986:13069:13311:13357:13894:14181:14384:14394:14687:14721:21080:21444:21451:21627:21666:21740:30034:30054,0,RBL:209.85.208.195:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.18.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:28,LUA_SUMMARY:none X-HE-Tag: cow40_53f342d92bd3d X-Filterd-Recvd-Size: 5071 Received: from mail-lj1-f195.google.com (mail-lj1-f195.google.com [209.85.208.195]) by imf39.hostedemail.com (Postfix) with ESMTP for ; Tue, 28 Apr 2020 20:59:36 +0000 (UTC) Received: by mail-lj1-f195.google.com with SMTP id b2so244670ljp.4 for ; Tue, 28 Apr 2020 13:59:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0CB7l/SclT28khEWhDaoUOOzb7zLYSSjDTiXz+d76vU=; b=Jpurp9ClV4jnTD0F9m48AGNqTXz1ipQrAnggVWpZFOF++j+brs2BKKdSPUbdX6aZOP BB8HYRVyTSPw91F6rc83G9mQtUbt8i+VsniMWASdZeP4lB0eHTKd1GAxaIoFracSoQ3+ F7NRMfQvoTQaOkMDbTk7RM6frPn7zX6fgnxmlX95hAJhKHI6P8XvSJz0jdrT954quNlp dc7nYW9b2hFclqqafaD225XwkP8oDazc0+OJhcOqqA2m1CoVJpFVKvB9+/mbSIQC6RmP 1sCltiRVjQyIl+DaJZ13b39p5EgL7BpUtO0Iexqfelc4wWFL2kl9G74dBy8g1jpDUACO T3lw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0CB7l/SclT28khEWhDaoUOOzb7zLYSSjDTiXz+d76vU=; b=qKJa/bm3Pold+kk3Qxc9afyucPdbENTyrXp9NfcQLrJ6/sIfBJK9eskWbwgTZ41aGX yCyHEaoshteA0NgCr6vTU1DbfWdG9qq1v7px7cEnMlo3SH9jL20mGIDAHKXRTfRBFmvP LKhUUO8hsqgyEse+HAMT5O2pbhcXNPbeCD3qZZ2eDdISLpH++2PxNLM4jdwQp0rswZMi sQUwOYgYcA7ycPin1iomV4Vp3M7xBn81lC4Fo7NL2d3xYMGPulRTqmd3IDkhiNySD2hw AzzJv6tnMj/TM6KMtVQpCn/CU1uEjFXG/TEKrEHDjhH4unCq3GXtG8aZBXICtdWL58BT n1qw== X-Gm-Message-State: AGi0PubG3Pol3WO7WDOswn+wqFCs7SbNAfCHuFygk9sz3Uh/cqzA68L2 l7hWldhJLT+GVj8eqVCpiAo= X-Google-Smtp-Source: APiQypLcBZKomWi/mld0ygXkGSXRF8+JLRFoqb07LLbjRoBNCtmRtZtFo7Ra7QnQjLvb3s+93krp6g== X-Received: by 2002:a2e:571a:: with SMTP id l26mr18387002ljb.12.1588107575418; Tue, 28 Apr 2020 13:59:35 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id z21sm295483ljh.42.2020.04.28.13.59.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Apr 2020 13:59:34 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko , Boqun Feng Subject: [PATCH 06/24] rcu/tree: Simplify KFREE_BULK_MAX_ENTR macro Date: Tue, 28 Apr 2020 22:58:45 +0200 Message-Id: <20200428205903.61704-7-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200428205903.61704-1-urezki@gmail.com> References: <20200428205903.61704-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We can simplify KFREE_BULK_MAX_ENTR macro and get rid of magic numbers which were used to make the structure to be exactly one page. Suggested-by: Boqun Feng Reviewed-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki (Sony) Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 767aed49d7fd..eebd7f627794 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2848,13 +2848,6 @@ EXPORT_SYMBOL_GPL(call_rcu); #define KFREE_DRAIN_JIFFIES (HZ / 50) #define KFREE_N_BATCHES 2 -/* - * This macro defines how many entries the "records" array - * will contain. It is based on the fact that the size of - * kfree_rcu_bulk_data structure becomes exactly one page. - */ -#define KFREE_BULK_MAX_ENTR ((PAGE_SIZE / sizeof(void *)) - 3) - /** * struct kfree_rcu_bulk_data - single block to store kfree_rcu() pointers * @nr_records: Number of active pointers in the array @@ -2863,10 +2856,18 @@ EXPORT_SYMBOL_GPL(call_rcu); */ struct kfree_rcu_bulk_data { unsigned long nr_records; - void *records[KFREE_BULK_MAX_ENTR]; struct kfree_rcu_bulk_data *next; + void *records[]; }; +/* + * This macro defines how many entries the "records" array + * will contain. It is based on the fact that the size of + * kfree_rcu_bulk_data structure becomes exactly one page. + */ +#define KFREE_BULK_MAX_ENTR \ + ((PAGE_SIZE - sizeof(struct kfree_rcu_bulk_data)) / sizeof(void *)) + /** * struct kfree_rcu_cpu_work - single batch of kfree_rcu() requests * @rcu_work: Let queue_rcu_work() invoke workqueue handler after grace period From patchwork Tue Apr 28 20:58:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11515567 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2130014DD for ; Tue, 28 Apr 2020 20:59:46 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E2C3521927 for ; Tue, 28 Apr 2020 20:59:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="mTgjdo+b" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E2C3521927 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 458458E000C; Tue, 28 Apr 2020 16:59:40 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 409998E0007; Tue, 28 Apr 2020 16:59:40 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2A9FF8E000C; Tue, 28 Apr 2020 16:59:40 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0057.hostedemail.com [216.40.44.57]) by kanga.kvack.org (Postfix) with ESMTP id EE6CD8E0007 for ; Tue, 28 Apr 2020 16:59:39 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 71F21824805A for ; Tue, 28 Apr 2020 20:59:39 +0000 (UTC) X-FDA: 76758480078.19.rice60_543d0a681a552 X-Spam-Summary: 2,0,0,bd060d6d4ec164fe,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:41:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1541:1711:1730:1747:1777:1792:2194:2196:2199:2200:2393:2559:2562:2901:3138:3139:3140:3141:3142:3353:3865:3867:3868:3870:3871:3872:4321:4385:5007:6117:6119:6261:6653:7514:9413:9592:10004:11026:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12683:12895:13069:13311:13357:13894:14096:14110:14181:14384:14394:14687:14721:21080:21433:21444:21451:21611:21627:21666:21740:21990:30054:30070,0,RBL:209.85.208.194:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: rice60_543d0a681a552 X-Filterd-Recvd-Size: 5181 Received: from mail-lj1-f194.google.com (mail-lj1-f194.google.com [209.85.208.194]) by imf10.hostedemail.com (Postfix) with ESMTP for ; Tue, 28 Apr 2020 20:59:38 +0000 (UTC) Received: by mail-lj1-f194.google.com with SMTP id h4so187067ljg.12 for ; Tue, 28 Apr 2020 13:59:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/hjFgrHN/AobfDhCW7uHxtGVQp6m3WKcxFSvew6M5pw=; b=mTgjdo+bMst6VnOjdPMHNcx3tuDQ+BDhhX1LfQSV21Z7Uad2qAHdhz6tArXTjHtGWl ryL105FvQsaXCwDc/qECPQ3udl9XQdUVbLE0LAdDz63i47h/7cb7vjI+6eqoIUP0xKsr g5zCswt3KHkAUxPHoA63BYFFGiLwPCNe3NeSZ0/96qaiby/d5cHjc1x6xI+B4aFyLBRx RofJAnQaSKp63kKtqXQ8yj2WYWRRnTdUpWISvo4OIJNJRF8RMuhGfOhhr+D1Nd1Vbgq5 gCRPyRachrNlHGaNEzoS7iOonxh8Ti62fbG8mG+5pWVItXTa0pw/kOk8udN/D99y03Ju 9JVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/hjFgrHN/AobfDhCW7uHxtGVQp6m3WKcxFSvew6M5pw=; b=MV1919CTsQ3/j0DJFI1cwwnAeGQ9mp2bEdO/zMkvpyorsarNFCYP/z90YysPOPFxy+ +X8D3O4bCYsCZCuEYkSiza69ZK4LkgGIZkZgdYP40YUPGzTWF1GxZfdWY7W4/LCLgT6v 3prM7M8WD4DjI2ZIlogEiP0zjr5I1l+4k/6qiFz2+90aOiYWjrrT3CM9YcNVOT/P0hDg 5s/7iYN/vr6z7ZnjR82O3lgqtnijZFn4FrVk4H4JZST2XkHxZdETBBvJ/PAAq0Dp63uf G5n3/IwUdlSL/k4nrqDQ8ox5X+jW9fumMl8wG80GfhNzNZi8ZjLNxGliVkqnylNn94JX zK+Q== X-Gm-Message-State: AGi0PuZeNPQH7R8iyYw2lKOUnys3NdCKq2/erYs72jgxXz7wICypweGI wKIez/v9Yv0fpXJIx2F5l+g= X-Google-Smtp-Source: APiQypLbRFLIsVOfuz0lNkpuDjqlP3snMnoaGJKNyj3Wqv3N49M2dESfMr2LkQ60hBGlniDH8cEjaA== X-Received: by 2002:a2e:9b0f:: with SMTP id u15mr18548017lji.272.1588107577075; Tue, 28 Apr 2020 13:59:37 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id z21sm295483ljh.42.2020.04.28.13.59.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Apr 2020 13:59:36 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 07/24] rcu/tree: move locking/unlocking to separate functions Date: Tue, 28 Apr 2020 22:58:46 +0200 Message-Id: <20200428205903.61704-8-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200428205903.61704-1-urezki@gmail.com> References: <20200428205903.61704-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce two helpers to lock and unlock an access to the per-cpu "kfree_rcu_cpu" structure. The reason is to make kfree_call_rcu() function to be more readable. Signed-off-by: Uladzislau Rezki (Sony) --- kernel/rcu/tree.c | 31 +++++++++++++++++++++++-------- 1 file changed, 23 insertions(+), 8 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index eebd7f627794..bc6c2bc8fa32 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2925,6 +2925,27 @@ debug_rcu_bhead_unqueue(struct kfree_rcu_bulk_data *bhead) #endif } +static inline struct kfree_rcu_cpu * +krc_this_cpu_lock(unsigned long *flags) +{ + struct kfree_rcu_cpu *krcp; + + local_irq_save(*flags); // For safely calling this_cpu_ptr(). + krcp = this_cpu_ptr(&krc); + if (likely(krcp->initialized)) + raw_spin_lock(&krcp->lock); + + return krcp; +} + +static inline void +krc_this_cpu_unlock(struct kfree_rcu_cpu *krcp, unsigned long flags) +{ + if (likely(krcp->initialized)) + raw_spin_unlock(&krcp->lock); + local_irq_restore(flags); +} + /* * This function is invoked in workqueue context after a grace period. * It frees all the objects queued on ->bhead_free or ->head_free. @@ -3150,11 +3171,7 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func) struct kfree_rcu_cpu *krcp; void *ptr; - local_irq_save(flags); /* For safely calling this_cpu_ptr(). */ - krcp = this_cpu_ptr(&krc); - if (krcp->initialized) - raw_spin_lock(&krcp->lock); - + krcp = krc_this_cpu_lock(&flags); ptr = (void *)head - (unsigned long)func; /* Queue the object but don't yet schedule the batch. */ @@ -3185,9 +3202,7 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func) } unlock_return: - if (krcp->initialized) - raw_spin_unlock(&krcp->lock); - local_irq_restore(flags); + krc_this_cpu_unlock(krcp, flags); } EXPORT_SYMBOL_GPL(kfree_call_rcu); From patchwork Tue Apr 28 20:58:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11515569 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3B0991805 for ; Tue, 28 Apr 2020 20:59:48 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 093FB20737 for ; Tue, 28 Apr 2020 20:59:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Jp/cGmJk" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 093FB20737 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E95848E000D; Tue, 28 Apr 2020 16:59:40 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E06FF8E0007; Tue, 28 Apr 2020 16:59:40 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CE2BC8E000D; Tue, 28 Apr 2020 16:59:40 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0124.hostedemail.com [216.40.44.124]) by kanga.kvack.org (Postfix) with ESMTP id B3CA78E0007 for ; Tue, 28 Apr 2020 16:59:40 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 708D85828 for ; Tue, 28 Apr 2020 20:59:40 +0000 (UTC) X-FDA: 76758480120.04.soup86_54689d2252551 X-Spam-Summary: 2,0,0,ca64c452bfa882fe,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:41:355:379:541:800:960:966:968:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1542:1711:1730:1747:1777:1792:2194:2196:2199:2200:2393:2559:2562:2901:3138:3139:3140:3141:3142:3354:3865:3866:3867:3868:3870:3871:3872:3874:4117:4385:5007:6117:6261:6653:7514:7576:9413:10004:11026:11473:11658:11914:12043:12296:12297:12438:12517:12519:12555:12679:12895:13161:13229:13868:13894:14181:14394:14687:14721:21080:21444:21451:21627:21666:21740:21990:30012:30054:30070,0,RBL:209.85.208.195:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: soup86_54689d2252551 X-Filterd-Recvd-Size: 6161 Received: from mail-lj1-f195.google.com (mail-lj1-f195.google.com [209.85.208.195]) by imf48.hostedemail.com (Postfix) with ESMTP for ; Tue, 28 Apr 2020 20:59:39 +0000 (UTC) Received: by mail-lj1-f195.google.com with SMTP id l19so197664lje.10 for ; Tue, 28 Apr 2020 13:59:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KpPRWZDvG8CgK8dZKCEsG9Emj5dGIpU0eqSTK3oTu4U=; b=Jp/cGmJks3FbPmO6cnSdkivz7OJHf87NIdc550TYVUo5oyct/fknb8SeQgG/Wa6AvT j1s4N78U0TRqqfWCoy4Cnwkj6gYaJjhBPoPA4SDXFmJLkSgX/GEs/heihzFRtuULBLuR ydXaywn1Lar7YFPryFl7cTeTeik4WiCd59r+Y9nk9wFiAtp1oezC4vFzgKC81AgGdBDL STIicmNXYpDyWAknzGXu+U4RdBzyt0gcZaLuVfHNt3i11xsecqtZeCVI8ivDe15kcRl0 qrTiYkXv2wP3a7zEMIu2PMB0JHWoYrUnSoEO4DbpVVTeg46tQ5FPN0LKzJqMmpMJ/Mzu a/yQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KpPRWZDvG8CgK8dZKCEsG9Emj5dGIpU0eqSTK3oTu4U=; b=OnWcX6vz4YP5CYDrFrrFa6+9tg5xbYn4q7M9IbmMRS1TzDF+E3hHvjmLzzd5dOXLNb bA3BIFCKWwXZc/LqfdGUuRE1kcqPfdlJ0hzz0SKhqXYOe/afnYE25MJvn9VOhYHp7E3q om42pdf59/xTeZXENPrOHTi6pIeIQK7rXFo+e3SNAommXaZR6Edx+ssdT6U+8MHjq41q NU1+QmRSxanAXPg/UzCmCiYfuW7Bt5+L9MyiyzH1Z0J46eSrQMCL1NfFFvfyGMo/L1lZ 1XxNHVPoCh27ugzUq0CoKLIiv6CQTA22t/z3mlKxpI3PKGeusbQnNKswiSXDhtlDouqG AkOw== X-Gm-Message-State: AGi0PuYL1pBKY0Zmc5P511f1MtBJHZhywZQYb3YQaLHu/r1ZiDNJ6XY4 YGDPg2EDcpUxxxdrIhOsIZ4= X-Google-Smtp-Source: APiQypIK0yNKxQMrJ+FmeTtM1yEhM2Bk+cTVBWd0YgRswTk8U0gkkBZkwM/NPoG1ukrhwMQjsmSyPQ== X-Received: by 2002:a2e:9713:: with SMTP id r19mr19379545lji.89.1588107578198; Tue, 28 Apr 2020 13:59:38 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id z21sm295483ljh.42.2020.04.28.13.59.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Apr 2020 13:59:37 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko , Sebastian Andrzej Siewior Subject: [PATCH 08/24] rcu/tree: Use static initializer for krc.lock Date: Tue, 28 Apr 2020 22:58:47 +0200 Message-Id: <20200428205903.61704-9-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200428205903.61704-1-urezki@gmail.com> References: <20200428205903.61704-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Sebastian Andrzej Siewior The per-CPU variable is initialized at runtime in kfree_rcu_batch_init(). This function is invoked before 'rcu_scheduler_active' is set to 'RCU_SCHEDULER_RUNNING'. After the initialisation, '->initialized' is to true. The raw_spin_lock is only acquired if '->initialized' is set to true. The worqueue item is only used if 'rcu_scheduler_active' set to RCU_SCHEDULER_RUNNING which happens after initialisation. Use a static initializer for krc.lock and remove the runtime initialisation of the lock. Since the lock can now be always acquired, remove the '->initialized' check. Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Uladzislau Rezki (Sony) --- kernel/rcu/tree.c | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index bc6c2bc8fa32..89e9ca3f4e3e 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2892,7 +2892,7 @@ struct kfree_rcu_cpu_work { * @lock: Synchronize access to this structure * @monitor_work: Promote @head to @head_free after KFREE_DRAIN_JIFFIES * @monitor_todo: Tracks whether a @monitor_work delayed work is pending - * @initialized: The @lock and @rcu_work fields have been initialized + * @initialized: The @rcu_work fields have been initialized * * This is a per-CPU structure. The reason that it is not included in * the rcu_data structure is to permit this code to be extracted from @@ -2912,7 +2912,9 @@ struct kfree_rcu_cpu { int count; }; -static DEFINE_PER_CPU(struct kfree_rcu_cpu, krc); +static DEFINE_PER_CPU(struct kfree_rcu_cpu, krc) = { + .lock = __RAW_SPIN_LOCK_UNLOCKED(krc.lock), +}; static __always_inline void debug_rcu_bhead_unqueue(struct kfree_rcu_bulk_data *bhead) @@ -2930,10 +2932,9 @@ krc_this_cpu_lock(unsigned long *flags) { struct kfree_rcu_cpu *krcp; - local_irq_save(*flags); // For safely calling this_cpu_ptr(). + local_irq_save(*flags); /* For safely calling this_cpu_ptr(). */ krcp = this_cpu_ptr(&krc); - if (likely(krcp->initialized)) - raw_spin_lock(&krcp->lock); + raw_spin_lock(&krcp->lock); return krcp; } @@ -2941,8 +2942,7 @@ krc_this_cpu_lock(unsigned long *flags) static inline void krc_this_cpu_unlock(struct kfree_rcu_cpu *krcp, unsigned long flags) { - if (likely(krcp->initialized)) - raw_spin_unlock(&krcp->lock); + raw_spin_unlock(&krcp->lock); local_irq_restore(flags); } @@ -4168,7 +4168,6 @@ static void __init kfree_rcu_batch_init(void) for_each_possible_cpu(cpu) { struct kfree_rcu_cpu *krcp = per_cpu_ptr(&krc, cpu); - raw_spin_lock_init(&krcp->lock); for (i = 0; i < KFREE_N_BATCHES; i++) { INIT_RCU_WORK(&krcp->krw_arr[i].rcu_work, kfree_rcu_work); krcp->krw_arr[i].krcp = krcp; From patchwork Tue Apr 28 20:58:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11515571 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 41BF392C for ; Tue, 28 Apr 2020 20:59:50 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0296220737 for ; Tue, 28 Apr 2020 20:59:50 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="EWBRAk5k" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0296220737 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0FD0D8E000E; Tue, 28 Apr 2020 16:59:42 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0AEF98E0007; Tue, 28 Apr 2020 16:59:42 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F04BE8E000E; Tue, 28 Apr 2020 16:59:41 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0170.hostedemail.com [216.40.44.170]) by kanga.kvack.org (Postfix) with ESMTP id D40768E0007 for ; Tue, 28 Apr 2020 16:59:41 -0400 (EDT) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id A10D4824805A for ; Tue, 28 Apr 2020 20:59:41 +0000 (UTC) X-FDA: 76758480162.16.sheet22_549340230a151 X-Spam-Summary: 2,0,0,b9a052fefbea6afe,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:2:41:355:379:541:560:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1605:1606:1730:1747:1777:1792:2196:2199:2393:2559:2562:2901:2907:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:4119:4250:4321:4385:5007:6117:6261:6653:7514:7875:7903:8603:9010:9040:9413:10004:11026:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12683:12895:13141:13161:13229:13230:13894:14394:14687:21080:21324:21433:21444:21451:21611:21627:21666:21740:21987:21990:30029:30034:30054:30070,0,RBL:209.85.208.195:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: sheet22_549340230a151 X-Filterd-Recvd-Size: 8225 Received: from mail-lj1-f195.google.com (mail-lj1-f195.google.com [209.85.208.195]) by imf05.hostedemail.com (Postfix) with ESMTP for ; Tue, 28 Apr 2020 20:59:41 +0000 (UTC) Received: by mail-lj1-f195.google.com with SMTP id h4so187384ljg.12 for ; Tue, 28 Apr 2020 13:59:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=GLSwCcFlhT2VhQKTC73KXNQ7m5lfVddOJ9NrXpevtsw=; b=EWBRAk5kGzdsYCoM7US1OsnIhAydgAQIeV2ADHVhbDIYUFCzjmHePpJIfhPdwMe0ZD kjLHUK9aENp8DOOf3Fd1kQNXJuX5HAHNlbv4hbNhCcA5o9Mn72gnGvfAVrDyjHyIL0PU 1XUf3HUpRQ94dzVqzsdqsxlKrAFWC+tTMHx5DWFJ+Pvq2rtBCvD4Kc5ER4tTd2FoSUnR HyOhVXNKFX7miwg9KAvYD2LVZYuJzeBoT8LRFkEBlPqoUzkmzDTLLWQrUP6ni9nIjU1D T2mtsu1K9aFpnb4aWigTJRdnHQP1nijHY8L8wITm7Y1MDko6M/KPYlOfV/x3k2utaP3H 1O8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=GLSwCcFlhT2VhQKTC73KXNQ7m5lfVddOJ9NrXpevtsw=; b=aj5kLl+O/OioL7ZUOt2JIdxELnJjJyWJbceqKvF1cmWo/lzr/49Q9uto0T/jS5sNrW H2Bizu+H6zvBiiTd9qNVhl7lIw/Z/8iDyPM/Hz1fk4D/8WTg7Z1qVW9wR4oq9m9oIkDt VgFoH9ImY/TsJKXMjvNXBgtYnt/qeBVyj4zZa23XZXGPsOSxX2YoA3QaIlL7t+s35yv0 H1aJN7TyromlscbdIsHybUtIi3gFmFXCbrhLjDGqamrSXkope6oZUJ6wwO1R/q+d2+Yi wu8/itnAd1bDfAns9Mivn1tohPuqA4/+TbvTHO3xUSqb8pSIibHenhQKVfK+uRirKchD rhbQ== X-Gm-Message-State: AGi0PuZkrmB8P8LHzUo0yIotoyBsU8ZkFIdZjuBl3+vfhg6MOvUXPKcn aFKgmI6XN04AfCcnzROWTyE= X-Google-Smtp-Source: APiQypLBd6ihKL4ASunwou7LRD3kt57nhTc35mEMNnUP9p5PGWGHowh92faeH1ivrYe/6p3mH9nN5w== X-Received: by 2002:ac2:48b1:: with SMTP id u17mr19859185lfg.187.1588107579619; Tue, 28 Apr 2020 13:59:39 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id z21sm295483ljh.42.2020.04.28.13.59.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Apr 2020 13:59:39 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 09/24] rcu/tree: cache specified number of objects Date: Tue, 28 Apr 2020 22:58:48 +0200 Message-Id: <20200428205903.61704-10-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200428205903.61704-1-urezki@gmail.com> References: <20200428205903.61704-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Cache some extra objects per-CPU. During reclaim process some pages are cached instead of releasing by linking them into the list. Such approach provides O(1) access time to the cache. That reduces number of requests to the page allocator, also that makes it more helpful if a low memory condition occurs. A parameter reflecting the minimum allowed pages to be cached per one CPU is propagated via sysfs, it is read only, the name is "rcu_min_cached_objs". Signed-off-by: Uladzislau Rezki (Sony) --- kernel/rcu/tree.c | 64 ++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 60 insertions(+), 4 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 89e9ca3f4e3e..d8975819b1c9 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -178,6 +178,14 @@ module_param(gp_init_delay, int, 0444); static int gp_cleanup_delay; module_param(gp_cleanup_delay, int, 0444); +/* + * This rcu parameter is read-only, but can be write also. + * It reflects the minimum allowed number of objects which + * can be cached per-CPU. Object size is equal to one page. + */ +int rcu_min_cached_objs = 2; +module_param(rcu_min_cached_objs, int, 0444); + /* Retrieve RCU kthreads priority for rcutorture */ int rcu_get_gp_kthreads_prio(void) { @@ -2887,7 +2895,6 @@ struct kfree_rcu_cpu_work { * struct kfree_rcu_cpu - batch up kfree_rcu() requests for RCU grace period * @head: List of kfree_rcu() objects not yet waiting for a grace period * @bhead: Bulk-List of kfree_rcu() objects not yet waiting for a grace period - * @bcached: Keeps at most one object for later reuse when build chain blocks * @krw_arr: Array of batches of kfree_rcu() objects waiting for a grace period * @lock: Synchronize access to this structure * @monitor_work: Promote @head to @head_free after KFREE_DRAIN_JIFFIES @@ -2902,7 +2909,6 @@ struct kfree_rcu_cpu_work { struct kfree_rcu_cpu { struct rcu_head *head; struct kfree_rcu_bulk_data *bhead; - struct kfree_rcu_bulk_data *bcached; struct kfree_rcu_cpu_work krw_arr[KFREE_N_BATCHES]; raw_spinlock_t lock; struct delayed_work monitor_work; @@ -2910,6 +2916,15 @@ struct kfree_rcu_cpu { bool initialized; // Number of objects for which GP not started int count; + + /* + * Number of cached objects which are queued into + * the lock-less list. This cache is used by the + * kvfree_call_rcu() function and as of now its + * size is static. + */ + struct llist_head bkvcache; + int nr_bkv_objs; }; static DEFINE_PER_CPU(struct kfree_rcu_cpu, krc) = { @@ -2946,6 +2961,31 @@ krc_this_cpu_unlock(struct kfree_rcu_cpu *krcp, unsigned long flags) local_irq_restore(flags); } +static inline struct kfree_rcu_bulk_data * +get_cached_bnode(struct kfree_rcu_cpu *krcp) +{ + if (!krcp->nr_bkv_objs) + return NULL; + + krcp->nr_bkv_objs--; + return (struct kfree_rcu_bulk_data *) + llist_del_first(&krcp->bkvcache); +} + +static inline bool +put_cached_bnode(struct kfree_rcu_cpu *krcp, + struct kfree_rcu_bulk_data *bnode) +{ + /* Check the limit. */ + if (krcp->nr_bkv_objs >= rcu_min_cached_objs) + return false; + + llist_add((struct llist_node *) bnode, &krcp->bkvcache); + krcp->nr_bkv_objs++; + return true; + +} + /* * This function is invoked in workqueue context after a grace period. * It frees all the objects queued on ->bhead_free or ->head_free. @@ -2981,7 +3021,12 @@ static void kfree_rcu_work(struct work_struct *work) kfree_bulk(bhead->nr_records, bhead->records); rcu_lock_release(&rcu_callback_map); - if (cmpxchg(&krcp->bcached, NULL, bhead)) + krcp = krc_this_cpu_lock(&flags); + if (put_cached_bnode(krcp, bhead)) + bhead = NULL; + krc_this_cpu_unlock(krcp, flags); + + if (bhead) free_page((unsigned long) bhead); cond_resched_tasks_rcu_qs(); @@ -3114,7 +3159,7 @@ kfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, /* Check if a new block is required. */ if (!krcp->bhead || krcp->bhead->nr_records == KFREE_BULK_MAX_ENTR) { - bnode = xchg(&krcp->bcached, NULL); + bnode = get_cached_bnode(krcp); if (!bnode) { WARN_ON_ONCE(sizeof(struct kfree_rcu_bulk_data) > PAGE_SIZE); @@ -4167,12 +4212,23 @@ static void __init kfree_rcu_batch_init(void) for_each_possible_cpu(cpu) { struct kfree_rcu_cpu *krcp = per_cpu_ptr(&krc, cpu); + struct kfree_rcu_bulk_data *bnode; for (i = 0; i < KFREE_N_BATCHES; i++) { INIT_RCU_WORK(&krcp->krw_arr[i].rcu_work, kfree_rcu_work); krcp->krw_arr[i].krcp = krcp; } + for (i = 0; i < rcu_min_cached_objs; i++) { + bnode = (struct kfree_rcu_bulk_data *) + __get_free_page(GFP_NOWAIT | __GFP_NOWARN); + + if (bnode) + put_cached_bnode(krcp, bnode); + else + pr_err("Failed to preallocate for %d CPU!\n", cpu); + } + INIT_DELAYED_WORK(&krcp->monitor_work, kfree_rcu_monitor); krcp->initialized = true; } From patchwork Tue Apr 28 20:58:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11515573 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C9A7B14DD for ; Tue, 28 Apr 2020 20:59:52 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 97C3520737 for ; Tue, 28 Apr 2020 20:59:52 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="p9Fr9GlC" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 97C3520737 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id F32828E000F; Tue, 28 Apr 2020 16:59:42 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E43248E0007; Tue, 28 Apr 2020 16:59:42 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CBAD88E000F; Tue, 28 Apr 2020 16:59:42 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0238.hostedemail.com [216.40.44.238]) by kanga.kvack.org (Postfix) with ESMTP id B0C7D8E0007 for ; Tue, 28 Apr 2020 16:59:42 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 79CF4824805A for ; Tue, 28 Apr 2020 20:59:42 +0000 (UTC) X-FDA: 76758480204.30.event61_54b7d1c667d46 X-Spam-Summary: 2,0,0,1ef52ac6088e2f62,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2393:2559:2562:2693:3138:3139:3140:3141:3142:3352:3865:3866:3867:3868:3870:3871:3872:4321:5007:6119:6261:6653:7514:8603:9413:10004:11026:11232:11658:11914:12043:12114:12296:12297:12438:12517:12519:12555:12895:13069:13200:13229:13311:13357:13894:14181:14384:14394:14687:14721:21080:21444:21451:21627:21666:21990:30034:30045:30054,0,RBL:209.85.208.193:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: event61_54b7d1c667d46 X-Filterd-Recvd-Size: 4291 Received: from mail-lj1-f193.google.com (mail-lj1-f193.google.com [209.85.208.193]) by imf35.hostedemail.com (Postfix) with ESMTP for ; Tue, 28 Apr 2020 20:59:42 +0000 (UTC) Received: by mail-lj1-f193.google.com with SMTP id y4so216501ljn.7 for ; Tue, 28 Apr 2020 13:59:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WD1ujdDPaZr2tel/SdXaMCcHYNwfEhTtkcv7u+ya4TM=; b=p9Fr9GlCZWdBfVeNmGFGRectLCP0tawZJ6HOk4/26gkt2xbuGisjL5HWK8oxenMLc3 WK7CFb3ruKHCtZ7DW6Lp8Fpzt4GHk+CGxiYQtgWPSlUcZZhyrkXg+x7Q/f57QULeG4g7 KBX/5c90KxyqqsIIlo8SJO7E7uG+2zSYj8AOCV2Ylk8oubDLEsK3ADImk2flhGOFkXI+ 87KwKJ/zH2FkeTIOkSsL2VQzig4qfyW01BDbny+JHyYdb8TNIJn+W5fs5fV4eB76ibiO CcBQtvZ48ahRBPCdPeE5xo7HRSuwQn5iX+QgHsxVfkr1T4EZH49PkqEz1HmXrNUWClic bbZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WD1ujdDPaZr2tel/SdXaMCcHYNwfEhTtkcv7u+ya4TM=; b=Mru9VUEwUO/3vSKSXArJd0WkWKbXap51Tc+PbzX/mdJ5sM9RnEpkOPssm41Dm7F/Kk OxnSK04zJZ/tMHy7WT2ywjRguI+ikmBtNYm7hhhW0dVYGPHN9cyijJUtrcIJshWwDX7w gTi8Ulr4t0p8W/IsIwcOmFQC0DzFegABl0yVFp7jClAJ6oKMw2xw1YEIukTUksnuQDH5 MQJ/4YYJF/1xHw45QxREJH392dpEyqwtekbeaKSblKder6v9BfyqLcmiUJI51GcAkMIY DioWJH4g7esO5oLP3ThDu+6Wl+AwK89qRv2E1reUzBysRC1V57IWVettzD00ZuxcQqWP LyAw== X-Gm-Message-State: AGi0PuaJoRespm3a5C5QHJokoJBSFhIwBz6bllJlrPu6nxfxqzSen2W1 FjMqpMO2c1Xz4mrPTuyVlbg= X-Google-Smtp-Source: APiQypJNL/we3uS/PpZeWdXqJOcuZv4g219qgtqBw9a68MWbrzBMMmSnncu/axBnMngVJ8BcXOksZA== X-Received: by 2002:a2e:b057:: with SMTP id d23mr19244054ljl.157.1588107580807; Tue, 28 Apr 2020 13:59:40 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id z21sm295483ljh.42.2020.04.28.13.59.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Apr 2020 13:59:40 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 10/24] rcu/tree: add rcutree.rcu_min_cached_objs description Date: Tue, 28 Apr 2020 22:58:49 +0200 Message-Id: <20200428205903.61704-11-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200428205903.61704-1-urezki@gmail.com> References: <20200428205903.61704-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Document the rcutree.rcu_min_cached_objs sysfs kernel parameter. Signed-off-by: Uladzislau Rezki (Sony) --- Documentation/admin-guide/kernel-parameters.txt | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 828ff975fbc6..b2b7022374af 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3977,6 +3977,14 @@ latencies, which will choose a value aligned with the appropriate hardware boundaries. + rcutree.rcu_min_cached_objs= [KNL] + Minimum number of objects which are cached and + maintained per one CPU. Object size is equal + to PAGE_SIZE. The cache allows to reduce the + pressure to page allocator, also it makes the + whole algorithm to behave better in low memory + condition. + rcutree.jiffies_till_first_fqs= [KNL] Set delay from grace-period initialization to first attempt to force quiescent states. From patchwork Tue Apr 28 20:58:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11515575 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 05F3F92C for ; Tue, 28 Apr 2020 20:59:55 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AB33D21744 for ; Tue, 28 Apr 2020 20:59:54 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="gNThmBlg" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AB33D21744 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 765438E0010; Tue, 28 Apr 2020 16:59:44 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7159F8E0007; Tue, 28 Apr 2020 16:59:44 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 62D0A8E0010; Tue, 28 Apr 2020 16:59:44 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0056.hostedemail.com [216.40.44.56]) by kanga.kvack.org (Postfix) with ESMTP id 4517A8E0007 for ; Tue, 28 Apr 2020 16:59:44 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 09EB8180AD80F for ; Tue, 28 Apr 2020 20:59:44 +0000 (UTC) X-FDA: 76758480288.03.trip13_54ea39b690018 X-Spam-Summary: 2,0,0,88306afce2b7fa70,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:4:41:69:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1605:1730:1747:1777:1792:2196:2199:2393:2559:2562:2639:2693:2901:2907:2914:2918:3138:3139:3140:3141:3142:3503:3504:3865:3866:3867:3868:3870:3871:3872:3874:4250:4321:4385:5007:6117:6119:6261:6653:7514:7875:7903:8603:9010:9413:9592:10004:11026:11232:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12683:12895:13141:13161:13229:13230:13868:13894:14394:14687:21080:21324:21433:21444:21451:21627:21666:21740:21789:21987:21990:30034:30045:30054:30070:30091,0,RBL:209.85.208.196:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.18.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:4,LUA_SUMMARY:none X-HE-Tag: trip13_54ea39b690018 X-Filterd-Recvd-Size: 17134 Received: from mail-lj1-f196.google.com (mail-lj1-f196.google.com [209.85.208.196]) by imf07.hostedemail.com (Postfix) with ESMTP for ; Tue, 28 Apr 2020 20:59:43 +0000 (UTC) Received: by mail-lj1-f196.google.com with SMTP id h4so187684ljg.12 for ; Tue, 28 Apr 2020 13:59:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Y7MAhKyzz3e/Fs+xMmtreUpZYUfkP//qliijuNNwt88=; b=gNThmBlgvLEzpTWBz/Tdzm5eAZCPB+MOs15tK50M58/qOBM52ODhszfoSq263IU3LT Ehgz57/3S2aIuF4Od5GJ4jtPkq7d1mltUSg1irz55gTBrk642USk/FSe2bHNVza5RYIk 5DrBJTG/1Lr4FA0cgEtfKvws1xajGL8GGtmrkHB8Lcnn3oikYKb8EqbUCZxeQ2urkXXV /KKrb0Nthxf2ertDsOBSpi+yibfYXOJrboiN3xus/oaau/rlrdLOBcTrtEqakkNQiQpv svLYJQoNUarBkkiHOsN5ElAfWEJ3uAiGrteOZSP2HWRnFJpHAVXg7kh7yFKZlB+kYIvN opSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Y7MAhKyzz3e/Fs+xMmtreUpZYUfkP//qliijuNNwt88=; b=lMEPDrF3eBLFUWUoZ34zLe3GGH4B2O0SKjB1NQQbcCqIx8GerOOxnsYgF/x9XTnh3p oFfgABAxFZRERqq+N8IWAG/6XmV5ed5J8jz1cocpV7OnjvRdNbmv3DhxryZNGez/JFJ9 DN2+ff5vmJJd6I1k/94cbE6NuRqblWsc0z3bqJV12UnW66wAEKjA/KWYPYBPb26s/wkq RWYu8NIUHL3bQe/cBvgnnlUoIGMiHlJDc238Wd2RpCR6yeMUTyLuxrPm+Ek+3VOZ7PvJ QX5/SsjKrludwKXOMqHa5ukHi2fWoKB+sRjVCcV2eGUjHJNfNN4/cRzBwe0DXI7mht4E HDFA== X-Gm-Message-State: AGi0PuZuMcs4ZxUWt4xLWXPSo3OrNrbIQHNZcIRZOPRWscM2vxode7VT NWa5LX1/CD3AEnCMl/wDXeg= X-Google-Smtp-Source: APiQypKy5Gc8UNnfKygMNixPeCTeypgmTqWb7IbpYgGy5qBBM4NMrym+NReeOhx65Lv2MWtL6RpzeA== X-Received: by 2002:a05:6512:318a:: with SMTP id i10mr20378110lfe.96.1588107582020; Tue, 28 Apr 2020 13:59:42 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id z21sm295483ljh.42.2020.04.28.13.59.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Apr 2020 13:59:41 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 11/24] rcu/tree: Maintain separate array for vmalloc ptrs Date: Tue, 28 Apr 2020 22:58:50 +0200 Message-Id: <20200428205903.61704-12-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200428205903.61704-1-urezki@gmail.com> References: <20200428205903.61704-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To do so we use an array of common kvfree_rcu_bulk_data structure. It consists of two elements, index number 0 corresponds to SLAB ptrs., whereas vmalloc pointers can be accessed by using index number 1. The reason of not mixing pointers is to have an easy way to to distinguish them. It is also the preparation patch for head-less objects support. When an object is head-less we can not queue it into any list, instead a pointer is placed directly into an array. Signed-off-by: Uladzislau Rezki (Sony) Signed-off-by: Joel Fernandes (Google) Reviewed-by: Joel Fernandes (Google) Co-developed-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 172 +++++++++++++++++++++++++++++----------------- 1 file changed, 109 insertions(+), 63 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index d8975819b1c9..7983926af95b 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -57,6 +57,7 @@ #include #include #include +#include #include "../time/tick-internal.h" #include "tree.h" @@ -2857,44 +2858,44 @@ EXPORT_SYMBOL_GPL(call_rcu); #define KFREE_N_BATCHES 2 /** - * struct kfree_rcu_bulk_data - single block to store kfree_rcu() pointers + * struct kvfree_rcu_bulk_data - single block to store kvfree_rcu() pointers * @nr_records: Number of active pointers in the array - * @records: Array of the kfree_rcu() pointers * @next: Next bulk object in the block chain + * @records: Array of the kvfree_rcu() pointers */ -struct kfree_rcu_bulk_data { +struct kvfree_rcu_bulk_data { unsigned long nr_records; - struct kfree_rcu_bulk_data *next; + struct kvfree_rcu_bulk_data *next; void *records[]; }; /* * This macro defines how many entries the "records" array * will contain. It is based on the fact that the size of - * kfree_rcu_bulk_data structure becomes exactly one page. + * kvfree_rcu_bulk_data structure becomes exactly one page. */ -#define KFREE_BULK_MAX_ENTR \ - ((PAGE_SIZE - sizeof(struct kfree_rcu_bulk_data)) / sizeof(void *)) +#define KVFREE_BULK_MAX_ENTR \ + ((PAGE_SIZE - sizeof(struct kvfree_rcu_bulk_data)) / sizeof(void *)) /** * struct kfree_rcu_cpu_work - single batch of kfree_rcu() requests * @rcu_work: Let queue_rcu_work() invoke workqueue handler after grace period * @head_free: List of kfree_rcu() objects waiting for a grace period - * @bhead_free: Bulk-List of kfree_rcu() objects waiting for a grace period + * @bkvhead_free: Bulk-List of kvfree_rcu() objects waiting for a grace period * @krcp: Pointer to @kfree_rcu_cpu structure */ struct kfree_rcu_cpu_work { struct rcu_work rcu_work; struct rcu_head *head_free; - struct kfree_rcu_bulk_data *bhead_free; + struct kvfree_rcu_bulk_data *bkvhead_free[2]; struct kfree_rcu_cpu *krcp; }; /** * struct kfree_rcu_cpu - batch up kfree_rcu() requests for RCU grace period * @head: List of kfree_rcu() objects not yet waiting for a grace period - * @bhead: Bulk-List of kfree_rcu() objects not yet waiting for a grace period + * @bkvhead: Bulk-List of kvfree_rcu() objects not yet waiting for a grace period * @krw_arr: Array of batches of kfree_rcu() objects waiting for a grace period * @lock: Synchronize access to this structure * @monitor_work: Promote @head to @head_free after KFREE_DRAIN_JIFFIES @@ -2908,7 +2909,7 @@ struct kfree_rcu_cpu_work { */ struct kfree_rcu_cpu { struct rcu_head *head; - struct kfree_rcu_bulk_data *bhead; + struct kvfree_rcu_bulk_data *bkvhead[2]; struct kfree_rcu_cpu_work krw_arr[KFREE_N_BATCHES]; raw_spinlock_t lock; struct delayed_work monitor_work; @@ -2932,7 +2933,7 @@ static DEFINE_PER_CPU(struct kfree_rcu_cpu, krc) = { }; static __always_inline void -debug_rcu_bhead_unqueue(struct kfree_rcu_bulk_data *bhead) +debug_rcu_bhead_unqueue(struct kvfree_rcu_bulk_data *bhead) { #ifdef CONFIG_DEBUG_OBJECTS_RCU_HEAD int i; @@ -2961,20 +2962,20 @@ krc_this_cpu_unlock(struct kfree_rcu_cpu *krcp, unsigned long flags) local_irq_restore(flags); } -static inline struct kfree_rcu_bulk_data * +static inline struct kvfree_rcu_bulk_data * get_cached_bnode(struct kfree_rcu_cpu *krcp) { if (!krcp->nr_bkv_objs) return NULL; krcp->nr_bkv_objs--; - return (struct kfree_rcu_bulk_data *) + return (struct kvfree_rcu_bulk_data *) llist_del_first(&krcp->bkvcache); } static inline bool put_cached_bnode(struct kfree_rcu_cpu *krcp, - struct kfree_rcu_bulk_data *bnode) + struct kvfree_rcu_bulk_data *bnode) { /* Check the limit. */ if (krcp->nr_bkv_objs >= rcu_min_cached_objs) @@ -2993,41 +2994,73 @@ put_cached_bnode(struct kfree_rcu_cpu *krcp, static void kfree_rcu_work(struct work_struct *work) { unsigned long flags; + struct kvfree_rcu_bulk_data *bkhead, *bvhead, *bnext; struct rcu_head *head, *next; - struct kfree_rcu_bulk_data *bhead, *bnext; struct kfree_rcu_cpu *krcp; struct kfree_rcu_cpu_work *krwp; + int i; krwp = container_of(to_rcu_work(work), struct kfree_rcu_cpu_work, rcu_work); krcp = krwp->krcp; + raw_spin_lock_irqsave(&krcp->lock, flags); + /* Channel 1. */ + bkhead = krwp->bkvhead_free[0]; + krwp->bkvhead_free[0] = NULL; + + /* Channel 2. */ + bvhead = krwp->bkvhead_free[1]; + krwp->bkvhead_free[1] = NULL; + + /* Channel 3. */ head = krwp->head_free; krwp->head_free = NULL; - bhead = krwp->bhead_free; - krwp->bhead_free = NULL; raw_spin_unlock_irqrestore(&krcp->lock, flags); - /* "bhead" is now private, so traverse locklessly. */ - for (; bhead; bhead = bnext) { - bnext = bhead->next; - - debug_rcu_bhead_unqueue(bhead); + /* kmalloc()/kfree() channel. */ + for (; bkhead; bkhead = bnext) { + bnext = bkhead->next; + debug_rcu_bhead_unqueue(bkhead); rcu_lock_acquire(&rcu_callback_map); trace_rcu_invoke_kfree_bulk_callback(rcu_state.name, - bhead->nr_records, bhead->records); + bkhead->nr_records, bkhead->records); + + kfree_bulk(bkhead->nr_records, bkhead->records); + rcu_lock_release(&rcu_callback_map); + + krcp = krc_this_cpu_lock(&flags); + if (put_cached_bnode(krcp, bkhead)) + bkhead = NULL; + krc_this_cpu_unlock(krcp, flags); + + if (bkhead) + free_page((unsigned long) bkhead); + + cond_resched_tasks_rcu_qs(); + } + + /* vmalloc()/vfree() channel. */ + for (; bvhead; bvhead = bnext) { + bnext = bvhead->next; + debug_rcu_bhead_unqueue(bvhead); - kfree_bulk(bhead->nr_records, bhead->records); + rcu_lock_acquire(&rcu_callback_map); + for (i = 0; i < bvhead->nr_records; i++) { + trace_rcu_invoke_kfree_callback(rcu_state.name, + (struct rcu_head *) bvhead->records[i], 0); + vfree(bvhead->records[i]); + } rcu_lock_release(&rcu_callback_map); krcp = krc_this_cpu_lock(&flags); - if (put_cached_bnode(krcp, bhead)) - bhead = NULL; + if (put_cached_bnode(krcp, bvhead)) + bvhead = NULL; krc_this_cpu_unlock(krcp, flags); - if (bhead) - free_page((unsigned long) bhead); + if (bvhead) + free_page((unsigned long) bvhead); cond_resched_tasks_rcu_qs(); } @@ -3047,7 +3080,7 @@ static void kfree_rcu_work(struct work_struct *work) trace_rcu_invoke_kfree_callback(rcu_state.name, head, offset); if (!WARN_ON_ONCE(!__is_kfree_rcu_offset(offset))) - kfree(ptr); + kvfree(ptr); rcu_lock_release(&rcu_callback_map); cond_resched_tasks_rcu_qs(); @@ -3072,21 +3105,34 @@ static inline bool queue_kfree_rcu_work(struct kfree_rcu_cpu *krcp) krwp = &(krcp->krw_arr[i]); /* - * Try to detach bhead or head and attach it over any + * Try to detach bkvhead or head and attach it over any * available corresponding free channel. It can be that * a previous RCU batch is in progress, it means that * immediately to queue another one is not possible so * return false to tell caller to retry. */ - if ((krcp->bhead && !krwp->bhead_free) || + if ((krcp->bkvhead[0] && !krwp->bkvhead_free[0]) || + (krcp->bkvhead[1] && !krwp->bkvhead_free[1]) || (krcp->head && !krwp->head_free)) { - /* Channel 1. */ - if (!krwp->bhead_free) { - krwp->bhead_free = krcp->bhead; - krcp->bhead = NULL; + /* + * Channel 1 corresponds to SLAB ptrs. + */ + if (!krwp->bkvhead_free[0]) { + krwp->bkvhead_free[0] = krcp->bkvhead[0]; + krcp->bkvhead[0] = NULL; } - /* Channel 2. */ + /* + * Channel 2 corresponds to vmalloc ptrs. + */ + if (!krwp->bkvhead_free[1]) { + krwp->bkvhead_free[1] = krcp->bkvhead[1]; + krcp->bkvhead[1] = NULL; + } + + /* + * Channel 3 corresponds to emergency path. + */ if (!krwp->head_free) { krwp->head_free = krcp->head; krcp->head = NULL; @@ -3095,16 +3141,17 @@ static inline bool queue_kfree_rcu_work(struct kfree_rcu_cpu *krcp) WRITE_ONCE(krcp->count, 0); /* - * One work is per one batch, so there are two "free channels", - * "bhead_free" and "head_free" the batch can handle. It can be - * that the work is in the pending state when two channels have - * been detached following each other, one by one. + * One work is per one batch, so there are three + * "free channels", the batch can handle. It can + * be that the work is in the pending state when + * channels have been detached following by each + * other. */ queue_rcu_work(system_wq, &krwp->rcu_work); } /* Repeat if any "free" corresponding channel is still busy. */ - if (krcp->bhead || krcp->head) + if (krcp->bkvhead[0] || krcp->bkvhead[1] || krcp->head) repeat = true; } @@ -3146,23 +3193,22 @@ static void kfree_rcu_monitor(struct work_struct *work) } static inline bool -kfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, - struct rcu_head *head, rcu_callback_t func) +kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr) { - struct kfree_rcu_bulk_data *bnode; + struct kvfree_rcu_bulk_data *bnode; + int idx; if (unlikely(!krcp->initialized)) return false; lockdep_assert_held(&krcp->lock); + idx = !!is_vmalloc_addr(ptr); /* Check if a new block is required. */ - if (!krcp->bhead || - krcp->bhead->nr_records == KFREE_BULK_MAX_ENTR) { + if (!krcp->bkvhead[idx] || + krcp->bkvhead[idx]->nr_records == KVFREE_BULK_MAX_ENTR) { bnode = get_cached_bnode(krcp); if (!bnode) { - WARN_ON_ONCE(sizeof(struct kfree_rcu_bulk_data) > PAGE_SIZE); - /* * To keep this path working on raw non-preemptible * sections, prevent the optional entry into the @@ -3175,7 +3221,7 @@ kfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, if (IS_ENABLED(CONFIG_PREEMPT_RT)) return false; - bnode = (struct kfree_rcu_bulk_data *) + bnode = (struct kvfree_rcu_bulk_data *) __get_free_page(GFP_NOWAIT | __GFP_NOWARN); } @@ -3185,30 +3231,30 @@ kfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, /* Initialize the new block. */ bnode->nr_records = 0; - bnode->next = krcp->bhead; + bnode->next = krcp->bkvhead[idx]; /* Attach it to the head. */ - krcp->bhead = bnode; + krcp->bkvhead[idx] = bnode; } /* Finally insert. */ - krcp->bhead->records[krcp->bhead->nr_records++] = - (void *) head - (unsigned long) func; + krcp->bkvhead[idx]->records + [krcp->bkvhead[idx]->nr_records++] = ptr; return true; } /* - * Queue a request for lazy invocation of kfree_bulk()/kfree() after a grace - * period. Please note there are two paths are maintained, one is the main one - * that uses kfree_bulk() interface and second one is emergency one, that is - * used only when the main path can not be maintained temporary, due to memory - * pressure. + * Queue a request for lazy invocation of appropriate free routine after a + * grace period. Please note there are three paths are maintained, two are the + * main ones that use array of pointers interface and third one is emergency + * one, that is used only when the main path can not be maintained temporary, + * due to memory pressure. * * Each kfree_call_rcu() request is added to a batch. The batch will be drained * every KFREE_DRAIN_JIFFIES number of jiffies. All the objects in the batch will * be free'd in workqueue context. This allows us to: batch requests together to - * reduce the number of grace periods during heavy kfree_rcu() load. + * reduce the number of grace periods during heavy kfree_rcu()/kvfree_rcu() load. */ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func) { @@ -3231,7 +3277,7 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func) * Under high memory pressure GFP_NOWAIT can fail, * in that case the emergency path is maintained. */ - if (unlikely(!kfree_call_rcu_add_ptr_to_bulk(krcp, head, func))) { + if (unlikely(!kvfree_call_rcu_add_ptr_to_bulk(krcp, ptr))) { head->func = func; head->next = krcp->head; krcp->head = head; @@ -4212,7 +4258,7 @@ static void __init kfree_rcu_batch_init(void) for_each_possible_cpu(cpu) { struct kfree_rcu_cpu *krcp = per_cpu_ptr(&krc, cpu); - struct kfree_rcu_bulk_data *bnode; + struct kvfree_rcu_bulk_data *bnode; for (i = 0; i < KFREE_N_BATCHES; i++) { INIT_RCU_WORK(&krcp->krw_arr[i].rcu_work, kfree_rcu_work); @@ -4220,7 +4266,7 @@ static void __init kfree_rcu_batch_init(void) } for (i = 0; i < rcu_min_cached_objs; i++) { - bnode = (struct kfree_rcu_bulk_data *) + bnode = (struct kvfree_rcu_bulk_data *) __get_free_page(GFP_NOWAIT | __GFP_NOWARN); if (bnode) From patchwork Tue Apr 28 20:58:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11515577 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AC1CB14DD for ; Tue, 28 Apr 2020 20:59:57 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6F1CA20775 for ; Tue, 28 Apr 2020 20:59:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="a6WjH7SP" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6F1CA20775 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6A43E8E0011; Tue, 28 Apr 2020 16:59:45 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 605AB8E0007; Tue, 28 Apr 2020 16:59:45 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4A8E68E0011; Tue, 28 Apr 2020 16:59:45 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 29A608E0007 for ; Tue, 28 Apr 2020 16:59:45 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id F1A585831 for ; Tue, 28 Apr 2020 20:59:44 +0000 (UTC) X-FDA: 76758480288.07.waste97_55160c4a13257 X-Spam-Summary: 2,0,0,0ac736f427abdcbe,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:41:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1540:1711:1714:1730:1747:1777:1792:2196:2199:2393:2559:2562:3138:3139:3140:3141:3142:3351:3867:3870:3872:4250:4385:5007:6261:6653:7514:7875:9413:10004:11026:11658:11914:12043:12296:12297:12438:12517:12519:12555:12895:13069:13311:13357:13894:14096:14181:14384:14394:14687:14721:21080:21444:21451:21627:21666:21987:21990:30054,0,RBL:209.85.208.194:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:1,LUA_SUMMARY:none X-HE-Tag: waste97_55160c4a13257 X-Filterd-Recvd-Size: 4125 Received: from mail-lj1-f194.google.com (mail-lj1-f194.google.com [209.85.208.194]) by imf39.hostedemail.com (Postfix) with ESMTP for ; Tue, 28 Apr 2020 20:59:44 +0000 (UTC) Received: by mail-lj1-f194.google.com with SMTP id f18so181131lja.13 for ; Tue, 28 Apr 2020 13:59:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=OuCOh3fsoWi6ozxFbpjNX5HY0vVVouy9K1NPB0pENcE=; b=a6WjH7SPzi25N3JUy2dkzivo4WGLRcCpkpzaqYd6Fw0ka1nWZ/UmlbhMBdnhfVQZYB HNBoFRwvjYE2EnmhQFmBD1dpL5CQuXbjQdRsGrIa4lkDWaJLuS69980noZub88/DhT1O qzO1eDhPuABdmhTdtRkUmQNtAZZAQtD2Z5G+BTVuESXENUVSC3W9Tu6W2wGy3d4KjRmS LX18i7FFkX5xC3XqnidCDkeJ5qNyHJxcVU1lclTp+09yAtpHLWXQFx+XJTqjf2H0WlCE W+Td6Bmf8S8OSSLGjZNe/shh34hNqnCMEUxV9L5lnNBNUgbUPwVTrRQ1BEsme+NE0eQB bJjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=OuCOh3fsoWi6ozxFbpjNX5HY0vVVouy9K1NPB0pENcE=; b=Mjcj+ZDO7CkUguQXsfYS4U9tNXM5d9MXuxt6oNQ8qt8Y3ys2fqdHFm9sV2WvG5gzmI PzIMtCnbLhQKquFjDVFTunuM+xAq33nOgekfKUSEXYMkE9L8rdcCu3vYFvIqglP3lPcJ 9+NiLoGfVQmjsVcco4mgbKXKdt18vVmm3b+f0ty1Njxs+308M6TRZ0o7pkGG8t+G2ucK oUZmLn5OGCs8N2AoWXHdNZcrrysPJ0KNJ38kH/N0f6K8zL/ujBUa3urmv4afF4MboE8W aQHTkWGtU4xFZElBWV/sRswcz+0SyHWu56w7x/WP5PIGwOVjGUk9B5uhNgQ2N2563Lqp TqnA== X-Gm-Message-State: AGi0PuZs2hzSfjFWio2nedwNpMULO735mQWbOe1tQ3uiSwTDeAhrT6u7 CTJxEbCf9qGvXd5p8TwFdDg= X-Google-Smtp-Source: APiQypLj2b3ByApuoJPSgTS0qYmKq2vkv6PNL4ow5it/sNzKhbIbo7wxp7EgYlwk1Y9ymdWadjFIbQ== X-Received: by 2002:a2e:9cd:: with SMTP id 196mr3759962ljj.226.1588107583396; Tue, 28 Apr 2020 13:59:43 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id z21sm295483ljh.42.2020.04.28.13.59.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Apr 2020 13:59:42 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 12/24] rcu/tiny: support vmalloc in tiny-RCU Date: Tue, 28 Apr 2020 22:58:51 +0200 Message-Id: <20200428205903.61704-13-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200428205903.61704-1-urezki@gmail.com> References: <20200428205903.61704-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Replace kfree() with kvfree() in rcu_reclaim_tiny(). So it becomes possible to release either SLAB memory or vmalloc one after a GP. Signed-off-by: Uladzislau Rezki (Sony) --- kernel/rcu/tiny.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c index dd572ce7c747..4b99f7b88bee 100644 --- a/kernel/rcu/tiny.c +++ b/kernel/rcu/tiny.c @@ -23,6 +23,7 @@ #include #include #include +#include #include "rcu.h" @@ -86,7 +87,7 @@ static inline bool rcu_reclaim_tiny(struct rcu_head *head) rcu_lock_acquire(&rcu_callback_map); if (__is_kfree_rcu_offset(offset)) { trace_rcu_invoke_kfree_callback("", head, offset); - kfree((void *)head - offset); + kvfree((void *)head - offset); rcu_lock_release(&rcu_callback_map); return true; } From patchwork Tue Apr 28 20:58:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11515579 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1AA631805 for ; Tue, 28 Apr 2020 21:00:00 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CF63920775 for ; Tue, 28 Apr 2020 20:59:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="kP02x+M/" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CF63920775 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id BD2C08E0012; Tue, 28 Apr 2020 16:59:46 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B83DB8E0007; Tue, 28 Apr 2020 16:59:46 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A990F8E0012; Tue, 28 Apr 2020 16:59:46 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0124.hostedemail.com [216.40.44.124]) by kanga.kvack.org (Postfix) with ESMTP id 7E8F98E0007 for ; Tue, 28 Apr 2020 16:59:46 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 4A629181AC9CB for ; Tue, 28 Apr 2020 20:59:46 +0000 (UTC) X-FDA: 76758480372.30.table07_554698de62b52 X-Spam-Summary: 2,0,0,6bc14bedb6a14386,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:41:355:379:541:800:960:965:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1543:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:2907:3138:3139:3140:3141:3142:3355:3865:3866:3867:3868:3871:4118:4250:4321:4385:4390:4395:4605:5007:6119:6261:6653:7514:7875:9413:10004:11026:11473:11658:11914:12043:12114:12296:12297:12438:12517:12519:12555:12895:12986:13894:13972:14181:14394:14687:14721:21080:21212:21444:21451:21627:21666:21740:21772:21987:21990:30034:30054:30070:30076,0,RBL:209.85.208.194:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.18.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:1,LUA_SUMMARY:none X-HE-Tag: table07_554698de62b52 X-Filterd-Recvd-Size: 7206 Received: from mail-lj1-f194.google.com (mail-lj1-f194.google.com [209.85.208.194]) by imf40.hostedemail.com (Postfix) with ESMTP for ; Tue, 28 Apr 2020 20:59:45 +0000 (UTC) Received: by mail-lj1-f194.google.com with SMTP id j3so210796ljg.8 for ; Tue, 28 Apr 2020 13:59:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=3fJ5TMdCQebVw9K/lB+iz58HUNfGtYvyfl9ym54rQlM=; b=kP02x+M/0VMqRYPckvkQ5KHhPgnUW+5je7IT6oUunk7FfGrXaOci+I2mXWMDeyhOV+ KCxCW5+E3z8CDdniIsgjw+Ujmj6woV9U2vpX7RROlqU9D0LbWlJfs+R59GhHyvBsxF+I L47755O8cNu7u2Z/RS+DscHR3jg7t2kEgk2+xKx+IeZvovvdyET6gUnF4P4jSVhGfSNE v+b+D8rAkEN2F79mxfA/MHMqmjojM/AFxTmpVrpecE23T7JYANaUUoy818jZ50BZmWok ge+DSSWtnaH79mXjgMAsH8fLpNXfKya00SxR3Ef21hbEWNN5D2wi/rAOzhUnL8HEykqb fE/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=3fJ5TMdCQebVw9K/lB+iz58HUNfGtYvyfl9ym54rQlM=; b=BYuNnjRVzDEtXoXxNHWSFls2p4fM9jUQHB3gzqmYT5ditVtwT89DLFx/X4GrVs/vAY nDy3xTnJvE7UzazU1rJrxktR+wGK2l7GzZ2MfQ9pL3zm1hWnaeiKW2PsyPd4FBK4qO5E RzxT+ctZCCRdV9grIW9PtgazUt0cvSyyvk7XZSyaQoPuu+Acl2Aj7AjYX9LUn0Y1KQij tHW4RNghuypujvmLk22Al2uj5W/Y2pEXKLT2e/RrQslXWhCtwaAHbXAet77OKVUtLHFN vqViuC8le8D8BlWRgvdpgnTVG1i+vzQYujkaYtx6dUZTGiPgV4l0djnELtSB3G+cI40g Edzw== X-Gm-Message-State: AGi0PuYqMiAse0xrvFN/jGv6pEMsFFGhKMhftiBll1NYPlkNt6O0dEif WHMsI240/VqPDvirz+PEs5Q= X-Google-Smtp-Source: APiQypLFK7pLf6v4TBbEd8q0pQMGD3hqv0CAeV1BrlcjzHb6iUbngmOpghi/7W8+zKnNzxEpk2jThw== X-Received: by 2002:a05:651c:2002:: with SMTP id s2mr17409411ljo.285.1588107584561; Tue, 28 Apr 2020 13:59:44 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id z21sm295483ljh.42.2020.04.28.13.59.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Apr 2020 13:59:43 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 13/24] rcu: Rename rcu_invoke_kfree_callback/rcu_kfree_callback Date: Tue, 28 Apr 2020 22:58:52 +0200 Message-Id: <20200428205903.61704-14-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200428205903.61704-1-urezki@gmail.com> References: <20200428205903.61704-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Rename rcu_invoke_kfree_callback to rcu_invoke_kvfree_callback. Do the same with second trace event, the rcu_kfree_callback, becomes rcu_kvfree_callback. The reason is to be aligned with kvfree notation. Signed-off-by: Uladzislau Rezki (Sony) Reviewed-by: Joel Fernandes (Google) Signed-off-by: Joel Fernandes (Google) --- include/trace/events/rcu.h | 8 ++++---- kernel/rcu/tiny.c | 2 +- kernel/rcu/tree.c | 6 +++--- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/include/trace/events/rcu.h b/include/trace/events/rcu.h index f9a7811148e2..0ee93d0b1daa 100644 --- a/include/trace/events/rcu.h +++ b/include/trace/events/rcu.h @@ -506,13 +506,13 @@ TRACE_EVENT_RCU(rcu_callback, /* * Tracepoint for the registration of a single RCU callback of the special - * kfree() form. The first argument is the RCU type, the second argument + * kvfree() form. The first argument is the RCU type, the second argument * is a pointer to the RCU callback, the third argument is the offset * of the callback within the enclosing RCU-protected data structure, * the fourth argument is the number of lazy callbacks queued, and the * fifth argument is the total number of callbacks queued. */ -TRACE_EVENT_RCU(rcu_kfree_callback, +TRACE_EVENT_RCU(rcu_kvfree_callback, TP_PROTO(const char *rcuname, struct rcu_head *rhp, unsigned long offset, long qlen), @@ -596,12 +596,12 @@ TRACE_EVENT_RCU(rcu_invoke_callback, /* * Tracepoint for the invocation of a single RCU callback of the special - * kfree() form. The first argument is the RCU flavor, the second + * kvfree() form. The first argument is the RCU flavor, the second * argument is a pointer to the RCU callback, and the third argument * is the offset of the callback within the enclosing RCU-protected * data structure. */ -TRACE_EVENT_RCU(rcu_invoke_kfree_callback, +TRACE_EVENT_RCU(rcu_invoke_kvfree_callback, TP_PROTO(const char *rcuname, struct rcu_head *rhp, unsigned long offset), diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c index 4b99f7b88bee..3dd8e6e207b0 100644 --- a/kernel/rcu/tiny.c +++ b/kernel/rcu/tiny.c @@ -86,7 +86,7 @@ static inline bool rcu_reclaim_tiny(struct rcu_head *head) rcu_lock_acquire(&rcu_callback_map); if (__is_kfree_rcu_offset(offset)) { - trace_rcu_invoke_kfree_callback("", head, offset); + trace_rcu_invoke_kvfree_callback("", head, offset); kvfree((void *)head - offset); rcu_lock_release(&rcu_callback_map); return true; diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 7983926af95b..821de8149928 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2794,7 +2794,7 @@ __call_rcu(struct rcu_head *head, rcu_callback_t func) // If no-CBs CPU gets here, rcu_nocb_try_bypass() acquired ->nocb_lock. rcu_segcblist_enqueue(&rdp->cblist, head); if (__is_kfree_rcu_offset((unsigned long)func)) - trace_rcu_kfree_callback(rcu_state.name, head, + trace_rcu_kvfree_callback(rcu_state.name, head, (unsigned long)func, rcu_segcblist_n_cbs(&rdp->cblist)); else @@ -3048,7 +3048,7 @@ static void kfree_rcu_work(struct work_struct *work) rcu_lock_acquire(&rcu_callback_map); for (i = 0; i < bvhead->nr_records; i++) { - trace_rcu_invoke_kfree_callback(rcu_state.name, + trace_rcu_invoke_kvfree_callback(rcu_state.name, (struct rcu_head *) bvhead->records[i], 0); vfree(bvhead->records[i]); } @@ -3077,7 +3077,7 @@ static void kfree_rcu_work(struct work_struct *work) next = head->next; debug_rcu_head_unqueue((struct rcu_head *)ptr); rcu_lock_acquire(&rcu_callback_map); - trace_rcu_invoke_kfree_callback(rcu_state.name, head, offset); + trace_rcu_invoke_kvfree_callback(rcu_state.name, head, offset); if (!WARN_ON_ONCE(!__is_kfree_rcu_offset(offset))) kvfree(ptr); From patchwork Tue Apr 28 20:58:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11515581 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8173A112C for ; Tue, 28 Apr 2020 21:00:02 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3053B2186A for ; Tue, 28 Apr 2020 21:00:02 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="s5+emd78" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3053B2186A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0FE568E0013; Tue, 28 Apr 2020 16:59:48 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 08A2A8E0007; Tue, 28 Apr 2020 16:59:47 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DF8EB8E0013; Tue, 28 Apr 2020 16:59:47 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0025.hostedemail.com [216.40.44.25]) by kanga.kvack.org (Postfix) with ESMTP id B3B188E0007 for ; Tue, 28 Apr 2020 16:59:47 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 78B1952CC for ; Tue, 28 Apr 2020 20:59:47 +0000 (UTC) X-FDA: 76758480414.15.rifle26_5575b1d259625 X-Spam-Summary: 40,2.5,0,6eb44055ce62cfae,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:41:355:379:541:800:960:965:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1542:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:3138:3139:3140:3141:3142:3352:3865:3866:3867:3872:3874:4117:4250:4321:4385:4390:4395:4605:5007:6119:6261:6653:7514:7875:7903:9413:10011:11026:11232:11473:11658:11914:12043:12114:12296:12297:12438:12517:12519:12555:12895:12986:13894:14096:14181:14394:14687:14721:21080:21433:21444:21451:21627:21666:21740:21772:21987:21990:30054,0,RBL:209.85.208.196:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:1:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: rifle26_5575b1d259625 X-Filterd-Recvd-Size: 6094 Received: from mail-lj1-f196.google.com (mail-lj1-f196.google.com [209.85.208.196]) by imf27.hostedemail.com (Postfix) with ESMTP for ; Tue, 28 Apr 2020 20:59:47 +0000 (UTC) Received: by mail-lj1-f196.google.com with SMTP id u6so221558ljl.6 for ; Tue, 28 Apr 2020 13:59:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=NnJAtzNPQwOuN35prMWYFplXmKr7NAfEhiUXNxLclG8=; b=s5+emd78kekw/TXnvyA/bAOGc0SGZJAVB/4WGiYByfbpRI2x2pn3OrAbtzIvYhDPjg Xn63HYYc1XoBn8Jw9nvyZZH6daZ4v6723VaraNGm/RUwC5omukWIHw46Zg1xWaUcGBNZ oWnGge8C2RJuhXkJKzxwv5ACwD18xJChSmTWAaQuteDz8wCucOZYeCqGzU+ew0ixZ70z +kd1806zFpfw7lsZtANv/4OyHov/Wf/l+TUIGtjCkSIFKvQqB6vbiM8TlUb6X74bPQ1U WNupVjZ7RXbsPY5YXgkc8vGnbMdvna2JCxbwpfg75EvHH9ftEwW/NUBpxR8OWCNq0wzp //nQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NnJAtzNPQwOuN35prMWYFplXmKr7NAfEhiUXNxLclG8=; b=e0Bv/LYN4L+llbxh30JdgumGVDafmdTvi/wrOliMRJ98ILAdld1pbj2kqFmlxA0AgE ihbuh+5yaUD1AF9btLF+LLAoZhMThJI1x+5pVyPHTm++upNGYzs3VCpbc2/Q2nH9YNwC t80ppon0xM0Dby7o5YEP3q2WmXcVsTYQeIGq3W7ZDlYIHtXkw7jwver71HfR4TwxNP2j MBIRuhtCuXy1KjCTizDCr0tiH1zIs1HtJo0+LKToGXzoSZ6FGs9Zs/PG+e1CXV+54b4o T7OL1b5OpvwGmtz0mzL2CX+R+uZLxBBel25Kk0BOeHJhvbxLgH7xhnifdTtIgXla6cs1 crrA== X-Gm-Message-State: AGi0PuYQLbh+aKcO05ELBwC1IB3h+m6J/VLVXzPYSdhUR44mIq92+CxE NCH2f9ijffS1qYM9LVf8xrY= X-Google-Smtp-Source: APiQypIVpmrbm6mASEgB0xwC50tr3MbUHiwxa5AkAqbG8qXM4FhLBQercE0GBtCprGgQO16dDjqtNQ== X-Received: by 2002:a2e:5813:: with SMTP id m19mr10296851ljb.230.1588107585895; Tue, 28 Apr 2020 13:59:45 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id z21sm295483ljh.42.2020.04.28.13.59.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Apr 2020 13:59:45 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 14/24] rcu: Rename __is_kfree_rcu_offset() macro Date: Tue, 28 Apr 2020 22:58:53 +0200 Message-Id: <20200428205903.61704-15-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200428205903.61704-1-urezki@gmail.com> References: <20200428205903.61704-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Rename __is_kfree_rcu_offset to __is_kvfree_rcu_offset. All RCU paths use kvfree() now instead of kfree(), thus rename it. Signed-off-by: Uladzislau Rezki (Sony) Reviewed-by: Joel Fernandes (Google) Signed-off-by: Joel Fernandes (Google) --- include/linux/rcupdate.h | 6 +++--- kernel/rcu/tiny.c | 2 +- kernel/rcu/tree.c | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 659cbfa7581a..1d25e6c23ebd 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -828,16 +828,16 @@ static inline notrace void rcu_read_unlock_sched_notrace(void) /* * Does the specified offset indicate that the corresponding rcu_head - * structure can be handled by kfree_rcu()? + * structure can be handled by kvfree_rcu()? */ -#define __is_kfree_rcu_offset(offset) ((offset) < 4096) +#define __is_kvfree_rcu_offset(offset) ((offset) < 4096) /* * Helper macro for kfree_rcu() to prevent argument-expansion eyestrain. */ #define __kfree_rcu(head, offset) \ do { \ - BUILD_BUG_ON(!__is_kfree_rcu_offset(offset)); \ + BUILD_BUG_ON(!__is_kvfree_rcu_offset(offset)); \ kfree_call_rcu(head, (rcu_callback_t)(unsigned long)(offset)); \ } while (0) diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c index 3dd8e6e207b0..aa897c3f2e92 100644 --- a/kernel/rcu/tiny.c +++ b/kernel/rcu/tiny.c @@ -85,7 +85,7 @@ static inline bool rcu_reclaim_tiny(struct rcu_head *head) unsigned long offset = (unsigned long)head->func; rcu_lock_acquire(&rcu_callback_map); - if (__is_kfree_rcu_offset(offset)) { + if (__is_kvfree_rcu_offset(offset)) { trace_rcu_invoke_kvfree_callback("", head, offset); kvfree((void *)head - offset); rcu_lock_release(&rcu_callback_map); diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 821de8149928..5f53368f2554 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2793,7 +2793,7 @@ __call_rcu(struct rcu_head *head, rcu_callback_t func) return; // Enqueued onto ->nocb_bypass, so just leave. // If no-CBs CPU gets here, rcu_nocb_try_bypass() acquired ->nocb_lock. rcu_segcblist_enqueue(&rdp->cblist, head); - if (__is_kfree_rcu_offset((unsigned long)func)) + if (__is_kvfree_rcu_offset((unsigned long)func)) trace_rcu_kvfree_callback(rcu_state.name, head, (unsigned long)func, rcu_segcblist_n_cbs(&rdp->cblist)); @@ -3079,7 +3079,7 @@ static void kfree_rcu_work(struct work_struct *work) rcu_lock_acquire(&rcu_callback_map); trace_rcu_invoke_kvfree_callback(rcu_state.name, head, offset); - if (!WARN_ON_ONCE(!__is_kfree_rcu_offset(offset))) + if (!WARN_ON_ONCE(!__is_kvfree_rcu_offset(offset))) kvfree(ptr); rcu_lock_release(&rcu_callback_map); From patchwork Tue Apr 28 20:58:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11515583 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DAC791805 for ; Tue, 28 Apr 2020 21:00:04 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9169E20775 for ; Tue, 28 Apr 2020 21:00:04 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="B7vt3KAj" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9169E20775 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 654D18E0014; Tue, 28 Apr 2020 16:59:49 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6065B8E0007; Tue, 28 Apr 2020 16:59:49 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 432CA8E0014; Tue, 28 Apr 2020 16:59:49 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0212.hostedemail.com [216.40.44.212]) by kanga.kvack.org (Postfix) with ESMTP id 276F08E0007 for ; Tue, 28 Apr 2020 16:59:49 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id D9D6A2C8B for ; Tue, 28 Apr 2020 20:59:48 +0000 (UTC) X-FDA: 76758480456.11.grip70_55a67e09d1f51 X-Spam-Summary: 2,0,0,8c5ced6be5b54fc8,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:41:355:379:421:541:800:960:965:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1544:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2559:2562:2693:2731:2914:3138:3139:3140:3141:3142:3355:3622:3865:3866:3867:3870:3871:3872:3873:3874:4118:4321:4385:4390:4395:4605:5007:6261:6653:7514:9413:10004:11026:11473:11658:11914:12043:12296:12297:12438:12517:12519:12555:12895:12986:13161:13229:13894:14096:14181:14394:14687:14721:21080:21212:21433:21444:21451:21627:21666:21740:30034:30054,0,RBL:209.85.208.193:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: grip70_55a67e09d1f51 X-Filterd-Recvd-Size: 7531 Received: from mail-lj1-f193.google.com (mail-lj1-f193.google.com [209.85.208.193]) by imf34.hostedemail.com (Postfix) with ESMTP for ; Tue, 28 Apr 2020 20:59:48 +0000 (UTC) Received: by mail-lj1-f193.google.com with SMTP id j3so211005ljg.8 for ; Tue, 28 Apr 2020 13:59:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ZcJCHqE992p8lsmYiNcd24tMBqJtWQITQ3c5M8uA1SE=; b=B7vt3KAj4STTD+aKk+oKLMBs8qtai99+vCn8DipCi33bQwwq8BigFQl81rlWBL8xxm 5XMRocUMMayL6zNBfm8OnD5TDM6DSKI0IpCsuXRept5MT/XdGUKtiPMkbvt60gd5dq7O J1fDvz0SaOMR9WdM0zfuglp5AG/s4mH9e85cU5CT18sNbyTiVWN7lhYDQVVQ3CDH1ehB X4ON7co7BedfN0OLXCG5hwNQabjo2tdqaRqOpbKUSSrYTH6tV9P1jPvDlf+sLgzoENaB T4kOKoIQxciE6zMh912qt9MwSEmpnpTfeuxC4eTn0tSzskHowIUzt3PV0p5W3M82w6MV 5+jQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ZcJCHqE992p8lsmYiNcd24tMBqJtWQITQ3c5M8uA1SE=; b=WUE/b3wQGCs2+5tjsp5y+trs48l5+i7wUlEJxMgxp9QW/nIGvkhiTkbyWgAzo+7wpd 9Mg5lr6idK/4hJwO7vjLfHRm4oitO76u7+oK5br2RlMhEF5nEPpWnavKWffBvWrNGbXJ YMgdvr0XHvFiLncn8HSNR9VJuKrD1VU8kJVOgHGXNx33zAsfg7FIWmoZ+uA/O7JYEymq wsEvuPuT7ie8Hu1UlHPWc6M7KsD8xJkSV8Yj+stOVnVH+FrYfGi3IPqBO2NBP6l26kPf aP+QOJBKxKMip5NLs8Ije4G5WhCMbTRVND6AUp76eoIeS0aPlIMtJVHua7Ht0+C2nc0+ yyrQ== X-Gm-Message-State: AGi0Pua0MlaI/4RIqgDwuqWpECauhiWDly3douGN62BV4ayA7KwIuRO4 x9E5JLVcBpYRLOJ+83KhQR4= X-Google-Smtp-Source: APiQypIZsg28yYvjCdQRmrdaF5z9OdGgbIG1meF5iEoqlsZSntvvbPCTGsJPR0M5i7zF9h+4vPIP2A== X-Received: by 2002:a2e:9a82:: with SMTP id p2mr19047796lji.279.1588107587186; Tue, 28 Apr 2020 13:59:47 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id z21sm295483ljh.42.2020.04.28.13.59.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Apr 2020 13:59:46 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 15/24] rcu: Rename kfree_call_rcu() to the kvfree_call_rcu(). Date: Tue, 28 Apr 2020 22:58:54 +0200 Message-Id: <20200428205903.61704-16-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200428205903.61704-1-urezki@gmail.com> References: <20200428205903.61704-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The reason is, it is capable of freeing vmalloc() memory now. Do the same with __kfree_rcu() macro, it becomes __kvfree_rcu(), the reason is the same as pointed above. Signed-off-by: Uladzislau Rezki (Sony) Reviewed-by: Joel Fernandes (Google) Signed-off-by: Joel Fernandes (Google) --- include/linux/rcupdate.h | 8 ++++---- include/linux/rcutiny.h | 2 +- include/linux/rcutree.h | 2 +- kernel/rcu/tree.c | 6 +++--- 4 files changed, 9 insertions(+), 9 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 1d25e6c23ebd..b344fc800a9b 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -835,10 +835,10 @@ static inline notrace void rcu_read_unlock_sched_notrace(void) /* * Helper macro for kfree_rcu() to prevent argument-expansion eyestrain. */ -#define __kfree_rcu(head, offset) \ +#define __kvfree_rcu(head, offset) \ do { \ BUILD_BUG_ON(!__is_kvfree_rcu_offset(offset)); \ - kfree_call_rcu(head, (rcu_callback_t)(unsigned long)(offset)); \ + kvfree_call_rcu(head, (rcu_callback_t)(unsigned long)(offset)); \ } while (0) /** @@ -857,7 +857,7 @@ static inline notrace void rcu_read_unlock_sched_notrace(void) * Because the functions are not allowed in the low-order 4096 bytes of * kernel virtual memory, offsets up to 4095 bytes can be accommodated. * If the offset is larger than 4095 bytes, a compile-time error will - * be generated in __kfree_rcu(). If this error is triggered, you can + * be generated in __kvfree_rcu(). If this error is triggered, you can * either fall back to use of call_rcu() or rearrange the structure to * position the rcu_head structure into the first 4096 bytes. * @@ -872,7 +872,7 @@ do { \ typeof (ptr) ___p = (ptr); \ \ if (___p) \ - __kfree_rcu(&((___p)->rhf), offsetof(typeof(*(ptr)), rhf)); \ + __kvfree_rcu(&((___p)->rhf), offsetof(typeof(*(ptr)), rhf)); \ } while (0) /* diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h index 3465ba704a11..0c6315c4a0fe 100644 --- a/include/linux/rcutiny.h +++ b/include/linux/rcutiny.h @@ -34,7 +34,7 @@ static inline void synchronize_rcu_expedited(void) synchronize_rcu(); } -static inline void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func) +static inline void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) { call_rcu(head, func); } diff --git a/include/linux/rcutree.h b/include/linux/rcutree.h index fbc26274af4d..4d2732442013 100644 --- a/include/linux/rcutree.h +++ b/include/linux/rcutree.h @@ -33,7 +33,7 @@ static inline void rcu_virt_note_context_switch(int cpu) } void synchronize_rcu_expedited(void); -void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func); +void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func); void rcu_barrier(void); bool rcu_eqs_special_set(int cpu); diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 5f53368f2554..51726e4c3b4d 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3251,12 +3251,12 @@ kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr) * one, that is used only when the main path can not be maintained temporary, * due to memory pressure. * - * Each kfree_call_rcu() request is added to a batch. The batch will be drained + * Each kvfree_call_rcu() request is added to a batch. The batch will be drained * every KFREE_DRAIN_JIFFIES number of jiffies. All the objects in the batch will * be free'd in workqueue context. This allows us to: batch requests together to * reduce the number of grace periods during heavy kfree_rcu()/kvfree_rcu() load. */ -void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func) +void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) { unsigned long flags; struct kfree_rcu_cpu *krcp; @@ -3295,7 +3295,7 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func) unlock_return: krc_this_cpu_unlock(krcp, flags); } -EXPORT_SYMBOL_GPL(kfree_call_rcu); +EXPORT_SYMBOL_GPL(kvfree_call_rcu); static unsigned long kfree_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) From patchwork Tue Apr 28 20:58:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11515585 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 68DA292C for ; Tue, 28 Apr 2020 21:00:07 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2C2B32072A for ; Tue, 28 Apr 2020 21:00:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="juHF/lso" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2C2B32072A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D80848E0015; Tue, 28 Apr 2020 16:59:50 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D32778E0007; Tue, 28 Apr 2020 16:59:50 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BD1008E0015; Tue, 28 Apr 2020 16:59:50 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0159.hostedemail.com [216.40.44.159]) by kanga.kvack.org (Postfix) with ESMTP id 9ED5A8E0007 for ; Tue, 28 Apr 2020 16:59:50 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 4890C181AC9CB for ; Tue, 28 Apr 2020 20:59:50 +0000 (UTC) X-FDA: 76758480540.17.sound24_55dc429a0c50b X-Spam-Summary: 2,0,0,01b49194acfb5b2c,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:41:355:379:541:800:960:965:966:968:973:988:989:1260:1311:1314:1345:1359:1431:1437:1515:1534:1541:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:3138:3139:3140:3141:3142:3352:3865:3867:3868:3870:3871:3872:3874:4250:4385:4390:4395:5007:6261:6653:7514:9413:10004:11026:11658:11914:12043:12296:12297:12438:12517:12519:12555:12895:13069:13311:13357:13846:13894:13972:14096:14181:14384:14394:14687:14721:21080:21444:21451:21627:21666:21990:30054,0,RBL:209.85.208.193:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.18.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: sound24_55dc429a0c50b X-Filterd-Recvd-Size: 4849 Received: from mail-lj1-f193.google.com (mail-lj1-f193.google.com [209.85.208.193]) by imf38.hostedemail.com (Postfix) with ESMTP for ; Tue, 28 Apr 2020 20:59:49 +0000 (UTC) Received: by mail-lj1-f193.google.com with SMTP id w20so269261ljj.0 for ; Tue, 28 Apr 2020 13:59:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Asw9tqE4/cLTmlTUoE8ln3vIOlzdWNXgzJMQ+xZMncE=; b=juHF/lso7CN/H2XZntLeP+Irbj7xZN6Dce9JzBFxRojjPE6dBRALNOQJr5cKWc8VCl GAJBSsQ/YZEVC3bE5B+0VvqbSTECPI8HsuhxuAafleWUngRE8nj5+or1xX4bH7xSOq/X td1GjGUpOXtUE5ViUGkt+QVERAU9/smj4Mj0lZn+dw0VyNEX7j9PjFHuV53aVDtF27yS Ah8w+nux9V5kT+I6bHmxQEjUyGfSOmoJMaETxviC6ikjLYdJn66EmmOiaG/GNF+No0vt saxA/aJO4TdWTGgd1ftgsFZh118rD7Ly4/ZSPP+NDfkIJSMjum6fhzFsojrRQt6CDdjJ Zzlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Asw9tqE4/cLTmlTUoE8ln3vIOlzdWNXgzJMQ+xZMncE=; b=YK3t6yU1yg1lCpqzR896UgJ4uDzrL401rtzEp5pIRaPepgGTC6r0Mi/H6NWLAAzDKA X944BgF+cWM4qz7oV3rPeztp7AC7CVGI5bFPSKY5JGUEfjsPOHbZvZvwyTmGn5Atr4CV M089GO2ksGho4ByoPk3NUj7fBJmt5xdIOk5iObaKODQBwUnhM4sC60HDKdjza2hRCisj Rn2acD7gv+xHB2SzczlXOXPp2IWS2hWs2uv0AbBfe9odAlNmRwJ9AyJjA0yu0iBs2VbI wN7d0a/bnuauKBGR2dGFnWfKwkHrbnvElsUCAM7VTMv00R27mLKJdMVSqR2Z12Qe5Sx0 3HrA== X-Gm-Message-State: AGi0Puaf+BuknzNMxukEd5Jjf4MAQ2IGYRvI/YyI+wTPUbfSCuRESOi3 oXF0V1n7VO4bR7CjovIX6DA= X-Google-Smtp-Source: APiQypKV+6RjXJzSL+v6wXMSNXoSgBbylgy6V8bSx38gvWAORGi68jbgQU0ttvjTYKtEMPZIu8JiJw== X-Received: by 2002:a05:651c:549:: with SMTP id q9mr14774417ljp.236.1588107588713; Tue, 28 Apr 2020 13:59:48 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id z21sm295483ljh.42.2020.04.28.13.59.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Apr 2020 13:59:48 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 16/24] mm/list_lru.c: Rename kvfree_rcu() to local variant Date: Tue, 28 Apr 2020 22:58:55 +0200 Message-Id: <20200428205903.61704-17-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200428205903.61704-1-urezki@gmail.com> References: <20200428205903.61704-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Rename kvfree_rcu() function to the kvfree_rcu_local() one. The aim is to introduce the public API that would conflict with this one. So we temporarily rename it and remove it in a later commit. Cc: linux-mm@kvack.org Cc: Andrew Morton Cc: rcu@vger.kernel.org Signed-off-by: Uladzislau Rezki (Sony) Reviewed-by: Joel Fernandes (Google) Signed-off-by: Joel Fernandes (Google) --- mm/list_lru.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/list_lru.c b/mm/list_lru.c index 4d5294c39bba..42c95bcb53ca 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -373,14 +373,14 @@ static void memcg_destroy_list_lru_node(struct list_lru_node *nlru) struct list_lru_memcg *memcg_lrus; /* * This is called when shrinker has already been unregistered, - * and nobody can use it. So, there is no need to use kvfree_rcu(). + * and nobody can use it. So, there is no need to use kvfree_rcu_local(). */ memcg_lrus = rcu_dereference_protected(nlru->memcg_lrus, true); __memcg_destroy_list_lru_node(memcg_lrus, 0, memcg_nr_cache_ids); kvfree(memcg_lrus); } -static void kvfree_rcu(struct rcu_head *head) +static void kvfree_rcu_local(struct rcu_head *head) { struct list_lru_memcg *mlru; @@ -419,7 +419,7 @@ static int memcg_update_list_lru_node(struct list_lru_node *nlru, rcu_assign_pointer(nlru->memcg_lrus, new); spin_unlock_irq(&nlru->lock); - call_rcu(&old->rcu, kvfree_rcu); + call_rcu(&old->rcu, kvfree_rcu_local); return 0; } From patchwork Tue Apr 28 20:58:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11515587 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DEBEC92C for ; Tue, 28 Apr 2020 21:00:09 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id ABFC42072A for ; Tue, 28 Apr 2020 21:00:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="o5++9uYp" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ABFC42072A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id DA8058E0016; Tue, 28 Apr 2020 16:59:51 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D596F8E0007; Tue, 28 Apr 2020 16:59:51 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B83328E0016; Tue, 28 Apr 2020 16:59:51 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0242.hostedemail.com [216.40.44.242]) by kanga.kvack.org (Postfix) with ESMTP id 9AB5E8E0007 for ; Tue, 28 Apr 2020 16:59:51 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 56EEF180AD802 for ; Tue, 28 Apr 2020 20:59:51 +0000 (UTC) X-FDA: 76758480582.03.drop95_5604d86f5e133 X-Spam-Summary: 2,0,0,64c31c302f2f3168,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:41:355:379:541:800:960:965:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:2736:3138:3139:3140:3141:3142:3352:3865:3866:3867:3868:3871:3872:3874:4250:4321:4385:4390:4395:4605:5007:6261:6653:7514:7875:7904:8603:9413:10004:11026:11658:11914:12296:12297:12517:12519:12555:12895:12986:13069:13311:13357:13894:14181:14384:14394:14687:14721:21080:21444:21451:21627:21666:21990:30029:30054:30070,0,RBL:209.85.208.195:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.18.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: drop95_5604d86f5e133 X-Filterd-Recvd-Size: 4699 Received: from mail-lj1-f195.google.com (mail-lj1-f195.google.com [209.85.208.195]) by imf27.hostedemail.com (Postfix) with ESMTP for ; Tue, 28 Apr 2020 20:59:51 +0000 (UTC) Received: by mail-lj1-f195.google.com with SMTP id b2so245993ljp.4 for ; Tue, 28 Apr 2020 13:59:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=bWLqr2jPDyEQcY7iSPb5llcri1orQdd6DnkilIcs6pQ=; b=o5++9uYpB0tejuE6SOx2AEJ39EGXaxcQzOa6BiVWFjcS7yoTvXbje74KUsJfyuRkbg fwJPdrquJbOLHCaxqz9Mwtl6FuSvM+KsXMnliBkFsADyVua+6KDH1fCS59qjPVtxRK0n dbipm1fl+eKq8q9CYpoaWufW0aRmtdGBT3+unSG/3WWIawdf3TxALo47FGzyiW/IPdJm nKA0oWU8najSTRkjsVD0TmMdzC33ujQapFp5uE/fLh1YLZ0yXHG+58rzxSzfgTt4eNSp 7A45DvCSkx6V0saTuCCdtrbmkF8QTbYJ3Hrf+TmzTvnWaNoe/+pbLk046gt2RE4nFrvf CVtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=bWLqr2jPDyEQcY7iSPb5llcri1orQdd6DnkilIcs6pQ=; b=rFXuE1FM1fHTcLG1X1NBBcMDetMLFxvUQWrPeDabc/M+Jm9xX22sM/6cL3hYSTdYTZ mqlQz+Rfu5zYA9L5EpUC+QnLzHEX+WdNRDFC2LIeQb64gYreZZXzXXHMzi3nWXrFj36r DpDcN9cjmbrgak8iILiGMUB+u2pHZW7Cu2cGlJ6o2VOL3me18a3a4+LwwRF8n8zQn+mo C+zIpizFJagO8L9s2Oo6Yuy1OwwRALTwPFgpjcgrYz0eSGJS203bQ5PbKYsMtxcCvrBg aTa6gfRkNRwJJ2Mn/v4XAHQlUeTVTPxkKfQIvURBIM/S4XYX2AN+nNtV/wB5YpgLj5cW 3JeA== X-Gm-Message-State: AGi0PuYpWwTw7H9moeNxS4L28pPICtQJ5sysc70J7JqbhgcuXbfbkfz4 xK4ZeLwkHCoXiKjblgoGtvY= X-Google-Smtp-Source: APiQypK9EZpQhwaV5M+lBBDIJenVe52+/F2F7dseEsf9lWqnTDKV8zoEcohaXMvNV1qR0laI6NaBXg== X-Received: by 2002:a2e:9018:: with SMTP id h24mr18683499ljg.217.1588107589805; Tue, 28 Apr 2020 13:59:49 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id z21sm295483ljh.42.2020.04.28.13.59.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Apr 2020 13:59:49 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 17/24] rcu: Introduce 2 arg kvfree_rcu() interface Date: Tue, 28 Apr 2020 22:58:56 +0200 Message-Id: <20200428205903.61704-18-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200428205903.61704-1-urezki@gmail.com> References: <20200428205903.61704-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: kvfree_rcu() can deal with an allocated memory that is obtained via kvmalloc(). It can return two types of allocated memory or "pointers", one can belong to regular SLAB allocator and another one can be vmalloc one. It depends on requested size and memory pressure. struct test_kvfree_rcu { struct rcu_head rcu; unsigned char array[100]; }; struct test_kvfree_rcu *p; p = kvmalloc(10 * PAGE_SIZE); if (p) kvfree_rcu(p, rcu); Signed-off-by: Uladzislau Rezki (Sony) Reviewed-by: Joel Fernandes (Google) Signed-off-by: Joel Fernandes (Google) --- include/linux/rcupdate.h | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index b344fc800a9b..51b26ab02878 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -875,6 +875,15 @@ do { \ __kvfree_rcu(&((___p)->rhf), offsetof(typeof(*(ptr)), rhf)); \ } while (0) +/** + * kvfree_rcu() - kvfree an object after a grace period. + * @ptr: pointer to kvfree + * @rhf: the name of the struct rcu_head within the type of @ptr. + * + * Same as kfree_rcu(), just simple alias. + */ +#define kvfree_rcu(ptr, rhf) kfree_rcu(ptr, rhf) + /* * Place this after a lock-acquisition primitive to guarantee that * an UNLOCK+LOCK pair acts as a full barrier. This guarantee applies From patchwork Tue Apr 28 20:58:57 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11515589 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 604111805 for ; Tue, 28 Apr 2020 21:00:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2101B2072A for ; Tue, 28 Apr 2020 21:00:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="CQtUZub7" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2101B2072A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5E0678E0017; Tue, 28 Apr 2020 16:59:53 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 595A78E0007; Tue, 28 Apr 2020 16:59:53 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 457FB8E0017; Tue, 28 Apr 2020 16:59:53 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0174.hostedemail.com [216.40.44.174]) by kanga.kvack.org (Postfix) with ESMTP id 28CB18E0007 for ; Tue, 28 Apr 2020 16:59:53 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id DEDA052AE for ; Tue, 28 Apr 2020 20:59:52 +0000 (UTC) X-FDA: 76758480624.07.hate18_563ec80929d13 X-Spam-Summary: 2,0,0,c01417b6bd80a99e,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:41:69:355:379:541:800:960:965:966:968:973:988:989:1260:1311:1314:1345:1359:1431:1437:1515:1535:1542:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:3138:3139:3140:3141:3142:3353:3865:3867:3868:3870:3871:3872:3874:4321:4385:4390:4395:4419:4605:5007:6261:6653:7514:7903:9413:9592:10004:11026:11473:11658:11914:12043:12296:12297:12438:12517:12519:12555:12895:13846:13894:13972:14096:14181:14394:14687:14721:21080:21444:21451:21627:21666:21990:30054,0,RBL:209.85.208.193:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:2,LUA_SUMMARY:none X-HE-Tag: hate18_563ec80929d13 X-Filterd-Recvd-Size: 5244 Received: from mail-lj1-f193.google.com (mail-lj1-f193.google.com [209.85.208.193]) by imf37.hostedemail.com (Postfix) with ESMTP for ; Tue, 28 Apr 2020 20:59:52 +0000 (UTC) Received: by mail-lj1-f193.google.com with SMTP id g4so258176ljl.2 for ; Tue, 28 Apr 2020 13:59:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5ldgJ4636CVVH5K09ieY2CjYskH7FkK4YIkv15ewvcg=; b=CQtUZub7pmYCoM9rgBDRF8mOSsdyPuWwmd58ngg3Hlx0WWJBUPyU6nMUtnNSnjSvFi 9PcXKfbggKYCYRv4M2BI1ivQVzjcHHOwxvfjfZv1cqmGM5J25fNWUENTN5J+ek9erpbJ SvkSqmrZut59DzXc+ag223V9aWC3r1f9IWUbXfRi3ACwJp5mu9xU0KxkgDSNH83xyk8x QLhcYVe1OvpcztTL865V1jatwBmH3vn/pkqoIJ7Jxj4NSloLaTRUzsuxaBjRsh1xXLeN hQVUWmQlPqcZEL4nguD8c7Vlkry1FfBNwpumcnUWV6vISLulSNYtcvhuKkiQPO6HLKG6 S9FA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5ldgJ4636CVVH5K09ieY2CjYskH7FkK4YIkv15ewvcg=; b=niJbGJBTF21QoUgxMIXz5UDCD/GtdpqZDrXqL8y5TMxDLiyt+no+pdpJh38HiuhVGQ MKfWlEShkGQpC5SMBJHu5cYGrrvr3G0ITIgnsA/4THdMlpllNZDfOsRYa7RZatXrSbY6 bV6LoWSypl6q8G1hdlPHbVo+Go8o6dY8ujBb0+YX5aZNLFWMj38c0EU9wys5YxS1h+Pb jwDrjDVPUIi1ZKLGz8RBUUPk+VGzNwk8cb6B9UtxxjmL7m9MnepnB7FVhRTUDXaBU4Z8 JBtAmLEDjKLwUGOvTZC0uGC0bZipVtG127QPed7OMYOYbQDjB+oYnnm9oMw/CciCWsjj TQIw== X-Gm-Message-State: AGi0PuakohZW6BkDZuYOTNt8QPVa+8fK88QZQg8azY+bULaQ8Wh6NBkk KR4HmDxVB7ce9m8OYa/IhPI= X-Google-Smtp-Source: APiQypI7Q10ox7ABU0RMMw3dY9sgNyJEZcaL1fC+sRAcqhh5hwFSEZtLMIISQ2ZVDZMjGMbpQT614Q== X-Received: by 2002:a05:651c:545:: with SMTP id q5mr14996061ljp.57.1588107591349; Tue, 28 Apr 2020 13:59:51 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id z21sm295483ljh.42.2020.04.28.13.59.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Apr 2020 13:59:50 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 18/24] mm/list_lru.c: Remove kvfree_rcu_local() function Date: Tue, 28 Apr 2020 22:58:57 +0200 Message-Id: <20200428205903.61704-19-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200428205903.61704-1-urezki@gmail.com> References: <20200428205903.61704-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since there is newly introduced kvfree_rcu() API, there is no need in queuing and using call_rcu() to kvfree() an object after the GP. Remove kvfree_rcu_local() function and replace call_rcu() by new kvfree_rcu() API that does the same but in more efficient way. Cc: linux-mm@kvack.org Cc: Andrew Morton Cc: rcu@vger.kernel.org Signed-off-by: Uladzislau Rezki (Sony) Reviewed-by: Joel Fernandes (Google) Signed-off-by: Joel Fernandes (Google) --- mm/list_lru.c | 13 +++---------- 1 file changed, 3 insertions(+), 10 deletions(-) diff --git a/mm/list_lru.c b/mm/list_lru.c index 42c95bcb53ca..a0b08b27a9b9 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -12,6 +12,7 @@ #include #include #include +#include #include "slab.h" #ifdef CONFIG_MEMCG_KMEM @@ -373,21 +374,13 @@ static void memcg_destroy_list_lru_node(struct list_lru_node *nlru) struct list_lru_memcg *memcg_lrus; /* * This is called when shrinker has already been unregistered, - * and nobody can use it. So, there is no need to use kvfree_rcu_local(). + * and nobody can use it. So, there is no need to use kvfree_rcu(). */ memcg_lrus = rcu_dereference_protected(nlru->memcg_lrus, true); __memcg_destroy_list_lru_node(memcg_lrus, 0, memcg_nr_cache_ids); kvfree(memcg_lrus); } -static void kvfree_rcu_local(struct rcu_head *head) -{ - struct list_lru_memcg *mlru; - - mlru = container_of(head, struct list_lru_memcg, rcu); - kvfree(mlru); -} - static int memcg_update_list_lru_node(struct list_lru_node *nlru, int old_size, int new_size) { @@ -419,7 +412,7 @@ static int memcg_update_list_lru_node(struct list_lru_node *nlru, rcu_assign_pointer(nlru->memcg_lrus, new); spin_unlock_irq(&nlru->lock); - call_rcu(&old->rcu, kvfree_rcu_local); + kvfree_rcu(old, rcu); return 0; } From patchwork Tue Apr 28 20:58:58 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11515591 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AC9591805 for ; Tue, 28 Apr 2020 21:00:14 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6C9AC2072A for ; Tue, 28 Apr 2020 21:00:14 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="d6UTV9hy" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6C9AC2072A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0150A8E0018; Tue, 28 Apr 2020 16:59:55 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id EDF6D8E0007; Tue, 28 Apr 2020 16:59:54 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DA51F8E0018; Tue, 28 Apr 2020 16:59:54 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0166.hostedemail.com [216.40.44.166]) by kanga.kvack.org (Postfix) with ESMTP id BE5958E0007 for ; Tue, 28 Apr 2020 16:59:54 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 7B5532493 for ; Tue, 28 Apr 2020 20:59:54 +0000 (UTC) X-FDA: 76758480708.13.rings59_5678998eaaf2e X-Spam-Summary: 2,0,0,547416e9be589050,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:2:41:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1605:1730:1747:1777:1792:2196:2199:2393:2559:2562:2895:2899:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:4049:4120:4250:4385:5007:6117:6119:6261:6653:7514:7875:7903:8660:9010:9413:10004:10394:11026:11473:11658:11914:12043:12291:12295:12296:12297:12438:12517:12519:12555:12683:12895:13148:13161:13229:13230:13894:14096:14394:14687:21080:21324:21433:21444:21451:21611:21627:21666:21740:21789:21796:21810:21939:21987:21990:30012:30036:30045:30054:30055:30070,0,RBL:209.85.208.196:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.18.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:2,LUA_SUMMARY:none X-HE-Tag: rings59_5678998eaaf2e X-Filterd-Recvd-Size: 9958 Received: from mail-lj1-f196.google.com (mail-lj1-f196.google.com [209.85.208.196]) by imf13.hostedemail.com (Postfix) with ESMTP for ; Tue, 28 Apr 2020 20:59:54 +0000 (UTC) Received: by mail-lj1-f196.google.com with SMTP id a21so191502ljj.11 for ; Tue, 28 Apr 2020 13:59:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=66vSg9Njt1sZCB7M+dj1UlLLZ6BR/zJPh4B9KJb+CwY=; b=d6UTV9hymGMssr4fjOWqvH+tmH1+vbRR4+2M9oD8d4EnJPh0SHe7iRFDylZiDMGjHA JlfOILnK2hmlYxoV47p3bLx2RydaXS6tJMVboKMWS3J8iNzcIpksYLgp/hd+/iSk52Bc FZhx5LS9U3kAaoLzsoB7gF3Q0P1aqNetJLXUwXOBjG3pjd0reRTStlPB2nNHtf9WfeoT 2EHBK9+AmhcjSLEoej6QDrJTRpP2yLaiB4jWVvMEkjn88sYmZ4aIjGiBMeZZ7I/uEEYs 9rUx8KLuPJKJ3kSF3zVDbspnnrRzPN9vo8j5JP4VGPefcIr7ikAmm1mmSg//aGSaCJU5 UzNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=66vSg9Njt1sZCB7M+dj1UlLLZ6BR/zJPh4B9KJb+CwY=; b=uF6s5Ie/aylwJbAn963oI+/wnnKh1VoJJysKFO2z1z4BC3jDhpl+ccap+kpizTIrVy GAs5PRzy+3TAXx8yCqHp9UWOkwiraTcG/jvN4ut3ovSHF9+mKuTcbOCJ1zZFBJR7+ZtE i2sISoAJ8tXtAKRt9ojZImEZkDlggOpLtajVIHVWj4cEvyxlrlq3wsj4Q9CC+d8P4gbG LmfplUBh5GDUV67wVrduhcrTYs+Wxr0XooRXHZUPnMexOWuODPnFtA7MHBA6CSPPImVo 0fAH+z7hOn/Q1opR3UQXBsc/JgnMWS+fk/CWcmd350PHuQIoEYL5H8D8eRIXbqg2lmCM zZzg== X-Gm-Message-State: AGi0PuYDHd4tW6IIOTUg4r/TgVEAeHPn/1iA09Gs4HOfBg3OPg45fAPO WnheqzhqZ00wXDtHANQsyzo= X-Google-Smtp-Source: APiQypJwT3gU3ag2Y4efeBkFASGSObeuAOyEJgCNhNbI+6xXdAvu4anqfWejz6vVzioHPMcJxp3dYQ== X-Received: by 2002:a2e:9818:: with SMTP id a24mr18725795ljj.126.1588107592738; Tue, 28 Apr 2020 13:59:52 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id z21sm295483ljh.42.2020.04.28.13.59.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Apr 2020 13:59:51 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 19/24] rcu/tree: Support reclaim for head-less object Date: Tue, 28 Apr 2020 22:58:58 +0200 Message-Id: <20200428205903.61704-20-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200428205903.61704-1-urezki@gmail.com> References: <20200428205903.61704-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Update the kvfree_call_rcu() with head-less support, it means an object without any rcu_head structure can be reclaimed after GP. To store pointers there are two chain-arrays maintained one for SLAB and another one is for vmalloc. Both types of objects(head-less variant and regular one) are placed there based on the type. It can be that maintaining of arrays becomes impossible due to high memory pressure. For such reason there is an emergency path. In that case objects with rcu_head inside are just queued building one way list. Later on that list is drained. As for head-less variant. Such objects do not have any rcu_head helper inside. Thus it is dynamically attached. As a result an object consists of back-pointer and regular rcu_head. It implies that emergency path can detect such object type, therefore they are tagged. So a back-pointer could be freed as well as dynamically attached wrapper. Even though such approach requires dynamic memory it needs only sizeof(unsigned long *) + sizeof(struct rcu_head) bytes, thus SLAB is used to obtain it. Finally if attaching of the rcu_head and queuing get failed, the current context has to follow might_sleep() annotation, thus below steps could be applied: a) wait until a grace period has elapsed; b) direct inlining of the kvfree() call. Reviewed-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki (Sony) Signed-off-by: Joel Fernandes (Google) Co-developed-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 102 ++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 98 insertions(+), 4 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 51726e4c3b4d..501cac02146d 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3072,15 +3072,31 @@ static void kfree_rcu_work(struct work_struct *work) */ for (; head; head = next) { unsigned long offset = (unsigned long)head->func; - void *ptr = (void *)head - offset; + bool headless; + void *ptr; next = head->next; + + /* We tag the headless object, if so adjust offset. */ + headless = (((unsigned long) head - offset) & BIT(0)); + if (headless) + offset -= 1; + + ptr = (void *) head - offset; + debug_rcu_head_unqueue((struct rcu_head *)ptr); rcu_lock_acquire(&rcu_callback_map); trace_rcu_invoke_kvfree_callback(rcu_state.name, head, offset); - if (!WARN_ON_ONCE(!__is_kvfree_rcu_offset(offset))) + if (!WARN_ON_ONCE(!__is_kvfree_rcu_offset(offset))) { + /* + * If headless free the back-pointer first. + */ + if (headless) + kvfree((void *) *((unsigned long *) ptr)); + kvfree(ptr); + } rcu_lock_release(&rcu_callback_map); cond_resched_tasks_rcu_qs(); @@ -3221,6 +3237,13 @@ kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr) if (IS_ENABLED(CONFIG_PREEMPT_RT)) return false; + /* + * TODO: For one argument of kvfree_rcu() we can + * drop the lock and get the page in sleepable + * context. That would allow to maintain an array + * for the CONFIG_PREEMPT_RT as well. Thus we could + * get rid of dynamic rcu_head attaching code. + */ bnode = (struct kvfree_rcu_bulk_data *) __get_free_page(GFP_NOWAIT | __GFP_NOWARN); } @@ -3244,6 +3267,23 @@ kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr) return true; } +static inline struct rcu_head * +attach_rcu_head_to_object(void *obj) +{ + unsigned long *ptr; + + ptr = kmalloc(sizeof(unsigned long *) + + sizeof(struct rcu_head), GFP_NOWAIT | + __GFP_RECLAIM | /* can do direct reclaim. */ + __GFP_NORETRY | /* only lightweight one. */ + __GFP_NOWARN); /* no failure reports. */ + if (!ptr) + return NULL; + + ptr[0] = (unsigned long) obj; + return ((struct rcu_head *) ++ptr); +} + /* * Queue a request for lazy invocation of appropriate free routine after a * grace period. Please note there are three paths are maintained, two are the @@ -3260,16 +3300,34 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) { unsigned long flags; struct kfree_rcu_cpu *krcp; + bool success; void *ptr; + if (head) { + ptr = (void *) head - (unsigned long) func; + } else { + /* + * Please note there is a limitation for the head-less + * variant, that is why there is a clear rule for such + * objects: + * + * it can be used from might_sleep() context only. For + * other places please embed an rcu_head to your data. + */ + might_sleep(); + ptr = (unsigned long *) func; + } + krcp = krc_this_cpu_lock(&flags); - ptr = (void *)head - (unsigned long)func; /* Queue the object but don't yet schedule the batch. */ if (debug_rcu_head_queue(ptr)) { /* Probable double kfree_rcu(), just leak. */ WARN_ONCE(1, "%s(): Double-freed call. rcu_head %p\n", __func__, head); + + /* Mark as success and leave. */ + success = true; goto unlock_return; } @@ -3277,10 +3335,34 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) * Under high memory pressure GFP_NOWAIT can fail, * in that case the emergency path is maintained. */ - if (unlikely(!kvfree_call_rcu_add_ptr_to_bulk(krcp, ptr))) { + success = kvfree_call_rcu_add_ptr_to_bulk(krcp, ptr); + if (!success) { + if (head == NULL) { + /* + * Headless(one argument kvfree_rcu()) can sleep. + * Drop the lock and tack it back. So it can do + * direct lightweight reclaim. + */ + krc_this_cpu_unlock(krcp, flags); + head = attach_rcu_head_to_object(ptr); + krcp = krc_this_cpu_lock(&flags); + + if (head == NULL) + goto unlock_return; + + /* + * Tag the headless object. Such objects have a + * back-pointer to the original allocated memory, + * that has to be freed as well as dynamically + * attached wrapper/head. + */ + func = (rcu_callback_t) (sizeof(unsigned long *) + 1); + } + head->func = func; head->next = krcp->head; krcp->head = head; + success = true; } WRITE_ONCE(krcp->count, krcp->count + 1); @@ -3294,6 +3376,18 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) unlock_return: krc_this_cpu_unlock(krcp, flags); + + /* + * High memory pressure, so inline kvfree() after + * synchronize_rcu(). We can do it from might_sleep() + * context only, so the current CPU can pass the QS + * state. + */ + if (!success) { + debug_rcu_head_unqueue(ptr); + synchronize_rcu(); + kvfree(ptr); + } } EXPORT_SYMBOL_GPL(kvfree_call_rcu); From patchwork Tue Apr 28 20:58:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11515593 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ED97392C for ; Tue, 28 Apr 2020 21:00:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B0E972176D for ; Tue, 28 Apr 2020 21:00:16 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="pYZBp9n1" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B0E972176D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 916448E0019; Tue, 28 Apr 2020 16:59:58 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8C87F8E0007; Tue, 28 Apr 2020 16:59:58 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6F0608E0019; Tue, 28 Apr 2020 16:59:58 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0137.hostedemail.com [216.40.44.137]) by kanga.kvack.org (Postfix) with ESMTP id 52DAD8E0007 for ; Tue, 28 Apr 2020 16:59:58 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 19BDF824805A for ; Tue, 28 Apr 2020 20:59:58 +0000 (UTC) X-FDA: 76758480876.15.goat76_56feb7322e54e X-Spam-Summary: 2,0,0,e9abf0331c949773,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:41:355:379:541:800:960:965:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:3138:3139:3140:3141:3142:3353:3865:3866:3867:3871:3874:4321:4385:4390:4395:5007:6119:6261:6653:7514:7576:7875:9413:10004:10394:11026:11473:11658:11914:12043:12296:12297:12438:12517:12519:12555:12895:13069:13311:13357:13894:14181:14384:14394:14687:14721:21080:21444:21451:21627:21666:21740:21789:30054,0,RBL:209.85.208.195:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: goat76_56feb7322e54e X-Filterd-Recvd-Size: 4903 Received: from mail-lj1-f195.google.com (mail-lj1-f195.google.com [209.85.208.195]) by imf42.hostedemail.com (Postfix) with ESMTP for ; Tue, 28 Apr 2020 20:59:57 +0000 (UTC) Received: by mail-lj1-f195.google.com with SMTP id b2so246625ljp.4 for ; Tue, 28 Apr 2020 13:59:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KoBD39TIcEF5om4wxAY9WABo2MzBmQNw/jZQCSmQt+E=; b=pYZBp9n19gsVOTGoO/Sk3hIwSChUFPAuqtoXgQSXrtWfKsilDS0sALg+vhbnSbxvGu VdobfSf7hyqLliIrSIaj0t9nu3iDqeFBds26B81XwdubbvYQCcjAnz1LX5w/82X1wPx0 i3WUHABqC94YnY2H4OpAR8l9zs8RfZn6Xxytlj/3Enc9gvzbOpi6Q9ZfrZjnEg2AK2Y7 bjxMx8yDoA4Lqo7Q/heylOx3S3xZqCXtya9oCmVVbX/kuQPaw+5HEq0pOfIC0iO4wlE3 CLAlOGXVfDTxBH8OKQS/IzkGrNB2gypGnYJx6H2v/aWX4ISqOJiSpDNvKf1FVizYkmOf SUQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KoBD39TIcEF5om4wxAY9WABo2MzBmQNw/jZQCSmQt+E=; b=leszkMstnX9makLouUJUUtqc3oGKU6tqXj/L4t0qt1kc4X4UnrZ+++lx9h0H/WfqPj Aat1vyHhkWShw6/OgTU9D7AEzW+QC1gqWY8mBSHkbnAXAngT3Qp8fiU8eHqTO8nLH2ZD 8PL32uGdhN0OtUbW7JK2O77D1rI9FtVj5a6UePOozpeXXf83UtP8GYJLcu4XHrfLTkBI LplynmPM072AUlxAbObPCjg/dewIFDdWEsTWlaFen2B+lsjojn5GCUARRU/XQf4SSk38 OFHPPl4rxwRoN+JlUg5du7GZFn4+6Yt2VFKTjFmHn+RjhOvrD0jtjsagcnCZjbQ1k34j YGAQ== X-Gm-Message-State: AGi0PuabgJE4oDZAwNMmHvnQzVMUpndAigix1RzV6JQwNZrtn5BT0iPR G37PH5gynqMQEvVVL83L9dY= X-Google-Smtp-Source: APiQypJZR1Ec3ulELhNCGSHkNwXT2m0NQRhUCSqyh7Fi0d8nw8f+9QCfLPXaFRVdPqbaxBXapCWV3w== X-Received: by 2002:a2e:3012:: with SMTP id w18mr19191207ljw.55.1588107596479; Tue, 28 Apr 2020 13:59:56 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id z21sm295483ljh.42.2020.04.28.13.59.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Apr 2020 13:59:53 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 20/24] rcu/tree: Make kvfree_rcu() tolerate any alignment Date: Tue, 28 Apr 2020 22:58:59 +0200 Message-Id: <20200428205903.61704-21-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200428205903.61704-1-urezki@gmail.com> References: <20200428205903.61704-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Joel Fernandes (Google)" Handle cases where the the object being kvfree_rcu()'d is not aligned by 2-byte boundaries. Signed-off-by: Uladzislau Rezki (Sony) Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 501cac02146d..649bad7ad0f0 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2877,6 +2877,9 @@ struct kvfree_rcu_bulk_data { #define KVFREE_BULK_MAX_ENTR \ ((PAGE_SIZE - sizeof(struct kvfree_rcu_bulk_data)) / sizeof(void *)) +/* Encoding the offset of a fake rcu_head to indicate the head is a wrapper. */ +#define RCU_HEADLESS_KFREE BIT(31) + /** * struct kfree_rcu_cpu_work - single batch of kfree_rcu() requests * @rcu_work: Let queue_rcu_work() invoke workqueue handler after grace period @@ -3078,9 +3081,9 @@ static void kfree_rcu_work(struct work_struct *work) next = head->next; /* We tag the headless object, if so adjust offset. */ - headless = (((unsigned long) head - offset) & BIT(0)); + headless = !!(offset & RCU_HEADLESS_KFREE); if (headless) - offset -= 1; + offset &= ~(RCU_HEADLESS_KFREE); ptr = (void *) head - offset; @@ -3356,7 +3359,7 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) * that has to be freed as well as dynamically * attached wrapper/head. */ - func = (rcu_callback_t) (sizeof(unsigned long *) + 1); + func = (rcu_callback_t)(sizeof(unsigned long *) | RCU_HEADLESS_KFREE); } head->func = func; From patchwork Tue Apr 28 20:59:00 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11515595 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 058C9112C for ; Tue, 28 Apr 2020 21:00:19 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C796C20731 for ; Tue, 28 Apr 2020 21:00:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="UYukFqdv" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C796C20731 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D0DBF8E001A; Tue, 28 Apr 2020 16:59:59 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C967F8E0007; Tue, 28 Apr 2020 16:59:59 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B87158E001A; Tue, 28 Apr 2020 16:59:59 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0127.hostedemail.com [216.40.44.127]) by kanga.kvack.org (Postfix) with ESMTP id 94ADB8E0007 for ; Tue, 28 Apr 2020 16:59:59 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 4E7CE824805A for ; Tue, 28 Apr 2020 20:59:59 +0000 (UTC) X-FDA: 76758480918.28.grain92_5728570e76a23 X-Spam-Summary: 2,0,0,db8b083dfe98675a,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:41:355:379:541:800:960:965:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:3138:3139:3140:3141:3142:3352:3865:3866:3871:4385:4390:4395:4605:5007:6261:6653:7514:9413:9592:10004:11026:11473:11658:11914:12043:12296:12297:12438:12517:12519:12555:12895:13069:13311:13357:13894:14096:14181:14384:14394:14687:14721:21080:21212:21444:21451:21627:21666:30054,0,RBL:209.85.208.195:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: grain92_5728570e76a23 X-Filterd-Recvd-Size: 4595 Received: from mail-lj1-f195.google.com (mail-lj1-f195.google.com [209.85.208.195]) by imf48.hostedemail.com (Postfix) with ESMTP for ; Tue, 28 Apr 2020 20:59:58 +0000 (UTC) Received: by mail-lj1-f195.google.com with SMTP id h4so189024ljg.12 for ; Tue, 28 Apr 2020 13:59:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=sBXzHuBlNIiG5WL15rHVAeI2UeCimXr5JALsutUTg9Y=; b=UYukFqdvOw/RYQ+Em+r2cvMx8ideTgu8rhd+I9bYqLwxcbgu9vNXU3uaca55s732ym CrEQXN6x38a8DRsG2xFDMK4jMsgvDPwo5MBF0NBh/mZ4wod0iMehj046iiSrKLhDhxTD OLv6OW+1XR9USm3IGF083IXEWzo5J8uRc39cB1SyQZ5W9JXESO9ihoIFmDJaTahQ8jis vjc9UkiSGkqOJqqtzYOpN6RBPV6P3Df43fHcETpcrBu2P0GzMYBv4gsYLtHb74LY7+/s InJNpW48Iukvavept/9+c6OE54pG3/uqM0Sl1LdoqWfgWT5ZBgHdYawwx2PKGzYxu77m pjMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=sBXzHuBlNIiG5WL15rHVAeI2UeCimXr5JALsutUTg9Y=; b=YIpbhQSp4PxIejwtg/a+4MknySGcBHeuiU0vy/IiRg3Bk0Jg8w2LerB+S4r/KDApSh Y0b4UnPC3CtRAGFqpcYdMhOZhrleu+FH2qpH7ffsbxXs/hfqPalorzopRvdz2woqGaMZ NVp1CgpRhf0hUkAcq1VcS4UxX4PJVgIJjZIrQRyFRIlYaO5Gn665qsrR53lHNGAjIHC5 u90sjByrH6LppwMJVdUW/1j2pKhmo7272DPoVqlVFYpssFAf04mGM0a8Eixpudyb9f6c IC3VxDcAUNjA8lxi8HFp5pgsJQdfbRh1x+iQpE8cnD7wmagkN0oQhQQwvYrQ4VcmVrm6 XE1Q== X-Gm-Message-State: AGi0PuZ/HjoRvxKVUkfr1AxVR5X7fPKWI9arrQB+l9EWBt4VtfXgRQVK LAOBTiG4JvOz/PAdrznqsz0= X-Google-Smtp-Source: APiQypKbwyjmz7XAKZZaGzDhUO4GGzDo1hAKGgC0IXSPp5QF7iC3hAYqf7fmKiCv2L6QXg614yuycw== X-Received: by 2002:ac2:5dc6:: with SMTP id x6mr20373376lfq.108.1588107597607; Tue, 28 Apr 2020 13:59:57 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id z21sm295483ljh.42.2020.04.28.13.59.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Apr 2020 13:59:57 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 21/24] rcu/tiny: move kvfree_call_rcu() out of header Date: Tue, 28 Apr 2020 22:59:00 +0200 Message-Id: <20200428205903.61704-22-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200428205903.61704-1-urezki@gmail.com> References: <20200428205903.61704-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move inlined kvfree_call_rcu() function out of the header file. This step is a preparation for head-less support. Reviewed-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki (Sony) --- include/linux/rcutiny.h | 6 +----- kernel/rcu/tiny.c | 6 ++++++ 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h index 0c6315c4a0fe..7eb66909ae1b 100644 --- a/include/linux/rcutiny.h +++ b/include/linux/rcutiny.h @@ -34,11 +34,7 @@ static inline void synchronize_rcu_expedited(void) synchronize_rcu(); } -static inline void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) -{ - call_rcu(head, func); -} - +void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func); void rcu_qs(void); static inline void rcu_softirq_qs(void) diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c index aa897c3f2e92..508c82faa45c 100644 --- a/kernel/rcu/tiny.c +++ b/kernel/rcu/tiny.c @@ -177,6 +177,12 @@ void call_rcu(struct rcu_head *head, rcu_callback_t func) } EXPORT_SYMBOL_GPL(call_rcu); +void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) +{ + call_rcu(head, func); +} +EXPORT_SYMBOL_GPL(kvfree_call_rcu); + void __init rcu_init(void) { open_softirq(RCU_SOFTIRQ, rcu_process_callbacks); From patchwork Tue Apr 28 20:59:01 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11515597 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 18B0B112C for ; Tue, 28 Apr 2020 21:00:21 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CE38E2072A for ; Tue, 28 Apr 2020 21:00:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="CWL6Mhxs" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CE38E2072A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2889C8E001B; Tue, 28 Apr 2020 17:00:01 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 210EC8E0007; Tue, 28 Apr 2020 17:00:01 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0D9348E001B; Tue, 28 Apr 2020 17:00:00 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0009.hostedemail.com [216.40.44.9]) by kanga.kvack.org (Postfix) with ESMTP id DE90B8E0007 for ; Tue, 28 Apr 2020 17:00:00 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 9FC8D180AD802 for ; Tue, 28 Apr 2020 21:00:00 +0000 (UTC) X-FDA: 76758480960.22.stone87_5759df62e0e36 X-Spam-Summary: 2,0,0,5837024122872888,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:2:41:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1605:1606:1730:1747:1777:1792:2196:2199:2393:2559:2562:2907:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:4119:4250:4321:4385:5007:6119:6261:6653:7514:7875:7903:8660:9413:10004:11026:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12683:12895:13148:13211:13229:13230:13894:14096:14394:14687:21080:21324:21433:21444:21451:21627:21666:21740:21789:21939:21987:21990:30045:30054:30070,0,RBL:209.85.208.193:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.18.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:16,LUA_SUMMARY:none X-HE-Tag: stone87_5759df62e0e36 X-Filterd-Recvd-Size: 8648 Received: from mail-lj1-f193.google.com (mail-lj1-f193.google.com [209.85.208.193]) by imf31.hostedemail.com (Postfix) with ESMTP for ; Tue, 28 Apr 2020 21:00:00 +0000 (UTC) Received: by mail-lj1-f193.google.com with SMTP id l19so199594lje.10 for ; Tue, 28 Apr 2020 14:00:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=kEunEa6ntH1HytcdQSdsY2CsdbEZEQ/S23NHaGYEDh0=; b=CWL6Mhxss7Hv7VG0ZlIx12jk8sHFvdJzDPhUwW40jXJ/R4y20EdTQ5Nqk1C7iyTu1A /NnaS20p/fcIuDjH62b+awTfCcP5fdc5Zo1ZuxyntzXrA0F7TkzLFSEPnyQQzk2x6bxU w/I7BKkaKWNxFO06OxepNfWb5QfIp+Sf8gJXVuVilwidIM+OorINFbKbFH5kz871t/6p FMX6JxEr+RNc9VK/aYO6sW9mmQQPTx2smO4BDGaqPkYIbtlYMAsOH9kNQoDbxhgNUB49 snoL2XjyusdH6OFmnR/L0Bo1kn5nTu1xdtc4R2+siuvSwuj/zYa49CuhQK+YFIhdclY8 jSog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kEunEa6ntH1HytcdQSdsY2CsdbEZEQ/S23NHaGYEDh0=; b=VDktAOTVuBAhXKIrf4gGuL8cNHlXuXS6J55GkLDNsNNXJPUlS3hbukfIu9cqE9Kz4c 3Rn9b+dEUeXOXAV0r7cuWNUb6oFvG/gGfneHc8KsQ2QJ9XIjOFs7JXujK2y56Wk66+Ug OImzfLXRlTXNtWuLjYCyf0dgWtipfgDkDvg8KU14cmIgXLSvspInih7gSyK1lhbFrgBd Ggl3+YDWxn6Avhzo7Uy3DwQDyelm5Bm9ExBjieA65Td/5j3lrizDpryHHXgCOuyZjOzl MmCMw9iragGowzgrcyTVT4ieCX63pv3bdZaaNS0+zz1bDJFP1HwZum2gpHR7UMgEy3EU qqgg== X-Gm-Message-State: AGi0PuYNgcWEyAHzQbo1j588A6A+u+gFQFywqXE07ctDwRGJ8DFX/ZoL C1SXOlCyMwqweneM1kLakVQ= X-Google-Smtp-Source: APiQypIuEVHqAxoNEWEpH8q58I4tuk4mhXqNnaVsvdWrBoPEevlMRUF3qOaGJoFkVwIyEeRklqxn/g== X-Received: by 2002:ac2:4426:: with SMTP id w6mr20163280lfl.8.1588107598830; Tue, 28 Apr 2020 13:59:58 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id z21sm295483ljh.42.2020.04.28.13.59.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Apr 2020 13:59:58 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 22/24] rcu/tiny: support reclaim for head-less object Date: Tue, 28 Apr 2020 22:59:01 +0200 Message-Id: <20200428205903.61704-23-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200428205903.61704-1-urezki@gmail.com> References: <20200428205903.61704-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make a kvfree_call_rcu() function to support head-less freeing. Same as for tree-RCU, for such purpose we store pointers in array. SLAB and vmalloc ptrs. are mixed and coexist together. Under high memory pressure it can be that maintaining of arrays becomes impossible. Objects with an rcu_head are released via call_rcu(). When it comes to the head-less variant, the kvfree() call is directly inlined, i.e. we do the same as for tree-RCU: a) wait until a grace period has elapsed; b) direct inlining of the kvfree() call. Thus the current context has to follow might_sleep() annotation. Also please note that for tiny-RCU any call of synchronize_rcu() is actually a quiescent state, therefore (a) does nothing. Reviewed-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki (Sony) --- kernel/rcu/tiny.c | 157 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 156 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c index 508c82faa45c..b1c31a935db9 100644 --- a/kernel/rcu/tiny.c +++ b/kernel/rcu/tiny.c @@ -40,6 +40,29 @@ static struct rcu_ctrlblk rcu_ctrlblk = { .curtail = &rcu_ctrlblk.rcucblist, }; +/* Can be common with tree-RCU. */ +#define KVFREE_DRAIN_JIFFIES (HZ / 50) + +/* Can be common with tree-RCU. */ +struct kvfree_rcu_bulk_data { + unsigned long nr_records; + struct kvfree_rcu_bulk_data *next; + void *records[]; +}; + +/* Can be common with tree-RCU. */ +#define KVFREE_BULK_MAX_ENTR \ + ((PAGE_SIZE - sizeof(struct kvfree_rcu_bulk_data)) / sizeof(void *)) + +static struct kvfree_rcu_bulk_data *kvhead; +static struct kvfree_rcu_bulk_data *kvhead_free; +static struct kvfree_rcu_bulk_data *kvcache; + +static DEFINE_STATIC_KEY_FALSE(rcu_init_done); +static struct delayed_work monitor_work; +static struct rcu_work rcu_work; +static bool monitor_todo; + void rcu_barrier(void) { wait_rcu_gp(call_rcu); @@ -177,9 +200,137 @@ void call_rcu(struct rcu_head *head, rcu_callback_t func) } EXPORT_SYMBOL_GPL(call_rcu); +static inline bool +kvfree_call_rcu_add_ptr_to_bulk(void *ptr) +{ + struct kvfree_rcu_bulk_data *bnode; + + if (!kvhead || kvhead->nr_records == KVFREE_BULK_MAX_ENTR) { + bnode = xchg(&kvcache, NULL); + if (!bnode) + bnode = (struct kvfree_rcu_bulk_data *) + __get_free_page(GFP_NOWAIT | __GFP_NOWARN); + + if (unlikely(!bnode)) + return false; + + /* Initialize the new block. */ + bnode->nr_records = 0; + bnode->next = kvhead; + + /* Attach it to the bvhead. */ + kvhead = bnode; + } + + /* Done. */ + kvhead->records[kvhead->nr_records++] = ptr; + return true; +} + +static void +kvfree_rcu_work(struct work_struct *work) +{ + struct kvfree_rcu_bulk_data *kvhead_tofree, *next; + unsigned long flags; + int i; + + local_irq_save(flags); + kvhead_tofree = kvhead_free; + kvhead_free = NULL; + local_irq_restore(flags); + + /* Reclaim process. */ + for (; kvhead_tofree; kvhead_tofree = next) { + next = kvhead_tofree->next; + + for (i = 0; i < kvhead_tofree->nr_records; i++) { + debug_rcu_head_unqueue((struct rcu_head *) + kvhead_tofree->records[i]); + kvfree(kvhead_tofree->records[i]); + } + + if (cmpxchg(&kvcache, NULL, kvhead_tofree)) + free_page((unsigned long) kvhead_tofree); + } +} + +static inline bool +queue_kvfree_rcu_work(void) +{ + /* Check if the free channel is available. */ + if (kvhead_free) + return false; + + kvhead_free = kvhead; + kvhead = NULL; + + /* + * Queue the job for memory reclaim after GP. + */ + queue_rcu_work(system_wq, &rcu_work); + return true; +} + +static void kvfree_rcu_monitor(struct work_struct *work) +{ + unsigned long flags; + bool queued; + + local_irq_save(flags); + queued = queue_kvfree_rcu_work(); + if (queued) + /* Success. */ + monitor_todo = false; + local_irq_restore(flags); + + /* + * If previous RCU reclaim process is still in progress, + * schedule the work one more time to try again later. + */ + if (monitor_todo) + schedule_delayed_work(&monitor_work, + KVFREE_DRAIN_JIFFIES); +} + void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) { - call_rcu(head, func); + unsigned long flags; + bool success; + void *ptr; + + if (head) { + ptr = (void *) head - (unsigned long) func; + } else { + might_sleep(); + ptr = (void *) func; + } + + if (debug_rcu_head_queue(ptr)) { + /* Probable double free, just leak. */ + WARN_ONCE(1, "%s(): Double-freed call. rcu_head %p\n", + __func__, head); + return; + } + + local_irq_save(flags); + success = kvfree_call_rcu_add_ptr_to_bulk(ptr); + if (static_branch_likely(&rcu_init_done)) { + if (success && !monitor_todo) { + monitor_todo = true; + schedule_delayed_work(&monitor_work, + KVFREE_DRAIN_JIFFIES); + } + } + local_irq_restore(flags); + + if (!success) { + if (!head) { + synchronize_rcu(); + kvfree(ptr); + } else { + call_rcu(head, func); + } + } } EXPORT_SYMBOL_GPL(kvfree_call_rcu); @@ -188,4 +339,8 @@ void __init rcu_init(void) open_softirq(RCU_SOFTIRQ, rcu_process_callbacks); rcu_early_boot_tests(); srcu_init(); + + INIT_DELAYED_WORK(&monitor_work, kvfree_rcu_monitor); + INIT_RCU_WORK(&rcu_work, kvfree_rcu_work); + static_branch_enable(&rcu_init_done); } From patchwork Tue Apr 28 20:59:02 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11515599 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 27CAA112C for ; Tue, 28 Apr 2020 21:00:23 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DFB1A2137B for ; Tue, 28 Apr 2020 21:00:22 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="uT3JK0wk" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DFB1A2137B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 877528E001C; Tue, 28 Apr 2020 17:00:02 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8279A8E0007; Tue, 28 Apr 2020 17:00:02 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 73E6F8E001C; Tue, 28 Apr 2020 17:00:02 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0024.hostedemail.com [216.40.44.24]) by kanga.kvack.org (Postfix) with ESMTP id 599BB8E0007 for ; Tue, 28 Apr 2020 17:00:02 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 16D224DA8 for ; Tue, 28 Apr 2020 21:00:02 +0000 (UTC) X-FDA: 76758481044.01.force50_578d1e54c820b X-Spam-Summary: 2,0,0,06afc2d5827a7a03,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:41:355:379:541:800:960:965:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1542:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2559:2562:3138:3139:3140:3141:3142:3355:3865:3866:3867:3868:3870:3871:3872:3874:4117:4321:4385:4390:4395:4605:5007:6119:6261:6653:7514:7875:7904:9413:10004:11026:11473:11658:11914:12043:12291:12295:12296:12297:12438:12517:12519:12555:12683:12895:13161:13229:13894:14181:14394:14687:14721:21063:21080:21433:21444:21451:21627:21666:21796:21939:30029:30036:30054:30070,0,RBL:209.85.208.195:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:44,LUA_SUMMARY:none X-HE-Tag: force50_578d1e54c820b X-Filterd-Recvd-Size: 6154 Received: from mail-lj1-f195.google.com (mail-lj1-f195.google.com [209.85.208.195]) by imf23.hostedemail.com (Postfix) with ESMTP for ; Tue, 28 Apr 2020 21:00:01 +0000 (UTC) Received: by mail-lj1-f195.google.com with SMTP id h4so189218ljg.12 for ; Tue, 28 Apr 2020 14:00:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=7dg4vvvuMGvUTLvHmWPN1Rrhqpd5aScShP8yFUvMy+A=; b=uT3JK0wksSzmJdp1AC64KvjdyD8rxvkukt3pYhWUvkn+XTCa1XhRcC+h0weAIylNWW 5f0SOqHWyVHCSEHwX0q9HOjNPsnQn42cRtWxBuBvHzaC4KYuVFI22rxKRLywnxBir/XK 7mo2M71F/Fmv3ZXo9S/zpiGNWQNvBtZzOpNyeJdWgvfgG3JM1QS8l895wLB4DQihVrdx 5EPpzOGrTUGGfDLSf5MINED8eLwhxcNiA2XALr0K0t/iQOUh4koXDkHGBZ1ILsTEC2DJ g+LhG8On/9yrFZsL4Prg7zlIOUN0NJPUPcjH+ZC/4g0/Cj4+e11UHQqtKK4OgncU4gKV gHsA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=7dg4vvvuMGvUTLvHmWPN1Rrhqpd5aScShP8yFUvMy+A=; b=j/YtX43hp3RU5nq5w6C6XMwtBc+sdZ4dar15rTczyan1WOX2aStGDEPY1cURc2yQxt skp9kiIZAMMXNuBIszSYEJ6yx1lZSWHjEUbH4I8l/B9Txu3mUPL3xqge341ouFU9q1ZD JTEs1hh8LD+Kcmp2sScVGc1ZPOYz7yMnBtURrTNilIFPxoZqTYstCGWics+eA3Sxrt2t HKqnU/NUiRuBAylkTFaLkppD2ZiVOYj6Df/rkiFDjZ1tB+Pu70Yxv9nsRYGMPS+O/rt2 TG3RCdOH6t/rmejsL+pm6knNvvOcusRSYjtw9wmRCqN0jodSmdNlELM07ySTsQ3z9hVq Lnog== X-Gm-Message-State: AGi0Pua5Bc3Iw7LRn3J3QPwjtwCmNxdMzHtJQrMdL5fgF/DqYi9CP5zU zHXxv+kNLBJYm+/ZoQzgpKI= X-Google-Smtp-Source: APiQypJ74RE+BeVO/a4be2iqMHLz/Sbx0gEUw9T9zhqkEcIjYEMEfzVvNufri7CsMeJjoPs58csz2Q== X-Received: by 2002:a19:644f:: with SMTP id b15mr274982lfj.28.1588107600253; Tue, 28 Apr 2020 14:00:00 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id z21sm295483ljh.42.2020.04.28.13.59.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Apr 2020 13:59:59 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 23/24] rcu: Introduce 1 arg kvfree_rcu() interface Date: Tue, 28 Apr 2020 22:59:02 +0200 Message-Id: <20200428205903.61704-24-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200428205903.61704-1-urezki@gmail.com> References: <20200428205903.61704-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make it possible to pass one or two arguments to the kvfree_rcu() macro what corresponds to either headless case or not, so it becomes a bit versatile. As a result we obtain two ways of using that macro, below are two examples: a) kvfree_rcu(ptr, rhf); struct X { struct rcu_head rhf; unsigned char data[100]; }; void *ptr = kvmalloc(sizeof(struct X), GFP_KERNEL); if (ptr) kvfree_rcu(ptr, rhf); b) kvfree_rcu(ptr); void *ptr = kvmalloc(some_bytes, GFP_KERNEL); if (ptr) kvfree_rcu(ptr); Last one, we name it headless variant, only needs one argument, means it does not require any rcu_head to be present within the type of ptr. There is a restriction the (b) context has to fall into might_sleep() annotation. To check that, please activate the CONFIG_DEBUG_ATOMIC_SLEEP option in your kernel. Reviewed-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki (Sony) --- include/linux/rcupdate.h | 38 ++++++++++++++++++++++++++++++++++---- 1 file changed, 34 insertions(+), 4 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 51b26ab02878..d15d46db61f7 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -877,12 +877,42 @@ do { \ /** * kvfree_rcu() - kvfree an object after a grace period. - * @ptr: pointer to kvfree - * @rhf: the name of the struct rcu_head within the type of @ptr. * - * Same as kfree_rcu(), just simple alias. + * This macro consists of one or two arguments and it is + * based on whether an object is head-less or not. If it + * has a head then a semantic stays the same as it used + * to be before: + * + * kvfree_rcu(ptr, rhf); + * + * where @ptr is a pointer to kvfree(), @rhf is the name + * of the rcu_head structure within the type of @ptr. + * + * When it comes to head-less variant, only one argument + * is passed and that is just a pointer which has to be + * freed after a grace period. Therefore the semantic is + * + * kvfree_rcu(ptr); + * + * where @ptr is a pointer to kvfree(). + * + * Please note, head-less way of freeing is permitted to + * use from a context that has to follow might_sleep() + * annotation. Otherwise, please switch and embed the + * rcu_head structure within the type of @ptr. */ -#define kvfree_rcu(ptr, rhf) kfree_rcu(ptr, rhf) +#define kvfree_rcu(...) KVFREE_GET_MACRO(__VA_ARGS__, \ + kvfree_rcu_arg_2, kvfree_rcu_arg_1)(__VA_ARGS__) + +#define KVFREE_GET_MACRO(_1, _2, NAME, ...) NAME +#define kvfree_rcu_arg_2(ptr, rhf) kfree_rcu(ptr, rhf) +#define kvfree_rcu_arg_1(ptr) \ +do { \ + typeof(ptr) ___p = (ptr); \ + \ + if (___p) \ + kvfree_call_rcu(NULL, (rcu_callback_t) (___p)); \ +} while (0) /* * Place this after a lock-acquisition primitive to guarantee that From patchwork Tue Apr 28 20:59:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11515601 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0A8D192C for ; Tue, 28 Apr 2020 21:00:25 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B62672072A for ; Tue, 28 Apr 2020 21:00:24 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="nMGZ+fe6" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B62672072A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B13FC8E001D; Tue, 28 Apr 2020 17:00:03 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A9E5D8E0007; Tue, 28 Apr 2020 17:00:03 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 89F738E001D; Tue, 28 Apr 2020 17:00:03 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0186.hostedemail.com [216.40.44.186]) by kanga.kvack.org (Postfix) with ESMTP id 6FAA38E0007 for ; Tue, 28 Apr 2020 17:00:03 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 2964C181AC9CB for ; Tue, 28 Apr 2020 21:00:03 +0000 (UTC) X-FDA: 76758481086.04.party54_57b75f02c2c02 X-Spam-Summary: 2,0,0,bbfdedbd8116665c,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:41:355:379:541:800:960:965:966:973:982:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1544:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:2987:3138:3139:3140:3141:3142:3354:3865:3866:3867:3868:3870:3871:3872:4118:4321:4385:4390:4395:5007:6261:6653:7514:9163:9413:10004:11026:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12683:12895:13161:13229:13870:13894:14181:14394:14687:14721:21080:21444:21451:21627:21666:21990:30029:30054:30056:30070,0,RBL:209.85.208.195:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: party54_57b75f02c2c02 X-Filterd-Recvd-Size: 7282 Received: from mail-lj1-f195.google.com (mail-lj1-f195.google.com [209.85.208.195]) by imf26.hostedemail.com (Postfix) with ESMTP for ; Tue, 28 Apr 2020 21:00:02 +0000 (UTC) Received: by mail-lj1-f195.google.com with SMTP id g4so259056ljl.2 for ; Tue, 28 Apr 2020 14:00:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=pFz29Ncgyaz8z/1Qp2+b1WJTbiL+UppT5lx2exYc8qs=; b=nMGZ+fe6xiYOw++aC//Z8OF1EzE9mTIAXdtY2wYZBb4quZdq+MAuktQTFSZimsBMfv yr21ycB+3x8EE/b74Vi1nlowx9liJrqN0hO2NzaxZ09mbYg4k5xi/sFv5/E/hf/J9SV/ e68LuotEDHAqBbYzGSwp7SvvkcwdYmbSeLMoXn5Qq0UJyehqB8pu8BGqUIsvl4CBGCPP usSTKwwlhyWEonVJtNoaJp5318qEf+frWelZvLRGS7Lrr9DEw+4lr1MSFUEtPNnVYlp2 ZiNz53ok5DGi27Ny8bImfXSRM6zS9vQSS6Ac2KO1RkonVz58KfwAw+Yk1vsf5eAAN/0f KU+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pFz29Ncgyaz8z/1Qp2+b1WJTbiL+UppT5lx2exYc8qs=; b=dp99ovC3vzVuYTR0yWatwc3cMOax3acvgh27GttMWq/3l7XeYNOFrSKq4hio7KQQed fzacABLZMdgJPVSnwhRF0L9n/dXl8i3V3KpqTzx2natuOMIETj9tUFLQZtsMlI+FmdfJ x6O7Cyk2WYI1n4IwB/YXlwVyoPvEjwcCkNdmy16jW3zZJMMOVjcaULsjkFQxstPjIbqL WWzm8ysOSCMhDN9Y+7i9/tLcLNyOpBKTBk4mJZFfWDSrS2ZuIhtINCRrjVzao7/Cvl6d Wa0aJnnjSxv77AY8J/XhmYuSihZCn2M/fA97o5D9sUowoEEEUiQOJ9obCwHmAdG8OJL9 uXXw== X-Gm-Message-State: AGi0PuYj/fVQ9v75OORk5g2IlB7IJshOCQMfrLb/eWppBARwYkmbke/7 h0X3RhFMknOw2VKgJrrWsM7ONkOaXrBGtQ== X-Google-Smtp-Source: APiQypLE5mxTFoU3KAMfpCj2cbiae08hkJfL0cpRwE+8diBE7xGLdv5s1UOvFjgkkhBVYoqWhWEgHQ== X-Received: by 2002:a2e:8083:: with SMTP id i3mr18677202ljg.175.1588107601389; Tue, 28 Apr 2020 14:00:01 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id z21sm295483ljh.42.2020.04.28.14.00.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Apr 2020 14:00:00 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 24/24] lib/test_vmalloc.c: Add test cases for kvfree_rcu() Date: Tue, 28 Apr 2020 22:59:03 +0200 Message-Id: <20200428205903.61704-25-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200428205903.61704-1-urezki@gmail.com> References: <20200428205903.61704-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce four new test cases. They are considered to cover and testing of kvfree_rcu() interface. Two of them belong to 1 argument functionality and another two for 2 arguments functionality. The aim is to stress it, to check how it behaves under different load and memory conditions, analysis of the performance throughput and its impact. Signed-off-by: Uladzislau Rezki (Sony) --- lib/test_vmalloc.c | 103 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 95 insertions(+), 8 deletions(-) diff --git a/lib/test_vmalloc.c b/lib/test_vmalloc.c index 8bbefcaddfe8..ec73561cda2e 100644 --- a/lib/test_vmalloc.c +++ b/lib/test_vmalloc.c @@ -15,6 +15,8 @@ #include #include #include +#include +#include #define __param(type, name, init, msg) \ static type name = init; \ @@ -35,14 +37,18 @@ __param(int, test_loop_count, 1000000, __param(int, run_test_mask, INT_MAX, "Set tests specified in the mask.\n\n" - "\t\tid: 1, name: fix_size_alloc_test\n" - "\t\tid: 2, name: full_fit_alloc_test\n" - "\t\tid: 4, name: long_busy_list_alloc_test\n" - "\t\tid: 8, name: random_size_alloc_test\n" - "\t\tid: 16, name: fix_align_alloc_test\n" - "\t\tid: 32, name: random_size_align_alloc_test\n" - "\t\tid: 64, name: align_shift_alloc_test\n" - "\t\tid: 128, name: pcpu_alloc_test\n" + "\t\tid: 1, name: fix_size_alloc_test\n" + "\t\tid: 2, name: full_fit_alloc_test\n" + "\t\tid: 4, name: long_busy_list_alloc_test\n" + "\t\tid: 8, name: random_size_alloc_test\n" + "\t\tid: 16, name: fix_align_alloc_test\n" + "\t\tid: 32, name: random_size_align_alloc_test\n" + "\t\tid: 64, name: align_shift_alloc_test\n" + "\t\tid: 128, name: pcpu_alloc_test\n" + "\t\tid: 256, name: kvfree_rcu_1_arg_vmalloc_test\n" + "\t\tid: 512, name: kvfree_rcu_2_arg_vmalloc_test\n" + "\t\tid: 1024, name: kvfree_rcu_1_arg_slab_test\n" + "\t\tid: 2048, name: kvfree_rcu_2_arg_slab_test\n" /* Add a new test case description here. */ ); @@ -328,6 +334,83 @@ pcpu_alloc_test(void) return rv; } +struct test_kvfree_rcu { + struct rcu_head rcu; + unsigned char array[20]; +}; + +static int +kvfree_rcu_1_arg_vmalloc_test(void) +{ + struct test_kvfree_rcu *p; + int i; + + for (i = 0; i < test_loop_count; i++) { + p = vmalloc(1 * PAGE_SIZE); + if (!p) + return -1; + + p->array[0] = 'a'; + kvfree_rcu(p); + } + + return 0; +} + +static int +kvfree_rcu_2_arg_vmalloc_test(void) +{ + struct test_kvfree_rcu *p; + int i; + + for (i = 0; i < test_loop_count; i++) { + p = vmalloc(1 * PAGE_SIZE); + if (!p) + return -1; + + p->array[0] = 'a'; + kvfree_rcu(p, rcu); + } + + return 0; +} + +static int +kvfree_rcu_1_arg_slab_test(void) +{ + struct test_kvfree_rcu *p; + int i; + + for (i = 0; i < test_loop_count; i++) { + p = kmalloc(sizeof(*p), GFP_KERNEL); + if (!p) + return -1; + + p->array[0] = 'a'; + kvfree_rcu(p); + } + + return 0; +} + +static int +kvfree_rcu_2_arg_slab_test(void) +{ + struct test_kvfree_rcu *p; + int i; + + for (i = 0; i < test_loop_count; i++) { + p = kmalloc(sizeof(*p), GFP_KERNEL); + if (!p) + return -1; + + p->array[0] = 'a'; + kvfree_rcu(p, rcu); + } + + return 0; +} + struct test_case_desc { const char *test_name; int (*test_func)(void); @@ -342,6 +425,10 @@ static struct test_case_desc test_case_array[] = { { "random_size_align_alloc_test", random_size_align_alloc_test }, { "align_shift_alloc_test", align_shift_alloc_test }, { "pcpu_alloc_test", pcpu_alloc_test }, + { "kvfree_rcu_1_arg_vmalloc_test", kvfree_rcu_1_arg_vmalloc_test }, + { "kvfree_rcu_2_arg_vmalloc_test", kvfree_rcu_2_arg_vmalloc_test }, + { "kvfree_rcu_1_arg_slab_test", kvfree_rcu_1_arg_slab_test }, + { "kvfree_rcu_2_arg_slab_test", kvfree_rcu_2_arg_slab_test }, /* Add a new test case here. */ };