From patchwork Fri Sep 29 18:00:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 13404691 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46761E728D7 for ; Fri, 29 Sep 2023 18:01:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CBD128D00F9; Fri, 29 Sep 2023 14:01:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C46CA8D002B; Fri, 29 Sep 2023 14:01:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ABF1D8D00F9; Fri, 29 Sep 2023 14:01:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 95DF38D002B for ; Fri, 29 Sep 2023 14:01:27 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 4242AC0ECA for ; Fri, 29 Sep 2023 18:01:27 +0000 (UTC) X-FDA: 81290402214.24.83D66CC Received: from out-199.mta0.migadu.com (out-199.mta0.migadu.com [91.218.175.199]) by imf12.hostedemail.com (Postfix) with ESMTP id 1542B40038 for ; Fri, 29 Sep 2023 18:01:24 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=iy2KaghV; spf=pass (imf12.hostedemail.com: domain of roman.gushchin@linux.dev designates 91.218.175.199 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696010485; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=i5Ua/wdmvBo/wu1nXQ0Cl5cuox9DFv/B5F3hb3WMJFo=; b=KAYqpSv/3MAP6iirul2pq9NvXF2s3pPyl5tswfaYlMFbgen0jV4aOkUj9x2k45XdBjKHRC i/4cPbalTENfnmlCPW3+XcZMhSp5CoXkyQhqpt20MQARA7KFU0F+arfFE0NNZAz9wkoe0z 0h+DQTeH11QmCB04cdOdcn3iylkELKE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696010485; a=rsa-sha256; cv=none; b=bv7G4zrTFf0P45wVNbQrheNpQeq4P0dDxdsipyMsK60+6FDOzD2bVyuJZAwlKW1rdjxBbF ctB64rMGCP6GjYTC9MDv0KsISjdPd1nAOoMV+bpD75qfnwOnWQaLk2f8UoNIL7muDrbBAe ZDIpK/0FdZRltqvRNY9SBUAEvvLlHAQ= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=iy2KaghV; spf=pass (imf12.hostedemail.com: domain of roman.gushchin@linux.dev designates 91.218.175.199 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1696010477; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=i5Ua/wdmvBo/wu1nXQ0Cl5cuox9DFv/B5F3hb3WMJFo=; b=iy2KaghVwmRlwfpu6b7+AOhBWIOkmf5hD6kyAjLMK5XlcnnjDizfH24Jr/mHe5sgea67k7 4WiyJ8S09oEuEbj4kODKMMxIn9KtFUE1gjaYCUzSTGsq7ELsKoT6yLcMwvrxUsNOMZ4P39 APJaRZfIVaJdT9iKHDsVsLui/4J0bOY= From: Roman Gushchin To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Dennis Zhou , Andrew Morton , David Rientjes , Vlastimil Babka , Roman Gushchin Subject: [PATCH v1 1/5] mm: kmem: optimize get_obj_cgroup_from_current() Date: Fri, 29 Sep 2023 11:00:51 -0700 Message-ID: <20230929180056.1122002-2-roman.gushchin@linux.dev> In-Reply-To: <20230929180056.1122002-1-roman.gushchin@linux.dev> References: <20230929180056.1122002-1-roman.gushchin@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 1542B40038 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 8kbs4ot5xr7ed8y9u9ya9zifrsks7yfq X-HE-Tag: 1696010484-126644 X-HE-Meta: U2FsdGVkX19crwk5ZlDv/GD9n9HFbj7NOTjHFsBhKR6Fzkk9GL9fokAUYloUJLm41Z4ymSZ0nFN0WH0zwjMwozkqTWxyZaskDyUF7oHHmM9iy5sUNtbnNTvf9DLwjquQvy7NmjcYiApx9kzYXwYuxmKge4TTlwblbiPXwKeDuxCgnGkCfH2Z2BYcyqUrWaKlpZ+l+/Z8otxjeAcuChz2LgQD6R0HSe67rGo3pEx4hoHOSPVEQNfbFgdbTRmCR0/xs9VI48dQ//jWrS9eQP2YSuUW5pXk1D0NWEsBw9F4viUaeOufrfICTmhB8fU/IOz2Z27ViOdukiSvTYLc9sKX/Pw4zEqecIJSyWG3N4iwSPSVouiGbFBI1WwbdQeO2FftiWk+CmFVCCLTQR187ISBS+K+tbOY5fgPkBvlGWANVw+tIT6BQ6d2NSZyLnRok/24XCbTj8ROy4N5T1fC8/0GtDdVx5Nd9M4TN0YQuB/kc9ebL9LdVRyF0DJWZa2pZwonLEHNcwHcDgfcTECVrMfGD7frxpfxZclMU6N/OgYekUQse1nFGAEgov/1R8NFdVAdrJIdMgEhJnawg0JiwEy06WAZDvkwdPhqg0+x/JDLXW7eNDtfzimVN69oOPCsuhK4t4dSxl/Nf5GkvtrxS4VFboef9B4SvpcbdntT8vz+J+ZnqOlFN2xgTBCDbnh0Z8J54mCeNVLWggIeg7Vqr+lcK+EgFcxZspXYcufI9vYuA1+9GLNIYih5l7tdo2k4ljxO6Lh9V6jzvWwPZ5rA2pXdcvwPTIINiOi5eW45FtNiS13bszkoFwRF1KfnUTsSAK2HLz+xccQJWGqTynLyDxHZ5r3a7jATPuYbiOR8PJhGmJ0W3qKlbf+8Ly7q7p+lknSYptW0BtqgGwuUh7GVZnySe31UfXmwLHva14p9UCqTzB+5i8TO75STVkDzHu6W+4kJChGLJMCQkrfPUCs0Len H/A2IysN vfYABuOYdhYVwRgNFqYiaXpgoH8zpEA/OF141Zm+1DhpZnoZ4FADrxo0wf/V5VnVy9MpX8Lp2HyAwwRXeaVmFtp+ujnzZNOVwPCnbxoqrXkZfMuXZK3St6ttKki390TndbFnJeQ3+qOBICuPNxvqB8h1E6Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Manually inline memcg_kmem_bypass() and active_memcg() to speed up get_obj_cgroup_from_current() by avoiding duplicate in_task() checks and active_memcg() readings. Also add a likely() macro to __get_obj_cgroup_from_memcg(): obj_cgroup_tryget() should succeed at almost all times except a very unlikely race with the memcg deletion path. Signed-off-by: Roman Gushchin (Cruise) Acked-by: Shakeel Butt Acked-by: Johannes Weiner --- mm/memcontrol.c | 34 ++++++++++++++-------------------- 1 file changed, 14 insertions(+), 20 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 9741d62d0424..16ac2a5838fb 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1068,19 +1068,6 @@ struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm) } EXPORT_SYMBOL(get_mem_cgroup_from_mm); -static __always_inline bool memcg_kmem_bypass(void) -{ - /* Allow remote memcg charging from any context. */ - if (unlikely(active_memcg())) - return false; - - /* Memcg to charge can't be determined. */ - if (!in_task() || !current->mm || (current->flags & PF_KTHREAD)) - return true; - - return false; -} - /** * mem_cgroup_iter - iterate over memory cgroup hierarchy * @root: hierarchy root @@ -3007,7 +2994,7 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg)) { objcg = rcu_dereference(memcg->objcg); - if (objcg && obj_cgroup_tryget(objcg)) + if (likely(objcg && obj_cgroup_tryget(objcg))) break; objcg = NULL; } @@ -3016,16 +3003,23 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) { - struct obj_cgroup *objcg = NULL; struct mem_cgroup *memcg; + struct obj_cgroup *objcg; - if (memcg_kmem_bypass()) - return NULL; + if (in_task()) { + memcg = current->active_memcg; + + /* Memcg to charge can't be determined. */ + if (likely(!memcg) && (!current->mm || (current->flags & PF_KTHREAD))) + return NULL; + } else { + memcg = this_cpu_read(int_active_memcg); + if (likely(!memcg)) + return NULL; + } rcu_read_lock(); - if (unlikely(active_memcg())) - memcg = active_memcg(); - else + if (!memcg) memcg = mem_cgroup_from_task(current); objcg = __get_obj_cgroup_from_memcg(memcg); rcu_read_unlock();