From patchwork Thu Sep 16 16:24:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 12499627 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E884CC4332F for ; Thu, 16 Sep 2021 16:27:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CE6136152B for ; Thu, 16 Sep 2021 16:27:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242509AbhIPQ2d (ORCPT ); Thu, 16 Sep 2021 12:28:33 -0400 Received: from mx0b-00082601.pphosted.com ([67.231.153.30]:65282 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S232660AbhIPQ01 (ORCPT ); Thu, 16 Sep 2021 12:26:27 -0400 Received: from pps.filterd (m0089730.ppops.net [127.0.0.1]) by m0089730.ppops.net (8.16.1.2/8.16.1.2) with SMTP id 18GFgtm3027960 for ; Thu, 16 Sep 2021 09:25:06 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=FvkZsdRnuDFdsFs6RIuDJK0lxpqem6zpUwmYQiIpIG4=; b=A+pK/+rZLVyk3N245w0cDN5aG0VOxd+gBaLAQqwoiLk7nrCCQsqw9vd12Lc3/zjUlkfb OnpuSAqHGioIdNO65LTJY/iG2rj8AbBNj3NvrJfRdzRUqpRfQVq14RqyxPFhW1LOKPXw BHU+uKzXLBbs76Xsp1h3kv4NL0b2YxbrDfU= Received: from mail.thefacebook.com ([163.114.132.120]) by m0089730.ppops.net with ESMTP id 3b42vxttgy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 16 Sep 2021 09:25:05 -0700 Received: from intmgw002.46.prn1.facebook.com (2620:10d:c085:208::11) by mail.thefacebook.com (2620:10d:c085:21d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.14; Thu, 16 Sep 2021 09:25:04 -0700 Received: by devvm3388.prn0.facebook.com (Postfix, from userid 111017) id 36BF6BE68AAE; Thu, 16 Sep 2021 09:25:02 -0700 (PDT) From: Roman Gushchin To: Peter Zijlstra , Ingo Molnar CC: Mel Gorman , , , Roman Gushchin Subject: [PATCH rfc 3/6] bpf: sched: introduce bpf_sched_enable() Date: Thu, 16 Sep 2021 09:24:48 -0700 Message-ID: <20210916162451.709260-4-guro@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210916162451.709260-1-guro@fb.com> References: <20210915213550.3696532-1-guro@fb.com> <20210916162451.709260-1-guro@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-FB-Source: Intern X-Proofpoint-GUID: N4OWnV7FrtyBrAEYYACzcI_dk7gikiGB X-Proofpoint-ORIG-GUID: N4OWnV7FrtyBrAEYYACzcI_dk7gikiGB X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475 definitions=2021-09-16_04,2021-09-16_01,2020-04-07_01 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 impostorscore=0 suspectscore=0 priorityscore=1501 mlxlogscore=999 adultscore=0 malwarescore=0 clxscore=1015 phishscore=0 bulkscore=0 lowpriorityscore=0 mlxscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2109030001 definitions=main-2109160098 X-FB-Internal: deliver Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Introduce a dedicated static key and the bpf_sched_enabled() wrapper to guard all invocations of bpf programs in the scheduler code. It will help to avoid any potential performance regression in a case when no scheduler bpf programs are attached. Signed-off-by: Roman Gushchin --- include/linux/bpf_sched.h | 24 ++++++++++++++++++++++++ kernel/bpf/syscall.c | 7 +++++++ kernel/sched/bpf_sched.c | 2 ++ 3 files changed, 33 insertions(+) diff --git a/include/linux/bpf_sched.h b/include/linux/bpf_sched.h index 0f8d3dae53df..6e773aecdff7 100644 --- a/include/linux/bpf_sched.h +++ b/include/linux/bpf_sched.h @@ -6,6 +6,8 @@ #ifdef CONFIG_BPF_SYSCALL +#include + #define BPF_SCHED_HOOK(RET, DEFAULT, NAME, ...) \ RET bpf_sched_##NAME(__VA_ARGS__); #include @@ -14,6 +16,23 @@ int bpf_sched_verify_prog(struct bpf_verifier_log *vlog, const struct bpf_prog *prog); +DECLARE_STATIC_KEY_FALSE(bpf_sched_enabled_key); + +static inline bool bpf_sched_enabled(void) +{ + return static_branch_unlikely(&bpf_sched_enabled_key); +} + +static inline void bpf_sched_inc(void) +{ + static_branch_inc(&bpf_sched_enabled_key); +} + +static inline void bpf_sched_dec(void) +{ + static_branch_dec(&bpf_sched_enabled_key); +} + #else /* CONFIG_BPF_SYSCALL */ #define BPF_SCHED_HOOK(RET, DEFAULT, NAME, ...) \ @@ -23,6 +42,11 @@ static inline RET bpf_sched_##NAME(__VA_ARGS__) \ } #undef BPF_SCHED_HOOK +static inline bool bpf_sched_enabled(void) +{ + return false; +} + #endif /* CONFIG_BPF_SYSCALL */ #endif /* _BPF_CGROUP_H */ diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 67e062376f22..aa5565110498 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -31,6 +31,7 @@ #include #include #include +#include #define IS_FD_ARRAY(map) ((map)->map_type == BPF_MAP_TYPE_PERF_EVENT_ARRAY || \ (map)->map_type == BPF_MAP_TYPE_CGROUP_ARRAY || \ @@ -2602,6 +2603,9 @@ static void bpf_tracing_link_release(struct bpf_link *link) struct bpf_tracing_link *tr_link = container_of(link, struct bpf_tracing_link, link); + if (link->prog->type == BPF_PROG_TYPE_SCHED) + bpf_sched_dec(); + WARN_ON_ONCE(bpf_trampoline_unlink_prog(link->prog, tr_link->trampoline)); @@ -2804,6 +2808,9 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, goto out_unlock; } + if (prog->type == BPF_PROG_TYPE_SCHED) + bpf_sched_inc(); + link->tgt_prog = tgt_prog; link->trampoline = tr; diff --git a/kernel/sched/bpf_sched.c b/kernel/sched/bpf_sched.c index ead691dc6e85..bf92cfb5ecf4 100644 --- a/kernel/sched/bpf_sched.c +++ b/kernel/sched/bpf_sched.c @@ -6,6 +6,8 @@ #include #include "sched.h" +DEFINE_STATIC_KEY_FALSE(bpf_sched_enabled_key); + /* * For every hook declare a nop function where a BPF program can be attached. */