From patchwork Thu Oct 31 21:09:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13858356 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1E0301CCB5E; Thu, 31 Oct 2024 21:09:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730408990; cv=none; b=DFasd46F95FmL2k/TiCDQRX/JR7SyaHi16RgRUWKOI3ftuwK9Rddy32i++/MUEKwdz2QZ5nimHCDCzjRxjRWcA0/52nEAhJCh/skXuyp3qhkY2GjC6hGmwR0jsRk9CXZr9YEfkYLz7/r3MS1Rt/nYQ60iCoW2wLpmmvPfBK38EA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730408990; c=relaxed/simple; bh=xJTzBD6BpyYMRten0/IhV3TKzrQ+ciwoPgRAuiCLgWQ=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=BYXLUXqh2Rc2QBNyikmyNDxtuem5swGImND2md7E8XaWxID9w5I22h52rdten6HrzIy058yXJgY2ZCM79kyxIMAg8xO0vbuLmizs2Ca3t8elEfI5H+nsUnr7GW0DQgsSHJuxxIV0CN6D4bDgEobAUNQz52mrt7RFKAHBdipi8X8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=DMmSnzG4; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="DMmSnzG4" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 81545C4CEC3; Thu, 31 Oct 2024 21:09:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1730408989; bh=xJTzBD6BpyYMRten0/IhV3TKzrQ+ciwoPgRAuiCLgWQ=; h=From:To:Cc:Subject:Date:From; b=DMmSnzG4loA5DacydVvt+7G3FA4+p8BSEYnMRcKB+WLGHWGE9BEXHso4bTYx7U+0k xBBAWNfE/MljIp60YTwR6joAoniDiI2Oksa1jDa0iv0/tiWhOuUcGAcDCxMNbIiddj hK05uvLv7B1lbsNRebel2cP3B65bED2smZRy9DSnFk1W5bOf4ZCv9HwyQu31uumgy3 8TSVAvbScK+Dvfw//5tZuIi7yO2i6bJXWxD9F6Op0VdM5ZW1GV/wKd0dmETzyFNaxm 6SSu5OU/Ta5etKDSNIyiqUDDOIcNE5jE9FXB/ly1sPWprpvAZ5c4bAag+XPu6scKWj zw9cQqjlooXzQ== From: Andrii Nakryiko To: linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, rostedt@goodmis.org, ast@kernel.org, daniel@iogearbox.net, martin.lau@kernel.org Cc: mathieu.desnoyers@efficios.com, linux-kernel@vger.kernel.org, mhiramat@kernel.org, peterz@infradead.org, paulmck@kernel.org, jrife@google.com, Andrii Nakryiko Subject: [PATCH trace/for-next 1/3] bpf: put bpf_link's program when link is safe to be deallocated Date: Thu, 31 Oct 2024 14:09:36 -0700 Message-ID: <20241031210938.1696639-1-andrii@kernel.org> X-Mailer: git-send-email 2.43.5 Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net In general, BPF link's underlying BPF program should be considered to be reachable through attach hook -> link -> prog chain, and, pessimistically, we have to assume that as long as link's memory is not safe to free, attach hook's code might hold a pointer to BPF program and use it. As such, it's not (generally) correct to put link's program early before waiting for RCU GPs to go through. More eager bpf_prog_put() that we currently do is mostly correct due to BPF program's release code doing similar RCU GP waiting, but as will be shown in the following patches, BPF program can be non-sleepable (and, thus, reliant on only "classic" RCU GP), while BPF link's attach hook can have sleepable semantics and needs to be protected by RCU Tasks Trace, and for such cases BPF link has to go through RCU Tasks Trace + "classic" RCU GPs before being deallocated. And so, if we put BPF program early, we might free BPF program before we free BPF link, leading to use-after-free situation. So, this patch defers bpf_prog_put() until we are ready to perform bpf_link's deallocation. At worst, this delays BPF program freeing by one extra RCU GP, but that seems completely acceptable. Alternatively, we'd need more elaborate ways to determine BPF hook, BPF link, and BPF program lifetimes, and how they relate to each other, which seems like an unnecessary complication. Note, for most BPF links we still will perform eager bpf_prog_put() and link dealloc, so for those BPF links there are no observable changes whatsoever. Only BPF links that use deferred dealloc might notice slightly delayed freeing of BPF programs. Also, to reduce code and logic duplication, extract program put + link dealloc logic into bpf_link_dealloc() helper. Signed-off-by: Andrii Nakryiko --- kernel/bpf/syscall.c | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-) diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index a8f1808a1ca5..aa7246a399f3 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -2976,12 +2976,24 @@ void bpf_link_inc(struct bpf_link *link) atomic64_inc(&link->refcnt); } +static void bpf_link_dealloc(struct bpf_link *link) +{ + /* now that we know that bpf_link itself can't be reached, put underlying BPF program */ + if (link->prog) + bpf_prog_put(link->prog); + + /* free bpf_link and its containing memory */ + if (link->ops->dealloc_deferred) + link->ops->dealloc_deferred(link); + else + link->ops->dealloc(link); +} + static void bpf_link_defer_dealloc_rcu_gp(struct rcu_head *rcu) { struct bpf_link *link = container_of(rcu, struct bpf_link, rcu); - /* free bpf_link and its containing memory */ - link->ops->dealloc_deferred(link); + bpf_link_dealloc(link); } static void bpf_link_defer_dealloc_mult_rcu_gp(struct rcu_head *rcu) @@ -3003,7 +3015,6 @@ static void bpf_link_free(struct bpf_link *link) sleepable = link->prog->sleepable; /* detach BPF program, clean up used resources */ ops->release(link); - bpf_prog_put(link->prog); } if (ops->dealloc_deferred) { /* schedule BPF link deallocation; if underlying BPF program @@ -3014,8 +3025,9 @@ static void bpf_link_free(struct bpf_link *link) call_rcu_tasks_trace(&link->rcu, bpf_link_defer_dealloc_mult_rcu_gp); else call_rcu(&link->rcu, bpf_link_defer_dealloc_rcu_gp); - } else if (ops->dealloc) - ops->dealloc(link); + } else if (ops->dealloc) { + bpf_link_dealloc(link); + } } static void bpf_link_put_deferred(struct work_struct *work) From patchwork Thu Oct 31 21:09:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13858357 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 864C21CF29F; Thu, 31 Oct 2024 21:09:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730408993; cv=none; b=C+9OapphInbdn0XOxDKu4uoiHHbXvZa0q/CUIkDloWhlLtPnCf5p4C8zEBEKAvu487k94lrS/wh13gfZbrQwEi7dLUPEopU6kU80ae48m5naCccRNJ10HG55V/0L14C/Jfu4r743LglHlIR02pfcmKexxzi6kNlAjo8p1A3iC28= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730408993; c=relaxed/simple; bh=THS3CaMzEiPMuO3ygX52Vs3wHc1JR87FIqj8W1MmvWM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=AQguAuLZlWjcpaPkJQhiPNcK+728yjTk6QYDxAr67hZpoyFvIMJeM+RJ0n71R48M33mUlGJXizd/AW7l8Mik+bSxLVRsgd/Q6dSDJXyiLmbTY5aZaE886lLZUPr3BGtHQl/y53vYwSL2niCZpUpdKflka6uuAMjNMR0ggioK+x8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=IdWZ20f0; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="IdWZ20f0" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C77B8C4CEC3; Thu, 31 Oct 2024 21:09:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1730408993; bh=THS3CaMzEiPMuO3ygX52Vs3wHc1JR87FIqj8W1MmvWM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IdWZ20f0PqzFVnxCi4YYyg+ruMSqjrRiQycOgpW0CkthT4HkPDUwEBoX2KveBAt/Q 6y4wI4EdkZHZA1uR5WZ0UtpGUfLB2xfAfrW7c1EmdKd2IFnOmsPCb+zXYH6FDLTOEm fNbF9IcqFKcF9YUgQF5AhL4YNDG7n2T2kj2+mRSy3vp7JIS9sRHr+xtc4FnT/Ku4Lx 9caQGlMSLXFBECYR9EQiQu3i+lMIvpZV2E/KFvwtGgEL9de85X4Hl8MkS3MnjK9GvS DGxbl+5zvhibV04FwtxJ9B7vxYDkK1gDvaYFMw2xd2Wb81c8RukPqAXAr6517EVHfH AFss6P2YtTT4A== From: Andrii Nakryiko To: linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, rostedt@goodmis.org, ast@kernel.org, daniel@iogearbox.net, martin.lau@kernel.org Cc: mathieu.desnoyers@efficios.com, linux-kernel@vger.kernel.org, mhiramat@kernel.org, peterz@infradead.org, paulmck@kernel.org, jrife@google.com, Andrii Nakryiko Subject: [PATCH trace/for-next 2/3] bpf: decouple BPF link/attach hook and BPF program sleepable semantics Date: Thu, 31 Oct 2024 14:09:37 -0700 Message-ID: <20241031210938.1696639-2-andrii@kernel.org> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241031210938.1696639-1-andrii@kernel.org> References: <20241031210938.1696639-1-andrii@kernel.org> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net BPF link's lifecycle protection scheme depends on both BPF hook and BPF program. If *either* of those require RCU Tasks Trace GP, then we need to go through a chain of GPs before putting BPF program refcount and deallocating BPF link memory. This patch adds bpf_link-specific sleepable flag, which can be set to true even if underlying BPF program is not sleepable itself. If either link->sleepable or link->prog->sleepable is true, we'll go through a chain of RCU Tasks Trace GP and RCU GP before putting BPF program and freeing memory. This will be used to protect BPF link for sleepable (faultable) raw tracepoints in the next patch. Signed-off-by: Andrii Nakryiko --- include/linux/bpf.h | 22 +++++++++++++++++++--- kernel/bpf/syscall.c | 39 ++++++++++++++++++++++++++++----------- 2 files changed, 47 insertions(+), 14 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 19d8ca8ac960..83f44caf47e1 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1598,6 +1598,11 @@ struct bpf_link { enum bpf_link_type type; const struct bpf_link_ops *ops; struct bpf_prog *prog; + /* whether BPF link itself has "sleepable" semantics, which can differ + * from underlying BPF program having a "sleepable" semantics, as BPF + * link's semantics is determined by target attach hook + */ + bool sleepable; /* rcu is used before freeing, work can be used to schedule that * RCU-based freeing before that, so they never overlap */ @@ -1614,8 +1619,10 @@ struct bpf_link_ops { */ void (*dealloc)(struct bpf_link *link); /* deallocate link resources callback, called after RCU grace period; - * if underlying BPF program is sleepable we go through tasks trace - * RCU GP and then "classic" RCU GP + * if either the underlying BPF program is sleepable or BPF link's + * target hook is sleepable, we'll go through tasks trace RCU GP and + * then "classic" RCU GP; this need for chaining tasks trace and + * classic RCU GPs is designated by setting bpf_link->sleepable flag */ void (*dealloc_deferred)(struct bpf_link *link); int (*detach)(struct bpf_link *link); @@ -2362,6 +2369,9 @@ int bpf_prog_new_fd(struct bpf_prog *prog); void bpf_link_init(struct bpf_link *link, enum bpf_link_type type, const struct bpf_link_ops *ops, struct bpf_prog *prog); +void bpf_link_init_sleepable(struct bpf_link *link, enum bpf_link_type type, + const struct bpf_link_ops *ops, struct bpf_prog *prog, + bool sleepable); int bpf_link_prime(struct bpf_link *link, struct bpf_link_primer *primer); int bpf_link_settle(struct bpf_link_primer *primer); void bpf_link_cleanup(struct bpf_link_primer *primer); @@ -2713,7 +2723,13 @@ bpf_prog_inc_not_zero(struct bpf_prog *prog) static inline void bpf_link_init(struct bpf_link *link, enum bpf_link_type type, const struct bpf_link_ops *ops, - struct bpf_prog *prog) + struct bpf_prog *prog, bool sleepable) +{ +} + +static inline void bpf_link_init_sleepable(struct bpf_link *link, enum bpf_link_type type, + const struct bpf_link_ops *ops, struct bpf_prog *prog, + bool sleepable) { } diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index aa7246a399f3..0f5540627911 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -2933,17 +2933,33 @@ static int bpf_obj_get(const union bpf_attr *attr) attr->file_flags); } -void bpf_link_init(struct bpf_link *link, enum bpf_link_type type, - const struct bpf_link_ops *ops, struct bpf_prog *prog) +/* bpf_link_init_sleepable() allows to specify whether BPF link itself has + * "sleepable" semantics, which normally would mean that BPF link's attach + * hook can dereference link or link's underlying program for some time after + * detachment due to RCU Tasks Trace-based lifetime protection scheme. + * BPF program itself can be non-sleepable, yet, because it's transitively + * reachable through BPF link, its freeing has to be delayed until after RCU + * Tasks Trace GP. + */ +void bpf_link_init_sleepable(struct bpf_link *link, enum bpf_link_type type, + const struct bpf_link_ops *ops, struct bpf_prog *prog, + bool sleepable) { WARN_ON(ops->dealloc && ops->dealloc_deferred); atomic64_set(&link->refcnt, 1); link->type = type; + link->sleepable = sleepable; link->id = 0; link->ops = ops; link->prog = prog; } +void bpf_link_init(struct bpf_link *link, enum bpf_link_type type, + const struct bpf_link_ops *ops, struct bpf_prog *prog) +{ + bpf_link_init_sleepable(link, type, ops, prog, false); +} + static void bpf_link_free_id(int id) { if (!id) @@ -3008,20 +3024,21 @@ static void bpf_link_defer_dealloc_mult_rcu_gp(struct rcu_head *rcu) static void bpf_link_free(struct bpf_link *link) { const struct bpf_link_ops *ops = link->ops; - bool sleepable = false; bpf_link_free_id(link->id); - if (link->prog) { - sleepable = link->prog->sleepable; - /* detach BPF program, clean up used resources */ + /* detach BPF program, clean up used resources */ + if (link->prog) ops->release(link); - } if (ops->dealloc_deferred) { - /* schedule BPF link deallocation; if underlying BPF program - * is sleepable, we need to first wait for RCU tasks trace - * sync, then go through "classic" RCU grace period + /* Schedule BPF link deallocation, which will only then + * trigger putting BPF program refcount. + * If underlying BPF program is sleepable or BPF link's target + * attach hookpoint is sleepable or otherwise requires RCU GPs + * to ensure link and its underlying BPF program is not + * reachable anymore, we need to first wait for RCU tasks + * trace sync, and then go through "classic" RCU grace period */ - if (sleepable) + if (link->sleepable || (link->prog && link->prog->sleepable)) call_rcu_tasks_trace(&link->rcu, bpf_link_defer_dealloc_mult_rcu_gp); else call_rcu(&link->rcu, bpf_link_defer_dealloc_rcu_gp); From patchwork Thu Oct 31 21:09:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13858358 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AE0F31D131E; Thu, 31 Oct 2024 21:09:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730408996; cv=none; b=UUTScjeb8scG7VEMpKlSvKg6LMWhEbzE2+G7xMauiyEYJl6X3O7Doa4/42baZvW/WZDP50tcVdWc4DPQhie8DWe3QJl8p7uHUJHw/9yZnKAG7b7Gh4/3Op0bF41E2/F0PGxw8li9QZuf/Vz8MXCt/d1Qrkqnv7itH4RR7tpz6ig= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730408996; c=relaxed/simple; bh=aa+8Ty0bzdDp3GnvBMFsb/72+qrBl2HEaLiCKUzVr9s=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PxeN+hOHHofXEFl5Pkpa5VvoqymfH3dHuvfINu4WFxEJxXArDLoVNvpDxqUTcZiJFvxsQ8UVMPtsbl+Cvlft4Mp8qUSxFZC3VaXK344HIeTWsBt4ViiauIlAjkDKy8Kjb+AIPjhdpowsn559CyVebALrr2/JH6f8tdMVaB/4ccI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=iPXjVi3g; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="iPXjVi3g" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 079B2C4CEC3; Thu, 31 Oct 2024 21:09:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1730408996; bh=aa+8Ty0bzdDp3GnvBMFsb/72+qrBl2HEaLiCKUzVr9s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=iPXjVi3g+TFRX4AgT+3dtWGZhUpkcUOXHMkv8+7QA0GuyC2X/+H9Dus4Sff3lkNfr ATHbTF48HMmm2wsXFQERU21dhyvVhzq7gg1YFIxNOxH6RUpjJ4d4o1UDkPxAFtRCvE 5V0gvZLYxGfAMmUFJs/AoMx5fUTqJ2yXYIr0uGSMrO6x7cLSFAogsBfzmwdYmdFRWo TJSsXDm+GLIDnShumoeUOsp+xQkV2Jpm1F/+KNhWps/db6/Lz8lJcVBsUBltBMP8FR yRWY5xY4E3jgwzgHyTkJP4FbCmZsWvhUlyH0YI8DsFDj5KwdSkT2cLFbFTwh1zoOvN yfguitZWq9Frw== From: Andrii Nakryiko To: linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, rostedt@goodmis.org, ast@kernel.org, daniel@iogearbox.net, martin.lau@kernel.org Cc: mathieu.desnoyers@efficios.com, linux-kernel@vger.kernel.org, mhiramat@kernel.org, peterz@infradead.org, paulmck@kernel.org, jrife@google.com, Andrii Nakryiko Subject: [PATCH trace/for-next 3/3] bpf: ensure RCU Tasks Trace GP for sleepable raw tracepoint BPF links Date: Thu, 31 Oct 2024 14:09:38 -0700 Message-ID: <20241031210938.1696639-3-andrii@kernel.org> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241031210938.1696639-1-andrii@kernel.org> References: <20241031210938.1696639-1-andrii@kernel.org> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Now that kernel supports sleepable tracepoints, the fact that bpf_probe_unregister() is asynchronous, i.e., that it doesn't wait for any in-flight tracepoints to conclude before returning, we now need to delay BPF raw tp link's deallocation and bpf_prog_put() of its underlying BPF program (regardless of program's own sleepable semantics) until after full RCU Tasks Trace GP. With that GP over, we'll have a guarantee that no tracepoint can reach BPF link and thus its BPF program. We use newly added tracepoint_is_faultable() check to know when this RCU Tasks Trace GP is necessary and utilize BPF link's own sleepable flag passed through bpf_link_init_sleepable() initializer. Reported-by: Jordan Rife Fixes: a363d27cdbc2 ("tracing: Allow system call tracepoints to handle page faults") Signed-off-by: Andrii Nakryiko --- kernel/bpf/syscall.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 0f5540627911..db2a987504b2 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -35,6 +35,7 @@ #include #include #include +#include #include #include @@ -3845,8 +3846,9 @@ static int bpf_raw_tp_link_attach(struct bpf_prog *prog, err = -ENOMEM; goto out_put_btp; } - bpf_link_init(&link->link, BPF_LINK_TYPE_RAW_TRACEPOINT, - &bpf_raw_tp_link_lops, prog); + bpf_link_init_sleepable(&link->link, BPF_LINK_TYPE_RAW_TRACEPOINT, + &bpf_raw_tp_link_lops, prog, + tracepoint_is_faultable(btp->tp)); link->btp = btp; link->cookie = cookie;