From patchwork Fri Feb 2 16:28:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anton Protopopov X-Patchwork-Id: 13543141 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-ed1-f46.google.com (mail-ed1-f46.google.com [209.85.208.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3E7581482F9 for ; Fri, 2 Feb 2024 16:34:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706891659; cv=none; b=PAs/MFRoztVVTZqPlCXitPmGRWN1VfxQNAYFHTEuk5M8FdL22C6cI4wwnpUQTTR1Iy1vNj2LNZM3FwayFEU40nt33zcoe+gotg58oiQqTnB2x6dmAli1DDss3vf8EtX7l55pAyBYlYwx7VEvi2Bm7Ce9yZ7cm2HOOnSP72FD7xE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706891659; c=relaxed/simple; bh=/5eT0KZzLpagKLj0WzCL4hsOrlVf0tQTTb5YC6kHKg0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=hndcBy7RoJMLMhMEI8r5M1gxEk9P2u8s9+kG6vTReOOL6KMS7Bfkp7kUAeQDvWhbROPBtp3lhPHWGb4iNG07sOdB13YLGYuMZUwc+zrW3VzetLiBcJZaojRnSYfBky5dTakl/XU3N/9IX0CVWOpEawAw3WGcBUd1CnsB9+j032E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=isovalent.com; spf=pass smtp.mailfrom=isovalent.com; dkim=pass (2048-bit key) header.d=isovalent.com header.i=@isovalent.com header.b=FzJ4mt5E; arc=none smtp.client-ip=209.85.208.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=isovalent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=isovalent.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=isovalent.com header.i=@isovalent.com header.b="FzJ4mt5E" Received: by mail-ed1-f46.google.com with SMTP id 4fb4d7f45d1cf-55f279dca99so3307669a12.3 for ; Fri, 02 Feb 2024 08:34:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=isovalent.com; s=google; t=1706891655; x=1707496455; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WJEGOO/b1+o1hTO6SMOu4g8owzzUlWXaPImP6sMPI1c=; b=FzJ4mt5EAq3qcW8StnvfUL2BlDBLznYITY2/s+AsjX3tNuRQqpZfa60w2cRIOQPP+K FrASyYtq5Mj/pFa0mJVsJhJKB1QxDS2YDjs+cegMw0ZPyJ2hSqHnrayDZ8ZC8YKI/cU4 fz1KaVIcNCvFAjOoSav/wncjkG4s4dyqT0UpKqBS/hDNhVvW9MyampdTYwB/FjKp50Sj Iu3AI2074PJabQTbvn3BRbiFFGtp2TWr9K9/qnXbQX0/DmUDR7HMyDO9zxvvm9jZYhMk QFr9O3y14ZxPWVKCLEdiJ0tzIFDV2+iOshVWKgAUWvGktm24RyX99djuwpGRNE7ZdNQA Fi8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706891655; x=1707496455; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WJEGOO/b1+o1hTO6SMOu4g8owzzUlWXaPImP6sMPI1c=; b=JpeFArUZ+S2dM0fEEe6bTAh9aWecGb2JPJsp+VHPHAQ4q4mJifP06YKecCRIt4027y nsEFR4WyiSFeOCI7flYgeZHTHfbLxJc++g6vzwz0xVq9N4Y5B1qrrNeuJQ0pP3bCfnOJ 4Xc+z06QyILQGA51wXzEBzH13oT+n4aDimtOwtTeQUyfDxFo+l0Vl3fkhRvFpNiUuCKU cxjuJDiiU7C9KVkmFj/YcTJL1HUDhY6mg+CC6PGl3OBdTkXByBoWVeMjVxFzeeoOW0Vv Bhfsx8bbIXboNVJnHPOKRo1mVRcQGMfjLtgtv5yruoon4UbLiIkNzE3dRF79LIH7Cwai BYjA== X-Gm-Message-State: AOJu0YwUV2IDz8RQq6LvX+zn/n78NMVY/Jbfb1esG4J41tFxibJ2ieZJ Ao7BUAE9ra/7lz+nf+gStzXXHNZq5Mpsj4PkLfKLKHE2N5ysA/SYyaK5r3rclA4= X-Google-Smtp-Source: AGHT+IHW4Mh4tlEA9yDiUqaSQhf9R004AQJtZ2fNeLGQadUe79BVZoJPhbCBUAstzW+j+HYYjlYnJQ== X-Received: by 2002:a05:6402:b23:b0:55f:e7e2:c963 with SMTP id bo3-20020a0564020b2300b0055fe7e2c963mr143996edb.15.1706891655438; Fri, 02 Feb 2024 08:34:15 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCX86VCnTrg8Gf6EVa4kb/04gNQbIPvc2+JRbzkhn9kgL46sbM6tQo9dLEDxXNn2xC/P8+KtJ0kjOiI1nRvsVpsOII5nGgtw6EkebpKRp68oKRpMryVdKNDBk75W2Tmnx2Kh68SoZByXyMgYM58xBCbqvefkPSTGNlRTGl+18ejBx/z5S/fSUcNA1iepaEriMnBuaX5Fx/p3fY0c9xTqx/s9azTMVyh5E6RSvtRCvQb46FCn5TwQUg/b1G7ScxWPElO465/guxj0OsQYXaXExn7xPJz2yv+3r0kdcS+p5M3bPZMQWS5gSYQ= Received: from zh-lab-node-5.home ([2a02:168:f656:0:1ac0:4dff:fe0f:3782]) by smtp.gmail.com with ESMTPSA id l19-20020aa7c313000000b0055edbe94b34sm952544edq.54.2024.02.02.08.34.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Feb 2024 08:34:14 -0800 (PST) From: Anton Protopopov To: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Jiri Olsa , Martin KaFai Lau , Stanislav Fomichev , Yonghong Song , Eduard Zingerman , Quentin Monnet , bpf@vger.kernel.org Cc: Anton Protopopov Subject: [PATCH v1 bpf-next 1/9] bpf: fix potential error return Date: Fri, 2 Feb 2024 16:28:05 +0000 Message-Id: <20240202162813.4184616-2-aspsk@isovalent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240202162813.4184616-1-aspsk@isovalent.com> References: <20240202162813.4184616-1-aspsk@isovalent.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net The bpf_remove_insns() function returns WARN_ON_ONCE(error), where error is a result of bpf_adj_branches(), and thus should be always 0 However, if for any reason it is not 0, then it will be converted to boolean by WARN_ON_ONCE and returned to user space as 1, not an actual error value. Fix this by returning the original err after the WARN check. Signed-off-by: Anton Protopopov Acked-by: Jiri Olsa --- kernel/bpf/core.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 71c459a51d9e..ad8e6f7e0886 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -532,6 +532,8 @@ struct bpf_prog *bpf_patch_insn_single(struct bpf_prog *prog, u32 off, int bpf_remove_insns(struct bpf_prog *prog, u32 off, u32 cnt) { + int err; + /* Branch offsets can't overflow when program is shrinking, no need * to call bpf_adj_branches(..., true) here */ @@ -539,7 +541,12 @@ int bpf_remove_insns(struct bpf_prog *prog, u32 off, u32 cnt) sizeof(struct bpf_insn) * (prog->len - off - cnt)); prog->len -= cnt; - return WARN_ON_ONCE(bpf_adj_branches(prog, off, off + cnt, off, false)); + err = bpf_adj_branches(prog, off, off + cnt, off, false); + WARN_ON_ONCE(err); + if (err) + return err; + + return 0; } static void bpf_prog_kallsyms_del_subprogs(struct bpf_prog *fp) From patchwork Fri Feb 2 16:28:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anton Protopopov X-Patchwork-Id: 13543142 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-ed1-f43.google.com (mail-ed1-f43.google.com [209.85.208.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A221A1487C3 for ; Fri, 2 Feb 2024 16:34:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706891660; cv=none; b=OCl0AmGC6qZwzh0OptS/ZA4XOsnfDSu2a1ob0HA00qTqw6GQVMtQohLILv3IJf23f/qIPW53o5WVsgZXO2z7cFeW0n3eXrCz+WT05FGrXZYFvvei8CkhICRlepYKzA8TZnmqi7DtyWdNxeHIxYwJG9W3iMfVYd5jtxUT6NOSFdY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706891660; c=relaxed/simple; bh=tmNyr/8TlE8ThVK8BbQOa/pYP0yZvj6lq52/DC4I8Cc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=UR4yF2mLroZma7aGrqFRpmSkQkoYyUR1bdI7utYk/NFcDvA8Pm2WaMXs2Hd2Ur/MUiIR7LsvA3m6eaL9FSdFkjboJwEbL/pKYu03aTFIQbXTzcpcevFUBGXFKospYrxDRejYkD6AkroIhfptSyRnv10DSGGc5Awqr2Z5mciHkTQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=isovalent.com; spf=pass smtp.mailfrom=isovalent.com; dkim=pass (2048-bit key) header.d=isovalent.com header.i=@isovalent.com header.b=QcnXrJc2; arc=none smtp.client-ip=209.85.208.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=isovalent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=isovalent.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=isovalent.com header.i=@isovalent.com header.b="QcnXrJc2" Received: by mail-ed1-f43.google.com with SMTP id 4fb4d7f45d1cf-56010d9aab4so21438a12.3 for ; Fri, 02 Feb 2024 08:34:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=isovalent.com; s=google; t=1706891657; x=1707496457; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Z6Av9rZofWvSWATbkpfszZbyRD5PHKzS0dqn78BDygg=; b=QcnXrJc2TvF1qMS+JYefKp2MKL3szCo379yS+QQL9ORBRLHr6JTJ/LebkIiTUnAbUC XrkYR1QmKjcOCxlyinY6y4pCX+wf8fV96rYt+ronUXZFEr8ZGxygP4zOfJPUDS4cwB5o pnWv9RaKE1lKZ6i7+EK+xQa+WcGJ3JivDVmn2suDInndXAN1F8Uxq5mF4QzMHVmA/qy3 Kyz650moOxNCQGmLp9LSFlriBgpyQmgijDPL8bLaWbyCmy/PS4vMTs+MmMSUYbQwEr3u YFWbHkfFDsvl/OO10yjUG6uCPxfdrlIixcfdIvdTVAzFqmXXMzghR+unrC6GGsE0g2DA W1JA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706891657; x=1707496457; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Z6Av9rZofWvSWATbkpfszZbyRD5PHKzS0dqn78BDygg=; b=qnHHQ6bSeE4E+s9XZGwHCPVvuC4SkcZdZS/CiJ8oYK/EVyk11ueP9CFwJzoCU3mAmd JKlKnz6mjWSRJvRt2NNyjY2f7ifDNftIp9Gw8bdRcH1G7pm1WB47/YanR0Z4ycABUxdc WTmSU0Ljs360A5fHeOkXfIAuPJRLLsOa+L4z/lI9XlSbQlNera3IH3IpFoyoYC/N/9YG UmBgR8VJUlzSi4/cPkc/qiGKDr9DJsfOHqMrDVOEmH1QVdPXBTiWtir41aLGY3i0Km6n hLAybffePbKvrBZR6SkH3Y7S1Iz+aGVaStEC6B2Vh4uF4apYyoRn8wDX7qmkL103MWuJ uFQg== X-Gm-Message-State: AOJu0Yy0SekY3OIO3v4Zuj5WP6pjjUT+qTYbI4dveWW5f7lXDDVHjBu9 0+C+QQXjXnhLtxGXBBJdLHPMVU04jIOLGYZu0tx0HSZR2swfOxxJn515OZkeZ7w= X-Google-Smtp-Source: AGHT+IFGla2H3cB8qnKRvziL8FkSXI/HTkNNQHMvA+aiOLT4g1AP3JtRZgaHdeXzuH703YTmrtvkrw== X-Received: by 2002:a05:6402:1647:b0:55f:8103:1943 with SMTP id s7-20020a056402164700b0055f81031943mr125783edx.38.1706891656939; Fri, 02 Feb 2024 08:34:16 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCXN+6452jaPC8iqqUt+L7GIjEsEHbsyRgMgd8pXT4StLt2Gg5objvvc0k69L5j8FkQLLe86gD0mnZhwSjq0Q6+WtjYlFXItWV7NHcaortBbD6nF91nGMwdFFEPByF1advd8BMIrFQG0CowlUUww4aPKZB8VZNiwz7I9h6UKo2TKSPjwhVxPyJG7v4lGyjz7TuUR4riK4P/09Sk5g52IJUyhfRw1moXXBi3S49+sDEUqSAHYwYPDxBxdcn0efqoab2Uo9L8zZ7Sy1q2oYs8CLX8Zq+AnhoxiTIidtSHtGRtRhzU+3X2R1tg= Received: from zh-lab-node-5.home ([2a02:168:f656:0:1ac0:4dff:fe0f:3782]) by smtp.gmail.com with ESMTPSA id l19-20020aa7c313000000b0055edbe94b34sm952544edq.54.2024.02.02.08.34.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Feb 2024 08:34:15 -0800 (PST) From: Anton Protopopov To: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Jiri Olsa , Martin KaFai Lau , Stanislav Fomichev , Yonghong Song , Eduard Zingerman , Quentin Monnet , bpf@vger.kernel.org Cc: Anton Protopopov Subject: [PATCH v1 bpf-next 2/9] bpf: keep track of and expose xlated insn offsets Date: Fri, 2 Feb 2024 16:28:06 +0000 Message-Id: <20240202162813.4184616-3-aspsk@isovalent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240202162813.4184616-1-aspsk@isovalent.com> References: <20240202162813.4184616-1-aspsk@isovalent.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net On bpf(BPF_PROG_LOAD) syscall a user-supplied program is translated by the verifier into an "xlated" program. During this process the original instruction offsets might be adjusted and/or individual instructions might be replaced by a new set of instructions: User-supplied prog: ---> Xlated prog: -- func 0 -- -- func 0 -- insn insn' ... ... insn insn' -- func 1 -- -- func 1 -- insn insn' ... ... insn insn' -- func N -- -- func N -- insn insn' ... ... insn insn' We want to provide users (and ourselves) with the off(insn') -> off(insn) mapping so that when an xlated program is returned to the userspace by the bpf_prog_get_info_by_fd() function, users can determine the real offsets of instructions of interest. Since commit 9e4c24e7ee7d ("bpf: verifier: record original instruction index") the verifier saves the original instruction index in env->insn_aux_data. This information was, however, lost when we patched instructions. Also, the information about original index was kept in the verifier env only, so was inaccessible by later stages, like constants blinding during the jit stage. To address the above issues save the information about the original indexes in a separate array inside the prog->aux so that it doesn't depend on the verifier environment and can be adjusted, and accessed, during later stages. To let users access the information after the program was loaded, add new fields, orig_idx_len and orig_idx to struct bpf_prog_info and patch the bpf_prog_get_info_by_fd function correspondingly. Example mapping would be something like this: Original prog: Xlated prog: 0: r1 = 0x0 0: r1 = 0 1: *(u32 *)(r10 - 0x4) = r1 1: *(u32 *)(r10 -4) = r1 2: r2 = r10 2: r2 = r10 3: r2 += -0x4 3: r2 += -4 4: r1 = 0x0 ll 4: r1 = map[id:88] 6: call 0x1 6: r1 += 272 7: r0 = *(u32 *)(r2 +0) 8: if r0 >= 0x1 goto pc+3 9: r0 <<= 3 10: r0 += r1 11: goto pc+1 12: r0 = 0 7: r6 = r0 13: r6 = r0 8: if r6 == 0x0 goto +0x2 14: if r6 == 0x0 goto pc+4 9: call 0x76 15: r0 = 0xffffffff8d2079c0 17: r0 = *(u64 *)(r0 +0) 10: *(u64 *)(r6 + 0x0) = r0 18: *(u64 *)(r6 +0) = r0 11: r0 = 0x0 19: r0 = 0x0 12: exit 20: exit Here the orig_idx array has length 21 and is equal to (0, 1, 2, 3, 4, 0/*undefined*/, 6, 6, 6, 6, 6, 6, 6, 7, 8, 9, 9, 10, 11, 12) The item 6 is undefined because the r1=0ll occupies 16 bytes. Signed-off-by: Anton Protopopov --- include/linux/bpf.h | 2 ++ include/linux/bpf_verifier.h | 1 - include/uapi/linux/bpf.h | 2 ++ kernel/bpf/core.c | 29 +++++++++++++++++++++++++++++ kernel/bpf/syscall.c | 30 ++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 6 ++---- tools/include/uapi/linux/bpf.h | 2 ++ 7 files changed, 67 insertions(+), 5 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 1ebbee1d648e..4def3dde35f6 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1522,6 +1522,8 @@ struct bpf_prog_aux { struct work_struct work; struct rcu_head rcu; }; + /* an array of original indexes for all xlated instructions */ + u32 *orig_idx; }; struct bpf_prog { diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 0dcde339dc7e..8348de569f11 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -553,7 +553,6 @@ struct bpf_insn_aux_data { u8 alu_state; /* used in combination with alu_limit */ /* below fields are initialized once */ - unsigned int orig_idx; /* original instruction index */ bool jmp_point; bool prune_point; /* ensure we check state equivalence and save state checkpoint and diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index d96708380e52..b929523444b0 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -6533,6 +6533,8 @@ struct bpf_prog_info { __u32 verified_insns; __u32 attach_btf_obj_id; __u32 attach_btf_id; + __u32 orig_idx_len; + __aligned_u64 orig_idx; } __attribute__((aligned(8))); struct bpf_map_info { diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index ad8e6f7e0886..f0086925b810 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -469,6 +469,30 @@ static void bpf_adj_linfo(struct bpf_prog *prog, u32 off, u32 delta) linfo[i].insn_off += delta; } +static int bpf_prog_realloc_orig_idx(struct bpf_prog *prog, u32 off, u32 patch_len) +{ + u32 *old_idx = prog->aux->orig_idx, *new_idx; + u32 new_prog_len = prog->len + patch_len - 1; + int i; + + if (patch_len <= 1) + return 0; + + new_idx = kzalloc(array_size(new_prog_len, sizeof(u32)), GFP_KERNEL); + if (!new_idx) + return -ENOMEM; + + memcpy(new_idx, old_idx, sizeof(*old_idx) * off); + for (i = off; i < off + patch_len; i++) + new_idx[i] = old_idx[off]; + memcpy(new_idx + off + patch_len, old_idx + off + 1, + sizeof(*old_idx) * (prog->len - off)); + + prog->aux->orig_idx = new_idx; + kfree(old_idx); + return 0; +} + struct bpf_prog *bpf_patch_insn_single(struct bpf_prog *prog, u32 off, const struct bpf_insn *patch, u32 len) { @@ -494,6 +518,10 @@ struct bpf_prog *bpf_patch_insn_single(struct bpf_prog *prog, u32 off, (err = bpf_adj_branches(prog, off, off + 1, off + len, true))) return ERR_PTR(err); + err = bpf_prog_realloc_orig_idx(prog, off, len); + if (err) + return ERR_PTR(err); + /* Several new instructions need to be inserted. Make room * for them. Likely, there's no need for a new allocation as * last page could have large enough tailroom. @@ -2778,6 +2806,7 @@ static void bpf_prog_free_deferred(struct work_struct *work) } else { bpf_jit_free(aux->prog); } + kfree(aux->orig_idx); } void bpf_prog_free(struct bpf_prog *fp) diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index b2750b79ac80..172bf8d3aef2 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -2625,6 +2625,18 @@ static bool is_perfmon_prog_type(enum bpf_prog_type prog_type) } } +static void *bpf_prog_alloc_orig_idx(u32 insn_cnt) +{ + u32 *orig_idx; + int i; + + orig_idx = kcalloc(insn_cnt, sizeof(*orig_idx), GFP_KERNEL); + if (orig_idx) + for (i = 0; i < insn_cnt; i++) + orig_idx[i] = i; + return orig_idx; +} + /* last field in 'union bpf_attr' used by this command */ #define BPF_PROG_LOAD_LAST_FIELD prog_token_fd @@ -2760,6 +2772,12 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) goto put_token; } + prog->aux->orig_idx = bpf_prog_alloc_orig_idx(attr->insn_cnt); + if (!prog->aux->orig_idx) { + err = -ENOMEM; + goto free_prog; + } + prog->expected_attach_type = attr->expected_attach_type; prog->aux->attach_btf = attach_btf; prog->aux->attach_btf_id = attr->attach_btf_id; @@ -4541,6 +4559,18 @@ static int bpf_prog_get_info_by_fd(struct file *file, return -EFAULT; } + ulen = info.orig_idx_len; + if (prog->aux->orig_idx) + info.orig_idx_len = prog->len * sizeof(*prog->aux->orig_idx); + else + info.orig_idx_len = 0; + if (info.orig_idx_len && ulen) { + if (copy_to_user(u64_to_user_ptr(info.orig_idx), + prog->aux->orig_idx, + min_t(u32, info.orig_idx_len, ulen))) + return -EFAULT; + } + if (bpf_prog_is_offloaded(prog->aux)) { err = bpf_prog_offload_info_fill(&info, prog); if (err) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index cd4d780e5400..2dc48f88f43c 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -18289,7 +18289,7 @@ static struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 of if (PTR_ERR(new_prog) == -ERANGE) verbose(env, "insn %d cannot be patched due to 16-bit range\n", - env->insn_aux_data[off].orig_idx); + env->prog->aux->orig_idx[off]); vfree(new_data); return NULL; } @@ -20829,7 +20829,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, __u3 { u64 start_time = ktime_get_ns(); struct bpf_verifier_env *env; - int i, len, ret = -EINVAL, err; + int len, ret = -EINVAL, err; u32 log_true_size; bool is_priv; @@ -20852,8 +20852,6 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, __u3 ret = -ENOMEM; if (!env->insn_aux_data) goto err_free_env; - for (i = 0; i < len; i++) - env->insn_aux_data[i].orig_idx = i; env->prog = *prog; env->ops = bpf_verifier_ops[env->prog->type]; env->fd_array = make_bpfptr(attr->fd_array, uattr.is_kernel); diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index d96708380e52..b929523444b0 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -6533,6 +6533,8 @@ struct bpf_prog_info { __u32 verified_insns; __u32 attach_btf_obj_id; __u32 attach_btf_id; + __u32 orig_idx_len; + __aligned_u64 orig_idx; } __attribute__((aligned(8))); struct bpf_map_info { From patchwork Fri Feb 2 16:28:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anton Protopopov X-Patchwork-Id: 13543144 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-lj1-f180.google.com (mail-lj1-f180.google.com [209.85.208.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1D9B814831F for ; Fri, 2 Feb 2024 16:34:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706891662; cv=none; b=nzvbP3G4J9hA/LErAyLPZc9IWHYaFigMxa2lmjajlSQIc28M7WHfd8cz7YpMOeurHuc2XZpjDEXpyYd8jTzPg0bq/BIEA+XeTRGOl88uhBSxNFlETwsIMIqmutJpcNQ2J0tD/i2sREyKSZiGA6rnRKYWuZEVCrxQBT4lxysEOis= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706891662; c=relaxed/simple; bh=eQRCvv/nvao4Ag3993uceu9j2Izh1r0MC4HF8KLpIAE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=bO1MAhCTIpEJR/4Nm2hCrx0hgBPbymwH6qbeB61saa6gYnZ6yfxVL+e2oO1HxefbatEayOS44L3IwpskqUEeXON7vnY4AjpAuJ+Ggjl+1zI/va8dIsUFrbZ1gA8qo8iaId7X5DUzEjxhBqAHWToLhBrylCfKO5VAYIhNdLt59VE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=isovalent.com; spf=pass smtp.mailfrom=isovalent.com; dkim=pass (2048-bit key) header.d=isovalent.com header.i=@isovalent.com header.b=JzGviSB0; arc=none smtp.client-ip=209.85.208.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=isovalent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=isovalent.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=isovalent.com header.i=@isovalent.com header.b="JzGviSB0" Received: by mail-lj1-f180.google.com with SMTP id 38308e7fff4ca-2cf45305403so28254361fa.2 for ; Fri, 02 Feb 2024 08:34:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=isovalent.com; s=google; t=1706891658; x=1707496458; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kDxa/0DzWT7yXuuXKohUyBoMTDCBisl0p5q7nAF3Gaw=; b=JzGviSB0scQWzusZ8wjZI0iBVEAUsEfTI6Osx7C4lUS0iXerV99oV86dXEAkL0HADG QV8Zly9JEp4mCtMs1x3dhUSnFXHv8IgwXwJ2N9fK9GLxc5FD7gNFpkB6LvE/Hnzjy11N H7/QzUav6ybyqCxABPuB6hEQ/ob4nnb3McCbzW2K8U4FJPsUkzcHpQrIXv2sRhAuNIFN 7yoWR7RhOEQgN4/cUkZRJ6wk+4Kl6bJDxsw9UVYojVhmK0Kn4fQPI4RKWKl9NJtI+jrE JPRbtdrRJwlvmBIvEglwelKMVnRbO+AIw+c3KVtF0Zyy5hSc18P9aLRztYslfnopF+zn 8Usg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706891658; x=1707496458; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kDxa/0DzWT7yXuuXKohUyBoMTDCBisl0p5q7nAF3Gaw=; b=mcRpcJZakcRcgg1c2XRWf4miSIZoa4zExtxEEjrTOt8QP9ZPDMPNYuQWuHsxF6B/19 RytAtSZzSuI2mx5k0gY66y/+ITlvymr722KNhmyjZ+lM3LmSlW+JtRepXITWhnnDfJV8 XAwMKg2UQMJu/qR6gSXC4hVpz8BCXaSSgKwVHuxWnr8sK61S6/5fyA6oc21E0l23SgDr ReWTOSZXTxZ2P+AloiKeWBXEMHPPtXpheC2tdzX1xUI1kTjazjLgowcHtOSYVJoOiyHP ZAn4TmQ1KTVm0Z29V2j+dXukJqB500pk7/waN86BVNstaSyYjsEo14PMRUIw8l4dkLVX NkSA== X-Gm-Message-State: AOJu0YyoDxqAAonKyRUommdQo/Kgsp+5RbKZ3cQeb/eN2eH86lZGLIso 9pTsx1c0hEE8xoxk9M07lQauASiYTEjR0NbpvHO141Ttv7OMQV5I8M1UjpEGP/g= X-Google-Smtp-Source: AGHT+IHJ91LEBI8SkIeU4NotnRCW9xrmQ5th/m8NP24jJ3MaropCq2+k6AGTuYrg+1lzNolQjujgYg== X-Received: by 2002:a2e:9c0c:0:b0:2cf:35d8:31ee with SMTP id s12-20020a2e9c0c000000b002cf35d831eemr3293122lji.16.1706891657899; Fri, 02 Feb 2024 08:34:17 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCUsoMwV0Pvr3lU0X5HSYWR4JVlTpiNTKS0JcNVSm3ysOkuenUbEGhHY1/cb0ACGp496szm/q/+mentJx0fluvKvO+NrJn/W5GTupkAcVGw5UEsxasT66vfJPSAY+T7+ehrj5w3f8hSiAqJNLQIOUtVMR4KuPlBurU5aOaj9VxLPxhyo2RYCKm16bYbX5ba1uoxDwLujapr90sjFocMGxdsMwZS1JcEueMa1iz2CiDrCoWWc+VuFsCJhQZBI60eBKa7ETw5nxC8bK/SU7qX3ruA98okV1JicMUqHvf5Yg6aLbo1Q2KolRU8= Received: from zh-lab-node-5.home ([2a02:168:f656:0:1ac0:4dff:fe0f:3782]) by smtp.gmail.com with ESMTPSA id l19-20020aa7c313000000b0055edbe94b34sm952544edq.54.2024.02.02.08.34.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Feb 2024 08:34:17 -0800 (PST) From: Anton Protopopov To: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Jiri Olsa , Martin KaFai Lau , Stanislav Fomichev , Yonghong Song , Eduard Zingerman , Quentin Monnet , bpf@vger.kernel.org Cc: Anton Protopopov Subject: [PATCH v1 bpf-next 3/9] bpf: expose how xlated insns map to jitted insns Date: Fri, 2 Feb 2024 16:28:07 +0000 Message-Id: <20240202162813.4184616-4-aspsk@isovalent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240202162813.4184616-1-aspsk@isovalent.com> References: <20240202162813.4184616-1-aspsk@isovalent.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Allow users to get the exact xlated -> jitted instructions mapping. This is done by using a new field xlated_to_jit in bpf_prog_info which can return up to prog->len struct bpf_xlated_to_jit { u32 off; u32 len; }; elements. The xlated_to_jit[insn_off] contains jitted offset within a function and the length of the resulting jitted instruction. Example: Original: Xlated: Jitted: 0: nopl (%rax,%rax) 5: nop 7: pushq %rbp 8: movq %rsp, %rbp 0: call 0x76 0: r0 = 0xfffffbeef b: movabsq $-1923847220, %rax 2: r0 = *(u64 *)(r0 +0) 15: movq (%rax), %rax 1: r1 = 0x9 ll 3: r1 = map[id:666][0]+9 19: movabsq $-102223334445559, %rdi 3: r2 = 0x6 5: r2 = 6 23: movl $6, %esi 4: r3 = r0 6: r3 = r0 28: movq %rax, %rdx 5: call 0x6 7: call bpf_trace_printk 2b: callq 0xffffffffcdead4dc 6: call pc+2 8: call pc+2 30: callq 0x7c 7: r0 = 0x0 9: r0 = 0 35: xorl %eax, %eax 8: exit 10: exit 37: leave 38: jmp 0xffffffffcbeeffbc --- --- --- 0: nopl (%rax,%rax) 5: nop 7: pushq %rbp 8: movq %rsp, %rbp 9: goto +0x1 11: goto pc+1 b: jmp 0xf 10: goto +0x1 12: goto pc+1 d: jmp 0x11 11: goto -0x2 13: goto pc-2 f: jmp 0xd 12: r0 = 0x0 14: r0 = 0 11: xorl %eax, %eax 13: exit 15: exit 13: leave 14: jmp 0xffffffffcbffbeef Here the xlated_to_jit array will be of length 16 (11 + 6) and equal to 0: (0xb, 10) 1: (0,0) /* undefined, as the previous instruction is 16 bytes */ 2: (0x15, 4) 3: (0x19, 10) 4: (0,0) /* undefined, as the previous instruction is 16 bytes */ 5: (0x23, 5) 6: (0x28, 3) 7: (0x2b, 5) 8: (0x30, 5) 9: (0x35, 2) 10: (0x37, 6) 11: (0xb, 2) 12: (0xd, 2) 13: (0xf, 2) 14: (0x11, 2) 15: (0x13, 6) The prologues are "unmapped": no mapping exists for xlated -> [0,b) Signed-off-by: Anton Protopopov --- arch/x86/net/bpf_jit_comp.c | 14 ++++++++++++++ include/linux/bpf.h | 7 +++++++ include/uapi/linux/bpf.h | 7 +++++++ kernel/bpf/core.c | 24 ++++++++++++++++++++++++ kernel/bpf/syscall.c | 25 +++++++++++++++++++++++++ kernel/bpf/verifier.c | 9 +++++++++ tools/include/uapi/linux/bpf.h | 7 +++++++ 7 files changed, 93 insertions(+) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index e1390d1e331b..a80b8c1e7afe 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -1186,6 +1186,7 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image const s32 imm32 = insn->imm; u32 dst_reg = insn->dst_reg; u32 src_reg = insn->src_reg; + int adjust_off = 0; u8 b2 = 0, b3 = 0; u8 *start_of_ldx; s64 jmp_offset; @@ -1290,6 +1291,7 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image emit_mov_imm64(&prog, dst_reg, insn[1].imm, insn[0].imm); insn++; i++; + adjust_off = 1; break; /* dst %= src, dst /= src, dst %= imm32, dst /= imm32 */ @@ -2073,6 +2075,18 @@ st: if (is_imm8(insn->off)) return -EFAULT; } memcpy(rw_image + proglen, temp, ilen); + + if (bpf_prog->aux->xlated_to_jit) { + u32 func_idx = bpf_prog->aux->func_idx; + int off; + + off = i - 1 - adjust_off; + if (func_idx) + off += bpf_prog->aux->func_info[func_idx].insn_off; + + bpf_prog->aux->xlated_to_jit[off].off = proglen; + bpf_prog->aux->xlated_to_jit[off].len = ilen; + } } proglen += ilen; addrs[i] = proglen; diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 4def3dde35f6..bdd6be718e82 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1524,6 +1524,13 @@ struct bpf_prog_aux { }; /* an array of original indexes for all xlated instructions */ u32 *orig_idx; + /* for every xlated instruction point to all generated jited + * instructions, if allocated + */ + struct { + u32 off; /* local offset in the jitted code */ + u32 len; /* the total len of generated jit code */ + } *xlated_to_jit; }; struct bpf_prog { diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index b929523444b0..c874f354c290 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -6493,6 +6493,11 @@ struct sk_reuseport_md { #define BPF_TAG_SIZE 8 +struct bpf_xlated_to_jit { + __u32 off; + __u32 len; +}; + struct bpf_prog_info { __u32 type; __u32 id; @@ -6535,6 +6540,8 @@ struct bpf_prog_info { __u32 attach_btf_id; __u32 orig_idx_len; __aligned_u64 orig_idx; + __u32 xlated_to_jit_len; + __aligned_u64 xlated_to_jit; } __attribute__((aligned(8))); struct bpf_map_info { diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index f0086925b810..8e99c1563a7f 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -493,6 +493,26 @@ static int bpf_prog_realloc_orig_idx(struct bpf_prog *prog, u32 off, u32 patch_l return 0; } +static void adjust_func_info(struct bpf_prog *prog, u32 off, u32 insn_delta) +{ + int i; + + if (insn_delta == 0) + return; + + for (i = 0; i < prog->aux->func_info_cnt; i++) { + if (prog->aux->func_info[i].insn_off <= off) + continue; + prog->aux->func_info[i].insn_off += insn_delta; + } +} + +static void bpf_prog_adj_orig_idx_after_remove(struct bpf_prog *prog, u32 off, u32 len) +{ + memmove(prog->aux->orig_idx + off, prog->aux->orig_idx + off + len, + sizeof(*prog->aux->orig_idx) * (prog->len - off)); +} + struct bpf_prog *bpf_patch_insn_single(struct bpf_prog *prog, u32 off, const struct bpf_insn *patch, u32 len) { @@ -554,6 +574,7 @@ struct bpf_prog *bpf_patch_insn_single(struct bpf_prog *prog, u32 off, BUG_ON(bpf_adj_branches(prog_adj, off, off + 1, off + len, false)); bpf_adj_linfo(prog_adj, off, insn_delta); + adjust_func_info(prog_adj, off, insn_delta); return prog_adj; } @@ -574,6 +595,8 @@ int bpf_remove_insns(struct bpf_prog *prog, u32 off, u32 cnt) if (err) return err; + bpf_prog_adj_orig_idx_after_remove(prog, off, cnt); + return 0; } @@ -2807,6 +2830,7 @@ static void bpf_prog_free_deferred(struct work_struct *work) bpf_jit_free(aux->prog); } kfree(aux->orig_idx); + kfree(aux->xlated_to_jit); } void bpf_prog_free(struct bpf_prog *fp) diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 172bf8d3aef2..36b8fdcfba75 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -4571,6 +4571,31 @@ static int bpf_prog_get_info_by_fd(struct file *file, return -EFAULT; } + ulen = info.xlated_to_jit_len; + if (prog->aux->xlated_to_jit) + info.xlated_to_jit_len = prog->len * sizeof(struct bpf_xlated_to_jit); + else + info.xlated_to_jit_len = 0; + if (info.xlated_to_jit_len && ulen) { + struct bpf_xlated_to_jit *xlated_to_jit; + int i; + + xlated_to_jit = kzalloc(info.xlated_to_jit_len, GFP_KERNEL); + if (!xlated_to_jit) + return -ENOMEM; + for (i = 0; i < prog->len; i++) { + xlated_to_jit[i].off = prog->aux->xlated_to_jit[i].off; + xlated_to_jit[i].len = prog->aux->xlated_to_jit[i].len; + } + if (copy_to_user(u64_to_user_ptr(info.xlated_to_jit), + xlated_to_jit, + min_t(u32, info.xlated_to_jit_len, ulen))) { + kfree(xlated_to_jit); + return -EFAULT; + } + kfree(xlated_to_jit); + } + if (bpf_prog_is_offloaded(prog->aux)) { err = bpf_prog_offload_info_fill(&info, prog); if (err) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 2dc48f88f43c..270dc0a26d03 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -18974,6 +18974,7 @@ static int jit_subprogs(struct bpf_verifier_env *env) func[i]->aux->exception_cb = env->subprog_info[i].is_exception_cb; if (!i) func[i]->aux->exception_boundary = env->seen_exception; + func[i]->aux->xlated_to_jit = prog->aux->xlated_to_jit; func[i] = bpf_int_jit_compile(func[i]); if (!func[i]->jited) { err = -ENOTSUPP; @@ -20832,6 +20833,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, __u3 int len, ret = -EINVAL, err; u32 log_true_size; bool is_priv; + u32 size; /* no program is valid */ if (ARRAY_SIZE(bpf_verifier_ops) == 0) @@ -20981,6 +20983,13 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, __u3 : false; } + if (ret == 0) { + size = array_size(sizeof(*env->prog->aux->xlated_to_jit), env->prog->len); + env->prog->aux->xlated_to_jit = kzalloc(size, GFP_KERNEL); + if (!env->prog->aux->xlated_to_jit) + ret = -ENOMEM; + } + if (ret == 0) ret = fixup_call_args(env); diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index b929523444b0..c874f354c290 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -6493,6 +6493,11 @@ struct sk_reuseport_md { #define BPF_TAG_SIZE 8 +struct bpf_xlated_to_jit { + __u32 off; + __u32 len; +}; + struct bpf_prog_info { __u32 type; __u32 id; @@ -6535,6 +6540,8 @@ struct bpf_prog_info { __u32 attach_btf_id; __u32 orig_idx_len; __aligned_u64 orig_idx; + __u32 xlated_to_jit_len; + __aligned_u64 xlated_to_jit; } __attribute__((aligned(8))); struct bpf_map_info { From patchwork Fri Feb 2 16:28:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anton Protopopov X-Patchwork-Id: 13543145 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-ed1-f44.google.com (mail-ed1-f44.google.com [209.85.208.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6DF451487E8 for ; Fri, 2 Feb 2024 16:34:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706891663; cv=none; b=UqMaY9PW0zZpzVOnec5a4TCOGZdNrJVQJkf7O71W4RsHlXVaCn4OBaBgUMDk3z1FW9zDnM61eqs04MmD/IcizZLIVS0L7lnI/xucjYv2mnreQGlUbGYBqVzWGCbcfs863XbqWp9ZcHmTvk07iM9rTcFEPk5eX9nl0TpEMqapVHo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706891663; c=relaxed/simple; bh=4CKXPN/OMsn2X7A9LSY1+30WXAoBGGBrtjADl9xmZDY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Vpjs+WpbmYh5yCBC+aDuSy6oJ/UiH+iQl0pwKehOsg3/sQ68fLt7PdOo6eq6gk5v+EqO4GQYf4hHjfoi7OT/llAJ41aOwLoUYkvTZ/Mp83xFV1vV4/Oa6xEFuFK3QDQD/YbVv/zLKjANKmoIzAm6+Ei0HTazZ6lOYCVdmIh7Auw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=isovalent.com; spf=pass smtp.mailfrom=isovalent.com; dkim=pass (2048-bit key) header.d=isovalent.com header.i=@isovalent.com header.b=d19JqL/y; arc=none smtp.client-ip=209.85.208.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=isovalent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=isovalent.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=isovalent.com header.i=@isovalent.com header.b="d19JqL/y" Received: by mail-ed1-f44.google.com with SMTP id 4fb4d7f45d1cf-55a8fd60af0so3221507a12.1 for ; Fri, 02 Feb 2024 08:34:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=isovalent.com; s=google; t=1706891659; x=1707496459; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=NcgWct5g6icTJo6bXdXUt1he4RpRZ3Wqq8A1ZwrEFbs=; b=d19JqL/yEHId0KpKeYtrf+dJB0WN6faSg7m3mHzjcaxnGl+pgFQjBnSv0ewWFGf4hz YWuJ3s0PnQJX10f4MIP3EWzxDR6ixL2IaPVxc2XrduC6/uxNiPBSzDdu5PVuA9Okjsd+ jXmtLKwdsl2x+3JcJhfYmSt411uvZoj7dyyN23yI6vVFVZebfqewdfVxU4va+oCo9ZI0 U8XErCOBqPCEXmXQqdDnwKdZPEm3RCDC9nDk9F4nb3bgLCLdvWv4POCdDNLGol98+7/p +18pjkwgBLxBx5ozrSXFfsAIilNlfijCdRZV+xZBTGrDQEvUJyVJUSjf4AY8lgMeZRPa 83wQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706891659; x=1707496459; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NcgWct5g6icTJo6bXdXUt1he4RpRZ3Wqq8A1ZwrEFbs=; b=g75Z+LSLcpe1Q6hAlrem+OfbFzRHAxbaVMM87kys0dt9I8AsYpGuwXZXpiXdxWH/Sv IG3Nu7LWBWBQUnjh9OgkHOurhQ31EwWFgCxQsHTRTcyz4TLf+NCaFZ3WvcCsvn/NqwGw 0l1ySKsQBpM8FefFnvGU5wNzNjt0Vtn0b/zwKsvXA3nxD0L3JkcS5vvnxUUV6mGu+zzk 17+UUsoCuLd/2tLEKKskErgGuqFn3veae4wDg6y5hnCNxB4Om5v9GClrd9HIW79pqvnP ts4U8UVAWw5N2xwBmc+OZocuSf0PMo1DykEz2b5iNDxlJruDgj5Hx7FrwaCmajAlKfY4 UbUw== X-Gm-Message-State: AOJu0Ywjxe/UATmzkQFR74ajO7JD+KEILmH8N471R0ZvR8vc0O/ewL99 xqJfVxiGgnkzwMaCXqRHI5zrPUDuC7iXNAxwKxFNTVyRBlu2Oqvvuh5wCwHjPdI= X-Google-Smtp-Source: AGHT+IHF2ZbsROoFWIZ/jZwJznQH7F92GuJ2EbKe9NJQwSU2N/ks7eObi/7JyXHGa54kzkkBpM1Mzg== X-Received: by 2002:a05:6402:60b:b0:55f:8054:93fb with SMTP id n11-20020a056402060b00b0055f805493fbmr129570edv.29.1706891659482; Fri, 02 Feb 2024 08:34:19 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCWEhHTGLnpKemETPZwGKPzCAeLj2TisX7+NAMYRmPMzdb/S8XqZ3wTOv4E479ViAwK3koRCjHqFYFSIVqryvvQCOizX6767VqzCIYhsxGCFcFBeKBUKDhc4JXr03KpsOJ5lLKAwWaliRpgPrLoHizOiBFuCNCdzGeeulhz7WmRGFpAPaOegM1Cypf4PolzrQrfwlAEaQwUlUjmKYlGaQ2vR92JVnWDpyEHxJfWyRXVgPakKF3P8DkZMBrZsy9NScD0d2/q2ib8JOEhQncxf9fzkGtxjYnRrlFtyTiuARNzgUfXbMwcHEWo= Received: from zh-lab-node-5.home ([2a02:168:f656:0:1ac0:4dff:fe0f:3782]) by smtp.gmail.com with ESMTPSA id l19-20020aa7c313000000b0055edbe94b34sm952544edq.54.2024.02.02.08.34.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Feb 2024 08:34:18 -0800 (PST) From: Anton Protopopov To: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Jiri Olsa , Martin KaFai Lau , Stanislav Fomichev , Yonghong Song , Eduard Zingerman , Quentin Monnet , bpf@vger.kernel.org Cc: Anton Protopopov Subject: [PATCH v1 bpf-next 4/9] selftests/bpf: Add tests for instructions mappings Date: Fri, 2 Feb 2024 16:28:08 +0000 Message-Id: <20240202162813.4184616-5-aspsk@isovalent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240202162813.4184616-1-aspsk@isovalent.com> References: <20240202162813.4184616-1-aspsk@isovalent.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Add several self-tests to test the instructions mappings from xlated to original: * check that mappings work for a program without patches * same for a program with patches to insns * same for a program with patches to insns and bpf-to-bpf calls * same for a program with patches and deletions * same for a program with patches, deletions, and bpf-to-bpf calls Signed-off-by: Anton Protopopov --- .../bpf/prog_tests/bpf_insns_mappings.c | 156 ++++++++++++++++++ .../selftests/bpf/progs/bpf_insns_mappings.c | 155 +++++++++++++++++ 2 files changed, 311 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/bpf_insns_mappings.c create mode 100644 tools/testing/selftests/bpf/progs/bpf_insns_mappings.c diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_insns_mappings.c b/tools/testing/selftests/bpf/prog_tests/bpf_insns_mappings.c new file mode 100644 index 000000000000..2a7b53231080 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/bpf_insns_mappings.c @@ -0,0 +1,156 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Isovalent */ + +#include +#include "bpf_insns_mappings.skel.h" + +#define MAX_INSNS 4096 + +static struct bpf_prog_info *prog_info_and_mappings(int prog_fd) +{ + static __thread struct bpf_prog_info prog_info; + static __thread char xlated_insns[MAX_INSNS]; + static __thread __u32 orig_idx[MAX_INSNS]; + __u32 prog_info_len; + __u32 orig_idx_len; + int err; + + prog_info_len = sizeof(prog_info); + + memset(&prog_info, 0, sizeof(prog_info)); + err = bpf_prog_get_info_by_fd(prog_fd, &prog_info, &prog_info_len); + if (!ASSERT_GE(err, 0, "bpf_prog_get_info_by_fd")) + return NULL; + + orig_idx_len = prog_info.orig_idx_len; + memset(&prog_info, 0, sizeof(prog_info)); + + if (orig_idx_len) { + prog_info.orig_idx_len = orig_idx_len; + prog_info.orig_idx = ptr_to_u64(orig_idx); + } + + prog_info.xlated_prog_insns = ptr_to_u64(xlated_insns); + prog_info.xlated_prog_len = sizeof(xlated_insns); + + err = bpf_prog_get_info_by_fd(prog_fd, &prog_info, &prog_info_len); + if (!ASSERT_GE(err, 0, "bpf_prog_get_info_by_fd")) + return NULL; + + return &prog_info; +} + +static int beef_search_original(const struct bpf_insn *insns, int n_insns, int *idx, int n_max) +{ + int i, n_found = 0; + + for (i = 0; i < n_insns; i++) { + if (insns[i].imm == 0xbeef) { + if (!ASSERT_LT(n_found, n_max, "beef")) + return -1; + idx[n_found++] = i; + } + } + + return n_found; +} + +static int beef_search_xlated(struct bpf_prog_info *info, int *idx, int len) +{ + struct bpf_insn *insns = u64_to_ptr(info->xlated_prog_insns); + int tot = info->xlated_prog_len / 8; + int i, n = 0; + + for (i = 0; i < tot; i++) { + if (insns[i].imm == 0xbeef) { + if (!ASSERT_LT(n, len, "beef")) + return -1; + idx[n++] = ((__u32 *)u64_to_ptr(info->orig_idx))[i]; + } + } + + return n; +} + +static void beef_check(const struct bpf_program *prog, int n_expected) +{ + struct bpf_prog_info *prog_info; + int idx_expected[MAX_INSNS]; + int idx[MAX_INSNS]; + int prog_fd; + int n, i; + + /* + * Find all beef instructions in the original program + */ + + n = beef_search_original(bpf_program__insns(prog), + bpf_program__insn_cnt(prog), + idx_expected, MAX_INSNS); + if (!ASSERT_EQ(n, n_expected, "search original insns")) + return; + + /* + * Now find all the beef instructions in the xlated program and extract + * corresponding orig_idx mappings + */ + prog_fd = bpf_program__fd(prog); + if (!ASSERT_GE(prog_fd, 0, "bpf_program__fd")) + return; + + prog_info = prog_info_and_mappings(prog_fd); + if (!ASSERT_OK_PTR(prog_info, "prog_info_and_mappings")) + return; + + if (!ASSERT_EQ(beef_search_xlated(prog_info, idx, n), n, "total # of beef")) + return; + + /* + * Check that the orig_idx points to the correct original indexes + */ + for (i = 0; i < n; i++) + ASSERT_EQ(idx[i], idx_expected[i], "beef index"); +} + +static void check_prog(const struct bpf_program *prog, int n_expected) +{ + struct bpf_link *link; + + link = bpf_program__attach(prog); + if (!ASSERT_OK_PTR(link, "link")) + return; + + beef_check(prog, n_expected); + + bpf_link__destroy(link); +} + +void test_bpf_insns_mappings(void) +{ + struct bpf_insns_mappings *skel; + + skel = bpf_insns_mappings__open(); + if (!ASSERT_OK_PTR(skel, "bpf_insns_mappings__open")) + return; + + if (!ASSERT_OK(bpf_insns_mappings__load(skel), + "bpf_insns_mappings__load")) + return; + + if (test__start_subtest("check_trivial_prog")) + check_prog(skel->progs.check_trivial_prog, 3); + + if (test__start_subtest("check_simple_prog")) + check_prog(skel->progs.check_simple_prog, 3); + + if (test__start_subtest("check_bpf_to_bpf")) + check_prog(skel->progs.check_bpf_to_bpf, 6); + + if (test__start_subtest("check_prog_dead_code")) + check_prog(skel->progs.check_prog_dead_code, 13); + + if (test__start_subtest("check_prog_dead_code_bpf_to_bpf")) + check_prog(skel->progs.check_prog_dead_code_bpf_to_bpf, 26); + + bpf_insns_mappings__destroy(skel); +} diff --git a/tools/testing/selftests/bpf/progs/bpf_insns_mappings.c b/tools/testing/selftests/bpf/progs/bpf_insns_mappings.c new file mode 100644 index 000000000000..f6ff690801ea --- /dev/null +++ b/tools/testing/selftests/bpf/progs/bpf_insns_mappings.c @@ -0,0 +1,155 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Isovalent */ + +#include "vmlinux.h" +#include +#include "bpf_misc.h" + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __uint(max_entries, 1); + __type(key, u32); + __type(value, u64); +} just_a_map SEC(".maps"); + +static inline void beef(void) +{ + asm volatile("r8 = 0xbeef" ::: "r8"); +} + +static inline void cafe(void) +{ + asm volatile("r7 = 0xcafe" ::: "r7"); +} + +/* + * Trivial program: every insn maps to the original index + */ +SEC("fentry/" SYS_PREFIX "sys_nanosleep") +int check_trivial_prog(void *ctx) +{ + beef(); + cafe(); + beef(); + cafe(); + beef(); + + return 0; +} + +/* Some random instructions which will be patched for sure */ +static inline void beefify(void) +{ + __u32 key = 0; + __u64 *x; + + beef(); + bpf_printk("%llx", bpf_jiffies64()); + beef(); + + key = !!bpf_jiffies64(); + x = bpf_map_lookup_elem(&just_a_map, &key); + if (!x) + return; + + beef(); +} + +/* + * Simple program: one section, no bpf-to-bpf calls, some patches + */ +SEC("fentry/" SYS_PREFIX "sys_nanosleep") +int check_simple_prog(void *ctx) +{ + beefify(); + return 0; +} + +int __noinline foobar(int x) +{ + beefify(); + return x; +} + +/* + * Same simple program + a bpf-to-bpf call + */ +SEC("fentry/" SYS_PREFIX "sys_nanosleep") +int check_bpf_to_bpf(void *ctx) +{ + beefify(); + + return foobar(0); +} + +static inline void dead_code1(void) +{ + asm volatile("goto +0"); +} + +static inline void dead_code100(void) +{ +# if defined(__clang__) +# pragma clang loop unroll_count(100) +# elif defined(__GNUC__) +# pragma GCC unroll 100 +# else +# error "unroll this loop, please" +# endif + for (int i = 0; i < 100; i++) + asm volatile("goto +0"); +} + +/* + * Some beef instructions, patches, plus dead code + */ +static __always_inline void dead_beef(void) +{ + beef(); /* 1 beef */ + dead_code1(); + beef(); /* 1 beef */ + dead_code1(); + beef(); /* 1 beef */ + dead_code100(); + beef(); /* 1 beef */ + + dead_code100(); + beefify(); /* 3 beef */ + dead_code100(); + beefify(); /* 3 beef */ + dead_code1(); + beefify(); /* 3 beef */ + + /* 13 beef insns total */ +} + +/* + * A program with some nops to be removed + */ +SEC("fentry/" SYS_PREFIX "sys_nanosleep") +int check_prog_dead_code(void *ctx) +{ + dead_beef(); + + return 0; +} + +int __noinline foobar2(int x) +{ + dead_beef(); + + return x; +} + +/* + * A program with some nops to be removed + a bpf-to-bpf call to a similar func + */ +SEC("fentry/" SYS_PREFIX "sys_nanosleep") +int check_prog_dead_code_bpf_to_bpf(void *ctx) +{ + dead_beef(); + + return foobar2(0); +} + +char _license[] SEC("license") = "GPL"; From patchwork Fri Feb 2 16:28:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anton Protopopov X-Patchwork-Id: 13543146 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-ed1-f51.google.com (mail-ed1-f51.google.com [209.85.208.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 53C0D1487F4 for ; Fri, 2 Feb 2024 16:34:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706891664; cv=none; b=ax8475eyMhQiE8dHgrWzT4SkBlJLQ3hGQwsRXCNyymB0edLhW4AshbdlCILD7xNI8Sfb8TWfj5kOw8vjIEiVxZFfAxRHrXjFaiJQIwRqBQCxRWMkSe92P4JvZoVQTJs8nhQQh1ryp/xDtGJsQfiTbVC8EmjxP4JRagG+vbhJ3N0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706891664; c=relaxed/simple; bh=TL9L4X+rUjQ91gN/VNqRNMR+Gb80qW9iWai2lkK8nyg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=NHlB2bWagAROmnzPPcgajoiSS9eTd+jOTuiGHnp/kk1eDXwdpVZQfLwMGZ+uXx/I8/3nlo3pzk3KQfSROO/HkQQgUhuN43W4rhi5GillNj91M/GQnr5TtfZLMLOq4vttwGdKbzSEfNtmRQOQwZHUe/O6RHDG2z9lQgbAZprZwtA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=isovalent.com; spf=pass smtp.mailfrom=isovalent.com; dkim=pass (2048-bit key) header.d=isovalent.com header.i=@isovalent.com header.b=a1qrDorq; arc=none smtp.client-ip=209.85.208.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=isovalent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=isovalent.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=isovalent.com header.i=@isovalent.com header.b="a1qrDorq" Received: by mail-ed1-f51.google.com with SMTP id 4fb4d7f45d1cf-55790581457so3388456a12.3 for ; Fri, 02 Feb 2024 08:34:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=isovalent.com; s=google; t=1706891660; x=1707496460; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5eEh07CcQmyPa939B6UJrl0j+WeVe+LqiVEymgHgF5I=; b=a1qrDorq2KcR0amqbXFOAdYKk8AtVm+FNmW+kGR+LWKEVh4OGeGKhagQ3KC5qxp4bR n6wGOJ+bQDPLLTY2M0hSib7/IdkR6lKbYFHNTNbAXQVWaRDZoejjcamwdRDa2s7DQUPt pslGCixUcFpsm+smFx4HKacedU67/preP9NvxFlpb8j1MWPDQXYPoxQzO14UGWRKFmQ7 i9a7L9obznj9/XlbKzLRAnxMGghntTktrrjOnozcsEbfCTsdwr/aQ3F124JzOaf6gfnh U1YXQ4rCLCR0ZQAgxBBVU+SAFi+dUoGKiqnugUONj0UvnlKHBB5sO/zJgJl2FdiFjxGO 07Kg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706891660; x=1707496460; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5eEh07CcQmyPa939B6UJrl0j+WeVe+LqiVEymgHgF5I=; b=NvqodFyoEjDWkQc9ST1BPG4uqlx0/4y2NI2KdNoaX5T54xegTH3y4Fsx1PS4KV1AMH GhLjVMErzmTJdFnhHmRq/S4oa0cHfqhaQxyhiUxoHM9xXSGoBBi5BnHp6kT8J1FYjKyn 80/XP0iDgURG/ipGE0eQ3K78zZFNJmVWJacEtd+n8j45pVrZyV6rhqxtQCOGo4kBgbCZ 7fXK3XhhAG3ieGqUlOfrQhGG4OdLAoR4iF8p0JEwkzle1JAIYwnWiv+8AP9M1G37st92 uIOoe/k8F334NlS16QhnrL/pkPcqGb/j/RbUe8oeUs9h28DKf/A60REe8//cbwLyBfqD CdKw== X-Gm-Message-State: AOJu0YzxTweZUGRcWxl10uMkW0zX4EX5II8YDqXKFwZe12RpCDW/8ziP CHf/6FfTOT6ATerZp6QjqDI2XX4f3TwC6HRSrcdw79Lt/zaD3HXj06Rc9SLS09U7xHogsOl1euJ f X-Google-Smtp-Source: AGHT+IFdZIe48zHhRIjAriwrZeL4gOML7pW6+xIz9F1EcMFy33EQz/xzxMvrDmcaelxrvietrZRprQ== X-Received: by 2002:aa7:c0c7:0:b0:55f:8c5d:e55e with SMTP id j7-20020aa7c0c7000000b0055f8c5de55emr120857edp.26.1706891660541; Fri, 02 Feb 2024 08:34:20 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCXZ55Mebdh/6cz2O3bKDVLT929R+EdMAANKiTaQebr0axKBuhqaJcp5BHI9sk5hIMq7cDEgHNfUxv3fDw+BDnDoIIHfiTZCVxlSOzn0KN0M5v76eda13KE8mvtVLrNwpwM9xI48bVlh3tWp/vXX8YXTo8GM0hq1ReqYVsq0hT94pcUw3ekFZdxHlIsbeoxQPyxpGAK03ZhfGfOIqYc5vPnW8TMSbxbPROflpAKluPLIOVLV5cj8NDTW5oHTvMWdVHMTSUQ2CFpH+1X5jSVgz3YDnOXeMyhQClxlkWcvSvRJUgH18PLaxJA= Received: from zh-lab-node-5.home ([2a02:168:f656:0:1ac0:4dff:fe0f:3782]) by smtp.gmail.com with ESMTPSA id l19-20020aa7c313000000b0055edbe94b34sm952544edq.54.2024.02.02.08.34.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Feb 2024 08:34:19 -0800 (PST) From: Anton Protopopov To: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Jiri Olsa , Martin KaFai Lau , Stanislav Fomichev , Yonghong Song , Eduard Zingerman , Quentin Monnet , bpf@vger.kernel.org Cc: Anton Protopopov Subject: [PATCH v1 bpf-next 5/9] bpftool: dump new fields of bpf prog info Date: Fri, 2 Feb 2024 16:28:09 +0000 Message-Id: <20240202162813.4184616-6-aspsk@isovalent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240202162813.4184616-1-aspsk@isovalent.com> References: <20240202162813.4184616-1-aspsk@isovalent.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net When dumping prog info in JSON format add two new fields: "orig_insn" and "jit". The former field maps the xlated instruction to the original instruction (which was loaded via the bpf(2) syscall). The latter maps the xlated instruction to the jitted instruction; as jited instructions lengths may vary both the offset and length are specified. Signed-off-by: Anton Protopopov --- tools/bpf/bpftool/prog.c | 14 ++++++++++++++ tools/bpf/bpftool/xlated_dumper.c | 18 ++++++++++++++++++ tools/bpf/bpftool/xlated_dumper.h | 2 ++ 3 files changed, 34 insertions(+) diff --git a/tools/bpf/bpftool/prog.c b/tools/bpf/bpftool/prog.c index 9cb42a3366c0..d3fd7d699574 100644 --- a/tools/bpf/bpftool/prog.c +++ b/tools/bpf/bpftool/prog.c @@ -119,6 +119,12 @@ static int prep_prog_info(struct bpf_prog_info *const info, enum dump_mode mode, holder.jited_line_info_rec_size = info->jited_line_info_rec_size; needed += info->nr_jited_line_info * info->jited_line_info_rec_size; + holder.orig_idx_len = info->orig_idx_len; + needed += info->orig_idx_len; + + holder.xlated_to_jit_len = info->xlated_to_jit_len; + needed += info->xlated_to_jit_len; + if (needed > *info_data_sz) { ptr = realloc(*info_data, needed); if (!ptr) @@ -152,6 +158,12 @@ static int prep_prog_info(struct bpf_prog_info *const info, enum dump_mode mode, holder.jited_line_info = ptr_to_u64(ptr); ptr += holder.nr_jited_line_info * holder.jited_line_info_rec_size; + holder.orig_idx = ptr_to_u64(ptr); + ptr += holder.orig_idx_len; + + holder.xlated_to_jit = ptr_to_u64(ptr); + ptr += holder.xlated_to_jit_len; + *info = holder; return 0; } @@ -852,6 +864,8 @@ prog_dump(struct bpf_prog_info *info, enum dump_mode mode, dd.func_info = func_info; dd.finfo_rec_size = info->func_info_rec_size; dd.prog_linfo = prog_linfo; + dd.orig_idx = u64_to_ptr(info->orig_idx); + dd.xlated_to_jit = u64_to_ptr(info->xlated_to_jit); if (json_output) dump_xlated_json(&dd, buf, member_len, opcodes, linum); diff --git a/tools/bpf/bpftool/xlated_dumper.c b/tools/bpf/bpftool/xlated_dumper.c index 567f56dfd9f1..fcc1c4a96178 100644 --- a/tools/bpf/bpftool/xlated_dumper.c +++ b/tools/bpf/bpftool/xlated_dumper.c @@ -270,6 +270,24 @@ void dump_xlated_json(struct dump_data *dd, void *buf, unsigned int len, jsonw_name(json_wtr, "disasm"); print_bpf_insn(&cbs, insn + i, true); + if (dd->orig_idx) { + jsonw_name(json_wtr, "orig_insn"); + jsonw_printf(json_wtr, "\"0x%x\"", dd->orig_idx[i]); + } + + if (dd->xlated_to_jit) { + jsonw_name(json_wtr, "jit"); + jsonw_start_object(json_wtr); + + jsonw_name(json_wtr, "off"); + jsonw_printf(json_wtr, "\"0x%x\"", dd->xlated_to_jit[i].off); + + jsonw_name(json_wtr, "len"); + jsonw_printf(json_wtr, "\"%d\"", dd->xlated_to_jit[i].len); + + jsonw_end_object(json_wtr); + } + if (opcodes) { jsonw_name(json_wtr, "opcodes"); jsonw_start_object(json_wtr); diff --git a/tools/bpf/bpftool/xlated_dumper.h b/tools/bpf/bpftool/xlated_dumper.h index db3ba0671501..078430fada17 100644 --- a/tools/bpf/bpftool/xlated_dumper.h +++ b/tools/bpf/bpftool/xlated_dumper.h @@ -26,6 +26,8 @@ struct dump_data { __u32 finfo_rec_size; const struct bpf_prog_linfo *prog_linfo; char scratch_buff[SYM_MAX_NAME + 8]; + unsigned int *orig_idx; + struct bpf_xlated_to_jit *xlated_to_jit; }; void kernel_syms_load(struct dump_data *dd); From patchwork Fri Feb 2 16:28:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anton Protopopov X-Patchwork-Id: 13543147 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-ed1-f42.google.com (mail-ed1-f42.google.com [209.85.208.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5B0722E3E1 for ; Fri, 2 Feb 2024 16:34:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706891665; cv=none; b=PYnJ+uYBYWha7yxVy0Igwc2/EdZsa0msUyIjZngD6eLWXjQ8TSia7FeRWZ9xQSevvS6Pz7I4fUlynsDRh3+1m2nhsLYf9G0U7jvy8bCx2KiaPjWCpHPMSqYUhmFws5otAFHe8rXjAijJMHskp68JHx+8hbYM+EF9waqtRAKUNI8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706891665; c=relaxed/simple; bh=mzSJZq3Jc5DiGLGTDSv6JFgvcsw52wo529lFsU7tDeo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=g2Yn+FvtzH+LfIdi1vhpAUZrqvfnmKFGf6+wpjDbOEWv1489ueSa2vfGMvJ6gn6NwpM5adyZtk1kVUgoCue+js0haS/zmFecAJoR+MJIFyblHIeR43m5JL7GmKQz2Of6p4MiePbqCzhhgYzo8TrNGs9dwZv7U+ohF3KHVZGqsXI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=isovalent.com; spf=pass smtp.mailfrom=isovalent.com; dkim=pass (2048-bit key) header.d=isovalent.com header.i=@isovalent.com header.b=Hbok9IzP; arc=none smtp.client-ip=209.85.208.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=isovalent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=isovalent.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=isovalent.com header.i=@isovalent.com header.b="Hbok9IzP" Received: by mail-ed1-f42.google.com with SMTP id 4fb4d7f45d1cf-55a179f5fa1so3126632a12.0 for ; Fri, 02 Feb 2024 08:34:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=isovalent.com; s=google; t=1706891661; x=1707496461; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lWrCvGoDQwOWMpVcb46eMyJCzvJuUZPcrKCneQA7xTs=; b=Hbok9IzPRVN1MaYQIFlA4O9xwM3VljW5Sa4i65wGlAgvzdg7LmepTQenk0/UAfNIUI 3YuKgPLBsr35BZUE9+ddhIqIjNrGXjoveYCNzgab0KXofBw5S7dLkhXwoy7MpLO3/v0e k5LluqMto43LlrR9EMhcALgQIEnu1AD6WpFuQjoY6cJSS8Nbslzm4NIkM9jD9e7C/JwV XVLmVIj2YcAz0fsGcTObwLTcRdNUql1cAp79d+zntj3s6IxHaDgkgMXYqCKu3PU4QMUy MjUD2XOTIdydqwoAUTtD0wlCpggaxnZlStTZ7bnc9iemZUkRM0eQ01EtPIs/6P0THVQH LTAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706891661; x=1707496461; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lWrCvGoDQwOWMpVcb46eMyJCzvJuUZPcrKCneQA7xTs=; b=LYapqbkC9VKJuNKS/a/ASZCGkk3cAil7TGAKtgpM/DAVDdMAa204Nj3sRHn3fbW6mK 49fFwPZ5sIZF8Uxwn3O3s14LVv160a49iGs0aW220/IzQIUdOZbYveX9B0LfF5ekqmdO zW0PDwklP33SpprU0vWWJkcTtZADwPYCJUr/M0Om0BgeSlDFAbpNLp2ttMMyW7dg4oFf X0CF8DQHIZIqzCfIJr61hora7YhWoVauxF/jSoOt6rUzm/rDnbyizJBt8hiXkpmAoaCB pjVk1MHoqqXBYeEFp1m/io7qs6GS+knSt21ychJjp3nLI8cF/73ocpGnCxGeWwACF6R2 0xsg== X-Gm-Message-State: AOJu0Yzuw/i13IPU/q+1vRz0lt1aYhW/uppOgRv/oOb721m21HmnUXQ+ NPjKA2gG6rJnD0seusM0hTm3T8GEXdDnhI9lhwe+ORyZGSiakxMx4NMZwF87/jg= X-Google-Smtp-Source: AGHT+IEafoDqngvIZ3JoIAvTtWUcJ5pbxsnL17NJuvWQPo6jRYYh5iq9u/2vNaPdBG2eaYeDHqDnzg== X-Received: by 2002:aa7:d710:0:b0:55f:1728:3b33 with SMTP id t16-20020aa7d710000000b0055f17283b33mr95019edq.40.1706891661474; Fri, 02 Feb 2024 08:34:21 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCWhTtlC7UMW0/mPCkyhSmvN0k9mS60j+hmiKGf4tS+l6EBb+WKP9uXUDlLnC1BtYYkMEslqQTakQHwY0LmAgb51X+sHOyzX/kS7nmQTDQdW+Wp5nv/JqNXu57IoqC7Ao7nu3LDlAVCjSFAjXfJ/MDTRUyX+b4K+MXP9M80N832ZQyZ8+LxuiBaUl2IJhxnxlHXOhxFPJlOhUzrixlh9M0p0FRs24PV8OCsch7E7Ll69udNEgGJpFX/QJd+dlEVknDQI86cwzMOuEulHVJOYsvsxgqGYzQmNfSqGZL0o0xbYrXx8qdcbaGI= Received: from zh-lab-node-5.home ([2a02:168:f656:0:1ac0:4dff:fe0f:3782]) by smtp.gmail.com with ESMTPSA id l19-20020aa7c313000000b0055edbe94b34sm952544edq.54.2024.02.02.08.34.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Feb 2024 08:34:21 -0800 (PST) From: Anton Protopopov To: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Jiri Olsa , Martin KaFai Lau , Stanislav Fomichev , Yonghong Song , Eduard Zingerman , Quentin Monnet , bpf@vger.kernel.org Cc: Anton Protopopov Subject: [PATCH v1 bpf-next 6/9] bpf: add support for an extended JA instruction Date: Fri, 2 Feb 2024 16:28:10 +0000 Message-Id: <20240202162813.4184616-7-aspsk@isovalent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240202162813.4184616-1-aspsk@isovalent.com> References: <20240202162813.4184616-1-aspsk@isovalent.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Add support for a new version of JA instruction, a static branch JA. Such instructions may either jump to the specified offset or act as nops. To distinguish such instructions from normal JA the BPF_STATIC_BRANCH_JA flag should be set for the SRC register. By default on program load such instructions are jitted as a normal JA. However, if the BPF_STATIC_BRANCH_NOP flag is set in the SRC register, then the instruction is jitted to a NOP. In order to generate BPF_STATIC_BRANCH_JA instructions using llvm two new instructions were added: asm volatile goto ("nop_or_gotol %l[label]" :::: label); will generate the BPF_STATIC_BRANCH_JA|BPF_STATIC_BRANCH_NOP instuction and asm volatile goto ("gotol_or_nop %l[label]" :::: label); will generate a BPF_STATIC_BRANCH_JA instruction, without an extra bit set. The reason for adding two instructions is that both are required to implement static keys functionality for BPF. The verifier logic is extended to check both possible paths: jump and nop. Signed-off-by: Anton Protopopov --- arch/x86/net/bpf_jit_comp.c | 19 +++++++++++++-- include/uapi/linux/bpf.h | 10 ++++++++ kernel/bpf/verifier.c | 43 +++++++++++++++++++++++++++------- tools/include/uapi/linux/bpf.h | 10 ++++++++ 4 files changed, 71 insertions(+), 11 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index a80b8c1e7afe..b291b5c79d26 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -1131,6 +1131,15 @@ static void emit_shiftx(u8 **pprog, u32 dst_reg, u8 src_reg, bool is64, u8 op) *pprog = prog; } +static bool is_static_ja_nop(const struct bpf_insn *insn) +{ + u8 code = insn->code; + + return (code == (BPF_JMP | BPF_JA) || code == (BPF_JMP32 | BPF_JA)) && + (insn->src_reg & BPF_STATIC_BRANCH_JA) && + (insn->src_reg & BPF_STATIC_BRANCH_NOP); +} + #define INSN_SZ_DIFF (((addrs[i] - addrs[i - 1]) - (prog - temp))) /* mov rax, qword ptr [rbp - rounded_stack_depth - 8] */ @@ -2016,9 +2025,15 @@ st: if (is_imm8(insn->off)) } emit_nops(&prog, INSN_SZ_DIFF - 2); } - EMIT2(0xEB, jmp_offset); + if (is_static_ja_nop(insn)) + emit_nops(&prog, 2); + else + EMIT2(0xEB, jmp_offset); } else if (is_simm32(jmp_offset)) { - EMIT1_off32(0xE9, jmp_offset); + if (is_static_ja_nop(insn)) + emit_nops(&prog, 5); + else + EMIT1_off32(0xE9, jmp_offset); } else { pr_err("jmp gen bug %llx\n", jmp_offset); return -EFAULT; diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index c874f354c290..aca5ed065731 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -1412,6 +1412,16 @@ struct bpf_stack_build_id { }; }; +/* Flags for JA insn, passed in SRC_REG */ +enum { + BPF_STATIC_BRANCH_JA = 1 << 0, + BPF_STATIC_BRANCH_NOP = 1 << 1, +}; + +#define BPF_STATIC_BRANCH_MASK (BPF_STATIC_BRANCH_JA | \ + BPF_STATIC_BRANCH_NOP) + + #define BPF_OBJ_NAME_LEN 16U union bpf_attr { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 270dc0a26d03..003b54fbc6d9 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -15630,14 +15630,24 @@ static int visit_insn(int t, struct bpf_verifier_env *env) else off = insn->imm; - /* unconditional jump with single edge */ - ret = push_insn(t, t + off + 1, FALLTHROUGH, env); - if (ret) - return ret; + if (insn->src_reg & BPF_STATIC_BRANCH_JA) { + /* static branch - jump with two edges */ + mark_prune_point(env, t); + + ret = push_insn(t, t + 1, FALLTHROUGH, env); + if (ret) + return ret; - mark_prune_point(env, t + off + 1); - mark_jmp_point(env, t + off + 1); + ret = push_insn(t, t + off + 1, BRANCH, env); + } else { + /* unconditional jump with single edge */ + ret = push_insn(t, t + off + 1, FALLTHROUGH, env); + if (ret) + return ret; + mark_prune_point(env, t + off + 1); + mark_jmp_point(env, t + off + 1); + } return ret; default: @@ -17607,8 +17617,11 @@ static int do_check(struct bpf_verifier_env *env) mark_reg_scratched(env, BPF_REG_0); } else if (opcode == BPF_JA) { + struct bpf_verifier_state *other_branch; + u32 jmp_offset; + if (BPF_SRC(insn->code) != BPF_K || - insn->src_reg != BPF_REG_0 || + (insn->src_reg & ~BPF_STATIC_BRANCH_MASK) || insn->dst_reg != BPF_REG_0 || (class == BPF_JMP && insn->imm != 0) || (class == BPF_JMP32 && insn->off != 0)) { @@ -17617,9 +17630,21 @@ static int do_check(struct bpf_verifier_env *env) } if (class == BPF_JMP) - env->insn_idx += insn->off + 1; + jmp_offset = insn->off + 1; else - env->insn_idx += insn->imm + 1; + jmp_offset = insn->imm + 1; + + /* Staic branch can either jump to +off or +0 */ + if (insn->src_reg & BPF_STATIC_BRANCH_JA) { + other_branch = push_stack(env, env->insn_idx + jmp_offset, + env->insn_idx, false); + if (!other_branch) + return -EFAULT; + + jmp_offset = 1; + } + + env->insn_idx += jmp_offset; continue; } else if (opcode == BPF_EXIT) { diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index c874f354c290..aca5ed065731 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -1412,6 +1412,16 @@ struct bpf_stack_build_id { }; }; +/* Flags for JA insn, passed in SRC_REG */ +enum { + BPF_STATIC_BRANCH_JA = 1 << 0, + BPF_STATIC_BRANCH_NOP = 1 << 1, +}; + +#define BPF_STATIC_BRANCH_MASK (BPF_STATIC_BRANCH_JA | \ + BPF_STATIC_BRANCH_NOP) + + #define BPF_OBJ_NAME_LEN 16U union bpf_attr { From patchwork Fri Feb 2 16:28:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anton Protopopov X-Patchwork-Id: 13543148 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-ed1-f42.google.com (mail-ed1-f42.google.com [209.85.208.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 27C731487E8 for ; Fri, 2 Feb 2024 16:34:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706891665; cv=none; b=amozlm73sYvzn5qqSsd0t4ERz0lgvPAhHm2REjYZlhwdT04S2WSb73NIkXDBLOgCPdMyPC6SypvEvdM+IrrTCrsiO7+pMuFmUpSi9ATDr7LGgwWql/wnYRDco2yRZDQR4dIJh9S+hWEmKZtsC9KKgNb6RAOKXcTlGtVXcvt5+GY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706891665; c=relaxed/simple; bh=NE1NLLzZr8CUrmLnmmaPmBcY9uHP2NUWNMFPX2IDo+Q=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=T+CKVNBnn+tU53Z6d3OdMmrCwscnxkXeaNlu1y+bwp9U3qmp1/3Tl31pTY9Dw2XH3n0ow4q1QDPKAKMRpeIHnkOv5eyJuGy8l+DwBicWOJQmT8Aun2ovUn7uTJ5TG78MEKhQ6ywWxTdrcjYA1814QQsLs1l5xMKsVvaCsrVBS0I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=isovalent.com; spf=pass smtp.mailfrom=isovalent.com; dkim=pass (2048-bit key) header.d=isovalent.com header.i=@isovalent.com header.b=QxkC6JmM; arc=none smtp.client-ip=209.85.208.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=isovalent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=isovalent.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=isovalent.com header.i=@isovalent.com header.b="QxkC6JmM" Received: by mail-ed1-f42.google.com with SMTP id 4fb4d7f45d1cf-55a9008c185so1831286a12.1 for ; Fri, 02 Feb 2024 08:34:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=isovalent.com; s=google; t=1706891662; x=1707496462; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6DsDRTsOyaVpHWTH/Ld3zgp4KeMMBnazkevDmIj8tFU=; b=QxkC6JmMKJWTymVCHkwmoBzkOjwD9Pps2ZtrSXJs5hbCsoYXnq2e+//1/a20xmOTQE ubzV+KQLYboqUNYu1CxuK7dJzW7iw9BbCMiUbEzPGQl3Ap2AVAUBBHqBTgKwLI970D+0 5auy1nADEUFZtPurCgS7q0ezyyvON26tTSxBsD7HeFPmeI75v1UkvzwIpxVoU28mgfP6 KNBKbzYZJQvl06h/+Swdd/bAldIV/rimWz1yTvj8B+3Ao+fTTKxx3YquJkgkS6PciXI0 Dx7zbsHCYLD6TUsgX1/VMDcdJfclIgnkzF43l2jEnFOViPV2fz0Of58mzuC4k3A22Glf 2mCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706891662; x=1707496462; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6DsDRTsOyaVpHWTH/Ld3zgp4KeMMBnazkevDmIj8tFU=; b=egxF1x2xhLB0AWd9INiEMrR+B/nrVVKfzUv8rgQw0lW1cOPFg0DYR389uf1B32cFNs pvQlvvW5Bfx7ImGRlaoDP//u81cyY6EdHXLheDwRs7Tt73D8mCqOV57kDgqMvg50qP+W WU9GFPn5IclFajr+HRml4lqRxylBMbl6AQIJQFq0tNmc2gutrOKFtxPIODwY417mqqN4 XH/1IfOzH7OPm21Ndyp4sp5mI/cjdw2LaEuwaZwCPiWoI20bVLxtyU5FJlrendApmnwk kgj4q19Lwx6nvEAJNkCAwiDIE41TkPEGOOT2U+W3Kp14R58SKt4NaE84+8Sqzo/m8FpK GvwA== X-Gm-Message-State: AOJu0YwRTdD8dwtCqn8hjvzXXGXNo5urxMIYEw57Y2uSsTeEkLZAfLLW PE5UzLo5H7BmibM4kjsS8CNP1tgs+eJ3oZvPjL2m5aiuuSUKcITk2oKKElfgn/s= X-Google-Smtp-Source: AGHT+IHZa716znm/na8SX0FjzVuyAy0KNC7WnyegWEwJbahSN3uyFiKAYMBB4pNrYx/w9c2+Q7yQ0g== X-Received: by 2002:a50:d516:0:b0:55f:ce14:337a with SMTP id u22-20020a50d516000000b0055fce14337amr185732edi.11.1706891662391; Fri, 02 Feb 2024 08:34:22 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCUY9zXBO/iSyh02hb1iVUYsygi4Jt1HYcIuIkXhPPdFBUiKGNlpWCy8hVg74hGC0cbiHFbI+rcMw6Tr3C8Mrg4sC7UfuogZYU1HSS6OatYiyDKecWnZf3yZ6qDRyfTHpJBu6CdeCgqcc97To9KU/Tg6SSsGsgtxj2KwuDtTBNPOGw2X2hk4gQT83kRzfbHmqR5BaZ/bQRHWRTKS+MTNfXeG//xC9D1ovtzuBEP7/FqbI87hC6U/rAouT2PWd35fg2+k1EQLdFc8xydJzg8qi1dGh5rmPahTjCIIuwBm1cRzrpOK55zSQ6s= Received: from zh-lab-node-5.home ([2a02:168:f656:0:1ac0:4dff:fe0f:3782]) by smtp.gmail.com with ESMTPSA id l19-20020aa7c313000000b0055edbe94b34sm952544edq.54.2024.02.02.08.34.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Feb 2024 08:34:21 -0800 (PST) From: Anton Protopopov To: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Jiri Olsa , Martin KaFai Lau , Stanislav Fomichev , Yonghong Song , Eduard Zingerman , Quentin Monnet , bpf@vger.kernel.org Cc: Anton Protopopov Subject: [PATCH v1 bpf-next 7/9] bpf: Add kernel/bpftool asm support for new instructions Date: Fri, 2 Feb 2024 16:28:11 +0000 Message-Id: <20240202162813.4184616-8-aspsk@isovalent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240202162813.4184616-1-aspsk@isovalent.com> References: <20240202162813.4184616-1-aspsk@isovalent.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Add asm support for new JA* instructions so kernel verifier and bpftool xlated insn dumps can have proper asm syntax. Signed-off-by: Anton Protopopov --- kernel/bpf/disasm.c | 33 +++++++++++++++++++++++++++------ 1 file changed, 27 insertions(+), 6 deletions(-) diff --git a/kernel/bpf/disasm.c b/kernel/bpf/disasm.c index 49940c26a227..5c8ee230ee5a 100644 --- a/kernel/bpf/disasm.c +++ b/kernel/bpf/disasm.c @@ -166,6 +166,30 @@ static bool is_movsx(const struct bpf_insn *insn) (insn->off == 8 || insn->off == 16 || insn->off == 32); } +static void print_bpf_ja_insn(bpf_insn_print_t verbose, + void *private_data, + const struct bpf_insn *insn) +{ + bool jmp32 = insn->code == (BPF_JMP32 | BPF_JA); + int off = jmp32 ? insn->imm : insn->off; + const char *suffix = jmp32 ? "l" : ""; + char op[16]; + + switch (insn->src_reg & BPF_STATIC_BRANCH_MASK) { + case BPF_STATIC_BRANCH_JA: + snprintf(op, sizeof(op), "goto%s_or_nop", suffix); + break; + case BPF_STATIC_BRANCH_JA | BPF_STATIC_BRANCH_NOP: + snprintf(op, sizeof(op), "nop_or_goto%s", suffix); + break; + default: + snprintf(op, sizeof(op), "goto%s", suffix); + break; + } + + verbose(private_data, "(%02x) %s pc%+d\n", insn->code, op, off); +} + void print_bpf_insn(const struct bpf_insn_cbs *cbs, const struct bpf_insn *insn, bool allow_ptr_leaks) @@ -319,12 +343,9 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs, tmp, sizeof(tmp)), insn->imm); } - } else if (insn->code == (BPF_JMP | BPF_JA)) { - verbose(cbs->private_data, "(%02x) goto pc%+d\n", - insn->code, insn->off); - } else if (insn->code == (BPF_JMP32 | BPF_JA)) { - verbose(cbs->private_data, "(%02x) gotol pc%+d\n", - insn->code, insn->imm); + } else if (insn->code == (BPF_JMP | BPF_JA) || + insn->code == (BPF_JMP32 | BPF_JA)) { + print_bpf_ja_insn(verbose, cbs->private_data, insn); } else if (insn->code == (BPF_JMP | BPF_EXIT)) { verbose(cbs->private_data, "(%02x) exit\n", insn->code); } else if (BPF_SRC(insn->code) == BPF_X) { From patchwork Fri Feb 2 16:28:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anton Protopopov X-Patchwork-Id: 13543149 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-ed1-f48.google.com (mail-ed1-f48.google.com [209.85.208.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 04C441487F4 for ; Fri, 2 Feb 2024 16:34:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706891666; cv=none; b=uo3/UGMCXEFLC9ybwkFyO4hhQD3DgEF+1kDCbHevQc2jyMIOm0dX5NURviHeJgrX9/Pj3zZUFkICBdNj5vt9IL3uH8BNhFbH47Ro22lEYW4XXqtNFvDnwPGWxuy9lc/mje6lbi5yT+upFsjGHf2DhLPcNueB/FoaYezhi+jwzkc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706891666; c=relaxed/simple; bh=bNowxqK6s8C3syKDPvlOUzdu7D3tdyI0c6VNHRN4uwU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=W5kxuxNii/x4IfWevt7rRc4jZ+o5cK9miS8cFROu0b+G+eLTY+BXClAzo1O2SZGxJh6AdHzEeQihXNVDSlZ0cTpCQ2HepvqDgfX79k3qWPXnlo3Ta4HsIk+2VyyB9EmJ9N1MvNL1AYCrNY/RCib3Ku0MNTnP0re/MGwNdZN8E/c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=isovalent.com; spf=pass smtp.mailfrom=isovalent.com; dkim=pass (2048-bit key) header.d=isovalent.com header.i=@isovalent.com header.b=LWRjhOey; arc=none smtp.client-ip=209.85.208.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=isovalent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=isovalent.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=isovalent.com header.i=@isovalent.com header.b="LWRjhOey" Received: by mail-ed1-f48.google.com with SMTP id 4fb4d7f45d1cf-55a9008c185so1831312a12.1 for ; Fri, 02 Feb 2024 08:34:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=isovalent.com; s=google; t=1706891663; x=1707496463; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mQnEiZYPethPwLa16b0xJfRGCxsPiWgVF9283FCozko=; b=LWRjhOeyEQdSf8t/h3OGcM3yxWeBFwVL61j9oDF2Eo4dCrg9MugfIs5AAX9SZdqVgZ EqFcwWbPUW6deFvxER2ixYcbgNFgNreYwfScqgXcsJthJhZ0t2gkXYxCltPtGhlfgRNT /oTO0YHyRBpKKRh414U2jtdAPMq5DyJ4OF3GB1MKFnKPdz450fYOVnm+3sOYhPuYlK9w dISIjFzMe/FDiUiOTHWEJQ1oVVWZZELpfPPa4SorxTl+QR3qrU89xCrVttGlxaVM8ma+ x2wD3xeZ4UCTH28DymtbyN5jCWUkmqxg6y5KotoylWFoHn2fSbwIloSD9zHZmDN34HMB vBZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706891663; x=1707496463; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mQnEiZYPethPwLa16b0xJfRGCxsPiWgVF9283FCozko=; b=XT0tv+JomHbCj8aO0PfNvACw6qr5BV+UmQJUNWAKbDnuv1USzfZbR4+mOjqvT1YXv0 OMsloXHQwgpwitKxl+krgAbJbq9NzsRlZ7R5GNGDyWnZYANhAP/4SAKmdzknJQu1HKHa eduPzJCtpfXzEkjPDfmEQfB+UjeTadsg/Oht54cf2y+yhK0bR0EToUFXKyEPmXOuQkCS ckKSGhMT1Qy6A978ScptBGjzrl/3l7rd+WcoZoWMcOrxtMOre1yDfAiLUgv+UoB83sR/ 2/Ae7Wy9pd0NLx4GU0KFxB1dphiSqC0BI8r7ALczCk3+FYFfxmFV3mfeLIvw2aT9RI3n ARgw== X-Gm-Message-State: AOJu0YwRxo/rBf5/jiSiMMCmneK32p8Mgt0irb0P1gRrOsTKmpxvEvnG XwuV8I/rabWmLnJJZ0Gj2LkfnQz1mC6Oa+rq1KeOxV9C2LExuVjK/muQhL0xchM= X-Google-Smtp-Source: AGHT+IEWc8r51wN7yXndGPjJxj2ElaKXc38dnVl8NQKn+hA36hSb+ukixAFEGb048dwsw1tZlUTI2g== X-Received: by 2002:a05:6402:1659:b0:55f:e543:b006 with SMTP id s25-20020a056402165900b0055fe543b006mr92975edx.13.1706891663273; Fri, 02 Feb 2024 08:34:23 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCVOFbEzg+JPWuzlKADSq9mckHaiFJ1NRxvt2YiIbYuOFK5MNskQWzswDO4xuFnEln4PVCYBc14GyyS8qglyOMCaXPhmJKuqRe56fn22q8rg6KFzKG92M9uJBQoXX6tzly36Iy+Ab1mOLHstMzhIB+l6n9DaLU4fRtS4Uw3Bh0IUsHcjAXTRkLi8gCA9AGULZikwk3Yp8lwguqxDlbMPwWPSjfEVf0B6PiUJseM8niedsLFfutlVmoVzT7sp6hx1sVylp4c7WZKeosWadIi334E6mAvLMGXApaoHNtvAFX2PIux57Zd54n4= Received: from zh-lab-node-5.home ([2a02:168:f656:0:1ac0:4dff:fe0f:3782]) by smtp.gmail.com with ESMTPSA id l19-20020aa7c313000000b0055edbe94b34sm952544edq.54.2024.02.02.08.34.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Feb 2024 08:34:22 -0800 (PST) From: Anton Protopopov To: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Jiri Olsa , Martin KaFai Lau , Stanislav Fomichev , Yonghong Song , Eduard Zingerman , Quentin Monnet , bpf@vger.kernel.org Cc: Anton Protopopov Subject: [PATCH v1 bpf-next 8/9] bpf: add BPF_STATIC_BRANCH_UPDATE syscall Date: Fri, 2 Feb 2024 16:28:12 +0000 Message-Id: <20240202162813.4184616-9-aspsk@isovalent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240202162813.4184616-1-aspsk@isovalent.com> References: <20240202162813.4184616-1-aspsk@isovalent.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Add a new bpf system call, BPF_STATIC_BRANCH_UPDATE, which allows users to update static branches in BPF. Namely, this system call is executed as bpf(BPF_STATIC_BRANCH_UPDATE, attrs={prog_fd, insn_off, on}) where prog_fd points to a BPF program, insn_off is an _xlated_ offset in this program, on is a boolean value to set this branch on or off. The instruction at insn_off must be a JA with SRC_REG or'ed with BPF_STATIC_BRANCH_JA and, optionally, with BPF_STATIC_BRANCH_INVERSE. To implement this for a particular architecture, re-define the weak bpf_arch_poke_static_branch() function in the corresponding bpf_jit_comp.c This patch adds x86 implementation. Signed-off-by: Anton Protopopov --- arch/x86/net/bpf_jit_comp.c | 40 +++++++++++++++++++++++ include/linux/bpf.h | 2 ++ include/linux/filter.h | 1 + include/uapi/linux/bpf.h | 7 ++++ kernel/bpf/core.c | 5 +++ kernel/bpf/syscall.c | 60 ++++++++++++++++++++++++++++++++++ tools/include/uapi/linux/bpf.h | 7 ++++ 7 files changed, 122 insertions(+) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index b291b5c79d26..2090713e4126 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -2099,8 +2099,17 @@ st: if (is_imm8(insn->off)) if (func_idx) off += bpf_prog->aux->func_info[func_idx].insn_off; + bpf_prog->aux->xlated_to_jit[off].ip = image + proglen; bpf_prog->aux->xlated_to_jit[off].off = proglen; bpf_prog->aux->xlated_to_jit[off].len = ilen; + + /* + * Save the offset so that it can later be accessed + * by the bpf(BPF_STATIC_BRANCH_UPDATE) syscall + */ + if (insn->code == (BPF_JMP | BPF_JA) || + insn->code == (BPF_JMP32 | BPF_JA)) + bpf_prog->aux->xlated_to_jit[off].jmp_offset = jmp_offset; } } proglen += ilen; @@ -3276,3 +3285,34 @@ bool bpf_jit_supports_ptr_xchg(void) { return true; } + +int bpf_arch_poke_static_branch(struct bpf_prog *prog, + u32 insn_off, + bool on) +{ + int jmp_offset = prog->aux->xlated_to_jit[insn_off].jmp_offset; + u32 len = prog->aux->xlated_to_jit[insn_off].len; + u8 op[5]; + + if (WARN_ON_ONCE(is_imm8(jmp_offset) && len != 2)) + return -EINVAL; + + if (WARN_ON_ONCE(!is_imm8(jmp_offset) && len != 5)) + return -EINVAL; + + if (on) { + if (len == 2) { + op[0] = 0xEB; + op[1] = jmp_offset; + } else { + op[0] = 0xE9; + memcpy(&op[1], &jmp_offset, 4); + } + } else { + memcpy(op, x86_nops[len], len); + } + + text_poke_bp(prog->aux->xlated_to_jit[insn_off].ip, op, len, NULL); + + return 0; +} diff --git a/include/linux/bpf.h b/include/linux/bpf.h index bdd6be718e82..1363b1fc8c09 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1528,8 +1528,10 @@ struct bpf_prog_aux { * instructions, if allocated */ struct { + void *ip; /* the address of the jitted insn */ u32 off; /* local offset in the jitted code */ u32 len; /* the total len of generated jit code */ + u32 jmp_offset; /* jitted jump offset for BPF_JA insns */ } *xlated_to_jit; }; diff --git a/include/linux/filter.h b/include/linux/filter.h index fee070b9826e..0dad44fa3af2 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -957,6 +957,7 @@ bool bpf_jit_supports_far_kfunc_call(void); bool bpf_jit_supports_exceptions(void); bool bpf_jit_supports_ptr_xchg(void); void arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie); +int bpf_arch_poke_static_branch(struct bpf_prog *prog, u32 off, bool on); bool bpf_helper_changes_pkt_data(void *func); static inline bool bpf_dump_raw_ok(const struct cred *cred) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index aca5ed065731..8aafb0eddd1c 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -932,6 +932,7 @@ enum bpf_cmd { BPF_LINK_DETACH, BPF_PROG_BIND_MAP, BPF_TOKEN_CREATE, + BPF_STATIC_BRANCH_UPDATE, __MAX_BPF_CMD, }; @@ -1787,6 +1788,12 @@ union bpf_attr { __u32 bpffs_fd; } token_create; + struct { /* struct used by BPF_STATIC_BRANCH_UPDATE command */ + __u32 prog_fd; + __u32 insn_off; + __u32 on; + } static_branch; + } __attribute__((aligned(8))); /* The description below is an attempt at providing documentation to eBPF diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 8e99c1563a7f..fec185354ea3 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -3042,6 +3042,11 @@ static int __init bpf_global_ma_init(void) late_initcall(bpf_global_ma_init); #endif +int __weak bpf_arch_poke_static_branch(struct bpf_prog *prog, u32 off, bool on) +{ + return -EOPNOTSUPP; +} + DEFINE_STATIC_KEY_FALSE(bpf_stats_enabled_key); EXPORT_SYMBOL(bpf_stats_enabled_key); diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 36b8fdcfba75..9e2e12a0bdfe 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -1540,6 +1540,63 @@ static int map_lookup_elem(union bpf_attr *attr) return err; } +static int parse_static_branch_insn(struct bpf_insn *insn, bool *inverse) +{ + __u8 code = insn->code; + + if (code != (BPF_JMP | BPF_JA) && code != (BPF_JMP32 | BPF_JA)) + return -EINVAL; + + if (insn->src_reg & ~BPF_STATIC_BRANCH_MASK) + return -EINVAL; + + if (!(insn->src_reg & BPF_STATIC_BRANCH_JA)) + return -EINVAL; + + if (insn->dst_reg) + return -EINVAL; + + *inverse = !(insn->src_reg & BPF_STATIC_BRANCH_NOP); + + return 0; +} + +#define BPF_STATIC_BRANCH_UPDATE_LAST_FIELD static_branch.on + +static int bpf_static_branch_update(union bpf_attr *attr) +{ + bool on = attr->static_branch.on & 1; + struct bpf_prog *prog; + u32 insn_off; + bool inverse; + int ret; + + if (CHECK_ATTR(BPF_STATIC_BRANCH_UPDATE)) + return -EINVAL; + + if (attr->static_branch.on & ~1) + return -EINVAL; + + prog = bpf_prog_get(attr->static_branch.prog_fd); + if (IS_ERR(prog)) + return PTR_ERR(prog); + + insn_off = attr->static_branch.insn_off; + if (insn_off >= prog->len) { + ret = -ERANGE; + goto put_prog; + } + + ret = parse_static_branch_insn(&prog->insnsi[insn_off], &inverse); + if (ret) + goto put_prog; + + ret = bpf_arch_poke_static_branch(prog, insn_off, on ^ inverse); + +put_prog: + bpf_prog_put(prog); + return ret; +} #define BPF_MAP_UPDATE_ELEM_LAST_FIELD flags @@ -5694,6 +5751,9 @@ static int __sys_bpf(int cmd, bpfptr_t uattr, unsigned int size) case BPF_MAP_DELETE_BATCH: err = bpf_map_do_batch(&attr, uattr.user, BPF_MAP_DELETE_BATCH); break; + case BPF_STATIC_BRANCH_UPDATE: + err = bpf_static_branch_update(&attr); + break; case BPF_LINK_CREATE: err = link_create(&attr, uattr); break; diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index aca5ed065731..8aafb0eddd1c 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -932,6 +932,7 @@ enum bpf_cmd { BPF_LINK_DETACH, BPF_PROG_BIND_MAP, BPF_TOKEN_CREATE, + BPF_STATIC_BRANCH_UPDATE, __MAX_BPF_CMD, }; @@ -1787,6 +1788,12 @@ union bpf_attr { __u32 bpffs_fd; } token_create; + struct { /* struct used by BPF_STATIC_BRANCH_UPDATE command */ + __u32 prog_fd; + __u32 insn_off; + __u32 on; + } static_branch; + } __attribute__((aligned(8))); /* The description below is an attempt at providing documentation to eBPF From patchwork Fri Feb 2 16:28:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anton Protopopov X-Patchwork-Id: 13543150 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-ed1-f44.google.com (mail-ed1-f44.google.com [209.85.208.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F332F148FEB for ; Fri, 2 Feb 2024 16:34:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706891667; cv=none; b=VW+2VtYvaeeVUy+f8NOqKe5QtOGGapQ/yGbto1JiDJGNDH4xVG1Y3fAhgy/0K52r3U1qZcpbuP80AzmSUZNdpFVgUB1359Biw8rSvCeaZ/o0nSV4vhrPf3NFGAJTLeQ9jFRcUM88VB5nATvNcaRRSLMJdrEZdXnLiCiP/TCDNcg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706891667; c=relaxed/simple; bh=UZhWlT5mcK3zCufZWDMll9/QiRC2NNAsSv3V3fN9KbE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=pitP8VNgODZ24aCzyanHj7OS9FclAPFgzZgx6Sru6ShCDdW0wBm5bw54cRc9Ly95CFExnc4L1XVIKUoEwD4lz6mr8A42g4aQR8SHgxZUA/Y67UHkWA/csgRCmxwuCMDPKn/31Z/iTx6NsCo7rCbwWrsBLpsa9KyA46ZX63mYARs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=isovalent.com; spf=pass smtp.mailfrom=isovalent.com; dkim=pass (2048-bit key) header.d=isovalent.com header.i=@isovalent.com header.b=GuKrmqWU; arc=none smtp.client-ip=209.85.208.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=isovalent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=isovalent.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=isovalent.com header.i=@isovalent.com header.b="GuKrmqWU" Received: by mail-ed1-f44.google.com with SMTP id 4fb4d7f45d1cf-55790581457so3388514a12.3 for ; Fri, 02 Feb 2024 08:34:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=isovalent.com; s=google; t=1706891664; x=1707496464; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=G4lZr8ol6h5LmQWBe/QEm3mdW7gWf8SzZeol8g09eZM=; b=GuKrmqWUs2Lfqg76hB9HS+JsK7R4gnxyZEtSXckfdxJiVvNOV9nBYtw6jjfuVXmDQg JkqyjXO+TuNKDi9Qn5yiUspnlCfGhtrNMXQn4YnyNYZ3Oo1CfSPSbMlkqgzY/QcRXJCh Z6Z0GCNtXGDIU3ZYPhjpIGvsdDQ+MYtnTqOClzjvjILWJCXxpiMIMOPDDhTHs/RIq9BM nfWUKARIBpf53VXMh9q4lSzMmXWTS4DOVOQW46EZ+uYeVQx0NwXV3YggeWZF7D8MnJhG dt1doQoHA/8Wz2eOaMdAMM15FP2M/kKNoP90hBWxTIBFIUphuSpzxb7mJ/u2HdRvgJmO KSCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706891664; x=1707496464; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=G4lZr8ol6h5LmQWBe/QEm3mdW7gWf8SzZeol8g09eZM=; b=Owhh1PtdmdrV1vNONlEfUuQeVdT1z8lTO75LNRuI3BW7jqCQwCfJTyiwQ7Hhkrm3Tb 6og/V8znQMq33oRm4HJcCJXC7JtIecqRzv7LHQ2Bq9G059UWT6dX/EwRCROH8t7JzYfL qpIQImyUPIAixQtagLLwtLj5OolIf7yWNGDEpYJT6N1S8arePGyKKoEvYzCwwnPHvG8X mkiv9GwFMMUK90UcXpB4HYO81fS+mMftSV+BSTCF+WFXQbLnOeFqMK8aSPGRzrQaZGXf DGdv9KtblxTYlJQ/93p4lFORBbqRCMc1T0drWdHG8gaLaoTBO4WL8cou4M4lFwS5eqEX uLSw== X-Gm-Message-State: AOJu0Yy04IvMLzO00Fca88MOxZWv7+P8rBKsj3UzMUKREXnVpH2Pu6ss 5f548f1Yn84iVEc6+2ph5/MaVnQJlr2rXZLjKwsDmvByog1Vu+fJX0dVEflIUJg= X-Google-Smtp-Source: AGHT+IHzN0RhAJg8oXbyRbO9nZuZd1EcDyiOQI9Vw/WzNQQ08pjNs4+vuc/6gI7NpXO68EmCZPE9WQ== X-Received: by 2002:aa7:d58b:0:b0:55e:f52d:1825 with SMTP id r11-20020aa7d58b000000b0055ef52d1825mr121513edq.35.1706891664241; Fri, 02 Feb 2024 08:34:24 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCX7k4zJbXNVAnB+f6mYz5GFJjM3lkV4CViVNTQaAq1Q5iTUrvd/k29GuWXWQI/jD8m6ewzJAckiGQlCfN+n999qtchTOExcMpFtF7kstmQGmUaLqX5dQ8PBNGtaV2mDmnp3MYZTM7ZD/M+Wtm+GXtY2L7yDyiOPAw3U7UgjI1A5R2M/fwlqLB82gvepsCxCDsSplLiYcsvIRAV2Ibk8Y4h8LH7+XvxPX+pbes5BQMYBcspNV5yeHrLQBZKdwEuv6dU5cbxB3RWe3AEmFVEjtvvAY+eBHD5UFORRS7qPd4g56I9jPGBRTko= Received: from zh-lab-node-5.home ([2a02:168:f656:0:1ac0:4dff:fe0f:3782]) by smtp.gmail.com with ESMTPSA id l19-20020aa7c313000000b0055edbe94b34sm952544edq.54.2024.02.02.08.34.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Feb 2024 08:34:23 -0800 (PST) From: Anton Protopopov To: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Jiri Olsa , Martin KaFai Lau , Stanislav Fomichev , Yonghong Song , Eduard Zingerman , Quentin Monnet , bpf@vger.kernel.org Cc: Anton Protopopov Subject: [PATCH v1 bpf-next 9/9] selftests/bpf: Add tests for new ja* instructions Date: Fri, 2 Feb 2024 16:28:13 +0000 Message-Id: <20240202162813.4184616-10-aspsk@isovalent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240202162813.4184616-1-aspsk@isovalent.com> References: <20240202162813.4184616-1-aspsk@isovalent.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Add several self-tests to test the new instructions and the new BPF_STATIC_BRANCH_UPDATE syscall. Signed-off-by: Anton Protopopov --- .../bpf/prog_tests/bpf_static_branches.c | 269 ++++++++++++++++++ 1 file changed, 269 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/bpf_static_branches.c diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_static_branches.c b/tools/testing/selftests/bpf/prog_tests/bpf_static_branches.c new file mode 100644 index 000000000000..6f54002e6e15 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/bpf_static_branches.c @@ -0,0 +1,269 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Isovalent */ + +#include + +#include +#include + +static inline int _bpf_prog_load(struct bpf_insn *insns, __u32 insn_cnt) +{ + union bpf_attr attr = { + .prog_type = BPF_PROG_TYPE_XDP, /* we don't care */ + .insns = ptr_to_u64(insns), + .insn_cnt = insn_cnt, + .license = ptr_to_u64("GPL"), + }; + + return syscall(__NR_bpf, BPF_PROG_LOAD, &attr, sizeof(attr)); +} + +enum { + OFF, + ON +}; + +static inline int bpf_static_branch_update(int prog_fd, __u32 insn_off, __u32 on) +{ + union bpf_attr attr = { + .static_branch.prog_fd = (__u32)prog_fd, + .static_branch.insn_off = insn_off, + .static_branch.on = on, + }; + + return syscall(__NR_bpf, BPF_STATIC_BRANCH_UPDATE, &attr, sizeof(attr)); +} + +#define BPF_JMP32_OR_NOP(IMM, OFF) \ + ((struct bpf_insn) { \ + .code = BPF_JMP32 | BPF_JA | BPF_K, \ + .dst_reg = 0, \ + .src_reg = BPF_STATIC_BRANCH_JA, \ + .off = OFF, \ + .imm = IMM }) + +#define BPF_JMP_OR_NOP(IMM, OFF) \ + ((struct bpf_insn) { \ + .code = BPF_JMP | BPF_JA | BPF_K, \ + .dst_reg = 0, \ + .src_reg = BPF_STATIC_BRANCH_JA, \ + .off = OFF, \ + .imm = IMM }) + +#define BPF_NOP_OR_JMP32(IMM, OFF) \ + ((struct bpf_insn) { \ + .code = BPF_JMP32 | BPF_JA | BPF_K, \ + .dst_reg = 0, \ + .src_reg = BPF_STATIC_BRANCH_JA | \ + BPF_STATIC_BRANCH_NOP, \ + .off = OFF, \ + .imm = IMM }) + +#define BPF_NOP_OR_JMP(IMM, OFF) \ + ((struct bpf_insn) { \ + .code = BPF_JMP | BPF_JA | BPF_K, \ + .dst_reg = 0, \ + .src_reg = BPF_STATIC_BRANCH_JA | \ + BPF_STATIC_BRANCH_NOP, \ + .off = OFF, \ + .imm = IMM }) + +static const struct bpf_insn insns0[] = { + BPF_JMP_OR_NOP(0, 1), + BPF_NOP_OR_JMP(0, 1), + BPF_JMP32_OR_NOP(1, 0), + BPF_NOP_OR_JMP32(1, 0), +}; + +static void check_ops(void) +{ + struct bpf_insn insns[] = { + {}, /* we will substitute this by insn0[i], i=0,1,2,3 */ + BPF_JMP_IMM(BPF_JA, 0, 0, 1), + BPF_JMP_IMM(BPF_JA, 0, 0, -2), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }; + union bpf_attr attr = { + .prog_type = BPF_PROG_TYPE_XDP, + .insns = ptr_to_u64(insns), + .insn_cnt = ARRAY_SIZE(insns), + .license = ptr_to_u64("GPL"), + }; + bool stop = false; + int prog_fd[4]; + int i; + + for (i = 0; i < 4; i++) { + insns[0] = insns0[i]; + prog_fd[i] = _bpf_prog_load(insns, ARRAY_SIZE(insns)); + if (!ASSERT_GE(prog_fd[i], 0, "correct program")) + stop = true; + } + + for (i = 0; i < 4; i++) + close(prog_fd[i]); + + if (stop) + return; + + /* load should fail: incorrect SRC */ + for (i = 0; i < 4; i++) { + insns[0] = insns0[i]; + insns[0].src_reg |= 4; + prog_fd[i] = _bpf_prog_load(insns, ARRAY_SIZE(insns)); + if (!ASSERT_EQ(prog_fd[i], -1, "incorrect src")) + return; + } + + /* load should fail: incorrect DST */ + for (i = 0; i < 4; i++) { + insns[0] = insns0[i]; + insns[0].dst_reg = i + 1; /* non-zero */ + prog_fd[i] = _bpf_prog_load(insns, ARRAY_SIZE(insns)); + if (!ASSERT_EQ(prog_fd[i], -1, "incorrect dst")) + return; + } + + /* load should fail: both off and imm are set */ + for (i = 0; i < 4; i++) { + insns[0] = insns0[i]; + insns[0].imm = insns[0].off = insns0[i].imm ?: insns0[i].off; + prog_fd[i] = _bpf_prog_load(insns, ARRAY_SIZE(insns)); + if (!ASSERT_EQ(prog_fd[i], -1, "incorrect imm/off")) + return; + } + + /* load should fail: offset is incorrect */ + for (i = 0; i < 4; i++) { + insns[0] = insns0[i]; + + if (insns0[i].imm) + insns[0].imm = -2; + else + insns[0].off = -2; + prog_fd[i] = _bpf_prog_load(insns, ARRAY_SIZE(insns)); + if (!ASSERT_EQ(prog_fd[i], -1, "incorrect imm/off")) + return; + + if (insns0[i].imm) + insns[0].imm = 42; + else + insns[0].off = 42; + prog_fd[i] = _bpf_prog_load(insns, ARRAY_SIZE(insns)); + if (!ASSERT_EQ(prog_fd[i], -1, "incorrect imm/off")) + return; + + /* 0 is not allowed */ + insns[0].imm = insns[0].off = 0; + prog_fd[i] = _bpf_prog_load(insns, ARRAY_SIZE(insns)); + if (!ASSERT_EQ(prog_fd[i], -1, "incorrect imm/off")) + return; + } + + /* incorrect field is used */ + for (i = 0; i < 4; i++) { + int tmp; + + insns[0] = insns0[i]; + + tmp = insns[0].imm; + insns[0].imm = insns[0].off; + insns[0].off = tmp; + + prog_fd[i] = _bpf_prog_load(insns, ARRAY_SIZE(insns)); + if (!ASSERT_EQ(prog_fd[i], -1, "incorrect field")) + return; + } +} + +static void check_syscall(void) +{ + struct bpf_insn insns[] = { + {}, /* we will substitute this by insn0[i], i=0,1,2,3 */ + BPF_JMP_IMM(BPF_JA, 0, 0, 1), + BPF_JMP_IMM(BPF_JA, 0, 0, -2), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }; + union bpf_attr attr = { + .prog_type = BPF_PROG_TYPE_XDP, + .insns = ptr_to_u64(insns), + .insn_cnt = ARRAY_SIZE(insns), + .license = ptr_to_u64("GPL"), + }; + bool stop = false; + int prog_fd[4]; + __u32 insn_off; + int ret; + int i; + + for (i = 0; i < 4; i++) { + insns[0] = insns0[i]; + prog_fd[i] = _bpf_prog_load(insns, ARRAY_SIZE(insns)); + if (!ASSERT_GE(prog_fd[i], 0, "correct program")) + stop = true; + } + + if (stop) + goto end; + + for (i = 0; i < 4; i++) { + /* we can set branch off */ + ret = bpf_static_branch_update(prog_fd[i], 0, OFF); + if (!ASSERT_EQ(ret, 0, "correct update")) + goto end; + + /* we can set branch on */ + ret = bpf_static_branch_update(prog_fd[i], 0, ON); + if (!ASSERT_EQ(ret, 0, "correct update")) + goto end; + + /* incorrect.static_branch.on: can only be 0|1 */ + ret = bpf_static_branch_update(prog_fd[i], 0, 2); + if (!ASSERT_EQ(ret, -1, "incorrect static_branch.on value")) + goto end; + + /* incorrect static_branch.insn_off: can only be 0 in this case */ + for (insn_off = 1; insn_off < 5; insn_off++) { + ret = bpf_static_branch_update(prog_fd[i], insn_off, OFF); + if (!ASSERT_EQ(ret, -1, "incorrect insn_off: not a correct insns")) + goto end; + if (!ASSERT_EQ(errno, EINVAL, "incorrect insn_off: not a correct insns")) + goto end; + } + ret = bpf_static_branch_update(prog_fd[i], 666, OFF); + if (!ASSERT_EQ(ret, -1, "incorrect insn_off: out of range")) + goto end; + if (!ASSERT_EQ(errno, ERANGE, "incorrect insn_off: out puf range")) + goto end; + + /* bad file descriptor: no open file */ + ret = bpf_static_branch_update(-1, 0, OFF); + if (!ASSERT_EQ(ret, -1, "incorrect prog_fd: no file")) + goto end; + if (!ASSERT_EQ(errno, EBADF, "incorrect prog_fd: no file")) + goto end; + + /* bad file descriptor: not a bpf prog */ + ret = bpf_static_branch_update(0, 0, OFF); + if (!ASSERT_EQ(ret, -1, "incorrect prog_fd: not a bpf prog")) + goto end; + if (!ASSERT_EQ(errno, EINVAL, "incorrect prog_fd: not a bpf prog")) + goto end; + } + +end: + for (i = 0; i < 4; i++) + close(prog_fd[i]); + +} + +void test_bpf_static_branches(void) +{ + if (test__start_subtest("check_ops")) + check_ops(); + + if (test__start_subtest("check_syscall")) + check_syscall(); +}