From patchwork Sat Jun 29 09:47:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eduard Zingerman X-Patchwork-Id: 13716878 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0E68C39AFD for ; Sat, 29 Jun 2024 09:48:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719654489; cv=none; b=ZOJu2jGxajf3nUKmdBKVFNEmPW9Os+2z/57QOAxBEWSk0Vowa3FjpNSfMkA9+T6Mb56UOj2NyTGcdPSvwFDd3QR4DkuHRvEqqWwxQvq6huf8lPspbouX6QD/gfSidJY4jkKLve0MXwdWL/tAzXLZcMIkfOjtLdbzOBBx4R2X1bA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719654489; c=relaxed/simple; bh=68p6jS1SGRbOID2C8G6NZjxcNDQc0YTynw+zk8Mq+H4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=UwRsB8n1tvk5m+fCMR1TJjD0nzpCzSxhyebS5igL21IZ+P2kZ2SRTehwY3wtQ9RDgEn3bFwEGs1j32Qr0Il8FHB2k7kGHRykMN/2Tp5fgXpIgU2KY5Q4vxhWCgHNZYp9zV19jNEkb64QmdWaa4cJb0y3VcWPkSwgd0TNS89Kn5g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=TtdpMIBZ; arc=none smtp.client-ip=209.85.210.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="TtdpMIBZ" Received: by mail-pf1-f171.google.com with SMTP id d2e1a72fcca58-7067435d376so1006333b3a.0 for ; Sat, 29 Jun 2024 02:48:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1719654487; x=1720259287; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jxZ1bxAM6zLFt/zwCqSvd+e2s/weo89oqqXWBW/BCjk=; b=TtdpMIBZdKOmFeP8zQE2liMAV8GyUG8NbYkz4HGGOCeSdjQhvHQvW+O/lg9lnGKRZr 4OoQBZw7kXnYAOPEiPM3QVvSFCGNludviYaHyDQGaWCUzKzNhaV0WeUyDX0L4UI3xkIM lAXW1NZc34Y0pK3XL21zOiqhifFh+5O8sQSzIpnvtlheJu/tCKiFgNOb8jpSLdPOwdvx FG3tznrKLIQf+ShaQn7g1pnt/RSZhZaJJ0KQp8haNni9MOgLpKE2Br6Ol28/QnCb68fU H/s3xLIgSYKHB4brlNWqJxtQLhlRi3Nk+/u7ot2G84aOsZ6sQgZIDobiGC2zTE3wiEeq k/Lw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719654487; x=1720259287; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jxZ1bxAM6zLFt/zwCqSvd+e2s/weo89oqqXWBW/BCjk=; b=AKzgCngWDssMD5B3lJEfjrw3TrVZIrXPNbeVAR8JOa0cxeon/bBpMyUUt1yBm0H45b vvVJo8WuDIQ8X8O4x2ulohPB9OhDA5gGTlm/N1/2Ou7gicrXo9vM/rSUk4mhdv3eMmYZ sTXlXHrv8yKUypsumRjK/GTKsPvG5/o5s2CYKD5b+B5rO3T+EsUuQ6rUNYypTxwYlCXj J8/wVqvjcrqjOKk5KB2MVRAIetP3JqWnaz80t/3qovQ0kWzmxebcuJw3y5N17fzyvzXm x4dpJTIKtOKXRCm5BuhSmooVtP4A5aQgg4iUXOVn6b+TzwMPSE2Mcau9y8bR2OVr/ES1 gXqQ== X-Gm-Message-State: AOJu0Yx4itmCdgaxAPCIyfwTN7WPDaCOA/zvD+WkCMM9ABSZXM7k3Npk vigP+qnmpWSP2KKmWhDW+mcno0NEZLOOdvsCk2vVX0tnfc3rkb0Dcjp61A== X-Google-Smtp-Source: AGHT+IFyRFLbVCo3VxCt/w1zexfL0Bx2HHOLqUN/7OHWcdgsH0sltla6Lbm61+VT075LEPXhWm+2TQ== X-Received: by 2002:a05:6a00:98f:b0:706:3f17:ca2 with SMTP id d2e1a72fcca58-70aaa95bfd8mr1114743b3a.0.1719654486890; Sat, 29 Jun 2024 02:48:06 -0700 (PDT) Received: from badger.. ([38.34.87.7]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70804989f5asm2948932b3a.195.2024.06.29.02.48.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 29 Jun 2024 02:48:06 -0700 (PDT) From: Eduard Zingerman To: bpf@vger.kernel.org, ast@kernel.org Cc: andrii@kernel.org, daniel@iogearbox.net, martin.lau@linux.dev, kernel-team@fb.com, yonghong.song@linux.dev, jose.marchesi@oracle.com, Eduard Zingerman Subject: [RFC bpf-next v1 1/8] bpf: add a get_helper_proto() utility function Date: Sat, 29 Jun 2024 02:47:26 -0700 Message-ID: <20240629094733.3863850-2-eddyz87@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240629094733.3863850-1-eddyz87@gmail.com> References: <20240629094733.3863850-1-eddyz87@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Extract the part of check_helper_call() as a utility function allowing to query 'struct bpf_func_proto' for a specific helper function id. Signed-off-by: Eduard Zingerman --- kernel/bpf/verifier.c | 37 ++++++++++++++++++++++++++----------- 1 file changed, 26 insertions(+), 11 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index d3927d819465..8dd3385cf925 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -10261,6 +10261,24 @@ static void update_loop_inline_state(struct bpf_verifier_env *env, u32 subprogno state->callback_subprogno == subprogno); } +static int get_helper_proto(struct bpf_verifier_env *env, int func_id, + const struct bpf_func_proto **ptr) +{ + const struct bpf_func_proto *result = NULL; + + if (func_id < 0 || func_id >= __BPF_FUNC_MAX_ID) + return -ERANGE; + + if (env->ops->get_func_proto) + result = env->ops->get_func_proto(func_id, env->prog); + + if (!result) + return -EINVAL; + + *ptr = result; + return 0; +} + static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn, int *insn_idx_p) { @@ -10277,17 +10295,14 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn /* find function prototype */ func_id = insn->imm; - if (func_id < 0 || func_id >= __BPF_FUNC_MAX_ID) { - verbose(env, "invalid func %s#%d\n", func_id_name(func_id), - func_id); - return -EINVAL; - } - - if (env->ops->get_func_proto) - fn = env->ops->get_func_proto(func_id, env->prog); - if (!fn) { - verbose(env, "program of this type cannot use helper %s#%d\n", - func_id_name(func_id), func_id); + err = get_helper_proto(env, insn->imm, &fn); + if (err) { + if (err == -ERANGE) + verbose(env, "invalid func %s#%d\n", func_id_name(func_id), + func_id); + else + verbose(env, "program of this type cannot use helper %s#%d\n", + func_id_name(func_id), func_id); return -EINVAL; } From patchwork Sat Jun 29 09:47:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eduard Zingerman X-Patchwork-Id: 13716879 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5229F3BBEB for ; Sat, 29 Jun 2024 09:48:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719654491; cv=none; b=q/Q/lluJy/ejn2LN3DRAkF4iYturz8gs68jTJxbX772Y0MdS7c32UUoGnWfn14BOJL88M/dhLf/2fo6TAO8hh56B9CtSi+8ej9HgUegE90EX6BVjAi/17/CLDMx4YDqZXijtrpVgeHx7M5Ok4aAgXO4FNc+JXdvby9tX+dU5+qA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719654491; c=relaxed/simple; bh=88t31omqelfZrG+7VgU81QVbocnRtC9afxZdpBzA+1c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YXrVBkmzU0/B6miPia/udau/IqvpAVTYlY3XxGv0ef6eNaNTaOz1BKr05o6bYj86HhOiSju1RtuPe+l8H5PrEH+qmF3ZJcByGrWFxw4iRsiFCYFJVEviE1A5Oknro5re2ZCNSvJNwFqonKuuahWRKma+lnv/ySumtRjYZWRRUEg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=m9o99T6R; arc=none smtp.client-ip=209.85.210.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="m9o99T6R" Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-7041053c0fdso802868b3a.3 for ; Sat, 29 Jun 2024 02:48:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1719654488; x=1720259288; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8oQkEOWKTyU+VvhBseK2bV41vKgqu0GApcClQiOuF64=; b=m9o99T6RJeOvzC/MxfKiVMw4PwhuBFilZH5BgngJCPkrtnog1U6W5R8/t07hX7KRil eu+I8t9oM9Vm0pJZGzFTJxBloYiZCBGwDhrbxiR3fPvtOx8Nb4s0c32a7I14q2Xlctc/ yeMEjKFQjKYWnxGjFpYG2s/n190nFuTANr8+E7faoCFnRbszp7UFAWwcbWQJXRV21JQj aKEcT2xgQF10FPdbO8S5Q9YE1lKu40mjvXIIczdJPZMEMZwnbrFTLdp5Ye/iliU5Z9cH C/+p7jQODqg6K9/hAUIprdPFmKSS4iPxULjhzqUiviLAd/P8qsNfRFkzMzSsbdI3wVGI NcZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719654488; x=1720259288; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8oQkEOWKTyU+VvhBseK2bV41vKgqu0GApcClQiOuF64=; b=D7NwdF8XIlqeALHkLSI+HywSqPSheKf+vnOr6YDOg3Lw/Xe1cYystWCY2Jc0PuA2Vd oEuAAWDXCRq6PD5xaxFYAGMHP528q6Ii2ICtvz++ey26dGShMw1K0V29XNRXoiLtQK58 lxHoJtkFu1nF8XN93o6n/BMl9usnyCatpCIQM0TKVwdcdoOTddPas8JwepEZ11qXt7N1 GHcsIVMmhjg0zfvra2Vkg8CyKFMD8Wk05hFjrr5HQdXAy4aNiz0mYaB4WUnqeki1Ub7R PCWkaxCnv2dgQSbSo1u7rGvscGi9bKGv/QYZDUH5BD6hctyT5BIfwx+HF66K4P9OInep 0u+g== X-Gm-Message-State: AOJu0YywvOgKbHEolq49+xEUay7cmBBB05b3Zaj1Drb8yMlbuAwpBcUX mn9bmCQ9bFbil4WJeMdzb8ELj/sS36TSJjCsIazsDh29LMGVJZnWKZW5JA== X-Google-Smtp-Source: AGHT+IEszEmBjNODGZU8jQMSXnRMtpkV9y1hmz4bHyclfXxlE+DQITKcO1UTeHdzi2dwRyY2XW8j/g== X-Received: by 2002:a05:6a00:8d6:b0:704:2bdd:82fe with SMTP id d2e1a72fcca58-70aaad5ca44mr585266b3a.15.1719654488009; Sat, 29 Jun 2024 02:48:08 -0700 (PDT) Received: from badger.. ([38.34.87.7]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70804989f5asm2948932b3a.195.2024.06.29.02.48.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 29 Jun 2024 02:48:07 -0700 (PDT) From: Eduard Zingerman To: bpf@vger.kernel.org, ast@kernel.org Cc: andrii@kernel.org, daniel@iogearbox.net, martin.lau@linux.dev, kernel-team@fb.com, yonghong.song@linux.dev, jose.marchesi@oracle.com, Eduard Zingerman , Alexei Starovoitov Subject: [RFC bpf-next v1 2/8] bpf: no_caller_saved_registers attribute for helper calls Date: Sat, 29 Jun 2024 02:47:27 -0700 Message-ID: <20240629094733.3863850-3-eddyz87@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240629094733.3863850-1-eddyz87@gmail.com> References: <20240629094733.3863850-1-eddyz87@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC GCC and LLVM define a no_caller_saved_registers function attribute. This attribute means that function scratches only some of the caller saved registers defined by ABI. For BPF the set of such registers could be defined as follows: - R0 is scratched only if function is non-void; - R1-R5 are scratched only if corresponding parameter type is defined in the function prototype. This commit introduces flag bpf_func_prot->nocsr. If this flag is set for some helper function, verifier assumes that it follows no_caller_saved_registers calling convention. The contract between kernel and clang allows to simultaneously use such functions and maintain backwards compatibility with old kernels that don't understand no_caller_saved_registers calls (nocsr for short): - clang generates a simple pattern for nocsr calls, e.g.: r1 = 1; r2 = 2; *(u64 *)(r10 - 8) = r1; *(u64 *)(r10 - 16) = r2; call %[to_be_inlined_by_jit] r2 = *(u64 *)(r10 - 16); r1 = *(u64 *)(r10 - 8); r0 = r1; r0 += r2; exit; - kernel removes unnecessary spills and fills, if called function is inlined by current JIT (with assumption that patch inserted by JIT honors nocsr contract, e.g. does not scratch r3-r5 for the example above), e.g. the code above would be transformed to: r1 = 1; r2 = 2; call %[to_be_inlined_by_jit] r0 = r1; r0 += r2; exit; Technically, the transformation is split into the following phases: - during check_cfg() function update_nocsr_pattern_marks() is used to find potential patterns; - upon stack read or write access, function check_nocsr_stack_contract() is used to verify if stack offsets, presumably reserved for nocsr patterns, are used only from those patterns; - function remove_nocsr_spills_fills(), called from bpf_check(), applies the rewrite for valid patterns. See comment in match_and_mark_nocsr_pattern() for more details. Suggested-by: Alexei Starovoitov Signed-off-by: Eduard Zingerman --- include/linux/bpf.h | 6 + include/linux/bpf_verifier.h | 9 ++ kernel/bpf/verifier.c | 300 ++++++++++++++++++++++++++++++++++- 3 files changed, 307 insertions(+), 8 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 960780ef04e1..f4faa6b8cb5b 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -807,6 +807,12 @@ struct bpf_func_proto { bool gpl_only; bool pkt_access; bool might_sleep; + /* set to true if helper follows contract for gcc/llvm + * attribute no_caller_saved_registers: + * - void functions do not scratch r0 + * - functions taking N arguments scratch only registers r1-rN + */ + bool nocsr; enum bpf_return_type ret_type; union { struct { diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 2b54e25d2364..67dbe4cb1529 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -585,6 +585,10 @@ struct bpf_insn_aux_data { * accepts callback function as a parameter. */ bool calls_callback; + /* true if STX or LDX instruction is a part of a spill/fill + * pattern for a no_caller_saved_registers call. + */ + bool nocsr_pattern; }; #define MAX_USED_MAPS 64 /* max number of maps accessed by one eBPF program */ @@ -641,6 +645,11 @@ struct bpf_subprog_info { u32 linfo_idx; /* The idx to the main_prog->aux->linfo */ u16 stack_depth; /* max. stack depth used by this function */ u16 stack_extra; + /* stack depth after which slots reserved for + * no_caller_saved_registers spills/fills start, + * value <= nocsr_stack_off belongs to the spill/fill area. + */ + s16 nocsr_stack_off; bool has_tail_call: 1; bool tail_call_reachable: 1; bool has_ld_abs: 1; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 8dd3385cf925..1340d3e60d30 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -2471,16 +2471,41 @@ static int cmp_subprogs(const void *a, const void *b) ((struct bpf_subprog_info *)b)->start; } -static int find_subprog(struct bpf_verifier_env *env, int off) +/* Find subprogram that contains instruction at 'off' */ +static int find_containing_subprog(struct bpf_verifier_env *env, int off) { - struct bpf_subprog_info *p; + struct bpf_subprog_info *vals = env->subprog_info; + int high = env->subprog_cnt - 1; + int low = 0, ret = -ENOENT; - p = bsearch(&off, env->subprog_info, env->subprog_cnt, - sizeof(env->subprog_info[0]), cmp_subprogs); - if (!p) + if (off >= env->prog->len || off < 0) return -ENOENT; - return p - env->subprog_info; + while (low <= high) { + int mid = (low + high)/2; + struct bpf_subprog_info *val = &vals[mid]; + int diff = off - val->start; + + if (diff < 0) { + high = mid - 1; + } else { + low = mid + 1; + /* remember last time mid.start <= off */ + ret = mid; + } + } + return ret; +} + +/* Find subprogram that starts exactly at 'off' */ +static int find_subprog(struct bpf_verifier_env *env, int off) +{ + int idx; + + idx = find_containing_subprog(env, off); + if (idx < 0 || env->subprog_info[idx].start != off) + return -ENOENT; + return idx; } static int add_subprog(struct bpf_verifier_env *env, int off) @@ -4501,6 +4526,23 @@ static int get_reg_width(struct bpf_reg_state *reg) return fls64(reg->umax_value); } +/* See comment for match_and_mark_nocsr_pattern() */ +static void check_nocsr_stack_contract(struct bpf_verifier_env *env, struct bpf_func_state *state, + int insn_idx, int off) +{ + struct bpf_subprog_info *subprog = &env->subprog_info[state->subprogno]; + struct bpf_insn_aux_data *aux = &env->insn_aux_data[insn_idx]; + + if (subprog->nocsr_stack_off <= off || aux->nocsr_pattern) + return; + /* access to the region [max_stack_depth .. nocsr_stack_off] + * from something that is not a part of the nocsr pattern, + * disable nocsr rewrites for current subprogram by setting + * nocsr_stack_off to a value smaller than any possible offset. + */ + subprog->nocsr_stack_off = S16_MIN; +} + /* check_stack_{read,write}_fixed_off functions track spill/fill of registers, * stack boundary and alignment are checked in check_mem_access() */ @@ -4549,6 +4591,7 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env, if (err) return err; + check_nocsr_stack_contract(env, state, insn_idx, off); mark_stack_slot_scratched(env, spi); if (reg && !(off % BPF_REG_SIZE) && reg->type == SCALAR_VALUE && env->bpf_capable) { bool reg_value_fits; @@ -4682,6 +4725,7 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env, return err; } + check_nocsr_stack_contract(env, state, insn_idx, min_off); /* Variable offset writes destroy any spilled pointers in range. */ for (i = min_off; i < max_off; i++) { u8 new_type, *stype; @@ -4820,6 +4864,7 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env, reg = ®_state->stack[spi].spilled_ptr; mark_stack_slot_scratched(env, spi); + check_nocsr_stack_contract(env, state, env->insn_idx, off); if (is_spilled_reg(®_state->stack[spi])) { u8 spill_size = 1; @@ -4980,6 +5025,7 @@ static int check_stack_read_var_off(struct bpf_verifier_env *env, min_off = reg->smin_value + off; max_off = reg->smax_value + off; mark_reg_stack_read(env, ptr_state, min_off, max_off + size, dst_regno); + check_nocsr_stack_contract(env, ptr_state, env->insn_idx, min_off); return 0; } @@ -15950,6 +15996,205 @@ static int visit_func_call_insn(int t, struct bpf_insn *insns, return ret; } +/* Bitmask with 1s for all caller saved registers */ +#define ALL_CALLER_SAVED_REGS ((1u << CALLER_SAVED_REGS) - 1) + +/* Return a bitmask specifying which caller saved registers are + * modified by a call to a helper. + * (Either as a return value or as scratch registers). + * + * For normal helpers registers R0-R5 are scratched. + * For helpers marked as no_csr: + * - scratch R0 if function is non-void; + * - scratch R1-R5 if corresponding parameter type is set + * in the function prototype. + */ +static u8 get_helper_reg_mask(const struct bpf_func_proto *fn) +{ + u8 mask; + int i; + + if (!fn->nocsr) + return ALL_CALLER_SAVED_REGS; + + mask = 0; + mask |= fn->ret_type == RET_VOID ? 0 : BIT(BPF_REG_0); + for (i = 0; i < ARRAY_SIZE(fn->arg_type); ++i) + mask |= fn->arg_type[i] == ARG_DONTCARE ? 0 : BIT(BPF_REG_1 + i); + return mask; +} + +/* True if do_misc_fixups() replaces calls to helper number 'imm', + * replacement patch is presumed to follow no_caller_saved_registers contract + * (see match_and_mark_nocsr_pattern() below). + */ +static bool verifier_inlines_helper_call(struct bpf_verifier_env *env, s32 imm) +{ + return false; +} + +/* If 'insn' is a call that follows no_caller_saved_registers contract + * and called function is inlined by current jit, return a mask with + * 1s corresponding to registers that are scratched by this call + * (depends on return type and number of return parameters). + * Otherwise return ALL_CALLER_SAVED_REGS mask. + */ +static u32 call_csr_mask(struct bpf_verifier_env *env, struct bpf_insn *insn) +{ + const struct bpf_func_proto *fn; + + if (bpf_helper_call(insn) && + verifier_inlines_helper_call(env, insn->imm) && + get_helper_proto(env, insn->imm, &fn) == 0 && + fn->nocsr) + return ~get_helper_reg_mask(fn); + + return ALL_CALLER_SAVED_REGS; +} + +/* GCC and LLVM define a no_caller_saved_registers function attribute. + * This attribute means that function scratches only some of + * the caller saved registers defined by ABI. + * For BPF the set of such registers could be defined as follows: + * - R0 is scratched only if function is non-void; + * - R1-R5 are scratched only if corresponding parameter type is defined + * in the function prototype. + * + * The contract between kernel and clang allows to simultaneously use + * such functions and maintain backwards compatibility with old + * kernels that don't understand no_caller_saved_registers calls + * (nocsr for short): + * + * - for nocsr calls clang allocates registers as-if relevant r0-r5 + * registers are not scratched by the call; + * + * - as a post-processing step, clang visits each nocsr call and adds + * spill/fill for every live r0-r5; + * + * - stack offsets used for the spill/fill are allocated as minimal + * stack offsets in whole function and are not used for any other + * purposes; + * + * - when kernel loads a program, it looks for such patterns + * (nocsr function surrounded by spills/fills) and checks if + * spill/fill stack offsets are used exclusively in nocsr patterns; + * + * - if so, and if current JIT inlines the call to the nocsr function + * (e.g. a helper call), kernel removes unnecessary spill/fill pairs; + * + * - when old kernel loads a program, presence of spill/fill pairs + * keeps BPF program valid, albeit slightly less efficient. + * + * For example: + * + * r1 = 1; + * r2 = 2; + * *(u64 *)(r10 - 8) = r1; r1 = 1; + * *(u64 *)(r10 - 16) = r2; r2 = 2; + * call %[to_be_inlined_by_jit] --> call %[to_be_inlined_by_jit] + * r2 = *(u64 *)(r10 - 16); r0 = r1; + * r1 = *(u64 *)(r10 - 8); r0 += r2; + * r0 = r1; exit; + * r0 += r2; + * exit; + * + * The purpose of match_and_mark_nocsr_pattern is to: + * - look for such patterns; + * - mark spill and fill instructions in env->insn_aux_data[*].nocsr_pattern; + * - update env->subprog_info[*]->nocsr_stack_off to find an offset + * at which nocsr spill/fill stack slots start. + * + * The .nocsr_pattern and .nocsr_stack_off are used by + * check_nocsr_stack_contract() to check if every stack access to + * nocsr spill/fill stack slot originates from spill/fill + * instructions, members of nocsr patterns. + * + * If such condition holds true for a subprogram, nocsr patterns could + * be rewritten by remove_nocsr_spills_fills(). + * Otherwise nocsr patterns are not changed in the subprogram + * (code, presumably, generated by an older clang version). + * + * For example, it is *not* safe to remove spill/fill below: + * + * r1 = 1; + * *(u64 *)(r10 - 8) = r1; r1 = 1; + * call %[to_be_inlined_by_jit] --> call %[to_be_inlined_by_jit] + * r1 = *(u64 *)(r10 - 8); r0 = *(u64 *)(r10 - 8); <---- wrong !!! + * r0 = *(u64 *)(r10 - 8); r0 += r1; + * r0 += r1; exit; + * exit; + */ +static int match_and_mark_nocsr_pattern(struct bpf_verifier_env *env, int t, bool mark) +{ + struct bpf_insn *insns = env->prog->insnsi, *stx, *ldx; + struct bpf_subprog_info *subprog; + u32 csr_mask = call_csr_mask(env, &insns[t]); + u32 reg_mask = ~csr_mask | ~ALL_CALLER_SAVED_REGS; + int s, i; + s16 off; + + if (csr_mask == ALL_CALLER_SAVED_REGS) + return false; + + for (i = 1, off = 0; i <= ARRAY_SIZE(caller_saved); ++i, off += BPF_REG_SIZE) { + if (t - i < 0 || t + i >= env->prog->len) + break; + stx = &insns[t - i]; + ldx = &insns[t + i]; + if (off == 0) { + off = stx->off; + if (off % BPF_REG_SIZE != 0) + break; + } + if (/* *(u64 *)(r10 - off) = r[0-5]? */ + stx->code != (BPF_STX | BPF_MEM | BPF_DW) || + stx->dst_reg != BPF_REG_10 || + /* r[0-5] = *(u64 *)(r10 - off)? */ + ldx->code != (BPF_LDX | BPF_MEM | BPF_DW) || + ldx->src_reg != BPF_REG_10 || + /* check spill/fill for the same reg and offset */ + stx->src_reg != ldx->dst_reg || + stx->off != ldx->off || + stx->off != off || + /* this should be a previously unseen register */ + BIT(stx->src_reg) & reg_mask) + break; + reg_mask |= BIT(stx->src_reg); + if (mark) { + env->insn_aux_data[t - i].nocsr_pattern = true; + env->insn_aux_data[t + i].nocsr_pattern = true; + } + } + if (i == 1) + return 0; + if (mark) { + s = find_containing_subprog(env, t); + /* can't happen */ + if (WARN_ON_ONCE(s < 0)) + return 0; + subprog = &env->subprog_info[s]; + subprog->nocsr_stack_off = min(subprog->nocsr_stack_off, off); + } + return i - 1; +} + +/* If instruction 't' is a nocsr call surrounded by spill/fill pairs, + * update env->subprog_info[_]->nocsr_stack_off and + * env->insn_aux_data[_].nocsr_pattern fields. + */ +static void update_nocsr_pattern_marks(struct bpf_verifier_env *env, int t) +{ + match_and_mark_nocsr_pattern(env, t, true); +} + +/* If instruction 't' is a nocsr call surrounded by spill/fill pairs, + * return the number of such pairs. + */ +static int match_nocsr_pattern(struct bpf_verifier_env *env, int t) +{ + return match_and_mark_nocsr_pattern(env, t, false); +} + /* Visits the instruction at index t and returns one of the following: * < 0 - an error occurred * DONE_EXPLORING - the instruction was fully explored @@ -16017,6 +16262,8 @@ static int visit_insn(int t, struct bpf_verifier_env *env) mark_force_checkpoint(env, t); } } + if (insn->src_reg == 0) + update_nocsr_pattern_marks(env, t); return visit_func_call_insn(t, insns, env, insn->src_reg == BPF_PSEUDO_CALL); case BPF_JA: @@ -19063,15 +19310,16 @@ static int opt_remove_dead_code(struct bpf_verifier_env *env) return 0; } +static const struct bpf_insn NOP = BPF_JMP_IMM(BPF_JA, 0, 0, 0); + static int opt_remove_nops(struct bpf_verifier_env *env) { - const struct bpf_insn ja = BPF_JMP_IMM(BPF_JA, 0, 0, 0); struct bpf_insn *insn = env->prog->insnsi; int insn_cnt = env->prog->len; int i, err; for (i = 0; i < insn_cnt; i++) { - if (memcmp(&insn[i], &ja, sizeof(ja))) + if (memcmp(&insn[i], &NOP, sizeof(NOP))) continue; err = verifier_remove_insns(env, i, 1); @@ -20801,6 +21049,39 @@ static int optimize_bpf_loop(struct bpf_verifier_env *env) return 0; } +/* Remove unnecessary spill/fill pairs, members of nocsr pattern. + * Do this as a separate pass to avoid interfering with helper/kfunc + * inlining logic in do_misc_fixups(). + * See comment for match_and_mark_nocsr_pattern(). + */ +static int remove_nocsr_spills_fills(struct bpf_verifier_env *env) +{ + struct bpf_subprog_info *subprogs = env->subprog_info; + int i, j, spills_num, cur_subprog = 0; + struct bpf_insn *insn = env->prog->insnsi; + int insn_cnt = env->prog->len; + + for (i = 0; i < insn_cnt; i++, insn++) { + spills_num = match_nocsr_pattern(env, i); + if (spills_num == 0) + goto next_insn; + for (j = 1; j <= spills_num; ++j) + if ((insn - j)->off >= subprogs[cur_subprog].nocsr_stack_off || + (insn + j)->off >= subprogs[cur_subprog].nocsr_stack_off) + goto next_insn; + /* NOPs are removed by opt_remove_nops() later */ + for (j = 1; j <= spills_num; ++j) { + *(insn - j) = NOP; + *(insn + j) = NOP; + } + +next_insn: + if (subprogs[cur_subprog + 1].start == i + 1) + cur_subprog++; + } + return 0; +} + static void free_states(struct bpf_verifier_env *env) { struct bpf_verifier_state_list *sl, *sln; @@ -21719,6 +22000,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, __u3 if (ret == 0) ret = optimize_bpf_loop(env); + if (ret == 0) + ret = remove_nocsr_spills_fills(env); + if (is_priv) { if (ret == 0) opt_hard_wire_dead_code_branches(env); From patchwork Sat Jun 29 09:47:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eduard Zingerman X-Patchwork-Id: 13716880 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-ot1-f41.google.com (mail-ot1-f41.google.com [209.85.210.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 32F4339AFD for ; Sat, 29 Jun 2024 09:48:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719654491; cv=none; b=hYXaKC5rH/OE1yZZ1m1jlcmLVT17BsZAcnVeYeBgS6EriidJoDa8Lk3IFXeUvWer72hr5oIaGrIxJ0p3FhA8/7pNmq5vHNY9WGoLf+CiF+qhZk2XNSs9uc/xR9Ucwj4xOQBJmLz2PCOW/Q9qyawltQupHl55L8z0YL5RCAUiwdM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719654491; c=relaxed/simple; bh=wW6VTQHBN1F8s86TJZF8xTChvV6uDYYIZg55KlKFRIk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FI05rFYbXdlsaKqD6A4opgSaHLYGiO0blgzh3xI1w4MuoWiWJMfV0LceiztU898TLUNrvMSlOEfmjM+5RD4sIMAGsfDFuJdhSk/lMnm1lTvlrMSeZpGbR5/JPRndT7GKXnZQHl1J+M5tTJyxRY26NuNwZUOAYYzrme1ER/o31oo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Exl3IffD; arc=none smtp.client-ip=209.85.210.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Exl3IffD" Received: by mail-ot1-f41.google.com with SMTP id 46e09a7af769-6f8d0a00a35so1293129a34.2 for ; Sat, 29 Jun 2024 02:48:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1719654489; x=1720259289; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9n72pGCNJB6/flRjs1h8f3ESxisHAQzCEWPPFbEsaoY=; b=Exl3IffDr3tn3AmgEDs1lYTC+iG4Mw7mESmU2gDHA+z7b7J1UEhImdEnC7SYBUxiJr S6jUvNDUKDKXKSH1bJQ4cHkEBvhsN8/BkH7SoVbsWXMH4qGlxBoVBSPk7D1fEFA5/1vY 9JVtDF1apLxHBSwQb1hDPjx7OXFiR9jWZWbqDbwFATlMtTjfD0RI8v2Okl5irWvgqvhj VelNnCssNCusqGNgAXK3ebuBeEvbkHzRV2Poj4l3VMGQFONrjAa90pQlK0bJEam5pMjp 7NRG/pHd+ANUX8BDS9YX/8//lQzs4ZAutYTNJ9CuPdDgLrptdcu+WkAurpTzdlnZpCOZ hTcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719654489; x=1720259289; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9n72pGCNJB6/flRjs1h8f3ESxisHAQzCEWPPFbEsaoY=; b=L69Oso9MKk6gpMGqt4c3MZ8zRdHcqJaX8lQPV37OFymXjhG9NeVNEY3Unkzxq/JNNp 2s/KOwCFFgHHCpKoU+AzcVe2QRgV/YceYVD2FhR3Am3VCJR8aMgfhxBxKsB4iSub9mWV dSl1OJIrviXOf99BcaMinKrJnNqYjJVwvCV6Uw2V7gTRHbAEmbLadfZMRPhB0oQVBuU2 FpwyKv8T9SZKGEXbWag3dUmANI19eNexT4vGQb3RGRQqless2QswD0sUv4QH/QVR7fDq Pfhhw7n2aMgRMKloWrrm6eQY7YzVIXXPnLfYiiKiKdgG2To3OxSLkXOVL36YWaJlNIpb zoIg== X-Gm-Message-State: AOJu0Yx3ZtLbsS9e2wkldT8uOWp12lEwQ1Qu0X1w8OmgLcp9bCT1923i 9K5GQsGViB7apTiQrznhNF1L/scUHzYLb0DA2PsujP4wWG5nyR9buKU8zQ== X-Google-Smtp-Source: AGHT+IGe1tf+3uz9E+eQrEITRLTqkkg3lD2v2ZsHFVwBGz6gNkD7TdDGMTk2tKs2SzPYmz+hE0Txzg== X-Received: by 2002:a05:6830:94:b0:6fa:732b:862e with SMTP id 46e09a7af769-7020763c466mr587307a34.9.1719654488862; Sat, 29 Jun 2024 02:48:08 -0700 (PDT) Received: from badger.. ([38.34.87.7]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70804989f5asm2948932b3a.195.2024.06.29.02.48.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 29 Jun 2024 02:48:08 -0700 (PDT) From: Eduard Zingerman To: bpf@vger.kernel.org, ast@kernel.org Cc: andrii@kernel.org, daniel@iogearbox.net, martin.lau@linux.dev, kernel-team@fb.com, yonghong.song@linux.dev, jose.marchesi@oracle.com, Eduard Zingerman Subject: [RFC bpf-next v1 3/8] bpf, x86: no_caller_saved_registers for bpf_get_smp_processor_id() Date: Sat, 29 Jun 2024 02:47:28 -0700 Message-ID: <20240629094733.3863850-4-eddyz87@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240629094733.3863850-1-eddyz87@gmail.com> References: <20240629094733.3863850-1-eddyz87@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC On x86 verifier replaces call to bpf_get_smp_processor_id() with a sequence of instructions that modify only R0. This satisfies attribute no_caller_saved_registers contract. Allow rewrite of no_caller_saved_registers patterns for bpf_get_smp_processor_id() in order to use this function as a canary for no_caller_saved_registers tests. Signed-off-by: Eduard Zingerman --- kernel/bpf/helpers.c | 1 + kernel/bpf/verifier.c | 11 +++++++++-- 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 229396172026..1df01f454590 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -158,6 +158,7 @@ const struct bpf_func_proto bpf_get_smp_processor_id_proto = { .func = bpf_get_smp_processor_id, .gpl_only = false, .ret_type = RET_INTEGER, + .nocsr = true, }; BPF_CALL_0(bpf_get_numa_node_id) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 1340d3e60d30..d68ed70186c8 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -16030,7 +16030,14 @@ static u8 get_helper_reg_mask(const struct bpf_func_proto *fn) */ static bool verifier_inlines_helper_call(struct bpf_verifier_env *env, s32 imm) { - return false; + switch (imm) { +#ifdef CONFIG_X86_64 + case BPF_FUNC_get_smp_processor_id: + return env->prog->jit_requested && bpf_jit_supports_percpu_insn(); +#endif + default: + return false; + } } /* If 'insn' is a call that follows no_caller_saved_registers contract @@ -20666,7 +20673,7 @@ static int do_misc_fixups(struct bpf_verifier_env *env) #ifdef CONFIG_X86_64 /* Implement bpf_get_smp_processor_id() inline. */ if (insn->imm == BPF_FUNC_get_smp_processor_id && - prog->jit_requested && bpf_jit_supports_percpu_insn()) { + verifier_inlines_helper_call(env, insn->imm)) { /* BPF_FUNC_get_smp_processor_id inlining is an * optimization, so if pcpu_hot.cpu_number is ever * changed in some incompatible and hard to support From patchwork Sat Jun 29 09:47:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eduard Zingerman X-Patchwork-Id: 13716881 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-oa1-f54.google.com (mail-oa1-f54.google.com [209.85.160.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0A7DF3CF7E for ; Sat, 29 Jun 2024 09:48:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719654492; cv=none; b=OX5Ous+MaPVkFLefni3Uh/1GEII/deCxPaCGguDdK0/xENwYQrafgAT6JbM+MLlIPN5dvl9+o4S32lONgQgvePUyGkVM26OV47FQcabxtPRu6IJWP/wkJN0xFLpV3GdJNHFSMakcowUdkLqAYrseg1WbyHuEjJjctyjXfTn0ANE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719654492; c=relaxed/simple; bh=hMoR0VsH6b/86oM4MC60TZBCv5ZEfsGMaDUt3aOIl2o=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=d4fb0WRJ7yyRna98yB0Pl2zWeQrCXrNU3os3gRseKOgON0izmxWo4YWDpqMHiJZWZFiKazMbYZZ73KSTXpCNuTAo8ivZjWhevabP4u350xYdNImoPABop3bh++MGHw2OR4ySDDcM8/epuQh6CEhD6MMScbQSwfsn27pH4tk+EmI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=kXLyuM0f; arc=none smtp.client-ip=209.85.160.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="kXLyuM0f" Received: by mail-oa1-f54.google.com with SMTP id 586e51a60fabf-25c9786835eso803580fac.3 for ; Sat, 29 Jun 2024 02:48:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1719654490; x=1720259290; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tF6+BoPqiTXE6WlMeUomLmOMPBTE3lwEWPLjkPv3CLc=; b=kXLyuM0feBKnhk1PKkOxowmHX/eB+3AmtO/si8ks4HWp+vthJNReH+K1hubJfSCxAg 2zrs7kZlZU3+bY4f23lSWPkPGfBlHNZVjtON0p2REEGCv64/pAdqYpLSmsLAbS2XWLwU ZKrxjJRf25GfdvzqQgJmw5KvWar9iBfKrSEwTEesgRU+oUiBjOM4sZDEY0ZnvGDzWG6a HQq4u2mggzzdCVwF46PohlDVvcSMkED1QIF0DxPugmADAjFq458iItK+xl2XgT8fWSRC fi7t2OM6nsXMC6efq972l7HzsL0xL5yHT3rwX9nEQ06diSMFBU/Tu+AP0CQ5JaiYzsHe +HyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719654490; x=1720259290; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tF6+BoPqiTXE6WlMeUomLmOMPBTE3lwEWPLjkPv3CLc=; b=mEF7qG8x/R8X9nGZIgKknxfpxa0Dk3fLJgrMYBABRp8aFwip7DtrQ8G5dJLYdsifNd k+aRVObZe4IsLfgHg5948M0X2q2rjJo9hJroa0lWzr7t4BfSv4B3vy3YVKykx2vqrKvL dcA+/KywtSsi+TXPp4lWO3eVwnCdraErXZY8XkbwRq6p7s5XsTKetUs2Leqd5AWNmIQj sGPXT+lcXbiOO1PyyXD6Blx96NkG4kk/e+TvZmBQGId5QkHWzwWPUFrEvDAKqgTwWMZo YFUC/iT1XACA8EemnIAWewDqb0XcoKgpNz1JZ2H6a14jqy6i44oZ1p1btEJWayiF4zhe lenQ== X-Gm-Message-State: AOJu0YwsnYZqmkvEw8DFhJkalMMrQ5lQVaHcBQblWuGw2muMT6r5Q76W Fe25VU3nXsY9aocWC1TNv2aiqaKI1i/WonSQNjb/fPheAnev7KBPGz1WWA== X-Google-Smtp-Source: AGHT+IGvQRPqdtitddhAH7dYOnsTmppCdPbG8itf0Ab7wZQh1Cw6Zi2zEZ225UNidjeKqRp2mO/oDQ== X-Received: by 2002:a05:6870:d202:b0:254:b4a6:958d with SMTP id 586e51a60fabf-25db338f565mr417899fac.2.1719654489738; Sat, 29 Jun 2024 02:48:09 -0700 (PDT) Received: from badger.. ([38.34.87.7]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70804989f5asm2948932b3a.195.2024.06.29.02.48.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 29 Jun 2024 02:48:09 -0700 (PDT) From: Eduard Zingerman To: bpf@vger.kernel.org, ast@kernel.org Cc: andrii@kernel.org, daniel@iogearbox.net, martin.lau@linux.dev, kernel-team@fb.com, yonghong.song@linux.dev, jose.marchesi@oracle.com, Eduard Zingerman Subject: [RFC bpf-next v1 4/8] selftests/bpf: extract utility function for BPF disassembly Date: Sat, 29 Jun 2024 02:47:29 -0700 Message-ID: <20240629094733.3863850-5-eddyz87@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240629094733.3863850-1-eddyz87@gmail.com> References: <20240629094733.3863850-1-eddyz87@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC uint32_t disasm_insn(struct bpf_insn *insn, char *buf, size_t buf_sz); Disassembles instruction 'insn' to a text buffer 'buf'. Removes insn->code hex prefix added by kernel disassemly routine. Returns the length of decoded instruction (either 1 or 2). Signed-off-by: Eduard Zingerman --- tools/testing/selftests/bpf/Makefile | 1 + tools/testing/selftests/bpf/disasm_helpers.c | 50 +++++++++++++ tools/testing/selftests/bpf/disasm_helpers.h | 12 ++++ .../selftests/bpf/prog_tests/ctx_rewrite.c | 71 +++---------------- tools/testing/selftests/bpf/testing_helpers.c | 1 + 5 files changed, 72 insertions(+), 63 deletions(-) create mode 100644 tools/testing/selftests/bpf/disasm_helpers.c create mode 100644 tools/testing/selftests/bpf/disasm_helpers.h diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index e0b3887b3d2d..5eb7b5eb89d2 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -636,6 +636,7 @@ TRUNNER_EXTRA_SOURCES := test_progs.c \ test_loader.c \ xsk.c \ disasm.c \ + disasm_helpers.c \ json_writer.c \ flow_dissector_load.h \ ip_check_defrag_frags.h diff --git a/tools/testing/selftests/bpf/disasm_helpers.c b/tools/testing/selftests/bpf/disasm_helpers.c new file mode 100644 index 000000000000..7c29d294a456 --- /dev/null +++ b/tools/testing/selftests/bpf/disasm_helpers.c @@ -0,0 +1,50 @@ +// SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) + +#include +#include "disasm.h" + +struct print_insn_context { + char *buf; + size_t sz; +}; + +static void print_insn_cb(void *private_data, const char *fmt, ...) +{ + struct print_insn_context *ctx = private_data; + va_list args; + + va_start(args, fmt); + vsnprintf(ctx->buf, ctx->sz, fmt, args); + va_end(args); +} + +uint32_t disasm_insn(struct bpf_insn *insn, char *buf, size_t buf_sz) +{ + struct print_insn_context ctx = { + .buf = buf, + .sz = buf_sz, + }; + struct bpf_insn_cbs cbs = { + .cb_print = print_insn_cb, + .private_data = &ctx, + }; + int pfx_end, sfx_start, len; + bool double_insn; + + print_bpf_insn(&cbs, insn, true); + /* We share code with kernel BPF disassembler, it adds '(FF) ' prefix + * for each instruction (FF stands for instruction `code` byte). + * Remove the prefix inplace, and also simplify call instructions. + * E.g.: "(85) call foo#10" -> "call foo". + */ + pfx_end = 0; + sfx_start = max((int)strlen(buf) - 1, 0); + /* For whatever reason %n is not counted in sscanf return value */ + sscanf(buf, "(%*[^)]) %n", &pfx_end); + sscanf(buf, "(%*[^)]) call %*[^#]%n", &sfx_start); + len = sfx_start - pfx_end; + memmove(buf, buf + pfx_end, len); + buf[len] = 0; + double_insn = insn->code == (BPF_LD | BPF_IMM | BPF_DW); + return double_insn ? 2 : 1; +} diff --git a/tools/testing/selftests/bpf/disasm_helpers.h b/tools/testing/selftests/bpf/disasm_helpers.h new file mode 100644 index 000000000000..db3dfe9f93dd --- /dev/null +++ b/tools/testing/selftests/bpf/disasm_helpers.h @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) */ + +#ifndef __DISASM_HELPERS_H +#define __DISASM_HELPERS_H + +#include + +struct bpf_insn; + +uint32_t disasm_insn(struct bpf_insn *insn, char *buf, size_t buf_sz); + +#endif /* __DISASM_HELPERS_H */ diff --git a/tools/testing/selftests/bpf/prog_tests/ctx_rewrite.c b/tools/testing/selftests/bpf/prog_tests/ctx_rewrite.c index 08b6391f2f56..55e41167f1f3 100644 --- a/tools/testing/selftests/bpf/prog_tests/ctx_rewrite.c +++ b/tools/testing/selftests/bpf/prog_tests/ctx_rewrite.c @@ -10,7 +10,8 @@ #include "bpf/btf.h" #include "bpf_util.h" #include "linux/filter.h" -#include "disasm.h" +#include "linux/kernel.h" +#include "disasm_helpers.h" #define MAX_PROG_TEXT_SZ (32 * 1024) @@ -628,63 +629,6 @@ static bool match_pattern(struct btf *btf, char *pattern, char *text, char *reg_ return false; } -static void print_insn(void *private_data, const char *fmt, ...) -{ - va_list args; - - va_start(args, fmt); - vfprintf((FILE *)private_data, fmt, args); - va_end(args); -} - -/* Disassemble instructions to a stream */ -static void print_xlated(FILE *out, struct bpf_insn *insn, __u32 len) -{ - const struct bpf_insn_cbs cbs = { - .cb_print = print_insn, - .cb_call = NULL, - .cb_imm = NULL, - .private_data = out, - }; - bool double_insn = false; - int i; - - for (i = 0; i < len; i++) { - if (double_insn) { - double_insn = false; - continue; - } - - double_insn = insn[i].code == (BPF_LD | BPF_IMM | BPF_DW); - print_bpf_insn(&cbs, insn + i, true); - } -} - -/* We share code with kernel BPF disassembler, it adds '(FF) ' prefix - * for each instruction (FF stands for instruction `code` byte). - * This function removes the prefix inplace for each line in `str`. - */ -static void remove_insn_prefix(char *str, int size) -{ - const int prefix_size = 5; - - int write_pos = 0, read_pos = prefix_size; - int len = strlen(str); - char c; - - size = min(size, len); - - while (read_pos < size) { - c = str[read_pos++]; - if (c == 0) - break; - str[write_pos++] = c; - if (c == '\n') - read_pos += prefix_size; - } - str[write_pos] = 0; -} - struct prog_info { char *prog_kind; enum bpf_prog_type prog_type; @@ -702,8 +646,10 @@ static void match_program(struct btf *btf, struct bpf_insn *buf = NULL; int err = 0, prog_fd = 0; FILE *prog_out = NULL; + char insn_buf[64]; char *text = NULL; __u32 cnt = 0; + int i; text = calloc(MAX_PROG_TEXT_SZ, 1); if (!text) { @@ -739,12 +685,11 @@ static void match_program(struct btf *btf, PRINT_FAIL("Can't open memory stream\n"); goto out; } - if (skip_first_insn) - print_xlated(prog_out, buf + 1, cnt - 1); - else - print_xlated(prog_out, buf, cnt); + for (i = skip_first_insn ? 1 : 0; i < cnt;) { + i += disasm_insn(buf + i, insn_buf, sizeof(insn_buf)); + fprintf(prog_out, "%s\n", insn_buf); + } fclose(prog_out); - remove_insn_prefix(text, MAX_PROG_TEXT_SZ); ASSERT_TRUE(match_pattern(btf, pattern, text, reg_map), pinfo->prog_kind); diff --git a/tools/testing/selftests/bpf/testing_helpers.c b/tools/testing/selftests/bpf/testing_helpers.c index d5379a0e6da8..ac7c66f4fc7b 100644 --- a/tools/testing/selftests/bpf/testing_helpers.c +++ b/tools/testing/selftests/bpf/testing_helpers.c @@ -7,6 +7,7 @@ #include #include #include +#include "disasm.h" #include "test_progs.h" #include "testing_helpers.h" #include From patchwork Sat Jun 29 09:47:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eduard Zingerman X-Patchwork-Id: 13716882 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-oa1-f47.google.com (mail-oa1-f47.google.com [209.85.160.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EF50D3F8C7 for ; Sat, 29 Jun 2024 09:48:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719654493; cv=none; b=pD+lHziA31bQsHVEL3lBtdGaEW94vQxM4XYzfwHQAYWFirTqpF5eH0kZEXJ+FUA35SPaT4vEAUm9BP5sM2Z6ERyjuVQaknm0TWeiVv7NNtkVRIaeTdhatPQuVjWn3WE73UrHE/6TFcS1GnG9NY+w/XWRQ6VSbjSXHNweO1HIXuE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719654493; c=relaxed/simple; bh=Q2Hf+dtKFKxCHAoFjupuQq6hLoNi/M+gDEEPRFrP3U4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dulR5/eo1JbgRlzxmQcXEahdUYdml3ZTNof/WR/ldkZQbxfFI6Xel/8JPU5ahBOky/Ikj/8TvDefojROhE2w2nPKCx7HiMZNy15baIVvQYGzGSG8X2pTHgGgqX8Zj14xf3SojhAFNko7hnt9X8o3XCYiIllNNpLzrLvzFttKm94= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=eebtCxGk; arc=none smtp.client-ip=209.85.160.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="eebtCxGk" Received: by mail-oa1-f47.google.com with SMTP id 586e51a60fabf-25982aa59efso683663fac.3 for ; Sat, 29 Jun 2024 02:48:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1719654491; x=1720259291; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rzJ32pOW9CYn6Orr21OKsQ8Oe31NgrzxMqAqSZGNdng=; b=eebtCxGkd6aLNN+yuX7zVjD+50QcOep6cq43SqlTYadDmCaoizZ9VFV2LmE2VktZJN 4zvRKwS6oGmtYwbykszLrv7Rli6sw0tlB66wV9NSFGxhIQS3ncU09k4b6s81Z2JGMW/v 0YmibnUe0FoBtLzOFvUtMoq2q64C1UOQJyZvV4PK1qIdkeP5cD91AGYGcYqXU/NhAfCw vLDOP6/ZK4Khav/uueqPB5dhpNsMsVraks3oliN+GlZuzHaZy9K+9mUgRECLUByXjsju soU3aM4li5nJyuZeeJik+0/m3z7VGaXWNVmBvhQMNR6rQ2PEz7X7VOkKXvSOeTB/ZyCr CMqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719654491; x=1720259291; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rzJ32pOW9CYn6Orr21OKsQ8Oe31NgrzxMqAqSZGNdng=; b=l32J5qETrHXBfaGfZaZ99sBG6+z3wIz+QY5kYWVgTqOiSVM7p8qVcU+R9Pr4DT+AK4 XBQnj6tFKzcgQkN8eGxdce7/uoMcxzxcPBncjCbWYcntDHR/nRuRICyIzkgkeP7bzYzl NfDZhYYKBK4PoZ9Bk5370Tnu4kcQmlVtCgFn30jcgmqGfssYEPpBF8/jW2Md6lStPJmU /6FqP3P6oNbuHWQfQtVAYVwXbhxzaLDn7ZlOGcaiLm3Ra7wDyj2UHs2IijcaHlkvc5H7 giTr4Ax7xyslN35H5gjcV8XPSq7YcTxN6WFKCBZJkkO9HC/UYOsg5UHicnOgHCvYEr0d 2Shw== X-Gm-Message-State: AOJu0YzHlTaQeLHmz2WmksGGrQz+C2kQS3PX5OENSO2FwzyetKBJvepP ITrSLD9B3pXO1cOdmR4eknMoYveZj3vyxwkGwdJKGm9Mve4kiqP0kvyIkg== X-Google-Smtp-Source: AGHT+IEUCCgJ1lulHD7x2QV4JXvmWvg35n2hUbqHnkw4A1x/acftH8pPSJo6LALXDP9nvxHK/BnQZw== X-Received: by 2002:a05:6870:b685:b0:23d:225a:9443 with SMTP id 586e51a60fabf-25db3592066mr476237fac.41.1719654490698; Sat, 29 Jun 2024 02:48:10 -0700 (PDT) Received: from badger.. ([38.34.87.7]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70804989f5asm2948932b3a.195.2024.06.29.02.48.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 29 Jun 2024 02:48:10 -0700 (PDT) From: Eduard Zingerman To: bpf@vger.kernel.org, ast@kernel.org Cc: andrii@kernel.org, daniel@iogearbox.net, martin.lau@linux.dev, kernel-team@fb.com, yonghong.song@linux.dev, jose.marchesi@oracle.com, Eduard Zingerman Subject: [RFC bpf-next v1 5/8] selftests/bpf: no need to track next_match_pos in struct test_loader Date: Sat, 29 Jun 2024 02:47:30 -0700 Message-ID: <20240629094733.3863850-6-eddyz87@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240629094733.3863850-1-eddyz87@gmail.com> References: <20240629094733.3863850-1-eddyz87@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC The call stack for validate_case() function looks as follows: - test_loader__run_subtests() - process_subtest() - run_subtest() - prepare_case(), which does 'tester->next_match_pos = 0'; - validate_case(), which increments tester->next_match_pos. Hence, each subtest is run with next_match_pos freshly set to zero. Meaning that there is no need to persist this variable in the struct test_loader, use local variable instead. Signed-off-by: Eduard Zingerman Acked-by: Andrii Nakryiko --- tools/testing/selftests/bpf/test_loader.c | 17 ++++++++--------- tools/testing/selftests/bpf/test_progs.h | 1 - 2 files changed, 8 insertions(+), 10 deletions(-) diff --git a/tools/testing/selftests/bpf/test_loader.c b/tools/testing/selftests/bpf/test_loader.c index f14e10b0de96..ac9d3e81abdb 100644 --- a/tools/testing/selftests/bpf/test_loader.c +++ b/tools/testing/selftests/bpf/test_loader.c @@ -434,7 +434,6 @@ static void prepare_case(struct test_loader *tester, bpf_program__set_flags(prog, prog_flags | spec->prog_flags); tester->log_buf[0] = '\0'; - tester->next_match_pos = 0; } static void emit_verifier_log(const char *log_buf, bool force) @@ -450,23 +449,23 @@ static void validate_case(struct test_loader *tester, struct bpf_program *prog, int load_err) { - int i, j, err; - char *match; regmatch_t reg_match[1]; + const char *match; + const char *log = tester->log_buf; + int i, j, err; for (i = 0; i < subspec->expect_msg_cnt; i++) { struct expect_msg *msg = &subspec->expect_msgs[i]; if (msg->substr) { - match = strstr(tester->log_buf + tester->next_match_pos, msg->substr); + match = strstr(log, msg->substr); if (match) - tester->next_match_pos = match - tester->log_buf + strlen(msg->substr); + log += strlen(msg->substr); } else { - err = regexec(&msg->regex, - tester->log_buf + tester->next_match_pos, 1, reg_match, 0); + err = regexec(&msg->regex, log, 1, reg_match, 0); if (err == 0) { - match = tester->log_buf + tester->next_match_pos + reg_match[0].rm_so; - tester->next_match_pos += reg_match[0].rm_eo; + match = log + reg_match[0].rm_so; + log += reg_match[0].rm_eo; } else { match = NULL; } diff --git a/tools/testing/selftests/bpf/test_progs.h b/tools/testing/selftests/bpf/test_progs.h index 0ba5a20b19ba..8e997de596db 100644 --- a/tools/testing/selftests/bpf/test_progs.h +++ b/tools/testing/selftests/bpf/test_progs.h @@ -438,7 +438,6 @@ typedef int (*pre_execution_cb)(struct bpf_object *obj); struct test_loader { char *log_buf; size_t log_buf_sz; - size_t next_match_pos; pre_execution_cb pre_execution_cb; struct bpf_object *obj; From patchwork Sat Jun 29 09:47:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eduard Zingerman X-Patchwork-Id: 13716883 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-oi1-f182.google.com (mail-oi1-f182.google.com [209.85.167.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C03FF39AFD for ; Sat, 29 Jun 2024 09:48:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719654494; cv=none; b=bH7IpvnrBaMXdJFVaUdv5JbK/QJ810xUXUBzEuMHP2BXDh/N320zvEbxbRx/JXgRKvt2AqF4tAufsdsB6DCX90EQ58LkkiHk7Yw9jGRIRMy98+9iBeAReZhvbiQFGhhrvWRsTmILrJgiirM0Cin2PJugit60RMuJkD6FhydbfDk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719654494; c=relaxed/simple; bh=yCZt+XVcSAr7lva+2GhMDW87L+I3GClfxPnsLYKLnrY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=IxaJNELpvEbrgGDv7cCGdexXdDFKC7l/ybWQdThaNgbIUpb/geXMKNIFuhI0IaMjyvO8KwFlcEpWvNq3oRaV+FqjDfqynWYbp2Q93jPIo1oPydYl/Zc2svldOn5ksCOIVImIaecIBBSYEZH/0ErcMEkYFI5t+SAN0bJAKs8Gnuc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Zfmxkhyn; arc=none smtp.client-ip=209.85.167.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Zfmxkhyn" Received: by mail-oi1-f182.google.com with SMTP id 5614622812f47-3d5666a4860so792818b6e.2 for ; Sat, 29 Jun 2024 02:48:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1719654492; x=1720259292; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KzejdGYduNITsXseCl4pjy4KNIRrUs1ao04Ddv+xHA8=; b=Zfmxkhyn43zsOBZOEr+KcHY753fRDWYLDmGughNanxMnNr8deUHeXzHwN3PmsHVyGz ZO5omXyXdgVfyodxM+PK+RVW9a2GF3q/Fx88/gnmedVVkP47OfO8iuVrpbjzChRHihpC jF4F+Mny5htyMv4aDf9t1BA3wiao6Bo7RGed+tQ6lg0D0zd/toHmcVHeYJNuEslvEoIm G4+CTRR+VbJzCRaCatR6Nww0Mbyn4LX+yAaE6F8IHpJqnqUup6gtlUk4pn2jFd117rYR VqxeIPfojjcNvY3tuuDX3Wl8prlCgOz4FDF7JMELjxN6QWaTbWkfbuVjrittNxx5G9vT H0Cw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719654492; x=1720259292; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KzejdGYduNITsXseCl4pjy4KNIRrUs1ao04Ddv+xHA8=; b=C6DAzUt+CFB8Se4RhbhkKxJNcI0vncBNCdXhbUx5pgFw9DPierfRJ7UxGUI7LallrE jNIAXs1DexEyYzM4Mz6WAmnb0dhoLPkdGCo1Cu+itohFa0eg83vAofsQfa1+95bY5Ajg GhMtz3hbWmYGGM286Gv0WFFquAiJmu3VmbimlVqnK23SfEOYbmjQ3IbLsXZPreCkLK3u A9nZKtVWFizDebjNK/LSE91X0wis8EpfXTmzhY4/mpigqsEosimPnkyGTCbgpkXm5S4H srtDz5BqdMWg3daXmLPYC6u1PV8jeJ3nWsB8u9pZgs+lsim0Gt4+WlGANU75uQ6VHB2v nkHg== X-Gm-Message-State: AOJu0YyM6DbPPCoUyC4UaJLN6h2Mju1PM7H9IIr22GacgTS/jGf/GKKs DehFheYFeXoTHhwJwvBbiilsBvOIoINI4AqoIR/rAgRuTMO/SxRej6hF1Q== X-Google-Smtp-Source: AGHT+IFNl2/wIr6iPus6I3mEapjgQi/KBQVDif3/F0wWl+ZeSHd4CoaDjdhLqBn+HtXybqDdLlNkSA== X-Received: by 2002:a05:6808:1925:b0:3d2:1a21:782 with SMTP id 5614622812f47-3d6b5499d99mr730774b6e.56.1719654491646; Sat, 29 Jun 2024 02:48:11 -0700 (PDT) Received: from badger.. ([38.34.87.7]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70804989f5asm2948932b3a.195.2024.06.29.02.48.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 29 Jun 2024 02:48:11 -0700 (PDT) From: Eduard Zingerman To: bpf@vger.kernel.org, ast@kernel.org Cc: andrii@kernel.org, daniel@iogearbox.net, martin.lau@linux.dev, kernel-team@fb.com, yonghong.song@linux.dev, jose.marchesi@oracle.com, Eduard Zingerman Subject: [RFC bpf-next v1 6/8] selftests/bpf: extract test_loader->expect_msgs as a data structure Date: Sat, 29 Jun 2024 02:47:31 -0700 Message-ID: <20240629094733.3863850-7-eddyz87@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240629094733.3863850-1-eddyz87@gmail.com> References: <20240629094733.3863850-1-eddyz87@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Non-functional change: use a separate data structure to represented expected messages in test_loader. This would allow to use the same functionality for expected set of disassembled instructions in the follow-up commit. Signed-off-by: Eduard Zingerman Acked-by: Andrii Nakryiko --- tools/testing/selftests/bpf/test_loader.c | 81 ++++++++++++----------- 1 file changed, 41 insertions(+), 40 deletions(-) diff --git a/tools/testing/selftests/bpf/test_loader.c b/tools/testing/selftests/bpf/test_loader.c index ac9d3e81abdb..d4bb68685ba5 100644 --- a/tools/testing/selftests/bpf/test_loader.c +++ b/tools/testing/selftests/bpf/test_loader.c @@ -55,11 +55,15 @@ struct expect_msg { regex_t regex; }; +struct msgs { + struct expect_msg *patterns; + size_t cnt; +}; + struct test_subspec { char *name; bool expect_failure; - struct expect_msg *expect_msgs; - size_t expect_msg_cnt; + struct msgs expect_msgs; int retval; bool execute; }; @@ -96,44 +100,45 @@ void test_loader_fini(struct test_loader *tester) free(tester->log_buf); } -static void free_test_spec(struct test_spec *spec) +static void free_msgs(struct msgs *msgs) { int i; + for (i = 0; i < msgs->cnt; i++) + if (msgs->patterns[i].regex_str) + regfree(&msgs->patterns[i].regex); + free(msgs->patterns); + msgs->patterns = NULL; + msgs->cnt = 0; +} + +static void free_test_spec(struct test_spec *spec) +{ /* Deallocate expect_msgs arrays. */ - for (i = 0; i < spec->priv.expect_msg_cnt; i++) - if (spec->priv.expect_msgs[i].regex_str) - regfree(&spec->priv.expect_msgs[i].regex); - for (i = 0; i < spec->unpriv.expect_msg_cnt; i++) - if (spec->unpriv.expect_msgs[i].regex_str) - regfree(&spec->unpriv.expect_msgs[i].regex); + free_msgs(&spec->priv.expect_msgs); + free_msgs(&spec->unpriv.expect_msgs); free(spec->priv.name); free(spec->unpriv.name); - free(spec->priv.expect_msgs); - free(spec->unpriv.expect_msgs); - spec->priv.name = NULL; spec->unpriv.name = NULL; - spec->priv.expect_msgs = NULL; - spec->unpriv.expect_msgs = NULL; } -static int push_msg(const char *substr, const char *regex_str, struct test_subspec *subspec) +static int push_msg(const char *substr, const char *regex_str, struct msgs *msgs) { void *tmp; int regcomp_res; char error_msg[100]; struct expect_msg *msg; - tmp = realloc(subspec->expect_msgs, - (1 + subspec->expect_msg_cnt) * sizeof(struct expect_msg)); + tmp = realloc(msgs->patterns, + (1 + msgs->cnt) * sizeof(struct expect_msg)); if (!tmp) { ASSERT_FAIL("failed to realloc memory for messages\n"); return -ENOMEM; } - subspec->expect_msgs = tmp; - msg = &subspec->expect_msgs[subspec->expect_msg_cnt]; + msgs->patterns = tmp; + msg = &msgs->patterns[msgs->cnt]; if (substr) { msg->substr = substr; @@ -150,7 +155,7 @@ static int push_msg(const char *substr, const char *regex_str, struct test_subsp } } - subspec->expect_msg_cnt += 1; + msgs->cnt += 1; return 0; } @@ -272,25 +277,25 @@ static int parse_test_spec(struct test_loader *tester, spec->mode_mask |= UNPRIV; } else if (str_has_pfx(s, TEST_TAG_EXPECT_MSG_PFX)) { msg = s + sizeof(TEST_TAG_EXPECT_MSG_PFX) - 1; - err = push_msg(msg, NULL, &spec->priv); + err = push_msg(msg, NULL, &spec->priv.expect_msgs); if (err) goto cleanup; spec->mode_mask |= PRIV; } else if (str_has_pfx(s, TEST_TAG_EXPECT_MSG_PFX_UNPRIV)) { msg = s + sizeof(TEST_TAG_EXPECT_MSG_PFX_UNPRIV) - 1; - err = push_msg(msg, NULL, &spec->unpriv); + err = push_msg(msg, NULL, &spec->unpriv.expect_msgs); if (err) goto cleanup; spec->mode_mask |= UNPRIV; } else if (str_has_pfx(s, TEST_TAG_EXPECT_REGEX_PFX)) { msg = s + sizeof(TEST_TAG_EXPECT_REGEX_PFX) - 1; - err = push_msg(NULL, msg, &spec->priv); + err = push_msg(NULL, msg, &spec->priv.expect_msgs); if (err) goto cleanup; spec->mode_mask |= PRIV; } else if (str_has_pfx(s, TEST_TAG_EXPECT_REGEX_PFX_UNPRIV)) { msg = s + sizeof(TEST_TAG_EXPECT_REGEX_PFX_UNPRIV) - 1; - err = push_msg(NULL, msg, &spec->unpriv); + err = push_msg(NULL, msg, &spec->unpriv.expect_msgs); if (err) goto cleanup; spec->mode_mask |= UNPRIV; @@ -387,11 +392,12 @@ static int parse_test_spec(struct test_loader *tester, spec->unpriv.execute = spec->priv.execute; } - if (!spec->unpriv.expect_msgs) { - for (i = 0; i < spec->priv.expect_msg_cnt; i++) { - struct expect_msg *msg = &spec->priv.expect_msgs[i]; + if (spec->unpriv.expect_msgs.cnt == 0) { + for (i = 0; i < spec->priv.expect_msgs.cnt; i++) { + struct expect_msg *msg = &spec->priv.expect_msgs.patterns[i]; - err = push_msg(msg->substr, msg->regex_str, &spec->unpriv); + err = push_msg(msg->substr, msg->regex_str, + &spec->unpriv.expect_msgs); if (err) goto cleanup; } @@ -443,19 +449,15 @@ static void emit_verifier_log(const char *log_buf, bool force) fprintf(stdout, "VERIFIER LOG:\n=============\n%s=============\n", log_buf); } -static void validate_case(struct test_loader *tester, - struct test_subspec *subspec, - struct bpf_object *obj, - struct bpf_program *prog, - int load_err) +static void validate_msgs(char *log_buf, struct msgs *msgs) { regmatch_t reg_match[1]; const char *match; - const char *log = tester->log_buf; + const char *log = log_buf; int i, j, err; - for (i = 0; i < subspec->expect_msg_cnt; i++) { - struct expect_msg *msg = &subspec->expect_msgs[i]; + for (i = 0; i < msgs->cnt; i++) { + struct expect_msg *msg = &msgs->patterns[i]; if (msg->substr) { match = strstr(log, msg->substr); @@ -473,9 +475,9 @@ static void validate_case(struct test_loader *tester, if (!ASSERT_OK_PTR(match, "expect_msg")) { if (env.verbosity == VERBOSE_NONE) - emit_verifier_log(tester->log_buf, true /*force*/); + emit_verifier_log(log_buf, true /*force*/); for (j = 0; j <= i; j++) { - msg = &subspec->expect_msgs[j]; + msg = &msgs->patterns[j]; fprintf(stderr, "%s %s: '%s'\n", j < i ? "MATCHED " : "EXPECTED", msg->substr ? "SUBSTR" : " REGEX", @@ -694,9 +696,8 @@ void run_subtest(struct test_loader *tester, goto tobj_cleanup; } } - emit_verifier_log(tester->log_buf, false /*force*/); - validate_case(tester, subspec, tobj, tprog, err); + validate_msgs(tester->log_buf, &subspec->expect_msgs); if (should_do_test_run(spec, subspec)) { /* For some reason test_verifier executes programs From patchwork Sat Jun 29 09:47:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eduard Zingerman X-Patchwork-Id: 13716884 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-oi1-f169.google.com (mail-oi1-f169.google.com [209.85.167.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BB4C940858 for ; Sat, 29 Jun 2024 09:48:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719654495; cv=none; b=jTgGc3KVQvjHwTpjykTmnmPscPto/+V3zBP59ZGg1WEwoqP9GH88mtwlcAdWvEsjbD/QbqctddX2aZi9INyV0/CEN5f+KKgdvf0O8DBaDi3gE7kqFnkk2nWY0RjTAJHGev6bL6ezCUu/ryfaiCdqIrGvq7v+O2wBX1od98K7ED8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719654495; c=relaxed/simple; bh=dUme/273h2bYDL94/PNwDiEcNu1qMYHju32TPFHb0jI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=e3+xQs05iBTyOgmKoSh8/sh3sOV0a2eFKSqPMhW5WqFWKI29/DVJVeM2G7hNXeAE94R/qOlqLwxb0hx5pTPFA5DrNB+FS/axZpwLGMEf4KegEf1yh+jTs7LyMs1ovJBLmzl29cvrmeD4gxV1wPCkQ3+yTp7nV9mrvHE7JxGocHM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=iZNbZhcI; arc=none smtp.client-ip=209.85.167.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="iZNbZhcI" Received: by mail-oi1-f169.google.com with SMTP id 5614622812f47-3d566d5eda9so730995b6e.0 for ; Sat, 29 Jun 2024 02:48:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1719654493; x=1720259293; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QHAnBbmnrS9ddngMLWtcJO0VwQYCAVjSLoLQMRLwmMU=; b=iZNbZhcI6dlU6WZAx9xBsusnpdcPpHmePIW8uSDZEc3vNhLC7pWbTJOXDAid0HrKCY cPo0KwIhTnI/y8Cnu6Js6hqSu994OWMeX7klKIZxvZ032C+iT/SzVbBIa/ldgcd7h06C mBv7YbAhhbYYdeUdWuaZIEV+im5iZtY/4lvbEo1/w84aLYdC1f4u0ZawmFNduQeuPme8 Cjn5jfbLl5RmQg5KhiL3phlVhF9uGSxiS4MUDKPVqniXKZFZQkQoDhbdMXozxCq1kdUq /k0PEO5Bk71ulo3oYX35dInqOuMH/JlERdxpseIf9FecjHRNS6UnKs+Pb7/QYAa6nl7a 4Pvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719654493; x=1720259293; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QHAnBbmnrS9ddngMLWtcJO0VwQYCAVjSLoLQMRLwmMU=; b=md+lE6T4efuZLdpm9V9vGwXXBtWF/+KNPNeJz45zW125HqDvW62jLQ4MHsci77PADl 5AZSt0AY++M01uL8BHMSrH/uCBAbOtjR8iYluuwzNOcHQHtNPBk3rHMKO6h1UCNy7nwt 82ufeolmFogrmdaMMGwslDfgBIRQlj86ymrk5Gd/YYAdWEohMYVvG5XebfP7Lrm7XoCN xKhKfeSTWPEeB/ohby2ilnmUVJKz/GHsCSqrjwn4HVB+4BpkJb1zKGj7YEfUI9pF6006 s70J2oHAJVPXwO2kx5SA6g98XCpH0CCdfvSFVznfk+Ho9apw5+7YMnoHCTb8w9Yh2mTt cu/g== X-Gm-Message-State: AOJu0Yxt3mCpk4EDVMgWw17aBTD1HbPQXrVREy5BwKqmFYFz3ICKurmo Mb3Z0x2ICv3HzlIJ2EpvwxqePV2J6jrHUU+/C0HJJLWqPNP8znqrRmKrTA== X-Google-Smtp-Source: AGHT+IFWEvIbklmFpsgiiqn6NU2EMaVSSexmVByBVUiymf+dAgpKmBOTWFwUHKzahSlzO2z1Vkb4ig== X-Received: by 2002:a05:6808:1789:b0:3d5:670f:baa5 with SMTP id 5614622812f47-3d6b4de2d0amr730205b6e.44.1719654492558; Sat, 29 Jun 2024 02:48:12 -0700 (PDT) Received: from badger.. ([38.34.87.7]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70804989f5asm2948932b3a.195.2024.06.29.02.48.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 29 Jun 2024 02:48:12 -0700 (PDT) From: Eduard Zingerman To: bpf@vger.kernel.org, ast@kernel.org Cc: andrii@kernel.org, daniel@iogearbox.net, martin.lau@linux.dev, kernel-team@fb.com, yonghong.song@linux.dev, jose.marchesi@oracle.com, Eduard Zingerman Subject: [RFC bpf-next v1 7/8] selftests/bpf: allow checking xlated programs in verifier_* tests Date: Sat, 29 Jun 2024 02:47:32 -0700 Message-ID: <20240629094733.3863850-8-eddyz87@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240629094733.3863850-1-eddyz87@gmail.com> References: <20240629094733.3863850-1-eddyz87@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Add a macro __xlated("...") for use with test_loader tests. When such annotations are present for the test case: - bpf_prog_get_info_by_fd() is used to get BPF program after all rewrites are applied by verifier. - the program is diassembled and patterns specified in __xlated are searched for in the disassembly text. __xlated matching follows the same mechanics as __msg: each subsequent pattern is matched from the point where previous pattern ended. This allows to write tests like below, where the goal is to verify the behavior of one of the of the transformations applied by verifier: SEC("raw_tp") __xlated("1: w0 = ") __xlated("2: r0 = &(void __percpu *)(r0)") __xlated("3: r0 = *(u32 *)(r0 +0)") __xlated("4: exit") __success __naked void simple(void) { asm volatile ( "call %[bpf_get_smp_processor_id];" "exit;" : : __imm(bpf_get_smp_processor_id) : __clobber_all); } Signed-off-by: Eduard Zingerman Acked-by: Andrii Nakryiko --- tools/testing/selftests/bpf/progs/bpf_misc.h | 6 ++ tools/testing/selftests/bpf/test_loader.c | 80 +++++++++++++++++++- 2 files changed, 83 insertions(+), 3 deletions(-) diff --git a/tools/testing/selftests/bpf/progs/bpf_misc.h b/tools/testing/selftests/bpf/progs/bpf_misc.h index 81097a3f15eb..fac131a23578 100644 --- a/tools/testing/selftests/bpf/progs/bpf_misc.h +++ b/tools/testing/selftests/bpf/progs/bpf_misc.h @@ -26,6 +26,9 @@ * * __regex Same as __msg, but using a regular expression. * __regex_unpriv Same as __msg_unpriv but using a regular expression. + * __xlated Expect a line in a disassembly log after verifier applies rewrites. + * Multiple __xlated attributes could be specified. + * __xlated_unpriv Same as __xlated but for unprivileged mode. * * __success Expect program load success in privileged mode. * __success_unpriv Expect program load success in unprivileged mode. @@ -63,11 +66,14 @@ */ #define __msg(msg) __attribute__((btf_decl_tag("comment:test_expect_msg=" msg))) #define __regex(regex) __attribute__((btf_decl_tag("comment:test_expect_regex=" regex))) +#define __xlated(msg) __attribute__((btf_decl_tag("comment:test_expect_xlated=" msg))) #define __failure __attribute__((btf_decl_tag("comment:test_expect_failure"))) #define __success __attribute__((btf_decl_tag("comment:test_expect_success"))) #define __description(desc) __attribute__((btf_decl_tag("comment:test_description=" desc))) #define __msg_unpriv(msg) __attribute__((btf_decl_tag("comment:test_expect_msg_unpriv=" msg))) #define __regex_unpriv(regex) __attribute__((btf_decl_tag("comment:test_expect_regex_unpriv=" regex))) +#define __xlated_unpriv(msg) \ + __attribute__((btf_decl_tag("comment:test_expect_xlated_unpriv=" msg))) #define __failure_unpriv __attribute__((btf_decl_tag("comment:test_expect_failure_unpriv"))) #define __success_unpriv __attribute__((btf_decl_tag("comment:test_expect_success_unpriv"))) #define __log_level(lvl) __attribute__((btf_decl_tag("comment:test_log_level="#lvl))) diff --git a/tools/testing/selftests/bpf/test_loader.c b/tools/testing/selftests/bpf/test_loader.c index d4bb68685ba5..8e5f051801db 100644 --- a/tools/testing/selftests/bpf/test_loader.c +++ b/tools/testing/selftests/bpf/test_loader.c @@ -7,6 +7,7 @@ #include #include "autoconf_helper.h" +#include "disasm_helpers.h" #include "unpriv_helpers.h" #include "cap_helpers.h" @@ -19,10 +20,12 @@ #define TEST_TAG_EXPECT_SUCCESS "comment:test_expect_success" #define TEST_TAG_EXPECT_MSG_PFX "comment:test_expect_msg=" #define TEST_TAG_EXPECT_REGEX_PFX "comment:test_expect_regex=" +#define TEST_TAG_EXPECT_XLATED_PFX "comment:test_expect_xlated=" #define TEST_TAG_EXPECT_FAILURE_UNPRIV "comment:test_expect_failure_unpriv" #define TEST_TAG_EXPECT_SUCCESS_UNPRIV "comment:test_expect_success_unpriv" #define TEST_TAG_EXPECT_MSG_PFX_UNPRIV "comment:test_expect_msg_unpriv=" #define TEST_TAG_EXPECT_REGEX_PFX_UNPRIV "comment:test_expect_regex_unpriv=" +#define TEST_TAG_EXPECT_XLATED_PFX_UNPRIV "comment:test_expect_xlated_unpriv=" #define TEST_TAG_LOG_LEVEL_PFX "comment:test_log_level=" #define TEST_TAG_PROG_FLAGS_PFX "comment:test_prog_flags=" #define TEST_TAG_DESCRIPTION_PFX "comment:test_description=" @@ -64,6 +67,7 @@ struct test_subspec { char *name; bool expect_failure; struct msgs expect_msgs; + struct msgs expect_xlated; int retval; bool execute; }; @@ -117,6 +121,8 @@ static void free_test_spec(struct test_spec *spec) /* Deallocate expect_msgs arrays. */ free_msgs(&spec->priv.expect_msgs); free_msgs(&spec->unpriv.expect_msgs); + free_msgs(&spec->priv.expect_xlated); + free_msgs(&spec->unpriv.expect_xlated); free(spec->priv.name); free(spec->unpriv.name); @@ -299,6 +305,18 @@ static int parse_test_spec(struct test_loader *tester, if (err) goto cleanup; spec->mode_mask |= UNPRIV; + } else if (str_has_pfx(s, TEST_TAG_EXPECT_XLATED_PFX)) { + msg = s + sizeof(TEST_TAG_EXPECT_XLATED_PFX) - 1; + err = push_msg(msg, NULL, &spec->priv.expect_xlated); + if (err) + goto cleanup; + spec->mode_mask |= PRIV; + } else if (str_has_pfx(s, TEST_TAG_EXPECT_XLATED_PFX_UNPRIV)) { + msg = s + sizeof(TEST_TAG_EXPECT_XLATED_PFX_UNPRIV) - 1; + err = push_msg(msg, NULL, &spec->unpriv.expect_xlated); + if (err) + goto cleanup; + spec->mode_mask |= UNPRIV; } else if (str_has_pfx(s, TEST_TAG_RETVAL_PFX)) { val = s + sizeof(TEST_TAG_RETVAL_PFX) - 1; err = parse_retval(val, &spec->priv.retval, "__retval"); @@ -402,6 +420,16 @@ static int parse_test_spec(struct test_loader *tester, goto cleanup; } } + if (spec->unpriv.expect_xlated.cnt == 0) { + for (i = 0; i < spec->priv.expect_xlated.cnt; i++) { + struct expect_msg *msg = &spec->priv.expect_xlated.patterns[i]; + + err = push_msg(msg->substr, msg->regex_str, + &spec->unpriv.expect_xlated); + if (err) + goto cleanup; + } + } } spec->valid = true; @@ -449,7 +477,15 @@ static void emit_verifier_log(const char *log_buf, bool force) fprintf(stdout, "VERIFIER LOG:\n=============\n%s=============\n", log_buf); } -static void validate_msgs(char *log_buf, struct msgs *msgs) +static void emit_xlated(const char *xlated, bool force) +{ + if (!force && env.verbosity == VERBOSE_NONE) + return; + fprintf(stdout, "XLATED:\n=============\n%s=============\n", xlated); +} + +static void validate_msgs(char *log_buf, struct msgs *msgs, + void (*emit_fn)(const char *buf, bool force)) { regmatch_t reg_match[1]; const char *match; @@ -475,7 +511,7 @@ static void validate_msgs(char *log_buf, struct msgs *msgs) if (!ASSERT_OK_PTR(match, "expect_msg")) { if (env.verbosity == VERBOSE_NONE) - emit_verifier_log(log_buf, true /*force*/); + emit_fn(log_buf, true /*force*/); for (j = 0; j <= i; j++) { msg = &msgs->patterns[j]; fprintf(stderr, "%s %s: '%s'\n", @@ -612,6 +648,35 @@ static bool should_do_test_run(struct test_spec *spec, struct test_subspec *subs return true; } +/* Get a disassembly of BPF program after verifier applies all rewrites */ +static int get_xlated_program_text(int prog_fd, char *text, size_t text_sz) +{ + __u32 insns_cnt = 0, i, insn_sz; + struct bpf_insn *insns = NULL; + char buf[64]; + FILE *out = NULL; + int err; + + err = get_xlated_program(prog_fd, &insns, &insns_cnt); + if (!ASSERT_OK(err, "get_xlated_program")) + goto out; + out = fmemopen(text, text_sz, "w"); + if (!ASSERT_OK_PTR(out, "open_memstream")) + goto out; + for (i = 0; i < insns_cnt;) { + insn_sz = disasm_insn(insns + i, buf, sizeof(buf)); + fprintf(out, "%d: %s\n", i, buf); + i += insn_sz; + } + fflush(out); + +out: + free(insns); + if (out) + fclose(out); + return err; +} + /* this function is forced noinline and has short generic name to look better * in test_progs output (in case of a failure) */ @@ -697,7 +762,16 @@ void run_subtest(struct test_loader *tester, } } emit_verifier_log(tester->log_buf, false /*force*/); - validate_msgs(tester->log_buf, &subspec->expect_msgs); + validate_msgs(tester->log_buf, &subspec->expect_msgs, emit_verifier_log); + + if (subspec->expect_xlated.cnt) { + err = get_xlated_program_text(bpf_program__fd(tprog), + tester->log_buf, tester->log_buf_sz); + if (err) + goto tobj_cleanup; + emit_xlated(tester->log_buf, false /*force*/); + validate_msgs(tester->log_buf, &subspec->expect_xlated, emit_xlated); + } if (should_do_test_run(spec, subspec)) { /* For some reason test_verifier executes programs From patchwork Sat Jun 29 09:47:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eduard Zingerman X-Patchwork-Id: 13716885 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-il1-f182.google.com (mail-il1-f182.google.com [209.85.166.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CAE4B41C67 for ; Sat, 29 Jun 2024 09:48:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719654496; cv=none; b=mmjdQGVKlLDlE9iRenPfzSr5njDdvxTGjPjYQ3Ld98RMXCobbHJSHIQMdTlUKStZ0bBUj7DTLZcr4oIIvPtuCUMKq/7hPVT84fp9ZF1/Ee1OLykcmbaP6tvF8+HYG+bELxsiVa3dXewY1t2VhnCTVrhOGX8dyWxkk6eK8q4G2lw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719654496; c=relaxed/simple; bh=9cS3JMvs1ouLlWFF2l5AwH8CGdOc7Scuk490FSU4RI8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=fLfLRxnWQ9yppQ6CCj7UZZ+iySkN5BUfsWFjgyjXD8YhRr1H0f8Ox6A5fO61+sNBZkDUm4g86ewPRa3DsmHL7hJUBcoHpSYXxETDI7VrZ6h+ZOU+Bb2la/htgm7bQbcs87HruOEW1iqUKLOkJ0IcHxTh+UciAbotGvPnjDMftWk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=erS07F0E; arc=none smtp.client-ip=209.85.166.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="erS07F0E" Received: by mail-il1-f182.google.com with SMTP id e9e14a558f8ab-376e6c5ce0eso5180775ab.1 for ; Sat, 29 Jun 2024 02:48:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1719654493; x=1720259293; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mQibrfHYTiGdeEU72XTGkGyEeLqNF+8UUhWKxDocg1k=; b=erS07F0E4BH7pgQ4MymfnrazLfPCtpcBEWWMB6BDV2BpwUgNWswmLSsHxfUKUS9lvJ zzfra3HAVPUg/bxeSBu1B1b02RXRJCRyOGBKinNp2lbv6omnw5m/nIM1tMjBNayIoeUW gDDxswi1O9vIkyaTnNRgKenxezMWq9drR5hlE+YbXmsl1mLn1+cl/ycXHmogAGVVkfgw GYKSm3t0mBwmSBA+ZJhmXaET2KkArmjUR9oT9jPHZQMHSoQN2/gv/ZvlNLuEpDuHqR5O dRX55ahLqp7rBXnvaW1oWxiCimTtyh+WBoR4fO2basBWk2hyngK1Vv/A5CGngTpJ9kS/ CWiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719654493; x=1720259293; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mQibrfHYTiGdeEU72XTGkGyEeLqNF+8UUhWKxDocg1k=; b=OLOcumOoud5cRbq0eXTA9LO8oQ7Oo/5lt5UhZACOB95mtI7gFdmDlhHdtYJTGDaYR9 QzN3Ik+W80hw7UVid8UCVk/NrmPODxXryapAkTS65VaAQmBQyD0GVWM/L4AZbCCwhNUT mrdNz/oBZka6tsWl1/oEsnIv4ovDbhgfitF8ukA67G+xkeEzU6IGtOtKgGGEFpiVdEub dt0xL0fJK9T8KxGgsgxpHoATMPdalQOXpR0IqwM5YIbMqVCFrKJ93rRftQEbGfMB6PhC sIU5yCozJGRZd0rRZD7Moaax58CoFozJsYUschISECc2QklnHrPNgjTqRSrquIJv0KHd Pohw== X-Gm-Message-State: AOJu0Yyo65DhtLjAl+U1FhR8fyP0AE9AJ19ThsTv/tRWARHkxiacIQx5 /Y63p3KtnmMQlnsO9KA134oEu0SprFfG+tL73AfTJdC2St5QiW/6nY6LYg== X-Google-Smtp-Source: AGHT+IENLa4vfQ4K4Bjs5fDaWUikrQFRRjYc9I7M1BPNSAR+ekyGoV0+bdfAohg75I0Lcm2pkMFN4Q== X-Received: by 2002:a05:6e02:1fe8:b0:374:aa60:a5c3 with SMTP id e9e14a558f8ab-37cd2bee935mr5634835ab.28.1719654493511; Sat, 29 Jun 2024 02:48:13 -0700 (PDT) Received: from badger.. ([38.34.87.7]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70804989f5asm2948932b3a.195.2024.06.29.02.48.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 29 Jun 2024 02:48:13 -0700 (PDT) From: Eduard Zingerman To: bpf@vger.kernel.org, ast@kernel.org Cc: andrii@kernel.org, daniel@iogearbox.net, martin.lau@linux.dev, kernel-team@fb.com, yonghong.song@linux.dev, jose.marchesi@oracle.com, Eduard Zingerman Subject: [RFC bpf-next v1 8/8] selftests/bpf: test no_caller_saved_registers spill/fill removal Date: Sat, 29 Jun 2024 02:47:33 -0700 Message-ID: <20240629094733.3863850-9-eddyz87@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240629094733.3863850-1-eddyz87@gmail.com> References: <20240629094733.3863850-1-eddyz87@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Tests for no_caller_saved_registers processing logic (see verifier.c:match_and_mark_nocsr_pattern()): - a canary positive test case; - various tests with broken patterns; - tests with read/write fixed/varying stack access that violate nocsr stack access contract; - tests with multiple subprograms. Signed-off-by: Eduard Zingerman --- .../selftests/bpf/prog_tests/verifier.c | 7 + .../selftests/bpf/progs/verifier_nocsr.c | 437 ++++++++++++++++++ 2 files changed, 444 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_nocsr.c diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c index 6816ff064516..8e056c36c549 100644 --- a/tools/testing/selftests/bpf/prog_tests/verifier.c +++ b/tools/testing/selftests/bpf/prog_tests/verifier.c @@ -53,6 +53,7 @@ #include "verifier_movsx.skel.h" #include "verifier_netfilter_ctx.skel.h" #include "verifier_netfilter_retcode.skel.h" +#include "verifier_nocsr.skel.h" #include "verifier_precision.skel.h" #include "verifier_prevent_map_lookup.skel.h" #include "verifier_raw_stack.skel.h" @@ -171,6 +172,12 @@ void test_verifier_meta_access(void) { RUN(verifier_meta_access); } void test_verifier_movsx(void) { RUN(verifier_movsx); } void test_verifier_netfilter_ctx(void) { RUN(verifier_netfilter_ctx); } void test_verifier_netfilter_retcode(void) { RUN(verifier_netfilter_retcode); } +void test_verifier_nocsr(void) +{ +#if defined(__x86_64__) + RUN(verifier_nocsr); +#endif /* __x86_64__ */ +} void test_verifier_precision(void) { RUN(verifier_precision); } void test_verifier_prevent_map_lookup(void) { RUN(verifier_prevent_map_lookup); } void test_verifier_raw_stack(void) { RUN(verifier_raw_stack); } diff --git a/tools/testing/selftests/bpf/progs/verifier_nocsr.c b/tools/testing/selftests/bpf/progs/verifier_nocsr.c new file mode 100644 index 000000000000..5ddc2c97ada6 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_nocsr.c @@ -0,0 +1,437 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include +#include "bpf_misc.h" + +#define __xlated_bpf_get_smp_processor_id \ + __xlated(": w0 = ") \ + __xlated(": r0 = &(void __percpu *)(r0)") \ + __xlated(": r0 = *(u32 *)(r0 +0)") + +SEC("raw_tp") +__xlated("4: r5 = 5") +__xlated_bpf_get_smp_processor_id +__xlated("8: exit") +__success +__naked void simple(void) +{ + asm volatile ( + "r1 = 1;" + "r2 = 2;" + "r3 = 3;" + "r4 = 4;" + "r5 = 5;" + "*(u64 *)(r10 - 16) = r1;" + "*(u64 *)(r10 - 24) = r2;" + "*(u64 *)(r10 - 32) = r3;" + "*(u64 *)(r10 - 40) = r4;" + "*(u64 *)(r10 - 48) = r5;" + "call %[bpf_get_smp_processor_id];" + "r5 = *(u64 *)(r10 - 48);" + "r4 = *(u64 *)(r10 - 40);" + "r3 = *(u64 *)(r10 - 32);" + "r2 = *(u64 *)(r10 - 24);" + "r1 = *(u64 *)(r10 - 16);" + "exit;" + : + : __imm(bpf_get_smp_processor_id) + : __clobber_all); +} + +SEC("raw_tp") +__xlated("1: *(u64 *)(r10 -16) = r1") +__xlated_bpf_get_smp_processor_id +__xlated("5: r2 = *(u64 *)(r10 -16)") +__success +__naked void wrong_reg_in_pattern1(void) +{ + asm volatile ( + "r1 = 1;" + "*(u64 *)(r10 - 16) = r1;" + "call %[bpf_get_smp_processor_id];" + "r2 = *(u64 *)(r10 - 16);" + "exit;" + : + : __imm(bpf_get_smp_processor_id) + : __clobber_all); +} + +SEC("raw_tp") +__xlated("1: *(u64 *)(r10 -16) = r6") +__xlated_bpf_get_smp_processor_id +__xlated("5: r6 = *(u64 *)(r10 -16)") +__success +__naked void wrong_reg_in_pattern2(void) +{ + asm volatile ( + "r6 = 1;" + "*(u64 *)(r10 - 16) = r6;" + "call %[bpf_get_smp_processor_id];" + "r6 = *(u64 *)(r10 - 16);" + "exit;" + : + : __imm(bpf_get_smp_processor_id) + : __clobber_all); +} + +SEC("raw_tp") +__xlated("1: *(u64 *)(r10 -16) = r0") +__xlated_bpf_get_smp_processor_id +__xlated("5: r0 = *(u64 *)(r10 -16)") +__success +__naked void wrong_reg_in_pattern3(void) +{ + asm volatile ( + "r0 = 1;" + "*(u64 *)(r10 - 16) = r0;" + "call %[bpf_get_smp_processor_id];" + "r0 = *(u64 *)(r10 - 16);" + "exit;" + : + : __imm(bpf_get_smp_processor_id) + : __clobber_all); +} + +SEC("raw_tp") +__xlated("2: *(u64 *)(r2 -16) = r1") +__xlated_bpf_get_smp_processor_id +__xlated("6: r1 = *(u64 *)(r10 -16)") +__success +__naked void wrong_base_in_pattern(void) +{ + asm volatile ( + "r1 = 1;" + "r2 = r10;" + "*(u64 *)(r2 - 16) = r1;" + "call %[bpf_get_smp_processor_id];" + "r1 = *(u64 *)(r10 - 16);" + "exit;" + : + : __imm(bpf_get_smp_processor_id) + : __clobber_all); +} + +SEC("raw_tp") +__xlated("1: *(u64 *)(r10 -16) = r1") +__xlated_bpf_get_smp_processor_id +__xlated("5: r2 = 1") +__success +__naked void wrong_insn_in_pattern(void) +{ + asm volatile ( + "r1 = 1;" + "*(u64 *)(r10 - 16) = r1;" + "call %[bpf_get_smp_processor_id];" + "r2 = 1;" + "r1 = *(u64 *)(r10 - 16);" + "exit;" + : + : __imm(bpf_get_smp_processor_id) + : __clobber_all); +} + +SEC("raw_tp") +__xlated("2: *(u64 *)(r10 -16) = r1") +__xlated_bpf_get_smp_processor_id +__xlated("6: r1 = *(u64 *)(r10 -8)") +__success +__naked void wrong_off_in_pattern(void) +{ + asm volatile ( + "r1 = 1;" + "*(u64 *)(r10 - 8) = r1;" + "*(u64 *)(r10 - 16) = r1;" + "call %[bpf_get_smp_processor_id];" + "r1 = *(u64 *)(r10 - 8);" + "exit;" + : + : __imm(bpf_get_smp_processor_id) + : __clobber_all); +} + +SEC("raw_tp") +__xlated("1: *(u32 *)(r10 -16) = r1") +__xlated_bpf_get_smp_processor_id +__xlated("5: r1 = *(u32 *)(r10 -16)") +__success +__naked void wrong_size_in_pattern(void) +{ + asm volatile ( + "r1 = 1;" + "*(u32 *)(r10 - 16) = r1;" + "call %[bpf_get_smp_processor_id];" + "r1 = *(u32 *)(r10 - 16);" + "exit;" + : + : __imm(bpf_get_smp_processor_id) + : __clobber_all); +} + +SEC("raw_tp") +__xlated("2: *(u32 *)(r10 -8) = r1") +__xlated_bpf_get_smp_processor_id +__xlated("6: r1 = *(u32 *)(r10 -8)") +__success +__naked void partial_pattern(void) +{ + asm volatile ( + "r1 = 1;" + "r2 = 2;" + "*(u32 *)(r10 - 8) = r1;" + "*(u64 *)(r10 - 16) = r2;" + "call %[bpf_get_smp_processor_id];" + "r2 = *(u64 *)(r10 - 16);" + "r1 = *(u32 *)(r10 - 8);" + "exit;" + : + : __imm(bpf_get_smp_processor_id) + : __clobber_all); +} + +SEC("raw_tp") +__xlated("0: r1 = 1") +__xlated("1: r2 = 2") +/* not patched, spills for -8, -16 not removed */ +__xlated("2: *(u64 *)(r10 -8) = r1") +__xlated("3: *(u64 *)(r10 -16) = r2") +__xlated_bpf_get_smp_processor_id +__xlated("7: r2 = *(u64 *)(r10 -16)") +__xlated("8: r1 = *(u64 *)(r10 -8)") +/* patched, spills for -16, -24 removed */ +__xlated_bpf_get_smp_processor_id +__xlated("12: exit") +__success +__naked void min_stack_offset(void) +{ + asm volatile ( + "r1 = 1;" + "r2 = 2;" + /* this call won't be patched */ + "*(u64 *)(r10 - 8) = r1;" + "*(u64 *)(r10 - 16) = r2;" + "call %[bpf_get_smp_processor_id];" + "r2 = *(u64 *)(r10 - 16);" + "r1 = *(u64 *)(r10 - 8);" + /* this call would be patched */ + "*(u64 *)(r10 - 16) = r1;" + "*(u64 *)(r10 - 24) = r2;" + "call %[bpf_get_smp_processor_id];" + "r2 = *(u64 *)(r10 - 24);" + "r1 = *(u64 *)(r10 - 16);" + "exit;" + : + : __imm(bpf_get_smp_processor_id) + : __clobber_all); +} + +SEC("raw_tp") +__xlated("1: *(u64 *)(r10 -8) = r1") +__xlated_bpf_get_smp_processor_id +__xlated("5: r1 = *(u64 *)(r10 -8)") +__success +__naked void bad_fixed_read(void) +{ + asm volatile ( + "r1 = 1;" + "*(u64 *)(r10 - 8) = r1;" + "call %[bpf_get_smp_processor_id];" + "r1 = *(u64 *)(r10 - 8);" + "r1 = r10;" + "r1 += -8;" + "r1 = *(u64 *)(r1 - 0);" + "exit;" + : + : __imm(bpf_get_smp_processor_id) + : __clobber_all); +} + +SEC("raw_tp") +__xlated("1: *(u64 *)(r10 -8) = r1") +__xlated_bpf_get_smp_processor_id +__xlated("5: r1 = *(u64 *)(r10 -8)") +__success +__naked void bad_fixed_write(void) +{ + asm volatile ( + "r1 = 1;" + "*(u64 *)(r10 - 8) = r1;" + "call %[bpf_get_smp_processor_id];" + "r1 = *(u64 *)(r10 - 8);" + "r1 = r10;" + "r1 += -8;" + "*(u64 *)(r1 - 0) = r1;" + "exit;" + : + : __imm(bpf_get_smp_processor_id) + : __clobber_all); +} + +SEC("raw_tp") +__xlated("6: *(u64 *)(r10 -16) = r1") +__xlated_bpf_get_smp_processor_id +__xlated("10: r1 = *(u64 *)(r10 -16)") +__success +__naked void bad_varying_read(void) +{ + asm volatile ( + "r6 = *(u64 *)(r1 + 0);" /* random scalar value */ + "r6 &= 0x7;" /* r6 range [0..7] */ + "r6 += 0x2;" /* r6 range [2..9] */ + "r7 = 0;" + "r7 -= r6;" /* r7 range [-9..-2] */ + "r1 = 1;" + "*(u64 *)(r10 - 16) = r1;" + "call %[bpf_get_smp_processor_id];" + "r1 = *(u64 *)(r10 - 16);" + "r1 = r10;" + "r1 += r7;" + "r1 = *(u8 *)(r1 - 0);" /* touches slot [-16..-9] where spills are stored */ + "exit;" + : + : __imm(bpf_get_smp_processor_id) + : __clobber_all); +} + +SEC("raw_tp") +__xlated("6: *(u64 *)(r10 -16) = r1") +__xlated_bpf_get_smp_processor_id +__xlated("10: r1 = *(u64 *)(r10 -16)") +__success +__naked void bad_varying_write(void) +{ + asm volatile ( + "r6 = *(u64 *)(r1 + 0);" /* random scalar value */ + "r6 &= 0x7;" /* r6 range [0..7] */ + "r6 += 0x2;" /* r6 range [2..9] */ + "r7 = 0;" + "r7 -= r6;" /* r7 range [-9..-2] */ + "r1 = 1;" + "*(u64 *)(r10 - 16) = r1;" + "call %[bpf_get_smp_processor_id];" + "r1 = *(u64 *)(r10 - 16);" + "r1 = r10;" + "r1 += r7;" + "*(u8 *)(r1 - 0) = r7;" /* touches slot [-16..-9] where spills are stored */ + "exit;" + : + : __imm(bpf_get_smp_processor_id) + : __clobber_all); +} + +SEC("raw_tp") +__xlated("1: *(u64 *)(r10 -8) = r1") +__xlated_bpf_get_smp_processor_id +__xlated("5: r1 = *(u64 *)(r10 -8)") +__success +__naked void bad_write_in_subprog(void) +{ + asm volatile ( + "r1 = 1;" + "*(u64 *)(r10 - 8) = r1;" + "call %[bpf_get_smp_processor_id];" + "r1 = *(u64 *)(r10 - 8);" + "r1 = r10;" + "r1 += -8;" + "call bad_write_in_subprog_aux;" + "exit;" + : + : __imm(bpf_get_smp_processor_id) + : __clobber_all); +} + +__used +__naked static void bad_write_in_subprog_aux(void) +{ + asm volatile ( + "r0 = 1;" + "*(u64 *)(r1 - 0) = r0;" /* invalidates nocsr contract for caller: */ + "exit;" /* caller stack at -8 used outside of the pattern */ + ::: __clobber_all); +} + +SEC("raw_tp") +/* main, not patched */ +__xlated("1: *(u64 *)(r10 -8) = r1") +__xlated_bpf_get_smp_processor_id +__xlated("5: r1 = *(u64 *)(r10 -8)") +__xlated("9: call pc+1") +__xlated("10: exit") +/* subprogram, patched */ +__xlated("11: r1 = 1") +__xlated_bpf_get_smp_processor_id +__xlated("15: exit") +__success +__naked void invalidate_one_subprog(void) +{ + asm volatile ( + "r1 = 1;" + "*(u64 *)(r10 - 8) = r1;" + "call %[bpf_get_smp_processor_id];" + "r1 = *(u64 *)(r10 - 8);" + "r1 = r10;" + "r1 += -8;" + "r1 = *(u64 *)(r1 - 0);" + "call invalidate_one_subprog_aux;" + "exit;" + : + : __imm(bpf_get_smp_processor_id) + : __clobber_all); +} + +__used +__naked static void invalidate_one_subprog_aux(void) +{ + asm volatile ( + "r1 = 1;" + "*(u64 *)(r10 - 8) = r1;" + "call %[bpf_get_smp_processor_id];" + "r1 = *(u64 *)(r10 - 8);" + "exit;" + : + : __imm(bpf_get_smp_processor_id) + : __clobber_all); +} + +SEC("raw_tp") +/* main */ +__xlated("0: r1 = 1") +__xlated_bpf_get_smp_processor_id +__xlated("4: call pc+1") +__xlated("5: exit") +/* subprogram */ +__xlated("6: r1 = 1") +__xlated_bpf_get_smp_processor_id +__xlated("10: *(u64 *)(r10 -16) = r1") +__xlated("11: exit") +__success +__naked void subprogs_use_independent_offsets(void) +{ + asm volatile ( + "r1 = 1;" + "*(u64 *)(r10 - 16) = r1;" + "call %[bpf_get_smp_processor_id];" + "r1 = *(u64 *)(r10 - 16);" + "call subprogs_use_independent_offsets_aux;" + "exit;" + : + : __imm(bpf_get_smp_processor_id) + : __clobber_all); +} + +__used +__naked static void subprogs_use_independent_offsets_aux(void) +{ + asm volatile ( + "r1 = 1;" + "*(u64 *)(r10 - 24) = r1;" + "call %[bpf_get_smp_processor_id];" + "r1 = *(u64 *)(r10 - 24);" + "*(u64 *)(r10 - 16) = r1;" + "exit;" + : + : __imm(bpf_get_smp_processor_id) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL";