From patchwork Fri Mar 29 09:46:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Uros Bizjak X-Patchwork-Id: 13610501 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-ed1-f48.google.com (mail-ed1-f48.google.com [209.85.208.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 657D33BBDB; Fri, 29 Mar 2024 09:49:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711705768; cv=none; b=VGWWXc/zg7h8SKjkJTwAiiIEjciBwndWPkPGSXjoUCi/GibVDAaAj+Lu+IuMRkykQ1Ijes/LnhSZz/FxIeV3+nHTw+ssGBq7g1pR7XhGEl6Fpvqb21R74xWpGpZDV3qjwJwQPb1g53VmkbVluULdTYpt+KKwJ6SbvYO1+2oQg18= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711705768; c=relaxed/simple; bh=1c0GfFEPUmmfZMeB83civMHb91mX5T3u/YH4/WLOdpY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=dnVRmh52G5WzZFGMijW+PjJ4PNpSW4MraytlU4Y8dPyKiyQIQo94ElLzyJNmPatOYYDF2/WWUehbPF4j2VQpgnDx3ASdhgSOL0VT6RSXAUoNbrwo0FkK6qco7zo2uUDBLw4DiowX3u7EeC3VT9B0zAgyhW/kYR89/nCug/w3YMA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=msykGo5F; arc=none smtp.client-ip=209.85.208.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="msykGo5F" Received: by mail-ed1-f48.google.com with SMTP id 4fb4d7f45d1cf-56c4ca65dc6so2221190a12.1; Fri, 29 Mar 2024 02:49:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711705765; x=1712310565; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+53nVtKSv5Jk7rKbaYFDJuz2r5jEU1fs9QHE79j+A6U=; b=msykGo5FCZaEasP28aBRd1Nx19ugBI/pwIPMN9qbtca0GUlQg1CQqlTqCPYAiQSMnZ Oi95LoFhQCyrz7IqBEaBJ/u3rbsn/e5mmj9ayH8DfAT8rCing8rKDaR+7DrTAOs87Mdy CWMXHXRB1TiK9oTIHE7vxVpZuxSHFkh11YpT4VO4LHyCHzOxdyrMWdMWSvMQcavOimdB yPDaqdZY8W6wn0K65zSOyl7tkOuv34O3BsT/LJtNx8eSNr6IyM0Un9MGoP95nROxxCRk nmKQUqjZLHyQcxi/0j+g5d6yYLPYxXRtmNksDMpNErt1SVmv6CzSGhLx64xWNs/Q4Pwd mEpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711705765; x=1712310565; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+53nVtKSv5Jk7rKbaYFDJuz2r5jEU1fs9QHE79j+A6U=; b=n/9JMR5/TgZrfIv83ALUHFVz5uqHMPj5YMwsctob2DFGaaYYbhX5sE2u6ppoTaf8aE 2wWV+cgJCj/RcPhcQRx42E880S9x8vg/7ilS6QqU2kKOCwKpchETJ8NbNVRP5GAPFwpH SQ92qny60HL3G+zX9tKApZ/RbzfRizHEOL7ykvlA+fBFN9BHLJgcotkfel+/9ydqhEgw EpPlmlmrh0w6YAFlwTbhBULPcWaGXGjyXtND4dn5Fy81tuUl0/olRV821TF7pmJgt1g2 +ZIzzlKcf4O7AsPVJgn9aoKP71L1qwQpE833fcYJIor7kFDM+Tr4GtdMzH3hxrfmBXfO 42ow== X-Forwarded-Encrypted: i=1; AJvYcCWKyfF9ubzs/Wh6MQTzrpKaNzRhe0lFtgy990BFDwcTxq1GK3zBExF+jyMnp75CSfkFybxTCGsjkoolT0gaozRkKXZsjGR2h2T2/P5wMYXtm8fqHb5tjdFBFi+nttJ+zoHvuvQGpadvMiiNsIinGCMvBiaNNZy4R+rg X-Gm-Message-State: AOJu0YxoZoR73OvPELyAP9aL/tuiip07Q8DTXAwAOcuWqkiGyWtmmrP9 6a9nWVdlkvG4js0JV54ncFQ4pg8/lDjgW10o94mSB6CbN12NkWKx X-Google-Smtp-Source: AGHT+IET1giK0nCDgaaYNoRofNa4LOOOB4uldHyP/oj/iDaFmAcLATci4GlKLRHJ6In34vRon/dcrw== X-Received: by 2002:a50:ccc3:0:b0:56b:9f82:4a40 with SMTP id b3-20020a50ccc3000000b0056b9f824a40mr1202811edj.11.1711705764508; Fri, 29 Mar 2024 02:49:24 -0700 (PDT) Received: from fedora.iskraemeco.si ([193.77.86.250]) by smtp.gmail.com with ESMTPSA id b4-20020a0564021f0400b0056c1cca33bfsm1829733edb.6.2024.03.29.02.49.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 29 Mar 2024 02:49:24 -0700 (PDT) From: Uros Bizjak To: x86@kernel.org, bpf@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Cc: =?utf-8?q?Joan_Bruguera_Mic=C3=B3?= , Alexei Starovoitov , Daniel Borkmann Subject: [PATCH RESEND bpf 1/2] x86/bpf: Fix IP after emitting call depth accounting Date: Fri, 29 Mar 2024 10:46:17 +0100 Message-ID: <20240329094906.18147-2-ubizjak@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240329094906.18147-1-ubizjak@gmail.com> References: <20240329094906.18147-1-ubizjak@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net From: Joan Bruguera Micó Adjust the IP passed to `emit_patch` so it calculates the correct offset for the CALL instruction if `x86_call_depth_emit_accounting` emits code. Otherwise we will skip some instructions and most likely crash. Fixes: b2e9dfe54be4 ("x86/bpf: Emit call depth accounting if required") Link: https://lore.kernel.org/lkml/20230105214922.250473-1-joanbrugueram@gmail.com/ Signed-off-by: Joan Bruguera Micó Cc: Alexei Starovoitov Cc: Daniel Borkmann --- arch/x86/net/bpf_jit_comp.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index a7ba8e178645..09f7dc9d4d65 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -479,9 +479,10 @@ static int emit_call(u8 **pprog, void *func, void *ip) static int emit_rsb_call(u8 **pprog, void *func, void *ip) { + void *adjusted_ip; OPTIMIZER_HIDE_VAR(func); - x86_call_depth_emit_accounting(pprog, func); - return emit_patch(pprog, func, ip, 0xE8); + adjusted_ip = ip + x86_call_depth_emit_accounting(pprog, func); + return emit_patch(pprog, func, adjusted_ip, 0xE8); } static int emit_jump(u8 **pprog, void *func, void *ip) From patchwork Fri Mar 29 09:46:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Uros Bizjak X-Patchwork-Id: 13610502 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-ej1-f41.google.com (mail-ej1-f41.google.com [209.85.218.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7DDA251C42; Fri, 29 Mar 2024 09:49:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711705770; cv=none; b=KNJwWR9XzkvuisRUQ+IuveOS/Gi39IJddkOjMNhjfSnlejuxky1En1JmgcLgqbYIkbNdGyPcOYXM+OZ0icCw6POYuEQRDvrjU/g7N/Digx3cctgqhnKM/8a8kZksAr4nWhcTvjNAQw+lcXWkB48zH+JxCxDrgNelQv8K8LI6idc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711705770; c=relaxed/simple; bh=hqCljvC32qy1aJQwAmFON1j9oPrqVbZVDNy4RJHdb/k=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=OJfoeq3xB6gj/Z3OKMB6Plrd4L1t46XGQLHywduhCTsx7Sm12jTUQxC8vyXco5cOOsBtVpNGS7AHaxV36WL0P9LbD4mLxW8OXlKpp0c0KFFa4VV9+A7Z941vX2RE3OnDv5GWToImbMQAPmybZKdT6/kYlUYztEre9Dx3Yt0aW7E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=XOuc8Zbn; arc=none smtp.client-ip=209.85.218.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="XOuc8Zbn" Received: by mail-ej1-f41.google.com with SMTP id a640c23a62f3a-a465ddc2c09so106847366b.2; Fri, 29 Mar 2024 02:49:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711705767; x=1712310567; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hHSOVIWxWM7IMY+bvKjJnZsIEx5Dhp5jPlyzSlliO1I=; b=XOuc8ZbnT4/sl8dT+PMdlpcpLVbvrYF8IQleYXBpogqn9A3HmkRQZBD4zl82jzi0hy iCrshPuTjeXNu1guMVvwsIKn/PZZ4PcPVKVrtmZSVD8LxdW3hRUcFNN3Z8a5dR94Govb zl4mLE4eC3+8LfRWnUULgEUfQdGLB8I5rfqS+pqz77HDV0hTRIzFV/hNSplQVm31hcth CqIN0Ex3fhjApKWbTfcGlgCXC0me6GvK62rjDxuyKsOcAJedsKMbf5VpU2I+UTrwUNgs gcyxbKx8DYuAIe5u7/kpU6/Tv/Mt2gU4p1IyQ/aixMXKpbob+1WjYt0g8tcING4A8JWH R32Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711705767; x=1712310567; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hHSOVIWxWM7IMY+bvKjJnZsIEx5Dhp5jPlyzSlliO1I=; b=p+22zyOZFBV40qTQMmjZkjkJX3oLlFncuSDdFLCTEeJtCB8oJsMiMg3hXrenjJnGfz P9Nr/rMmiSNXO9gqkveGBNxLZ08EwMzUNeTVTiXFKfIIZIRySt53wwGIgJgFfhCRp9Om bhGcw5BkQtrh0zqVgIXQFECCiOENAOR3ZOtpQ+staVuG0Qz+KETlWMZh5MZ7raPgx2Vm GNIH6dc/DGoaZyg+UIIrtrBne1pd9fZz30uBXjC8pLbxze+NTV4Yhn5UL5km/oCN4gV5 GN9GT+kEOg457bZKKlqPPduDkM7hA+xq4PoVPt306dwFk86skpa10VnyoVEq7a08IuAD tAXA== X-Forwarded-Encrypted: i=1; AJvYcCWpQ3yKvp4UjjNTBYmMpfFN0aMIWtC9sSjYMtvQcQjGtII+9WIYQIBZ+LxtU4kEh+V5vzMy64t1M3GdzpDsQCCy/4rv20rXqedQnzR4lve2Dnqk3Bqbf8yN+CR7tBaGTFwomnIptl8DUNpgh+a8s3ty8nPFZjy8iC0f X-Gm-Message-State: AOJu0YxjF5D59pPmtImtLdw/MlfbkkeEZAT5ySucdFKET4IJhQC3k2Fb LdqjRCJrSlvrKp9Boii5HU4FyM4Zq17WDLXujeqbyZFO/CWwCe1a X-Google-Smtp-Source: AGHT+IFACKnbtBeJLF5ik9tpo0Ya/WArpgeSL7iNa9UO8o7ombLR0YWPBn0KSIsUuamOWRvaMhglxg== X-Received: by 2002:a05:6402:13d6:b0:56c:5a7b:5dbf with SMTP id a22-20020a05640213d600b0056c5a7b5dbfmr2019197edx.2.1711705766578; Fri, 29 Mar 2024 02:49:26 -0700 (PDT) Received: from fedora.iskraemeco.si ([193.77.86.250]) by smtp.gmail.com with ESMTPSA id b4-20020a0564021f0400b0056c1cca33bfsm1829733edb.6.2024.03.29.02.49.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 29 Mar 2024 02:49:25 -0700 (PDT) From: Uros Bizjak To: x86@kernel.org, bpf@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Cc: =?utf-8?q?Joan_Bruguera_Mic=C3=B3?= , Uros Bizjak , Ingo Molnar , Alexei Starovoitov , Daniel Borkmann Subject: [PATCH RESEND bpf 2/2] x86/bpf: Fix IP for relocating call depth accounting Date: Fri, 29 Mar 2024 10:46:18 +0100 Message-ID: <20240329094906.18147-3-ubizjak@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240329094906.18147-1-ubizjak@gmail.com> References: <20240329094906.18147-1-ubizjak@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net From: Joan Bruguera Micó The recently introduced support for %rip-relative relocations in the call thunk template assumes that the code is being patched in-place, so the destination of the relocation matches the address of the code. This is not true for the call depth accounting emitted by the BPF JIT, so the calculated address is wrong and usually causes a page fault. Pass the destination IP when the BPF JIT emits call depth accounting. Fixes: 17bce3b2ae2d ("x86/callthunks: Handle %rip-relative relocations in call thunk template") Signed-off-by: Joan Bruguera Micó Reviewed-by: Uros Bizjak Acked-by: Ingo Molnar Cc: Alexei Starovoitov Cc: Daniel Borkmann --- arch/x86/include/asm/alternative.h | 4 ++-- arch/x86/kernel/callthunks.c | 4 ++-- arch/x86/net/bpf_jit_comp.c | 19 ++++++++----------- 3 files changed, 12 insertions(+), 15 deletions(-) diff --git a/arch/x86/include/asm/alternative.h b/arch/x86/include/asm/alternative.h index fcd20c6dc7f9..67b68d0d17d1 100644 --- a/arch/x86/include/asm/alternative.h +++ b/arch/x86/include/asm/alternative.h @@ -117,7 +117,7 @@ extern void callthunks_patch_builtin_calls(void); extern void callthunks_patch_module_calls(struct callthunk_sites *sites, struct module *mod); extern void *callthunks_translate_call_dest(void *dest); -extern int x86_call_depth_emit_accounting(u8 **pprog, void *func); +extern int x86_call_depth_emit_accounting(u8 **pprog, void *func, void *ip); #else static __always_inline void callthunks_patch_builtin_calls(void) {} static __always_inline void @@ -128,7 +128,7 @@ static __always_inline void *callthunks_translate_call_dest(void *dest) return dest; } static __always_inline int x86_call_depth_emit_accounting(u8 **pprog, - void *func) + void *func, void *ip) { return 0; } diff --git a/arch/x86/kernel/callthunks.c b/arch/x86/kernel/callthunks.c index 30335182b6b0..e92ff0c11db8 100644 --- a/arch/x86/kernel/callthunks.c +++ b/arch/x86/kernel/callthunks.c @@ -314,7 +314,7 @@ static bool is_callthunk(void *addr) return !bcmp(pad, insn_buff, tmpl_size); } -int x86_call_depth_emit_accounting(u8 **pprog, void *func) +int x86_call_depth_emit_accounting(u8 **pprog, void *func, void *ip) { unsigned int tmpl_size = SKL_TMPL_SIZE; u8 insn_buff[MAX_PATCH_LEN]; @@ -327,7 +327,7 @@ int x86_call_depth_emit_accounting(u8 **pprog, void *func) return 0; memcpy(insn_buff, skl_call_thunk_template, tmpl_size); - apply_relocation(insn_buff, tmpl_size, *pprog, + apply_relocation(insn_buff, tmpl_size, ip, skl_call_thunk_template, tmpl_size); memcpy(*pprog, insn_buff, tmpl_size); diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 09f7dc9d4d65..f2e8769f5eee 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -481,7 +481,7 @@ static int emit_rsb_call(u8 **pprog, void *func, void *ip) { void *adjusted_ip; OPTIMIZER_HIDE_VAR(func); - adjusted_ip = ip + x86_call_depth_emit_accounting(pprog, func); + adjusted_ip = ip + x86_call_depth_emit_accounting(pprog, func, ip); return emit_patch(pprog, func, adjusted_ip, 0xE8); } @@ -1973,20 +1973,17 @@ st: if (is_imm8(insn->off)) /* call */ case BPF_JMP | BPF_CALL: { - int offs; + u8 *ip = image + addrs[i - 1]; func = (u8 *) __bpf_call_base + imm32; if (tail_call_reachable) { RESTORE_TAIL_CALL_CNT(bpf_prog->aux->stack_depth); - if (!imm32) - return -EINVAL; - offs = 7 + x86_call_depth_emit_accounting(&prog, func); - } else { - if (!imm32) - return -EINVAL; - offs = x86_call_depth_emit_accounting(&prog, func); + ip += 7; } - if (emit_call(&prog, func, image + addrs[i - 1] + offs)) + if (!imm32) + return -EINVAL; + ip += x86_call_depth_emit_accounting(&prog, func, ip); + if (emit_call(&prog, func, ip)) return -EINVAL; break; } @@ -2836,7 +2833,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im * Direct-call fentry stub, as such it needs accounting for the * __fentry__ call. */ - x86_call_depth_emit_accounting(&prog, NULL); + x86_call_depth_emit_accounting(&prog, NULL, image); } EMIT1(0x55); /* push rbp */ EMIT3(0x48, 0x89, 0xE5); /* mov rbp, rsp */