From patchwork Mon Nov 11 11:59:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13870710 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 784EF153BED; Mon, 11 Nov 2024 12:54:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731329678; cv=none; b=cMYomgV9AcNeRRV0hunrMzqSiif7caCCX59PfCWH74HtikFHDUl39I5A7+cnxp/N7AyOM7JN+tPaUbVPKQBX8tWxb75e0OxvBOCk7+7W4IbvInGwNaCZ1KwlDW+V6mY0jtYyyOYtSrBUeytytYYtX9713fU/LV1vH6AkyMa+xr8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731329678; c=relaxed/simple; bh=m01SsjIwow88FybogNsi6AHLyy3wvV6DFlakmMfrMvg=; h=Message-Id:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=Nvtd3tVOivmhrHuzp1hdeWUZTIZl8MweThPnV3WYgO5BswDCxwoQQjw3TOnvKhNCqCh+4HkhQ4ffYjTLkNGJ3SBRvS3C3kEmOVer/cIQYe9ifQ6O6FPR3Dn+OPKRx9lXcPtq/4l/T9hQqQjbycpgmiPnNGRCnGKq44wcUbApHCA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=hHuSfZU4; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="hHuSfZU4" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=VOeUTBZ+q/b21Yztg4np/MJbr8kcEsMNi/GgleRqC8c=; b=hHuSfZU4qfshHtFwsRYTWHKlui i0T/ToOi/iVVsFknbAo0HOEEbEss/W1BRlPjTIkPN5+9JPDMCpBp6WQDeKlc/RK47w0Q8T2OTNgWV vqfl6hZGXJS4mRdJTqdLni+IBLD813TlveU24OeavrJ7IKujxtmf5zYtWE8aJaS/wgcj4vsMIoiQr LXpt0F7tMUV6cmJsbXnARn/UNVqcG2QRo+OjNmebMJm7IZX5326/JhZ2vF//YqHrDanXXV1/ov4mg RelKt66BKdlOgWcMHuuIpIUtU7erwRgSAc0EsK80n+w4e2yFBYgAwXV6u5k9sQylvrD7+femcxfcH DZmnbXfg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tATw9-0000000Coeh-2cJi; Mon, 11 Nov 2024 12:54:33 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id C53CB30083E; Mon, 11 Nov 2024 13:54:32 +0100 (CET) Message-Id: <20241111125218.113053713@infradead.org> User-Agent: quilt/0.65 Date: Mon, 11 Nov 2024 12:59:36 +0100 From: Peter Zijlstra To: seanjc@google.com, pbonzini@redhat.com, jpoimboe@redhat.com, tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, x86@kernel.org, kvm@vger.kernel.org, jthoughton@google.com, "Peter Zijlstra (Intel)" Subject: [PATCH v2 01/12] objtool: Generic annotation infrastructure References: <20241111115935.796797988@infradead.org> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Avoid endless .discard.foo sections for each annotation, create a single .discard.annotate section that takes an annotation type along with the instruction. Signed-off-by: Peter Zijlstra (Intel) --- include/linux/objtool.h | 18 ++++++++++++++++++ tools/objtool/check.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 64 insertions(+) --- a/include/linux/objtool.h +++ b/include/linux/objtool.h @@ -57,6 +57,13 @@ ".long 998b\n\t" \ ".popsection\n\t" +#define ASM_ANNOTATE(x) \ + "911:\n\t" \ + ".pushsection .discard.annotate,\"M\",@progbits,8\n\t" \ + ".long 911b - .\n\t" \ + ".long " __stringify(x) "\n\t" \ + ".popsection\n\t" + #else /* __ASSEMBLY__ */ /* @@ -146,6 +153,14 @@ .popsection .endm +.macro ANNOTATE type:req +.Lhere_\@: + .pushsection .discard.annotate,"M",@progbits,8 + .long .Lhere_\@ - . + .long \type + .popsection +.endm + #endif /* __ASSEMBLY__ */ #else /* !CONFIG_OBJTOOL */ @@ -155,6 +170,7 @@ #define UNWIND_HINT(type, sp_reg, sp_offset, signal) "\n\t" #define STACK_FRAME_NON_STANDARD(func) #define STACK_FRAME_NON_STANDARD_FP(func) +#define ASM_ANNOTATE(x) #define ANNOTATE_NOENDBR #define ASM_REACHABLE #else @@ -167,6 +183,8 @@ .endm .macro REACHABLE .endm +.macro ANNOTATE type:req +.endm #endif #endif /* CONFIG_OBJTOOL */ --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -2373,6 +2373,50 @@ static int read_unwind_hints(struct objt return 0; } +static int read_annotate(struct objtool_file *file, void (*func)(int type, struct instruction *insn)) +{ + struct section *rsec, *sec; + struct instruction *insn; + struct reloc *reloc; + int type; + + rsec = find_section_by_name(file->elf, ".rela.discard.annotate"); + if (!rsec) + return 0; + + sec = find_section_by_name(file->elf, ".discard.annotate"); + if (!sec) + return 0; + + if (sec->sh.sh_entsize != 8) { + static bool warn = false; + if (!warn) { + WARN("%s: dodgy linker, sh_entsize != 8", sec->name); + warn = true; + } + sec->sh.sh_entsize = 8; + } + + for_each_reloc(rsec, reloc) { + insn = find_insn(file, reloc->sym->sec, + reloc->sym->offset + reloc_addend(reloc)); + if (!insn) { + WARN("bad .discard.annotate entry: %d", reloc_idx(reloc)); + return -1; + } + + type = *(u32 *)(sec->data->d_buf + (reloc_idx(reloc) * sec->sh.sh_entsize) + 4); + + func(type, insn); + } + + return 0; +} + +static void __annotate_nop(int type, struct instruction *insn) +{ +} + static int read_noendbr_hints(struct objtool_file *file) { struct instruction *insn; @@ -2670,6 +2714,8 @@ static int decode_sections(struct objtoo if (ret) return ret; + ret = read_annotate(file, __annotate_nop); + /* * Must be before read_unwind_hints() since that needs insn->noendbr. */ From patchwork Mon Nov 11 11:59:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13870716 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 53B5019CD0E; Mon, 11 Nov 2024 12:54:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731329679; cv=none; b=DHIXopa74fjTmzA3SIwGbtqT0zWfocW7kl+3KnoV16jD/V+1zsNA30Dcuz3xhZYVTmEFS/Bi8MP8h5CAeSojf4yrxdxdIALjgvET+QPER8lwf7vEsv1jKs+zyGMWt1Xc0wp+07CtoiyNJzCQupmp7zuJq20DQFq2VJVzW7PBfvA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731329679; c=relaxed/simple; bh=53DniVL2kNJAh6YH3okCJEaYKY7wFXMzVRHLvbrhu4k=; h=Message-Id:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=o2f0QGHQLVUOAVfihHOHMCp9g0Tv9xU2bgH1/m/z4ip2hS4TEyNKkokFXi7i5oAskUJV0Py4fIcrPzK4Tz+dQsoyPRnZZ1VaAkPdtUCFb/g3HML9sYcHmuiJIQpS3ADl2Gx4tNgDS6GwqBMMcmvMtP5RVnOO3xL9XWwro7TypEs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=EOLiYe0z; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="EOLiYe0z" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=dYFXJA1/fZVGzxQ5AQj4cq8phZVmCHY7eA+a7xfnNgw=; b=EOLiYe0zU+So1jL4PO+NCMf/Cn zT6rzJR6daK1Nh8Y8Skab6HGr1A5aVJAc3G2gstHJgsZCIakK9a83uf1IoHEgbsRZY/ALcXxVoF2p nIpaB/0V0s5G1nazrw9z0V5bKNFgcoeqSMY2l8/8u7xBC2qQ0e32c6/IwWs1ZqEBCHM87ay/XoZ39 BOhicg/vcR8ZKkMtYx43GQI2dQ4oVBoLheJMusOoeELwEUU2YX1j3UPiy3JjI0NJzXehtYKhGQZGB VQFq15QT8riG39vpECe/6LLyIAlVvDG1FTgyz5VSkiMg6+3Jl6qG8YT6dU/+xOTnn30hVF7eF01gD 11C0gq8Q==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tATw9-0000000Cqjc-2ZQt; Mon, 11 Nov 2024 12:54:33 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id C98D4300C1F; Mon, 11 Nov 2024 13:54:32 +0100 (CET) Message-Id: <20241111125218.222910882@infradead.org> User-Agent: quilt/0.65 Date: Mon, 11 Nov 2024 12:59:37 +0100 From: Peter Zijlstra To: seanjc@google.com, pbonzini@redhat.com, jpoimboe@redhat.com, tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, x86@kernel.org, kvm@vger.kernel.org, jthoughton@google.com, "Peter Zijlstra (Intel)" Subject: [PATCH v2 02/12] objtool: Convert ANNOTATE_NOENDBR to ANNOTATE References: <20241111115935.796797988@infradead.org> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Signed-off-by: Peter Zijlstra (Intel) --- include/linux/objtool.h | 17 ++++------------- include/linux/objtool_types.h | 5 +++++ tools/include/linux/objtool_types.h | 5 +++++ tools/objtool/check.c | 32 +++++--------------------------- 4 files changed, 19 insertions(+), 40 deletions(-) --- a/include/linux/objtool.h +++ b/include/linux/objtool.h @@ -45,12 +45,6 @@ #define STACK_FRAME_NON_STANDARD_FP(func) #endif -#define ANNOTATE_NOENDBR \ - "986: \n\t" \ - ".pushsection .discard.noendbr\n\t" \ - ".long 986b\n\t" \ - ".popsection\n\t" - #define ASM_REACHABLE \ "998:\n\t" \ ".pushsection .discard.reachable\n\t" \ @@ -64,6 +58,8 @@ ".long " __stringify(x) "\n\t" \ ".popsection\n\t" +#define ANNOTATE_NOENDBR ASM_ANNOTATE(ANNOTYPE_NOENDBR) + #else /* __ASSEMBLY__ */ /* @@ -122,13 +118,6 @@ #endif .endm -.macro ANNOTATE_NOENDBR -.Lhere_\@: - .pushsection .discard.noendbr - .long .Lhere_\@ - .popsection -.endm - /* * Use objtool to validate the entry requirement that all code paths do * VALIDATE_UNRET_END before RET. @@ -161,6 +150,8 @@ .popsection .endm +#define ANNOTATE_NOENDBR ANNOTATE type=ANNOTYPE_NOENDBR + #endif /* __ASSEMBLY__ */ #else /* !CONFIG_OBJTOOL */ --- a/include/linux/objtool_types.h +++ b/include/linux/objtool_types.h @@ -54,4 +54,9 @@ struct unwind_hint { #define UNWIND_HINT_TYPE_SAVE 6 #define UNWIND_HINT_TYPE_RESTORE 7 +/* + * Annotate types + */ +#define ANNOTYPE_NOENDBR 1 + #endif /* _LINUX_OBJTOOL_TYPES_H */ --- a/tools/include/linux/objtool_types.h +++ b/tools/include/linux/objtool_types.h @@ -54,4 +54,9 @@ struct unwind_hint { #define UNWIND_HINT_TYPE_SAVE 6 #define UNWIND_HINT_TYPE_RESTORE 7 +/* + * Annotate types + */ +#define ANNOTYPE_NOENDBR 1 + #endif /* _LINUX_OBJTOOL_TYPES_H */ --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -2339,32 +2339,12 @@ static int read_annotate(struct objtool_ return 0; } -static void __annotate_nop(int type, struct instruction *insn) +static void __annotate_noendbr(int type, struct instruction *insn) { -} - -static int read_noendbr_hints(struct objtool_file *file) -{ - struct instruction *insn; - struct section *rsec; - struct reloc *reloc; - - rsec = find_section_by_name(file->elf, ".rela.discard.noendbr"); - if (!rsec) - return 0; + if (type != ANNOTYPE_NOENDBR) + return; - for_each_reloc(rsec, reloc) { - insn = find_insn(file, reloc->sym->sec, - reloc->sym->offset + reloc_addend(reloc)); - if (!insn) { - WARN("bad .discard.noendbr entry"); - return -1; - } - - insn->noendbr = 1; - } - - return 0; + insn->noendbr = 1; } static int read_retpoline_hints(struct objtool_file *file) @@ -2637,12 +2617,10 @@ static int decode_sections(struct objtoo if (ret) return ret; - ret = read_annotate(file, __annotate_nop); - /* * Must be before read_unwind_hints() since that needs insn->noendbr. */ - ret = read_noendbr_hints(file); + ret = read_annotate(file, __annotate_noendbr); if (ret) return ret; From patchwork Mon Nov 11 11:59:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13870711 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9097D158DC8; Mon, 11 Nov 2024 12:54:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731329678; cv=none; b=bYZjO3GrW6tUSvWkms2CGRfRd5nmahKJM1hmOlANH/4IlZXm1p49geMVdq0Dzu5fm5JIrTznlI+HzPaa/NKdE0gtPMrDPY6+fmKS0ijzPNjmIwZe130mE8wyZaUZ7upqSURcK/atZ5c+mA5qdMZr1CDHwsTajfERqPc0kTEA4DQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731329678; c=relaxed/simple; bh=oYrIFwvJfrI7jAQgaqfbw5jx8yrxxSmXJ4S7CQo7fxg=; h=Message-Id:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=ju2bIDpkhIOq9Kr7ZR4XdjdllmgGf1H+OzRxsOfdZrDRc/+Dj9wdAkU8ouTsgMFq4IZKTKEFD7g7QG3I3wGEQ1DXHqX54g+JscP4fjgnuGgXKHAnjCGpjD1P9BT3VEu62Wg5uogScrWGbK/VIjW6Pik4a9CARyZl3mUBA3ty7cM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=Y/SaczTt; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="Y/SaczTt" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=O8rCIvUkqNpAaMxQiAO/KoN2MOG0w6rqWm9LpswbbCQ=; b=Y/SaczTt2VLJeFlQy4NUTmJoFv os+shIddrDIK8eHeP/2ce+6zBS4frieguidXLlaqzvKdNkfyrQQTvDffiQ84947/rWiKM4D7/q0W3 pxG6uONvpyw7OSY4AaKuMLW/rCtiryO0tmTzZjAOzl1rCVsjRSTfIseQwQ7veP4+UwU0iH7c2wZhl jGLq7GttfpzYISIggXLWvMa7tXYrMUwBAzLAhu5LaBfIsKBKplLdyKKTGaYZ4gVtPQRoN47QYl3hi njeasJTnIcQmqlhYpbw9rAMUp49Uc3AzGSX72A3jS4mmeHRNCIKaWDThXjvuZvzJ2P0G1vnk3mdvM zwieHMVA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tATw9-0000000Cqjf-2bS1; Mon, 11 Nov 2024 12:54:33 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id CD60F300DDC; Mon, 11 Nov 2024 13:54:32 +0100 (CET) Message-Id: <20241111125218.357848045@infradead.org> User-Agent: quilt/0.65 Date: Mon, 11 Nov 2024 12:59:38 +0100 From: Peter Zijlstra To: seanjc@google.com, pbonzini@redhat.com, jpoimboe@redhat.com, tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, x86@kernel.org, kvm@vger.kernel.org, jthoughton@google.com, "Peter Zijlstra (Intel)" Subject: [PATCH v2 03/12] objtool: Convert ANNOTATE_RETPOLINE_SAFE to ANNOTATE References: <20241111115935.796797988@infradead.org> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/nospec-branch.h | 13 +------- include/linux/objtool_types.h | 1 tools/include/linux/objtool_types.h | 1 tools/objtool/check.c | 52 ++++++++++++----------------------- 4 files changed, 22 insertions(+), 45 deletions(-) --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -193,12 +193,7 @@ * objtool the subsequent indirect jump/call is vouched safe for retpoline * builds. */ -.macro ANNOTATE_RETPOLINE_SAFE -.Lhere_\@: - .pushsection .discard.retpoline_safe - .long .Lhere_\@ - .popsection -.endm +#define ANNOTATE_RETPOLINE_SAFE ANNOTATE type=ANNOTYPE_RETPOLINE_SAFE /* * (ab)use RETPOLINE_SAFE on RET to annotate away 'bare' RET instructions @@ -317,11 +312,7 @@ #else /* __ASSEMBLY__ */ -#define ANNOTATE_RETPOLINE_SAFE \ - "999:\n\t" \ - ".pushsection .discard.retpoline_safe\n\t" \ - ".long 999b\n\t" \ - ".popsection\n\t" +#define ANNOTATE_RETPOLINE_SAFE ASM_ANNOTATE(ANNOTYPE_RETPOLINE_SAFE) typedef u8 retpoline_thunk_t[RETPOLINE_THUNK_SIZE]; extern retpoline_thunk_t __x86_indirect_thunk_array[]; --- a/include/linux/objtool_types.h +++ b/include/linux/objtool_types.h @@ -58,5 +58,6 @@ struct unwind_hint { * Annotate types */ #define ANNOTYPE_NOENDBR 1 +#define ANNOTYPE_RETPOLINE_SAFE 2 #endif /* _LINUX_OBJTOOL_TYPES_H */ --- a/tools/include/linux/objtool_types.h +++ b/tools/include/linux/objtool_types.h @@ -58,5 +58,6 @@ struct unwind_hint { * Annotate types */ #define ANNOTYPE_NOENDBR 1 +#define ANNOTYPE_RETPOLINE_SAFE 2 #endif /* _LINUX_OBJTOOL_TYPES_H */ --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -2308,12 +2308,12 @@ static int read_unwind_hints(struct objt return 0; } -static int read_annotate(struct objtool_file *file, void (*func)(int type, struct instruction *insn)) +static int read_annotate(struct objtool_file *file, int (*func)(int type, struct instruction *insn)) { struct section *rsec, *sec; struct instruction *insn; struct reloc *reloc; - int type; + int type, ret; rsec = find_section_by_name(file->elf, ".rela.discard.annotate"); if (!rsec) @@ -2333,53 +2333,37 @@ static int read_annotate(struct objtool_ type = *(u32 *)(sec->data->d_buf + (reloc_idx(reloc) * sec->sh.sh_entsize) + 4); - func(type, insn); + ret = func(type, insn); + if (ret < 0) + return ret; } return 0; } -static void __annotate_noendbr(int type, struct instruction *insn) +static int __annotate_noendbr(int type, struct instruction *insn) { if (type != ANNOTYPE_NOENDBR) - return; + return 0; insn->noendbr = 1; + return 0; } -static int read_retpoline_hints(struct objtool_file *file) +static int __annotate_retpoline_safe(int type, struct instruction *insn) { - struct section *rsec; - struct instruction *insn; - struct reloc *reloc; - - rsec = find_section_by_name(file->elf, ".rela.discard.retpoline_safe"); - if (!rsec) + if (type != ANNOTYPE_RETPOLINE_SAFE) return 0; - for_each_reloc(rsec, reloc) { - if (reloc->sym->type != STT_SECTION) { - WARN("unexpected relocation symbol type in %s", rsec->name); - return -1; - } - - insn = find_insn(file, reloc->sym->sec, reloc_addend(reloc)); - if (!insn) { - WARN("bad .discard.retpoline_safe entry"); - return -1; - } - - if (insn->type != INSN_JUMP_DYNAMIC && - insn->type != INSN_CALL_DYNAMIC && - insn->type != INSN_RETURN && - insn->type != INSN_NOP) { - WARN_INSN(insn, "retpoline_safe hint not an indirect jump/call/ret/nop"); - return -1; - } - - insn->retpoline_safe = true; + if (insn->type != INSN_JUMP_DYNAMIC && + insn->type != INSN_CALL_DYNAMIC && + insn->type != INSN_RETURN && + insn->type != INSN_NOP) { + WARN_INSN(insn, "retpoline_safe hint not an indirect jump/call/ret/nop"); + return -1; } + insn->retpoline_safe = true; return 0; } @@ -2666,7 +2650,7 @@ static int decode_sections(struct objtoo if (ret) return ret; - ret = read_retpoline_hints(file); + ret = read_annotate(file, __annotate_retpoline_safe); if (ret) return ret; From patchwork Mon Nov 11 11:59:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13870719 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 55B0219D075; Mon, 11 Nov 2024 12:54:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731329679; cv=none; b=CXKU+isRV+enR+WJtBejgCfFK0UAfWGZUfZP6514XPF2qjrcWpanIPumdv2TzO5vLy2ELqG+i4kH0oJpXI13nXm5si3yAaQJb/DCiBaZv7SAFJi3GObQbQjNL+xU7NJzh1jetTkH/VmgcqtkuIT/D5BgrxOJUq2gkzB1LLbn3os= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731329679; c=relaxed/simple; bh=xCKBCWveVHzvPaCu6KcEW6e5D2t80IxrzBBBa7ugjmE=; h=Message-Id:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=AUTXGyCNbZHdj8noIwJ0WrrQDPdlwoAXgsykzre36RTfC7uT4BhzR0Jp4GnlO56TKD6nILBokyiK71v1o12vUtVjg0ep8owGCGhbsyIlJHTjDcBes2oQpsu6RiNP85kl66fOr5qDP+p5Z3/TTCAtJMIph+W8n1PuJNNq1DiyTKs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=ZEktBubi; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="ZEktBubi" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=gFiDthjTb2jV5OQYX2IoNEzMxtX8OE63urpzPswRdWY=; b=ZEktBubiLEUyquOyzWHYXGjm3n Aa0Axe6XO0Kz4ruDdUSO7hkfdNJgkwmAWwQ8adlFgCdzVf6x2qy+Csg43woEpJNhd1PYrdTAoq68R hZYiai+JvTFunN1sNG3zX1NcBc314JVDfVNX6fGyw6Li9sG7OXKIYDjuTZW1qxat20PQDsshmNbHZ KFGvlU7KJ/NlOUMlBxT8zfG2dfM8EL8uvvgfamNsl+UmI6cwu8IBGe3SV7204tGy7UWUXZ9ZoiOWT DMdrIsHPMQUo0dhPAzCApoEPXljeDymBBPN4TrjbSDk4BVR7E+59cgAShA0dY/PSZj5PmQs/4qZoo YcL79ucg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tATw9-0000000Cqje-2bIy; Mon, 11 Nov 2024 12:54:34 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id D1453300DF3; Mon, 11 Nov 2024 13:54:32 +0100 (CET) Message-Id: <20241111125218.469665219@infradead.org> User-Agent: quilt/0.65 Date: Mon, 11 Nov 2024 12:59:39 +0100 From: Peter Zijlstra To: seanjc@google.com, pbonzini@redhat.com, jpoimboe@redhat.com, tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, x86@kernel.org, kvm@vger.kernel.org, jthoughton@google.com, "Peter Zijlstra (Intel)" Subject: [PATCH v2 04/12] objtool: Convert instrumentation_{begin,end}() to ANNOTATE References: <20241111115935.796797988@infradead.org> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Signed-off-by: Peter Zijlstra (Intel) --- include/linux/instrumentation.h | 11 +++----- include/linux/objtool.h | 9 ++++-- include/linux/objtool_types.h | 2 + tools/include/linux/objtool_types.h | 2 + tools/objtool/check.c | 49 +++++++----------------------------- 5 files changed, 25 insertions(+), 48 deletions(-) --- a/include/linux/instrumentation.h +++ b/include/linux/instrumentation.h @@ -4,14 +4,14 @@ #ifdef CONFIG_NOINSTR_VALIDATION +#include #include /* Begin/end of an instrumentation safe region */ #define __instrumentation_begin(c) ({ \ asm volatile(__stringify(c) ": nop\n\t" \ - ".pushsection .discard.instr_begin\n\t" \ - ".long " __stringify(c) "b - .\n\t" \ - ".popsection\n\t" : : "i" (c)); \ + __ASM_ANNOTATE(c, ANNOTYPE_INSTR_BEGIN) \ + : : "i" (c)); \ }) #define instrumentation_begin() __instrumentation_begin(__COUNTER__) @@ -48,9 +48,8 @@ */ #define __instrumentation_end(c) ({ \ asm volatile(__stringify(c) ": nop\n\t" \ - ".pushsection .discard.instr_end\n\t" \ - ".long " __stringify(c) "b - .\n\t" \ - ".popsection\n\t" : : "i" (c)); \ + __ASM_ANNOTATE(c, ANNOTYPE_INSTR_END) \ + : : "i" (c)); \ }) #define instrumentation_end() __instrumentation_end(__COUNTER__) #else /* !CONFIG_NOINSTR_VALIDATION */ --- a/include/linux/objtool.h +++ b/include/linux/objtool.h @@ -51,13 +51,16 @@ ".long 998b\n\t" \ ".popsection\n\t" -#define ASM_ANNOTATE(x) \ - "911:\n\t" \ +#define __ASM_ANNOTATE(s, x) \ ".pushsection .discard.annotate,\"M\",@progbits,8\n\t" \ - ".long 911b - .\n\t" \ + ".long " __stringify(s) "b - .\n\t" \ ".long " __stringify(x) "\n\t" \ ".popsection\n\t" +#define ASM_ANNOTATE(x) \ + "911:\n\t" \ + __ASM_ANNOTATE(911, x) + #define ANNOTATE_NOENDBR ASM_ANNOTATE(ANNOTYPE_NOENDBR) #else /* __ASSEMBLY__ */ --- a/include/linux/objtool_types.h +++ b/include/linux/objtool_types.h @@ -59,5 +59,7 @@ struct unwind_hint { */ #define ANNOTYPE_NOENDBR 1 #define ANNOTYPE_RETPOLINE_SAFE 2 +#define ANNOTYPE_INSTR_BEGIN 3 +#define ANNOTYPE_INSTR_END 4 #endif /* _LINUX_OBJTOOL_TYPES_H */ --- a/tools/include/linux/objtool_types.h +++ b/tools/include/linux/objtool_types.h @@ -59,5 +59,7 @@ struct unwind_hint { */ #define ANNOTYPE_NOENDBR 1 #define ANNOTYPE_RETPOLINE_SAFE 2 +#define ANNOTYPE_INSTR_BEGIN 3 +#define ANNOTYPE_INSTR_END 4 #endif /* _LINUX_OBJTOOL_TYPES_H */ --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -2367,48 +2367,19 @@ static int __annotate_retpoline_safe(int return 0; } -static int read_instr_hints(struct objtool_file *file) +static int __annotate_instr(int type, struct instruction *insn) { - struct section *rsec; - struct instruction *insn; - struct reloc *reloc; - - rsec = find_section_by_name(file->elf, ".rela.discard.instr_end"); - if (!rsec) - return 0; - - for_each_reloc(rsec, reloc) { - if (reloc->sym->type != STT_SECTION) { - WARN("unexpected relocation symbol type in %s", rsec->name); - return -1; - } - - insn = find_insn(file, reloc->sym->sec, reloc_addend(reloc)); - if (!insn) { - WARN("bad .discard.instr_end entry"); - return -1; - } + switch (type) { + case ANNOTYPE_INSTR_BEGIN: + insn->instr++; + break; + case ANNOTYPE_INSTR_END: insn->instr--; - } - - rsec = find_section_by_name(file->elf, ".rela.discard.instr_begin"); - if (!rsec) - return 0; + break; - for_each_reloc(rsec, reloc) { - if (reloc->sym->type != STT_SECTION) { - WARN("unexpected relocation symbol type in %s", rsec->name); - return -1; - } - - insn = find_insn(file, reloc->sym->sec, reloc_addend(reloc)); - if (!insn) { - WARN("bad .discard.instr_begin entry"); - return -1; - } - - insn->instr++; + default: + break; } return 0; @@ -2654,7 +2625,7 @@ static int decode_sections(struct objtoo if (ret) return ret; - ret = read_instr_hints(file); + ret = read_annotate(file, __annotate_instr); if (ret) return ret; From patchwork Mon Nov 11 11:59:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13870713 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D43C619B3EE; Mon, 11 Nov 2024 12:54:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731329678; cv=none; b=bojob/cg+pYunfHn9HXDuaX/yKZq17+uCZ4mz6XuvukE+oi51FcF/PIwNtDpD9/3j0VzZ+FO1o8Qu8QjAHZo8NBtHJEm/spRjnCEE5mQw/yp6usrXFzk3ZBc+6rWtiG0Y9NGUQjjC62P+UTLV/30GU25FsbLyr+1xsbZZFM/rMc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731329678; c=relaxed/simple; bh=Yn8q4QXUFlxAK46m/412Srx0seyBg8HOXUGBVXU1ZX4=; h=Message-Id:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=f35ly1jmbPHYfbWGs+OZ0SnpmXNCPH+2Zw2jfMKMBpSVc0oex/DEFa7zOoqPiQl4a0i9S7K665yN1rbObfLCcg3bNdySMzh2MkRVilBHIulq7FIBO7b9s2yrdZVy+quw0JPckrDgVRenNOD9nS6eABz2VlTKtETcAA9UrElHtAI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=fgfgYhAo; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="fgfgYhAo" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=v+L6BbfOg0jwO5KFbIbrKS0ECVjAfimoVQrz+6GrR9E=; b=fgfgYhAojh5l7OFcvabZtkuelJ SvRXnmQxVkY1TOKWuBUL1X6PIhn2CUA8tEI0y6DouOGHmNcY+8AI7BPYVIglWJ0nrq4Nb7OmnDozr nj37M9SBSEshZ5s0PtuhRNYTpf6IhipDGA6MqzSX0f8nMYFp1HdH216nihk9HmjRGI0QT6+6+vjCL KYXS7rONdbPAmHXS8BHEwvBnbR7dRPHUXlhASkSp3kEy/+33/zAk0PHCEHzHKCqMMH0qcPCwrU/qL CaJonk9RIrRDAknTlyoS37ZEqfkLF6RlPV0SVw5QU2LmbtmIKEAb/MpH2aG8T9YsE03NKc+hND4Lf 72CdLj0A==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tATwA-0000000Coek-18VT; Mon, 11 Nov 2024 12:54:34 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id D57A7300F1A; Mon, 11 Nov 2024 13:54:32 +0100 (CET) Message-Id: <20241111125218.580632025@infradead.org> User-Agent: quilt/0.65 Date: Mon, 11 Nov 2024 12:59:40 +0100 From: Peter Zijlstra To: seanjc@google.com, pbonzini@redhat.com, jpoimboe@redhat.com, tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, x86@kernel.org, kvm@vger.kernel.org, jthoughton@google.com, "Peter Zijlstra (Intel)" Subject: [PATCH v2 05/12] objtool: Convert VALIDATE_UNRET_BEGIN to ANNOTATE References: <20241111115935.796797988@infradead.org> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Signed-off-by: Peter Zijlstra (Intel) --- include/linux/objtool.h | 9 +++------ include/linux/objtool_types.h | 1 + tools/include/linux/objtool_types.h | 1 + tools/objtool/check.c | 28 +++++----------------------- 4 files changed, 10 insertions(+), 29 deletions(-) --- a/include/linux/objtool.h +++ b/include/linux/objtool.h @@ -128,15 +128,12 @@ * NOTE: The macro must be used at the beginning of a global symbol, otherwise * it will be ignored. */ -.macro VALIDATE_UNRET_BEGIN #if defined(CONFIG_NOINSTR_VALIDATION) && \ (defined(CONFIG_MITIGATION_UNRET_ENTRY) || defined(CONFIG_MITIGATION_SRSO)) -.Lhere_\@: - .pushsection .discard.validate_unret - .long .Lhere_\@ - . - .popsection +#define VALIDATE_UNRET_BEGIN ANNOTATE type=ANNOTYPE_UNRET_BEGIN +#else +#define VALIDATE_UNRET_BEGIN #endif -.endm .macro REACHABLE .Lhere_\@: --- a/include/linux/objtool_types.h +++ b/include/linux/objtool_types.h @@ -61,5 +61,6 @@ struct unwind_hint { #define ANNOTYPE_RETPOLINE_SAFE 2 #define ANNOTYPE_INSTR_BEGIN 3 #define ANNOTYPE_INSTR_END 4 +#define ANNOTYPE_UNRET_BEGIN 5 #endif /* _LINUX_OBJTOOL_TYPES_H */ --- a/tools/include/linux/objtool_types.h +++ b/tools/include/linux/objtool_types.h @@ -61,5 +61,6 @@ struct unwind_hint { #define ANNOTYPE_RETPOLINE_SAFE 2 #define ANNOTYPE_INSTR_BEGIN 3 #define ANNOTYPE_INSTR_END 4 +#define ANNOTYPE_UNRET_BEGIN 5 #endif /* _LINUX_OBJTOOL_TYPES_H */ --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -2450,33 +2450,15 @@ static int __annotate_instr(int type, st return 0; } -static int read_validate_unret_hints(struct objtool_file *file) +static int __annotate_unret(int type, struct instruction *insn) { - struct section *rsec; - struct instruction *insn; - struct reloc *reloc; - - rsec = find_section_by_name(file->elf, ".rela.discard.validate_unret"); - if (!rsec) + if (type != ANNOTYPE_UNRET_BEGIN) return 0; - for_each_reloc(rsec, reloc) { - if (reloc->sym->type != STT_SECTION) { - WARN("unexpected relocation symbol type in %s", rsec->name); - return -1; - } - - insn = find_insn(file, reloc->sym->sec, reloc_addend(reloc)); - if (!insn) { - WARN("bad .discard.instr_end entry"); - return -1; - } - insn->unret = 1; - } - + insn->unret = 1; return 0; -} +} static int read_intra_function_calls(struct objtool_file *file) { @@ -2697,7 +2679,7 @@ static int decode_sections(struct objtoo if (ret) return ret; - ret = read_validate_unret_hints(file); + ret = read_annotate(file, __annotate_unret); if (ret) return ret; From patchwork Mon Nov 11 11:59:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13870709 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 784A1139578; Mon, 11 Nov 2024 12:54:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731329678; cv=none; b=SnmVAh5wTxyjhKBSeTDwT1jTE3S6J5A3BCbNlbKf52VvXHreruXYtG2qukyPkXTx1sKssHAIHr+1i0AuGIJ7HWdZyubpq2qw+fc1VBirnxUj5cuzA3e2R0UU1oqUwmYMWQJPtF/tW+oe0lgko6ELGHzt2aCDKr3ww4gDQJSNm8Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731329678; c=relaxed/simple; bh=J4qe9kzMeaC+48JT9UPo1D22Cwz02M19mcNneZ0NCOo=; h=Message-Id:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=hpXxm+DYdVXkWAWUSXB0IqCFC+ewlBckKdK/jkBv0i3FdQlsJ3zVWhDsGLBuhW0Jyv1g7LEFnT4I6N+UL8f62qbiKV9YWAo+sqt3B84qgfk6MenR+XuoOpQuBXG9miSAG7EBL38WJ346rPraoI6LcSp3J3ZrADko58crDYJs3Bc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=LLK/QY0i; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="LLK/QY0i" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=iT8rZBU1kFBBncA0eT4LO89fZPLO0VJsqubseEy0Iso=; b=LLK/QY0iORoCXTv/AgbBtVKyil 0gq6D6JmKAd/7DHvVvQfvCU61Zc6ove7tjrpSKxu6itDNj7Q1B8YXmxampxz8aWRKiJ2b1AY1FwiB py5MeXoF8ChtYhfMgkt8SYQXMFbyYmEEmAfQdPuoGxibGSUQfo3QNB3D802A2mqidE+yfTyX2e/0V EhAeGxwd5XRjgQwqL/U3cY57ALPV3q0VK+4gq5xbpAAPCwdBoKBQ3Sb2+zqdx894tyhphjIPGb7V1 VT8v2UvcfsrIoNVsiBV2aJve+uM0yoSXWiKkCndiVgRnEoW/NTOSy44ELkpDdE4sGkaadRnFM/4Kr ek27ukxA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tATwA-0000000Coel-18OY; Mon, 11 Nov 2024 12:54:34 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id D9E8930114C; Mon, 11 Nov 2024 13:54:32 +0100 (CET) Message-Id: <20241111125218.684543983@infradead.org> User-Agent: quilt/0.65 Date: Mon, 11 Nov 2024 12:59:41 +0100 From: Peter Zijlstra To: seanjc@google.com, pbonzini@redhat.com, jpoimboe@redhat.com, tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, x86@kernel.org, kvm@vger.kernel.org, jthoughton@google.com, "Peter Zijlstra (Intel)" Subject: [PATCH v2 06/12] objtool: Convert ANNOTATE_IGNORE_ALTERNATIVE to ANNOTATE References: <20241111115935.796797988@infradead.org> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/alternative.h | 14 ++--------- include/linux/objtool_types.h | 1 tools/include/linux/objtool_types.h | 1 tools/objtool/check.c | 45 ++++++++---------------------------- 4 files changed, 15 insertions(+), 46 deletions(-) --- a/arch/x86/include/asm/alternative.h +++ b/arch/x86/include/asm/alternative.h @@ -4,6 +4,7 @@ #include #include +#include #include #define ALT_FLAGS_SHIFT 16 @@ -55,11 +56,7 @@ * objtool annotation to ignore the alternatives and only consider the original * instruction(s). */ -#define ANNOTATE_IGNORE_ALTERNATIVE \ - "999:\n\t" \ - ".pushsection .discard.ignore_alts\n\t" \ - ".long 999b\n\t" \ - ".popsection\n\t" +#define ANNOTATE_IGNORE_ALTERNATIVE ASM_ANNOTATE(ANNOTYPE_IGNORE_ALTS) /* * The patching flags are part of the upper bits of the @ft_flags parameter when @@ -349,12 +346,7 @@ static inline int alternatives_text_rese * objtool annotation to ignore the alternatives and only consider the original * instruction(s). */ -.macro ANNOTATE_IGNORE_ALTERNATIVE - .Lannotate_\@: - .pushsection .discard.ignore_alts - .long .Lannotate_\@ - .popsection -.endm +#define ANNOTATE_IGNORE_ALTERNATIVE ANNOTATE type=ANNOTYPE_IGNORE_ALTS /* * Issue one struct alt_instr descriptor entry (need to put it into --- a/include/linux/objtool_types.h +++ b/include/linux/objtool_types.h @@ -62,5 +62,6 @@ struct unwind_hint { #define ANNOTYPE_INSTR_BEGIN 3 #define ANNOTYPE_INSTR_END 4 #define ANNOTYPE_UNRET_BEGIN 5 +#define ANNOTYPE_IGNORE_ALTS 6 #endif /* _LINUX_OBJTOOL_TYPES_H */ --- a/tools/include/linux/objtool_types.h +++ b/tools/include/linux/objtool_types.h @@ -62,5 +62,6 @@ struct unwind_hint { #define ANNOTYPE_INSTR_BEGIN 3 #define ANNOTYPE_INSTR_END 4 #define ANNOTYPE_UNRET_BEGIN 5 +#define ANNOTYPE_IGNORE_ALTS 6 #endif /* _LINUX_OBJTOOL_TYPES_H */ --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -1255,40 +1255,6 @@ static void add_uaccess_safe(struct objt } /* - * FIXME: For now, just ignore any alternatives which add retpolines. This is - * a temporary hack, as it doesn't allow ORC to unwind from inside a retpoline. - * But it at least allows objtool to understand the control flow *around* the - * retpoline. - */ -static int add_ignore_alternatives(struct objtool_file *file) -{ - struct section *rsec; - struct reloc *reloc; - struct instruction *insn; - - rsec = find_section_by_name(file->elf, ".rela.discard.ignore_alts"); - if (!rsec) - return 0; - - for_each_reloc(rsec, reloc) { - if (reloc->sym->type != STT_SECTION) { - WARN("unexpected relocation symbol type in %s", rsec->name); - return -1; - } - - insn = find_insn(file, reloc->sym->sec, reloc_addend(reloc)); - if (!insn) { - WARN("bad .discard.ignore_alts entry"); - return -1; - } - - insn->ignore_alts = true; - } - - return 0; -} - -/* * Symbols that replace INSN_CALL_DYNAMIC, every (tail) call to such a symbol * will be added to the .retpoline_sites section. */ @@ -2341,6 +2307,15 @@ static int read_annotate(struct objtool_ return 0; } +static int __annotate_ignore_alts(int type, struct instruction *insn) +{ + if (type != ANNOTYPE_IGNORE_ALTS) + return 0; + + insn->ignore_alts = true; + return 0; +} + static int __annotate_noendbr(int type, struct instruction *insn) { if (type != ANNOTYPE_NOENDBR) @@ -2550,7 +2525,7 @@ static int decode_sections(struct objtoo add_ignores(file); add_uaccess_safe(file); - ret = add_ignore_alternatives(file); + ret = read_annotate(file, __annotate_ignore_alts); if (ret) return ret; From patchwork Mon Nov 11 11:59:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13870715 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 064DF19CCEC; Mon, 11 Nov 2024 12:54:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731329678; cv=none; b=gHdeLoGtL2ejGdNS7Y0L/UjQNjaMZXj1cWfZQ0I74pOz3WJL44qqTYC60eyUA4OTHm+te3/jav39EVeFhWERNl1ARR8MfJeIUgu10jAVmlUPx4t6qI4qqqmz538ClUbohEOmzaf0/Ttqk3Fu/74Mti02fwZCKYjsMHcTZvU4qBI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731329678; c=relaxed/simple; bh=teh3Enru7lZzQgzh8lrSHN3JIulND/XbK060Y3FnIJ4=; h=Message-Id:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=sN0LuRJ4TFlwMMLvEjtXHRA2KFz7louEMxakLZ0YVALRMko8f9oiy291Rezq1pT5YOpk8tBlyOSvM69rmRIJnz5ieTHE7sxjKSAC/zZp4h/zZWauFbdokkIPuSdBMLhgrJ+eNQBRBtMPM0U5dNvNTU3ANzEQ+AFlh9CGYWD4T+Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=tCoslt4o; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="tCoslt4o" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=Jgweoy9v1ry+vJ+0xQvkdC/DCJR/HfQjV7GgwvonHBA=; b=tCoslt4oQvJBkKyqO2AKiyeXgM iCWODgLz3ZeKlKH/Toidjq+Tkfa5pe0ljO6URG9/rjg7c8XayO0akN7FWeVKUQgwGWXo6eKdCIH0C wGurOUbm5cNKepwIFtb1U3f/ivU9biYpUSejYADugUan+BlPT49uFTsTvYJAyGrMpl/z1pvhc1XQp 3AcCeJsKbCHfrFl7+ZTsqYokSji7jETIUUANDT3eCYM09yYQVnWRYx/+mnZtIZsL/3wCCuADmXFFR 9WCLpPvsBhAYbGSuI2W6ee5HVUJxa0wXxrDnIfOOeoi/Gon9EJilORKV+5Y+ZvCbucN6pSxajFwKG sq0tRYLQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tATwA-0000000Coep-1uVM; Mon, 11 Nov 2024 12:54:34 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id DDA8330122C; Mon, 11 Nov 2024 13:54:32 +0100 (CET) Message-Id: <20241111125218.788965667@infradead.org> User-Agent: quilt/0.65 Date: Mon, 11 Nov 2024 12:59:42 +0100 From: Peter Zijlstra To: seanjc@google.com, pbonzini@redhat.com, jpoimboe@redhat.com, tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, x86@kernel.org, kvm@vger.kernel.org, jthoughton@google.com, "Peter Zijlstra (Intel)" Subject: [PATCH v2 07/12] objtool: Convert ANNOTATE_INTRA_FUNCTION_CALLS to ANNOTATE References: <20241111115935.796797988@infradead.org> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Signed-off-by: Peter Zijlstra (Intel) --- include/linux/objtool.h | 16 ++---- include/linux/objtool_types.h | 1 tools/include/linux/objtool_types.h | 1 tools/objtool/check.c | 96 ++++++++++++++---------------------- 4 files changed, 47 insertions(+), 67 deletions(-) --- a/include/linux/objtool.h +++ b/include/linux/objtool.h @@ -66,16 +66,6 @@ #else /* __ASSEMBLY__ */ /* - * This macro indicates that the following intra-function call is valid. - * Any non-annotated intra-function call will cause objtool to issue a warning. - */ -#define ANNOTATE_INTRA_FUNCTION_CALL \ - 999: \ - .pushsection .discard.intra_function_calls; \ - .long 999b; \ - .popsection; - -/* * In asm, there are two kinds of code: normal C-type callable functions and * the rest. The normal callable functions can be called by other code, and * don't do anything unusual with the stack. Such normal callable functions @@ -152,6 +142,12 @@ #define ANNOTATE_NOENDBR ANNOTATE type=ANNOTYPE_NOENDBR +/* + * This macro indicates that the following intra-function call is valid. + * Any non-annotated intra-function call will cause objtool to issue a warning. + */ +#define ANNOTATE_INTRA_FUNCTION_CALL ANNOTATE type=ANNOTYPE_INTRA_FUNCTION_CALLS + #endif /* __ASSEMBLY__ */ #else /* !CONFIG_OBJTOOL */ --- a/include/linux/objtool_types.h +++ b/include/linux/objtool_types.h @@ -63,5 +63,6 @@ struct unwind_hint { #define ANNOTYPE_INSTR_END 4 #define ANNOTYPE_UNRET_BEGIN 5 #define ANNOTYPE_IGNORE_ALTS 6 +#define ANNOTYPE_INTRA_FUNCTION_CALLS 7 #endif /* _LINUX_OBJTOOL_TYPES_H */ --- a/tools/include/linux/objtool_types.h +++ b/tools/include/linux/objtool_types.h @@ -63,5 +63,6 @@ struct unwind_hint { #define ANNOTYPE_INSTR_END 4 #define ANNOTYPE_UNRET_BEGIN 5 #define ANNOTYPE_IGNORE_ALTS 6 +#define ANNOTYPE_INTRA_FUNCTION_CALLS 7 #endif /* _LINUX_OBJTOOL_TYPES_H */ --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -2274,7 +2274,8 @@ static int read_unwind_hints(struct objt return 0; } -static int read_annotate(struct objtool_file *file, int (*func)(int type, struct instruction *insn)) +static int read_annotate(struct objtool_file *file, + int (*func)(struct objtool_file *file, int type, struct instruction *insn)) { struct section *rsec, *sec; struct instruction *insn; @@ -2299,7 +2300,7 @@ static int read_annotate(struct objtool_ type = *(u32 *)(sec->data->d_buf + (reloc_idx(reloc) * sec->sh.sh_entsize) + 4); - ret = func(type, insn); + ret = func(file, type, insn); if (ret < 0) return ret; } @@ -2307,7 +2308,7 @@ static int read_annotate(struct objtool_ return 0; } -static int __annotate_ignore_alts(int type, struct instruction *insn) +static int __annotate_ignore_alts(struct objtool_file *file, int type, struct instruction *insn) { if (type != ANNOTYPE_IGNORE_ALTS) return 0; @@ -2316,7 +2317,7 @@ static int __annotate_ignore_alts(int ty return 0; } -static int __annotate_noendbr(int type, struct instruction *insn) +static int __annotate_noendbr(struct objtool_file *file, int type, struct instruction *insn) { if (type != ANNOTYPE_NOENDBR) return 0; @@ -2325,7 +2326,37 @@ static int __annotate_noendbr(int type, return 0; } -static int __annotate_retpoline_safe(int type, struct instruction *insn) +static int __annotate_ifc(struct objtool_file *file, int type, struct instruction *insn) +{ + unsigned long dest_off; + + if (type != ANNOTYPE_INTRA_FUNCTION_CALLS) + return 0; + + if (insn->type != INSN_CALL) { + WARN_INSN(insn, "intra_function_call not a direct call"); + return -1; + } + + /* + * Treat intra-function CALLs as JMPs, but with a stack_op. + * See add_call_destinations(), which strips stack_ops from + * normal CALLs. + */ + insn->type = INSN_JUMP_UNCONDITIONAL; + + dest_off = arch_jump_destination(insn); + insn->jump_dest = find_insn(file, insn->sec, dest_off); + if (!insn->jump_dest) { + WARN_INSN(insn, "can't find call dest at %s+0x%lx", + insn->sec->name, dest_off); + return -1; + } + + return 0; +} + +static int __annotate_retpoline_safe(struct objtool_file *file, int type, struct instruction *insn) { if (type != ANNOTYPE_RETPOLINE_SAFE) return 0; @@ -2342,7 +2373,7 @@ static int __annotate_retpoline_safe(int return 0; } -static int __annotate_instr(int type, struct instruction *insn) +static int __annotate_instr(struct objtool_file *file, int type, struct instruction *insn) { switch (type) { case ANNOTYPE_INSTR_BEGIN: @@ -2360,7 +2391,7 @@ static int __annotate_instr(int type, st return 0; } -static int __annotate_unret(int type, struct instruction *insn) +static int __annotate_unret(struct objtool_file *file, int type, struct instruction *insn) { if (type != ANNOTYPE_UNRET_BEGIN) return 0; @@ -2370,55 +2401,6 @@ static int __annotate_unret(int type, st } -static int read_intra_function_calls(struct objtool_file *file) -{ - struct instruction *insn; - struct section *rsec; - struct reloc *reloc; - - rsec = find_section_by_name(file->elf, ".rela.discard.intra_function_calls"); - if (!rsec) - return 0; - - for_each_reloc(rsec, reloc) { - unsigned long dest_off; - - if (reloc->sym->type != STT_SECTION) { - WARN("unexpected relocation symbol type in %s", - rsec->name); - return -1; - } - - insn = find_insn(file, reloc->sym->sec, reloc_addend(reloc)); - if (!insn) { - WARN("bad .discard.intra_function_call entry"); - return -1; - } - - if (insn->type != INSN_CALL) { - WARN_INSN(insn, "intra_function_call not a direct call"); - return -1; - } - - /* - * Treat intra-function CALLs as JMPs, but with a stack_op. - * See add_call_destinations(), which strips stack_ops from - * normal CALLs. - */ - insn->type = INSN_JUMP_UNCONDITIONAL; - - dest_off = arch_jump_destination(insn); - insn->jump_dest = find_insn(file, insn->sec, dest_off); - if (!insn->jump_dest) { - WARN_INSN(insn, "can't find call dest at %s+0x%lx", - insn->sec->name, dest_off); - return -1; - } - } - - return 0; -} - /* * Return true if name matches an instrumentation function, where calls to that * function from noinstr code can safely be removed, but compilers won't do so. @@ -2554,7 +2536,7 @@ static int decode_sections(struct objtoo * Must be before add_call_destination(); it changes INSN_CALL to * INSN_JUMP. */ - ret = read_intra_function_calls(file); + ret = read_annotate(file, __annotate_ifc); if (ret) return ret; From patchwork Mon Nov 11 11:59:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13870717 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3CD9C19CD0B; Mon, 11 Nov 2024 12:54:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731329679; cv=none; b=HpFqKAkeXF30StgKIYTpgO+slo0sUBWOIOMFEhLWCYD+umwW3w2nS6NiNCR/y2SMzoiRSFCinDhE7Dg/ezYgFpi0gXGPTON91w1VtQWdR1Q5fS/RZfg904nSlm6c84vdEWSYC/nBP5ENm4RCvZWobNGATUBKdp6p6dRUHQt850o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731329679; c=relaxed/simple; bh=jybyn6XZ2fRp0B9YYZfk+IZqXuh7HT6e1Vw11iBq/m4=; h=Message-Id:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=PLJRwlpNYJQX7lyWZlmApa7vHXGgEcJJ7QNq/dw5fveWCl5q4zUWXuFRsUjLksB6L7ofoSU9PQkyscSca7pV+GDYSjZMfe9obRh3Z7lrhOv2X84BBDayPn+TykLg5yLauKAlK/CnXsu/CMc+tpmzsvRLkViOY1K4P0U7a5mLJN4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=OMijxigB; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="OMijxigB" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=8ZmAUMpjWfI1B/Ma4euS1POKJrZM+zq/pwDtgORWvEc=; b=OMijxigB62AK9YJX8h2s49/609 SlOn+fs2Y+kKMXv5BRm155kzL7xXcU/z48hB8rjGwQocaEJZduO7lWfurmIeqU80zVz7mDLB8mJuW GMOUGZJPNYKvrn9J7Dy+zd5H0Y5tOGXlwxqQ/3/MShgOhzE5uxCzn/u8/BfuOWxspcGnRaujRjKK6 a8eg+5oeEvnx44Nsaf2jjNK3F/m3klo/j+XbLmxfVbuNWci0jMO3VeLVEn2iM6DYWZbGeAdJsIZpq lEjdp/LzqOlEjrz7WrGArJbWQxU6gD+/O0C6bZsdjZsYU+sLpfZ3GEJGK1aJhkFZsKAUqaUbMWsR5 KcLgfsBg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tATwA-0000000Cqjo-1utq; Mon, 11 Nov 2024 12:54:34 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id E19963021D3; Mon, 11 Nov 2024 13:54:32 +0100 (CET) Message-Id: <20241111125218.921110073@infradead.org> User-Agent: quilt/0.65 Date: Mon, 11 Nov 2024 12:59:43 +0100 From: Peter Zijlstra To: seanjc@google.com, pbonzini@redhat.com, jpoimboe@redhat.com, tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, x86@kernel.org, kvm@vger.kernel.org, jthoughton@google.com, "Peter Zijlstra (Intel)" Subject: [PATCH v2 08/12] objtool: Collapse annotate sequences References: <20241111115935.796797988@infradead.org> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Reduce read_annotate() runs by collapsing subsequent runs into a single call. Signed-off-by: Peter Zijlstra (Intel) --- tools/objtool/check.c | 87 ++++++++++++++++++-------------------------------- 1 file changed, 32 insertions(+), 55 deletions(-) --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -2308,21 +2308,24 @@ static int read_annotate(struct objtool_ return 0; } -static int __annotate_ignore_alts(struct objtool_file *file, int type, struct instruction *insn) +static int __annotate_early(struct objtool_file *file, int type, struct instruction *insn) { - if (type != ANNOTYPE_IGNORE_ALTS) - return 0; + switch (type) { + case ANNOTYPE_IGNORE_ALTS: + insn->ignore_alts = true; + break; - insn->ignore_alts = true; - return 0; -} + /* + * Must be before read_unwind_hints() since that needs insn->noendbr. + */ + case ANNOTYPE_NOENDBR: + insn->noendbr = 1; + break; -static int __annotate_noendbr(struct objtool_file *file, int type, struct instruction *insn) -{ - if (type != ANNOTYPE_NOENDBR) - return 0; + default: + break; + } - insn->noendbr = 1; return 0; } @@ -2356,26 +2359,21 @@ static int __annotate_ifc(struct objtool return 0; } -static int __annotate_retpoline_safe(struct objtool_file *file, int type, struct instruction *insn) +static int __annotate_late(struct objtool_file *file, int type, struct instruction *insn) { - if (type != ANNOTYPE_RETPOLINE_SAFE) - return 0; - - if (insn->type != INSN_JUMP_DYNAMIC && - insn->type != INSN_CALL_DYNAMIC && - insn->type != INSN_RETURN && - insn->type != INSN_NOP) { - WARN_INSN(insn, "retpoline_safe hint not an indirect jump/call/ret/nop"); - return -1; - } + switch (type) { + case ANNOTYPE_RETPOLINE_SAFE: + if (insn->type != INSN_JUMP_DYNAMIC && + insn->type != INSN_CALL_DYNAMIC && + insn->type != INSN_RETURN && + insn->type != INSN_NOP) { + WARN_INSN(insn, "retpoline_safe hint not an indirect jump/call/ret/nop"); + return -1; + } - insn->retpoline_safe = true; - return 0; -} + insn->retpoline_safe = true; + break; -static int __annotate_instr(struct objtool_file *file, int type, struct instruction *insn) -{ - switch (type) { case ANNOTYPE_INSTR_BEGIN: insn->instr++; break; @@ -2384,6 +2382,10 @@ static int __annotate_instr(struct objto insn->instr--; break; + case ANNOTYPE_UNRET_BEGIN: + insn->unret = 1; + break; + default: break; } @@ -2391,16 +2393,6 @@ static int __annotate_instr(struct objto return 0; } -static int __annotate_unret(struct objtool_file *file, int type, struct instruction *insn) -{ - if (type != ANNOTYPE_UNRET_BEGIN) - return 0; - - insn->unret = 1; - return 0; - -} - /* * Return true if name matches an instrumentation function, where calls to that * function from noinstr code can safely be removed, but compilers won't do so. @@ -2507,14 +2499,7 @@ static int decode_sections(struct objtoo add_ignores(file); add_uaccess_safe(file); - ret = read_annotate(file, __annotate_ignore_alts); - if (ret) - return ret; - - /* - * Must be before read_unwind_hints() since that needs insn->noendbr. - */ - ret = read_annotate(file, __annotate_noendbr); + ret = read_annotate(file, __annotate_early); if (ret) return ret; @@ -2560,15 +2545,7 @@ static int decode_sections(struct objtoo if (ret) return ret; - ret = read_annotate(file, __annotate_retpoline_safe); - if (ret) - return ret; - - ret = read_annotate(file, __annotate_instr); - if (ret) - return ret; - - ret = read_annotate(file, __annotate_unret); + ret = read_annotate(file, __annotate_late); if (ret) return ret; From patchwork Mon Nov 11 11:59:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13870708 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7853615853B; Mon, 11 Nov 2024 12:54:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731329678; cv=none; b=jLnuK1htfIjprsxY3/fs27yI81SeFNg06Mc/SCr2FzylImBcxZqIFjimqt1D59WIJ/skL5mxqg2FxbfsvrjSjRV60bsMTwlrXbyoXKGZ9cNj8pcojxHMexFFSg0SQMMH94zXMyoK5hfi4KellEpLf6APzf5gH8ZpUvD9WnO+CkI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731329678; c=relaxed/simple; bh=re7YlqIlSiYJgwFG46YU4SbL85meBq6Fe4HTgqcFdvk=; h=Message-Id:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=QfKa9jnJsBihnKM9g8/LdOe71RERj6FmiNdbfcM7Qhai0vxcVuJKkIZoXuPPlbCBZNG+K5aqzSx62bAQmthfYrUXHa9LvatrGunvFFtbB+irSysoRtlMt7BwQnzFE5yIINix3d8rRUwzTmfDRb5FDlyiRQLkYgDGMPaZeoIGYzI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=ibVwk2A9; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="ibVwk2A9" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=5G7tZh92XjzM00TzgErAKrW0ED3Ua0OvGoNXzUhcSY0=; b=ibVwk2A9uRdQ5Dm4vPQYhrvld0 L4VN0NGeWV0uHk24mDyhPoyF2y7YF6XhDybzQj/2zpEyqUgMYAv1KFMNpWCYayl+MLRsmBgNQkmSQ Pa9OY6HyVG3Oh79AuiMtkomYkteCcvz6gEcVNPresdIqJpZYjJE1FVfFAgmFz3NH5JBqkEwjx1mX3 +E9gANxKibfLsdY0fXtEwOk3BBb9tKlxggcBepR5VFEHnitlfywbFbKQqRkd5YlUvHKvquzl4u4cK KqDn8C+UOYarHprqt9AOxC2A0B3thkqtQa4axNRbob2iIFSxNOFEIEbA/qzD6D+SU0kb8gG1HR5Qz 3ejlvmIg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tATwA-0000000Coer-207x; Mon, 11 Nov 2024 12:54:34 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id E56AB3021DA; Mon, 11 Nov 2024 13:54:32 +0100 (CET) Message-Id: <20241111125219.033699387@infradead.org> User-Agent: quilt/0.65 Date: Mon, 11 Nov 2024 12:59:44 +0100 From: Peter Zijlstra To: seanjc@google.com, pbonzini@redhat.com, jpoimboe@redhat.com, tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, x86@kernel.org, kvm@vger.kernel.org, jthoughton@google.com, "Peter Zijlstra (Intel)" Subject: [PATCH v2 09/12] x86/nospec: JMP_NOSPEC References: <20241111115935.796797988@infradead.org> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/nospec-branch.h | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -403,6 +403,17 @@ static inline void call_depth_return_thu "call *%[thunk_target]\n", \ X86_FEATURE_RETPOLINE_LFENCE) +# define JMP_NOSPEC \ + ALTERNATIVE_2( \ + ANNOTATE_RETPOLINE_SAFE \ + "jmp *%[thunk_target]\n", \ + "jmp __x86_indirect_thunk_%V[thunk_target]\n", \ + X86_FEATURE_RETPOLINE, \ + "lfence;\n" \ + ANNOTATE_RETPOLINE_SAFE \ + "jmp *%[thunk_target]\n", \ + X86_FEATURE_RETPOLINE_LFENCE) + # define THUNK_TARGET(addr) [thunk_target] "r" (addr) #else /* CONFIG_X86_32 */ @@ -433,10 +444,31 @@ static inline void call_depth_return_thu "call *%[thunk_target]\n", \ X86_FEATURE_RETPOLINE_LFENCE) +# define JMP_NOSPEC \ + ALTERNATIVE_2( \ + ANNOTATE_RETPOLINE_SAFE \ + "jmp *%[thunk_target]\n", \ + " jmp 901f;\n" \ + " .align 16\n" \ + "901: call 903f;\n" \ + "902: pause;\n" \ + " lfence;\n" \ + " jmp 902b;\n" \ + " .align 16\n" \ + "903: lea 4(%%esp), %%esp;\n" \ + " pushl %[thunk_target];\n" \ + " ret;\n", \ + X86_FEATURE_RETPOLINE, \ + "lfence;\n" \ + ANNOTATE_RETPOLINE_SAFE \ + "jmp *%[thunk_target]\n", \ + X86_FEATURE_RETPOLINE_LFENCE) + # define THUNK_TARGET(addr) [thunk_target] "rm" (addr) #endif #else /* No retpoline for C / inline asm */ # define CALL_NOSPEC "call *%[thunk_target]\n" +# define JMP_NOSPEC "jmp *%[thunk_target]\n" # define THUNK_TARGET(addr) [thunk_target] "rm" (addr) #endif From patchwork Mon Nov 11 11:59:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13870718 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E4FA219CC22; Mon, 11 Nov 2024 12:54:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731329679; cv=none; b=EjezGO8zd7tGLkIZdalUHw0kyJuGTf6ObJiltdLJLYf3wagdvJR6DQAUQHMfkeOq9atOoG4WLY0Gj4ZGP2u6s7UhHCRG1IbpgRZwrSrBFJCSpq9fNQ+OGtUB7f+QpLx8mQHeHLBJ4L4/qnyVEAtFgLhX4wHR9l1y9Ue6fAYyqk4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731329679; c=relaxed/simple; bh=m4wNEd9VZkRxLFzfitrd/BccDMwMsl2i+sWH+ZJYUu8=; h=Message-Id:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=Z3PUyF+yTCxwd7QmRcFm9St+T6e/1e7nxSvxxOjhlDbDp04orY1oLoicpZBFFtpG0TTg8cKsrz9CB+ul5VU4dtsuHAz66Gjf6jiKSho2cUyjgfrAbMD8smqWOu5myGM5/bMNIcKqgS/QeIsqzwPEomn+Qv7QXSjTry0tNVDm1zo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=tKMoI+CP; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="tKMoI+CP" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=6BeHqEa2jz3IbAPf4WR7i0ahbBf1txiVvqLKV6vDwuM=; b=tKMoI+CP/PLxiqcur0M0UoG0pN xYxe/ni2cjlN/+mDic/H+036MOmGoS0rMmgC4CyxiRpz4slRc2GJj7BS7DqRU592VLyTselmcyqat bFGRHrM1uh5Gcxl0DmhUybDQI1cSQqTjF/MTcAclqKeOnK+mUgcyJ2KW2woK6AVqGOoWCn3wxrOuc OGdzlmA+09HDpJsmjzVBo3AIGrMI4leOt+TIX8n8TEodWuRol+vaNxTvjpjp2dKWFtcL5u3SK89Jk gsax5mCH5oTMS4vbH4qGZ/N3+cdb/TpROmlAl9ficPHXRevmGQHuT1pm0yz829b02yhaVyS7XfO0R AM46fd/Q==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tATwA-0000000Coes-1zxz; Mon, 11 Nov 2024 12:54:34 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id E98873021E5; Mon, 11 Nov 2024 13:54:32 +0100 (CET) Message-Id: <20241111125219.140262800@infradead.org> User-Agent: quilt/0.65 Date: Mon, 11 Nov 2024 12:59:45 +0100 From: Peter Zijlstra To: seanjc@google.com, pbonzini@redhat.com, jpoimboe@redhat.com, tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, x86@kernel.org, kvm@vger.kernel.org, jthoughton@google.com, "Peter Zijlstra (Intel)" Subject: [PATCH v2 10/12] x86,nospec: Simplify {JMP,CALL}_NOSPEC (part 2) References: <20241111115935.796797988@infradead.org> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Counterpart to 09d09531a51a ("x86,nospec: Simplify {JMP,CALL}_NOSPEC"), x86_64 will rewrite all this anyway, see apply_retpoline. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/nospec-branch.h | 29 +++++++++++------------------ 1 file changed, 11 insertions(+), 18 deletions(-) --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -429,31 +429,24 @@ static inline void call_depth_return_thu #ifdef CONFIG_X86_64 +#define __CS_PREFIX \ + ".irp rs,r8,r9,r10,r11,r12,r13,r14,r15\n" \ + ".ifc %V[thunk_target],\\rs\n" \ + ".byte 0x2e\n" \ + ".endif\n" \ + ".endr\n" + /* * Inline asm uses the %V modifier which is only in newer GCC * which is ensured when CONFIG_MITIGATION_RETPOLINE is defined. */ # define CALL_NOSPEC \ - ALTERNATIVE_2( \ - ANNOTATE_RETPOLINE_SAFE \ - "call *%[thunk_target]\n", \ - "call __x86_indirect_thunk_%V[thunk_target]\n", \ - X86_FEATURE_RETPOLINE, \ - "lfence;\n" \ - ANNOTATE_RETPOLINE_SAFE \ - "call *%[thunk_target]\n", \ - X86_FEATURE_RETPOLINE_LFENCE) + __CS_PREFIX \ + "call __x86_indirect_thunk_%V[thunk_target]\n" # define JMP_NOSPEC \ - ALTERNATIVE_2( \ - ANNOTATE_RETPOLINE_SAFE \ - "jmp *%[thunk_target]\n", \ - "jmp __x86_indirect_thunk_%V[thunk_target]\n", \ - X86_FEATURE_RETPOLINE, \ - "lfence;\n" \ - ANNOTATE_RETPOLINE_SAFE \ - "jmp *%[thunk_target]\n", \ - X86_FEATURE_RETPOLINE_LFENCE) + __CS_PREFIX \ + "jmp __x86_indirect_thunk_%V[thunk_target]\n" # define THUNK_TARGET(addr) [thunk_target] "r" (addr) From patchwork Mon Nov 11 11:59:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13870712 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 909D01850AF; Mon, 11 Nov 2024 12:54:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731329678; cv=none; b=bcm2TtidgbHk5Ern4qDBLvpWhgQ+/Bsr1hFw+FCVCXFkpuIuTIUZ4NlT6WK3DxPyG2OgLWn2duML+CHWxojs/wSenAwUhi0Cae6gA4ZBJdru3I2hv6DoyzcxDsXT9OVu/c2VwY52Vg/j2bPKDN+RYgI8hwXNbc4TRXfmqhmcD4o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731329678; c=relaxed/simple; bh=njWlbghBF0JH2VFA38VkKHvuHFbC5tXgU00ibS8oZCU=; h=Message-Id:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=H2EuOebJwqfrgbwB1mAIjsg1IAe6T++RA0o2TyMou4dDpPqBYvfnWn941jR19FBEb75TyMxSA6cW4M8X/tigWVOWh6/rt2YTYNE5oRxo5gRwBeMqNhle8qYRRrGDC/aHV52du0xqn2m8sHsXFmh3xsgOMLy47yp15ui7czoLfIs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=HWkCkwGw; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="HWkCkwGw" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=VD8wRttp+FatX9b+/O+oQABiC0DBtKQAPEWfVxMeFu0=; b=HWkCkwGwu54kQmqDRRp5thWhRy q/Cg0HbirAnYelPnYcOMQ7J4qfUpaRLqVAEPP55JHrFi63/arieiI0UUx8NS//qxe4fwFLOEefZX3 POv2oUk7HT7mp38LoXXiZFyJdHMoBdsi6b8t3iSnBzBvIethWw1VxSdY6AN3UjJOpokR4XzjzzrVS yp+B/oxlvvzNorpb0/VPCGL5PGf3vh/WzTZWeB+zpjMeas1i2GxQeI2sT7ZP8d13CIZqYPs1oT8lX EpUWofcELOTEgLn3GmMNcbyyL4if2ysXyiXqRjzMFPPZwW0jPyQU+gzPhJ93BY/a899OxzwmoT1DF DB8wySfw==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tATwA-0000000Cqjp-1yP1; Mon, 11 Nov 2024 12:54:34 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id ED37930221D; Mon, 11 Nov 2024 13:54:32 +0100 (CET) Message-Id: <20241111125219.248649120@infradead.org> User-Agent: quilt/0.65 Date: Mon, 11 Nov 2024 12:59:46 +0100 From: Peter Zijlstra To: seanjc@google.com, pbonzini@redhat.com, jpoimboe@redhat.com, tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, x86@kernel.org, kvm@vger.kernel.org, jthoughton@google.com, "Peter Zijlstra (Intel)" Subject: [PATCH v2 11/12] x86/kvm/emulate: Implement test_cc() in C References: <20241111115935.796797988@infradead.org> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Current test_cc() uses the fastop infrastructure to test flags using SETcc instructions. However, int3_emulate_jcc() already fully implements the flags->CC mapping, use that. Removes a pile of gnarly asm. Signed-off-by: Peter Zijlstra (Intel) Acked-by: Sean Christopherson --- arch/x86/include/asm/text-patching.h | 20 +++++++++++++------- arch/x86/kvm/emulate.c | 34 ++-------------------------------- 2 files changed, 15 insertions(+), 39 deletions(-) --- a/arch/x86/include/asm/text-patching.h +++ b/arch/x86/include/asm/text-patching.h @@ -176,9 +176,9 @@ void int3_emulate_ret(struct pt_regs *re } static __always_inline -void int3_emulate_jcc(struct pt_regs *regs, u8 cc, unsigned long ip, unsigned long disp) +bool __emulate_cc(unsigned long flags, u8 cc) { - static const unsigned long jcc_mask[6] = { + static const unsigned long cc_mask[6] = { [0] = X86_EFLAGS_OF, [1] = X86_EFLAGS_CF, [2] = X86_EFLAGS_ZF, @@ -191,15 +191,21 @@ void int3_emulate_jcc(struct pt_regs *re bool match; if (cc < 0xc) { - match = regs->flags & jcc_mask[cc >> 1]; + match = flags & cc_mask[cc >> 1]; } else { - match = ((regs->flags & X86_EFLAGS_SF) >> X86_EFLAGS_SF_BIT) ^ - ((regs->flags & X86_EFLAGS_OF) >> X86_EFLAGS_OF_BIT); + match = ((flags & X86_EFLAGS_SF) >> X86_EFLAGS_SF_BIT) ^ + ((flags & X86_EFLAGS_OF) >> X86_EFLAGS_OF_BIT); if (cc >= 0xe) - match = match || (regs->flags & X86_EFLAGS_ZF); + match = match || (flags & X86_EFLAGS_ZF); } - if ((match && !invert) || (!match && invert)) + return (match && !invert) || (!match && invert); +} + +static __always_inline +void int3_emulate_jcc(struct pt_regs *regs, u8 cc, unsigned long ip, unsigned long disp) +{ + if (__emulate_cc(regs->flags, cc)) ip += disp; int3_emulate_jmp(regs, ip); --- a/arch/x86/kvm/emulate.c +++ b/arch/x86/kvm/emulate.c @@ -26,6 +26,7 @@ #include #include #include +#include #include "x86.h" #include "tss.h" @@ -416,31 +417,6 @@ static int fastop(struct x86_emulate_ctx ON64(FOP3E(op##q, rax, rdx, cl)) \ FOP_END -/* Special case for SETcc - 1 instruction per cc */ -#define FOP_SETCC(op) \ - FOP_FUNC(op) \ - #op " %al \n\t" \ - FOP_RET(op) - -FOP_START(setcc) -FOP_SETCC(seto) -FOP_SETCC(setno) -FOP_SETCC(setc) -FOP_SETCC(setnc) -FOP_SETCC(setz) -FOP_SETCC(setnz) -FOP_SETCC(setbe) -FOP_SETCC(setnbe) -FOP_SETCC(sets) -FOP_SETCC(setns) -FOP_SETCC(setp) -FOP_SETCC(setnp) -FOP_SETCC(setl) -FOP_SETCC(setnl) -FOP_SETCC(setle) -FOP_SETCC(setnle) -FOP_END; - FOP_START(salc) FOP_FUNC(salc) "pushf; sbb %al, %al; popf \n\t" @@ -1064,13 +1040,7 @@ static int em_bsr_c(struct x86_emulate_c static __always_inline u8 test_cc(unsigned int condition, unsigned long flags) { - u8 rc; - void (*fop)(void) = (void *)em_setcc + FASTOP_SIZE * (condition & 0xf); - - flags = (flags & EFLAGS_MASK) | X86_EFLAGS_IF; - asm("push %[flags]; popf; " CALL_NOSPEC - : "=a"(rc), ASM_CALL_CONSTRAINT : [thunk_target]"r"(fop), [flags]"r"(flags)); - return rc; + return __emulate_cc(flags, condition & 0xf); } static void fetch_register_operand(struct operand *op) From patchwork Mon Nov 11 11:59:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13870714 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E34A119CC20; Mon, 11 Nov 2024 12:54:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731329679; cv=none; b=dav2UBF9ZUgDEqs8DzVOSyEHDm4ycGUX/5dDduwRz+UTWlFS2817GC/o50xL3XUKrVO8pq+7uwvY2i5BRjV9WCIY/XLftF9cqX9FSODGF+livEGlxGJc5WQPR7PZynFvE6MjBHQRBjVOY+QP02RPShiOtBJOxxlStut6KxqbsJ0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731329679; c=relaxed/simple; bh=kDi+TVCXm76xbQAD1/onHdkrODKXEEtiHXcBLawYGW4=; h=Message-Id:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=pjaoi/11X1/zt0OCW0idZmgOUPV9OBXcqRUrriYRzp8I8Y32fEw9i5UsjHaLbUL1xk27uNpDEcbkPoGzz7jDHdZkAbn1jJD6lQ40Xpj7DkwS9YfbbNWEDdwTuSNlK09Kt/NI6yOQ8+QuAeKKz5I5RBAucqzBoClK2Ii87tOSJe4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=WoxR2q26; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="WoxR2q26" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=cKtSfZ5sWaSTj7tZINYGDLIwyP/HVC5Nz5I81xWPCwk=; b=WoxR2q26i+JGKuI+u/OTnBS/As fN2FnAawEqX+BeWD0xU8yoKIHkTbWmmIgJNS0w0wfaX+hAMsSCpueVLcPkHDCLSCBTY7T9lzNugWl yBNCYhG9B6gpLKORP0MAw4J2cByVteC6b1QeZ6xT7/ZPP//bDHr16bWYcsEnoGHIx7lE+dxIqBU9h y6xnub51UBoGZ67ajic4ja9gVG3eRS1IUqNKXFDyhAu35eRydrLEkgoOl+ndQdKPzgeqpvtHvwlXX tBkjjrwbbbhug7Ix6u0q1XTf46MXqCT+bq+MUwAn2RqIr8heZZesK3QEp7zyRPyG+E0amY/eO4AcE h5kq3Z3A==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tATwA-0000000Coet-23KF; Mon, 11 Nov 2024 12:54:34 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id F102C302795; Mon, 11 Nov 2024 13:54:32 +0100 (CET) Message-Id: <20241111125219.361243118@infradead.org> User-Agent: quilt/0.65 Date: Mon, 11 Nov 2024 12:59:47 +0100 From: Peter Zijlstra To: seanjc@google.com, pbonzini@redhat.com, jpoimboe@redhat.com, tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, x86@kernel.org, kvm@vger.kernel.org, jthoughton@google.com, "Peter Zijlstra (Intel)" Subject: [PATCH v2 12/12] x86/kvm/emulate: Avoid RET for fastops References: <20241111115935.796797988@infradead.org> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Since there is only a single fastop() function, convert the FASTOP stuff from CALL_NOSPEC+RET to JMP_NOSPEC+JMP, avoiding the return thunks and all that jazz. Specifically FASTOPs rely on the return thunk to preserve EFLAGS, which not all of them can trivially do (call depth tracing suffers here). Objtool strenuously complains about this: - indirect call without a .rodata, fails to determine JUMP_TABLE, annotate - fastop functions fall through, exception - unreachable instruction after fastop_return, save/restore Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/kvm/emulate.c | 20 +++++++++++++++----- include/linux/objtool_types.h | 1 + tools/include/linux/objtool_types.h | 1 + tools/objtool/check.c | 11 ++++++++++- 4 files changed, 27 insertions(+), 6 deletions(-) --- a/arch/x86/kvm/emulate.c +++ b/arch/x86/kvm/emulate.c @@ -285,8 +285,8 @@ static void invalidate_registers(struct * different operand sizes can be reached by calculation, rather than a jump * table (which would be bigger than the code). * - * The 16 byte alignment, considering 5 bytes for the RET thunk, 3 for ENDBR - * and 1 for the straight line speculation INT3, leaves 7 bytes for the + * The 16 byte alignment, considering 5 bytes for the JMP, 4 for ENDBR + * and 1 for the straight line speculation INT3, leaves 6 bytes for the * body of the function. Currently none is larger than 4. */ static int fastop(struct x86_emulate_ctxt *ctxt, fastop_t fop); @@ -304,7 +304,7 @@ static int fastop(struct x86_emulate_ctx __FOP_FUNC(#name) #define __FOP_RET(name) \ - "11: " ASM_RET \ + "11: jmp fastop_return; int3 \n\t" \ ".size " name ", .-" name "\n\t" #define FOP_RET(name) \ @@ -5071,14 +5071,24 @@ static void fetch_possible_mmx_operand(s kvm_read_mmx_reg(op->addr.mm, &op->mm_val); } -static int fastop(struct x86_emulate_ctxt *ctxt, fastop_t fop) +/* + * All the FASTOP magic above relies on there being *one* instance of this + * so it can JMP back, avoiding RET and it's various thunks. + */ +static noinline int fastop(struct x86_emulate_ctxt *ctxt, fastop_t fop) { ulong flags = (ctxt->eflags & EFLAGS_MASK) | X86_EFLAGS_IF; if (!(ctxt->d & ByteOp)) fop += __ffs(ctxt->dst.bytes) * FASTOP_SIZE; - asm("push %[flags]; popf; " CALL_NOSPEC " ; pushf; pop %[flags]\n" + asm("push %[flags]; popf \n\t" + UNWIND_HINT(UNWIND_HINT_TYPE_SAVE, 0, 0, 0) + ASM_ANNOTATE(ANNOTYPE_JUMP_TABLE) + JMP_NOSPEC + "fastop_return: \n\t" + UNWIND_HINT(UNWIND_HINT_TYPE_RESTORE, 0, 0, 0) + "pushf; pop %[flags]\n" : "+a"(ctxt->dst.val), "+d"(ctxt->src.val), [flags]"+D"(flags), [thunk_target]"+S"(fop), ASM_CALL_CONSTRAINT : "c"(ctxt->src2.val)); --- a/include/linux/objtool_types.h +++ b/include/linux/objtool_types.h @@ -64,5 +64,6 @@ struct unwind_hint { #define ANNOTYPE_UNRET_BEGIN 5 #define ANNOTYPE_IGNORE_ALTS 6 #define ANNOTYPE_INTRA_FUNCTION_CALLS 7 +#define ANNOTYPE_JUMP_TABLE 8 #endif /* _LINUX_OBJTOOL_TYPES_H */ --- a/tools/include/linux/objtool_types.h +++ b/tools/include/linux/objtool_types.h @@ -64,5 +64,6 @@ struct unwind_hint { #define ANNOTYPE_UNRET_BEGIN 5 #define ANNOTYPE_IGNORE_ALTS 6 #define ANNOTYPE_INTRA_FUNCTION_CALLS 7 +#define ANNOTYPE_JUMP_TABLE 8 #endif /* _LINUX_OBJTOOL_TYPES_H */ --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -2386,6 +2386,14 @@ static int __annotate_late(struct objtoo insn->unret = 1; break; + /* + * Must be after add_jump_table(); for it doesn't set a sane + * _jump_table value. + */ + case ANNOTYPE_JUMP_TABLE: + insn->_jump_table = (void *)1; + break; + default: break; } @@ -3459,7 +3467,8 @@ static int validate_branch(struct objtoo if (func && insn_func(insn) && func != insn_func(insn)->pfunc) { /* Ignore KCFI type preambles, which always fall through */ if (!strncmp(func->name, "__cfi_", 6) || - !strncmp(func->name, "__pfx_", 6)) + !strncmp(func->name, "__pfx_", 6) || + !strcmp(insn_func(insn)->name, "fastop")) return 0; WARN("%s() falls through to next function %s()",