From patchwork Wed Oct 4 21:19:56 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Garnier X-Patchwork-Id: 9985551 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id A308760586 for ; Wed, 4 Oct 2017 21:23:26 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 924C828C2C for ; Wed, 4 Oct 2017 21:23:26 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 85B8B28C2E; Wed, 4 Oct 2017 21:23:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=2.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, RCVD_IN_DNSWL_MED, RCVD_IN_SORBS_SPAM, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 7BF8F28C2F for ; Wed, 4 Oct 2017 21:23:25 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dzr6g-0003Nq-7z; Wed, 04 Oct 2017 21:21:30 +0000 Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dzr6e-0003Kw-NZ for xen-devel@lists.xenproject.org; Wed, 04 Oct 2017 21:21:28 +0000 Received: from [85.158.143.35] by server-3.bemta-6.messagelabs.com id 7E/12-03093-8D055D95; Wed, 04 Oct 2017 21:21:28 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrIIsWRWlGSWpSXmKPExsXiVRusr3sl4Gq kQe8lPYvvWyYzOTB6HP5whSWAMYo1My8pvyKBNWP7vfSCtU2MFRcP6TUwXsvpYuTiEBKYwSjx 7+MxRhCHReAVi8TFE+2sII6EQD+rxLzWp+xdjJxATpbE9e75LF2MHEB2msSq72oQZrnElP3xI BVCAkoSWzcsZYYY+pdR4mb/XiaQBJuAlsSehvlMIAkRgVkiErde97CBOMwCZ5gkjvedZAWpEh bwktiy9iwLiM0ioCqx5eUBsG5eAUuJ5rvfGCGOMJF4eOQaWD0nUPzIiTfMEKstJFrP3GeawCi 4gJFhFaN6cWpRWWqRrrFeUlFmekZJbmJmjq6hgZlebmpxcWJ6ak5iUrFecn7uJkZgwDEAwQ7G jn9OhxglOZiURHk3+V6NFOJLyk+pzEgszogvKs1JLT7EKMPBoSTBGwAMYCHBotT01Iq0zBxg6 MOkJTh4lER4+UHSvMUFibnFmekQqVOMxhz79tz6w8TRcfPuHyYhlrz8vFQpcd7N/kClAiClGa V5cINgMXmJUVZKmJcR6DQhnoLUotzMElT5V4ziHIxKwryyIAt5MvNK4Pa9AjqFCeiUOU1XQE4 pSURISTUwrrrMmxXmaeu2/7LYbrmPqxi7WYyPe5be5qg9zBUUsGKJVSVz6P3gYG7+yxJ3hRaW N/1k3+Hhf/p32GzNdF6nulUmZ5zStXjbuH+G9sdt/K++6V1TbrwrU90+h8P6M1/lruye+uzSz 88T9prEn5P+/O2RTMRWz+k7lzT0RE36ddB3+6ctBQ2nlViKMxINtZiLihMB7SPMwcQCAAA= X-Env-Sender: thgarnie@google.com X-Msg-Ref: server-15.tower-21.messagelabs.com!1507152083!76237966!1 X-Originating-IP: [74.125.83.47] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 9.4.45; banners=-,-,- X-VirusChecked: Checked Received: (qmail 19783 invoked from network); 4 Oct 2017 21:21:24 -0000 Received: from mail-pg0-f47.google.com (HELO mail-pg0-f47.google.com) (74.125.83.47) by server-15.tower-21.messagelabs.com with AES128-GCM-SHA256 encrypted SMTP; 4 Oct 2017 21:21:24 -0000 Received: by mail-pg0-f47.google.com with SMTP id j70so7163727pgc.10 for ; Wed, 04 Oct 2017 14:21:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=mOfT8muX75rkB7Ny2s9Dkoc37g8kZoDDxOwCk7TBrj4=; b=NfH1jMq0v6V1y5Qsxv/umrYhXd2/Zdo9UN3shYBkJ4PZWMpJA7bXN/L5YfvPRZbLoy h86X1N2pxZZbRgOr4H4T+EnoZIuM2ngaz87AeVlbNeX4jMZEJ/2h9ACKKWZayoAkhZ0M mSjFruNlb0tgfFfs7RVbL2OBZ0OVnLHCCQ7xZQGoC11fvsAfl6JGwJ0z7v/ox/Udx5t1 SM1QfNqbrSgl58PSpIEYP5NnO2ILvWg988IdLPr0cGs/cX7ID0DOWCgyHAQaat0/15vS LmSjBM4VQGXAYborzHfjDOXn3ZuFEUttUKvgK7mrGb07j/J8j5+gEx5ws+bEWM2NmjAa udSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=mOfT8muX75rkB7Ny2s9Dkoc37g8kZoDDxOwCk7TBrj4=; b=D7Xc79bjApPIPdBI0bnTmsi9RG981Atlk5fJzFq//R5bTbypIMxLeSi5JmickfHxKM 10gsXtMm3OhjiYLGrVWHfm6wC/ZKDh3k6Az86DzVB7AAUNfHUlrsk6nKM/hnWPmuCJ7X /nVhfS2qfFM64e96RS9wn2tFojosNNY70ltIJNdH95HjJqqNYAVUloCIi+WqdmTJRbWv kK7FmMemJZbuLQtLrV2LF5kLm+klFGo+NU6i3wHJjWu6Ub2q5vesxtjBy43kxAnvG86v sk8TyVGNJopsHFsWZa04vdUPWmPH4RkWCzPhPREHa4N0HsZU/5HEBVur/RLelj43FF97 qhig== X-Gm-Message-State: AMCzsaUozR+9AwrrfkcrLbjix6RRnZ9HoulYVMk9VoksMa/tdmn1JadH At0LDvLLVmNr00G8XlaUyfDsKA== X-Google-Smtp-Source: AOwi7QA2beO7hensj0vl6qmFf/Lk91DrUg4n08DkaalcKtxk/9zKYMqJeIdLdYRe89pzG+qliLpX6w== X-Received: by 10.98.15.211 with SMTP id 80mr7809918pfp.24.1507152081511; Wed, 04 Oct 2017 14:21:21 -0700 (PDT) Received: from skynet.sea.corp.google.com ([172.31.92.33]) by smtp.gmail.com with ESMTPSA id s68sm30204019pfd.72.2017.10.04.14.21.20 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 04 Oct 2017 14:21:20 -0700 (PDT) From: Thomas Garnier To: Herbert Xu , "David S . Miller" , Thomas Gleixner , Ingo Molnar , "H . Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Thomas Garnier , Arnd Bergmann , Kees Cook , Matthias Kaehlcke , Tom Lendacky , Andy Lutomirski , "Kirill A . Shutemov" , Borislav Petkov , "Rafael J . Wysocki" , Len Brown , Pavel Machek , Juergen Gross , Chris Wright , Alok Kataria , Rusty Russell , Tejun Heo , Christoph Lameter , Boris Ostrovsky , Alexey Dobriyan , Andrew Morton , Paul Gortmaker , Chris Metcalf , "Paul E . McKenney" , Nicolas Pitre , Borislav Petkov , "Luis R . Rodriguez" , Greg Kroah-Hartman , Christopher Li , Steven Rostedt , Jason Baron , Dou Liyang , "Rafael J . Wysocki" , Mika Westerberg , Lukas Wunner , Masahiro Yamada , Alexei Starovoitov , Daniel Borkmann , Markus Trippelsdorf , Paolo Bonzini , =?UTF-8?q?Radim=20Kr=C4=8Dm=C3=A1=C5=99?= , Joerg Roedel , Rik van Riel , David Howells , Ard Biesheuvel , Waiman Long , Kyle Huey , Andrey Ryabinin , Jonathan Corbet , Matthew Wilcox , Michal Hocko , Peter Foley , Paul Bolle , Jiri Kosina , Rob Landley , "H . J . Lu" , Baoquan He , =?UTF-8?q?Jan=20H=20=2E=20Sch=C3=B6nherr?= , Daniel Micay Date: Wed, 4 Oct 2017 14:19:56 -0700 Message-Id: <20171004212003.28296-21-thgarnie@google.com> X-Mailer: git-send-email 2.14.2.920.gcf0c67979c-goog In-Reply-To: <20171004212003.28296-1-thgarnie@google.com> References: <20171004212003.28296-1-thgarnie@google.com> Cc: linux-arch@vger.kernel.org, kvm@vger.kernel.org, linux-pm@vger.kernel.org, x86@kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-sparse@vger.kernel.org, linux-crypto@vger.kernel.org, kernel-hardening@lists.openwall.com, xen-devel@lists.xenproject.org Subject: [Xen-devel] [RFC v3 20/27] x86/ftrace: Adapt function tracing for PIE support X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP When using -fPIE/PIC with function tracing, the compiler generates a call through the GOT (call *__fentry__@GOTPCREL). This instruction takes 6 bytes instead of 5 on the usual relative call. With this change, function tracing supports 6 bytes on traceable function and can still replace relative calls on the ftrace assembly functions. Position Independent Executable (PIE) support will allow to extended the KASLR randomization range below the -2G memory limit. Signed-off-by: Thomas Garnier --- arch/x86/include/asm/ftrace.h | 23 +++++- arch/x86/include/asm/sections.h | 4 + arch/x86/kernel/ftrace.c | 168 ++++++++++++++++++++++++++-------------- arch/x86/kernel/module.lds | 3 + 4 files changed, 139 insertions(+), 59 deletions(-) create mode 100644 arch/x86/kernel/module.lds diff --git a/arch/x86/include/asm/ftrace.h b/arch/x86/include/asm/ftrace.h index eccd0ac6bc38..b8bbcc7fad7f 100644 --- a/arch/x86/include/asm/ftrace.h +++ b/arch/x86/include/asm/ftrace.h @@ -1,6 +1,7 @@ #ifndef _ASM_X86_FTRACE_H #define _ASM_X86_FTRACE_H + #ifdef CONFIG_FUNCTION_TRACER #ifdef CC_USING_FENTRY # define MCOUNT_ADDR ((unsigned long)(__fentry__)) @@ -8,7 +9,19 @@ # define MCOUNT_ADDR ((unsigned long)(mcount)) # define HAVE_FUNCTION_GRAPH_FP_TEST #endif -#define MCOUNT_INSN_SIZE 5 /* sizeof mcount call */ + +#define MCOUNT_RELINSN_SIZE 5 /* sizeof relative (call or jump) */ +#define MCOUNT_GOTCALL_SIZE 6 /* sizeof call *got */ + +/* + * MCOUNT_INSN_SIZE is the highest size of instructions based on the + * configuration. + */ +#ifdef CONFIG_X86_PIE +#define MCOUNT_INSN_SIZE MCOUNT_GOTCALL_SIZE +#else +#define MCOUNT_INSN_SIZE MCOUNT_RELINSN_SIZE +#endif #ifdef CONFIG_DYNAMIC_FTRACE #define ARCH_SUPPORTS_FTRACE_OPS 1 @@ -17,6 +30,8 @@ #define HAVE_FUNCTION_GRAPH_RET_ADDR_PTR #ifndef __ASSEMBLY__ +#include + extern void mcount(void); extern atomic_t modifying_ftrace_code; extern void __fentry__(void); @@ -24,9 +39,11 @@ extern void __fentry__(void); static inline unsigned long ftrace_call_adjust(unsigned long addr) { /* - * addr is the address of the mcount call instruction. - * recordmcount does the necessary offset calculation. + * addr is the address of the mcount call instruction. PIE has always a + * byte added to the start of the function. */ + if (IS_ENABLED(CONFIG_X86_PIE)) + addr -= 1; return addr; } diff --git a/arch/x86/include/asm/sections.h b/arch/x86/include/asm/sections.h index 2f75f30cb2f6..6b2d496cf1aa 100644 --- a/arch/x86/include/asm/sections.h +++ b/arch/x86/include/asm/sections.h @@ -11,4 +11,8 @@ extern struct exception_table_entry __stop___ex_table[]; extern char __end_rodata_hpage_align[]; #endif +#if defined(CONFIG_X86_PIE) +extern char __start_got[], __end_got[]; +#endif + #endif /* _ASM_X86_SECTIONS_H */ diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c index 9bef1bbeba63..41d8c4c4306d 100644 --- a/arch/x86/kernel/ftrace.c +++ b/arch/x86/kernel/ftrace.c @@ -58,12 +58,17 @@ static int ftrace_calc_offset(long ip, long addr) return (int)(addr - ip); } -static unsigned char *ftrace_call_replace(unsigned long ip, unsigned long addr) +static unsigned char *ftrace_call_replace(unsigned long ip, unsigned long addr, + unsigned int size) { static union ftrace_code_union calc; + /* On PIE, fill the rest of the buffer with nops */ + if (IS_ENABLED(CONFIG_X86_PIE)) + memset(calc.code, ideal_nops[1][0], sizeof(calc.code)); + calc.e8 = 0xe8; - calc.offset = ftrace_calc_offset(ip + MCOUNT_INSN_SIZE, addr); + calc.offset = ftrace_calc_offset(ip + MCOUNT_RELINSN_SIZE, addr); /* * No locking needed, this must be called via kstop_machine @@ -72,6 +77,44 @@ static unsigned char *ftrace_call_replace(unsigned long ip, unsigned long addr) return calc.code; } +#ifdef CONFIG_X86_PIE +union ftrace_code_got_union { + char code[MCOUNT_INSN_SIZE]; + struct { + unsigned short ff15; + int offset; + } __attribute__((packed)); +}; + +/* Used to identify a mcount GOT call on PIE */ +static unsigned char *ftrace_original_call(struct module* mod, unsigned long ip, + unsigned long addr, + unsigned int size) +{ + static union ftrace_code_got_union calc; + unsigned long gotaddr; + + calc.ff15 = 0x15ff; + + gotaddr = module_find_got_entry(mod, addr); + if (!gotaddr) { + pr_err("Failed to find GOT entry for 0x%lx\n", addr); + return NULL; + } + + calc.offset = ftrace_calc_offset(ip + MCOUNT_GOTCALL_SIZE, gotaddr); + return calc.code; +} +#else +static unsigned char *ftrace_original_call(struct module* mod, unsigned long ip, + unsigned long addr, + unsigned int size) +{ + return ftrace_call_replace(ip, addr, size); +} + +#endif + static inline int within(unsigned long addr, unsigned long start, unsigned long end) { @@ -94,16 +137,18 @@ static unsigned long text_ip_addr(unsigned long ip) return ip; } -static const unsigned char *ftrace_nop_replace(void) +static const unsigned char *ftrace_nop_replace(unsigned int size) { - return ideal_nops[NOP_ATOMIC5]; + return ideal_nops[size == 5 ? NOP_ATOMIC5 : size]; } static int -ftrace_modify_code_direct(unsigned long ip, unsigned const char *old_code, - unsigned const char *new_code) +ftrace_modify_code_direct(struct dyn_ftrace *rec, unsigned const char *old_code, + unsigned const char *new_code) { unsigned char replaced[MCOUNT_INSN_SIZE]; + unsigned long ip = rec->ip; + unsigned int size = MCOUNT_INSN_SIZE; ftrace_expected = old_code; @@ -116,17 +161,17 @@ ftrace_modify_code_direct(unsigned long ip, unsigned const char *old_code, */ /* read the text we want to modify */ - if (probe_kernel_read(replaced, (void *)ip, MCOUNT_INSN_SIZE)) + if (probe_kernel_read(replaced, (void *)ip, size)) return -EFAULT; /* Make sure it is what we expect it to be */ - if (memcmp(replaced, old_code, MCOUNT_INSN_SIZE) != 0) + if (memcmp(replaced, old_code, size) != 0) return -EINVAL; ip = text_ip_addr(ip); /* replace the text with the new text */ - if (probe_kernel_write((void *)ip, new_code, MCOUNT_INSN_SIZE)) + if (probe_kernel_write((void *)ip, new_code, size)) return -EPERM; sync_core(); @@ -139,9 +184,7 @@ int ftrace_make_nop(struct module *mod, { unsigned const char *new, *old; unsigned long ip = rec->ip; - - old = ftrace_call_replace(ip, addr); - new = ftrace_nop_replace(); + unsigned int size = MCOUNT_INSN_SIZE; /* * On boot up, and when modules are loaded, the MCOUNT_ADDR @@ -151,14 +194,20 @@ int ftrace_make_nop(struct module *mod, * We do not want to use the breakpoint version in this case, * just modify the code directly. */ - if (addr == MCOUNT_ADDR) - return ftrace_modify_code_direct(rec->ip, old, new); + if (addr != MCOUNT_ADDR) { + ftrace_expected = NULL; - ftrace_expected = NULL; + /* Normal cases use add_brk_on_nop */ + WARN_ONCE(1, "invalid use of ftrace_make_nop"); + return -EINVAL; + } - /* Normal cases use add_brk_on_nop */ - WARN_ONCE(1, "invalid use of ftrace_make_nop"); - return -EINVAL; + old = ftrace_original_call(mod, ip, addr, size); + if (!old) + return -EINVAL; + new = ftrace_nop_replace(size); + + return ftrace_modify_code_direct(rec, old, new); } int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) @@ -166,11 +215,11 @@ int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) unsigned const char *new, *old; unsigned long ip = rec->ip; - old = ftrace_nop_replace(); - new = ftrace_call_replace(ip, addr); + old = ftrace_nop_replace(MCOUNT_INSN_SIZE); + new = ftrace_call_replace(ip, addr, MCOUNT_INSN_SIZE); /* Should only be called when module is loaded */ - return ftrace_modify_code_direct(rec->ip, old, new); + return ftrace_modify_code_direct(rec, old, new); } /* @@ -233,7 +282,7 @@ static int update_ftrace_func(unsigned long ip, void *new) unsigned char old[MCOUNT_INSN_SIZE]; int ret; - memcpy(old, (void *)ip, MCOUNT_INSN_SIZE); + memcpy(old, (void *)ip, MCOUNT_RELINSN_SIZE); ftrace_update_func = ip; /* Make sure the breakpoints see the ftrace_update_func update */ @@ -255,13 +304,14 @@ int ftrace_update_ftrace_func(ftrace_func_t func) unsigned char *new; int ret; - new = ftrace_call_replace(ip, (unsigned long)func); + new = ftrace_call_replace(ip, (unsigned long)func, MCOUNT_RELINSN_SIZE); ret = update_ftrace_func(ip, new); /* Also update the regs callback function */ if (!ret) { ip = (unsigned long)(&ftrace_regs_call); - new = ftrace_call_replace(ip, (unsigned long)func); + new = ftrace_call_replace(ip, (unsigned long)func, + MCOUNT_RELINSN_SIZE); ret = update_ftrace_func(ip, new); } @@ -309,18 +359,18 @@ static int ftrace_write(unsigned long ip, const char *val, int size) return 0; } -static int add_break(unsigned long ip, const char *old) +static int add_break(unsigned long ip, const char *old, unsigned int size) { unsigned char replaced[MCOUNT_INSN_SIZE]; unsigned char brk = BREAKPOINT_INSTRUCTION; - if (probe_kernel_read(replaced, (void *)ip, MCOUNT_INSN_SIZE)) + if (probe_kernel_read(replaced, (void *)ip, size)) return -EFAULT; ftrace_expected = old; /* Make sure it is what we expect it to be */ - if (memcmp(replaced, old, MCOUNT_INSN_SIZE) != 0) + if (memcmp(replaced, old, size) != 0) return -EINVAL; return ftrace_write(ip, &brk, 1); @@ -330,20 +380,22 @@ static int add_brk_on_call(struct dyn_ftrace *rec, unsigned long addr) { unsigned const char *old; unsigned long ip = rec->ip; + unsigned int size = MCOUNT_INSN_SIZE; - old = ftrace_call_replace(ip, addr); + old = ftrace_call_replace(ip, addr, size); - return add_break(rec->ip, old); + return add_break(rec->ip, old, size); } static int add_brk_on_nop(struct dyn_ftrace *rec) { unsigned const char *old; + unsigned int size = MCOUNT_INSN_SIZE; - old = ftrace_nop_replace(); + old = ftrace_nop_replace(size); - return add_break(rec->ip, old); + return add_break(rec->ip, old, size); } static int add_breakpoints(struct dyn_ftrace *rec, int enable) @@ -386,22 +438,23 @@ static int remove_breakpoint(struct dyn_ftrace *rec) const unsigned char *nop; unsigned long ftrace_addr; unsigned long ip = rec->ip; + unsigned int size = MCOUNT_INSN_SIZE; /* If we fail the read, just give up */ - if (probe_kernel_read(ins, (void *)ip, MCOUNT_INSN_SIZE)) + if (probe_kernel_read(ins, (void *)ip, size)) return -EFAULT; /* If this does not have a breakpoint, we are done */ if (ins[0] != brk) return 0; - nop = ftrace_nop_replace(); + nop = ftrace_nop_replace(size); /* * If the last 4 bytes of the instruction do not match * a nop, then we assume that this is a call to ftrace_addr. */ - if (memcmp(&ins[1], &nop[1], MCOUNT_INSN_SIZE - 1) != 0) { + if (memcmp(&ins[1], &nop[1], size - 1) != 0) { /* * For extra paranoidism, we check if the breakpoint is on * a call that would actually jump to the ftrace_addr. @@ -409,18 +462,18 @@ static int remove_breakpoint(struct dyn_ftrace *rec) * a disaster. */ ftrace_addr = ftrace_get_addr_new(rec); - nop = ftrace_call_replace(ip, ftrace_addr); + nop = ftrace_call_replace(ip, ftrace_addr, size); - if (memcmp(&ins[1], &nop[1], MCOUNT_INSN_SIZE - 1) == 0) + if (memcmp(&ins[1], &nop[1], size - 1) == 0) goto update; /* Check both ftrace_addr and ftrace_old_addr */ ftrace_addr = ftrace_get_addr_curr(rec); - nop = ftrace_call_replace(ip, ftrace_addr); + nop = ftrace_call_replace(ip, ftrace_addr, size); ftrace_expected = nop; - if (memcmp(&ins[1], &nop[1], MCOUNT_INSN_SIZE - 1) != 0) + if (memcmp(&ins[1], &nop[1], size - 1) != 0) return -EINVAL; } @@ -428,30 +481,33 @@ static int remove_breakpoint(struct dyn_ftrace *rec) return ftrace_write(ip, nop, 1); } -static int add_update_code(unsigned long ip, unsigned const char *new) +static int add_update_code(unsigned long ip, unsigned const char *new, + unsigned int size) { /* skip breakpoint */ ip++; new++; - return ftrace_write(ip, new, MCOUNT_INSN_SIZE - 1); + return ftrace_write(ip, new, size - 1); } static int add_update_call(struct dyn_ftrace *rec, unsigned long addr) { unsigned long ip = rec->ip; + unsigned int size = MCOUNT_INSN_SIZE; unsigned const char *new; - new = ftrace_call_replace(ip, addr); - return add_update_code(ip, new); + new = ftrace_call_replace(ip, addr, size); + return add_update_code(ip, new, size); } static int add_update_nop(struct dyn_ftrace *rec) { unsigned long ip = rec->ip; + unsigned int size = MCOUNT_INSN_SIZE; unsigned const char *new; - new = ftrace_nop_replace(); - return add_update_code(ip, new); + new = ftrace_nop_replace(size); + return add_update_code(ip, new, size); } static int add_update(struct dyn_ftrace *rec, int enable) @@ -485,7 +541,7 @@ static int finish_update_call(struct dyn_ftrace *rec, unsigned long addr) unsigned long ip = rec->ip; unsigned const char *new; - new = ftrace_call_replace(ip, addr); + new = ftrace_call_replace(ip, addr, MCOUNT_INSN_SIZE); return ftrace_write(ip, new, 1); } @@ -495,7 +551,7 @@ static int finish_update_nop(struct dyn_ftrace *rec) unsigned long ip = rec->ip; unsigned const char *new; - new = ftrace_nop_replace(); + new = ftrace_nop_replace(MCOUNT_INSN_SIZE); return ftrace_write(ip, new, 1); } @@ -619,13 +675,13 @@ ftrace_modify_code(unsigned long ip, unsigned const char *old_code, { int ret; - ret = add_break(ip, old_code); + ret = add_break(ip, old_code, MCOUNT_RELINSN_SIZE); if (ret) goto out; run_sync(); - ret = add_update_code(ip, new_code); + ret = add_update_code(ip, new_code, MCOUNT_RELINSN_SIZE); if (ret) goto fail_update; @@ -670,7 +726,7 @@ static unsigned char *ftrace_jmp_replace(unsigned long ip, unsigned long addr) /* Jmp not a call (ignore the .e8) */ calc.e8 = 0xe9; - calc.offset = ftrace_calc_offset(ip + MCOUNT_INSN_SIZE, addr); + calc.offset = ftrace_calc_offset(ip + MCOUNT_RELINSN_SIZE, addr); /* * ftrace external locks synchronize the access to the static variable. @@ -766,11 +822,11 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) * the jmp to ftrace_epilogue, as well as the address of * the ftrace_ops this trampoline is used for. */ - trampoline = alloc_tramp(size + MCOUNT_INSN_SIZE + sizeof(void *)); + trampoline = alloc_tramp(size + MCOUNT_RELINSN_SIZE + sizeof(void *)); if (!trampoline) return 0; - *tramp_size = size + MCOUNT_INSN_SIZE + sizeof(void *); + *tramp_size = size + MCOUNT_RELINSN_SIZE + sizeof(void *); /* Copy ftrace_caller onto the trampoline memory */ ret = probe_kernel_read(trampoline, (void *)start_offset, size); @@ -783,7 +839,7 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) /* The trampoline ends with a jmp to ftrace_epilogue */ jmp = ftrace_jmp_replace(ip, (unsigned long)ftrace_epilogue); - memcpy(trampoline + size, jmp, MCOUNT_INSN_SIZE); + memcpy(trampoline + size, jmp, MCOUNT_RELINSN_SIZE); /* * The address of the ftrace_ops that is used for this trampoline @@ -793,7 +849,7 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) * the global function_trace_op variable. */ - ptr = (unsigned long *)(trampoline + size + MCOUNT_INSN_SIZE); + ptr = (unsigned long *)(trampoline + size + MCOUNT_RELINSN_SIZE); *ptr = (unsigned long)ops; op_offset -= start_offset; @@ -868,7 +924,7 @@ void arch_ftrace_update_trampoline(struct ftrace_ops *ops) func = ftrace_ops_get_func(ops); /* Do a safe modify in case the trampoline is executing */ - new = ftrace_call_replace(ip, (unsigned long)func); + new = ftrace_call_replace(ip, (unsigned long)func, MCOUNT_RELINSN_SIZE); ret = update_ftrace_func(ip, new); set_memory_ro(ops->trampoline, npages); @@ -882,7 +938,7 @@ static void *addr_from_call(void *ptr) union ftrace_code_union calc; int ret; - ret = probe_kernel_read(&calc, ptr, MCOUNT_INSN_SIZE); + ret = probe_kernel_read(&calc, ptr, MCOUNT_RELINSN_SIZE); if (WARN_ON_ONCE(ret < 0)) return NULL; @@ -892,7 +948,7 @@ static void *addr_from_call(void *ptr) return NULL; } - return ptr + MCOUNT_INSN_SIZE + calc.offset; + return ptr + MCOUNT_RELINSN_SIZE + calc.offset; } void prepare_ftrace_return(unsigned long self_addr, unsigned long *parent, diff --git a/arch/x86/kernel/module.lds b/arch/x86/kernel/module.lds new file mode 100644 index 000000000000..fd6e95a4b454 --- /dev/null +++ b/arch/x86/kernel/module.lds @@ -0,0 +1,3 @@ +SECTIONS { + .got (NOLOAD) : { BYTE(0) } +}