From patchwork Sun Sep 11 20:35:13 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Konrad Rzeszutek Wilk X-Patchwork-Id: 9325575 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id A97576048B for ; Sun, 11 Sep 2016 20:38:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9511C284EF for ; Sun, 11 Sep 2016 20:38:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 88FC928AB5; Sun, 11 Sep 2016 20:38:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id C9FA2284EF for ; Sun, 11 Sep 2016 20:38:19 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bjBTm-0001wu-7w; Sun, 11 Sep 2016 20:35:54 +0000 Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bjBTk-0001w0-5A for xen-devel@lists.xenproject.org; Sun, 11 Sep 2016 20:35:52 +0000 Received: from [85.158.139.211] by server-10.bemta-5.messagelabs.com id 7F/BB-16745-720C5D75; Sun, 11 Sep 2016 20:35:51 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrDLMWRWlGSWpSXmKPExsUyZ7p8oK7agav hBrfuqFh83zKZyYHR4/CHKywBjFGsmXlJ+RUJrBkff39iKTjdylSxdXJyA+Oiw4xdjFwcQgKT mSR2LVnMCuH8ZZSYumcylLORUeLK+ddQZRMYJaYfPMvSxcjBwSZgIvFmlSNIXESgjVHi24mt7 F2MnBzMAr4St569YgaxhQWsJXZtnM0IYrMIqEocOnsarIZXwE3i89IJjCBzJATkJZ7drgcJcw q4S2yc8x5svBBQyYkeY5CwhICxRPvbi2wTGPkWMDKsYtQoTi0qSy3SNTTUSyrKTM8oyU3MzNE 1NDDVy00tLk5MT81JTCrWS87P3cQIDBUGINjBuLLd+RCjJAeTkiiv/4Kr4UJ8SfkplRmJxRnx RaU5qcWHGGU4OJQkeE/vA8oJFqWmp1akZeYAgxYmLcHBoyTC+xckzVtckJhbnJkOkTrFqCglz vsFJCEAksgozYNrg0XKJUZZKWFeRqBDhHgKUotyM0tQ5V8xinMwKgnzcuwHmsKTmVcCN/0V0G ImoMVPt14GWVySiJCSAsbeE77ZVTuWLlbfPDO+XEi3ctWWgwfNvBv2lJibCP3sFa3+uy2uWKX ou8NX26Anqm3PefxiVri29Xuf8Ch+c+eBe0l6xLSgeqUvilyqnr8eGx3+Xh/xpdjF4YMHX29J 6e1nAh7MzN0vDnK6pzEUnO8WYD3Gt/pmi98ye8ebcWoV0x8ncL76q8RSnJFoqMVcVJwIAMqeI zqPAgAA X-Env-Sender: konrad.wilk@oracle.com X-Msg-Ref: server-16.tower-206.messagelabs.com!1473626148!43028526!1 X-Originating-IP: [156.151.31.81] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogMTU2LjE1MS4zMS44MSA9PiAyODgzMzk=\n X-StarScan-Received: X-StarScan-Version: 8.84; banners=-,-,- X-VirusChecked: Checked Received: (qmail 56948 invoked from network); 11 Sep 2016 20:35:50 -0000 Received: from userp1040.oracle.com (HELO userp1040.oracle.com) (156.151.31.81) by server-16.tower-206.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 11 Sep 2016 20:35:50 -0000 Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u8BKZeL6021157 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 11 Sep 2016 20:35:40 GMT Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id u8BKZe1r014205 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 11 Sep 2016 20:35:40 GMT Received: from abhmp0016.oracle.com (abhmp0016.oracle.com [141.146.116.22]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id u8BKZdPB025589; Sun, 11 Sep 2016 20:35:39 GMT Received: from localhost.localdomain.com (/209.6.196.81) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Sun, 11 Sep 2016 13:35:39 -0700 From: Konrad Rzeszutek Wilk To: xen-devel@lists.xenproject.org, konrad@kernel.org, julien.grall@arm.com, sstabellini@kernel.org, ross.lagerwall@citrix.com Date: Sun, 11 Sep 2016 16:35:13 -0400 Message-Id: <1473626125-13683-7-git-send-email-konrad.wilk@oracle.com> X-Mailer: git-send-email 2.4.11 In-Reply-To: <1473626125-13683-1-git-send-email-konrad.wilk@oracle.com> References: <1473626125-13683-1-git-send-email-konrad.wilk@oracle.com> X-Source-IP: aserv0022.oracle.com [141.146.126.234] Cc: andrew.cooper3@citrix.com, Konrad Rzeszutek Wilk Subject: [Xen-devel] [PATCH v3 06/18] livepatch: Initial ARM64 support. X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP As compared to x86 the va of the hypervisor .text is locked down - we cannot modify the running pagetables to have the .ro flag unset. We borrow the same idea that alternative patching has - which is to vmap the entire .text region and use the alternative virtual address for patching. Since we are doing vmap we may fail, hence the arch_livepatch_quiesce was changed (see "x86,arm: Change arch_livepatch_quiesce() declaration") to return an error value which will be bubbled in payload->rc and provided to the user (along with messages in the ring buffer). The livepatch virtual address space (where the new functions are) needs to be close to the hypervisor virtual address so that the trampoline can reach it. As such we re-use the BOOT_RELOC_VIRT_START which is not used after bootup (alternatively we can also use the space after the _end to FIXMAP_ADDR(0), but that may be too small). The ELF relocation engine at the start was coded from the "ELF for the ARM 64-bit Architecture (AArch64)" (http://infocenter.arm.com/help/topic/com.arm.doc.ihi0056b/IHI0056B_aaelf64.pdf) but after a while of trying to write the correct bit shifting and masking from scratch I ended up borrowing from Linux, the 'reloc_insn_imm' (Linux v4.7 arch/arm64/kernel/module.c function. See 257cb251925f854da435cbf79b140984413871ac "arm64: Loadable modules") And while at it - we also utilize code from Linux to construct the right branch instruction (see "arm64/insn: introduce aarch64_insn_gen_{nop|branch_imm}() helper functions"). In the livepatch payload loading code we tweak the #ifdef to only exclude ARM_32. The exceptions are not part of ARM 32/64 hence they are still behind the #ifdef. We also expand the MAINTAINERS file to include the arm64 and arm32 platform specific livepatch file. Signed-off-by: Konrad Rzeszutek Wilk Acked-by: Jan Beulich --- Cc: Ross Lagerwall Cc: Stefano Stabellini Cc: Julien Grall Cc Jan Beulich Cc: Andrew Cooper RFC: Wholy cow! It works! v1: Finished the various TODOs and reviews outlined by Julien in RFC. v2: Call check_for_livepatch_work in leave_hypervisor_tail not in reset_stack_and_jump - Move ARM 32 components in other patch - Blacklist platform options in Kconfig - Added R_AARCH64_LDST128_ABS_LO12_NC, R_AARCH64_TSTBR14, and R_AARCH64_ADR_PREL_PG_HI21_NC - Added do_reloc and reloc_data along with overflow checks from Linux. - Use apply_alternatives without any #ifdef - Use leave_hypervisor_tail to call check_for_livepatch_work - Add ASSERT that isns is not AARCH64_BREAK_FAULT - Spun out arch_livepatch_quiesce changes in seperate patch. - Spun out changes to config.h (document ones) to seperate patch. - Move the livepatch.h to include/xen/asm-arm. - Add LIVEPATCH_VMAP_END and LIVEPATCH_VMAP_START in config.h - In arch_livepatch_secure use switch for va_type. - Drop the #ifndef CONFIG_ARM for .ex_table (see "arm/x86: Add HAS_ALTERNATIVE and HAS_EX_TABLE") - Piggyback on "x86: change modify_xen_mappings to return error" so that arch_livepatch_secure can return errors. - Moves comment about branch predictor out of this patch. v3: Fix StyleGuide violations (switch statements) - Fix incorrect cast of addr when reverting. - Drop old_ptr variable. - Use bool_t values instead of numbers. - Sprinkle \n as requested by Julien. - In arch_livepatch_quiesce use 1U instead of 1. - Use C99 #defines for [U,]INT[16,32]_[MIN,MAX] instead of Linux kernel ones ([S,U][16,32]_[MIN,MAX]). - Include the ELF relocations for R_AARCH64_[ABS,PRE]16, and all the various groupings for R_AARCH64_MOVW_[UABS,SABS,PREL]_* family. - Redo the NOP patching to use more of the opaque size (so up to 7 instructions in one go). - Drop the cpu_to_le32 macros as they are not needed (and can allow use to share more code with ARM32). - Flush out func->old_addr instead of vmap pointer when reverting. --- MAINTAINERS | 1 + xen/arch/arm/Makefile | 13 +- xen/arch/arm/arm32/Makefile | 1 + xen/arch/arm/arm32/livepatch.c | 38 ++++ xen/arch/arm/arm64/Makefile | 1 + xen/arch/arm/arm64/livepatch.c | 477 ++++++++++++++++++++++++++++++++++++++++ xen/arch/arm/domain.c | 6 + xen/arch/arm/livepatch.c | 103 +++++++-- xen/arch/arm/traps.c | 6 + xen/common/Kconfig | 2 +- xen/include/asm-arm/config.h | 5 + xen/include/asm-arm/livepatch.h | 28 +++ xen/include/xen/elfstructs.h | 57 ++++- xen/include/xen/types.h | 9 + 14 files changed, 722 insertions(+), 25 deletions(-) create mode 100644 xen/arch/arm/arm32/livepatch.c create mode 100644 xen/arch/arm/arm64/livepatch.c create mode 100644 xen/include/asm-arm/livepatch.h diff --git a/MAINTAINERS b/MAINTAINERS index 97720a8..ae0b6bc 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -269,6 +269,7 @@ S: Supported F: docs/misc/livepatch.markdown F: tools/misc/xen-livepatch.c F: xen/arch/*/livepatch* +F: xen/arch/*/*/livepatch* F: xen/common/livepatch* F: xen/include/xen/livepatch* diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile index 64fdf41..c3ce5a3 100644 --- a/xen/arch/arm/Makefile +++ b/xen/arch/arm/Makefile @@ -59,6 +59,15 @@ ALL_OBJS := $(TARGET_SUBARCH)/head.o $(ALL_OBJS) DEPS += $(TARGET_SUBARCH)/.head.o.d +ifdef CONFIG_LIVEPATCH +all_symbols = --all-symbols +ifdef CONFIG_FAST_SYMBOL_LOOKUP +all_symbols = --all-symbols --sort-by-name +endif +else +all_symbols = +endif + $(TARGET): $(TARGET)-syms $(TARGET).axf $(OBJCOPY) -O binary -S $< $@ ifeq ($(CONFIG_ARM_64),y) @@ -94,12 +103,12 @@ $(TARGET)-syms: prelink.o xen.lds $(BASEDIR)/common/symbols-dummy.o $(LD) $(LDFLAGS) -T xen.lds -N prelink.o \ $(BASEDIR)/common/symbols-dummy.o -o $(@D)/.$(@F).0 $(NM) -pa --format=sysv $(@D)/.$(@F).0 \ - | $(BASEDIR)/tools/symbols --sysv --sort >$(@D)/.$(@F).0.S + | $(BASEDIR)/tools/symbols $(all_symbols) --sysv --sort >$(@D)/.$(@F).0.S $(MAKE) -f $(BASEDIR)/Rules.mk $(@D)/.$(@F).0.o $(LD) $(LDFLAGS) -T xen.lds -N prelink.o \ $(@D)/.$(@F).0.o -o $(@D)/.$(@F).1 $(NM) -pa --format=sysv $(@D)/.$(@F).1 \ - | $(BASEDIR)/tools/symbols --sysv --sort >$(@D)/.$(@F).1.S + | $(BASEDIR)/tools/symbols $(all_symbols) --sysv --sort >$(@D)/.$(@F).1.S $(MAKE) -f $(BASEDIR)/Rules.mk $(@D)/.$(@F).1.o $(LD) $(LDFLAGS) -T xen.lds -N prelink.o $(build_id_linker) \ $(@D)/.$(@F).1.o -o $@ diff --git a/xen/arch/arm/arm32/Makefile b/xen/arch/arm/arm32/Makefile index b20db64..4395693 100644 --- a/xen/arch/arm/arm32/Makefile +++ b/xen/arch/arm/arm32/Makefile @@ -4,6 +4,7 @@ obj-$(EARLY_PRINTK) += debug.o obj-y += domctl.o obj-y += domain.o obj-y += entry.o +obj-$(CONFIG_LIVEPATCH) += livepatch.o obj-y += proc-v7.o proc-caxx.o obj-y += smpboot.o obj-y += traps.o diff --git a/xen/arch/arm/arm32/livepatch.c b/xen/arch/arm/arm32/livepatch.c new file mode 100644 index 0000000..c33b68d --- /dev/null +++ b/xen/arch/arm/arm32/livepatch.c @@ -0,0 +1,38 @@ +/* + * Copyright (c) 2016 Oracle and/or its affiliates. All rights reserved. + */ + +#include +#include +#include +#include + +void arch_livepatch_apply_jmp(struct livepatch_func *func) +{ +} + +void arch_livepatch_revert_jmp(const struct livepatch_func *func) +{ +} + +int arch_livepatch_verify_elf(const struct livepatch_elf *elf) +{ + return -EOPNOTSUPP; +} + +int arch_livepatch_perform_rela(struct livepatch_elf *elf, + const struct livepatch_elf_sec *base, + const struct livepatch_elf_sec *rela) +{ + return -ENOSYS; +} + +/* + * Local variables: + * mode: C + * c-file-style: "BSD" + * c-basic-offset: 4 + * tab-width: 4 + * indent-tabs-mode: nil + * End: + */ diff --git a/xen/arch/arm/arm64/Makefile b/xen/arch/arm/arm64/Makefile index c1fa43f..149b6b3 100644 --- a/xen/arch/arm/arm64/Makefile +++ b/xen/arch/arm/arm64/Makefile @@ -6,6 +6,7 @@ obj-y += domctl.o obj-y += domain.o obj-y += entry.o obj-y += insn.o +obj-$(CONFIG_LIVEPATCH) += livepatch.o obj-y += smpboot.o obj-y += traps.o obj-y += vfp.o diff --git a/xen/arch/arm/arm64/livepatch.c b/xen/arch/arm/arm64/livepatch.c new file mode 100644 index 0000000..ecaf3f6 --- /dev/null +++ b/xen/arch/arm/arm64/livepatch.c @@ -0,0 +1,477 @@ +/* + * Copyright (c) 2016 Oracle and/or its affiliates. All rights reserved. + */ + +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include + +void arch_livepatch_apply_jmp(struct livepatch_func *func) +{ + uint32_t insn; + uint32_t *new_ptr; + unsigned int i, len; + + BUILD_BUG_ON(PATCH_INSN_SIZE > sizeof(func->opaque)); + BUILD_BUG_ON(PATCH_INSN_SIZE != sizeof(insn)); + + ASSERT(vmap_of_xen_text); + + len = arch_livepatch_insn_len(func); + if ( !len ) + return; + + /* Save old ones. */ + memcpy(func->opaque, func->old_addr, len); + + if ( func->new_addr ) + insn = aarch64_insn_gen_branch_imm((unsigned long)func->old_addr, + (unsigned long)func->new_addr, + AARCH64_INSN_BRANCH_NOLINK); + else + insn = aarch64_insn_gen_nop(); + + ASSERT(insn != AARCH64_BREAK_FAULT); + + new_ptr = func->old_addr - (void *)_start + vmap_of_xen_text; + len = len / sizeof(uint32_t); + + /* PATCH! */ + for ( i = 0; i < len; i++ ) + *(new_ptr + i) = insn; + + clean_and_invalidate_dcache_va_range(new_ptr, sizeof(*new_ptr) * len); +} + +void arch_livepatch_revert_jmp(const struct livepatch_func *func) +{ + uint32_t *new_ptr; + unsigned int i, len; + + new_ptr = func->old_addr - (void *)_start + vmap_of_xen_text; + len = arch_livepatch_insn_len(func) / sizeof(uint32_t); + for ( i = 0; i < len; i++ ) + { + uint32_t insn; + + memcpy(&insn, func->opaque + (i * sizeof(uint32_t)), PATCH_INSN_SIZE); + /* PATCH! */ + *(new_ptr + i) = insn; + } + + clean_and_invalidate_dcache_va_range(func->old_addr, sizeof(*new_ptr) * len); +} + +int arch_livepatch_verify_elf(const struct livepatch_elf *elf) +{ + const Elf_Ehdr *hdr = elf->hdr; + + if ( hdr->e_machine != EM_AARCH64 || + hdr->e_ident[EI_CLASS] != ELFCLASS64 ) + { + dprintk(XENLOG_ERR, LIVEPATCH "%s: Unsupported ELF Machine type!\n", + elf->name); + return -EOPNOTSUPP; + } + + return 0; +} + +enum aarch64_reloc_op { + RELOC_OP_NONE, + RELOC_OP_ABS, + RELOC_OP_PREL, + RELOC_OP_PAGE, +}; + +static u64 do_reloc(enum aarch64_reloc_op reloc_op, void *place, u64 val) +{ + switch ( reloc_op ) + { + case RELOC_OP_ABS: + return val; + + case RELOC_OP_PREL: + return val - (u64)place; + + case RELOC_OP_PAGE: + return (val & ~0xfff) - ((u64)place & ~0xfff); + + case RELOC_OP_NONE: + return 0; + + } + + dprintk(XENLOG_DEBUG, LIVEPATCH "do_reloc: unknown relocation operation %d\n", reloc_op); + + return 0; +} + +static int reloc_data(enum aarch64_reloc_op op, void *place, u64 val, int len) +{ + s64 sval = do_reloc(op, place, val); + + switch ( len ) + { + case 16: + *(s16 *)place = sval; + if ( sval < INT16_MIN || sval > UINT16_MAX ) + return -EOVERFLOW; + break; + + case 32: + *(s32 *)place = sval; + if ( sval < INT32_MIN || sval > UINT32_MAX ) + return -EOVERFLOW; + break; + + case 64: + *(s64 *)place = sval; + break; + + default: + dprintk(XENLOG_DEBUG, LIVEPATCH "Invalid length (%d) for data relocation\n", len); + return 0; + } + + return 0; +} + +enum aarch64_insn_movw_imm_type { + AARCH64_INSN_IMM_MOVNZ, + AARCH64_INSN_IMM_MOVKZ, +}; + +static int reloc_insn_movw(enum aarch64_reloc_op op, void *dest, u64 val, + int lsb, enum aarch64_insn_movw_imm_type imm_type) +{ + u64 imm; + s64 sval; + u32 insn = *(u32 *)dest; + + sval = do_reloc(op, dest, val); + imm = sval >> lsb; + + if ( imm_type == AARCH64_INSN_IMM_MOVNZ ) + { + /* + * For signed MOVW relocations, we have to manipulate the + * instruction encoding depending on whether or not the + * immediate is less than zero. + */ + insn &= ~(3 << 29); + if ( sval >= 0 ) + { + /* >=0: Set the instruction to MOVZ (opcode 10b). */ + insn |= 2 << 29; + } + else + { + /* + * <0: Set the instruction to MOVN (opcode 00b). + * Since we've masked the opcode already, we + * don't need to do anything other than + * inverting the new immediate field. + */ + imm = ~imm; + } + } + + /* Update the instruction with the new encoding. */ + insn = aarch64_insn_encode_immediate(AARCH64_INSN_IMM_16, insn, imm); + *(u32 *)dest = insn; + + if ( imm > UINT16_MAX ) + return -EOVERFLOW; + + return 0; +} + +static int reloc_insn_imm(enum aarch64_reloc_op op, void *dest, u64 val, + int lsb, int len, enum aarch64_insn_imm_type imm_type) +{ + u64 imm, imm_mask; + s64 sval; + u32 insn = *(u32 *)dest; + + /* Calculate the relocation value. */ + sval = do_reloc(op, dest, val); + sval >>= lsb; + + /* Extract the value bits and shift them to bit 0. */ + imm_mask = (BIT(lsb + len) - 1) >> lsb; + imm = sval & imm_mask; + + /* Update the instruction's immediate field. */ + insn = aarch64_insn_encode_immediate(imm_type, insn, imm); + *(u32 *)dest = insn; + + /* + * Extract the upper value bits (including the sign bit) and + * shift them to bit 0. + */ + sval = (s64)(sval & ~(imm_mask >> 1)) >> (len - 1); + + /* + * Overflow has occurred if the upper bits are not all equal to + * the sign bit of the value. + */ + if ( (u64)(sval + 1) >= 2 ) + return -EOVERFLOW; + return 0; +} + +int arch_livepatch_perform_rela(struct livepatch_elf *elf, + const struct livepatch_elf_sec *base, + const struct livepatch_elf_sec *rela) +{ + const Elf_RelA *r; + unsigned int symndx, i; + uint64_t val; + void *dest; + bool_t overflow_check; + + for ( i = 0; i < (rela->sec->sh_size / rela->sec->sh_entsize); i++ ) + { + int ovf = 0; + + r = rela->data + i * rela->sec->sh_entsize; + + symndx = ELF64_R_SYM(r->r_info); + + if ( symndx > elf->nsym ) + { + dprintk(XENLOG_ERR, LIVEPATCH "%s: Relative relocation wants symbol@%u which is past end!\n", + elf->name, symndx); + return -EINVAL; + } + + dest = base->load_addr + r->r_offset; /* P */ + val = elf->sym[symndx].sym->st_value + r->r_addend; /* S+A */ + + overflow_check = true; + + /* ARM64 operations at minimum are always 32-bit. */ + if ( r->r_offset >= base->sec->sh_size || + (r->r_offset + sizeof(uint32_t)) > base->sec->sh_size ) + goto bad_offset; + + switch ( ELF64_R_TYPE(r->r_info) ) + { + /* Data */ + case R_AARCH64_ABS64: + if ( r->r_offset + sizeof(uint64_t) > base->sec->sh_size ) + goto bad_offset; + overflow_check = false; + ovf = reloc_data(RELOC_OP_ABS, dest, val, 64); + break; + + case R_AARCH64_ABS32: + ovf = reloc_data(RELOC_OP_ABS, dest, val, 32); + break; + + case R_AARCH64_ABS16: + ovf = reloc_data(RELOC_OP_ABS, dest, val, 16); + break; + + case R_AARCH64_PREL64: + if ( r->r_offset + sizeof(uint64_t) > base->sec->sh_size ) + goto bad_offset; + overflow_check = false; + ovf = reloc_data(RELOC_OP_PREL, dest, val, 64); + break; + + case R_AARCH64_PREL32: + ovf = reloc_data(RELOC_OP_PREL, dest, val, 32); + break; + + case R_AARCH64_PREL16: + ovf = reloc_data(RELOC_OP_PREL, dest, val, 16); + break; + + /* MOVW instruction relocations. */ + case R_AARCH64_MOVW_UABS_G0_NC: + overflow_check = false; + + case R_AARCH64_MOVW_UABS_G0: + ovf = reloc_insn_movw(RELOC_OP_ABS, dest, val, 0, + AARCH64_INSN_IMM_MOVKZ); + break; + + case R_AARCH64_MOVW_UABS_G1_NC: + overflow_check = false; + + case R_AARCH64_MOVW_UABS_G1: + ovf = reloc_insn_movw(RELOC_OP_ABS, dest, val, 16, + AARCH64_INSN_IMM_MOVKZ); + break; + + case R_AARCH64_MOVW_UABS_G2_NC: + overflow_check = false; + + case R_AARCH64_MOVW_UABS_G2: + ovf = reloc_insn_movw(RELOC_OP_ABS, dest, val, 32, + AARCH64_INSN_IMM_MOVKZ); + break; + + case R_AARCH64_MOVW_UABS_G3: + /* We're using the top bits so we can't overflow. */ + overflow_check = false; + ovf = reloc_insn_movw(RELOC_OP_ABS, dest, val, 48, + AARCH64_INSN_IMM_MOVKZ); + break; + + case R_AARCH64_MOVW_SABS_G0: + ovf = reloc_insn_movw(RELOC_OP_ABS, dest, val, 0, + AARCH64_INSN_IMM_MOVNZ); + break; + + case R_AARCH64_MOVW_SABS_G1: + ovf = reloc_insn_movw(RELOC_OP_ABS, dest, val, 16, + AARCH64_INSN_IMM_MOVNZ); + break; + + case R_AARCH64_MOVW_SABS_G2: + ovf = reloc_insn_movw(RELOC_OP_ABS, dest, val, 32, + AARCH64_INSN_IMM_MOVNZ); + break; + + case R_AARCH64_MOVW_PREL_G0_NC: + overflow_check = false; + ovf = reloc_insn_movw(RELOC_OP_PREL, dest, val, 0, + AARCH64_INSN_IMM_MOVKZ); + break; + + case R_AARCH64_MOVW_PREL_G0: + ovf = reloc_insn_movw(RELOC_OP_PREL, dest, val, 0, + AARCH64_INSN_IMM_MOVNZ); + break; + + case R_AARCH64_MOVW_PREL_G1_NC: + overflow_check = false; + ovf = reloc_insn_movw(RELOC_OP_PREL, dest, val, 16, + AARCH64_INSN_IMM_MOVKZ); + break; + + case R_AARCH64_MOVW_PREL_G1: + ovf = reloc_insn_movw(RELOC_OP_PREL, dest, val, 16, + AARCH64_INSN_IMM_MOVNZ); + break; + + case R_AARCH64_MOVW_PREL_G2_NC: + overflow_check = false; + ovf = reloc_insn_movw(RELOC_OP_PREL, dest, val, 32, + AARCH64_INSN_IMM_MOVKZ); + break; + + case R_AARCH64_MOVW_PREL_G2: + ovf = reloc_insn_movw(RELOC_OP_PREL, dest, val, 32, + AARCH64_INSN_IMM_MOVNZ); + break; + + case R_AARCH64_MOVW_PREL_G3: + /* We're using the top bits so we can't overflow. */ + overflow_check = false; + ovf = reloc_insn_movw(RELOC_OP_PREL, dest, val, 48, + AARCH64_INSN_IMM_MOVNZ); + break; + + /* Instructions. */ + case R_AARCH64_ADR_PREL_LO21: + ovf = reloc_insn_imm(RELOC_OP_PREL, dest, val, 0, 21, + AARCH64_INSN_IMM_ADR); + break; + + case R_AARCH64_ADR_PREL_PG_HI21_NC: + overflow_check = false; + case R_AARCH64_ADR_PREL_PG_HI21: + ovf = reloc_insn_imm(RELOC_OP_PAGE, dest, val, 12, 21, + AARCH64_INSN_IMM_ADR); + break; + + case R_AARCH64_LDST8_ABS_LO12_NC: + case R_AARCH64_ADD_ABS_LO12_NC: + overflow_check = false; + ovf = reloc_insn_imm(RELOC_OP_ABS, dest, val, 0, 12, + AARCH64_INSN_IMM_12); + break; + + case R_AARCH64_LDST16_ABS_LO12_NC: + overflow_check = false; + ovf = reloc_insn_imm(RELOC_OP_ABS, dest, val, 1, 11, + AARCH64_INSN_IMM_12); + break; + + case R_AARCH64_LDST32_ABS_LO12_NC: + overflow_check = false; + ovf = reloc_insn_imm(RELOC_OP_ABS, dest, val, 2, 10, + AARCH64_INSN_IMM_12); + break; + + case R_AARCH64_LDST64_ABS_LO12_NC: + overflow_check = false; + ovf = reloc_insn_imm(RELOC_OP_ABS, dest, val, 3, 9, + AARCH64_INSN_IMM_12); + break; + + case R_AARCH64_LDST128_ABS_LO12_NC: + overflow_check = false; + ovf = reloc_insn_imm(RELOC_OP_ABS, dest, val, 4, 8, + AARCH64_INSN_IMM_12); + break; + + case R_AARCH64_TSTBR14: + ovf = reloc_insn_imm(RELOC_OP_PREL, dest, val, 2, 19, + AARCH64_INSN_IMM_14); + break; + + case R_AARCH64_CONDBR19: + ovf = reloc_insn_imm(RELOC_OP_PREL, dest, val, 2, 19, + AARCH64_INSN_IMM_19); + break; + + case R_AARCH64_JUMP26: + case R_AARCH64_CALL26: + ovf = reloc_insn_imm(RELOC_OP_PREL, dest, val, 2, 26, + AARCH64_INSN_IMM_26); + break; + + default: + dprintk(XENLOG_ERR, LIVEPATCH "%s: Unhandled relocation %lu\n", + elf->name, ELF64_R_TYPE(r->r_info)); + return -EOPNOTSUPP; + } + + if ( overflow_check && ovf == -EOVERFLOW ) + { + dprintk(XENLOG_ERR, LIVEPATCH "%s: Overflow in relocation %u in %s for %s!\n", + elf->name, i, rela->name, base->name); + return ovf; + } + } + return 0; + + bad_offset: + dprintk(XENLOG_ERR, LIVEPATCH "%s: Relative relocation offset is past %s section!\n", + elf->name, base->name); + return -EINVAL; +} + +/* + * Local variables: + * mode: C + * c-file-style: "BSD" + * c-basic-offset: 4 + * tab-width: 4 + * indent-tabs-mode: nil + * End: + */ diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c index 20bb2ba..607ee59 100644 --- a/xen/arch/arm/domain.c +++ b/xen/arch/arm/domain.c @@ -13,6 +13,7 @@ #include #include #include +#include #include #include #include @@ -55,6 +56,11 @@ void idle_loop(void) do_tasklet(); do_softirq(); + /* + * We MUST be last (or before dsb, wfi). Otherwise after we get the + * softirq we would execute dsb,wfi (and sleep) and not patch. + */ + check_for_livepatch_work(); } } diff --git a/xen/arch/arm/livepatch.c b/xen/arch/arm/livepatch.c index 755f596..7a3e2f7 100644 --- a/xen/arch/arm/livepatch.c +++ b/xen/arch/arm/livepatch.c @@ -6,44 +6,82 @@ #include #include #include +#include + +#include +#include + +void *vmap_of_xen_text; int arch_livepatch_quiesce(void) { - return -ENOSYS; + mfn_t text_mfn; + unsigned int text_order; + + if ( vmap_of_xen_text ) + return -EINVAL; + + text_mfn = _mfn(virt_to_mfn(_start)); + text_order = get_order_from_bytes(_end - _start); + + /* + * The text section is read-only. So re-map Xen to be able to patch + * the code. + */ + vmap_of_xen_text = __vmap(&text_mfn, 1U << text_order, 1, 1, PAGE_HYPERVISOR, + VMAP_DEFAULT); + + if ( !vmap_of_xen_text ) + { + printk(XENLOG_ERR LIVEPATCH "Failed to setup vmap of hypervisor! (order=%u)\n", + text_order); + return -ENOMEM; + } + + return 0; } void arch_livepatch_revive(void) { + /* + * Nuke the instruction cache. Data cache has been cleaned before in + * arch_livepatch_apply_jmp. + */ + invalidate_icache(); + + if ( vmap_of_xen_text ) + vunmap(vmap_of_xen_text); + + vmap_of_xen_text = NULL; } int arch_livepatch_verify_func(const struct livepatch_func *func) { - return -ENOSYS; -} + /* If NOPing only do up to maximum amount we can put in the ->opaque. */ + if ( !func->new_addr && func->new_size > sizeof(func->opaque) && + func->new_size % PATCH_INSN_SIZE ) + return -EOPNOTSUPP; -void arch_livepatch_apply_jmp(struct livepatch_func *func) -{ -} + if ( func->old_size < PATCH_INSN_SIZE ) + return -EINVAL; -void arch_livepatch_revert_jmp(const struct livepatch_func *func) -{ + return 0; } void arch_livepatch_post_action(void) { + /* arch_livepatch_revive has nuked the instruction cache. */ } void arch_livepatch_mask(void) { + /* Mask System Error (SError) */ + local_abort_disable(); } void arch_livepatch_unmask(void) { -} - -int arch_livepatch_verify_elf(const struct livepatch_elf *elf) -{ - return -ENOSYS; + local_abort_enable(); } int arch_livepatch_perform_rel(struct livepatch_elf *elf, @@ -53,20 +91,43 @@ int arch_livepatch_perform_rel(struct livepatch_elf *elf, return -ENOSYS; } -int arch_livepatch_perform_rela(struct livepatch_elf *elf, - const struct livepatch_elf_sec *base, - const struct livepatch_elf_sec *rela) -{ - return -ENOSYS; -} - int arch_livepatch_secure(const void *va, unsigned int pages, enum va_type type) { - return -ENOSYS; + unsigned long start = (unsigned long)va; + unsigned int flags = 0; + + ASSERT(va); + ASSERT(pages); + + switch ( type ) + { + case LIVEPATCH_VA_RX: + flags = PTE_RO; /* R set, NX clear */ + break; + + case LIVEPATCH_VA_RW: + flags = PTE_NX; /* R clear, NX set */ + break; + + case LIVEPATCH_VA_RO: + flags = PTE_NX | PTE_RO; /* R set, NX set */ + break; + + default: + return -EINVAL; + } + + return modify_xen_mappings(start, start + pages * PAGE_SIZE, flags); } void __init arch_livepatch_init(void) { + void *start, *end; + + start = (void *)LIVEPATCH_VMAP_START; + end = (void *)LIVEPATCH_VMAP_END; + + vm_init_type(VMAP_XEN, start, end); } /* diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c index 39a05fd..cd6c222 100644 --- a/xen/arch/arm/traps.c +++ b/xen/arch/arm/traps.c @@ -24,6 +24,7 @@ #include #include #include +#include #include #include #include @@ -2689,6 +2690,11 @@ asmlinkage void leave_hypervisor_tail(void) } local_irq_enable(); do_softirq(); + /* + * Must be the last one - as the IPI will trigger us to come here + * and we want to patch the hypervisor with almost no stack. + */ + check_for_livepatch_work(); } } diff --git a/xen/common/Kconfig b/xen/common/Kconfig index 81e0017..0f26027 100644 --- a/xen/common/Kconfig +++ b/xen/common/Kconfig @@ -217,7 +217,7 @@ config CRYPTO config LIVEPATCH bool "Live patching support (TECH PREVIEW)" default n - depends on X86 && HAS_BUILD_ID = "y" + depends on !ARM_32 && HAS_BUILD_ID = "y" ---help--- Allows a running Xen hypervisor to be dynamically patched using binary patches without rebooting. This is primarily used to binarily diff --git a/xen/include/asm-arm/config.h b/xen/include/asm-arm/config.h index 6772555..ba61f65 100644 --- a/xen/include/asm-arm/config.h +++ b/xen/include/asm-arm/config.h @@ -80,6 +80,7 @@ * 4M - 6M Fixmap: special-purpose 4K mapping slots * 6M - 8M Early boot mapping of FDT * 8M - 10M Early relocation address (used when relocating Xen) + * and later for livepatch vmap (if compiled in) * * ARM32 layout: * 0 - 10M @@ -113,6 +114,10 @@ #define FIXMAP_ADDR(n) (_AT(vaddr_t,0x00400000) + (n) * PAGE_SIZE) #define BOOT_FDT_VIRT_START _AT(vaddr_t,0x00600000) #define BOOT_RELOC_VIRT_START _AT(vaddr_t,0x00800000) +#ifdef CONFIG_LIVEPATCH +#define LIVEPATCH_VMAP_START _AT(vaddr_t,0x00800000) +#define LIVEPATCH_VMAP_END (LIVEPATCH_VMAP_START + MB(2)) +#endif #define HYPERVISOR_VIRT_START XEN_VIRT_START diff --git a/xen/include/asm-arm/livepatch.h b/xen/include/asm-arm/livepatch.h new file mode 100644 index 0000000..8c8d625 --- /dev/null +++ b/xen/include/asm-arm/livepatch.h @@ -0,0 +1,28 @@ +/* + * Copyright (c) 2016 Oracle and/or its affiliates. All rights reserved. + * + */ + +#ifndef __XEN_ARM_LIVEPATCH_H__ +#define __XEN_ARM_LIVEPATCH_H__ + +/* On ARM32,64 instructions are always 4 bytes long. */ +#define PATCH_INSN_SIZE 4 + +/* + * The va of the hypervisor .text region. We need this as the + * normal va are write protected. + */ +extern void *vmap_of_xen_text; + +#endif /* __XEN_ARM_LIVEPATCH_H__ */ + +/* + * Local variables: + * mode: C + * c-file-style: "BSD" + * c-basic-offset: 4 + * tab-width: 4 + * indent-tabs-mode: nil + * End: + */ diff --git a/xen/include/xen/elfstructs.h b/xen/include/xen/elfstructs.h index 5f2082e..7329987 100644 --- a/xen/include/xen/elfstructs.h +++ b/xen/include/xen/elfstructs.h @@ -177,6 +177,7 @@ typedef struct { #define EM_IA_64 50 /* Intel Merced */ #define EM_X86_64 62 /* AMD x86-64 architecture */ #define EM_VAX 75 /* DEC VAX */ +#define EM_AARCH64 183 /* ARM 64-bit */ /* Version */ #define EV_NONE 0 /* Invalid */ @@ -353,12 +354,66 @@ typedef struct { #define ELF64_R_TYPE(info) ((info) & 0xFFFFFFFF) #define ELF64_R_INFO(s,t) (((s) << 32) + (u_int32_t)(t)) -/* x86-64 relocation types. We list only the ones Live Patch implements. */ +/* + * Relocation types for x86_64 and ARM 64. We list only the ones Live Patch + * implements. + */ #define R_X86_64_NONE 0 /* No reloc */ #define R_X86_64_64 1 /* Direct 64 bit */ #define R_X86_64_PC32 2 /* PC relative 32 bit signed */ #define R_X86_64_PLT32 4 /* 32 bit PLT address */ +/* + * S - address of symbol. + * A - addend for relocation (r_addend) + * P - address of the dest being relocated (derieved from r_offset) + * NC - No check for overflow. + * + * The defines also use _PREL for PC-relative address, and _NC is No Check. + */ +#define R_AARCH64_ABS64 257 /* Direct 64 bit. S+A, NC*/ +#define R_AARCH64_ABS32 258 /* Direct 32 bit. S+A */ +#define R_AARCH64_ABS16 259 /* Direct 16 bit, S+A */ +#define R_AARCH64_PREL64 260 /* S+A-P, NC */ +#define R_AARCH64_PREL32 261 /* S+A-P */ +#define R_AARCH64_PREL16 262 /* S+A-P */ + +/* Instructions. */ +#define R_AARCH64_MOVW_UABS_G0 263 +#define R_AARCH64_MOVW_UABS_G0_NC 264 +#define R_AARCH64_MOVW_UABS_G1 265 +#define R_AARCH64_MOVW_UABS_G1_NC 266 +#define R_AARCH64_MOVW_UABS_G2 267 +#define R_AARCH64_MOVW_UABS_G2_NC 268 +#define R_AARCH64_MOVW_UABS_G3 269 + +#define R_AARCH64_MOVW_SABS_G0 270 +#define R_AARCH64_MOVW_SABS_G1 271 +#define R_AARCH64_MOVW_SABS_G2 272 + +#define R_AARCH64_ADR_PREL_LO21 274 /* ADR imm, [20:0]. S+A-P */ +#define R_AARCH64_ADR_PREL_PG_HI21 275 /* ADRP imm, [32:12]. Page(S+A) - Page(P).*/ +#define R_AARCH64_ADR_PREL_PG_HI21_NC 276 +#define R_AARCH64_ADD_ABS_LO12_NC 277 /* ADD imm. [11:0]. S+A, NC */ + +#define R_AARCH64_TSTBR14 279 +#define R_AARCH64_CONDBR19 280 /* Bits 20:2, S+A-P */ +#define R_AARCH64_JUMP26 282 /* Bits 27:2, S+A-P */ +#define R_AARCH64_CALL26 283 /* Bits 27:2, S+A-P */ +#define R_AARCH64_LDST16_ABS_LO12_NC 284 /* LD/ST to bits 11:1, S+A, NC */ +#define R_AARCH64_LDST32_ABS_LO12_NC 285 /* LD/ST to bits 11:2, S+A, NC */ +#define R_AARCH64_LDST64_ABS_LO12_NC 286 /* LD/ST to bits 11:3, S+A, NC */ +#define R_AARCH64_LDST8_ABS_LO12_NC 278 /* LD/ST to bits 11:0, S+A, NC */ +#define R_AARCH64_LDST128_ABS_LO12_NC 299 + +#define R_AARCH64_MOVW_PREL_G0 287 +#define R_AARCH64_MOVW_PREL_G0_NC 288 +#define R_AARCH64_MOVW_PREL_G1 289 +#define R_AARCH64_MOVW_PREL_G1_NC 290 +#define R_AARCH64_MOVW_PREL_G2 291 +#define R_AARCH64_MOVW_PREL_G2_NC 292 +#define R_AARCH64_MOVW_PREL_G3 293 + /* Program Header */ typedef struct { Elf32_Word p_type; /* segment type */ diff --git a/xen/include/xen/types.h b/xen/include/xen/types.h index 7bdc83b..c79c353 100644 --- a/xen/include/xen/types.h +++ b/xen/include/xen/types.h @@ -14,6 +14,15 @@ #define NULL ((void*)0) #endif +#define INT16_MIN (-32767-1) +#define INT32_MIN (-2147483647-1) + +#define INT16_MAX (32767) +#define INT32_MAX (2147483647) + +#define UINT16_MAX (65535) +#define UINT32_MAX (4294967295U) + #define INT_MAX ((int)(~0U>>1)) #define INT_MIN (-INT_MAX - 1) #define UINT_MAX (~0U)