From patchwork Fri Jul 1 17:19:23 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 9210325 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 312A1607D6 for ; Fri, 1 Jul 2016 17:58:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 24A982852F for ; Fri, 1 Jul 2016 17:58:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 195DD28533; Fri, 1 Jul 2016 17:58:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 8D0202852F for ; Fri, 1 Jul 2016 17:58:37 +0000 (UTC) Received: from localhost ([::1]:34990 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bJ2i4-0008RM-Px for patchwork-qemu-devel@patchwork.kernel.org; Fri, 01 Jul 2016 13:58:36 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59166) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bJ26H-0003OD-9O for qemu-devel@nongnu.org; Fri, 01 Jul 2016 13:19:35 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bJ26D-0001f5-Mg for qemu-devel@nongnu.org; Fri, 01 Jul 2016 13:19:32 -0400 Received: from mail-pa0-x244.google.com ([2607:f8b0:400e:c03::244]:33718) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bJ26D-0001eH-6P for qemu-devel@nongnu.org; Fri, 01 Jul 2016 13:19:29 -0400 Received: by mail-pa0-x244.google.com with SMTP id ts6so10191672pac.0 for ; Fri, 01 Jul 2016 10:19:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=G4LfzuDUNQkWgYN2eAhJMFuElWTeQpK2dp7mcqCjjaw=; b=voK2G/5wr99dCKI6DRz9pH7bvVbXxYrmff3/65vTUEs9PW9mIf7CyloFrmC94hZGEt RxO9H31EuGGuSpf1P5HWhPd/VmkOoRKQpsKFZe+lvnnO0ENDY1FJr6VxdwOIrDluLG5e 9WfA/NZNgfeQbZa7uHYHhDUhzsLhBk4xCqpBWMbZftruWgH9d0/dFuuEi3Ij2gwn3Bze 5DFVc0TkF9fSNM3jtToPqazcw7F244V1lRf6N34tm3YryK3EGqQ5UU5e9TsN6hihx8Rv XyPPJC7ohIFQzsemj/891N10hSEwzYUdW+I3p477KFr/8B/+KYKAQAtnfRAPgwcKu7Lh DKBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=G4LfzuDUNQkWgYN2eAhJMFuElWTeQpK2dp7mcqCjjaw=; b=I4aHvf/cPXkbb9o2JBC51sAnmwzaH0C//uGvjTGH0Uy/sb8YymDmzY0GmFUD6dvQUg BSm6OifS2qR+2i4f2js1Qc9JR/FDmAmqrGTg/n91hxLmB9I7pEtMELcymBiRIo7uKQqo aD2gdWMN8H74sMY7rE+N5jzU+8TL82EumsX3ywFUlGyaksinGSlnrOa0hdRJyneyb1xz +C+jlRMN9QuSQbIreMwjaecpURDoUkRcHAMVJGJQdf/8rHZQdr1zP+PNPTCfKpExtapG MF6A0d1yKB80VtjX5gveAC6Jw+UxaJqL3xr2oCszeJxznfHbpXo4JIDVx96OqDw16AyG DXDw== X-Gm-Message-State: ALyK8tK5ayG0h2jeN/ahloT6yIPotT5P2dF0r6QZR81E2mIOHMl5cU2PXIRvtlJsEwOD3g== X-Received: by 10.66.135.40 with SMTP id pp8mr33653679pab.113.1467393567874; Fri, 01 Jul 2016 10:19:27 -0700 (PDT) Received: from bigtime.twiddle.net (71-37-54-227.tukw.qwest.net. [71.37.54.227]) by smtp.gmail.com with ESMTPSA id b71sm7102860pfc.51.2016.07.01.10.19.26 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 01 Jul 2016 10:19:27 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Fri, 1 Jul 2016 10:19:23 -0700 Message-Id: <1467393563-29447-5-git-send-email-rth@twiddle.net> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1467393563-29447-1-git-send-email-rth@twiddle.net> References: <1467393563-29447-1-git-send-email-rth@twiddle.net> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2607:f8b0:400e:c03::244 Subject: [Qemu-devel] [PULL 4/4] tcg: Improve the alignment check infrastructure X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, Sergey Sorokin Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: Sergey Sorokin Some architectures (e.g. ARMv8) need the address which is aligned to a size more than the size of the memory access. To support such check it's enough the current costless alignment check implementation in QEMU, but we need to support an alignment size specifying. Signed-off-by: Sergey Sorokin Message-Id: <1466705806-679898-1-git-send-email-afarallax@yandex.ru> Signed-off-by: Richard Henderson [rth: Assert in tcg_canonicalize_memop. Leave get_alignment_bits available for, though unused by, user-mode. Retain logging difference based on ALIGNED_ONLY.] --- include/exec/cpu-all.h | 16 ++++++-- softmmu_template.h | 88 ++++++++++++-------------------------------- tcg/aarch64/tcg-target.inc.c | 9 +++-- tcg/i386/tcg-target.inc.c | 15 +++++--- tcg/ppc/tcg-target.inc.c | 14 ++++--- tcg/s390/tcg-target.inc.c | 9 +++-- tcg/tcg-op.c | 3 ++ tcg/tcg.c | 26 +++++++++---- tcg/tcg.h | 87 ++++++++++++++++++++++++++++++++++++------- 9 files changed, 159 insertions(+), 108 deletions(-) diff --git a/include/exec/cpu-all.h b/include/exec/cpu-all.h index 6a6796d..b6a7059 100644 --- a/include/exec/cpu-all.h +++ b/include/exec/cpu-all.h @@ -288,14 +288,22 @@ CPUArchState *cpu_copy(CPUArchState *env); #if !defined(CONFIG_USER_ONLY) /* Flags stored in the low bits of the TLB virtual address. These are - defined so that fast path ram access is all zeros. */ + * defined so that fast path ram access is all zeros. + * The flags all must be between TARGET_PAGE_BITS and + * maximum address alignment bit. + */ /* Zero if TLB entry is valid. */ -#define TLB_INVALID_MASK (1 << 3) +#define TLB_INVALID_MASK (1 << (TARGET_PAGE_BITS - 1)) /* Set if TLB entry references a clean RAM page. The iotlb entry will contain the page physical address. */ -#define TLB_NOTDIRTY (1 << 4) +#define TLB_NOTDIRTY (1 << (TARGET_PAGE_BITS - 2)) /* Set if TLB entry is an IO callback. */ -#define TLB_MMIO (1 << 5) +#define TLB_MMIO (1 << (TARGET_PAGE_BITS - 3)) + +/* Use this mask to check interception with an alignment mask + * in a TCG backend. + */ +#define TLB_FLAGS_MASK (TLB_INVALID_MASK | TLB_NOTDIRTY | TLB_MMIO) void dump_exec_info(FILE *f, fprintf_function cpu_fprintf); void dump_opcount_info(FILE *f, fprintf_function cpu_fprintf); diff --git a/softmmu_template.h b/softmmu_template.h index 208f808..4d378ca 100644 --- a/softmmu_template.h +++ b/softmmu_template.h @@ -171,20 +171,21 @@ WORD_TYPE helper_le_ld_name(CPUArchState *env, target_ulong addr, unsigned mmu_idx = get_mmuidx(oi); int index = (addr >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); target_ulong tlb_addr = env->tlb_table[mmu_idx][index].ADDR_READ; + int a_bits = get_alignment_bits(get_memop(oi)); uintptr_t haddr; DATA_TYPE res; /* Adjust the given return address. */ retaddr -= GETPC_ADJ; + if (a_bits > 0 && (addr & ((1 << a_bits) - 1)) != 0) { + cpu_unaligned_access(ENV_GET_CPU(env), addr, READ_ACCESS_TYPE, + mmu_idx, retaddr); + } + /* If the TLB entry is for a different page, reload and try again. */ if ((addr & TARGET_PAGE_MASK) != (tlb_addr & (TARGET_PAGE_MASK | TLB_INVALID_MASK))) { - if ((addr & (DATA_SIZE - 1)) != 0 - && (get_memop(oi) & MO_AMASK) == MO_ALIGN) { - cpu_unaligned_access(ENV_GET_CPU(env), addr, READ_ACCESS_TYPE, - mmu_idx, retaddr); - } if (!VICTIM_TLB_HIT(ADDR_READ)) { tlb_fill(ENV_GET_CPU(env), addr, READ_ACCESS_TYPE, mmu_idx, retaddr); @@ -215,10 +216,6 @@ WORD_TYPE helper_le_ld_name(CPUArchState *env, target_ulong addr, DATA_TYPE res1, res2; unsigned shift; do_unaligned_access: - if ((get_memop(oi) & MO_AMASK) == MO_ALIGN) { - cpu_unaligned_access(ENV_GET_CPU(env), addr, READ_ACCESS_TYPE, - mmu_idx, retaddr); - } addr1 = addr & ~(DATA_SIZE - 1); addr2 = addr1 + DATA_SIZE; /* Note the adjustment at the beginning of the function. @@ -232,13 +229,6 @@ WORD_TYPE helper_le_ld_name(CPUArchState *env, target_ulong addr, return res; } - /* Handle aligned access or unaligned access in the same page. */ - if ((addr & (DATA_SIZE - 1)) != 0 - && (get_memop(oi) & MO_AMASK) == MO_ALIGN) { - cpu_unaligned_access(ENV_GET_CPU(env), addr, READ_ACCESS_TYPE, - mmu_idx, retaddr); - } - haddr = addr + env->tlb_table[mmu_idx][index].addend; #if DATA_SIZE == 1 res = glue(glue(ld, LSUFFIX), _p)((uint8_t *)haddr); @@ -255,20 +245,21 @@ WORD_TYPE helper_be_ld_name(CPUArchState *env, target_ulong addr, unsigned mmu_idx = get_mmuidx(oi); int index = (addr >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); target_ulong tlb_addr = env->tlb_table[mmu_idx][index].ADDR_READ; + int a_bits = get_alignment_bits(get_memop(oi)); uintptr_t haddr; DATA_TYPE res; /* Adjust the given return address. */ retaddr -= GETPC_ADJ; + if (a_bits > 0 && (addr & ((1 << a_bits) - 1)) != 0) { + cpu_unaligned_access(ENV_GET_CPU(env), addr, READ_ACCESS_TYPE, + mmu_idx, retaddr); + } + /* If the TLB entry is for a different page, reload and try again. */ if ((addr & TARGET_PAGE_MASK) != (tlb_addr & (TARGET_PAGE_MASK | TLB_INVALID_MASK))) { - if ((addr & (DATA_SIZE - 1)) != 0 - && (get_memop(oi) & MO_AMASK) == MO_ALIGN) { - cpu_unaligned_access(ENV_GET_CPU(env), addr, READ_ACCESS_TYPE, - mmu_idx, retaddr); - } if (!VICTIM_TLB_HIT(ADDR_READ)) { tlb_fill(ENV_GET_CPU(env), addr, READ_ACCESS_TYPE, mmu_idx, retaddr); @@ -299,10 +290,6 @@ WORD_TYPE helper_be_ld_name(CPUArchState *env, target_ulong addr, DATA_TYPE res1, res2; unsigned shift; do_unaligned_access: - if ((get_memop(oi) & MO_AMASK) == MO_ALIGN) { - cpu_unaligned_access(ENV_GET_CPU(env), addr, READ_ACCESS_TYPE, - mmu_idx, retaddr); - } addr1 = addr & ~(DATA_SIZE - 1); addr2 = addr1 + DATA_SIZE; /* Note the adjustment at the beginning of the function. @@ -316,13 +303,6 @@ WORD_TYPE helper_be_ld_name(CPUArchState *env, target_ulong addr, return res; } - /* Handle aligned access or unaligned access in the same page. */ - if ((addr & (DATA_SIZE - 1)) != 0 - && (get_memop(oi) & MO_AMASK) == MO_ALIGN) { - cpu_unaligned_access(ENV_GET_CPU(env), addr, READ_ACCESS_TYPE, - mmu_idx, retaddr); - } - haddr = addr + env->tlb_table[mmu_idx][index].addend; res = glue(glue(ld, LSUFFIX), _be_p)((uint8_t *)haddr); return res; @@ -376,19 +356,20 @@ void helper_le_st_name(CPUArchState *env, target_ulong addr, DATA_TYPE val, unsigned mmu_idx = get_mmuidx(oi); int index = (addr >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); target_ulong tlb_addr = env->tlb_table[mmu_idx][index].addr_write; + int a_bits = get_alignment_bits(get_memop(oi)); uintptr_t haddr; /* Adjust the given return address. */ retaddr -= GETPC_ADJ; + if (a_bits > 0 && (addr & ((1 << a_bits) - 1)) != 0) { + cpu_unaligned_access(ENV_GET_CPU(env), addr, MMU_DATA_STORE, + mmu_idx, retaddr); + } + /* If the TLB entry is for a different page, reload and try again. */ if ((addr & TARGET_PAGE_MASK) != (tlb_addr & (TARGET_PAGE_MASK | TLB_INVALID_MASK))) { - if ((addr & (DATA_SIZE - 1)) != 0 - && (get_memop(oi) & MO_AMASK) == MO_ALIGN) { - cpu_unaligned_access(ENV_GET_CPU(env), addr, MMU_DATA_STORE, - mmu_idx, retaddr); - } if (!VICTIM_TLB_HIT(addr_write)) { tlb_fill(ENV_GET_CPU(env), addr, MMU_DATA_STORE, mmu_idx, retaddr); } @@ -416,10 +397,6 @@ void helper_le_st_name(CPUArchState *env, target_ulong addr, DATA_TYPE val, >= TARGET_PAGE_SIZE)) { int i; do_unaligned_access: - if ((get_memop(oi) & MO_AMASK) == MO_ALIGN) { - cpu_unaligned_access(ENV_GET_CPU(env), addr, MMU_DATA_STORE, - mmu_idx, retaddr); - } /* XXX: not efficient, but simple */ /* Note: relies on the fact that tlb_fill() does not remove the * previous page from the TLB cache. */ @@ -434,13 +411,6 @@ void helper_le_st_name(CPUArchState *env, target_ulong addr, DATA_TYPE val, return; } - /* Handle aligned access or unaligned access in the same page. */ - if ((addr & (DATA_SIZE - 1)) != 0 - && (get_memop(oi) & MO_AMASK) == MO_ALIGN) { - cpu_unaligned_access(ENV_GET_CPU(env), addr, MMU_DATA_STORE, - mmu_idx, retaddr); - } - haddr = addr + env->tlb_table[mmu_idx][index].addend; #if DATA_SIZE == 1 glue(glue(st, SUFFIX), _p)((uint8_t *)haddr, val); @@ -456,19 +426,20 @@ void helper_be_st_name(CPUArchState *env, target_ulong addr, DATA_TYPE val, unsigned mmu_idx = get_mmuidx(oi); int index = (addr >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); target_ulong tlb_addr = env->tlb_table[mmu_idx][index].addr_write; + int a_bits = get_alignment_bits(get_memop(oi)); uintptr_t haddr; /* Adjust the given return address. */ retaddr -= GETPC_ADJ; + if (a_bits > 0 && (addr & ((1 << a_bits) - 1)) != 0) { + cpu_unaligned_access(ENV_GET_CPU(env), addr, MMU_DATA_STORE, + mmu_idx, retaddr); + } + /* If the TLB entry is for a different page, reload and try again. */ if ((addr & TARGET_PAGE_MASK) != (tlb_addr & (TARGET_PAGE_MASK | TLB_INVALID_MASK))) { - if ((addr & (DATA_SIZE - 1)) != 0 - && (get_memop(oi) & MO_AMASK) == MO_ALIGN) { - cpu_unaligned_access(ENV_GET_CPU(env), addr, MMU_DATA_STORE, - mmu_idx, retaddr); - } if (!VICTIM_TLB_HIT(addr_write)) { tlb_fill(ENV_GET_CPU(env), addr, MMU_DATA_STORE, mmu_idx, retaddr); } @@ -496,10 +467,6 @@ void helper_be_st_name(CPUArchState *env, target_ulong addr, DATA_TYPE val, >= TARGET_PAGE_SIZE)) { int i; do_unaligned_access: - if ((get_memop(oi) & MO_AMASK) == MO_ALIGN) { - cpu_unaligned_access(ENV_GET_CPU(env), addr, MMU_DATA_STORE, - mmu_idx, retaddr); - } /* XXX: not efficient, but simple */ /* Note: relies on the fact that tlb_fill() does not remove the * previous page from the TLB cache. */ @@ -514,13 +481,6 @@ void helper_be_st_name(CPUArchState *env, target_ulong addr, DATA_TYPE val, return; } - /* Handle aligned access or unaligned access in the same page. */ - if ((addr & (DATA_SIZE - 1)) != 0 - && (get_memop(oi) & MO_AMASK) == MO_ALIGN) { - cpu_unaligned_access(ENV_GET_CPU(env), addr, MMU_DATA_STORE, - mmu_idx, retaddr); - } - haddr = addr + env->tlb_table[mmu_idx][index].addend; glue(glue(st, SUFFIX), _be_p)((uint8_t *)haddr, val); } diff --git a/tcg/aarch64/tcg-target.inc.c b/tcg/aarch64/tcg-target.inc.c index 5ac0091..08b2d03 100644 --- a/tcg/aarch64/tcg-target.inc.c +++ b/tcg/aarch64/tcg-target.inc.c @@ -1081,19 +1081,20 @@ static void tcg_out_tlb_read(TCGContext *s, TCGReg addr_reg, TCGMemOp opc, int tlb_offset = is_read ? offsetof(CPUArchState, tlb_table[mem_index][0].addr_read) : offsetof(CPUArchState, tlb_table[mem_index][0].addr_write); - int s_mask = (1 << (opc & MO_SIZE)) - 1; + int a_bits = get_alignment_bits(opc); TCGReg base = TCG_AREG0, x3; uint64_t tlb_mask; /* For aligned accesses, we check the first byte and include the alignment bits within the address. For unaligned access, we check that we don't cross pages using the address of the last byte of the access. */ - if ((opc & MO_AMASK) == MO_ALIGN || s_mask == 0) { - tlb_mask = TARGET_PAGE_MASK | s_mask; + if (a_bits >= 0) { + /* A byte access or an alignment check required */ + tlb_mask = TARGET_PAGE_MASK | ((1 << a_bits) - 1); x3 = addr_reg; } else { tcg_out_insn(s, 3401, ADDI, TARGET_LONG_BITS == 64, - TCG_REG_X3, addr_reg, s_mask); + TCG_REG_X3, addr_reg, (1 << (opc & MO_SIZE)) - 1); tlb_mask = TARGET_PAGE_MASK; x3 = TCG_REG_X3; } diff --git a/tcg/i386/tcg-target.inc.c b/tcg/i386/tcg-target.inc.c index bc34535..6f8cdca 100644 --- a/tcg/i386/tcg-target.inc.c +++ b/tcg/i386/tcg-target.inc.c @@ -1202,8 +1202,8 @@ static inline void tcg_out_tlb_load(TCGContext *s, TCGReg addrlo, TCGReg addrhi, TCGType ttype = TCG_TYPE_I32; TCGType tlbtype = TCG_TYPE_I32; int trexw = 0, hrexw = 0, tlbrexw = 0; - int s_mask = (1 << (opc & MO_SIZE)) - 1; - bool aligned = (opc & MO_AMASK) == MO_ALIGN || s_mask == 0; + int a_bits = get_alignment_bits(opc); + target_ulong tlb_mask; if (TCG_TARGET_REG_BITS == 64) { if (TARGET_LONG_BITS == 64) { @@ -1220,19 +1220,22 @@ static inline void tcg_out_tlb_load(TCGContext *s, TCGReg addrlo, TCGReg addrhi, } tcg_out_mov(s, tlbtype, r0, addrlo); - if (aligned) { + if (a_bits >= 0) { + /* A byte access or an alignment check required */ tcg_out_mov(s, ttype, r1, addrlo); + tlb_mask = TARGET_PAGE_MASK | ((1 << a_bits) - 1); } else { /* For unaligned access check that we don't cross pages using the page address of the last byte. */ - tcg_out_modrm_offset(s, OPC_LEA + trexw, r1, addrlo, s_mask); + tcg_out_modrm_offset(s, OPC_LEA + trexw, r1, addrlo, + (1 << (opc & MO_SIZE)) - 1); + tlb_mask = TARGET_PAGE_MASK; } tcg_out_shifti(s, SHIFT_SHR + tlbrexw, r0, TARGET_PAGE_BITS - CPU_TLB_ENTRY_BITS); - tgen_arithi(s, ARITH_AND + trexw, r1, - TARGET_PAGE_MASK | (aligned ? s_mask : 0), 0); + tgen_arithi(s, ARITH_AND + trexw, r1, tlb_mask, 0); tgen_arithi(s, ARITH_AND + tlbrexw, r0, (CPU_TLB_SIZE - 1) << CPU_TLB_ENTRY_BITS, 0); diff --git a/tcg/ppc/tcg-target.inc.c b/tcg/ppc/tcg-target.inc.c index dba954c..eaf1bd9 100644 --- a/tcg/ppc/tcg-target.inc.c +++ b/tcg/ppc/tcg-target.inc.c @@ -1405,6 +1405,7 @@ static TCGReg tcg_out_tlb_read(TCGContext *s, TCGMemOp opc, int add_off = offsetof(CPUArchState, tlb_table[mem_index][0].addend); TCGReg base = TCG_AREG0; TCGMemOp s_bits = opc & MO_SIZE; + int a_bits = get_alignment_bits(opc); /* Extract the page index, shifted into place for tlb index. */ if (TCG_TARGET_REG_BITS == 64) { @@ -1462,14 +1463,17 @@ static TCGReg tcg_out_tlb_read(TCGContext *s, TCGMemOp opc, * the bottom bits and thus trigger a comparison failure on * unaligned accesses */ + if (a_bits < 0) { + a_bits = s_bits; + } tcg_out_rlw(s, RLWINM, TCG_REG_R0, addrlo, 0, - (32 - s_bits) & 31, 31 - TARGET_PAGE_BITS); - } else if (s_bits) { - /* > byte access, we need to handle alignment */ - if ((opc & MO_AMASK) == MO_ALIGN) { + (32 - a_bits) & 31, 31 - TARGET_PAGE_BITS); + } else if (a_bits) { + /* More than byte access, we need to handle alignment */ + if (a_bits > 0) { /* Alignment required by the front-end, same as 32-bits */ tcg_out_rld(s, RLDICL, TCG_REG_R0, addrlo, - 64 - TARGET_PAGE_BITS, TARGET_PAGE_BITS - s_bits); + 64 - TARGET_PAGE_BITS, TARGET_PAGE_BITS - a_bits); tcg_out_rld(s, RLDICL, TCG_REG_R0, TCG_REG_R0, TARGET_PAGE_BITS, 0); } else { /* We support unaligned accesses, we need to make sure we fail diff --git a/tcg/s390/tcg-target.inc.c b/tcg/s390/tcg-target.inc.c index 314c183..5a7495b 100644 --- a/tcg/s390/tcg-target.inc.c +++ b/tcg/s390/tcg-target.inc.c @@ -1505,18 +1505,19 @@ QEMU_BUILD_BUG_ON(offsetof(CPUArchState, tlb_table[NB_MMU_MODES - 1][1]) static TCGReg tcg_out_tlb_read(TCGContext* s, TCGReg addr_reg, TCGMemOp opc, int mem_index, bool is_ld) { - int s_mask = (1 << (opc & MO_SIZE)) - 1; + int a_bits = get_alignment_bits(opc); int ofs, a_off; uint64_t tlb_mask; /* For aligned accesses, we check the first byte and include the alignment bits within the address. For unaligned access, we check that we don't cross pages using the address of the last byte of the access. */ - if ((opc & MO_AMASK) == MO_ALIGN || s_mask == 0) { + if (a_bits >= 0) { + /* A byte access or an alignment check required */ a_off = 0; - tlb_mask = TARGET_PAGE_MASK | s_mask; + tlb_mask = TARGET_PAGE_MASK | ((1 << a_bits) - 1); } else { - a_off = s_mask; + a_off = (1 << (opc & MO_SIZE)) - 1; tlb_mask = TARGET_PAGE_MASK; } diff --git a/tcg/tcg-op.c b/tcg/tcg-op.c index 569cdc6..293b854 100644 --- a/tcg/tcg-op.c +++ b/tcg/tcg-op.c @@ -1851,6 +1851,9 @@ void tcg_gen_goto_tb(unsigned idx) static inline TCGMemOp tcg_canonicalize_memop(TCGMemOp op, bool is64, bool st) { + /* Trigger the asserts within as early as possible. */ + (void)get_alignment_bits(op); + switch (op & MO_SIZE) { case MO_8: op &= ~MO_BSWAP; diff --git a/tcg/tcg.c b/tcg/tcg.c index 635e4e9..f21a8ab 100644 --- a/tcg/tcg.c +++ b/tcg/tcg.c @@ -999,6 +999,22 @@ static const char * const ldst_name[] = [MO_BEQ] = "beq", }; +static const char * const alignment_name[(MO_AMASK >> MO_ASHIFT) + 1] = { +#ifdef ALIGNED_ONLY + [MO_UNALN >> MO_ASHIFT] = "un+", + [MO_ALIGN >> MO_ASHIFT] = "", +#else + [MO_UNALN >> MO_ASHIFT] = "", + [MO_ALIGN >> MO_ASHIFT] = "al+", +#endif + [MO_ALIGN_2 >> MO_ASHIFT] = "al2+", + [MO_ALIGN_4 >> MO_ASHIFT] = "al4+", + [MO_ALIGN_8 >> MO_ASHIFT] = "al8+", + [MO_ALIGN_16 >> MO_ASHIFT] = "al16+", + [MO_ALIGN_32 >> MO_ASHIFT] = "al32+", + [MO_ALIGN_64 >> MO_ASHIFT] = "al64+", +}; + void tcg_dump_ops(TCGContext *s) { char buf[128]; @@ -1100,14 +1116,8 @@ void tcg_dump_ops(TCGContext *s) if (op & ~(MO_AMASK | MO_BSWAP | MO_SSIZE)) { qemu_log(",$0x%x,%u", op, ix); } else { - const char *s_al = "", *s_op; - if (op & MO_AMASK) { - if ((op & MO_AMASK) == MO_ALIGN) { - s_al = "al+"; - } else { - s_al = "un+"; - } - } + const char *s_al, *s_op; + s_al = alignment_name[(op & MO_AMASK) >> MO_ASHIFT]; s_op = ldst_name[op & (MO_BSWAP | MO_SSIZE)]; qemu_log(",%s%s,%u", s_al, s_op, ix); } diff --git a/tcg/tcg.h b/tcg/tcg.h index 66d7fc0..66ae0c7 100644 --- a/tcg/tcg.h +++ b/tcg/tcg.h @@ -191,6 +191,15 @@ typedef uint64_t tcg_insn_unit; #endif +#ifdef CONFIG_DEBUG_TCG +# define tcg_debug_assert(X) do { assert(X); } while (0) +#elif QEMU_GNUC_PREREQ(4, 5) +# define tcg_debug_assert(X) \ + do { if (!(X)) { __builtin_unreachable(); } } while (0) +#else +# define tcg_debug_assert(X) do { (void)(X); } while (0) +#endif + typedef struct TCGRelocation { struct TCGRelocation *next; int type; @@ -275,10 +284,26 @@ typedef enum TCGMemOp { #endif /* MO_UNALN accesses are never checked for alignment. - MO_ALIGN accesses will result in a call to the CPU's - do_unaligned_access hook if the guest address is not aligned. - The default depends on whether the target CPU defines ALIGNED_ONLY. */ - MO_AMASK = 16, + * MO_ALIGN accesses will result in a call to the CPU's + * do_unaligned_access hook if the guest address is not aligned. + * The default depends on whether the target CPU defines ALIGNED_ONLY. + * Some architectures (e.g. ARMv8) need the address which is aligned + * to a size more than the size of the memory access. + * To support such check it's enough the current costless alignment + * check implementation in QEMU, but we need to support + * an alignment size specifying. + * MO_ALIGN supposes a natural alignment + * (i.e. the alignment size is the size of a memory access). + * Note that an alignment size must be equal or greater + * than an access size. + * There are three options: + * - an alignment to the size of an access (MO_ALIGN); + * - an alignment to the specified size that is equal or greater than + * an access size (MO_ALIGN_x where 'x' is a size in bytes); + * - unaligned access permitted (MO_UNALN). + */ + MO_ASHIFT = 4, + MO_AMASK = 7 << MO_ASHIFT, #ifdef ALIGNED_ONLY MO_ALIGN = 0, MO_UNALN = MO_AMASK, @@ -286,6 +311,12 @@ typedef enum TCGMemOp { MO_ALIGN = MO_AMASK, MO_UNALN = 0, #endif + MO_ALIGN_2 = 1 << MO_ASHIFT, + MO_ALIGN_4 = 2 << MO_ASHIFT, + MO_ALIGN_8 = 3 << MO_ASHIFT, + MO_ALIGN_16 = 4 << MO_ASHIFT, + MO_ALIGN_32 = 5 << MO_ASHIFT, + MO_ALIGN_64 = 6 << MO_ASHIFT, /* Combinations of the above, for ease of use. */ MO_UB = MO_8, @@ -317,6 +348,45 @@ typedef enum TCGMemOp { MO_SSIZE = MO_SIZE | MO_SIGN, } TCGMemOp; +/** + * get_alignment_bits + * @memop: TCGMemOp value + * + * Extract the alignment size from the memop. + * + * Returns: 0 in case of byte access (which is always aligned); + * positive value - number of alignment bits; + * negative value if unaligned access enabled + * and this is not a byte access. + */ +static inline int get_alignment_bits(TCGMemOp memop) +{ + int a = memop & MO_AMASK; + int s = memop & MO_SIZE; + int r; + + if (a == MO_UNALN) { + /* Negative value if unaligned access enabled, + * or zero value in case of byte access. + */ + return -s; + } else if (a == MO_ALIGN) { + /* A natural alignment: return a number of access size bits */ + r = s; + } else { + /* Specific alignment size. It must be equal or greater + * than the access size. + */ + r = a >> MO_ASHIFT; + tcg_debug_assert(r >= s); + } +#if defined(CONFIG_SOFTMMU) + /* The requested alignment cannot overlap the TLB flags. */ + tcg_debug_assert((TLB_FLAGS_MASK & ((1 << r) - 1)) == 0); +#endif + return r; +} + typedef tcg_target_ulong TCGArg; /* Define a type and accessor macros for variables. Using pointer types @@ -790,15 +860,6 @@ do {\ abort();\ } while (0) -#ifdef CONFIG_DEBUG_TCG -# define tcg_debug_assert(X) do { assert(X); } while (0) -#elif QEMU_GNUC_PREREQ(4, 5) -# define tcg_debug_assert(X) \ - do { if (!(X)) { __builtin_unreachable(); } } while (0) -#else -# define tcg_debug_assert(X) do { (void)(X); } while (0) -#endif - void tcg_add_target_add_op_defs(const TCGTargetOpDef *tdefs); #if UINTPTR_MAX == UINT32_MAX