From patchwork Fri Nov 23 17:29:02 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 10696235 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D49EA13BF for ; Fri, 23 Nov 2018 17:29:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B7DC22C146 for ; Fri, 23 Nov 2018 17:29:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AC2DE2C247; Fri, 23 Nov 2018 17:29:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id E5AB72C146 for ; Fri, 23 Nov 2018 17:29:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-Id:Date:Subject:To :From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=ODn9sk6nxZwEdLjfxa+kppgvj+0XQBTd8GmRtgaBB6Y=; b=riSIjAB9fXx89G Z5nADvM/H+YLz0NmR0OFulIi+flko7AHeNAuFAHiSZd/QH/pQ3QhpmNcHTLwk/dn9LHN47QAhX53Q D8rGrphinvNMEOoikq7YgX5UbvBv3nvZdwhoCd4fBPeN4UWuTNBAsJtXjzW9DSSHv9P6avPrU0nNd WQTl7pgAH+Zfl7wsrvmD3j+N2hBfUuvrHpJbv65zslm+/xsd5Ox0srtDP+KPSoEMaAS+gr5yUTx0X Qk+/9QmKWxSR6nsNulrfu9ooNMT5wO0J/BUOjUykrV/ObykJGHvNtvE1MVO/N04Nwn0XhW2rp/qY7 axmKF6iWrYXjuGMLGw/A==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gQFGk-0007kd-E7; Fri, 23 Nov 2018 17:29:30 +0000 Received: from mail-wr1-x442.google.com ([2a00:1450:4864:20::442]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1gQFGh-0007jq-4O for linux-arm-kernel@lists.infradead.org; Fri, 23 Nov 2018 17:29:28 +0000 Received: by mail-wr1-x442.google.com with SMTP id t3so13060050wrr.3 for ; Fri, 23 Nov 2018 09:29:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=vsz8kXb6CVVsBzyFCRZkaXVWSBUyNLZRbPvH2/3m1zI=; b=Cii11l+5xtXHlR9rXHCV63qwX5vwMHzHo8g/av47h2GbA8ZB0n8xnNUVYoZix5gzSL EzhMannNO+RLhXEirMVjjyRDZGZlauqrsA+zseqpgT/NgjAXE7IsHm8rOLtC0hJ5mLLI r94bw82iV4vBldk9UgMQWQN583nbS/ZvAkQCU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=vsz8kXb6CVVsBzyFCRZkaXVWSBUyNLZRbPvH2/3m1zI=; b=rAhYPnUKLtbPX9yFoIEZPbnPz+J/3lwN7YxS3TkTit+km+CFSQW7cIu7rUdC/FxUdN CAW7u0oI3Epm3Ga7rC5gni2SoxOdL1kEL7PGV5hRe8eYwFfoszO9wFXyCvMiSeLCqOvV 1+/DnFEuLAhm/y9qxrwRpp4EVSkQiRWAAW466jZnYa1wvUvU0ul4gbKYhy7210eVzxPU 9dMWQXgoYU6fuIR2/TFothl0IgyGoRMORuoIdUoXd8FrP7EaNib7Fht8lViQ7kAbxhHk ghMVSLOmvaYCyZef2bsX9c32PVnajt93IQp2uhsAAmGFiKj58A3p9pAJjgp47Id52A3i fD2Q== X-Gm-Message-State: AA+aEWbNhW5ktGxZ4TV2Fqt0Cq08oOzAfPzKxmRcegb4+m6PkiUyADTq X0+Uj6IrBs5/YwuFA+zuaeo9Ww== X-Google-Smtp-Source: AFSGD/W8AV7FoFifqisBZaV4fzOZN1nhgsgcHa2BfC2aXU9iZ+TBe2vik5IRRlXpJVueQD3qTlyi5A== X-Received: by 2002:adf:c6c3:: with SMTP id c3-v6mr14341030wrh.243.1542994153948; Fri, 23 Nov 2018 09:29:13 -0800 (PST) Received: from harold.home ([2a01:cb1d:112:6f00:6913:f64b:5e59:5ba5]) by smtp.gmail.com with ESMTPSA id 125sm5041606wml.35.2018.11.23.09.29.12 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 23 Nov 2018 09:29:12 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Subject: [PATCH] arm64/bpf: use movn/movk/movk sequence to generate kernel addresses Date: Fri, 23 Nov 2018 18:29:02 +0100 Message-Id: <20181123172902.21480-1-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.19.1 MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20181123_092927_170720_1833CCCD X-CRM114-Status: GOOD ( 14.59 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: daniel@iogearbox.net, Ard Biesheuvel , catalin.marinas@arm.com, ast@fb.com, will.deacon@arm.com, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP On arm64, all executable code is guaranteed to reside in the vmalloc space (or the module space), and so jump targets will only use 48 bits at most, and the remaining bits are guaranteed to be 0x1. This means we can generate an immediate jump address using a sequence of one MOVN (move wide negated) and two MOVK instructions, where the first one sets the lower 16 bits but also sets all top bits to 0x1. Signed-off-by: Ard Biesheuvel Acked-by: Will Deacon --- I looked into using ADRP/ADD pairs, but this is very fiddly, since it requires knowledge about where the ADRP instruction ends up in memory. (ADRP produces a PC-relative address with bits [11:0] cleared, and so in addition to the distance between the instruction and the target, we also need to know their offsets modulo 4096 and I wasn't sure whether the offsets are guaranteed to be relative to the start of a page or not) arch/arm64/net/bpf_jit_comp.c | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index a6fdaea07c63..3b4d2c6fc133 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -134,10 +134,9 @@ static inline void emit_a64_mov_i64(const int reg, const u64 val, } /* - * This is an unoptimized 64 immediate emission used for BPF to BPF call - * addresses. It will always do a full 64 bit decomposition as otherwise - * more complexity in the last extra pass is required since we previously - * reserved 4 instructions for the address. + * Kernel addresses in the vmalloc space use at most 48 bits, and the + * remaining bits are guaranteed to be 0x1. So we can compose the address + * with a fixed length movn/movk/movk sequence. */ static inline void emit_addr_mov_i64(const int reg, const u64 val, struct jit_ctx *ctx) @@ -145,8 +144,8 @@ static inline void emit_addr_mov_i64(const int reg, const u64 val, u64 tmp = val; int shift = 0; - emit(A64_MOVZ(1, reg, tmp & 0xffff, shift), ctx); - for (;shift < 48;) { + emit(A64_MOVN(1, reg, ~tmp & 0xffff, shift), ctx); + while (shift < 32) { tmp >>= 16; shift += 16; emit(A64_MOVK(1, reg, tmp & 0xffff, shift), ctx); @@ -627,10 +626,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx) const u8 r0 = bpf2a64[BPF_REG_0]; const u64 func = (u64)__bpf_call_base + imm; - if (ctx->prog->is_func) - emit_addr_mov_i64(tmp, func, ctx); - else - emit_a64_mov_i64(tmp, func, ctx); + emit_addr_mov_i64(tmp, func, ctx); emit(A64_BLR(tmp), ctx); emit(A64_MOV(1, r0, A64_R(0)), ctx); break;