From patchwork Mon Mar 9 08:22:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zong Li X-Patchwork-Id: 11426245 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1CEC314B4 for ; Mon, 9 Mar 2020 08:23:06 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id EEDE520848 for ; Mon, 9 Mar 2020 08:23:05 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="rwf8U1xU"; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="kMXhmKUj" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EEDE520848 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=sifive.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-riscv-bounces+patchwork-linux-riscv=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-Id:Date :Subject:To:From:Reply-To:Content-Type:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=SUgPxjUbuZfwtdr/gn2mVzPVC5s6lk1ipDYoBiMhXl8=; b=rwf8U1xUmw+Gny icskq7i+abMM/IG6SWh+SOnwC3ceBboA1JE8ZC2JbL4R7DRdMjdbgn3GiY2gKTx2jFTMe4Tmt3PJP IgsyBf8rPSyISo/ahHioHS76N4Xk9S5wO6Hfkw9TzSNPmmvCzsF8UuWTminZ7Cp7JopDMhwPsvm3r 4E/ntlZnyWyy6zhP6GQK3U2C3G2cL+ozYWh8AECKH14epmf/Hf1on5T/E3jK3OomJP09M7nhsIBP7 NImh2z8/RpyGaCkLpe/zKPo9BVqkX3Pu1EYCMbn5a/eIierIqC33YrtL7nIYLYvJDPRd5CA2avj1R 8cF0gQRoDtDVLNVtz2HA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jBDgl-0006KE-0A; Mon, 09 Mar 2020 08:23:03 +0000 Received: from mail-pg1-x544.google.com ([2607:f8b0:4864:20::544]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jBDgc-00066l-9m for linux-riscv@lists.infradead.org; Mon, 09 Mar 2020 08:22:56 +0000 Received: by mail-pg1-x544.google.com with SMTP id t3so4384217pgn.1 for ; Mon, 09 Mar 2020 01:22:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=SUgPxjUbuZfwtdr/gn2mVzPVC5s6lk1ipDYoBiMhXl8=; b=kMXhmKUjjLuTmELTTBo2VSu+j5H+0yCyM/etwFY9KFEyI06KLuI5SOuJF4o0jIyhy1 E+hNEFryXrxMvEvFIDUIQXKFcPQMBjSKEyu5zVEz9eP/UZTlo02MDIYxxeCjefKf9Mpz d2+inu7g+F0IvbvLCpjo82rktXLgY8EKKIHUVVKxVcD4AUemdwohaXfCIxzSy2SPzHb3 2UfdVbjWLJIOCvJYKPbxBm0z+JW6f58FtkYv2HKU6nSGXvNshfVYz+YhvpN0eI1MowRo 6eeokIqGQlrlIgCzEzVQJvU9OSrjSs1uK4sui+YQgJPWuRMie/4X9+b7mOYQfakI9MEl wzvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=SUgPxjUbuZfwtdr/gn2mVzPVC5s6lk1ipDYoBiMhXl8=; b=D6WKBmkovdJXmkhaZmhqkibXMTcuGb7eKN9xWpxrhPPzm2vqtUl34enZdg1R3eS3wh 6SZbHibt4EIvSEU+SRnxJeETbonlFRFMdJpbicilJG3x4EAFi+jY1bFW+Z9IZib1Kplo ApU78Xnnun82ut2XjERIEA/npcXY79UYrIO1ciPpoc9FXFwkreHiNnqrNi6/KwRJUW2v vylZkEnhO127EMUUWe7czaARotbME/Jnq5NVt9KUTvdBKTG3wNl5CQHGDPqEN5jO7ALa 1HXtlsTSZJAQKvy5iGPpbXtui48P+mk8jjWNYGVBbwHHWooJt+Vg9w99oYPy6JtOzXKu IkmQ== X-Gm-Message-State: ANhLgQ1qcUwV6Ti/T3bib8zMSGH5yMvwKCWZoDn5zeDTWyOjARdhTTao AzPLITV1chVl/UBDeO8pdtD+oA== X-Google-Smtp-Source: ADFU+vvWldbxBUJVZjNo3MZvwwVUc0e56Ke+qyKeIFd/Xff4hPsQhVXqcSZfRxRIgLhE/pts7VP3Ew== X-Received: by 2002:a63:2c50:: with SMTP id s77mr14822444pgs.182.1583742173481; Mon, 09 Mar 2020 01:22:53 -0700 (PDT) Received: from hsinchu02.internal.sifive.com (220-132-236-182.HINET-IP.hinet.net. [220.132.236.182]) by smtp.gmail.com with ESMTPSA id v5sm18364779pfn.64.2020.03.09.01.22.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Mar 2020 01:22:52 -0700 (PDT) From: Zong Li To: palmer@dabbelt.com, paul.walmsley@sifive.com, aou@eecs.berkeley.edu, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 8/9] riscv: introduce interfaces to patch kernel code Date: Mon, 9 Mar 2020 16:22:28 +0800 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200309_012254_374788_D5437FEB X-CRM114-Status: GOOD ( 15.77 ) X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.3 on bombadil.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:544 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zong Li Sender: "linux-riscv" Errors-To: linux-riscv-bounces+patchwork-linux-riscv=patchwork.kernel.org@lists.infradead.org On strict kernel memory permission, we couldn't patch code without writable permission. Preserve two holes in fixmap area, so we can map the kernel code temporarily to fixmap area, then patch the instructions. We need two pages here because we support the compressed instruction, so the instruction might be align to 2 bytes. When patching the 32-bit length instruction which is 2 bytes alignment, it will across two pages. Introduce two interfaces to patch kernel code: riscv_patch_text_nosync: - patch code without synchronization, it's caller's responsibility to synchronize all CPUs if needed. riscv_patch_text: - patch code and always synchronize with stop_machine() Signed-off-by: Zong Li Reported-by: kbuild test robot Reported-by: kbuild test robot --- arch/riscv/include/asm/fixmap.h | 2 + arch/riscv/include/asm/patch.h | 12 ++++ arch/riscv/kernel/Makefile | 4 +- arch/riscv/kernel/patch.c | 124 ++++++++++++++++++++++++++++++++ 4 files changed, 141 insertions(+), 1 deletion(-) create mode 100644 arch/riscv/include/asm/patch.h create mode 100644 arch/riscv/kernel/patch.c diff --git a/arch/riscv/include/asm/fixmap.h b/arch/riscv/include/asm/fixmap.h index 42d2c42f3cc9..2368d49eb4ef 100644 --- a/arch/riscv/include/asm/fixmap.h +++ b/arch/riscv/include/asm/fixmap.h @@ -27,6 +27,8 @@ enum fixed_addresses { FIX_FDT = FIX_FDT_END + FIX_FDT_SIZE / PAGE_SIZE - 1, FIX_PTE, FIX_PMD, + FIX_TEXT_POKE1, + FIX_TEXT_POKE0, FIX_EARLYCON_MEM_BASE, __end_of_fixed_addresses }; diff --git a/arch/riscv/include/asm/patch.h b/arch/riscv/include/asm/patch.h new file mode 100644 index 000000000000..b5918a6e0615 --- /dev/null +++ b/arch/riscv/include/asm/patch.h @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2020 SiFive + */ + +#ifndef _ASM_RISCV_PATCH_H +#define _ASM_RISCV_PATCH_H + +int riscv_patch_text_nosync(void *addr, const void *insns, size_t len); +int riscv_patch_text(void *addr, u32 insn); + +#endif /* _ASM_RISCV_PATCH_H */ diff --git a/arch/riscv/kernel/Makefile b/arch/riscv/kernel/Makefile index f40205cb9a22..d189bd3d8501 100644 --- a/arch/riscv/kernel/Makefile +++ b/arch/riscv/kernel/Makefile @@ -4,7 +4,8 @@ # ifdef CONFIG_FTRACE -CFLAGS_REMOVE_ftrace.o = -pg +CFLAGS_REMOVE_ftrace.o = -pg +CFLAGS_REMOVE_patch.o = -pg endif extra-y += head.o @@ -26,6 +27,7 @@ obj-y += traps.o obj-y += riscv_ksyms.o obj-y += stacktrace.o obj-y += cacheinfo.o +obj-y += patch.o obj-$(CONFIG_MMU) += vdso.o vdso/ obj-$(CONFIG_RISCV_M_MODE) += clint.o diff --git a/arch/riscv/kernel/patch.c b/arch/riscv/kernel/patch.c new file mode 100644 index 000000000000..1c61a8595839 --- /dev/null +++ b/arch/riscv/kernel/patch.c @@ -0,0 +1,124 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2020 SiFive + */ + +#include +#include +#include +#include +#include +#include +#include + +struct riscv_insn_patch { + void *addr; + u32 insn; + atomic_t cpu_count; +}; + +#ifdef CONFIG_MMU +static DEFINE_RAW_SPINLOCK(patch_lock); + +static void __kprobes *patch_map(void *addr, int fixmap) +{ + uintptr_t uintaddr = (uintptr_t) addr; + struct page *page; + + if (core_kernel_text(uintaddr)) + page = phys_to_page(__pa_symbol(addr)); + else if (IS_ENABLED(CONFIG_STRICT_MODULE_RWX)) + page = vmalloc_to_page(addr); + else + return addr; + + BUG_ON(!page); + + return (void *)set_fixmap_offset(fixmap, page_to_phys(page) + + (uintaddr & ~PAGE_MASK)); +} + +static void __kprobes patch_unmap(int fixmap) +{ + clear_fixmap(fixmap); +} +#else +static void __kprobes *patch_map(void *addr, int fixmap) +{ + return addr; +} + +static void __kprobes patch_unmap(int fixmap) +{ +} +#endif /* CONFIG_MMU */ + +static int __kprobes riscv_insn_write(void *addr, const void *insn, size_t len) +{ + void *waddr = addr; + bool across_pages = (((uintptr_t) addr & ~PAGE_MASK) + len) > PAGE_SIZE; + unsigned long flags = 0; + int ret; + + raw_spin_lock_irqsave(&patch_lock, flags); + + if (across_pages) + patch_map(addr + len, FIX_TEXT_POKE1); + + waddr = patch_map(addr, FIX_TEXT_POKE0); + + ret = probe_kernel_write(waddr, insn, len); + + patch_unmap(FIX_TEXT_POKE0); + + if (across_pages) + patch_unmap(FIX_TEXT_POKE1); + + raw_spin_unlock_irqrestore(&patch_lock, flags); + + return ret; +} + +int __kprobes riscv_patch_text_nosync(void *addr, const void *insns, size_t len) +{ + u32 *tp = addr; + int ret; + + ret = riscv_insn_write(tp, insns, len); + + if (!ret) + flush_icache_range((uintptr_t) tp, (uintptr_t) tp + len); + + return ret; +} + +static int __kprobes riscv_patch_text_cb(void *data) +{ + struct riscv_insn_patch *patch = data; + int ret = 0; + + if (atomic_inc_return(&patch->cpu_count) == 1) { + ret = + riscv_patch_text_nosync(patch->addr, &patch->insn, + GET_INSN_LENGTH(patch->insn)); + atomic_inc(&patch->cpu_count); + } else { + while (atomic_read(&patch->cpu_count) <= num_online_cpus()) + cpu_relax(); + smp_mb(); + } + + return ret; +} + +int __kprobes riscv_patch_text(void *addr, u32 insn) +{ + struct riscv_insn_patch patch = { + .addr = addr, + .insn = insn, + .cpu_count = ATOMIC_INIT(0), + }; + + return stop_machine_cpuslocked(riscv_patch_text_cb, + &patch, cpu_online_mask); +}