From patchwork Fri Apr 22 05:18:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Song Liu X-Patchwork-Id: 12822748 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3293AC433EF for ; Fri, 22 Apr 2022 05:24:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C9FA86B0075; Fri, 22 Apr 2022 01:24:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C4F906B0078; Fri, 22 Apr 2022 01:24:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B65726B007B; Fri, 22 Apr 2022 01:24:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.26]) by kanga.kvack.org (Postfix) with ESMTP id A6B0A6B0075 for ; Fri, 22 Apr 2022 01:24:37 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 522A6923 for ; Fri, 22 Apr 2022 05:24:33 +0000 (UTC) X-FDA: 79383374826.08.D581308 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf04.hostedemail.com (Postfix) with ESMTP id 9FF794001A for ; Fri, 22 Apr 2022 05:24:30 +0000 (UTC) Received: from pps.filterd (m0109334.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 23LNWiuG017164 for ; Thu, 21 Apr 2022 22:24:31 -0700 Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3fjtd41jad-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 21 Apr 2022 22:24:31 -0700 Received: from twshared13345.18.frc3.facebook.com (2620:10d:c085:208::f) by mail.thefacebook.com (2620:10d:c085:11d::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Thu, 21 Apr 2022 22:24:30 -0700 Received: by devbig932.frc1.facebook.com (Postfix, from userid 4523) id D979360E40E5; Thu, 21 Apr 2022 22:18:18 -0700 (PDT) From: Song Liu To: , , CC: , , , , , , , , Song Liu Subject: [PATCH bpf 1/4] bpf: invalidate unused part of bpf_prog_pack Date: Thu, 21 Apr 2022 22:18:10 -0700 Message-ID: <20220422051813.1989257-2-song@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220422051813.1989257-1-song@kernel.org> References: <20220422051813.1989257-1-song@kernel.org> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: hkpjtgRcHCoxcKeu5X_bEAAwSVyYfbdb X-Proofpoint-GUID: hkpjtgRcHCoxcKeu5X_bEAAwSVyYfbdb X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.858,Hydra:6.0.486,FMLib:17.11.64.514 definitions=2022-04-22_01,2022-04-21_01,2022-02-23_01 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 9FF794001A X-Rspam-User: Authentication-Results: imf04.hostedemail.com; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=kernel.org (policy=none); spf=none (imf04.hostedemail.com: domain of "prvs=5111fe83e2=songliubraving@fb.com" has no SPF policy when checking 67.231.145.42) smtp.mailfrom="prvs=5111fe83e2=songliubraving@fb.com" X-Stat-Signature: bfyke8bbsyrpdjorjw9fmsf4iqsm4h94 X-HE-Tag: 1650605070-968616 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: bpf_prog_pack enables sharing huge pages among multiple BPF programs. These pages are marked as executable, but some part of these huge page may not contain proper BPF programs. To make these pages safe, fill such unused part of these pages with invalid instructions. This is done when a pack is first allocated, and when a bpf program is freed.. Fixes: 57631054fae6 ("bpf: Introduce bpf_prog_pack allocator") Fixes: 33c9805860e5 ("bpf: Introduce bpf_jit_binary_pack_[alloc|finalize|free]") Signed-off-by: Song Liu --- arch/x86/net/bpf_jit_comp.c | 22 ++++++++++++++++++++++ include/linux/bpf.h | 2 ++ kernel/bpf/core.c | 21 +++++++++++++++++---- 3 files changed, 41 insertions(+), 4 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 16b6efacf7c6..ff3f0811c9a1 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -228,6 +228,28 @@ static void jit_fill_hole(void *area, unsigned int size) memset(area, 0xcc, size); } +#define INVALID_BUF_SIZE PAGE_SIZE +static char invalid_insn_buf[INVALID_BUF_SIZE]; + +static int __init bpf_init_invalid_insn_buf(void) +{ + jit_fill_hole(invalid_insn_buf, INVALID_BUF_SIZE); + return 0; +} +pure_initcall(bpf_init_invalid_insn_buf); + +void bpf_arch_invalidate_text(void *dst, size_t len) +{ + size_t i = 0; + + while (i < len) { + size_t s = min_t(size_t, len - i, INVALID_BUF_SIZE); + + bpf_arch_text_copy(dst + i, invalid_insn_buf, s); + i += s; + } +} + struct jit_context { int cleanup_addr; /* Epilogue code offset */ diff --git a/include/linux/bpf.h b/include/linux/bpf.h index bdb5298735ce..2157392a94d1 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -2382,6 +2382,8 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t, void *bpf_arch_text_copy(void *dst, void *src, size_t len); +void bpf_arch_invalidate_text(void *dst, size_t len); + struct btf_id_set; bool btf_id_set_contains(const struct btf_id_set *set, u32 id); diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 13e9dbeeedf3..2be2d73fb6a8 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -873,7 +873,7 @@ static size_t select_bpf_prog_pack_size(void) return size; } -static struct bpf_prog_pack *alloc_new_pack(void) +static struct bpf_prog_pack *alloc_new_pack(bpf_jit_fill_hole_t bpf_fill_ill_insns) { struct bpf_prog_pack *pack; @@ -886,6 +886,7 @@ static struct bpf_prog_pack *alloc_new_pack(void) kfree(pack); return NULL; } + bpf_fill_ill_insns(pack->ptr, bpf_prog_pack_size); bitmap_zero(pack->bitmap, bpf_prog_pack_size / BPF_PROG_CHUNK_SIZE); list_add_tail(&pack->list, &pack_list); @@ -895,7 +896,7 @@ static struct bpf_prog_pack *alloc_new_pack(void) return pack; } -static void *bpf_prog_pack_alloc(u32 size) +static void *bpf_prog_pack_alloc(u32 size, bpf_jit_fill_hole_t bpf_fill_ill_insns) { unsigned int nbits = BPF_PROG_SIZE_TO_NBITS(size); struct bpf_prog_pack *pack; @@ -910,6 +911,7 @@ static void *bpf_prog_pack_alloc(u32 size) size = round_up(size, PAGE_SIZE); ptr = module_alloc(size); if (ptr) { + bpf_fill_ill_insns(ptr, size); set_vm_flush_reset_perms(ptr); set_memory_ro((unsigned long)ptr, size / PAGE_SIZE); set_memory_x((unsigned long)ptr, size / PAGE_SIZE); @@ -923,7 +925,7 @@ static void *bpf_prog_pack_alloc(u32 size) goto found_free_area; } - pack = alloc_new_pack(); + pack = alloc_new_pack(bpf_fill_ill_insns); if (!pack) goto out; @@ -966,6 +968,7 @@ static void bpf_prog_pack_free(struct bpf_binary_header *hdr) nbits = BPF_PROG_SIZE_TO_NBITS(hdr->size); pos = ((unsigned long)hdr - (unsigned long)pack_ptr) >> BPF_PROG_CHUNK_SHIFT; + bpf_arch_invalidate_text(hdr, hdr->size); bitmap_clear(pack->bitmap, pos, nbits); if (bitmap_find_next_zero_area(pack->bitmap, bpf_prog_chunk_count(), 0, bpf_prog_chunk_count(), 0) == 0) { @@ -1102,7 +1105,7 @@ bpf_jit_binary_pack_alloc(unsigned int proglen, u8 **image_ptr, if (bpf_jit_charge_modmem(size)) return NULL; - ro_header = bpf_prog_pack_alloc(size); + ro_header = bpf_prog_pack_alloc(size, bpf_fill_ill_insns); if (!ro_header) { bpf_jit_uncharge_modmem(size); return NULL; @@ -1203,6 +1206,16 @@ void __weak bpf_jit_free(struct bpf_prog *fp) bpf_prog_unlock_free(fp); } +void __weak bpf_arch_invalidate_text(void *dst, size_t len) +{ + char buf[1] = {}; + int i; + + WARN_ONCE(1, "Please override %s for bpf_prog_pack\n", __func__); + for (i = 0; i < len; i++) + bpf_arch_text_copy(dst + i, buf, 1); +} + int bpf_jit_get_func_addr(const struct bpf_prog *prog, const struct bpf_insn *insn, bool extra_pass, u64 *func_addr, bool *func_addr_fixed)