From patchwork Fri Mar 8 01:08:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13586414 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06F89C54E49 for ; Fri, 8 Mar 2024 01:08:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7FA806B0302; Thu, 7 Mar 2024 20:08:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 783136B0301; Thu, 7 Mar 2024 20:08:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6087D6B0302; Thu, 7 Mar 2024 20:08:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 4C1126B02FF for ; Thu, 7 Mar 2024 20:08:47 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 1CF60C1204 for ; Fri, 8 Mar 2024 01:08:47 +0000 (UTC) X-FDA: 81872087094.12.C5086CA Received: from mail-oa1-f45.google.com (mail-oa1-f45.google.com [209.85.160.45]) by imf03.hostedemail.com (Postfix) with ESMTP id 6B74C2001A for ; Fri, 8 Mar 2024 01:08:45 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=k8ECFwpu; spf=pass (imf03.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.160.45 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709860125; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zJ3wipPJkzBXkAf+0tGonUTvUit9U85GVfiOeVf+bQg=; b=20/FStcWIpfJ0/PrlnIU3zsreJq7RDYDzikhKdQ3fo4cC5sBwvVE5M7+CEvoRWGEegEfMz /7te1RVNnvk/K8mIbO3mb7VqOy4gZckLlNgaxLXCRwI42kB891TP/z/50v0s5uQAVHr25A xA3LkaWxK73Qq+qsGqOFToG+s3X5RjE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709860125; a=rsa-sha256; cv=none; b=Qy+RSg4+D8fk/Grpf78ctOTo37XNudnRm9HvgYpdXkro72MYCKMC4MqJf1kA91jFfPP6xa 143Fengr2GmXjhxuT+1Xq65oegj8e1iLvUUbCJdxwz9iYDVvfZIi/biKAMvDnaUIfhvavR gJpQWS3zskQLvK5GJ3x2QKnOEi1PJMI= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=k8ECFwpu; spf=pass (imf03.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.160.45 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-oa1-f45.google.com with SMTP id 586e51a60fabf-220461b7597so700399fac.3 for ; Thu, 07 Mar 2024 17:08:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709860124; x=1710464924; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=zJ3wipPJkzBXkAf+0tGonUTvUit9U85GVfiOeVf+bQg=; b=k8ECFwpuGewYO8ESVQof05XC8UMjVWWjL3QcNynAXIn26AH0WHzOV2LLx3wrU9J9tf 1pewpqyQtCn0iieBbUGf2DgQoF0PFzGxf3CSUGEz+V8zDzRCLPaddqQbv+c29vbTpCIo hGn1CA2Hm4EldvWnDWa1lVZ437Fm1o5vG3QE7x4+MgoicNPXilrzCD3xjozD0muot4QV HzGvPWJJpPt2b0rbF2nYB8iI5Hz3m85LzXKujo4UEMFB2o/k0fBj6XnLgItBpr+TPK9y IEc0QLw6vO05n5/iAFjDPggFPm1x0qaG/QdyJW3XqyryVm5l7qlaZ0NRm+Fh/7x0S+Y0 ZbzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709860124; x=1710464924; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zJ3wipPJkzBXkAf+0tGonUTvUit9U85GVfiOeVf+bQg=; b=tednf3ps4ocf4xuUul4pulU7UEUNyxltNiMK0FKgArz53UKUZvHVgA68E96aijeLNi j7UspsrQbo4T+Kd/vg2QBU5GGnWYKcQT88t7brdAa/XfHU8ZQ+1oLx0oBKdqzkHEdgGi LmPaFnVBEH02jnSittKVY2SYKUVcQcYXUYdP1awA0Ua8TID8TczVsa299NeGaEwjlU3e nGQ4SnPrM+wHE+OHh1KBc/cEz/yvLgDEvcXIzXOrKtDlwGlpQ4GY7PmEisl+GFBQeNGO 0Wux7RK/GMNv0fis8u4wrtROlK5rmLb91dCtNm5yvPOjCmWCdbA3EcGOs31sLhZJWPM8 5pjg== X-Forwarded-Encrypted: i=1; AJvYcCX7eYemC0PWlLPa99saaKkBCCHioh4hHcz/aewXSXPjUECqA6SXE1kn9l4lHjjuN6aal0By8GeJLpy9tlUYu/xF3Iw= X-Gm-Message-State: AOJu0YzfWYR8QsryOlYtkne4iAGf6LvPgzfwJfZYDrYAJg6OWjMHiPOl pcMI4pGELUj9LsavP07reBJvxAYEA8c3nI3YAsfWFiolc+xjLZl3 X-Google-Smtp-Source: AGHT+IEHBzX1FEefqo9orjWOgHuRn9YByzCkzqsCH0sciZEM8nYEiDIls562lf4GmBYo0ky+H2anKA== X-Received: by 2002:a05:6871:411:b0:220:8ddb:3f6 with SMTP id d17-20020a056871041100b002208ddb03f6mr1832565oag.12.1709860124584; Thu, 07 Mar 2024 17:08:44 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:428c]) by smtp.gmail.com with ESMTPSA id k30-20020a63ba1e000000b005d6b5934deesm11679456pgf.48.2024.03.07.17.08.43 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 07 Mar 2024 17:08:44 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, torvalds@linux-foundation.org, brho@google.com, hannes@cmpxchg.org, akpm@linux-foundation.org, urezki@gmail.com, hch@infradead.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH v3 bpf-next 08/14] libbpf: Add support for bpf_arena. Date: Thu, 7 Mar 2024 17:08:06 -0800 Message-Id: <20240308010812.89848-9-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240308010812.89848-1-alexei.starovoitov@gmail.com> References: <20240308010812.89848-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 6B74C2001A X-Rspam-User: X-Stat-Signature: 4i1911g31zbf9786ana8t6xcnek5ngbi X-Rspamd-Server: rspam03 X-HE-Tag: 1709860125-389149 X-HE-Meta: U2FsdGVkX1+oT6YCOY9cj4g6vwszHWIOx/iBVCHp4o8I//VIMYWI2gtEcWYNS7XX5aobPBaq6WO2fWfy9rgiBf8oFKIwos9X1OVCSmHhhoiuEnsJpSo4sDop9cPcMBywscZiRjE6YqMyhWjJ3WTttmP5Vu5ZVL3gVYdxr0mQ9kGuEz8/L/Px8UCXh/TYf2HJ3F5F0OFFXkgIvtddNzPMmmW36ubSvZK5YKDQQ7JsxqtRouigmkrvDw50BNCMWjgdZrk8tJ0ibKRl8Mep95R/GGbykZ19KfI6FN4IuM8G8qjcEQwvBpaotuF4qwak5Jp9fD03E8SnM1LuyAtD09I+2I3uprbzz+ZZVGknWqxNXIywntqtqdpitO/A/zXwfmKF13eSYxKDZnCGxqmDI1Q0Wb46R6UmBmAfGHn/eMSReMBVosZn3KK2f6llyqsnzLeJP2EeLpXUIs6qsLqqzxlzN7ODE0wcNMs/sRS9Eny28oa7sO6A0Ugv5qlG8xml1t4zNTUqwo53qhEP3ydU3u4IX1UONcqAJw1vWHmo1H/iqmNhzP5h7DSswR1Ii49ZxsidGB3PZ244DdSjrzuCyATD4fGJqy7F6hAOoUIVDwoPLojMY8GOQQfL7718M+58lttN13cD+SD+qh3oA5cSGC1tZDTOGjR5bj4xCpdWiYiq55amOYVar4KhU+vcimL7J9Wb6c3Rk2/B7sly61cl6OS78UL01XW8+7mSnoaUbZe90JGEIs+9VT8GAt775+BCGCXkp1aJE/BEX32bonjygC7qMIkUlQtVZDHm1qLB5k4I0ulYav171+eEDDnhr7wEF6LwWurgB+MIFdeC/W0I4P5N4GslPEiN0R6dBtrFDN6YJrVL6sEVM6suknOLmRlSIkwY4he3GrvKEO4ZCY1Xq/WJAjkc2a46lY6VWImLNk131q4Aft6tDioHBQoilUtnff6sz4hh6Z1x2hf8T2r6KTD utPM9Lzg QrnScxOrzxGwv6Ayj3bjM7wdKdqhXQeUfoDSLnkiNzawr66IlbMvdJ3YUFYXzT06RclIfbfBM8sAsUBmtBvAIZKETnH1LGyIakZ9XTv2Rt+SnINIfVIJBPkperK0sV97sOyrWrUbSm5fDczDaLNUPGf46ywbHLyNrrw4sL6edC/xxHbERpe8lbdXo9NIV9wqp1Abo X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov mmap() bpf_arena right after creation, since the kernel needs to remember the address returned from mmap. This is user_vm_start. LLVM will generate bpf_arena_cast_user() instructions where necessary and JIT will add upper 32-bit of user_vm_start to such pointers. Fix up bpf_map_mmap_sz() to compute mmap size as map->value_size * map->max_entries for arrays and PAGE_SIZE * map->max_entries for arena. Don't set BTF at arena creation time, since it doesn't support it. Signed-off-by: Alexei Starovoitov --- tools/lib/bpf/libbpf.c | 47 +++++++++++++++++++++++++++++------ tools/lib/bpf/libbpf_probes.c | 7 ++++++ 2 files changed, 46 insertions(+), 8 deletions(-) diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 567ad367e7aa..34b722071874 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -185,6 +185,7 @@ static const char * const map_type_name[] = { [BPF_MAP_TYPE_BLOOM_FILTER] = "bloom_filter", [BPF_MAP_TYPE_USER_RINGBUF] = "user_ringbuf", [BPF_MAP_TYPE_CGRP_STORAGE] = "cgrp_storage", + [BPF_MAP_TYPE_ARENA] = "arena", }; static const char * const prog_type_name[] = { @@ -1684,7 +1685,7 @@ static struct bpf_map *bpf_object__add_map(struct bpf_object *obj) return map; } -static size_t bpf_map_mmap_sz(unsigned int value_sz, unsigned int max_entries) +static size_t array_map_mmap_sz(unsigned int value_sz, unsigned int max_entries) { const long page_sz = sysconf(_SC_PAGE_SIZE); size_t map_sz; @@ -1694,6 +1695,20 @@ static size_t bpf_map_mmap_sz(unsigned int value_sz, unsigned int max_entries) return map_sz; } +static size_t bpf_map_mmap_sz(const struct bpf_map *map) +{ + const long page_sz = sysconf(_SC_PAGE_SIZE); + + switch (map->def.type) { + case BPF_MAP_TYPE_ARRAY: + return array_map_mmap_sz(map->def.value_size, map->def.max_entries); + case BPF_MAP_TYPE_ARENA: + return page_sz * map->def.max_entries; + default: + return 0; /* not supported */ + } +} + static int bpf_map_mmap_resize(struct bpf_map *map, size_t old_sz, size_t new_sz) { void *mmaped; @@ -1847,7 +1862,7 @@ bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type, pr_debug("map '%s' (global data): at sec_idx %d, offset %zu, flags %x.\n", map->name, map->sec_idx, map->sec_offset, def->map_flags); - mmap_sz = bpf_map_mmap_sz(map->def.value_size, map->def.max_entries); + mmap_sz = bpf_map_mmap_sz(map); map->mmaped = mmap(NULL, mmap_sz, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0); if (map->mmaped == MAP_FAILED) { @@ -5017,6 +5032,7 @@ static int bpf_object__create_map(struct bpf_object *obj, struct bpf_map *map, b case BPF_MAP_TYPE_SOCKHASH: case BPF_MAP_TYPE_QUEUE: case BPF_MAP_TYPE_STACK: + case BPF_MAP_TYPE_ARENA: create_attr.btf_fd = 0; create_attr.btf_key_type_id = 0; create_attr.btf_value_type_id = 0; @@ -5261,7 +5277,19 @@ bpf_object__create_maps(struct bpf_object *obj) if (err < 0) goto err_out; } - + if (map->def.type == BPF_MAP_TYPE_ARENA) { + map->mmaped = mmap((void *)map->map_extra, bpf_map_mmap_sz(map), + PROT_READ | PROT_WRITE, + map->map_extra ? MAP_SHARED | MAP_FIXED : MAP_SHARED, + map->fd, 0); + if (map->mmaped == MAP_FAILED) { + err = -errno; + map->mmaped = NULL; + pr_warn("map '%s': failed to mmap arena: %d\n", + map->name, err); + return err; + } + } if (map->init_slots_sz && map->def.type != BPF_MAP_TYPE_PROG_ARRAY) { err = init_map_in_map_slots(obj, map); if (err < 0) @@ -8761,7 +8789,7 @@ static void bpf_map__destroy(struct bpf_map *map) if (map->mmaped) { size_t mmap_sz; - mmap_sz = bpf_map_mmap_sz(map->def.value_size, map->def.max_entries); + mmap_sz = bpf_map_mmap_sz(map); munmap(map->mmaped, mmap_sz); map->mmaped = NULL; } @@ -9995,11 +10023,14 @@ int bpf_map__set_value_size(struct bpf_map *map, __u32 size) return libbpf_err(-EBUSY); if (map->mmaped) { - int err; size_t mmap_old_sz, mmap_new_sz; + int err; + + if (map->def.type != BPF_MAP_TYPE_ARRAY) + return -EOPNOTSUPP; - mmap_old_sz = bpf_map_mmap_sz(map->def.value_size, map->def.max_entries); - mmap_new_sz = bpf_map_mmap_sz(size, map->def.max_entries); + mmap_old_sz = bpf_map_mmap_sz(map); + mmap_new_sz = array_map_mmap_sz(size, map->def.max_entries); err = bpf_map_mmap_resize(map, mmap_old_sz, mmap_new_sz); if (err) { pr_warn("map '%s': failed to resize memory-mapped region: %d\n", @@ -13530,7 +13561,7 @@ int bpf_object__load_skeleton(struct bpf_object_skeleton *s) for (i = 0; i < s->map_cnt; i++) { struct bpf_map *map = *s->maps[i].map; - size_t mmap_sz = bpf_map_mmap_sz(map->def.value_size, map->def.max_entries); + size_t mmap_sz = bpf_map_mmap_sz(map); int prot, map_fd = map->fd; void **mmaped = s->maps[i].mmaped; diff --git a/tools/lib/bpf/libbpf_probes.c b/tools/lib/bpf/libbpf_probes.c index ee9b1dbea9eb..302188122439 100644 --- a/tools/lib/bpf/libbpf_probes.c +++ b/tools/lib/bpf/libbpf_probes.c @@ -338,6 +338,13 @@ static int probe_map_create(enum bpf_map_type map_type) key_size = 0; max_entries = 1; break; + case BPF_MAP_TYPE_ARENA: + key_size = 0; + value_size = 0; + max_entries = 1; /* one page */ + opts.map_extra = 0; /* can mmap() at any address */ + opts.map_flags = BPF_F_MMAPABLE; + break; case BPF_MAP_TYPE_HASH: case BPF_MAP_TYPE_ARRAY: case BPF_MAP_TYPE_PROG_ARRAY: