From patchwork Fri Feb 9 04:06:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13550853 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 74FDDC48297 for ; Fri, 9 Feb 2024 04:07:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0167A8D0009; Thu, 8 Feb 2024 23:07:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EE2C78D0001; Thu, 8 Feb 2024 23:07:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D5C188D0009; Thu, 8 Feb 2024 23:07:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id BF8598D0001 for ; Thu, 8 Feb 2024 23:07:06 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 944A8A066B for ; Fri, 9 Feb 2024 04:07:06 +0000 (UTC) X-FDA: 81770930052.06.FC4F8B2 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf14.hostedemail.com (Postfix) with ESMTP id BAE94100002 for ; Fri, 9 Feb 2024 04:07:04 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=MSRpVZ0v; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf14.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707451624; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Nd9ldHvznK5bBX0WOOiLjO64Z4v1t4JC5wlS+JmAlpg=; b=1cmGGv/hOwyIWYzTHl0W3hxCdMSu2LwLR6HuVPeHFvgXUBSxGyyAMNCScSJ2bhL67IQPIp MGRFqO2QfTiE4RXvhCtpYOT+hoY8HVRevmPNH7ddK5fOe/57xwl/qmrlBUSifZy3RGvJk5 TXZUJCy8Zv4gbuky6MMxSM0Ay/0/ejc= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=MSRpVZ0v; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf14.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707451624; a=rsa-sha256; cv=none; b=xFLj6HKIiqRbCpG3FIduwhNvT63WwNV6hO8P2s76a5ErLY3WS/5UqvHEmfFWNAOAZt0Nks a4l666t8otlehFudKbAYfOKLJgFS3XPU8MM5Qg9I26+Aq98sbOhKvwB3SeSeWXYu1pX3DQ I9mKmSrhrRoW0LfUpkS8FdYClECon/s= Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-1d958e0d73dso4425525ad.1 for ; Thu, 08 Feb 2024 20:07:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707451623; x=1708056423; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Nd9ldHvznK5bBX0WOOiLjO64Z4v1t4JC5wlS+JmAlpg=; b=MSRpVZ0vknJGFoHewi2d3ikruX5c6ugOu+diDGVPdr5VHKfjKxl1HMYzDr2v/qKhAD KnLo+Nmq80LeMCKAetnFqMvB3RRAsCfA8vC0O89bisvWjByzvj7waVgJhibUOT7BDg3h HJS+4PSdRqLPMYsp2r8up89Kwm1M3zYuLE8GFkQy0mpG+lxSZKdO9byqX0NzHsKgZH/t 07P15b0hMvFR/1wdJ2WpokJQTzQV1+WFTewZXCHgrEpzP80O9PbYyRkcp9Uknq9XtCzr 9r1veqXt9WvyGcfrMpPdw+tEbhzH/XzZ0kgr+Uzasylv27qHxfPPJVDkP+zekVpWgV+Y JbvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707451623; x=1708056423; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Nd9ldHvznK5bBX0WOOiLjO64Z4v1t4JC5wlS+JmAlpg=; b=Tp3guQ0zx4SrWvl80r1fh9TB1GuvDu3k/We4/m+OnTLoPdbgf966UFDK8BqzBQrRBs kISSVkVlow6XkFOnycq1Rf79al/NOMLPJ+qq8H/TjqEyTxyjJ2Xp4Rf8oUFR5R2QfEta 2vdSi0TqB9nkkZgaRzBMrVdk62aAJR7w0TGJ1LSvcmjbMxrgJ5JExWbTw2plkVAdGMvr CufhQNKAYBboqjCl6yB243WpXpJu54ki/hskktm2gx22RTTuEXWVkKJtzPzjV+EAcbXD T0GDKrXVrdxr0hrapG0KbhEmRvr/Z8SD7uyYVofTcZd2fNnT1hg1r38pdgkdum9cKOgd 2Adg== X-Forwarded-Encrypted: i=1; AJvYcCU6DW/3oZdK52e/bMRy0Rg1AIkAhq5EfAyRFx9uLmXguk4eni02TeewnV0zeuYJ86Wvef2cEFBBlltMB7HCXP+hxvc= X-Gm-Message-State: AOJu0YyVWdojZDuB2ffiZ1Pg/HQ//QaywzOLo8rvI/sEZbfj/wL2VcnU roSoNodfyeiNS2Rn65GWoxuN+CMo2iwU5BGGdVQajX2hM35yFmyU X-Google-Smtp-Source: AGHT+IEoPNaoXNST3bY7EQ3Xy37Qb7fTssbofLabaCcz0TDvqlgviwuOkE+S47i6w80xgnTbu99/eA== X-Received: by 2002:a17:902:ce91:b0:1d9:34ff:f807 with SMTP id f17-20020a170902ce9100b001d934fff807mr727831plg.31.1707451623582; Thu, 08 Feb 2024 20:07:03 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCU4y6CYVGbOKIVod4G5Bg5GTdHTlGslPYpsljRN9o3Z+8SFl6eHgSF+Plus5zKv7ifiilN8+3Rjaqfp4tPFWlgiGm9U4WYTgptRd2cVgf0dvvrSyQcj/sI1ZPcywe5wsAK74e+moJ3nEcZD20bSvOl4UhYy9gcv8vpm3VgPJnpPhUu4irxBkQwSGw2QH6I3K6McMgvIFFbwqhzJxep1QZGhuyXkrRiSjJTp3dKWrYgJ878B2ObgfAyj29Ns+1kI/6tpzGl9I94oP1IN+y3mDfcH7Qk1OeqG6dNbq3FmYOFCEvdYqPdiXRYRMtkXLZqiyMVp1rjFBtFgEcyRZH6cMEDc1G3cXwnNMCz4r4xrV67tuUAMd7uW7g== Received: from macbook-pro-49.dhcp.thefacebook.com ([2620:10d:c090:400::4:a894]) by smtp.gmail.com with ESMTPSA id g12-20020a170902f74c00b001d9ba3b2b33sm541375plw.163.2024.02.08.20.07.01 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 08 Feb 2024 20:07:03 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, lstoakes@gmail.com, akpm@linux-foundation.org, urezki@gmail.com, hch@infradead.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH v2 bpf-next 12/20] libbpf: Add support for bpf_arena. Date: Thu, 8 Feb 2024 20:06:00 -0800 Message-Id: <20240209040608.98927-13-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240209040608.98927-1-alexei.starovoitov@gmail.com> References: <20240209040608.98927-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: BAE94100002 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: t8cjftzksyrmimqaqyzgeth9ojg9y9x6 X-HE-Tag: 1707451624-992582 X-HE-Meta: U2FsdGVkX18PhrS0nVVhiHOU4KPI/6bagD6yx0yipAL2zMUMDxmSM8hGWeiTQ5+MLARbr8xZURP9MNnd5eoYSLoOIUg2EK11GiHH+co8hNqbv6Pnu93jMC7sWJQdrPh7zmiM/ByYEaICJASu4ljcBaae5NT4A1pZulpwr59y7WiPshXwbUK7tavJpJCGxBe5T2fFYME5kHZgm7sJ+faSa2Tgjrzi0FxeeTafcKvSvV5bDMLBsb3TcjLzh8EopWf4p/uWoeT54l2w/Hd3Z1hwzzlSZOiHNs/286W/SRoMOBHKi2wrlBG/+1Kq5LgE0m+hYNzzfK32aSNo6KD09SqdIfrmak9+JieL4deMCHc5NXzXBCzM2wYwGJCDW/VezskJP6ltVtHsVlPj8Wo4fU/nr+zr0/Qj0HUurBzkoDflkHaJK1vrOTOjCtmMHbeKy7CIMVJHapFwBSq7bgi2s1FazqZLRQEuCjvbi9M46VVpuNi+AcqgiYrnweLn5awoN/q7auaa90onFdz8bZ9YHfkrozFUAnc1ha+kX+F5LLOaEuVFlMLbKRV96Oe6vU1mtjZ1yLL9T2zXgVMsW9AZ4rJS5i8e4H/LcEqUuK55UV+sVE5cC3fiwbH7fpQsLYqlkSd++jQ6lOvgd5nsL4P8p2FCDxn9Qu7wVtuvGGpF3Ybsri1UG60hAjF/ZnunibHo66bN7pQPMdnpicfCkG4ktdFSK3vAdxYeY20lNG2ug6H26vvqc2JOlABRcg9AGf2woGaJ4HMtiFR0Mi7cClHzFarmab+A+ahmtnNH2zmUBYYuh6ZoBAUiVmy6At+ciasIKaBurYdekdZzQDxBaRcyy7pZWx7RFGg/VePfvlXQhtC+Kl+zY2gv2zYjzeBmm0dURwt8LXSopAsbd06GuCC8FBKY75IXPw8DyxF76z64lLV5zM0w1jtGk1LnF/Zn1zYPfykZEsE6peOZpANtcRYF4JJ ZF/VkLX7 s2O7UbbKGcZMql0bOIXhXhyhxfYZIzFxVqZzJUtPWdV6x+P4D+hkHfu1YbJ8LQZdV6lyTa48F8S5YG5kj0hKWKjA+lCDjuIZplAj3nJoG9XIKEBpZah/hP/kJPlr9PVwxlVipBcRe1VimuCErstN4AMllWw+D2epZSymagYCo2e+dT8FxJf4TmY3nPxbVwhXc6AmCaYEvfGoXvZZXYjWpNPPSNxYsgBumA4DXDIyXK2uq/RutELwDc2wazgJmA0nKL1A7B/6k9U8DiyuBZNgj2l/DKr9bu74p8s5qlRcyHDzKJFNefQNMmnW5g4DVrUsbYzT8CDaUY2su6ieLigL4Xq1sWXrBTMPlc4q4bkpY8vdVAQ8uSfkpjK9Z8J8N3+QxSQI2tNp58qe2Rdzso2kSzDFbRFz98aVyl7de908didVCPkY2nCs6X+8sJJC8H73SBXScxyOr3pvpdfojFGaR7qUW1xvUkECVcLp2L5kYJzFq9qiLjJKVDstaoh5xBdvYh4CM91sOafHFbBTSFs6MmI2U0N+Tu3Zep1FDVkCCIyBddtSBkxexRTb1HSH+OVUljgQTTlaTJtKjAESkJpl2I0cWP6ijdQr3RNKn X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov mmap() bpf_arena right after creation, since the kernel needs to remember the address returned from mmap. This is user_vm_start. LLVM will generate bpf_arena_cast_user() instructions where necessary and JIT will add upper 32-bit of user_vm_start to such pointers. Fix up bpf_map_mmap_sz() to compute mmap size as map->value_size * map->max_entries for arrays and PAGE_SIZE * map->max_entries for arena. Don't set BTF at arena creation time, since it doesn't support it. Signed-off-by: Alexei Starovoitov --- tools/lib/bpf/libbpf.c | 43 ++++++++++++++++++++++++++++++----- tools/lib/bpf/libbpf_probes.c | 7 ++++++ 2 files changed, 44 insertions(+), 6 deletions(-) diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 01f407591a92..4880d623098d 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -185,6 +185,7 @@ static const char * const map_type_name[] = { [BPF_MAP_TYPE_BLOOM_FILTER] = "bloom_filter", [BPF_MAP_TYPE_USER_RINGBUF] = "user_ringbuf", [BPF_MAP_TYPE_CGRP_STORAGE] = "cgrp_storage", + [BPF_MAP_TYPE_ARENA] = "arena", }; static const char * const prog_type_name[] = { @@ -1577,7 +1578,7 @@ static struct bpf_map *bpf_object__add_map(struct bpf_object *obj) return map; } -static size_t bpf_map_mmap_sz(unsigned int value_sz, unsigned int max_entries) +static size_t __bpf_map_mmap_sz(unsigned int value_sz, unsigned int max_entries) { const long page_sz = sysconf(_SC_PAGE_SIZE); size_t map_sz; @@ -1587,6 +1588,20 @@ static size_t bpf_map_mmap_sz(unsigned int value_sz, unsigned int max_entries) return map_sz; } +static size_t bpf_map_mmap_sz(const struct bpf_map *map) +{ + const long page_sz = sysconf(_SC_PAGE_SIZE); + + switch (map->def.type) { + case BPF_MAP_TYPE_ARRAY: + return __bpf_map_mmap_sz(map->def.value_size, map->def.max_entries); + case BPF_MAP_TYPE_ARENA: + return page_sz * map->def.max_entries; + default: + return 0; /* not supported */ + } +} + static int bpf_map_mmap_resize(struct bpf_map *map, size_t old_sz, size_t new_sz) { void *mmaped; @@ -1740,7 +1755,7 @@ bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type, pr_debug("map '%s' (global data): at sec_idx %d, offset %zu, flags %x.\n", map->name, map->sec_idx, map->sec_offset, def->map_flags); - mmap_sz = bpf_map_mmap_sz(map->def.value_size, map->def.max_entries); + mmap_sz = bpf_map_mmap_sz(map); map->mmaped = mmap(NULL, mmap_sz, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0); if (map->mmaped == MAP_FAILED) { @@ -4852,6 +4867,7 @@ static int bpf_object__create_map(struct bpf_object *obj, struct bpf_map *map, b case BPF_MAP_TYPE_SOCKHASH: case BPF_MAP_TYPE_QUEUE: case BPF_MAP_TYPE_STACK: + case BPF_MAP_TYPE_ARENA: create_attr.btf_fd = 0; create_attr.btf_key_type_id = 0; create_attr.btf_value_type_id = 0; @@ -4908,6 +4924,21 @@ static int bpf_object__create_map(struct bpf_object *obj, struct bpf_map *map, b if (map->fd == map_fd) return 0; + if (def->type == BPF_MAP_TYPE_ARENA) { + map->mmaped = mmap((void *)map->map_extra, bpf_map_mmap_sz(map), + PROT_READ | PROT_WRITE, + map->map_extra ? MAP_SHARED | MAP_FIXED : MAP_SHARED, + map_fd, 0); + if (map->mmaped == MAP_FAILED) { + err = -errno; + map->mmaped = NULL; + close(map_fd); + pr_warn("map '%s': failed to mmap bpf_arena: %d\n", + bpf_map__name(map), err); + return err; + } + } + /* Keep placeholder FD value but now point it to the BPF map object. * This way everything that relied on this map's FD (e.g., relocated * ldimm64 instructions) will stay valid and won't need adjustments. @@ -8582,7 +8613,7 @@ static void bpf_map__destroy(struct bpf_map *map) if (map->mmaped) { size_t mmap_sz; - mmap_sz = bpf_map_mmap_sz(map->def.value_size, map->def.max_entries); + mmap_sz = bpf_map_mmap_sz(map); munmap(map->mmaped, mmap_sz); map->mmaped = NULL; } @@ -9830,8 +9861,8 @@ int bpf_map__set_value_size(struct bpf_map *map, __u32 size) int err; size_t mmap_old_sz, mmap_new_sz; - mmap_old_sz = bpf_map_mmap_sz(map->def.value_size, map->def.max_entries); - mmap_new_sz = bpf_map_mmap_sz(size, map->def.max_entries); + mmap_old_sz = bpf_map_mmap_sz(map); + mmap_new_sz = __bpf_map_mmap_sz(size, map->def.max_entries); err = bpf_map_mmap_resize(map, mmap_old_sz, mmap_new_sz); if (err) { pr_warn("map '%s': failed to resize memory-mapped region: %d\n", @@ -13356,7 +13387,7 @@ int bpf_object__load_skeleton(struct bpf_object_skeleton *s) for (i = 0; i < s->map_cnt; i++) { struct bpf_map *map = *s->maps[i].map; - size_t mmap_sz = bpf_map_mmap_sz(map->def.value_size, map->def.max_entries); + size_t mmap_sz = bpf_map_mmap_sz(map); int prot, map_fd = map->fd; void **mmaped = s->maps[i].mmaped; diff --git a/tools/lib/bpf/libbpf_probes.c b/tools/lib/bpf/libbpf_probes.c index ee9b1dbea9eb..302188122439 100644 --- a/tools/lib/bpf/libbpf_probes.c +++ b/tools/lib/bpf/libbpf_probes.c @@ -338,6 +338,13 @@ static int probe_map_create(enum bpf_map_type map_type) key_size = 0; max_entries = 1; break; + case BPF_MAP_TYPE_ARENA: + key_size = 0; + value_size = 0; + max_entries = 1; /* one page */ + opts.map_extra = 0; /* can mmap() at any address */ + opts.map_flags = BPF_F_MMAPABLE; + break; case BPF_MAP_TYPE_HASH: case BPF_MAP_TYPE_ARRAY: case BPF_MAP_TYPE_PROG_ARRAY: