From patchwork Wed Oct 23 23:47:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Martin KaFai Lau X-Patchwork-Id: 13848112 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4FF4ED0BB4A for ; Wed, 23 Oct 2024 23:48:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CF02A6B0093; Wed, 23 Oct 2024 19:48:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CA0086B0098; Wed, 23 Oct 2024 19:48:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B8E926B0099; Wed, 23 Oct 2024 19:48:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 9C43D6B0093 for ; Wed, 23 Oct 2024 19:48:45 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id E0F7BC0CAA for ; Wed, 23 Oct 2024 23:48:25 +0000 (UTC) X-FDA: 82706508822.26.F609DC4 Received: from out-182.mta0.migadu.com (out-182.mta0.migadu.com [91.218.175.182]) by imf28.hostedemail.com (Postfix) with ESMTP id 658A8C0010 for ; Wed, 23 Oct 2024 23:48:25 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=uYMLf+3m; spf=pass (imf28.hostedemail.com: domain of martin.lau@linux.dev designates 91.218.175.182 as permitted sender) smtp.mailfrom=martin.lau@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729727170; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DYleCe0rXNM8/wG/Behf846w4EfWFqIOGOhFhREBDMk=; b=JAXPQWETNldippHwFRJchBUdym7jAqrWwg5DqQ9Taf2UVJDDcBFw2fOPAlB4KunKZ1hpLN otwmaTtCyO4KNRyohSXM0d4atlxQrrLzWW5Qgh1zLUcteKa+t+twTrGzJ0L4wJK6TynJP0 ZSn8AMkMsbJ7ISLc+WVVKwx5/q5TkSk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729727170; a=rsa-sha256; cv=none; b=2OHB/Vgu7hoAPV+srnwDb9Z7379b78N8+OWpVHnPFCPdWDoDCqXcfhUDJMLkopULA31INn ASydhphJf9QGwV/M4tt/vDJ/GH+9zYyBr4XedMYpiRl8VeqOA2wZv+ehBItEZk8mJSz0tn Mkg+U/mVxqX7XcNlXnid3gfazvHUgDo= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=uYMLf+3m; spf=pass (imf28.hostedemail.com: domain of martin.lau@linux.dev designates 91.218.175.182 as permitted sender) smtp.mailfrom=martin.lau@linux.dev; dmarc=pass (policy=none) header.from=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1729727320; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DYleCe0rXNM8/wG/Behf846w4EfWFqIOGOhFhREBDMk=; b=uYMLf+3mJBBTtBOb7cpO5qR6yzpFI9FeRHmt8hOUuBbtQV2nxwNEFusIgB8faJ+nSbxjrs yGSihxAyuhNMXWOPxA1Jecx/P++FmhOOnfP6A2IULAAZsnYVIgHfjSZFewT/UVyUlABAij N7RGkTwcJGcvY9j1iKvBgbpYGdi+9Zc= From: Martin KaFai Lau To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Kui-Feng Lee , kernel-team@meta.com, linux-mm@kvack.org, Shakeel Butt Subject: [PATCH v6 bpf-next 06/12] bpf: Add uptr support in the map_value of the task local storage. Date: Wed, 23 Oct 2024 16:47:53 -0700 Message-ID: <20241023234759.860539-7-martin.lau@linux.dev> In-Reply-To: <20241023234759.860539-1-martin.lau@linux.dev> References: <20241023234759.860539-1-martin.lau@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Stat-Signature: 1qo3169bsf5s7rr9uodgyrapygcmqoja X-Rspamd-Queue-Id: 658A8C0010 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1729727305-739958 X-HE-Meta: U2FsdGVkX1/l6yslko1NKUE4kgF5zAtBd6A/FosDHV2rDE4qhhxTRh2Nj0spT8cY/QChp6NjkMqnbTMTnhJX2E8lpbZReX3SX/Qnivf8VFWE1jvCJQP3UXUhPa9rj0DLD3XmaGIykmikkxhNwHQlacwtdWQtsqMxDfYg7a1O7rUScKi+ZzTQQ6MZdx8P9KDkvfnZHxMJPTVO+7tdOIsEKMzNJF6yK2YsfoPyXjKRSlRVlxj8rttv7PWlG4ZQCagNxTbLAncdpB0r1D55jOGKW3kv7FWS43wHfFEMz4eDHlrafPi81AlkcDnPvcLC++TzDO15qvehpnmvk2BDCkNhDVAWCMMLbI3IQF0P5aPm4RMq6ebsIZsxQpjkt920zRcqz/EOmZ4e1g+2ZI2n9W+GglRC/ai7ZeCiXFVECuz5NtRMK5QCg2R0wlQ0BmKCTM5n5cLEyUYOY6NJsL1AJHSiy4mgEelWgEtJUGXo5v9gcigl6zlbZG/Tr4AOjXceTuNs/2KUsApgEKUoOW7w7LLkXp3FhgDiiKX67KlvnHyDbWtKYB4CVU/QZFHoVVp2ZNJyDVOsiozYPsVqmTfg1t65/6380MLXkjIzQt82D6J7SB52iKs/Iv399RqkoIS0gvP5x9lXHjqYYOLMvC8htthlXEdmIZrR2z2qchsm2KP2Qss2mMfGuDVy1meNfveuBCJb9K5uMpn0dnCWpGxTIodWWHxgnerm24PKZCa/fFAdup3ohGK/CoiqhMtZeN/iURVrwTUjg6drG5h/vBTq1L30r3RsBNhsNgxvCRdpnx4qHsFEPniucVdL0CMmjg84wQiKTNSaDSfx3VpHMEku/BKqWZzRyPhIFMjfVSfSDmn8odNy/Z9Bc61FyeZS5XclOkXiPc4a25Ud9pmrOOsx+Zee26QE7u7v09fhWyjaGbJ10iEG+F+7R/Th3lgQ8UFdc0+j6aX+DJyUtXRpV/7yi2P xrYrakuv ZYXUW9VeVYSj3tbtnP1eotvXwNWQCKzJjqHfC2LwxozhNyTWixOlvfodZTmsrRId2dlmvuBRvGRJW35Ow08I6DY4ysDtCHExvoYEcYZAGdrO7gouU4CvoRU5YJ+q/s/AkKU94YKbbVBAVw0eu/S74nVbHO7nfOoU+rr601ARzRgdDOmRzWV1FBqZQFGlOUc04rBAg7YNIp8K8ixvDBP7LlQVq2Ad20mB9vaoqFgm/96uPUe35IA8p4O9Pm7bYgGarMxmeRyfuJvHex8Itd6RKjysEtWVeLkj/38EcPx7BrQ+pELRK7BMUOQmPD25BvvAAgbuySYD0NHSiXbzc4xkIB/yxwg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Martin KaFai Lau This patch adds uptr support in the map_value of the task local storage. struct map_value { struct user_data __uptr *uptr; }; struct { __uint(type, BPF_MAP_TYPE_TASK_STORAGE); __uint(map_flags, BPF_F_NO_PREALLOC); __type(key, int); __type(value, struct value_type); } datamap SEC(".maps"); A new bpf_obj_pin_uptrs() is added to pin the user page and also stores the kernel address back to the uptr for the bpf prog to use later. It currently does not support the uptr pointing to a user struct across two pages. It also excludes PageHighMem support to keep it simple. As of now, the 32bit bpf jit is missing other more crucial bpf features. For example, many important bpf features depend on bpf kfunc now but so far only one arch (x86-32) supports it which was added by me as an example when kfunc was first introduced to bpf. The uptr can only be stored to the task local storage by the syscall update_elem. Meaning the uptr will not be considered if it is provided by the bpf prog through bpf_task_storage_get(BPF_LOCAL_STORAGE_GET_F_CREATE). This is enforced by only calling bpf_local_storage_update(swap_uptrs==true) in bpf_pid_task_storage_update_elem. Everywhere else will have swap_uptrs==false. This will pump down to bpf_selem_alloc(swap_uptrs==true). It is the only case that bpf_selem_alloc() will take the uptr value when updating the newly allocated selem. bpf_obj_swap_uptrs() is added to swap the uptr between the SDATA(selem)->data and the user provided map_value in "void *value". bpf_obj_swap_uptrs() makes the SDATA(selem)->data takes the ownership of the uptr and the user space provided map_value will have NULL in the uptr. The bpf_obj_unpin_uptrs() is called after map->ops->map_update_elem() returning error. If the map->ops->map_update_elem has reached a state that the local storage has taken the uptr ownership, the bpf_obj_unpin_uptrs() will be a no op because the uptr is NULL. A "__"bpf_obj_unpin_uptrs is added to make this error path unpin easier such that it does not have to check the map->record is NULL or not. BPF_F_LOCK is not supported when the map_value has uptr. This can be revisited later if there is a use case. A similar swap_uptrs idea can be considered. The final bit is to do unpin_user_page in the bpf_obj_free_fields(). The earlier patch has ensured that the bpf_obj_free_fields() has gone through the rcu gp when needed. Cc: linux-mm@kvack.org Cc: Shakeel Butt Signed-off-by: Martin KaFai Lau Acked-by: Shakeel Butt --- include/linux/bpf.h | 20 +++++++ kernel/bpf/bpf_local_storage.c | 7 ++- kernel/bpf/bpf_task_storage.c | 5 +- kernel/bpf/syscall.c | 106 +++++++++++++++++++++++++++++++-- 4 files changed, 131 insertions(+), 7 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index bb31bc6d0c4d..8888689aa917 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -424,6 +424,7 @@ static inline void bpf_obj_init_field(const struct btf_field *field, void *addr) case BPF_KPTR_UNREF: case BPF_KPTR_REF: case BPF_KPTR_PERCPU: + case BPF_UPTR: break; default: WARN_ON_ONCE(1); @@ -512,6 +513,25 @@ static inline void copy_map_value_long(struct bpf_map *map, void *dst, void *src bpf_obj_memcpy(map->record, dst, src, map->value_size, true); } +static inline void bpf_obj_swap_uptrs(const struct btf_record *rec, void *dst, void *src) +{ + unsigned long *src_uptr, *dst_uptr; + const struct btf_field *field; + int i; + + if (!btf_record_has_field(rec, BPF_UPTR)) + return; + + for (i = 0, field = rec->fields; i < rec->cnt; i++, field++) { + if (field->type != BPF_UPTR) + continue; + + src_uptr = src + field->offset; + dst_uptr = dst + field->offset; + swap(*src_uptr, *dst_uptr); + } +} + static inline void bpf_obj_memzero(struct btf_record *rec, void *dst, u32 size) { u32 curr_off = 0; diff --git a/kernel/bpf/bpf_local_storage.c b/kernel/bpf/bpf_local_storage.c index ca871be1c42d..7e6a0af0afc1 100644 --- a/kernel/bpf/bpf_local_storage.c +++ b/kernel/bpf/bpf_local_storage.c @@ -99,9 +99,12 @@ bpf_selem_alloc(struct bpf_local_storage_map *smap, void *owner, } if (selem) { - if (value) + if (value) { + /* No need to call check_and_init_map_value as memory is zero init */ copy_map_value(&smap->map, SDATA(selem)->data, value); - /* No need to call check_and_init_map_value as memory is zero init */ + if (swap_uptrs) + bpf_obj_swap_uptrs(smap->map.record, SDATA(selem)->data, value); + } return selem; } diff --git a/kernel/bpf/bpf_task_storage.c b/kernel/bpf/bpf_task_storage.c index 45dc3ca334d3..09705f9988f3 100644 --- a/kernel/bpf/bpf_task_storage.c +++ b/kernel/bpf/bpf_task_storage.c @@ -129,6 +129,9 @@ static long bpf_pid_task_storage_update_elem(struct bpf_map *map, void *key, struct pid *pid; int fd, err; + if ((map_flags & BPF_F_LOCK) && btf_record_has_field(map->record, BPF_UPTR)) + return -EOPNOTSUPP; + fd = *(int *)key; pid = pidfd_get_pid(fd, &f_flags); if (IS_ERR(pid)) @@ -147,7 +150,7 @@ static long bpf_pid_task_storage_update_elem(struct bpf_map *map, void *key, bpf_task_storage_lock(); sdata = bpf_local_storage_update( task, (struct bpf_local_storage_map *)map, value, map_flags, - false, GFP_ATOMIC); + true, GFP_ATOMIC); bpf_task_storage_unlock(); err = PTR_ERR_OR_ZERO(sdata); diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 2d2935d9c096..426a52e5c7da 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -155,6 +155,89 @@ static void maybe_wait_bpf_programs(struct bpf_map *map) synchronize_rcu(); } +static void unpin_uptr_kaddr(void *kaddr) +{ + if (kaddr) + unpin_user_page(virt_to_page(kaddr)); +} + +static void __bpf_obj_unpin_uptrs(struct btf_record *rec, u32 cnt, void *obj) +{ + const struct btf_field *field; + void **uptr_addr; + int i; + + for (i = 0, field = rec->fields; i < cnt; i++, field++) { + if (field->type != BPF_UPTR) + continue; + + uptr_addr = obj + field->offset; + unpin_uptr_kaddr(*uptr_addr); + } +} + +static void bpf_obj_unpin_uptrs(struct btf_record *rec, void *obj) +{ + if (!btf_record_has_field(rec, BPF_UPTR)) + return; + + __bpf_obj_unpin_uptrs(rec, rec->cnt, obj); +} + +static int bpf_obj_pin_uptrs(struct btf_record *rec, void *obj) +{ + const struct btf_field *field; + const struct btf_type *t; + unsigned long start, end; + struct page *page; + void **uptr_addr; + int i, err; + + if (!btf_record_has_field(rec, BPF_UPTR)) + return 0; + + for (i = 0, field = rec->fields; i < rec->cnt; i++, field++) { + if (field->type != BPF_UPTR) + continue; + + uptr_addr = obj + field->offset; + start = *(unsigned long *)uptr_addr; + if (!start) + continue; + + t = btf_type_by_id(field->kptr.btf, field->kptr.btf_id); + /* t->size was checked for zero before */ + if (check_add_overflow(start, t->size - 1, &end)) { + err = -EFAULT; + goto unpin_all; + } + + /* The uptr's struct cannot span across two pages */ + if ((start & PAGE_MASK) != (end & PAGE_MASK)) { + err = -EOPNOTSUPP; + goto unpin_all; + } + + err = pin_user_pages_fast(start, 1, FOLL_LONGTERM | FOLL_WRITE, &page); + if (err != 1) + goto unpin_all; + + if (PageHighMem(page)) { + err = -EOPNOTSUPP; + unpin_user_page(page); + goto unpin_all; + } + + *uptr_addr = page_address(page) + offset_in_page(start); + } + + return 0; + +unpin_all: + __bpf_obj_unpin_uptrs(rec, i, obj); + return err; +} + static int bpf_map_update_value(struct bpf_map *map, struct file *map_file, void *key, void *value, __u64 flags) { @@ -199,9 +282,14 @@ static int bpf_map_update_value(struct bpf_map *map, struct file *map_file, map->map_type == BPF_MAP_TYPE_BLOOM_FILTER) { err = map->ops->map_push_elem(map, value, flags); } else { - rcu_read_lock(); - err = map->ops->map_update_elem(map, key, value, flags); - rcu_read_unlock(); + err = bpf_obj_pin_uptrs(map->record, value); + if (!err) { + rcu_read_lock(); + err = map->ops->map_update_elem(map, key, value, flags); + rcu_read_unlock(); + if (err) + bpf_obj_unpin_uptrs(map->record, value); + } } bpf_enable_instrumentation(); @@ -716,6 +804,10 @@ void bpf_obj_free_fields(const struct btf_record *rec, void *obj) field->kptr.dtor(xchgd_field); } break; + case BPF_UPTR: + /* The caller ensured that no one is using the uptr */ + unpin_uptr_kaddr(*(void **)field_ptr); + break; case BPF_LIST_HEAD: if (WARN_ON_ONCE(rec->spin_lock_off < 0)) continue; @@ -1107,7 +1199,7 @@ static int map_check_btf(struct bpf_map *map, struct bpf_token *token, map->record = btf_parse_fields(btf, value_type, BPF_SPIN_LOCK | BPF_TIMER | BPF_KPTR | BPF_LIST_HEAD | - BPF_RB_ROOT | BPF_REFCOUNT | BPF_WORKQUEUE, + BPF_RB_ROOT | BPF_REFCOUNT | BPF_WORKQUEUE | BPF_UPTR, map->value_size); if (!IS_ERR_OR_NULL(map->record)) { int i; @@ -1163,6 +1255,12 @@ static int map_check_btf(struct bpf_map *map, struct bpf_token *token, goto free_map_tab; } break; + case BPF_UPTR: + if (map->map_type != BPF_MAP_TYPE_TASK_STORAGE) { + ret = -EOPNOTSUPP; + goto free_map_tab; + } + break; case BPF_LIST_HEAD: case BPF_RB_ROOT: if (map->map_type != BPF_MAP_TYPE_HASH &&