From patchwork Wed Aug 14 03:30:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kui-Feng Lee X-Patchwork-Id: 13762811 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7DD58C52D7C for ; Wed, 14 Aug 2024 03:30:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0F2C86B0085; Tue, 13 Aug 2024 23:30:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0A1A66B0088; Tue, 13 Aug 2024 23:30:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E83B46B0089; Tue, 13 Aug 2024 23:30:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id C99106B0085 for ; Tue, 13 Aug 2024 23:30:21 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 763B3C0B9E for ; Wed, 14 Aug 2024 03:30:21 +0000 (UTC) X-FDA: 82449423042.29.FF6DE30 Received: from mail-yw1-f172.google.com (mail-yw1-f172.google.com [209.85.128.172]) by imf26.hostedemail.com (Postfix) with ESMTP id A5DD7140005 for ; Wed, 14 Aug 2024 03:30:19 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=HYhM6AIs; spf=pass (imf26.hostedemail.com: domain of thinker.li@gmail.com designates 209.85.128.172 as permitted sender) smtp.mailfrom=thinker.li@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723606184; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/WlaZ8Qk4gImq5cOnYJqxcFCkNXR6u9I92jjLqb+PSo=; b=B0P6+csL8q/ucWqh+KSZoIdwuut34jB3Yi4ge2WHxVxmhhuk5+Q5hEKgDaOs364xgl820z k37+Ewe0UNs25OaNYajebe+nxc9zVIyukyPe4bBsWo+NKX/Ldj33lzQ3wJspZfHaUltexK PECBuTAxVb3GaSG/ayB7P4D7N45Vljg= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=HYhM6AIs; spf=pass (imf26.hostedemail.com: domain of thinker.li@gmail.com designates 209.85.128.172 as permitted sender) smtp.mailfrom=thinker.li@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723606184; a=rsa-sha256; cv=none; b=uweDTSpr30D4sUw7jUONr/4vyKvDCz/B+eyroQR/DB+BZUGPAsM89bQ9nlkw/namTkdufA nKZ2TlXSUZpOnTyRFOTd0aR0B3FDfIKKUY2Uo8h5uNuIS7icmIr5fOsO4r75/N0Ljhngmm gAVpPWotf20p9lbelC8QZC+OTjd1s8s= Received: by mail-yw1-f172.google.com with SMTP id 00721157ae682-699d8dc6744so4169747b3.0 for ; Tue, 13 Aug 2024 20:30:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1723606219; x=1724211019; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/WlaZ8Qk4gImq5cOnYJqxcFCkNXR6u9I92jjLqb+PSo=; b=HYhM6AIssMcXLmH3F3OriW04toBbjql6KDwEyLuJWQ83r+OpN2/bpJH/cQgwc/eRZF f8cC29KUrM0X0jjZm/vyQSwVCQmm9AXKtFgKE7lTnLkNWwsmjkvbH3n+piIw33HCWd4z 76hVd4wcKA2AvYgPfWRZ35Pps7HqMGiwLLR0dx6zr8S+yDn4ey731kxqFke1WWcg1BY3 IJyzKQnjtEZneNmu7sNRvT1sk2E87ccPfjDgpIyRxqhsPg1OuPIkQKsFPdnI1zzFWj3c 8gAIFioXBcPd5leQXv2zM3h1yrppzOVeIQz7m7ozCGyQIaU5aSZWMHOpMU5j2+sBciOf ky5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723606219; x=1724211019; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/WlaZ8Qk4gImq5cOnYJqxcFCkNXR6u9I92jjLqb+PSo=; b=Eh0T95lFw2QOtSr9vuR+K7o9CvsvylA/t9CcM2HCdSq23Dt4DfDQ1qyvrHar5U6dZj 4DiQV3nov2QZhX9uAd+64uzAu4GGDKbj31IdBzqIA0z9ndYR6TitVfbJ50KYxzSx7/kg sdBaGWOUh3AbtxVV6eNVObb6mYvRygNPS0mPtqhl2B4+IvKbTGeYAN5XjO/bwiWUk/3V 1VfQocPpfPrq9ny5iBnRfAzctUKhCGBLzmKl/A5rvID1PVnGYo2Qa+/ewU0fPRsw5FUw 6au9U3HDhI7/HAl63SWCPkSyesd1S1j0ZlPNR4F51Bij3RhmCoJRKT9/NI2qGPMR6eOT cDkg== X-Forwarded-Encrypted: i=1; AJvYcCXC8KGkb3USvePzHMlVHxM/ymMWnuubDv+LJiZUz11MnHWIqPKMmeQaOBXyfhaJ66Q6HC6DUsOL12hmDwT7L5bbalU= X-Gm-Message-State: AOJu0Yx61coxQINj5uVhYaChb2wfBmh8aqgSGSP40FqKxMYGc4RmKNUL xTNddeEqStpQXzFqtcmklRGTnekVhzBIGBnadlgNPxN0izGJoQva X-Google-Smtp-Source: AGHT+IG5ACL4Q/2MmoPXNV2oWkutSyNCViq3kQcmU5wjEwc7tftN0jDn9wF8InfAfMMzhsDH58rX8w== X-Received: by 2002:a05:690c:dd5:b0:65f:8d14:d37d with SMTP id 00721157ae682-6ad19995235mr5406827b3.1.1723606217949; Tue, 13 Aug 2024 20:30:17 -0700 (PDT) Received: from kickker.attlocal.net ([2600:1700:6cf8:1240:3c23:99cc:16a9:8b68]) by smtp.gmail.com with ESMTPSA id 00721157ae682-6a0a451b597sm15109587b3.117.2024.08.13.20.30.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Aug 2024 20:30:17 -0700 (PDT) From: Kui-Feng Lee To: bpf@vger.kernel.org, ast@kernel.org, martin.lau@linux.dev, song@kernel.org, kernel-team@meta.com, andrii@kernel.org Cc: sinquersw@gmail.com, kuifeng@meta.com, Kui-Feng Lee , linux-mm@kvack.org Subject: [RFC bpf-next v3 4/7] bpf: add helper functions of pinning and converting BPF_UPTR. Date: Tue, 13 Aug 2024 20:30:07 -0700 Message-Id: <20240814033010.2980635-5-thinker.li@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240814033010.2980635-1-thinker.li@gmail.com> References: <20240814033010.2980635-1-thinker.li@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: A5DD7140005 X-Stat-Signature: oui4u3otpcc8kr6s9bjhq4qfhzaqknfk X-HE-Tag: 1723606219-829206 X-HE-Meta: U2FsdGVkX1+DY2O7WvIFXrfuYsPq6pn1EcJH1HlF3b/1GQ3hkbcXAf/4v3pvYg+vMF1O4/S6UF6AliBVVtajFTXPZm18b36UeoWplbs55OHV6f3FjS23Ub52Q9/aIfsq28tmmLtbBRkGnjUNMmzdBQmTy75bgJL6Vzf/7T7PeKBZkLsSsuoOQnVIOn9g87zXJvubbLzfouS5X1Kq9gOu8N9iLIGju2mYFGzZCSo9UQh6s+3Fw1jzY2cjtzd0ccHgo6mvenrJ74wFbVqqF/P7uYDvc4nRRNNb861KopV7NQ/bZN4ssoxgJjWYFEi3UDIUNGgSkD4hs3F9fYr5j039n6dLaI7wYIeyAp24VWgxJBpVc0eGF9uUumIWDI0SwcYOKrio+xjXdXxY/gJtpi/2QC/GFADccB50s7/v9vwEPVDG4gjcR1wUxmOvpS62YZ/Xvu2+6MKkTXzpVAXfxzRuX3RoEkRl803NP07nusv+Hcvp1a1a27DHww4ih4nQPLZ5oHqqk3vAHAyFuZw9LmoS++G2asK2y0h2Gbj+eqlQj4O3d+Qg1MRNuQj1kgVQfO5WGRAeRZfT7TR1HhdGnztQo/uwREs56ibksNDL4sOmtv+d1c15ZAuC5Qv/NNC62iIRRAQGHW2VlX+tbGN5Te8veQ/2vttwYJ5DSfgNHxZHCYIrhlqtqkD3lQGG55LpFM/HK27du4K540UdX8YM8FA+zTTEDZc5hBgJObEDvmBeRnE1/eNSMNaCyJJLeb7Ktyw1zAzGBlNw2i5zFsocJTN9qqCxP5rz8Bp8uSrldEXGw6IS4L/Fgv5dTW0oHE3v2VLGXbDv3qzcc1AY0mZ0DNM18YYhT3C8aBIYl9sdXKshPv+3Ewz7R6mZTPK5VM+EB8a19X8SablkZQxxKYk0YzF6yxIzmkfkr11qvM4Pw3UfPRZfCY+SEUq1r9bRHE2++X6Hvmnw74HzhHSXa9gExNW T3mym3RO GkKCp7hbE5biTL8GGuy4HIihE/9AD22mgBofU9tM/Te+rNRPc2gOUGDbqqdb3RtOeVElCLtEvlWGelVyOt0etKNY4ERSgQ31gAYXEu4ISxrdPDyKwWdmWFOIU9dZFzaw+B68Keds3eRbxMNN2Ph4fNq40IDXg0OvyN8U0T0Fzm8okwKHbQlhUFp3WUypYdFlBEKUhAaKfiJMHsJSGDWukiS5bP+fsoBsgpjgbdDj6o2dni3ouD7NmPLbVnVTLnuI3KrUNxNgBaJboJvY+c6qRvWvFIsN13eDDgDJVwdLNmLIH1Qg7gwaAtEAfWPUuGO1Ci/DkVgrbBIfsjvXLjJrdr6ZZXE85SrPa+7K5CqnASa021Une3du746l2tCP3vwgXvPVH9HU1taP4At5qdV6hnRJdTJuo7H1KyXwwouKHGGIMNDu7AqaZ6I47Iotisl9lQ5OfR1r4tHwBG0tQMQ2AEEfLvxBmXV6zkxvEVgHGI6vz9wqcwgit7N1tH7TyaMeR18PZVp8bxiHTh70WraDS1l9Y2tHXSx/PmyLaXvFZgFBViTsQslskLZUqHAPTiKLsP/dV X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The helper functions manage uptrs in BPF map values from userspace. These uptrs point to user memory, so we must pin and convert them into kernel address space for BPF programs to access them in the kernel. These helper functions will be utilized by the upcoming patches. To access uptrs in BPF programs, they are pinned using pin_user_pages_fast(), but the conversion to kernel addresses is actually done by page_address(). The uptrs can be unpinned using unpin_user_pages(). Currently, the memory block pointed to by a uptr must reside in a single memory page, as crossing multiple pages is not supported. Cc: linux-mm@kvack.org Signed-off-by: Kui-Feng Lee --- include/linux/bpf.h | 30 ++++++++++ kernel/bpf/helpers.c | 20 +++++++ kernel/bpf/syscall.c | 132 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 182 insertions(+) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 954e476b5605..886c818ff555 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -477,6 +477,8 @@ static inline void bpf_long_memcpy(void *dst, const void *src, u32 size) data_race(*ldst++ = *lsrc++); } +void bpf_obj_unpin_uptr(const struct btf_field *field, void *addr); + /* copy everything but bpf_spin_lock, bpf_timer, and kptrs. There could be one of each. */ static inline void bpf_obj_memcpy(struct btf_record *rec, void *dst, void *src, u32 size, @@ -503,6 +505,34 @@ static inline void bpf_obj_memcpy(struct btf_record *rec, memcpy(dst + curr_off, src + curr_off, size - curr_off); } +static inline void bpf_obj_uptrcpy(struct btf_record *rec, + void *dst, void *src) +{ + int i; + + if (IS_ERR_OR_NULL(rec)) + return; + + for (i = 0; i < rec->cnt; i++) { + u32 next_off = rec->fields[i].offset; + void *addr; + + if (rec->fields[i].type == BPF_UPTR) { + /* Unpin old address. + * + * Alignments are guaranteed by btf_find_field_one(). + */ + addr = *(void **)(dst + next_off); + if (addr) + bpf_obj_unpin_uptr(&rec->fields[i], addr); + + *(void **)(dst + next_off) = *(void **)(src + next_off); + } + } +} + +void copy_map_uptr_locked(struct bpf_map *map, void *dst, void *src, bool lock_src); + static inline void copy_map_value(struct bpf_map *map, void *dst, void *src) { bpf_obj_memcpy(map->record, dst, src, map->value_size, false); diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index d02ae323996b..d588b52605b9 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -388,6 +388,26 @@ void copy_map_value_locked(struct bpf_map *map, void *dst, void *src, preempt_enable(); } +/* Copy map value and uptr from src to dst, with lock_src indicating + * whether src or dst is locked. + */ +void copy_map_uptr_locked(struct bpf_map *map, void *src, void *dst, + bool lock_src) +{ + struct bpf_spin_lock *lock; + + if (lock_src) + lock = src + map->record->spin_lock_off; + else + lock = dst + map->record->spin_lock_off; + preempt_disable(); + __bpf_spin_lock_irqsave(lock); + copy_map_value(map, dst, src); + bpf_obj_uptrcpy(map->record, dst, src); + __bpf_spin_unlock_irqrestore(lock); + preempt_enable(); +} + BPF_CALL_0(bpf_jiffies64) { return get_jiffies_64(); diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index fed4a2145f81..d504f5eb955a 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -155,6 +155,138 @@ static void maybe_wait_bpf_programs(struct bpf_map *map) synchronize_rcu(); } +void bpf_obj_unpin_uptr(const struct btf_field *field, void *addr) +{ + struct page *pages[1]; + u32 size, type_id; + int npages; + void *ptr; + + type_id = field->kptr.btf_id; + btf_type_id_size(field->kptr.btf, &type_id, &size); + if (size == 0) + return; + + ptr = (void *)((intptr_t)addr & PAGE_MASK); + + npages = (((intptr_t)addr + size + ~PAGE_MASK) - (intptr_t)ptr) >> PAGE_SHIFT; + if (WARN_ON_ONCE(npages > 1)) + return; + + pages[0] = virt_to_page(ptr); + unpin_user_pages(pages, 1); +} + +/* Unpin uptr fields in the record up to cnt */ +static void bpf_obj_unpin_uptrs_cnt(struct btf_record *rec, int cnt, void *src) +{ + u32 next_off; + void **kaddr_ptr; + int i; + + for (i = 0; i < cnt; i++) { + if (rec->fields[i].type != BPF_UPTR) + continue; + + next_off = rec->fields[i].offset; + kaddr_ptr = src + next_off; + if (*kaddr_ptr) { + bpf_obj_unpin_uptr(&rec->fields[i], *kaddr_ptr); + *kaddr_ptr = NULL; + } + } +} + +/* Find all BPF_UPTR fields in the record, pin the user memory, map it + * to kernel space, and update the addresses in the source memory. + * + * The map value passing from userspace may contain user kptrs pointing to + * user memory. This function pins the user memory and maps it to kernel + * memory so that BPF programs can access it. + */ +static int bpf_obj_trans_pin_uptrs(struct btf_record *rec, void *src, u32 size) +{ + u32 type_id, tsz, npages, next_off; + void *uaddr, *kaddr, **uaddr_ptr; + const struct btf_type *t; + struct page *pages[1]; + int i, err; + + if (IS_ERR_OR_NULL(rec)) + return 0; + + if (!btf_record_has_field(rec, BPF_UPTR)) + return 0; + + for (i = 0; i < rec->cnt; i++) { + if (rec->fields[i].type != BPF_UPTR) + continue; + + next_off = rec->fields[i].offset; + if (next_off + sizeof(void *) > size) { + err = -EFAULT; + goto rollback; + } + uaddr_ptr = src + next_off; + uaddr = *uaddr_ptr; + if (!uaddr) + continue; + + /* Make sure the user memory takes up at most one page */ + type_id = rec->fields[i].kptr.btf_id; + t = btf_type_id_size(rec->fields[i].kptr.btf, &type_id, &tsz); + if (!t) { + err = -EFAULT; + goto rollback; + } + if (tsz == 0) { + *uaddr_ptr = NULL; + continue; + } + npages = (((intptr_t)uaddr + tsz + ~PAGE_MASK) - + ((intptr_t)uaddr & PAGE_MASK)) >> PAGE_SHIFT; + if (npages > 1) { + /* Allow only one page */ + err = -EFAULT; + goto rollback; + } + + /* Pin the user memory */ + err = pin_user_pages_fast((intptr_t)uaddr, 1, FOLL_LONGTERM | FOLL_WRITE, pages); + if (err < 0) + goto rollback; + + /* Map to kernel space */ + kaddr = page_address(pages[0]); + if (unlikely(!kaddr)) { + WARN_ON_ONCE(1); + unpin_user_pages(pages, 1); + err = -EFAULT; + goto rollback; + } + *uaddr_ptr = kaddr + ((intptr_t)uaddr & ~PAGE_MASK); + } + + return 0; + +rollback: + /* Unpin the user memory of earlier fields */ + bpf_obj_unpin_uptrs_cnt(rec, i, src); + + return err; +} + +static void bpf_obj_unpin_uptrs(struct btf_record *rec, void *src) +{ + if (IS_ERR_OR_NULL(rec)) + return; + + if (!btf_record_has_field(rec, BPF_UPTR)) + return; + + bpf_obj_unpin_uptrs_cnt(rec, rec->cnt, src); +} + static int bpf_map_update_value(struct bpf_map *map, struct file *map_file, void *key, void *value, __u64 flags) {