From patchwork Sun Feb 19 15:52:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13145960 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89DA5C61DA4 for ; Sun, 19 Feb 2023 15:52:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230099AbjBSPw5 (ORCPT ); Sun, 19 Feb 2023 10:52:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43904 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230148AbjBSPw4 (ORCPT ); Sun, 19 Feb 2023 10:52:56 -0500 Received: from mail-ed1-x543.google.com (mail-ed1-x543.google.com [IPv6:2a00:1450:4864:20::543]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8E29A46A3 for ; Sun, 19 Feb 2023 07:52:54 -0800 (PST) Received: by mail-ed1-x543.google.com with SMTP id fd2so2644320edb.8 for ; Sun, 19 Feb 2023 07:52:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+dM2SLqS21zDKJx6x4MhUEm4dpwG3TWg25MgSQwFL7Y=; b=p/Cx8BQ/bNWHyOq19eMxASOaPwam3r8iHCShtYDfmIvQdNnls1XOBpamCAZVTLJy+n ht5a6AZ3Gp3H5TIrVZtSCE3yhgWDJ76YOh0ECyGzYoHJMO9H+Nfs9GYgLvws/g+U3jm5 4WrdhfvUfq7zcsruokgnE9oQnsMENQ98qeny0wqHSOECXOwEygpHP2zadXQuqrxR4Y0w a2BkYoN1pFbx1WxdwzFDlN0u31X7tvHHzeaT7rLptwn9RL5LoIP7NxhjNHBiqtUZuhcq civRwfCXN8EzWJk/WbfehcZ8Ag0pEnJYSqx8IQoT085ax67wA5PPtvKYUAu3Ddkbovfn SS9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+dM2SLqS21zDKJx6x4MhUEm4dpwG3TWg25MgSQwFL7Y=; b=EvpyaPeQTwTLmncNDImVeUNQ/rAAI5b5RXunypEqr7f9M9Ulxdu7LBqck3zNLuJFqm D7FzEyyFXJoeaDg0O2A/JOfsactuWQuOeLZEir5VjkJ88tjFUMqjUPsZ1wWvgOz/7oug cQNvkX3hzzEjStL2n4XJCXM8E2S41g9YEpdOWRmwVLo7md6ShU03W/YPHHeYR+LJwTq0 zjTfVu2EzplpnBUP2whXzVTLVzEB0w9LaQsY9jiRcaGt124neb4L6quFRpqFaqIB4HAV XnLtjJw5hy/WBZ9Y3ntG3Y/brBDgwSlR2lxiYKC3hZ6nT6nuUzpYsuvHH5GXs/pswPtL UkKQ== X-Gm-Message-State: AO0yUKVf0UWy97OU6oEdVnvZRfyVE14GGZDGgmI21BgBzIvm1vDf05RT ef+23U73Lm6r6NrkXRCPgVKWTZlV9TsNmw== X-Google-Smtp-Source: AK7set9ndrfBZfMG7t0SfgNxMRqLIjG5utl+uj0TjZ2HK+lC+MX5PuL6urC+P+PGgqKxVnuPwThLvw== X-Received: by 2002:a17:906:4f84:b0:8b1:32b0:2a25 with SMTP id o4-20020a1709064f8400b008b132b02a25mr7209943eju.10.1676821972226; Sun, 19 Feb 2023 07:52:52 -0800 (PST) Received: from localhost ([2001:620:618:580:2:80b3:0:8d0]) by smtp.gmail.com with ESMTPSA id jy28-20020a170907763c00b008b17b123a47sm3702778ejc.208.2023.02.19.07.52.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Feb 2023 07:52:51 -0800 (PST) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Martin KaFai Lau , KP Singh , Dave Marchevsky , David Vernet Subject: [PATCH bpf-next v1 1/7] bpf: Support kptrs in percpu hashmap and percpu LRU hashmap Date: Sun, 19 Feb 2023 16:52:43 +0100 Message-Id: <20230219155249.1755998-2-memxor@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230219155249.1755998-1-memxor@gmail.com> References: <20230219155249.1755998-1-memxor@gmail.com> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=6181; i=memxor@gmail.com; h=from:subject; bh=Y9p91t4feL/dpDYVE2Cl/S2CG69w6xu2l7fuqwPvi+k=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBj8kUeGndslOT9Tib4peABYZTK9T21wOGKJ4ZLkWdi FU0l8O6JAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCY/JFHgAKCRBM4MiGSL8RyhbNEA CaMQBzosGGX+I2vdqvxt8SW9ldwRwtiPeDVRX53iNeWVZGtj8RmTg4A6Bombpbq/MOMdItIEImpQkJ lVz1H4DAYosLZLRwLPWubn4FnRQqcBA+GYQGFqysuaExptNuhkOWhb4qWD01gHOXJ4XcnN3T33BF6p 79PfpQCSXmEywRe7jqtHNPGI60CqzIVNaIEzIjwHV692VT1jTYIGRY3t8ofRKa8dvck+KrcQyPqJi0 ajusM56iZgQ1eAdDeIJIUwpxj1npa41aApd6X7t85H3D4cNFo0U/mmztJ6MzSEBWrscn0nkSBWzpOm 0hiYTEyH7Gs/gyjcVJbkluKHepy+gffWHdlM6FQceHszg1C2KNPMKuyR4mTxclg4DscOcMkTKTGTbm Tj3raoTMIFZwiMyyNSrigkNSKhS98vxIJb583TVPX7VK+CGWZGb1oeSCgkPGUtWLvd0vCUBKWZIQa3 hCfkVzsBBPKUHaQ64m2WNQrL5v4eNy5gRWtiPMOWHXJkyH4UXLBtDjTZoEL2SPt0IryC1sQxRhHrBy UoRF4r5K0ZYwcFVvhtLMTeBkGIqZ0ZWCgKyf/3jEKZLmfQogfVx4t6pwWdOiR6vY0YZJivFP09RhZJ kvEe1aAIXU9ghycNTNJCvjbsEFM+8z1hgFIyAlqkOJifgFmttUKEZmulqC3Q== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Enable support for kptrs in percpu BPF hashmap and percpu BPF LRU hashmap by wiring up the freeing of these kptrs from percpu map elements. Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/hashtab.c | 59 +++++++++++++++++++++++++++----------------- kernel/bpf/syscall.c | 2 ++ 2 files changed, 39 insertions(+), 22 deletions(-) diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index 5dfcb5ad0d06..653aeb481c79 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -249,7 +249,18 @@ static void htab_free_prealloced_fields(struct bpf_htab *htab) struct htab_elem *elem; elem = get_htab_elem(htab, i); - bpf_obj_free_fields(htab->map.record, elem->key + round_up(htab->map.key_size, 8)); + if (htab_is_percpu(htab)) { + void __percpu *pptr = htab_elem_get_ptr(elem, htab->map.key_size); + int cpu; + + for_each_possible_cpu(cpu) { + bpf_obj_free_fields(htab->map.record, per_cpu_ptr(pptr, cpu)); + cond_resched(); + } + } else { + bpf_obj_free_fields(htab->map.record, elem->key + round_up(htab->map.key_size, 8)); + cond_resched(); + } cond_resched(); } } @@ -759,9 +770,17 @@ static int htab_lru_map_gen_lookup(struct bpf_map *map, static void check_and_free_fields(struct bpf_htab *htab, struct htab_elem *elem) { - void *map_value = elem->key + round_up(htab->map.key_size, 8); + if (htab_is_percpu(htab)) { + void __percpu *pptr = htab_elem_get_ptr(elem, htab->map.key_size); + int cpu; - bpf_obj_free_fields(htab->map.record, map_value); + for_each_possible_cpu(cpu) + bpf_obj_free_fields(htab->map.record, per_cpu_ptr(pptr, cpu)); + } else { + void *map_value = elem->key + round_up(htab->map.key_size, 8); + + bpf_obj_free_fields(htab->map.record, map_value); + } } /* It is called from the bpf_lru_list when the LRU needs to delete @@ -858,9 +877,9 @@ static int htab_map_get_next_key(struct bpf_map *map, void *key, void *next_key) static void htab_elem_free(struct bpf_htab *htab, struct htab_elem *l) { + check_and_free_fields(htab, l); if (htab->map.map_type == BPF_MAP_TYPE_PERCPU_HASH) bpf_mem_cache_free(&htab->pcpu_ma, l->ptr_to_pptr); - check_and_free_fields(htab, l); bpf_mem_cache_free(&htab->ma, l); } @@ -918,14 +937,13 @@ static void pcpu_copy_value(struct bpf_htab *htab, void __percpu *pptr, { if (!onallcpus) { /* copy true value_size bytes */ - memcpy(this_cpu_ptr(pptr), value, htab->map.value_size); + copy_map_value(&htab->map, this_cpu_ptr(pptr), value); } else { u32 size = round_up(htab->map.value_size, 8); int off = 0, cpu; for_each_possible_cpu(cpu) { - bpf_long_memcpy(per_cpu_ptr(pptr, cpu), - value + off, size); + copy_map_value_long(&htab->map, per_cpu_ptr(pptr, cpu), value + off); off += size; } } @@ -940,16 +958,14 @@ static void pcpu_init_value(struct bpf_htab *htab, void __percpu *pptr, * (onallcpus=false always when coming from bpf prog). */ if (!onallcpus) { - u32 size = round_up(htab->map.value_size, 8); int current_cpu = raw_smp_processor_id(); int cpu; for_each_possible_cpu(cpu) { if (cpu == current_cpu) - bpf_long_memcpy(per_cpu_ptr(pptr, cpu), value, - size); - else - memset(per_cpu_ptr(pptr, cpu), 0, size); + copy_map_value_long(&htab->map, per_cpu_ptr(pptr, cpu), value); + else /* Since elem is preallocated, we cannot touch special fields */ + zero_map_value(&htab->map, per_cpu_ptr(pptr, cpu)); } } else { pcpu_copy_value(htab, pptr, value, onallcpus); @@ -1575,9 +1591,8 @@ static int __htab_map_lookup_and_delete_elem(struct bpf_map *map, void *key, pptr = htab_elem_get_ptr(l, key_size); for_each_possible_cpu(cpu) { - bpf_long_memcpy(value + off, - per_cpu_ptr(pptr, cpu), - roundup_value_size); + copy_map_value_long(&htab->map, value + off, per_cpu_ptr(pptr, cpu)); + check_and_init_map_value(&htab->map, value + off); off += roundup_value_size; } } else { @@ -1772,8 +1787,8 @@ __htab_map_lookup_and_delete_batch(struct bpf_map *map, pptr = htab_elem_get_ptr(l, map->key_size); for_each_possible_cpu(cpu) { - bpf_long_memcpy(dst_val + off, - per_cpu_ptr(pptr, cpu), size); + copy_map_value_long(&htab->map, dst_val + off, per_cpu_ptr(pptr, cpu)); + check_and_init_map_value(&htab->map, dst_val + off); off += size; } } else { @@ -2046,9 +2061,9 @@ static int __bpf_hash_map_seq_show(struct seq_file *seq, struct htab_elem *elem) roundup_value_size = round_up(map->value_size, 8); pptr = htab_elem_get_ptr(elem, map->key_size); for_each_possible_cpu(cpu) { - bpf_long_memcpy(info->percpu_value_buf + off, - per_cpu_ptr(pptr, cpu), - roundup_value_size); + copy_map_value_long(map, info->percpu_value_buf + off, + per_cpu_ptr(pptr, cpu)); + check_and_init_map_value(map, info->percpu_value_buf + off); off += roundup_value_size; } ctx.value = info->percpu_value_buf; @@ -2292,8 +2307,8 @@ int bpf_percpu_hash_copy(struct bpf_map *map, void *key, void *value) */ pptr = htab_elem_get_ptr(l, map->key_size); for_each_possible_cpu(cpu) { - bpf_long_memcpy(value + off, - per_cpu_ptr(pptr, cpu), size); + copy_map_value_long(map, value + off, per_cpu_ptr(pptr, cpu)); + check_and_init_map_value(map, value + off); off += size; } ret = 0; diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index e3fcdc9836a6..da117a2a83b2 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -1059,7 +1059,9 @@ static int map_check_btf(struct bpf_map *map, const struct btf *btf, case BPF_KPTR_UNREF: case BPF_KPTR_REF: if (map->map_type != BPF_MAP_TYPE_HASH && + map->map_type != BPF_MAP_TYPE_PERCPU_HASH && map->map_type != BPF_MAP_TYPE_LRU_HASH && + map->map_type != BPF_MAP_TYPE_LRU_PERCPU_HASH && map->map_type != BPF_MAP_TYPE_ARRAY && map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) { ret = -EOPNOTSUPP; From patchwork Sun Feb 19 15:52:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13145961 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2EA1C64EC4 for ; Sun, 19 Feb 2023 15:52:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230148AbjBSPw6 (ORCPT ); Sun, 19 Feb 2023 10:52:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43916 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230146AbjBSPw5 (ORCPT ); Sun, 19 Feb 2023 10:52:57 -0500 Received: from mail-ed1-x542.google.com (mail-ed1-x542.google.com [IPv6:2a00:1450:4864:20::542]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C59D8658B for ; Sun, 19 Feb 2023 07:52:55 -0800 (PST) Received: by mail-ed1-x542.google.com with SMTP id ek25so2801645edb.7 for ; Sun, 19 Feb 2023 07:52:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PUNvfkVE+LmadH1Vh2pOdRNMkls/oHd969nUTmNAQ1g=; b=omIkfPU90OuLYTUlPza9jnVGuAp5dqfqgnR8v3LoUfilQzF0CkR9wUfz8Q2H2COVwv 0tZ1gHvMi06RRj5HuHSWb4PkrPsaF2T13q6CcYk5/rZQ68SAsr+4qexYYegJUhDZvROe LOTyvGgiD2Z8bj3+jBZsWqNtNrXtWfS6RY/zjqlcjaJeaNCFXmUdLf75HzPshcvxUYXO EruQFxCWnINxu7rdRkwNrQ6OzTl1Ox7/iEwBC3b/oTRUoYPnDxpYVvB/tVZLlosASItX SKuFYRiNmMRMXkXRY0lMQljavW8OPGO4iw+0IY9YskMPc5ykUZnrbypAQdvqMf/lY67v NSOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PUNvfkVE+LmadH1Vh2pOdRNMkls/oHd969nUTmNAQ1g=; b=wIHJg0kaaGnQcgiRwhV0lrCp14lYkV1HDK5gl1Yq/E4Th+fQilDgLaQ75eFf8pmB1/ AFlwk0Yy4lUMdgzEXGKpqUsRtyHAOf1zxIJgOuOpLoqEbGJxQkJI58Epi2syRaPsV8oo 233/OqwSuyxz/8sGA8Y9afA/2YjObWNlKPjxgaF5A9bfCj0niFgcACS/jx0HzMsSoxDi 8MyNHXESEDW5z6k2V9GatvUBEph11iWGUeQJ8zLrDBDRSEQzlVVEW7V+J6vS0FJTQ/4R 9xu45BkDOvHRId2uu9P8gYpkE6uQdam4NX91hz30Ezy8k+8dtcLGbLyDUrUVBrfefAiz o4bg== X-Gm-Message-State: AO0yUKXZtbtOg74+7Ibw33Keeb07GJJfrWYcMKMHyfHFOXpBZEcwbiIe 6xjTydOfo2LDLKufg7ZM46iq5b0SCpaEqQ== X-Google-Smtp-Source: AK7set8arj5whtYu2ObjEIxZJ2KW76Y2MqmYdLK9WUOv7xZuC7zteFPTbBgfaNYSklEV+W/DMkq+rQ== X-Received: by 2002:a05:6402:7c2:b0:4ae:eae1:21f7 with SMTP id u2-20020a05640207c200b004aeeae121f7mr2650670edy.37.1676821973774; Sun, 19 Feb 2023 07:52:53 -0800 (PST) Received: from localhost ([2001:620:618:580:2:80b3:0:8d0]) by smtp.gmail.com with ESMTPSA id m11-20020a50930b000000b004a27046b7a7sm655605eda.73.2023.02.19.07.52.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Feb 2023 07:52:53 -0800 (PST) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Martin KaFai Lau , KP Singh , "Paul E . McKenney" , Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Dave Marchevsky , David Vernet Subject: [PATCH bpf-next v1 2/7] bpf: Support kptrs in local storage maps Date: Sun, 19 Feb 2023 16:52:44 +0100 Message-Id: <20230219155249.1755998-3-memxor@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230219155249.1755998-1-memxor@gmail.com> References: <20230219155249.1755998-1-memxor@gmail.com> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5277; i=memxor@gmail.com; h=from:subject; bh=IZXotWh+grGo1i9hxQm1S3jtz8ac/rLZBYeMZigpPWw=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBj8kUej1ABQo1uQPzbzgNBzwI5sojqXW7Ud/sm9ahT AaaZ+HyJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCY/JFHgAKCRBM4MiGSL8Ryh+AD/ 9I4J0xLbi+t3dXHuMcDvGy6kyCfXmpKGoNl5xj2tIQ2D3yuAsyC0CGm5iflESj+d0WZqL1T4SY2TrD epvRbo1+2UxKpzJasx3QmhURVhbkFfHBTZeu2b1njN0ICMPjw6a30DQX9Du4wU5igaXlsQDCOTohS9 M93SHvv1Ii/1SXty8jnJH/CzNk38Etzhvj15Hy/izwt0OiAd6hbi7bPl1PTnq5KlbL0SsxGaMyEmLX jBx3r5RER0KfrWmNK+785wwhClYZa9oEGkoM7aTXl7koqvD2Y1XJeviJ8g0lNX+MQZFmI7Ne80/Jn6 sr6bj/MO6pFw7f9Ke8QkA8eIgKsxbjKfU4FgIX+X1UqJm7D1XalaaMXuDw412Fc1XkpLvzituK+h8l UyTExbz1GK8/p7kyRNHD6a9/4MgFsoAZBC/1PQU+hCEbskNLfOH5lZXxpVUpsBBqis233hXbRtYE05 cZYpSfGA3AsO9G+WN9ugdEd5QGQpNNtcdDT+1UoICBXdxl+9yR+GeYz43Oe0nzP2590d2lhMMwzRaM rZFVSHOhrXYA3M7c0p9jw3zuE+BpqakF5xu16n4hsq4n8lOeH9Yw8VSPJRUbUDR32waLZ+sxXCVY+s LypEpnDBXFEttk/IJSkJcbFaEJ17kuH3hTNyVmEfaER9//9eWWIDohUdxk/w== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Enable support for kptrs in local storage maps by wiring up the freeing of these kptrs from map value. Cc: Martin KaFai Lau Cc: KP Singh Cc: Paul E. McKenney Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/bpf_local_storage.c | 35 ++++++++++++++++++++++++++++++---- kernel/bpf/syscall.c | 6 +++++- kernel/bpf/verifier.c | 12 ++++++++---- 3 files changed, 44 insertions(+), 9 deletions(-) diff --git a/kernel/bpf/bpf_local_storage.c b/kernel/bpf/bpf_local_storage.c index 35f4138a54dc..4521d53d7d4d 100644 --- a/kernel/bpf/bpf_local_storage.c +++ b/kernel/bpf/bpf_local_storage.c @@ -75,6 +75,7 @@ bpf_selem_alloc(struct bpf_local_storage_map *smap, void *owner, if (selem) { if (value) copy_map_value(&smap->map, SDATA(selem)->data, value); + /* No need to call check_and_init_map_value as memory is zero init */ return selem; } @@ -103,10 +104,17 @@ static void bpf_selem_free_rcu(struct rcu_head *rcu) struct bpf_local_storage_elem *selem; selem = container_of(rcu, struct bpf_local_storage_elem, rcu); + bpf_obj_free_fields(SDATA(selem)->smap->map.record, SDATA(selem)); + kfree(selem); +} + +static void bpf_selem_free_tasks_trace_rcu(struct rcu_head *rcu) +{ + /* Free directly if Tasks Trace RCU GP also implies RCU GP */ if (rcu_trace_implies_rcu_gp()) - kfree(selem); + bpf_selem_free_rcu(rcu); else - kfree_rcu(selem, rcu); + call_rcu(rcu, bpf_selem_free_rcu); } /* local_storage->lock must be held and selem->local_storage == local_storage. @@ -160,9 +168,9 @@ static bool bpf_selem_unlink_storage_nolock(struct bpf_local_storage *local_stor RCU_INIT_POINTER(local_storage->cache[smap->cache_idx], NULL); if (use_trace_rcu) - call_rcu_tasks_trace(&selem->rcu, bpf_selem_free_rcu); + call_rcu_tasks_trace(&selem->rcu, bpf_selem_free_tasks_trace_rcu); else - kfree_rcu(selem, rcu); + call_rcu(&selem->rcu, bpf_selem_free_rcu); return free_local_storage; } @@ -713,6 +721,25 @@ void bpf_local_storage_map_free(struct bpf_map *map, */ synchronize_rcu(); + /* Only delay freeing of smap, buckets are not needed anymore */ kvfree(smap->buckets); + + /* When local storage has special fields, callbacks for + * bpf_selem_free_rcu and bpf_selem_free_tasks_trace_rcu will keep using + * the map BTF record, we need to execute an RCU barrier to wait for + * them as the record will be freed right after our map_free callback. + */ + if (!IS_ERR_OR_NULL(smap->map.record)) { + rcu_barrier_tasks_trace(); + /* We cannot skip rcu_barrier() when rcu_trace_implies_rcu_gp() + * is true, because while call_rcu invocation is skipped in that + * case in bpf_selem_free_tasks_trace_rcu (and all local storage + * maps pass use_trace_rcu = true), there can be call_rcu + * callbacks based on use_trace_rcu = false in the earlier while + * ((selem = ...)) loop or from bpf_local_storage_unlink_nolock + * called from owner's free path. + */ + rcu_barrier(); + } bpf_map_area_free(smap); } diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index da117a2a83b2..eb50025b03c1 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -1063,7 +1063,11 @@ static int map_check_btf(struct bpf_map *map, const struct btf *btf, map->map_type != BPF_MAP_TYPE_LRU_HASH && map->map_type != BPF_MAP_TYPE_LRU_PERCPU_HASH && map->map_type != BPF_MAP_TYPE_ARRAY && - map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) { + map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY && + map->map_type != BPF_MAP_TYPE_SK_STORAGE && + map->map_type != BPF_MAP_TYPE_INODE_STORAGE && + map->map_type != BPF_MAP_TYPE_TASK_STORAGE && + map->map_type != BPF_MAP_TYPE_CGRP_STORAGE) { ret = -EOPNOTSUPP; goto free_map_tab; } diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 272563a0b770..9a4e7efaf28f 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -7126,22 +7126,26 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, break; case BPF_MAP_TYPE_SK_STORAGE: if (func_id != BPF_FUNC_sk_storage_get && - func_id != BPF_FUNC_sk_storage_delete) + func_id != BPF_FUNC_sk_storage_delete && + func_id != BPF_FUNC_kptr_xchg) goto error; break; case BPF_MAP_TYPE_INODE_STORAGE: if (func_id != BPF_FUNC_inode_storage_get && - func_id != BPF_FUNC_inode_storage_delete) + func_id != BPF_FUNC_inode_storage_delete && + func_id != BPF_FUNC_kptr_xchg) goto error; break; case BPF_MAP_TYPE_TASK_STORAGE: if (func_id != BPF_FUNC_task_storage_get && - func_id != BPF_FUNC_task_storage_delete) + func_id != BPF_FUNC_task_storage_delete && + func_id != BPF_FUNC_kptr_xchg) goto error; break; case BPF_MAP_TYPE_CGRP_STORAGE: if (func_id != BPF_FUNC_cgrp_storage_get && - func_id != BPF_FUNC_cgrp_storage_delete) + func_id != BPF_FUNC_cgrp_storage_delete && + func_id != BPF_FUNC_kptr_xchg) goto error; break; case BPF_MAP_TYPE_BLOOM_FILTER: From patchwork Sun Feb 19 15:52:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13145962 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10CA6C61DA4 for ; Sun, 19 Feb 2023 15:53:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230175AbjBSPxA (ORCPT ); Sun, 19 Feb 2023 10:53:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43938 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230146AbjBSPw6 (ORCPT ); Sun, 19 Feb 2023 10:52:58 -0500 Received: from mail-ed1-x543.google.com (mail-ed1-x543.google.com [IPv6:2a00:1450:4864:20::543]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 172B346A3 for ; Sun, 19 Feb 2023 07:52:57 -0800 (PST) Received: by mail-ed1-x543.google.com with SMTP id ec23so2812256edb.1 for ; Sun, 19 Feb 2023 07:52:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=D/1so9i3HBYkTX95EWj7UWd4gR62OCbsarRt0LHkNFw=; b=S/z7EYh8pjP8qKoa4uixIseuCZo84G+Z4w2hpMQLnIO+jLyd6QfXsAasNB+veyo3nq 3rVL35V6IrQVRj+OtBqFlQ1FpqXlITxX1i5+igcERZdk3/5YbD5Mru7x3XzYUq16K4Jg jiZcLSgGGhh60yyMC2S3VOorXruyoDNsUDcb5QKO3UnLqARt1VRASzh+Z+aj3fjBXtdx vXwSroLg1+22rTAQ2hU6rkOQuygNGAdxIzbaVUO4oJ9X+biJOiTpEzKQ3HL4Gtp1+0jk KCxPaVFaYplNu5f1I9tiYh8cZtNn029dFPPnRae2/AUo+bzvNNw7Kg5pkBjHlEq1Y9bg o16g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=D/1so9i3HBYkTX95EWj7UWd4gR62OCbsarRt0LHkNFw=; b=5FEuLFpXlbI25e6N0OjlAHMCkcq90XE9oPqe3apo2MSRfQGcMzAQzEKfRp2n8oBZNv 17xCtWtWmp9/gPEJOdpgCdL8opNqapbNsLc5emZkvLHNBwxhnPXPwCjDblB+nfGym32A 3ZzFRAaLajkQmT5RX8uHdEqJiE7saKfVwkp8F2n7y6yBorG7DPI1JOA+4mRXGZ6IOiso qwcxWAORLbHQduG/MUt6hhE0ww6N/QKZUvk3+dOrGONTXs+mA4/Tq5OlkoMQcEiDGf8S HVLQkmxJqYCcOjaZ0cFNjtHSASjkfLywAh2EY7pxJC5Up+KETFUYpcnTUJ7OaZq4l6Zy BO9w== X-Gm-Message-State: AO0yUKWfh/n5C6iODRP+wqzRY+tIpRTk4wGtaIbF0tzrJYIhh9bsdLJ6 EnKTWG/3jtNeizutjj712j1sbcAclZDe4g== X-Google-Smtp-Source: AK7set+qnTEBr9ffYkb0sxhyIgtmA7CXgbd4MNZKyBoz+tkg0zo23f3EjPvN6Bx2IWsH2TJHcJf+9w== X-Received: by 2002:a17:907:2da2:b0:889:33ca:70c6 with SMTP id gt34-20020a1709072da200b0088933ca70c6mr11462725ejc.2.1676821975187; Sun, 19 Feb 2023 07:52:55 -0800 (PST) Received: from localhost ([2001:620:618:580:2:80b3:0:8d0]) by smtp.gmail.com with ESMTPSA id fx11-20020a170906b74b00b008bc8ad41646sm2092397ejb.157.2023.02.19.07.52.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Feb 2023 07:52:54 -0800 (PST) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Martin KaFai Lau , KP Singh , Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Dave Marchevsky , David Vernet Subject: [PATCH bpf-next v1 3/7] bpf: Annotate data races in bpf_local_storage Date: Sun, 19 Feb 2023 16:52:45 +0100 Message-Id: <20230219155249.1755998-4-memxor@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230219155249.1755998-1-memxor@gmail.com> References: <20230219155249.1755998-1-memxor@gmail.com> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2660; i=memxor@gmail.com; h=from:subject; bh=s/dXuZ/+okm2ypKP+JtyKlFMQDFsjWGusg00Wy+v0v8=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBj8kUe1zJfouqHU/2z1yOGqj668QLam336h9BlxOhO 1KWys3uJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCY/JFHgAKCRBM4MiGSL8RymSlD/ kB782p3N22XxbPysej6CQdL9T8Iw4N1U2Yo2cx8B2endI7W4MeSuu1uct55udrvgZ0lpepJMTYAj0V 89s6dcM3WvjxvvsWHpSAUvsVikMg1p+nWba67gW0g7athpxlkcazYeAsSOeX0roSLxMhP06Tx7mKFs 3Ow9uyIgv2iD5EndyUAP2XH7E9Xlgxx95EkpMYGJXnAFRXmYmmjvLBuyYKyny+3GKQM0Z7mE9mB4Nr jT8+gFAFlTPpQavyv9esGnX3bmhiuda7vLQTWfoSF3qW5uWVnSTTC1r6tBwiAhaUrO7GLobuZi7oN8 Oyx7Fum+XlqCdDUsPTMXcdgl2LEyTx4haOej0+1LwOJ3OqAAKOD7FEM3h0b/AIpl3fxMgmmXyM8few IBYKCK/rRVzbozj2ntjtAfv4jUVPDFbQektmuHAyJBSw4osunWRJmPh0kwjyeomQ39Ilv/wol8kKHh DXEtMl8bC/diPUqF4938m+O1iSViWisGPrO6pbSVdebibyA1bQjrKBU2nLWICOAWUUZmeMa4aZViym //xaUJJY9tyZZeSFWEljxrt6gnJGxSQ0p+qXjRgipZcj4FvBaOOZS229YiS6wqzKB6YyXwTMPdoKHZ v2B4mSIzw4QK8DV1FGNJtKl/ZXm6DsAywKZ8tGB/9O+D8M/OganminEED/Aw== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net There are a few cases where hlist_node is checked to be unhashed without holding the lock protecting its modification. In this case, one must use hlist_unhashed_lockless to avoid load tearing and KCSAN reports. Fix this by using lockless variant in places not protected by the lock. Since this is not prompted by any actual KCSAN reports but only from code review, I have not included a fixes tag. Cc: Martin KaFai Lau Cc: KP Singh Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/bpf_local_storage.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/kernel/bpf/bpf_local_storage.c b/kernel/bpf/bpf_local_storage.c index 4521d53d7d4d..6b9039460968 100644 --- a/kernel/bpf/bpf_local_storage.c +++ b/kernel/bpf/bpf_local_storage.c @@ -51,11 +51,21 @@ owner_storage(struct bpf_local_storage_map *smap, void *owner) return map->ops->map_owner_storage_ptr(owner); } +static bool selem_linked_to_storage_lockless(const struct bpf_local_storage_elem *selem) +{ + return !hlist_unhashed_lockless(&selem->snode); +} + static bool selem_linked_to_storage(const struct bpf_local_storage_elem *selem) { return !hlist_unhashed(&selem->snode); } +static bool selem_linked_to_map_lockless(const struct bpf_local_storage_elem *selem) +{ + return !hlist_unhashed_lockless(&selem->map_node); +} + static bool selem_linked_to_map(const struct bpf_local_storage_elem *selem) { return !hlist_unhashed(&selem->map_node); @@ -182,7 +192,7 @@ static void __bpf_selem_unlink_storage(struct bpf_local_storage_elem *selem, bool free_local_storage = false; unsigned long flags; - if (unlikely(!selem_linked_to_storage(selem))) + if (unlikely(!selem_linked_to_storage_lockless(selem))) /* selem has already been unlinked from sk */ return; @@ -216,7 +226,7 @@ void bpf_selem_unlink_map(struct bpf_local_storage_elem *selem) struct bpf_local_storage_map_bucket *b; unsigned long flags; - if (unlikely(!selem_linked_to_map(selem))) + if (unlikely(!selem_linked_to_map_lockless(selem))) /* selem has already be unlinked from smap */ return; @@ -428,7 +438,7 @@ bpf_local_storage_update(void *owner, struct bpf_local_storage_map *smap, err = check_flags(old_sdata, map_flags); if (err) return ERR_PTR(err); - if (old_sdata && selem_linked_to_storage(SELEM(old_sdata))) { + if (old_sdata && selem_linked_to_storage_lockless(SELEM(old_sdata))) { copy_map_value_locked(&smap->map, old_sdata->data, value, false); return old_sdata; From patchwork Sun Feb 19 15:52:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13145964 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 58A3EC05027 for ; Sun, 19 Feb 2023 15:53:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230169AbjBSPxB (ORCPT ); Sun, 19 Feb 2023 10:53:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43966 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230151AbjBSPw7 (ORCPT ); Sun, 19 Feb 2023 10:52:59 -0500 Received: from mail-ed1-x542.google.com (mail-ed1-x542.google.com [IPv6:2a00:1450:4864:20::542]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 595957ED4 for ; Sun, 19 Feb 2023 07:52:58 -0800 (PST) Received: by mail-ed1-x542.google.com with SMTP id eg19so3933234edb.0 for ; Sun, 19 Feb 2023 07:52:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yLBEn3jLcMOSmLSoNKVjC/08G00nx8V0vNuukGuHwCI=; b=HmxT8xbIuqoTvXfyLS9/1Q+qRwubsnGBNESzNRjFFAU1U2Hy4gBTgr7dUJSTdAqa58 IXaiyaSpMAq3TDGKj5WrZed9sgStkEz4GPjChPdgD+MpS8qYFr6p4iiHwA44mcGuPp2B yNnIY1q1DqaQvDf+eGHbPgiA8VY76PJMMVvqp5XYAobSk0fuk+t/dgUGISYpQgSPzApO Zx8upATMP4FVQ49lVYD3JRE8X06Gc2+uKkhHD3WOqWoacP/hm9uAHCs+LiZgVv90Ph74 jDKCXaApuUofG5jGVXZnHQghKxA/R7s4mscsg0RiJLUuPfcMo3O65DgtkBOTbm1S4L/t KAUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yLBEn3jLcMOSmLSoNKVjC/08G00nx8V0vNuukGuHwCI=; b=O70l1N+VsoKO7eYlii+EZjCrqqX6KYw91PxuynVYvSuv9q+6oiVT+1XjnmcRUeXPqb SqfqRxld1baf2nywTZWqqNxFWhEUCQRszCrSfTsjIeGFc3xhycFfMsrWODTMwxmuByCD fzbmYm/8PbKfOp+fpqRfeDSXNOaBZOKFOdmreIWbojdTIAM0VMYrDDYa3Rk76dH9ajeV STw9YI+rtnsfY3aBNmskfMuDvi8AtLdrM6t1GeAhXOElIYnZtm20YwHYvpgsi3H1G9xn EOz+kzCrK06Y2tYJFTuKLLnhklrUQGOMbLEY5BvYMscWSf+9FjZnQsNrQLcxw7Mu/0wd voPQ== X-Gm-Message-State: AO0yUKVF+QvTgyMEhiUs13igTk4rJ33aezMfCTbDnxMBLLzjtH6Tk5fY cRwBDlMJQMXNtMdaY2DZTat0nuwbKJY7KA== X-Google-Smtp-Source: AK7set/AyM50RWWGYYLNpB9sSNWuYQT8NxkWizARAkN7MvBh7gDsRech0AzElvskkDNurY6gIDk4fw== X-Received: by 2002:a17:906:4dce:b0:8a4:e0a2:e77f with SMTP id f14-20020a1709064dce00b008a4e0a2e77fmr6887300ejw.34.1676821976457; Sun, 19 Feb 2023 07:52:56 -0800 (PST) Received: from localhost ([2001:620:618:580:2:80b3:0:8d0]) by smtp.gmail.com with ESMTPSA id ay14-20020a170906d28e00b008d2d2d617ccsm104312ejb.17.2023.02.19.07.52.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Feb 2023 07:52:55 -0800 (PST) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Martin KaFai Lau , KP Singh , Dave Marchevsky , David Vernet Subject: [PATCH bpf-next v1 4/7] bpf: Remove unused MEM_ALLOC | PTR_TRUSTED checks Date: Sun, 19 Feb 2023 16:52:46 +0100 Message-Id: <20230219155249.1755998-5-memxor@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230219155249.1755998-1-memxor@gmail.com> References: <20230219155249.1755998-1-memxor@gmail.com> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1446; i=memxor@gmail.com; h=from:subject; bh=sLl/ONgY3LjqhuNq2QP2UePimgBpCudu+JdJnC28uVc=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBj8kUePWgTTP1c6F+b6jgW9VwGoxjIc4mGmqcUALza J7YGkT6JAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCY/JFHgAKCRBM4MiGSL8RyoqtD/ 9DMLVZ87MIVQqXKQ/vgr9HlhM3QTWHYCwgnbXeKmXr9p7X3TBtAE0UW9Hwrmxro3mH6nohlGOLNW4G k8Lxyu35T5V2+oxfbdgUcKtS5vOibJuItIvJNd7QW7l9cz6/nH2c3iIsS6FyceVi5an3x4fTekxHC3 IH12/vmQ7Bly+f05m5KDQa/r3W1dj+8hylcQNu9eqQTFzNnB6RpiK4WomN8TdjOYj3pNtkClK8yZh4 Uxv3s2lLktG5S58PHd4IS34o967V1NbrIloE3G3MjXDw8beTZ/aTqUGjlV+ZIsBjkinTB1CnJ4t+5m VhHVVTRjPxvip+VMCp5R6w56DGUhJKv/ww2CB8HJbL/+b3diekNEeEyZhSTXCgIe9CpNPZcjIZlH8h WkjbGj7G+lQq3QpMJ7Pz5pjbLetiTxueEjY7Ro0fSAfgTPql7Qq8JCPNUfamjSuYrEfWNJkXTg325F HpHKRFPdLlVQx1eZ73togeRCX4ZMCE4IyVAD7qj0s2E+Qwkws/xHsncOxUoDCuF++bAVnYTe044GZo rKyqeLP/bZ6dDERzYPgFYJepOc13jMHZyOA3YkemrSsEls2Z6ikK/8ajtFKHCR8KyCEMZBLA+MNR5k D4G5Lm1EGLM1AA+wqMRn8NCsAaS9MnRSJo3h54lVjmGrae0hZ8i0Wan0zA7w== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net The plan is to supposedly tag everything with PTR_TRUSTED eventually, however those changes should bring in their respective code, instead of leaving it around right now. It is arguable whether PTR_TRUSTED is required for all types, when it's only use case is making PTR_TO_BTF_ID a bit stronger, while all other types are trusted by default. Hence, just drop the two instances which do not occur in the verifier for now to avoid reader confusion. Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 9a4e7efaf28f..6837657b46bf 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -6651,7 +6651,6 @@ int check_func_arg_reg_off(struct bpf_verifier_env *env, case PTR_TO_BTF_ID | MEM_ALLOC: case PTR_TO_BTF_ID | PTR_TRUSTED: case PTR_TO_BTF_ID | MEM_RCU: - case PTR_TO_BTF_ID | MEM_ALLOC | PTR_TRUSTED: case PTR_TO_BTF_ID | MEM_ALLOC | NON_OWN_REF: /* When referenced PTR_TO_BTF_ID is passed to release function, * its fixed offset must be 0. In the other cases, fixed offset @@ -9210,7 +9209,6 @@ static int check_reg_allocation_locked(struct bpf_verifier_env *env, struct bpf_ ptr = reg->map_ptr; break; case PTR_TO_BTF_ID | MEM_ALLOC: - case PTR_TO_BTF_ID | MEM_ALLOC | PTR_TRUSTED: ptr = reg->btf; break; default: From patchwork Sun Feb 19 15:52:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13145963 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 807B5C64EC4 for ; Sun, 19 Feb 2023 15:53:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230146AbjBSPxB (ORCPT ); Sun, 19 Feb 2023 10:53:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43986 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230173AbjBSPxA (ORCPT ); Sun, 19 Feb 2023 10:53:00 -0500 Received: from mail-ed1-x542.google.com (mail-ed1-x542.google.com [IPv6:2a00:1450:4864:20::542]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 83D817EF8 for ; Sun, 19 Feb 2023 07:52:58 -0800 (PST) Received: by mail-ed1-x542.google.com with SMTP id ek25so2801950edb.7 for ; Sun, 19 Feb 2023 07:52:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1gjATKTPjeef4UxhZLE+OVD3fftepy3M/KUpIRkKPPE=; b=g00tRZJ0/d0UGGZoHGw7+f7RjEZhEaelfXjKPE9IXnWveIDEAjscu7F+x7kFnLepf9 psK9eqqwa3dSaqSCMHwoUTsbq8ZVmuCfW3zTJTMe2GSPlwExkglfht/GE20T1HTgxy9R FwnSU+kVhNsxsxDk1etakUtGKzp5YOg9y9NG+qg3yeSmPwMEb+VnBBTBqBUwXIrnyVkf g45207gmKSNeMN5G4xPOic1D0wG4DKDxDyCxTj9sewtuVEOE6QMT7XULmPs2CaQEndOz MSeoEmMtJUpt6nKhOd3H0IP1AwQb5VGri/uI5wMO6H5eBZWEebEZEhYr39umpGmjc3qz 3vVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1gjATKTPjeef4UxhZLE+OVD3fftepy3M/KUpIRkKPPE=; b=t6tOvDqxm+Md/N0FUWbtvXwaaFcdGz5S1KdmylY/bo4BygdQ3OJMASYuRR66c+KCQz JWlnulEMhcIoeDYvQFRyf9C/bPSkAtDrnu1Y42IrWZridviyHfwmN2cUm8Kxt4NsDUz9 ncK1yH6OA39JgLokJyAncVJLtJUMsWPDchBqgA97K09atn+4tEIkCATySpl5XDvfgHSk ZMV7lHvmjPN8AVlNljEwKNborl16/Xsvou1hBtx7ExlqO0lYJjAVC7X1jO6CBc0ykeLt jL2Stbj/vOpVsOT0WsbvNroq8wxMErQZ8o9wl6a5Hn99mnWjrXAQ6RM2R21MBEUHo1Fz uuYQ== X-Gm-Message-State: AO0yUKWRxmsFg3U5vJDQx6pfqoY5dzT1DxfJQllZEYgb1m4xIIyvY0L6 mYe1zsUeIE+5mJUgBHHiVuiISE3oGcrHLw== X-Google-Smtp-Source: AK7set/u7MO5NrXNupA9p+qTIRldZEG7CsD3tfEqtgio2Djkevm2jOne6vNDiFDQmopPv3Kbsg8s9A== X-Received: by 2002:a17:906:f55:b0:8b1:315c:be04 with SMTP id h21-20020a1709060f5500b008b1315cbe04mr5398774ejj.27.1676821977635; Sun, 19 Feb 2023 07:52:57 -0800 (PST) Received: from localhost ([2001:620:618:580:2:80b3:0:8d0]) by smtp.gmail.com with ESMTPSA id k3-20020a50ce43000000b004acbecf091esm947446edj.17.2023.02.19.07.52.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Feb 2023 07:52:57 -0800 (PST) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Martin KaFai Lau , KP Singh , Dave Marchevsky , David Vernet Subject: [PATCH bpf-next v1 5/7] bpf: Fix check_reg_type for PTR_TO_BTF_ID Date: Sun, 19 Feb 2023 16:52:47 +0100 Message-Id: <20230219155249.1755998-6-memxor@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230219155249.1755998-1-memxor@gmail.com> References: <20230219155249.1755998-1-memxor@gmail.com> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2598; i=memxor@gmail.com; h=from:subject; bh=Lqdy/IKzOnseTIegz0Zq4nxx9kSW/9SvKf/gr8GzPMU=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBj8kUewXAfDptv4JkR1DmGQNTV3abjCUocDju4yb7q Apnj8QeJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCY/JFHgAKCRBM4MiGSL8RyqL2D/ oDFwrfV9bvglckONid0Jpv4yTmNkPU48blpV7vYoMMaXzMVE+xNcoEQ9+BohIsLY7a4cp9JWpsBa5Q z3cHfIJjgZnTjBmTa8zMun1wZ72YJuBqVspfBRan4KJAvKOoQ16SGaqS+tDvqj7I7BHCCEmv30kzB9 6rgmH6P8NxHo84bnHfOGhcLmJeIp426A0EjuTgalnOIrhSu8iIhjh3b9eSCmNCtHTe26GEcTurPOWY U76x8t48fH5ZpDkt5vQoxFKaXY4dEr/dm++RaUzKPXyRFTiI3BFQtfmqJzHpqAUNutORBEfjhTGNAI WBbC6eKWtRGB+iDXgRTF2+1xyv2ZydgJbk+azzROUqCv5oao8eYyFgFVwg9787Z9ZHFAoKcIuWySW8 bcqpQ6QL43vmcyNPkBjfkZ18qveLhx8sCgTF8+Xta79rjoWI3aGoeLc4K6dUh13Ime2bzLhionywJN FknrW4tCTx7gHKPeVNRsmkUTLwI5JyZr+Ehj8q8ODIqXbYXTTl9GKDLvm51neunXD5K7/HGXQHdY1B XZA5CUD0Fz4hastaOtaOcmdYOIVkTKZ93ZLqaXsRpxuQubggBmzaTDS/50jba07Ye81ZqRURQJ3or3 kwez35Xv+AROOxSn95VxrNAzXIkpv4FyGLftMvFLWw8D96x/Pzv08tUzgFsw== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net The current code does type matching for the case where reg->type is PTR_TO_BTF_ID or has the PTR_TRUSTED flag. However, this only needs to occur for non-MEM_ALLOC and non-MEM_PERCPU cases, but will include both as per the current code. The MEM_ALLOC case with or without PTR_TRUSTED needs to be handled specially by the code for type_is_alloc case, while MEM_PERCPU case must be ignored. Hence, to restore correct behavior and for clarity, explicitly list out the handled PTR_TO_BTF_ID types which should be handled for each case using a switch statement. Helpers currently only take: PTR_TO_BTF_ID PTR_TO_BTF_ID | PTR_TRUSTED PTR_TO_BTF_ID | MEM_RCU PTR_TO_BTF_ID | MEM_ALLOC PTR_TO_BTF_ID | MEM_PERCPU PTR_TO_BTF_ID | MEM_PERCPU | PTR_TRUSTED This fix was also described (for the MEM_ALLOC case) in [0]. [0]: https://lore.kernel.org/bpf/20221121160657.h6z7xuvedybp5y7s@apollo Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 6837657b46bf..8dbd20735e92 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -6522,7 +6522,14 @@ static int check_reg_type(struct bpf_verifier_env *env, u32 regno, return -EACCES; found: - if (reg->type == PTR_TO_BTF_ID || reg->type & PTR_TRUSTED) { + if (base_type(reg->type) != PTR_TO_BTF_ID) + return 0; + + switch ((int)reg->type) { + case PTR_TO_BTF_ID: + case PTR_TO_BTF_ID | PTR_TRUSTED: + case PTR_TO_BTF_ID | MEM_RCU: + { /* For bpf_sk_release, it needs to match against first member * 'struct sock_common', hence make an exception for it. This * allows bpf_sk_release to work for multiple socket types. @@ -6558,13 +6565,23 @@ static int check_reg_type(struct bpf_verifier_env *env, u32 regno, return -EACCES; } } - } else if (type_is_alloc(reg->type)) { + break; + } + case PTR_TO_BTF_ID | MEM_ALLOC: if (meta->func_id != BPF_FUNC_spin_lock && meta->func_id != BPF_FUNC_spin_unlock) { verbose(env, "verifier internal error: unimplemented handling of MEM_ALLOC\n"); return -EFAULT; } + /* Handled by helper specific checks */ + break; + case PTR_TO_BTF_ID | MEM_PERCPU: + case PTR_TO_BTF_ID | MEM_PERCPU | PTR_TRUSTED: + /* Handled by helper specific checks */ + break; + default: + verbose(env, "verifier internal error: invalid PTR_TO_BTF_ID register for type match\n"); + return -EFAULT; } - return 0; } From patchwork Sun Feb 19 15:52:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13145965 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BCBC2C61DA4 for ; Sun, 19 Feb 2023 15:53:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230151AbjBSPxD (ORCPT ); Sun, 19 Feb 2023 10:53:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43972 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230043AbjBSPxC (ORCPT ); Sun, 19 Feb 2023 10:53:02 -0500 Received: from mail-ed1-x541.google.com (mail-ed1-x541.google.com [IPv6:2a00:1450:4864:20::541]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1C4FF7EF8 for ; Sun, 19 Feb 2023 07:53:01 -0800 (PST) Received: by mail-ed1-x541.google.com with SMTP id er22so2963419edb.4 for ; Sun, 19 Feb 2023 07:53:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1r/OJAG6pYjz3uYMx2mH7/j/3RiixHDF+PC6j6HMZ2A=; b=R3/tLS/aK7t15/GR6+5/J9WhzGFbhNEPBo2sSnXNzZhKT26EHcLhw5MaYa6jsz23Pp J8+FMCvV/yWE5AaPTfCoLpuOa1hwLwuZkR/L6mT+WuR3B3UxJAzu5t/Bv5X34IbOuuZ0 ZIhflgcTZbUwEvHS0jhC56nq+KXQw55esCZCAIRQyY89r5Tn5hN+AJ29TC6D7uewwoGL OhCxknTLQzscXL1aa4Jysnj6kWjYT3TCjpmlYCQMapp5TrWGErKFn9iz8Nrf6MizxFvI r0Wscf/R4pDNQjZqhCoGlTs7pp8diu7P+jK5g+aQUXO6QtbneHiu08FIN5TLFqQbbsZ6 BETQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1r/OJAG6pYjz3uYMx2mH7/j/3RiixHDF+PC6j6HMZ2A=; b=T2x6cJBO/q3/l5fB7vgjIOiiLE/mO7dcyKpeRUz0NXYBvabQo+j++xSvLmy/nBXhsG iWviDhOvlhVxv+DSAT+fUTcBLPF96yGo8w3/a9xl/dFgJFI9FFkyNygcPmlkWOSuxhnt dAC10aikYYzEJJCEtpX+Tldrn/XeqbIS72DnmlYRy2X+7m61vyhDJ3tZ9ot0+dqr4sdA Z0QcKRnDLa1KjljbBKD/j/Zp04IWy7iCb/l4k6TBMkPrVwbVMUBTqhLIPLQKd0KU51vp M7zgqU4Wc+y34DtDvM4OEZGtyBTRfx+VRTOCz19HNoSDILoU+/SWiGe+PEKcBRbTy/CX ksIA== X-Gm-Message-State: AO0yUKWoFYY1zPKmD/j8a2PKOeFP85lAuaUHyGhYAYRLi6nchAuv8p2y D/xW4ZOSnOwP1s+8xz8wNeoU7t8sDOMEBQ== X-Google-Smtp-Source: AK7set8zslRMvvKZM5/a8MZmshgqZe4YD+UwW9OMT8kb0UVn9zTpAAsP2AfiqkcpCe+5r4/YJME9GA== X-Received: by 2002:a17:906:3c2:b0:8b1:38a0:215c with SMTP id c2-20020a17090603c200b008b138a0215cmr8368578eja.76.1676821978962; Sun, 19 Feb 2023 07:52:58 -0800 (PST) Received: from localhost ([2001:620:618:580:2:80b3:0:8d0]) by smtp.gmail.com with ESMTPSA id z7-20020a170906714700b008b17b0f5d07sm3734080ejj.217.2023.02.19.07.52.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Feb 2023 07:52:58 -0800 (PST) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Martin KaFai Lau , KP Singh , Dave Marchevsky , David Vernet Subject: [PATCH bpf-next v1 6/7] bpf: Wrap register invalidation with a helper Date: Sun, 19 Feb 2023 16:52:48 +0100 Message-Id: <20230219155249.1755998-7-memxor@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230219155249.1755998-1-memxor@gmail.com> References: <20230219155249.1755998-1-memxor@gmail.com> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3186; i=memxor@gmail.com; h=from:subject; bh=nH2wX2lbjSnpaunG5MSM50qcXyLVo8LAtKNm1jgtDWo=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBj8kUeq+lsdIPAVzWlfBe5DxeOpg2l+MjZPHhrixuj esKP2luJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCY/JFHgAKCRBM4MiGSL8RyqPwD/ 45P2EoBKjf+bYNCAK9FlUuT8wMRpAiVJhzyj4RjxCHdh8u8lUQIzL5tvyxLvAnSQEZCKRetFRvusT2 kSI9fzWNFV/f2M3AMFkFCwEU5pGheCfj0eSNp2gSVoXjZCO8yy2E5YrtXspboEmZqLfcv+qUoGf1TE Vv0bPiihJ0VwoONFp8CdqACL+VavtZikP/zee4Vr9aFJVBF+xXy3EfKMzCFivvux/FSKGgG17NYOxr dIG66FLuhiLdziyCYwaZ+OiyXxTMAVKoAKEdW3vNmmmkX+OydGkMqBbq8Cmd6GMo6+Uo4JQ0f6ca55 Rcy/lNSmJfMNba667rIe8EhkUPY0t77cOFj25jpdpt2NoijG0+gqLM9ewAgUtC5bmt9tkUAbGgkiq7 EAZmPPkkgs1bsLegg1mHY/SuOYh7FaTMKD2UABbcUbJ6l8Ksr0ahVnVyZ3pzqqw6qiYOPrVNepQ4Dv eivyZF4nxi9s+tuOJTywwZrXxodgCIXj8GzzEzzF+jqdYXhlMkLU3DmUvXhze9AQeoJsCrgj9F9OYR 5ZdrHkNNqBCgj5URuPn9ml52phFwLrxRkvCsDigNbdobVUg9UBzCrnCoP2ENvQ9QYUA1JmeZzddUZM Us2YDYJh1bBW0pcsOx3WceYuGmMFPZCzR6eC88UQ4Fv6/fqrbe66pWlFr2Gg== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Typically, verifier should use env->allow_ptr_leaks when invaliding registers for users that don't have CAP_PERFMON or CAP_SYS_ADMIN to avoid leaking the pointer value. This is similar in spirit to c67cae551f0d ("bpf: Tighten ptr_to_btf_id checks."). In a lot of the existing checks, we know the capabilities are present, hence we don't do the check. Instead of being inconsistent in the application of the check, wrap the action of invalidating a register into a helper named 'mark_invalid_reg' and use it in a uniform fashion to replace open coded invalidation operations, so that the check is always made regardless of the call site and we don't have to remember whether it needs to be done or not for each case. Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 8dbd20735e92..d856ee74ad63 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -895,6 +895,14 @@ static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_re static void __mark_reg_unknown(const struct bpf_verifier_env *env, struct bpf_reg_state *reg); +static void mark_reg_invalid(const struct bpf_verifier_env *env, struct bpf_reg_state *reg) +{ + if (!env->allow_ptr_leaks) + __mark_reg_not_init(env, reg); + else + __mark_reg_unknown(env, reg); +} + static int destroy_if_dynptr_stack_slot(struct bpf_verifier_env *env, struct bpf_func_state *state, int spi) { @@ -934,12 +942,8 @@ static int destroy_if_dynptr_stack_slot(struct bpf_verifier_env *env, /* Dynptr slices are only PTR_TO_MEM_OR_NULL and PTR_TO_MEM */ if (dreg->type != (PTR_TO_MEM | PTR_MAYBE_NULL) && dreg->type != PTR_TO_MEM) continue; - if (dreg->dynptr_id == dynptr_id) { - if (!env->allow_ptr_leaks) - __mark_reg_not_init(env, dreg); - else - __mark_reg_unknown(env, dreg); - } + if (dreg->dynptr_id == dynptr_id) + mark_reg_invalid(env, dreg); })); /* Do not release reference state, we are destroying dynptr on stack, @@ -7383,7 +7387,7 @@ static void clear_all_pkt_pointers(struct bpf_verifier_env *env) bpf_for_each_reg_in_vstate(env->cur_state, state, reg, ({ if (reg_is_pkt_pointer_any(reg)) - __mark_reg_unknown(env, reg); + mark_reg_invalid(env, reg); })); } @@ -7428,12 +7432,8 @@ static int release_reference(struct bpf_verifier_env *env, return err; bpf_for_each_reg_in_vstate(env->cur_state, state, reg, ({ - if (reg->ref_obj_id == ref_obj_id) { - if (!env->allow_ptr_leaks) - __mark_reg_not_init(env, reg); - else - __mark_reg_unknown(env, reg); - } + if (reg->ref_obj_id == ref_obj_id) + mark_reg_invalid(env, reg); })); return 0; @@ -7446,7 +7446,7 @@ static void invalidate_non_owning_refs(struct bpf_verifier_env *env) bpf_for_each_reg_in_vstate(env->cur_state, unused, reg, ({ if (type_is_non_owning_ref(reg->type)) - __mark_reg_unknown(env, reg); + mark_reg_invalid(env, reg); })); } From patchwork Sun Feb 19 15:52:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13145966 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 843FAC61DA4 for ; Sun, 19 Feb 2023 15:53:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230011AbjBSPxG (ORCPT ); Sun, 19 Feb 2023 10:53:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44132 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230043AbjBSPxE (ORCPT ); Sun, 19 Feb 2023 10:53:04 -0500 Received: from mail-ed1-x541.google.com (mail-ed1-x541.google.com [IPv6:2a00:1450:4864:20::541]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4D27DA26E for ; Sun, 19 Feb 2023 07:53:02 -0800 (PST) Received: by mail-ed1-x541.google.com with SMTP id x22so3116477edd.10 for ; Sun, 19 Feb 2023 07:53:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=EbNQdHZY9pOZP+hGnRs1uyln8Su8LLk8qDTO4PuyI74=; b=J5MsPYZKis1BgTyCKJGkHBkd6NL8y9QZoGrxAGKUchYoXfKCrYDniJgRkm+1RCUfps NBRQxrlhwRua8enBgC3lo/kCt2zJed/kNVP6LN8EpkulQKmkkWpx0g5SPS3th0ThTTAi Q5hHBAH60BFu1Jl+c84aUW0rss/eO1onaSSSwA8wcrIDMH8f3r5+8OpztZabW1iSgWvA oOw1Kce8LnDNuRpsR5peP8G2QeKYVXNa0nIb4Z6jAFZ+ktynxHP7Ssu616KODlcjCbAL nGvcrWZkZgMOZZFmlU8UOok79tIJxZ7lWVy6GBo+CZq81/H18pMkk/hsP6mVAne3m6uY Rs8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EbNQdHZY9pOZP+hGnRs1uyln8Su8LLk8qDTO4PuyI74=; b=R4odE1M0zp6eIlZb223fGPQOaDspOsMD7XobxQ0laxPyohrm04h6nkMQgPo+xgCnxQ G8QDQBJSbmcctzLkCHl27V4I2lZnep1MIAn/naMbvwpyv5m+M4WDFmIX5x7Z1PkQTyrY h3ohnjGQKOddp9NhgCpZ1CEEjOJ2f3N8zhyc7JVpIONTxoiyVclRZxZ62Z/dO/FO/PZy djY3gPIpnr+VyZtQUc6lLBm5Y8JEQd9u5Ou4QZz/v5sTygOqHa3tPFlWnoB1AYruXpfK NTMinW7JpCOPwIv0knFhgEBi1nuECBiBjdcg4BMPdRrEwLemr9wZc//nrw+fnnzz2l14 GHuw== X-Gm-Message-State: AO0yUKWAHogWj/Df9UAiX/rOg6HEW0IXAFVNn7MI2XQC0lbVYLIlWW4u lyEg5kJ5zXIDxjOedIr4q8uKxkUS0694ww== X-Google-Smtp-Source: AK7set9XmeMYmnUcBfS/XzsphRBoNyPV8loCHT/uO0Be+gAycbQulOvX3qt7iwSRY8HOtirNpHiYhQ== X-Received: by 2002:a17:906:198f:b0:8b1:77bf:5b9f with SMTP id g15-20020a170906198f00b008b177bf5b9fmr9116526ejd.13.1676821980319; Sun, 19 Feb 2023 07:53:00 -0800 (PST) Received: from localhost ([2001:620:618:580:2:80b3:0:8d0]) by smtp.gmail.com with ESMTPSA id mf26-20020a170906cb9a00b008c44438734csm1599546ejb.113.2023.02.19.07.52.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Feb 2023 07:52:59 -0800 (PST) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Martin KaFai Lau , KP Singh , Dave Marchevsky , David Vernet Subject: [PATCH bpf-next v1 7/7] selftests/bpf: Add more tests for kptrs in maps Date: Sun, 19 Feb 2023 16:52:49 +0100 Message-Id: <20230219155249.1755998-8-memxor@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230219155249.1755998-1-memxor@gmail.com> References: <20230219155249.1755998-1-memxor@gmail.com> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=18274; i=memxor@gmail.com; h=from:subject; bh=FhwdAx8rDnzeIWJ0B+rZSR04vKpallaEP1UcGwT7cQU=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBj8kUflnSINva7DePLxyI+8ZSKm3L+3sKKsCPcu0Jp AEL+AFqJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCY/JFHwAKCRBM4MiGSL8RyifCD/ 4nQnLf0DpAERe6/NM5NM1Ou5rBRa2Mt3QGwsc3JbJbmMH0fZJHFoIjQP1hRs0YeMvXFHXo15ZAZa1I OzDZqEszRNP/uqw3sig6YjP8nyqmGnRjI+PCubqGq3eV66kdqro5nWBeRgtcII6dN7SSCQx4OA7rmD sAG6mXWxUV0o+Kvry5kPxzgMVZjcllDU8dqcqbzH+bxWHHcOG497FUB1RAgLTFfxhAWhNRCGlKeZYJ Y/eEj7KXTvntMu2N5SfYJ/T5yYLipnYiCKXDd9MWRGKh9O40sbvp2JOxL6EbNDU0xFTXe+HIciSwPR ABkzlqohvmkOpwyhi+huRng2lNUPyHZaDfB0hKEyhc9SflIvpJWWDcpluyUyV/fcjwMK5DDmlj1+7v UWmlRq8vI5mlqJ6yOtYUd52zdQ2ZBiIq/Q0D8ev8sA5NVFmrZFZSFZBbumKyzgEWCmc/kmXcFTVpwN huVsZwQr5IrgQz6lNnf2UB6W9WCaG5MnaN9Ki8f07BoiJfpVst3h9XHKeaZdt0BgpPNjdlrUKCf1se IcJPGt+prbxHVxvzg6ADj76zL/W4ebKpi3dMH4Gc1n8XRqnxvrMFHHwdqk3vkZuN2JERP+BUTZz3Uc wWqmp2c6+SVAl5XzZsj2otisCa9qu4cnFu+bW4N+mDdK2pmnVJI1s+mNl4oQ== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Firstly, ensure programs successfully load when using all of the supported maps. Then, extend existing tests to test more cases at runtime. We are currently testing both the synchronous freeing of items and asynchronous destruction when map is freed, but the code needs to be adjusted a bit to be able to also accomodate support for percpu maps. We now do a delete on the item (and update for array maps which has a similar effect for kptrs) to perform a synchronous free of the kptr, and test destruction both for the synchronous and asynchronous deletion. Next time the program runs, it should observe the refcount as 1 since all existing references should have been released by then. By running the program after both possible paths freeing kptrs, we establish that they correctly release resources. Next, we augment the existing test to also test the same code path shared by all local storage maps using a task local storage map. Signed-off-by: Kumar Kartikeya Dwivedi --- .../selftests/bpf/prog_tests/map_kptr.c | 96 +++-- tools/testing/selftests/bpf/progs/map_kptr.c | 333 +++++++++++++++--- 2 files changed, 364 insertions(+), 65 deletions(-) diff --git a/tools/testing/selftests/bpf/prog_tests/map_kptr.c b/tools/testing/selftests/bpf/prog_tests/map_kptr.c index 3533a4ecad01..588d6a79b7ac 100644 --- a/tools/testing/selftests/bpf/prog_tests/map_kptr.c +++ b/tools/testing/selftests/bpf/prog_tests/map_kptr.c @@ -7,67 +7,117 @@ static void test_map_kptr_success(bool test_run) { + LIBBPF_OPTS(bpf_test_run_opts, lopts); LIBBPF_OPTS(bpf_test_run_opts, opts, .data_in = &pkt_v4, .data_size_in = sizeof(pkt_v4), .repeat = 1, ); + int key = 0, ret, cpu; struct map_kptr *skel; - int key = 0, ret; - char buf[16]; + char buf[16], *pbuf; skel = map_kptr__open_and_load(); if (!ASSERT_OK_PTR(skel, "map_kptr__open_and_load")) return; - ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.test_map_kptr_ref), &opts); - ASSERT_OK(ret, "test_map_kptr_ref refcount"); - ASSERT_OK(opts.retval, "test_map_kptr_ref retval"); + ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.test_map_kptr_ref1), &opts); + ASSERT_OK(ret, "test_map_kptr_ref1 refcount"); + ASSERT_OK(opts.retval, "test_map_kptr_ref1 retval"); ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.test_map_kptr_ref2), &opts); ASSERT_OK(ret, "test_map_kptr_ref2 refcount"); ASSERT_OK(opts.retval, "test_map_kptr_ref2 retval"); + ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.test_ls_map_kptr_ref1), &lopts); + ASSERT_OK(ret, "test_ls_map_kptr_ref1 refcount"); + ASSERT_OK(opts.retval, "test_ls_map_kptr_ref1 retval"); + ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.test_ls_map_kptr_ref2), &lopts); + ASSERT_OK(ret, "test_ls_map_kptr_ref2 refcount"); + ASSERT_OK(opts.retval, "test_ls_map_kptr_ref2 retval"); + if (test_run) goto exit; + cpu = libbpf_num_possible_cpus(); + if (!ASSERT_GT(cpu, 0, "libbpf_num_possible_cpus")) + goto exit; + + pbuf = calloc(cpu, sizeof(buf)); + if (!ASSERT_OK_PTR(pbuf, "calloc(pbuf)")) + goto exit; + ret = bpf_map__update_elem(skel->maps.array_map, &key, sizeof(key), buf, sizeof(buf), 0); ASSERT_OK(ret, "array_map update"); - ret = bpf_map__update_elem(skel->maps.array_map, - &key, sizeof(key), buf, sizeof(buf), 0); - ASSERT_OK(ret, "array_map update2"); + ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.test_map_kptr_ref3), &opts); + ASSERT_OK(ret, "test_map_kptr_ref3 refcount"); + ASSERT_OK(opts.retval, "test_map_kptr_ref3 retval"); + + ret = bpf_map__update_elem(skel->maps.pcpu_array_map, + &key, sizeof(key), pbuf, cpu * sizeof(buf), 0); + ASSERT_OK(ret, "pcpu_array_map update"); + ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.test_map_kptr_ref3), &opts); + ASSERT_OK(ret, "test_map_kptr_ref3 refcount"); + ASSERT_OK(opts.retval, "test_map_kptr_ref3 retval"); - ret = bpf_map__update_elem(skel->maps.hash_map, - &key, sizeof(key), buf, sizeof(buf), 0); - ASSERT_OK(ret, "hash_map update"); ret = bpf_map__delete_elem(skel->maps.hash_map, &key, sizeof(key), 0); ASSERT_OK(ret, "hash_map delete"); + ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.test_map_kptr_ref3), &opts); + ASSERT_OK(ret, "test_map_kptr_ref3 refcount"); + ASSERT_OK(opts.retval, "test_map_kptr_ref3 retval"); + + ret = bpf_map__delete_elem(skel->maps.pcpu_hash_map, &key, sizeof(key), 0); + ASSERT_OK(ret, "pcpu_hash_map delete"); + ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.test_map_kptr_ref3), &opts); + ASSERT_OK(ret, "test_map_kptr_ref3 refcount"); + ASSERT_OK(opts.retval, "test_map_kptr_ref3 retval"); - ret = bpf_map__update_elem(skel->maps.hash_malloc_map, - &key, sizeof(key), buf, sizeof(buf), 0); - ASSERT_OK(ret, "hash_malloc_map update"); ret = bpf_map__delete_elem(skel->maps.hash_malloc_map, &key, sizeof(key), 0); ASSERT_OK(ret, "hash_malloc_map delete"); + ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.test_map_kptr_ref3), &opts); + ASSERT_OK(ret, "test_map_kptr_ref3 refcount"); + ASSERT_OK(opts.retval, "test_map_kptr_ref3 retval"); + + ret = bpf_map__delete_elem(skel->maps.pcpu_hash_malloc_map, &key, sizeof(key), 0); + ASSERT_OK(ret, "pcpu_hash_malloc_map delete"); + ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.test_map_kptr_ref3), &opts); + ASSERT_OK(ret, "test_map_kptr_ref3 refcount"); + ASSERT_OK(opts.retval, "test_map_kptr_ref3 retval"); - ret = bpf_map__update_elem(skel->maps.lru_hash_map, - &key, sizeof(key), buf, sizeof(buf), 0); - ASSERT_OK(ret, "lru_hash_map update"); ret = bpf_map__delete_elem(skel->maps.lru_hash_map, &key, sizeof(key), 0); ASSERT_OK(ret, "lru_hash_map delete"); + ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.test_map_kptr_ref3), &opts); + ASSERT_OK(ret, "test_map_kptr_ref3 refcount"); + ASSERT_OK(opts.retval, "test_map_kptr_ref3 retval"); + + ret = bpf_map__delete_elem(skel->maps.lru_pcpu_hash_map, &key, sizeof(key), 0); + ASSERT_OK(ret, "lru_pcpu_hash_map delete"); + ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.test_map_kptr_ref3), &opts); + ASSERT_OK(ret, "test_map_kptr_ref3 refcount"); + ASSERT_OK(opts.retval, "test_map_kptr_ref3 retval"); + ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.test_ls_map_kptr_ref_del), &lopts); + ASSERT_OK(ret, "test_ls_map_kptr_ref_del delete"); + ASSERT_OK(opts.retval, "test_ls_map_kptr_ref_del retval"); + + free(pbuf); exit: map_kptr__destroy(skel); } void test_map_kptr(void) { - if (test__start_subtest("success")) { + RUN_TESTS(map_kptr_fail); + + if (test__start_subtest("success-map")) { + test_map_kptr_success(true); + + ASSERT_OK(kern_sync_rcu(), "sync rcu"); + /* Observe refcount dropping to 1 on bpf_map_free_deferred */ test_map_kptr_success(false); - /* Do test_run twice, so that we see refcount going back to 1 - * after we leave it in map from first iteration. - */ + + ASSERT_OK(kern_sync_rcu(), "sync rcu"); + /* Observe refcount dropping to 1 on synchronous delete elem */ test_map_kptr_success(true); } - - RUN_TESTS(map_kptr_fail); } diff --git a/tools/testing/selftests/bpf/progs/map_kptr.c b/tools/testing/selftests/bpf/progs/map_kptr.c index 228ec45365a8..de7337abfbcc 100644 --- a/tools/testing/selftests/bpf/progs/map_kptr.c +++ b/tools/testing/selftests/bpf/progs/map_kptr.c @@ -15,6 +15,13 @@ struct array_map { __uint(max_entries, 1); } array_map SEC(".maps"); +struct pcpu_array_map { + __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); + __type(key, int); + __type(value, struct map_value); + __uint(max_entries, 1); +} pcpu_array_map SEC(".maps"); + struct hash_map { __uint(type, BPF_MAP_TYPE_HASH); __type(key, int); @@ -22,6 +29,13 @@ struct hash_map { __uint(max_entries, 1); } hash_map SEC(".maps"); +struct pcpu_hash_map { + __uint(type, BPF_MAP_TYPE_PERCPU_HASH); + __type(key, int); + __type(value, struct map_value); + __uint(max_entries, 1); +} pcpu_hash_map SEC(".maps"); + struct hash_malloc_map { __uint(type, BPF_MAP_TYPE_HASH); __type(key, int); @@ -30,6 +44,14 @@ struct hash_malloc_map { __uint(map_flags, BPF_F_NO_PREALLOC); } hash_malloc_map SEC(".maps"); +struct pcpu_hash_malloc_map { + __uint(type, BPF_MAP_TYPE_PERCPU_HASH); + __type(key, int); + __type(value, struct map_value); + __uint(max_entries, 1); + __uint(map_flags, BPF_F_NO_PREALLOC); +} pcpu_hash_malloc_map SEC(".maps"); + struct lru_hash_map { __uint(type, BPF_MAP_TYPE_LRU_HASH); __type(key, int); @@ -37,6 +59,41 @@ struct lru_hash_map { __uint(max_entries, 1); } lru_hash_map SEC(".maps"); +struct lru_pcpu_hash_map { + __uint(type, BPF_MAP_TYPE_LRU_PERCPU_HASH); + __type(key, int); + __type(value, struct map_value); + __uint(max_entries, 1); +} lru_pcpu_hash_map SEC(".maps"); + +struct cgrp_ls_map { + __uint(type, BPF_MAP_TYPE_CGRP_STORAGE); + __uint(map_flags, BPF_F_NO_PREALLOC); + __type(key, int); + __type(value, struct map_value); +} cgrp_ls_map SEC(".maps"); + +struct task_ls_map { + __uint(type, BPF_MAP_TYPE_TASK_STORAGE); + __uint(map_flags, BPF_F_NO_PREALLOC); + __type(key, int); + __type(value, struct map_value); +} task_ls_map SEC(".maps"); + +struct inode_ls_map { + __uint(type, BPF_MAP_TYPE_INODE_STORAGE); + __uint(map_flags, BPF_F_NO_PREALLOC); + __type(key, int); + __type(value, struct map_value); +} inode_ls_map SEC(".maps"); + +struct sk_ls_map { + __uint(type, BPF_MAP_TYPE_SK_STORAGE); + __uint(map_flags, BPF_F_NO_PREALLOC); + __type(key, int); + __type(value, struct map_value); +} sk_ls_map SEC(".maps"); + #define DEFINE_MAP_OF_MAP(map_type, inner_map_type, name) \ struct { \ __uint(type, map_type); \ @@ -160,6 +217,58 @@ int test_map_kptr(struct __sk_buff *ctx) return 0; } +SEC("tp_btf/cgroup_mkdir") +int BPF_PROG(test_cgrp_map_kptr, struct cgroup *cgrp, const char *path) +{ + struct map_value *v; + + v = bpf_cgrp_storage_get(&cgrp_ls_map, cgrp, NULL, BPF_LOCAL_STORAGE_GET_F_CREATE); + if (v) + test_kptr(v); + return 0; +} + +SEC("lsm/inode_unlink") +int BPF_PROG(test_task_map_kptr, struct inode *inode, struct dentry *victim) +{ + struct task_struct *task; + struct map_value *v; + + task = bpf_get_current_task_btf(); + if (!task) + return 0; + v = bpf_task_storage_get(&task_ls_map, task, NULL, BPF_LOCAL_STORAGE_GET_F_CREATE); + if (v) + test_kptr(v); + return 0; +} + +SEC("lsm/inode_unlink") +int BPF_PROG(test_inode_map_kptr, struct inode *inode, struct dentry *victim) +{ + struct map_value *v; + + v = bpf_inode_storage_get(&inode_ls_map, inode, NULL, BPF_LOCAL_STORAGE_GET_F_CREATE); + if (v) + test_kptr(v); + return 0; +} + +SEC("tc") +int test_sk_map_kptr(struct __sk_buff *ctx) +{ + struct map_value *v; + struct bpf_sock *sk; + + sk = ctx->sk; + if (!sk) + return 0; + v = bpf_sk_storage_get(&sk_ls_map, sk, NULL, BPF_LOCAL_STORAGE_GET_F_CREATE); + if (v) + test_kptr(v); + return 0; +} + SEC("tc") int test_map_in_map_kptr(struct __sk_buff *ctx) { @@ -189,66 +298,59 @@ int test_map_in_map_kptr(struct __sk_buff *ctx) return 0; } -SEC("tc") -int test_map_kptr_ref(struct __sk_buff *ctx) +static __always_inline +int test_map_kptr_ref_pre(struct map_value *v, int off) { struct prog_test_ref_kfunc *p, *p_st; unsigned long arg = 0; - struct map_value *v; - int key = 0, ret; + int ret; p = bpf_kfunc_call_test_acquire(&arg); if (!p) return 1; p_st = p->next; - if (p_st->cnt.refs.counter != 2) { + if (p_st->cnt.refs.counter != 2 + off) { ret = 2; goto end; } - v = bpf_map_lookup_elem(&array_map, &key); - if (!v) { - ret = 3; - goto end; - } - p = bpf_kptr_xchg(&v->ref_ptr, p); if (p) { - ret = 4; + ret = 3; goto end; } - if (p_st->cnt.refs.counter != 2) - return 5; + if (p_st->cnt.refs.counter != 2 + off) + return 4; p = bpf_kfunc_call_test_kptr_get(&v->ref_ptr, 0, 0); if (!p) - return 6; - if (p_st->cnt.refs.counter != 3) { - ret = 7; + return 5; + if (p_st->cnt.refs.counter != 3 + off) { + ret = 6; goto end; } bpf_kfunc_call_test_release(p); - if (p_st->cnt.refs.counter != 2) - return 8; + if (p_st->cnt.refs.counter != 2 + off) + return 7; p = bpf_kptr_xchg(&v->ref_ptr, NULL); if (!p) - return 9; + return 8; bpf_kfunc_call_test_release(p); - if (p_st->cnt.refs.counter != 1) - return 10; + if (p_st->cnt.refs.counter != 1 + off) + return 9; p = bpf_kfunc_call_test_acquire(&arg); if (!p) - return 11; + return 10; p = bpf_kptr_xchg(&v->ref_ptr, p); if (p) { - ret = 12; + ret = 11; goto end; } - if (p_st->cnt.refs.counter != 2) - return 13; + if (p_st->cnt.refs.counter != 2 + off) + return 12; /* Leave in map */ return 0; @@ -257,38 +359,185 @@ int test_map_kptr_ref(struct __sk_buff *ctx) return ret; } -SEC("tc") -int test_map_kptr_ref2(struct __sk_buff *ctx) +static __always_inline +int test_map_kptr_ref_post(struct map_value *v, int off) { struct prog_test_ref_kfunc *p, *p_st; - struct map_value *v; - int key = 0; - - v = bpf_map_lookup_elem(&array_map, &key); - if (!v) - return 1; p_st = v->ref_ptr; - if (!p_st || p_st->cnt.refs.counter != 2) - return 2; + if (!p_st || p_st->cnt.refs.counter != 2 + off) + return 1; p = bpf_kptr_xchg(&v->ref_ptr, NULL); if (!p) - return 3; - if (p_st->cnt.refs.counter != 2) { + return 2; + if (p_st->cnt.refs.counter != 2 + off) { bpf_kfunc_call_test_release(p); - return 4; + return 3; } p = bpf_kptr_xchg(&v->ref_ptr, p); if (p) { bpf_kfunc_call_test_release(p); - return 5; + return 4; } - if (p_st->cnt.refs.counter != 2) - return 6; + if (p_st->cnt.refs.counter != 2 + off) + return 5; + + return 0; +} + +#define TEST(map, off) \ + v = bpf_map_lookup_elem(&map, &key); \ + if (!v) \ + return -1; \ + ret = test_map_kptr_ref_pre(v, off); \ + if (ret) \ + return ret; + +#define TEST_PCPU(map, off) \ + v = bpf_map_lookup_percpu_elem(&map, &key, 0); \ + if (!v) \ + return -1; \ + ret = test_map_kptr_ref_pre(v, off); \ + if (ret) \ + return ret; + +int off = 0; + +SEC("tc") +int test_map_kptr_ref1(struct __sk_buff *ctx) +{ + struct map_value *v, val = {}; + int key = 0, ret; + + bpf_map_update_elem(&hash_map, &key, &val, 0); + bpf_map_update_elem(&hash_malloc_map, &key, &val, 0); + bpf_map_update_elem(&lru_hash_map, &key, &val, 0); + + bpf_map_update_elem(&pcpu_hash_map, &key, &val, 0); + bpf_map_update_elem(&pcpu_hash_malloc_map, &key, &val, 0); + bpf_map_update_elem(&lru_pcpu_hash_map, &key, &val, 0); + + TEST(array_map, off++); + TEST(hash_map, off++); + TEST(hash_malloc_map, off++); + TEST(lru_hash_map, off++); + + TEST_PCPU(pcpu_array_map, off++); + TEST_PCPU(pcpu_hash_map, off++); + TEST_PCPU(pcpu_hash_malloc_map, off++); + TEST_PCPU(lru_pcpu_hash_map, off++); return 0; } +#undef TEST +#undef TEST_PCPU + +#define TEST(map, off) \ + v = bpf_map_lookup_elem(&map, &key); \ + if (!v) \ + return -1; \ + ret = test_map_kptr_ref_post(v, off); \ + if (ret) \ + return ret; + +#define TEST_PCPU(map, off) \ + v = bpf_map_lookup_percpu_elem(&map, &key, 0); \ + if (!v) \ + return -1; \ + ret = test_map_kptr_ref_post(v, off); \ + if (ret) \ + return ret; + +SEC("tc") +int test_map_kptr_ref2(struct __sk_buff *ctx) +{ + struct map_value *v; + int key = 0, ret; + + off--; + + TEST(array_map, off); + TEST(hash_map, off); + TEST(hash_malloc_map, off); + TEST(lru_hash_map, off); + + TEST_PCPU(pcpu_array_map, off); + TEST_PCPU(pcpu_hash_map, off); + TEST_PCPU(pcpu_hash_malloc_map, off); + TEST_PCPU(lru_pcpu_hash_map, off); + + return 0; +} + +#undef TEST +#undef TEST_PCPU + +SEC("tc") +int test_map_kptr_ref3(struct __sk_buff *ctx) +{ + struct prog_test_ref_kfunc *p; + unsigned long sp = 0; + + p = bpf_kfunc_call_test_acquire(&sp); + if (!p) + return 1; + if (p->cnt.refs.counter != 2 + off) { + bpf_kfunc_call_test_release(p); + return 2; + } + bpf_kfunc_call_test_release(p); + off--; + return 0; +} + +SEC("fmod_ret/bpf_modify_return_test") +int BPF_PROG(test_ls_map_kptr_ref1, int a, int *b) +{ + struct task_struct *current; + struct map_value *v; + + current = bpf_get_current_task_btf(); + if (!current) + return 100; + v = bpf_task_storage_get(&task_ls_map, current, NULL, BPF_LOCAL_STORAGE_GET_F_CREATE); + if (!v) + return 200; + return test_map_kptr_ref_pre(v, off++); +} + +SEC("fmod_ret/bpf_modify_return_test") +int BPF_PROG(test_ls_map_kptr_ref2, int a, int *b) +{ + struct task_struct *current; + struct map_value *v; + + current = bpf_get_current_task_btf(); + if (!current) + return 100; + v = bpf_task_storage_get(&task_ls_map, current, NULL, BPF_LOCAL_STORAGE_GET_F_CREATE); + if (!v) + return 200; + return test_map_kptr_ref_post(v, off); +} + +SEC("fmod_ret/bpf_modify_return_test") +int BPF_PROG(test_ls_map_kptr_ref_del, int a, int *b) +{ + struct task_struct *current; + struct map_value *v; + + current = bpf_get_current_task_btf(); + if (!current) + return 100; + v = bpf_task_storage_get(&task_ls_map, current, NULL, 0); + if (!v) + return 200; + if (!v->ref_ptr) + return 300; + return bpf_task_storage_delete(&task_ls_map, current); +} + char _license[] SEC("license") = "GPL";