From patchwork Sun Mar 20 15:54:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 12786551 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1C21C433EF for ; Sun, 20 Mar 2022 15:55:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233295AbiCTP4n (ORCPT ); Sun, 20 Mar 2022 11:56:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48034 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241132AbiCTP4m (ORCPT ); Sun, 20 Mar 2022 11:56:42 -0400 Received: from mail-pf1-x441.google.com (mail-pf1-x441.google.com [IPv6:2607:f8b0:4864:20::441]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5E8C854194 for ; Sun, 20 Mar 2022 08:55:18 -0700 (PDT) Received: by mail-pf1-x441.google.com with SMTP id s8so13525901pfk.12 for ; Sun, 20 Mar 2022 08:55:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=NwoPr6n9CiEpsCjhfdSpjJzEgcYF2gF9N1XiTmOWdWg=; b=Z6TgBcldp1PS5CkUEOkJI/uRjoDI4JIZpWUU6ZUH778cNIJyJ7aW2Kh5ZF71qf5K9o Yi76M8VnPr5zFwJxZWxjtbep+UZlgmuTjCV9DaX88FSH3vOBjYboc+og/lau60OEyGvo aDohrB8Hr3h4HhD9487KjSNAj5ATCZEGmktBzB+FMj52NxY+1Nz7WUKhzcwoc4q60xoT s4fWaUqN9PnjwQpP2Npnfb7XmwhOgFhApna9UiPIzJVRoI7ajWI5PdDeYt/8BlAoz5HU lEqcvapNhnyi/1iiBaxV0ZUj88+UnK9y9QC8bX0cNluAS+1ptazF0covrM8VzMrDosKz 4NHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NwoPr6n9CiEpsCjhfdSpjJzEgcYF2gF9N1XiTmOWdWg=; b=HfDyMxmmKBB/SIa4C8OGRdt+HbL4Xv+QaLrO8aLLW39EIhuPRmVLQ9tDROJAcxc0WW jpYY91a+iOiOTNsareGKytEfccSEdaiCwIGrKZGy/SMw5ClvwoihoXi4bRsnltl+mOZu meVGidFYzK8UbPYw8j7pSvOXBdok8nE848/L4s4UxznWaJQR9wa00Ibzl5ANlwDqWTMn yB5yUm5aUSuNiQEp9Y5uqgnUBQvz0XfsfFLmYyG1uqHKrG0kfczGDJIl11QCNoPM22mk MrdkZYjSKB4n4WYXxaDGpGA0oxDsIga4CDTJZfdj8zbB/Ts1Lrnbfh+CghV6pzc6Zqoi IsWg== X-Gm-Message-State: AOAM532E1hj4pSdpfJORqAEUDtk1+hLXUzddRY5iIl/l7I40Rzd2jMkj uQyivM8CaCm9lG0HKW3RL/Yzb/KWKKQ= X-Google-Smtp-Source: ABdhPJwO7BokpbA0aITbTTybvrF8wg1MBO5z34cm6RtQMt7i1e8IeqhCFDsWKa0djIcaezECoBfbAA== X-Received: by 2002:a05:6a02:282:b0:381:6565:e080 with SMTP id bk2-20020a056a02028200b003816565e080mr15182208pgb.272.1647791717776; Sun, 20 Mar 2022 08:55:17 -0700 (PDT) Received: from localhost ([14.139.187.71]) by smtp.gmail.com with ESMTPSA id c5-20020a056a00248500b004f6b5ddcc65sm16444880pfv.199.2022.03.20.08.55.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 20 Mar 2022 08:55:17 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8?= =?utf-8?q?rgensen?= , Jesper Dangaard Brouer Subject: [PATCH bpf-next v3 01/13] bpf: Make btf_find_field more generic Date: Sun, 20 Mar 2022 21:24:58 +0530 Message-Id: <20220320155510.671497-2-memxor@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220320155510.671497-1-memxor@gmail.com> References: <20220320155510.671497-1-memxor@gmail.com> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=6265; h=from:subject; bh=5CZILHsi0H8bmXTq6JsyQsOEXM6zpEO+nsoKlQ4MfiY=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBiN00xKPRcbXeBKVx9ex+7ev9ty9+fuOscRkVxjpXl 0bau3X+JAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCYjdNMQAKCRBM4MiGSL8RykcvD/ 0bxqmrZtE/einKi4GKUsfA4O13qPIr2R9dZO6BdG5JemXW9rhfhkEvltEG1u2ZFHtJzMT92qVBrCYj nyoyQny2ZH3lk+OyAHWADmpFBdTxOfESJ01iHa4AoCFjp/svhjykTGaOqSaBPmZV3krpz+JdzjXs4l RznhSw+jB0CYazyIfzYg9ITsuKnS1HeYXnkjdbz019NcJM3mnNPcLr7yTkcgyyolNf4WQfDtm1+rUI Do+lntJW/wgZC3xiTGpgqNEsiXmoE9e6CvVFC5wYo1lF1N7vaPUZZPaku4N8N5+NR2vZqZiOtOirbO waJ5ohbCnLAhV33r9490k3yoS4RHWmvgXVvpD8GpXWebRa4OwY3TKloEOGq5Psv07a8kUbqDenTpSK XLNEPXpN5OpKsbPxUJgRMtlZRxVjn6CG6ZY+uOsWDIuzFcaI6MyhP7j9pUxcJ65ia1G0Q4kxV523ST dFZgeSgWnZlYqRBD2j2IFhIq19jNPM7c6yaOnGBwlySQ4pnYKlDVf1hvM/6JBQ5b3ImVrWZEFzhrGI f+UG3vddj8VLvwpawhKoUDbYpcLtoOqnepDV1Z8vkNDZ2XaIwGI3GEaxa1kcX/bVBst4Yg7JO49r2v FpflkbCzihKzvQuvHhmTzaW8tf8r2ESXzcVrhFdYWd3U7+kpqxPXoDYD+P9Q== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Next commit's field type will not be struct, but pointer, and it will not be limited to one offset, but multiple ones. Make existing btf_find_struct_field and btf_find_datasec_var functions amenable to use for finding BTF ID pointers in map value, by taking a moving spin_lock and timer specific checks into their own function. The alignment, and name are checked before the function is called, so it is the last point where we can skip field or return an error before the next loop iteration happens. The name parameter is now optional, and only checked if it is not NULL. The size must be checked in the function, because in case of PTR it will instead point to the underlying BTF ID it is pointing to (or modifiers), so the check becomes wrong to do outside of function, and the base type has to be obtained by removing modifiers. Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/btf.c | 129 +++++++++++++++++++++++++++++++++++------------ 1 file changed, 96 insertions(+), 33 deletions(-) diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 6d9e711cb5d4..9e17af936a7a 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -3161,71 +3161,126 @@ static void btf_struct_log(struct btf_verifier_env *env, btf_verifier_log(env, "size=%u vlen=%u", t->size, btf_type_vlen(t)); } +enum { + BTF_FIELD_SPIN_LOCK, + BTF_FIELD_TIMER, +}; + +struct btf_field_info { + u32 off; +}; + +static int btf_find_field_struct(const struct btf *btf, const struct btf_type *t, + u32 off, int sz, struct btf_field_info *info) +{ + if (!__btf_type_is_struct(t)) + return 0; + if (t->size != sz) + return 0; + if (info->off != -ENOENT) + /* only one such field is allowed */ + return -E2BIG; + info->off = off; + return 0; +} + static int btf_find_struct_field(const struct btf *btf, const struct btf_type *t, - const char *name, int sz, int align) + const char *name, int sz, int align, int field_type, + struct btf_field_info *info) { const struct btf_member *member; - u32 i, off = -ENOENT; + u32 i, off; + int ret; for_each_member(i, t, member) { const struct btf_type *member_type = btf_type_by_id(btf, member->type); - if (!__btf_type_is_struct(member_type)) - continue; - if (member_type->size != sz) - continue; - if (strcmp(__btf_name_by_offset(btf, member_type->name_off), name)) - continue; - if (off != -ENOENT) - /* only one such field is allowed */ - return -E2BIG; + off = __btf_member_bit_offset(t, member); + + if (name && strcmp(__btf_name_by_offset(btf, member_type->name_off), name)) + continue; if (off % 8) /* valid C code cannot generate such BTF */ return -EINVAL; off /= 8; if (off % align) return -EINVAL; + + switch (field_type) { + case BTF_FIELD_SPIN_LOCK: + case BTF_FIELD_TIMER: + ret = btf_find_field_struct(btf, member_type, off, sz, info); + if (ret < 0) + return ret; + break; + default: + return -EFAULT; + } } - return off; + return 0; } static int btf_find_datasec_var(const struct btf *btf, const struct btf_type *t, - const char *name, int sz, int align) + const char *name, int sz, int align, int field_type, + struct btf_field_info *info) { const struct btf_var_secinfo *vsi; - u32 i, off = -ENOENT; + u32 i, off; + int ret; for_each_vsi(i, t, vsi) { const struct btf_type *var = btf_type_by_id(btf, vsi->type); const struct btf_type *var_type = btf_type_by_id(btf, var->type); - if (!__btf_type_is_struct(var_type)) - continue; - if (var_type->size != sz) + off = vsi->offset; + + if (name && strcmp(__btf_name_by_offset(btf, var_type->name_off), name)) continue; if (vsi->size != sz) continue; - if (strcmp(__btf_name_by_offset(btf, var_type->name_off), name)) - continue; - if (off != -ENOENT) - /* only one such field is allowed */ - return -E2BIG; - off = vsi->offset; if (off % align) return -EINVAL; + + switch (field_type) { + case BTF_FIELD_SPIN_LOCK: + case BTF_FIELD_TIMER: + ret = btf_find_field_struct(btf, var_type, off, sz, info); + if (ret < 0) + return ret; + break; + default: + return -EFAULT; + } } - return off; + return 0; } static int btf_find_field(const struct btf *btf, const struct btf_type *t, - const char *name, int sz, int align) + int field_type, struct btf_field_info *info) { + const char *name; + int sz, align; + + switch (field_type) { + case BTF_FIELD_SPIN_LOCK: + name = "bpf_spin_lock"; + sz = sizeof(struct bpf_spin_lock); + align = __alignof__(struct bpf_spin_lock); + break; + case BTF_FIELD_TIMER: + name = "bpf_timer"; + sz = sizeof(struct bpf_timer); + align = __alignof__(struct bpf_timer); + break; + default: + return -EFAULT; + } if (__btf_type_is_struct(t)) - return btf_find_struct_field(btf, t, name, sz, align); + return btf_find_struct_field(btf, t, name, sz, align, field_type, info); else if (btf_type_is_datasec(t)) - return btf_find_datasec_var(btf, t, name, sz, align); + return btf_find_datasec_var(btf, t, name, sz, align, field_type, info); return -EINVAL; } @@ -3235,16 +3290,24 @@ static int btf_find_field(const struct btf *btf, const struct btf_type *t, */ int btf_find_spin_lock(const struct btf *btf, const struct btf_type *t) { - return btf_find_field(btf, t, "bpf_spin_lock", - sizeof(struct bpf_spin_lock), - __alignof__(struct bpf_spin_lock)); + struct btf_field_info info = { .off = -ENOENT }; + int ret; + + ret = btf_find_field(btf, t, BTF_FIELD_SPIN_LOCK, &info); + if (ret < 0) + return ret; + return info.off; } int btf_find_timer(const struct btf *btf, const struct btf_type *t) { - return btf_find_field(btf, t, "bpf_timer", - sizeof(struct bpf_timer), - __alignof__(struct bpf_timer)); + struct btf_field_info info = { .off = -ENOENT }; + int ret; + + ret = btf_find_field(btf, t, BTF_FIELD_TIMER, &info); + if (ret < 0) + return ret; + return info.off; } static void __btf_struct_show(const struct btf *btf, const struct btf_type *t, From patchwork Sun Mar 20 15:54:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 12786552 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4240FC433FE for ; Sun, 20 Mar 2022 15:55:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242460AbiCTP4q (ORCPT ); Sun, 20 Mar 2022 11:56:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48194 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241132AbiCTP4p (ORCPT ); Sun, 20 Mar 2022 11:56:45 -0400 Received: from mail-pl1-x644.google.com (mail-pl1-x644.google.com [IPv6:2607:f8b0:4864:20::644]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6E18A54188 for ; Sun, 20 Mar 2022 08:55:22 -0700 (PDT) Received: by mail-pl1-x644.google.com with SMTP id h5so10794740plf.7 for ; Sun, 20 Mar 2022 08:55:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=dc0XA9cd0KjRJbm53uQjS1eODXyKwY8M2o8ADmLlRbQ=; b=YUrDqA+WgfKOlkX4QcSGLldGBHO0OaeppL3IEfDuAO1MX/9lxQ7O/R7QNf2DYi9H7K uj4TPZ1TKgRciN8MfhGZnF+1G+K8cQY0aA2/gI8l7v+pJ5FR/7Zinwdf3w4AinAN7U7D gC1e1Q7QuPx+dFlE4MCzyJ8/VoCpVlWQAgll1mmhcS3ilxIooL/WR9t62RDcGW6uuelk atJf5BL+K81mSUbhbY+gZFROcXfbNL6fnvSDQSNzka3MLiR81hIHy9mB1WCNmSgS+L2p Krx1dbqhujDT6+hXXWarV7Ste7QT183xCTP2w4h0886VOCyTX0/yOLr5nS4FLfgZN0pk os4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=dc0XA9cd0KjRJbm53uQjS1eODXyKwY8M2o8ADmLlRbQ=; b=UocM76f6ovI6WEvdTRTV0RJYh2PesqOa8t8Wd/BWFvUTWbezCvVpo2mYn+9P1Vi0jS 4AdGkIoWg9pnPttImT2AOGS4w+4mo91N0DRw7pp/WlzFEjY2UsCf6W13iL0WDLNepgL3 +gFfhINspK18Vwc7RmaNBwxK0lKgJvSMOAlglXUDjYI75eGt1icNZyyff1uRbpbkS4WF VRrJDySkOH4thmgGIgBQqomnkM49q9xaeB9cC6Wy9uN9k+VcFU4LPQ+Ze5s0CQlsPLif nrf/f/wECP+cIFK7/yUYz6XNc3Ch1Fr7Ky0oh/aUlfQSe3eI15RQ/BhQMamsxLykM1+m rxRQ== X-Gm-Message-State: AOAM533uBH2VmuY6/er/9A4ZkTwfLnd+LJ+p/PT33lFOcYsar2Y5pyB2 VgNjVLPY9aUlcF+bD4ZFHWRIEA9dt5M= X-Google-Smtp-Source: ABdhPJwCZ1g406KzDogl56XmSUR4buLEwvnAVXu9mf9gRDMgVqZTJCxH+fDklvV0UmblJMAHRz5rwg== X-Received: by 2002:a17:90a:aa8c:b0:1bf:5273:ba28 with SMTP id l12-20020a17090aaa8c00b001bf5273ba28mr21724366pjq.226.1647791721564; Sun, 20 Mar 2022 08:55:21 -0700 (PDT) Received: from localhost ([14.139.187.71]) by smtp.gmail.com with ESMTPSA id f16-20020a056a00239000b004fa7103e13csm8870647pfc.41.2022.03.20.08.55.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 20 Mar 2022 08:55:21 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8?= =?utf-8?q?rgensen?= , Jesper Dangaard Brouer Subject: [PATCH bpf-next v3 02/13] bpf: Move check_ptr_off_reg before check_map_access Date: Sun, 20 Mar 2022 21:24:59 +0530 Message-Id: <20220320155510.671497-3-memxor@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220320155510.671497-1-memxor@gmail.com> References: <20220320155510.671497-1-memxor@gmail.com> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3314; h=from:subject; bh=QbQlORCSFeRYUc5m/Gowp55NAeCIJOQQJnuoZODwvvU=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBiN00xtP4ESyYu1uhjwOS+LsewrKAjJPu3OEtIjf7S os4g65WJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCYjdNMQAKCRBM4MiGSL8Ryo1/D/ 96QrwEQTQSiTi/FE9FLmDcrXdwA/MMDLWBLWEB85NX4Z4NFMI1BI0qEM6Av7emxCnIjI1SkFkS45qx rU6M9TGDAZAij5SBbcrSQ8p1OSkOli/RZwgUPgsvIdUKtHOOguKVIXh06xBM1LOJUwAypYwv5teL+R I1kqjhlu5KUdg+geMsFdS+PF4NdWWAZlZRBID1i02Jm35oWsZwwnP4mOnAMI5TM39GoEf3Q/usC08g o/0CLVWKLale76+lYfk19a/6IRtuvlF+5UqV6lhPaVS+D2COUIMWyRGw0E2fNXHPPq3pmBXq0v5sN9 52pAAx3G3SirXcebPHJo8pTAvBZKVLXDZ4txqVOnA9+AEJ0mAXlu1ibZBA+eO1mtUSGwib1DGddWAr r74e4fZwPYeto0mgWXRsgbtYt+PWBs1TC1QKOuRm0HMmjjBnTBVnt8/aR/mGBU1vmTkzX8N1fHQIuC kWyoYB96IlQcVNV/z+wbNX+OlGB9q8Fqrh2SadBnhl6KaMpM5WBtxRoj8uXgkhzM7m9K/69vYaD0Gf L5B76RKoNgwulS8dyvlDxUyB7FqDwRNLfYTWBE9IksJSs/hkh/gvoTfqX/uQi5sr/W5JDlDBPkmR4T MSMBU3VmCUKRVgnjClyHQ99FLxhiOy9ATE0SkwESFz9U5a6E/bAKprN2Iw6Q== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Some functions in next patch want to use this function, and those functions will be called by check_map_access, hence move it before check_map_access. Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 76 +++++++++++++++++++++---------------------- 1 file changed, 38 insertions(+), 38 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 0287176bfe9a..4ce9a528fb63 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -3469,6 +3469,44 @@ static int check_mem_region_access(struct bpf_verifier_env *env, u32 regno, return 0; } +static int __check_ptr_off_reg(struct bpf_verifier_env *env, + const struct bpf_reg_state *reg, int regno, + bool fixed_off_ok) +{ + /* Access to this pointer-typed register or passing it to a helper + * is only allowed in its original, unmodified form. + */ + + if (reg->off < 0) { + verbose(env, "negative offset %s ptr R%d off=%d disallowed\n", + reg_type_str(env, reg->type), regno, reg->off); + return -EACCES; + } + + if (!fixed_off_ok && reg->off) { + verbose(env, "dereference of modified %s ptr R%d off=%d disallowed\n", + reg_type_str(env, reg->type), regno, reg->off); + return -EACCES; + } + + if (!tnum_is_const(reg->var_off) || reg->var_off.value) { + char tn_buf[48]; + + tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off); + verbose(env, "variable %s access var_off=%s disallowed\n", + reg_type_str(env, reg->type), tn_buf); + return -EACCES; + } + + return 0; +} + +int check_ptr_off_reg(struct bpf_verifier_env *env, + const struct bpf_reg_state *reg, int regno) +{ + return __check_ptr_off_reg(env, reg, regno, false); +} + /* check read/write into a map element with possible variable offset */ static int check_map_access(struct bpf_verifier_env *env, u32 regno, int off, int size, bool zero_size_allowed) @@ -3980,44 +4018,6 @@ static int get_callee_stack_depth(struct bpf_verifier_env *env, } #endif -static int __check_ptr_off_reg(struct bpf_verifier_env *env, - const struct bpf_reg_state *reg, int regno, - bool fixed_off_ok) -{ - /* Access to this pointer-typed register or passing it to a helper - * is only allowed in its original, unmodified form. - */ - - if (reg->off < 0) { - verbose(env, "negative offset %s ptr R%d off=%d disallowed\n", - reg_type_str(env, reg->type), regno, reg->off); - return -EACCES; - } - - if (!fixed_off_ok && reg->off) { - verbose(env, "dereference of modified %s ptr R%d off=%d disallowed\n", - reg_type_str(env, reg->type), regno, reg->off); - return -EACCES; - } - - if (!tnum_is_const(reg->var_off) || reg->var_off.value) { - char tn_buf[48]; - - tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off); - verbose(env, "variable %s access var_off=%s disallowed\n", - reg_type_str(env, reg->type), tn_buf); - return -EACCES; - } - - return 0; -} - -int check_ptr_off_reg(struct bpf_verifier_env *env, - const struct bpf_reg_state *reg, int regno) -{ - return __check_ptr_off_reg(env, reg, regno, false); -} - static int __check_buffer_access(struct bpf_verifier_env *env, const char *buf_info, const struct bpf_reg_state *reg, From patchwork Sun Mar 20 15:55:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 12786553 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E87FC433EF for ; Sun, 20 Mar 2022 15:55:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241132AbiCTP4w (ORCPT ); Sun, 20 Mar 2022 11:56:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48540 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243814AbiCTP4v (ORCPT ); Sun, 20 Mar 2022 11:56:51 -0400 Received: from mail-pg1-x542.google.com (mail-pg1-x542.google.com [IPv6:2607:f8b0:4864:20::542]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 07B5754194 for ; Sun, 20 Mar 2022 08:55:27 -0700 (PDT) Received: by mail-pg1-x542.google.com with SMTP id c2so8649565pga.10 for ; Sun, 20 Mar 2022 08:55:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fmPeyZCPlKlbVwkubovKQbyar3AnXQAgnBLhtoi4kOI=; b=hE0OcRieVCob2blzInRJfeqB/oG7y8BLsYUEnfrn77OXhaMW3fS84HjsEiMoPaSNiH p+jmqu6buL2RydPvvhFOG0WW07jduhQ5bopbNjwf5+9qjBj2ABrJd/1g6ED3YfI5eTSv 8p8IVyU/gHCOZIJfef8nNhqLEbKlUawvbSBhR6i22tjhxh3oTk3ck+C4ZfqkOY2cn5CM GlMPssDYJ4bQ1Itu/NzZrklYeG/HsWqVnfqrTOoUeINuvrarfnl/WctItuHz4C4chbWf 7B2o8C3mG/FoD1t7lGcItatcAJrOOJ/ZPMH6SXnqy1jM8+wjNpL+nn/aUBtT3yUhlnId imEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fmPeyZCPlKlbVwkubovKQbyar3AnXQAgnBLhtoi4kOI=; b=25NkvxzinHPLfzv2SK6hHaJN8X4BcwX0VkU5rFI6br7HfzK3F2DwrJTlUbppwOP0Gb 7aqCRKW3sbw0UUeVu68cFQU/UMLV2oEtiRSDjT9nZHuFEc+5B1r7CVhde61H7FV0W4Ew 7d695jFYbVV9fQI3HA3sL8alJYBGosuphHofpK8ZZon22SXU54qj3GgqgST1cmocExYU nW1we6X732haP8TM48ILlKwo2zSiwtv7thqyNkSfhOYPdlXHFIeHEd1Dhr6ueWEdJTis 9eJG+3k4+6GuTHRdkPMBEhF0rABHW1ZrITftfNJdJf/EXppgHg6te8HKo5yoHM8VDssk 7bFQ== X-Gm-Message-State: AOAM531ySjnYJLneXkP1nUUMPvNOzShcgHnIG3zDN4bNdiQ8WYxnnbqr CUUs55ndu4DTKWS0CHsgYTk/AXQvXZY= X-Google-Smtp-Source: ABdhPJxRmNMmfbyya4teucQXJgV0HKuT3tocVPzQBQFFVNBBolIta3QTcf97nVgRS1cZKLsAPgjmqQ== X-Received: by 2002:a05:6a00:3387:b0:4fa:9063:dfed with SMTP id cm7-20020a056a00338700b004fa9063dfedmr4289207pfb.15.1647791726025; Sun, 20 Mar 2022 08:55:26 -0700 (PDT) Received: from localhost ([14.139.187.71]) by smtp.gmail.com with ESMTPSA id e16-20020aa78c50000000b004f76c255e92sm15620422pfd.101.2022.03.20.08.55.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 20 Mar 2022 08:55:25 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8?= =?utf-8?q?rgensen?= , Jesper Dangaard Brouer Subject: [PATCH bpf-next v3 03/13] bpf: Allow storing unreferenced kptr in map Date: Sun, 20 Mar 2022 21:25:00 +0530 Message-Id: <20220320155510.671497-4-memxor@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220320155510.671497-1-memxor@gmail.com> References: <20220320155510.671497-1-memxor@gmail.com> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=25220; h=from:subject; bh=fqwEr3t1IR90yKl1BCopBN/9z+ETNQ1PpEWTHd1ix8o=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBiN00xwH9bfm3NcRZnvLjveZiAC3S4oXWdC5i8bZcH jrpDJpaJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCYjdNMQAKCRBM4MiGSL8RytiyD/ 9cfjCdjIs0vukN+2SUxm1NZg59BtcEW/Xvuy8OluBNSYez1KFHkRs+ky8ZcKWH2odD+Jv54G9zFVBh wVX48tOwsWyd6q/5kZJ0OP022TVl0iuMkGLAOa70LPoTdBmV8fVyEdQtzD4eIVLasKEYW7tvSo6wk2 VOmUFHatiu9o/ooGOOOpW+7jIJHyS7cEBB8WzzOsr+HFPIDKn1uVidU/nByfYxx7CmNegFoh5X1Uel VL7kLij6ngZYemIjtc3gI0/SVin0ptGm2m3SG87s56yNdcd92GnpCaIrTWVs+hAjDAd7Y/EOJBM1J/ eHV2EoUGeo1cU9ykOCUv1fetV0aZarKlhhjpvY9H9dREtq4h3o5wnld8SD7vUJNXJmFPsbpTv+v8Va VqLrZfOL3KChxVwj65NpTQ/ziGgMEQ4ftQJEEbpGgH3TIfkwxfhewUcmcl7JlrZ6qjUMruFGh9nzAG 2MX28mk7Y/YmXfqWvTahndL1jfbMZzC7Q5ktqbQ7ar3l2BwXcQGF5EmOE4+3fQwi3rauG2CZRe9fcW kH06Y6i9men2KXMAS9+LMB91t20pN+G8eqeYbA4vpoelXGVWp+fDXM4LhfGCWm2l9yHQK2sGKemknJ j4SwRPBjOgv6WmVWfknXsvgcaNpvgDswlnMFBwEusLGVPP0icuE/BCl+WNSA== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net This commit introduces a new pointer type 'kptr' which can be embedded in a map value as holds a PTR_TO_BTF_ID stored by a BPF program during its invocation. Storing to such a kptr, BPF program's PTR_TO_BTF_ID register must have the same type as in the map value's BTF, and loading a kptr marks the destination register as PTR_TO_BTF_ID with the correct kernel BTF and BTF ID. Such kptr are unreferenced, i.e. by the time another invocation of the BPF program loads this pointer, the object which the pointer points to may not longer exist. Since PTR_TO_BTF_ID loads (using BPF_LDX) are patched to PROBE_MEM loads by the verifier, it would safe to allow user to still access such invalid pointer, but passing such pointers into BPF helpers and kfuncs should not be permitted. A future patch in this series will close this gap. The flexibility offered by allowing programs to dereference such invalid pointers while being safe at runtime frees the verifier from doing complex lifetime tracking. As long as the user may ensure that the object remains valid, it can ensure data read by it from the kernel object is valid. The user indicates that a certain pointer must be treated as kptr capable of accepting stores of PTR_TO_BTF_ID of a certain type, by using a BTF type tag 'kptr' on the pointed to type of the pointer. Then, this information is recorded in the object BTF which will be passed into the kernel by way of map's BTF information. The name and kind from the map value BTF is used to look up the in-kernel type, and the actual BTF and BTF ID is recorded in the map struct in a new kptr_off_tab member. For now, only storing pointers to structs is permitted. An example of this specification is shown below: #define __kptr __attribute__((btf_type_tag("kptr"))) struct map_value { ... struct task_struct __kptr *task; ... }; Then, in a BPF program, user may store PTR_TO_BTF_ID with the type task_struct into the map, and then load it later. Note that the destination register is marked PTR_TO_BTF_ID_OR_NULL, as the verifier cannot know whether the value is NULL or not statically, it must treat all potential loads at that map value offset as loading a possibly NULL pointer. Only BPF_LDX, BPF_STX, and BPF_ST with insn->imm = 0 (to denote NULL) are allowed instructions that can access such a pointer. On BPF_LDX, the destination register is updated to be a PTR_TO_BTF_ID, and on BPF_STX, it is checked whether the source register type is a PTR_TO_BTF_ID with same BTF type as specified in the map BTF. The access size must always be BPF_DW. For the map in map support, the kptr_off_tab for outer map is copied from the inner map's kptr_off_tab. It was chosen to do a deep copy instead of introducing a refcount to kptr_off_tab, because the copy only needs to be done when paramterizing using inner_map_fd in the map in map case, hence would be unnecessary for all other users. It is not permitted to use MAP_FREEZE command and mmap for BPF map having kptr, similar to the bpf_timer case. Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/bpf.h | 29 +++++++- include/linux/btf.h | 2 + kernel/bpf/btf.c | 161 ++++++++++++++++++++++++++++++++++------ kernel/bpf/map_in_map.c | 5 +- kernel/bpf/syscall.c | 112 +++++++++++++++++++++++++++- kernel/bpf/verifier.c | 120 ++++++++++++++++++++++++++++++ 6 files changed, 401 insertions(+), 28 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 88449fbbe063..f35920d279dd 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -155,6 +155,22 @@ struct bpf_map_ops { const struct bpf_iter_seq_info *iter_seq_info; }; +enum { + /* Support at most 8 pointers in a BPF map value */ + BPF_MAP_VALUE_OFF_MAX = 8, +}; + +struct bpf_map_value_off_desc { + u32 offset; + u32 btf_id; + struct btf *btf; +}; + +struct bpf_map_value_off { + u32 nr_off; + struct bpf_map_value_off_desc off[]; +}; + struct bpf_map { /* The first two cachelines with read-mostly members of which some * are also accessed in fast-path (e.g. ops, max_entries). @@ -171,6 +187,7 @@ struct bpf_map { u64 map_extra; /* any per-map-type extra fields */ u32 map_flags; int spin_lock_off; /* >=0 valid offset, <0 error */ + struct bpf_map_value_off *kptr_off_tab; int timer_off; /* >=0 valid offset, <0 error */ u32 id; int numa_node; @@ -184,7 +201,7 @@ struct bpf_map { char name[BPF_OBJ_NAME_LEN]; bool bypass_spec_v1; bool frozen; /* write-once; write-protected by freeze_mutex */ - /* 14 bytes hole */ + /* 6 bytes hole */ /* The 3rd and 4th cacheline with misc members to avoid false sharing * particularly with refcounting. @@ -217,6 +234,11 @@ static inline bool map_value_has_timer(const struct bpf_map *map) return map->timer_off >= 0; } +static inline bool map_value_has_kptr(const struct bpf_map *map) +{ + return !IS_ERR_OR_NULL(map->kptr_off_tab); +} + static inline void check_and_init_map_value(struct bpf_map *map, void *dst) { if (unlikely(map_value_has_spin_lock(map))) @@ -1497,6 +1519,11 @@ void bpf_prog_put(struct bpf_prog *prog); void bpf_prog_free_id(struct bpf_prog *prog, bool do_idr_lock); void bpf_map_free_id(struct bpf_map *map, bool do_idr_lock); +struct bpf_map_value_off_desc *bpf_map_kptr_off_contains(struct bpf_map *map, u32 offset); +void bpf_map_free_kptr_off_tab(struct bpf_map *map); +struct bpf_map_value_off *bpf_map_copy_kptr_off_tab(const struct bpf_map *map); +bool bpf_map_equal_kptr_off_tab(const struct bpf_map *map_a, const struct bpf_map *map_b); + struct bpf_map *bpf_map_get(u32 ufd); struct bpf_map *bpf_map_get_with_uref(u32 ufd); struct bpf_map *__bpf_map_get(struct fd f); diff --git a/include/linux/btf.h b/include/linux/btf.h index 36bc09b8e890..5b578dc81c04 100644 --- a/include/linux/btf.h +++ b/include/linux/btf.h @@ -123,6 +123,8 @@ bool btf_member_is_reg_int(const struct btf *btf, const struct btf_type *s, u32 expected_offset, u32 expected_size); int btf_find_spin_lock(const struct btf *btf, const struct btf_type *t); int btf_find_timer(const struct btf *btf, const struct btf_type *t); +struct bpf_map_value_off *btf_find_kptr(const struct btf *btf, + const struct btf_type *t); bool btf_type_is_void(const struct btf_type *t); s32 btf_find_by_name_kind(const struct btf *btf, const char *name, u8 kind); const struct btf_type *btf_type_skip_modifiers(const struct btf *btf, diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 9e17af936a7a..92afbec0a887 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -3164,9 +3164,16 @@ static void btf_struct_log(struct btf_verifier_env *env, enum { BTF_FIELD_SPIN_LOCK, BTF_FIELD_TIMER, + BTF_FIELD_KPTR, +}; + +enum { + BTF_FIELD_IGNORE = 0, + BTF_FIELD_FOUND = 1, }; struct btf_field_info { + const struct btf_type *type; u32 off; }; @@ -3174,23 +3181,48 @@ static int btf_find_field_struct(const struct btf *btf, const struct btf_type *t u32 off, int sz, struct btf_field_info *info) { if (!__btf_type_is_struct(t)) - return 0; + return BTF_FIELD_IGNORE; if (t->size != sz) - return 0; - if (info->off != -ENOENT) - /* only one such field is allowed */ - return -E2BIG; + return BTF_FIELD_IGNORE; info->off = off; - return 0; + return BTF_FIELD_FOUND; +} + +static int btf_find_field_kptr(const struct btf *btf, const struct btf_type *t, + u32 off, int sz, struct btf_field_info *info) +{ + /* For PTR, sz is always == 8 */ + if (!btf_type_is_ptr(t)) + return BTF_FIELD_IGNORE; + t = btf_type_by_id(btf, t->type); + + if (!btf_type_is_type_tag(t)) + return BTF_FIELD_IGNORE; + /* Reject extra tags */ + if (btf_type_is_type_tag(btf_type_by_id(btf, t->type))) + return -EINVAL; + if (strcmp("kptr", __btf_name_by_offset(btf, t->name_off))) + return -EINVAL; + + /* Get the base type */ + if (btf_type_is_modifier(t)) + t = btf_type_skip_modifiers(btf, t->type, NULL); + /* Only pointer to struct is allowed */ + if (!__btf_type_is_struct(t)) + return -EINVAL; + + info->type = t; + info->off = off; + return BTF_FIELD_FOUND; } static int btf_find_struct_field(const struct btf *btf, const struct btf_type *t, const char *name, int sz, int align, int field_type, - struct btf_field_info *info) + struct btf_field_info *info, int info_cnt) { const struct btf_member *member; + int ret, idx = 0; u32 i, off; - int ret; for_each_member(i, t, member) { const struct btf_type *member_type = btf_type_by_id(btf, @@ -3210,24 +3242,35 @@ static int btf_find_struct_field(const struct btf *btf, const struct btf_type *t switch (field_type) { case BTF_FIELD_SPIN_LOCK: case BTF_FIELD_TIMER: - ret = btf_find_field_struct(btf, member_type, off, sz, info); + ret = btf_find_field_struct(btf, member_type, off, sz, &info[idx]); + if (ret < 0) + return ret; + break; + case BTF_FIELD_KPTR: + ret = btf_find_field_kptr(btf, member_type, off, sz, &info[idx]); if (ret < 0) return ret; break; default: return -EFAULT; } + + if (ret == BTF_FIELD_FOUND && idx >= info_cnt) + return -E2BIG; + else if (ret == BTF_FIELD_IGNORE) + continue; + ++idx; } - return 0; + return idx; } static int btf_find_datasec_var(const struct btf *btf, const struct btf_type *t, const char *name, int sz, int align, int field_type, - struct btf_field_info *info) + struct btf_field_info *info, int info_cnt) { const struct btf_var_secinfo *vsi; + int ret, idx = 0; u32 i, off; - int ret; for_each_vsi(i, t, vsi) { const struct btf_type *var = btf_type_by_id(btf, vsi->type); @@ -3245,19 +3288,30 @@ static int btf_find_datasec_var(const struct btf *btf, const struct btf_type *t, switch (field_type) { case BTF_FIELD_SPIN_LOCK: case BTF_FIELD_TIMER: - ret = btf_find_field_struct(btf, var_type, off, sz, info); + ret = btf_find_field_struct(btf, var_type, off, sz, &info[idx]); + if (ret < 0) + return ret; + break; + case BTF_FIELD_KPTR: + ret = btf_find_field_kptr(btf, var_type, off, sz, &info[idx]); if (ret < 0) return ret; break; default: return -EFAULT; } + + if (ret == BTF_FIELD_FOUND && idx >= info_cnt) + return -E2BIG; + if (ret == BTF_FIELD_IGNORE) + continue; + ++idx; } - return 0; + return idx; } static int btf_find_field(const struct btf *btf, const struct btf_type *t, - int field_type, struct btf_field_info *info) + int field_type, struct btf_field_info *info, int info_cnt) { const char *name; int sz, align; @@ -3273,14 +3327,20 @@ static int btf_find_field(const struct btf *btf, const struct btf_type *t, sz = sizeof(struct bpf_timer); align = __alignof__(struct bpf_timer); break; + case BTF_FIELD_KPTR: + name = NULL; + sz = sizeof(u64); + align = __alignof__(u64); + break; default: return -EFAULT; } + /* The maximum allowed fields of a certain type will be info_cnt - 1 */ if (__btf_type_is_struct(t)) - return btf_find_struct_field(btf, t, name, sz, align, field_type, info); + return btf_find_struct_field(btf, t, name, sz, align, field_type, info, info_cnt - 1); else if (btf_type_is_datasec(t)) - return btf_find_datasec_var(btf, t, name, sz, align, field_type, info); + return btf_find_datasec_var(btf, t, name, sz, align, field_type, info, info_cnt - 1); return -EINVAL; } @@ -3290,24 +3350,79 @@ static int btf_find_field(const struct btf *btf, const struct btf_type *t, */ int btf_find_spin_lock(const struct btf *btf, const struct btf_type *t) { - struct btf_field_info info = { .off = -ENOENT }; + /* btf_find_field requires array of size max + 1 */ + struct btf_field_info info_arr[2]; int ret; - ret = btf_find_field(btf, t, BTF_FIELD_SPIN_LOCK, &info); + ret = btf_find_field(btf, t, BTF_FIELD_SPIN_LOCK, info_arr, ARRAY_SIZE(info_arr)); if (ret < 0) return ret; - return info.off; + if (!ret) + return -ENOENT; + return info_arr[0].off; } int btf_find_timer(const struct btf *btf, const struct btf_type *t) { - struct btf_field_info info = { .off = -ENOENT }; + /* btf_find_field requires array of size max + 1 */ + struct btf_field_info info_arr[2]; int ret; - ret = btf_find_field(btf, t, BTF_FIELD_TIMER, &info); + ret = btf_find_field(btf, t, BTF_FIELD_TIMER, info_arr, ARRAY_SIZE(info_arr)); if (ret < 0) return ret; - return info.off; + if (!ret) + return -ENOENT; + return info_arr[0].off; +} + +struct bpf_map_value_off *btf_find_kptr(const struct btf *btf, + const struct btf_type *t) +{ + /* btf_find_field requires array of size max + 1 */ + struct btf_field_info info_arr[BPF_MAP_VALUE_OFF_MAX + 1]; + struct bpf_map_value_off *tab; + int ret, i, nr_off; + + /* Revisit stack usage when bumping BPF_MAP_VALUE_OFF_MAX */ + BUILD_BUG_ON(BPF_MAP_VALUE_OFF_MAX != 8); + + ret = btf_find_field(btf, t, BTF_FIELD_KPTR, info_arr, ARRAY_SIZE(info_arr)); + if (ret < 0) + return ERR_PTR(ret); + if (!ret) + return NULL; + + nr_off = ret; + tab = kzalloc(offsetof(struct bpf_map_value_off, off[nr_off]), GFP_KERNEL | __GFP_NOWARN); + if (!tab) + return ERR_PTR(-ENOMEM); + + tab->nr_off = 0; + for (i = 0; i < nr_off; i++) { + const struct btf_type *t; + struct btf *off_btf; + s32 id; + + t = info_arr[i].type; + id = bpf_find_btf_id(__btf_name_by_offset(btf, t->name_off), BTF_INFO_KIND(t->info), + &off_btf); + if (id < 0) { + ret = id; + goto end; + } + + tab->off[i].offset = info_arr[i].off; + tab->off[i].btf_id = id; + tab->off[i].btf = off_btf; + tab->nr_off = i + 1; + } + return tab; +end: + while (tab->nr_off--) + btf_put(tab->off[tab->nr_off].btf); + kfree(tab); + return ERR_PTR(ret); } static void __btf_struct_show(const struct btf *btf, const struct btf_type *t, diff --git a/kernel/bpf/map_in_map.c b/kernel/bpf/map_in_map.c index 5cd8f5277279..135205d0d560 100644 --- a/kernel/bpf/map_in_map.c +++ b/kernel/bpf/map_in_map.c @@ -52,6 +52,7 @@ struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd) inner_map_meta->max_entries = inner_map->max_entries; inner_map_meta->spin_lock_off = inner_map->spin_lock_off; inner_map_meta->timer_off = inner_map->timer_off; + inner_map_meta->kptr_off_tab = bpf_map_copy_kptr_off_tab(inner_map); if (inner_map->btf) { btf_get(inner_map->btf); inner_map_meta->btf = inner_map->btf; @@ -71,6 +72,7 @@ struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd) void bpf_map_meta_free(struct bpf_map *map_meta) { + bpf_map_free_kptr_off_tab(map_meta); btf_put(map_meta->btf); kfree(map_meta); } @@ -83,7 +85,8 @@ bool bpf_map_meta_equal(const struct bpf_map *meta0, meta0->key_size == meta1->key_size && meta0->value_size == meta1->value_size && meta0->timer_off == meta1->timer_off && - meta0->map_flags == meta1->map_flags; + meta0->map_flags == meta1->map_flags && + bpf_map_equal_kptr_off_tab(meta0, meta1); } void *bpf_map_fd_get_ptr(struct bpf_map *map, diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index cdaa1152436a..5990d6fa97ab 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -6,6 +6,7 @@ #include #include #include +#include #include #include #include @@ -473,12 +474,95 @@ static void bpf_map_release_memcg(struct bpf_map *map) } #endif +static int bpf_map_kptr_off_cmp(const void *a, const void *b) +{ + const struct bpf_map_value_off_desc *off_desc1 = a, *off_desc2 = b; + + if (off_desc1->offset < off_desc2->offset) + return -1; + else if (off_desc1->offset > off_desc2->offset) + return 1; + return 0; +} + +struct bpf_map_value_off_desc *bpf_map_kptr_off_contains(struct bpf_map *map, u32 offset) +{ + /* Since members are iterated in btf_find_field in increasing order, + * offsets appended to kptr_off_tab are in increasing order, so we can + * do bsearch to find exact match. + */ + struct bpf_map_value_off *tab; + + if (!map_value_has_kptr(map)) + return NULL; + tab = map->kptr_off_tab; + return bsearch(&offset, tab->off, tab->nr_off, sizeof(tab->off[0]), bpf_map_kptr_off_cmp); +} + +void bpf_map_free_kptr_off_tab(struct bpf_map *map) +{ + struct bpf_map_value_off *tab = map->kptr_off_tab; + int i; + + if (!map_value_has_kptr(map)) + return; + for (i = 0; i < tab->nr_off; i++) { + struct btf *btf = tab->off[i].btf; + + btf_put(btf); + } + kfree(tab); + map->kptr_off_tab = NULL; +} + +struct bpf_map_value_off *bpf_map_copy_kptr_off_tab(const struct bpf_map *map) +{ + struct bpf_map_value_off *tab = map->kptr_off_tab, *new_tab; + int size, i, ret; + + if (!map_value_has_kptr(map)) + return ERR_PTR(-ENOENT); + /* Do a deep copy of the kptr_off_tab */ + for (i = 0; i < tab->nr_off; i++) + btf_get(tab->off[i].btf); + + size = offsetof(struct bpf_map_value_off, off[tab->nr_off]); + new_tab = kzalloc(size, GFP_KERNEL | __GFP_NOWARN); + if (!new_tab) { + ret = -ENOMEM; + goto end; + } + memcpy(new_tab, tab, size); + return new_tab; +end: + while (i--) + btf_put(tab->off[i].btf); + return ERR_PTR(ret); +} + +bool bpf_map_equal_kptr_off_tab(const struct bpf_map *map_a, const struct bpf_map *map_b) +{ + struct bpf_map_value_off *tab_a = map_a->kptr_off_tab, *tab_b = map_b->kptr_off_tab; + bool a_has_kptr = map_value_has_kptr(map_a), b_has_kptr = map_value_has_kptr(map_b); + int size; + + if (!a_has_kptr && !b_has_kptr) + return true; + if ((a_has_kptr && !b_has_kptr) || (!a_has_kptr && b_has_kptr)) + return false; + if (tab_a->nr_off != tab_b->nr_off) + return false; + size = offsetof(struct bpf_map_value_off, off[tab_a->nr_off]); + return !memcmp(tab_a, tab_b, size); +} + /* called from workqueue */ static void bpf_map_free_deferred(struct work_struct *work) { struct bpf_map *map = container_of(work, struct bpf_map, work); security_bpf_map_free(map); + bpf_map_free_kptr_off_tab(map); bpf_map_release_memcg(map); /* implementation dependent freeing */ map->ops->map_free(map); @@ -640,7 +724,7 @@ static int bpf_map_mmap(struct file *filp, struct vm_area_struct *vma) int err; if (!map->ops->map_mmap || map_value_has_spin_lock(map) || - map_value_has_timer(map)) + map_value_has_timer(map) || map_value_has_kptr(map)) return -ENOTSUPP; if (!(vma->vm_flags & VM_SHARED)) @@ -820,9 +904,31 @@ static int map_check_btf(struct bpf_map *map, const struct btf *btf, return -EOPNOTSUPP; } - if (map->ops->map_check_btf) + map->kptr_off_tab = btf_find_kptr(btf, value_type); + if (map_value_has_kptr(map)) { + if (!bpf_capable()) + return -EPERM; + if (map->map_flags & BPF_F_RDONLY_PROG) { + ret = -EACCES; + goto free_map_tab; + } + if (map->map_type != BPF_MAP_TYPE_HASH && + map->map_type != BPF_MAP_TYPE_LRU_HASH && + map->map_type != BPF_MAP_TYPE_ARRAY) { + ret = -EOPNOTSUPP; + goto free_map_tab; + } + } + + if (map->ops->map_check_btf) { ret = map->ops->map_check_btf(map, btf, key_type, value_type); + if (ret < 0) + goto free_map_tab; + } + return ret; +free_map_tab: + bpf_map_free_kptr_off_tab(map); return ret; } @@ -1639,7 +1745,7 @@ static int map_freeze(const union bpf_attr *attr) return PTR_ERR(map); if (map->map_type == BPF_MAP_TYPE_STRUCT_OPS || - map_value_has_timer(map)) { + map_value_has_timer(map) || map_value_has_kptr(map)) { fdput(f); return -ENOTSUPP; } diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 4ce9a528fb63..744b7362e52e 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -3507,6 +3507,94 @@ int check_ptr_off_reg(struct bpf_verifier_env *env, return __check_ptr_off_reg(env, reg, regno, false); } +static int map_kptr_match_type(struct bpf_verifier_env *env, + struct bpf_map_value_off_desc *off_desc, + struct bpf_reg_state *reg, u32 regno) +{ + const char *targ_name = kernel_type_name(off_desc->btf, off_desc->btf_id); + const char *reg_name = ""; + + if (reg->type != PTR_TO_BTF_ID && reg->type != PTR_TO_BTF_ID_OR_NULL) + goto bad_type; + + if (!btf_is_kernel(reg->btf)) { + verbose(env, "R%d must point to kernel BTF\n", regno); + return -EINVAL; + } + /* We need to verify reg->type and reg->btf, before accessing reg->btf */ + reg_name = kernel_type_name(reg->btf, reg->btf_id); + + if (__check_ptr_off_reg(env, reg, regno, true)) + return -EACCES; + + if (!btf_struct_ids_match(&env->log, reg->btf, reg->btf_id, reg->off, + off_desc->btf, off_desc->btf_id)) + goto bad_type; + return 0; +bad_type: + verbose(env, "invalid kptr access, R%d type=%s%s ", regno, + reg_type_str(env, reg->type), reg_name); + verbose(env, "expected=%s%s\n", reg_type_str(env, PTR_TO_BTF_ID), targ_name); + return -EINVAL; +} + +/* Returns an error, or 0 if ignoring the access, or 1 if register state was + * updated, in which case later updates must be skipped. + */ +static int check_map_kptr_access(struct bpf_verifier_env *env, u32 regno, + int off, int size, int value_regno, + enum bpf_access_type t, int insn_idx) +{ + struct bpf_reg_state *reg = reg_state(env, regno), *val_reg; + struct bpf_insn *insn = &env->prog->insnsi[insn_idx]; + struct bpf_map_value_off_desc *off_desc; + struct bpf_map *map = reg->map_ptr; + int class = BPF_CLASS(insn->code); + + /* Things we already checked for in check_map_access: + * - Reject cases where variable offset may touch BTF ID pointer + * - size of access (must be BPF_DW) + * - off_desc->offset == off + reg->var_off.value + */ + if (!tnum_is_const(reg->var_off)) + return 0; + + off_desc = bpf_map_kptr_off_contains(map, off + reg->var_off.value); + if (!off_desc) + return 0; + + /* Only BPF_[LDX,STX,ST] | BPF_MEM | BPF_DW is supported */ + if (BPF_MODE(insn->code) != BPF_MEM) + goto end; + + if (class == BPF_LDX) { + val_reg = reg_state(env, value_regno); + /* We can simply mark the value_regno receiving the pointer + * value from map as PTR_TO_BTF_ID, with the correct type. + */ + mark_btf_ld_reg(env, cur_regs(env), value_regno, PTR_TO_BTF_ID, off_desc->btf, + off_desc->btf_id, PTR_MAYBE_NULL); + val_reg->id = ++env->id_gen; + } else if (class == BPF_STX) { + val_reg = reg_state(env, value_regno); + if (!register_is_null(val_reg) && + map_kptr_match_type(env, off_desc, val_reg, value_regno)) + return -EACCES; + } else if (class == BPF_ST) { + if (insn->imm) { + verbose(env, "BPF_ST imm must be 0 when storing to kptr at off=%u\n", + off_desc->offset); + return -EACCES; + } + } else { + goto end; + } + return 1; +end: + verbose(env, "kptr in map can only be accessed using BPF_LDX/BPF_STX/BPF_ST\n"); + return -EACCES; +} + /* check read/write into a map element with possible variable offset */ static int check_map_access(struct bpf_verifier_env *env, u32 regno, int off, int size, bool zero_size_allowed) @@ -3545,6 +3633,32 @@ static int check_map_access(struct bpf_verifier_env *env, u32 regno, return -EACCES; } } + if (map_value_has_kptr(map)) { + struct bpf_map_value_off *tab = map->kptr_off_tab; + int i; + + for (i = 0; i < tab->nr_off; i++) { + u32 p = tab->off[i].offset; + + if (reg->smin_value + off < p + sizeof(u64) && + p < reg->umax_value + off + size) { + if (!tnum_is_const(reg->var_off)) { + verbose(env, "kptr access cannot have variable offset\n"); + return -EACCES; + } + if (p != off + reg->var_off.value) { + verbose(env, "kptr access misaligned expected=%u off=%llu\n", + p, off + reg->var_off.value); + return -EACCES; + } + if (size != bpf_size_to_bytes(BPF_DW)) { + verbose(env, "kptr access size must be BPF_DW\n"); + return -EACCES; + } + break; + } + } + } return err; } @@ -4421,6 +4535,10 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn if (err) return err; err = check_map_access(env, regno, off, size, false); + err = err ?: check_map_kptr_access(env, regno, off, size, value_regno, t, insn_idx); + if (err < 0) + return err; + /* if err == 0, check_map_kptr_access ignored the access */ if (!err && t == BPF_READ && value_regno >= 0) { struct bpf_map *map = reg->map_ptr; @@ -4442,6 +4560,8 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn mark_reg_unknown(env, regs, value_regno); } } + /* clear err == 1 */ + err = err < 0 ? err : 0; } else if (base_type(reg->type) == PTR_TO_MEM) { bool rdonly_mem = type_is_rdonly_mem(reg->type); From patchwork Sun Mar 20 15:55:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 12786554 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C5EDC433EF for ; Sun, 20 Mar 2022 15:55:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243814AbiCTP4z (ORCPT ); Sun, 20 Mar 2022 11:56:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48676 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244429AbiCTP4y (ORCPT ); Sun, 20 Mar 2022 11:56:54 -0400 Received: from mail-pf1-x444.google.com (mail-pf1-x444.google.com [IPv6:2607:f8b0:4864:20::444]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 378BC54194 for ; Sun, 20 Mar 2022 08:55:31 -0700 (PDT) Received: by mail-pf1-x444.google.com with SMTP id s11so13522610pfu.13 for ; Sun, 20 Mar 2022 08:55:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+oy/G68PKrNRew4ZKE6GOJcSvayUTQIzyyfAhHUa3f0=; b=A3w7oMmayb1GgfhYIWQPQ/5CAtSL1PwPR61JebJAsl8f/1grG5X/UYTmAzTL9Mc9Fj +5pkd5kqheZelXgi8vXrbOM8LW81tBloVgiaZIh6j/+tvT2ZRJV1ZWRRaOkSy9m81jp2 W8uMrwau9WeOceL7PZuZnpDy9ZiHlY15oWKa5WtmqK1fleCyJPQKShJsHE8OC6auqm8t KnTfT/RYidReu/OF03vSo7qgeypkRbFAdnZjVhHtPvG9qHDdV9L5Wxfo2T6wABDul9vt bDMN5aInYv9XQY4KXMfHgtSLUnn9Oru20Ty+n//cerHNvrL6dppbDjSw4qzI2evKym7J jQXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+oy/G68PKrNRew4ZKE6GOJcSvayUTQIzyyfAhHUa3f0=; b=AJje+wAL/wbnLYUsBxO4hoWJSCHyo6W5ChH8v2z3TwXNGq4NUj/pCmLo0YxR0VXRoH wvoF/Ulxo0hl8C0swHnop8KGfds7xWxiWdj+b2vYMimfZjVpLGUrf5Qq0bTx3Jy8gyjn +DDWDC4uWZ8ElnzFYCOUVATv/ZJ2lvFpVDcPOn1yVLdXlCZVhupXimf9O4V2KD+uxRUk bcKwixiqJBmohp7XfSrl7qSVVfdXNT8OafMHjnMO1bJVuwHhV9K79vYnlcw8VzL2MYX3 wEDFowVclz6/BnvJ5iKYkCtJkvEJiPSlnAawvKeNgWludrYG1SqGb/joAG9Z66+xUS8c M66g== X-Gm-Message-State: AOAM532tXaArMTCsLodfHZTMnlMRNtmfKV15ULN9V9lEgHYLIMe3aklx BPRXkJ5JM6EagKgd505HZRyf9M0CC80= X-Google-Smtp-Source: ABdhPJzfttmJ7BGGpM6CtXG9njrQoF206lznaYjSAhrCiBhjb+SJWZMAIdbWtJbLycyt2QwbV0Nu7w== X-Received: by 2002:a05:6a00:c93:b0:4f7:c76:921f with SMTP id a19-20020a056a000c9300b004f70c76921fmr19478924pfv.73.1647791730564; Sun, 20 Mar 2022 08:55:30 -0700 (PDT) Received: from localhost ([14.139.187.71]) by smtp.gmail.com with ESMTPSA id d24-20020a637358000000b003823aefde04sm5613621pgn.86.2022.03.20.08.55.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 20 Mar 2022 08:55:30 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8?= =?utf-8?q?rgensen?= , Jesper Dangaard Brouer Subject: [PATCH bpf-next v3 04/13] bpf: Indicate argument that will be released in bpf_func_proto Date: Sun, 20 Mar 2022 21:25:01 +0530 Message-Id: <20220320155510.671497-5-memxor@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220320155510.671497-1-memxor@gmail.com> References: <20220320155510.671497-1-memxor@gmail.com> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5395; h=from:subject; bh=JQxBrJfBnHlT/mhhv+KRR8AAKWTEKqO5dyF4aYXpZ88=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBiN00xTBpeHaT8slJR3mSTCefOFDeg6lABXodJTP+h 36EuxqmJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCYjdNMQAKCRBM4MiGSL8RyhuaD/ 9A1tG+4NWo5jNkjmWN6WRfrXSI2HOoFut8hoBr+OdEClQueQGwIXdVxo9ciwVlceO0jRfLrlA9y+px Ao+P6F4M3L3kt2fr9MAYIQrvwdsAXaHLBeD0NZgArlShAndFIz5Y2i0wyW1CYWAem9FE5lK9ZmwIge s4xqzITjcSu580eE+AafPIrkNLMOIP+oJ1MgOV7yzlAj1WQ4gY6Vlx5HeQPkizughhxP0MosAxsbQq CDzZJEAoC8vIe55ruA/z3RdoXCCSMdgeOLQiAHZmFuqxcYgtnw0RI6QjWWv6HV2hc84BPD5MxAVWki hazdHXfr84SpZw+/vEWGWSYGJYeHP9J1IJEEMoGUFNvWp1QbFwMzY5Bp+CuuJJcoY6JdZkxfNFUYLd Nxe/chRCjzMe8ugNWL/oc2zql5tEt6fapHkkDySm55Gu8PJR8q0bYpZiuISPm8Q1b/mqcvbZ22lzem 5MlZrGadFP4ZvydCh8KtOx9V8Eym3RRB/PIcrddBXmqSwfd/GGuuc2zumz+6Zp6P8ZH4K3+z2RIjbG DHn1qb6+CX9rX5MZbFSQzT1ukcETr3O7luNy8KtbES7Hk8dowBXinfEsjuavzcOsgP/Z27gtNMelu/ kFip6Rap6pqfgm6/naYreaV5v99IHcquaTQi8kykgyyAY2lkLIIXtrMCi1IA== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Add a few fields for each arg (argN_release) that when set to true, tells verifier that for a release function, that argument's register will be the one for which meta.ref_obj_id will be set, and which will then be released using release_reference. To capture the regno, introduce a release_regno field in bpf_call_arg_meta. This would be required in the next patch, where we may either pass NULL or a refcounted pointer as an argument to the release function bpf_kptr_xchg. Just releasing only when meta.ref_obj_id is set is not enough, as there is a case where the type of argument needed matches, but the ref_obj_id is set to 0. Hence, we must enforce that whenever meta.ref_obj_id is zero, the register that is to be released can only be NULL for a release function. Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/bpf.h | 10 ++++++++++ kernel/bpf/ringbuf.c | 2 ++ kernel/bpf/verifier.c | 39 +++++++++++++++++++++++++++++++++------ net/core/filter.c | 1 + 4 files changed, 46 insertions(+), 6 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index f35920d279dd..48ddde854d67 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -487,6 +487,16 @@ struct bpf_func_proto { }; u32 *arg_btf_id[5]; }; + union { + struct { + bool arg1_release; + bool arg2_release; + bool arg3_release; + bool arg4_release; + bool arg5_release; + }; + bool arg_release[5]; + }; int *ret_btf_id; /* return value btf_id */ bool (*allowed)(const struct bpf_prog *prog); }; diff --git a/kernel/bpf/ringbuf.c b/kernel/bpf/ringbuf.c index 710ba9de12ce..f40ce718630e 100644 --- a/kernel/bpf/ringbuf.c +++ b/kernel/bpf/ringbuf.c @@ -405,6 +405,7 @@ const struct bpf_func_proto bpf_ringbuf_submit_proto = { .func = bpf_ringbuf_submit, .ret_type = RET_VOID, .arg1_type = ARG_PTR_TO_ALLOC_MEM, + .arg1_release = true, .arg2_type = ARG_ANYTHING, }; @@ -418,6 +419,7 @@ const struct bpf_func_proto bpf_ringbuf_discard_proto = { .func = bpf_ringbuf_discard, .ret_type = RET_VOID, .arg1_type = ARG_PTR_TO_ALLOC_MEM, + .arg1_release = true, .arg2_type = ARG_ANYTHING, }; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 744b7362e52e..b8cd34607215 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -245,6 +245,7 @@ struct bpf_call_arg_meta { struct bpf_map *map_ptr; bool raw_mode; bool pkt_access; + u8 release_regno; int regno; int access_size; int mem_size; @@ -6101,12 +6102,31 @@ static bool check_btf_id_ok(const struct bpf_func_proto *fn) return true; } -static int check_func_proto(const struct bpf_func_proto *fn, int func_id) +static bool check_release_regno(const struct bpf_func_proto *fn, int func_id, + struct bpf_call_arg_meta *meta) +{ + int i; + + for (i = 0; i < ARRAY_SIZE(fn->arg_release); i++) { + if (fn->arg_release[i]) { + if (!is_release_function(func_id)) + return false; + if (meta->release_regno) + return false; + meta->release_regno = i + 1; + } + } + return !is_release_function(func_id) || meta->release_regno; +} + +static int check_func_proto(const struct bpf_func_proto *fn, int func_id, + struct bpf_call_arg_meta *meta) { return check_raw_mode_ok(fn) && check_arg_pair_ok(fn) && check_btf_id_ok(fn) && - check_refcount_ok(fn, func_id) ? 0 : -EINVAL; + check_refcount_ok(fn, func_id) && + check_release_regno(fn, func_id, meta) ? 0 : -EINVAL; } /* Packet data might have moved, any old PTR_TO_PACKET[_META,_END] @@ -6785,7 +6805,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn memset(&meta, 0, sizeof(meta)); meta.pkt_access = fn->pkt_access; - err = check_func_proto(fn, func_id); + err = check_func_proto(fn, func_id, &meta); if (err) { verbose(env, "kernel subsystem misconfigured func %s#%d\n", func_id_name(func_id), func_id); @@ -6818,8 +6838,17 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn return err; } + regs = cur_regs(env); + if (is_release_function(func_id)) { - err = release_reference(env, meta.ref_obj_id); + err = -EINVAL; + if (meta.ref_obj_id) + err = release_reference(env, meta.ref_obj_id); + /* meta.ref_obj_id can only be 0 if register that is meant to be + * released is NULL, which must be > R0. + */ + else if (meta.release_regno && register_is_null(®s[meta.release_regno])) + err = 0; if (err) { verbose(env, "func %s#%d reference has not been acquired before\n", func_id_name(func_id), func_id); @@ -6827,8 +6856,6 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn } } - regs = cur_regs(env); - switch (func_id) { case BPF_FUNC_tail_call: err = check_reference_leak(env); diff --git a/net/core/filter.c b/net/core/filter.c index 03655f2074ae..17eff4731b06 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -6622,6 +6622,7 @@ static const struct bpf_func_proto bpf_sk_release_proto = { .gpl_only = false, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_BTF_ID_SOCK_COMMON, + .arg1_release = true, }; BPF_CALL_5(bpf_xdp_sk_lookup_udp, struct xdp_buff *, ctx, From patchwork Sun Mar 20 15:55:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 12786555 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C5A6C433F5 for ; Sun, 20 Mar 2022 15:55:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245407AbiCTP5E (ORCPT ); Sun, 20 Mar 2022 11:57:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49234 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244429AbiCTP5D (ORCPT ); Sun, 20 Mar 2022 11:57:03 -0400 Received: from mail-pl1-x644.google.com (mail-pl1-x644.google.com [IPv6:2607:f8b0:4864:20::644]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2C9E054188 for ; Sun, 20 Mar 2022 08:55:38 -0700 (PDT) Received: by mail-pl1-x644.google.com with SMTP id d18so10791755plr.6 for ; Sun, 20 Mar 2022 08:55:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=qhcosv2Yhs3ivJRhHYG6HNFk8rBMBV1CSROMY3Yepw0=; b=c3l48mnTc/lNGdTzF5UUp9HfkxOHhnAVt+6FpAv3zGuG3W82wjX2EUlkH8Nu0MawfK k3CtL17ZCNPy9CZjXl9bIYzl+QVuHQF2Y33ZYZBz9gGxrmHH2TgEMLsyxU9JtAdD/XoL TgQX+3ckd/nm92ZNB1mMs+NnN+V7Tta6fP+6cYf5eVYilibGrSrHFoBlpzgEH9CFDOhU 6HSsY2197a8YSPQMSJ7LFgN+IYvTrmhC04Ri8BO5MFTaO32F1eqLwPFWo4iiSrxMILes gZjnzqObj2IwvYluz16FzDhHAMdAyDAafqHTGAL8fszMZkoxwCKg1KWywuj/+QpPDDGQ BgAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=qhcosv2Yhs3ivJRhHYG6HNFk8rBMBV1CSROMY3Yepw0=; b=KYYiF5mn4lbNGT4qOvD1rzdF9hfbANOpt5/JN23FlSlEdiuCTeVxsIblTM2HscI0iq FqX+qbQvR9PXMVrJG8ifjJvikly+vFCYufiITAt14Vw52nZ+4liqfkvU70ogLf8ntpiv VILU/Nhd9Vs9v7BtVOTh089+f9IFEyKSiQgzufwM3uHP44TKeqjU9lHD1de3gDMGoAIu 1sqn8i1h+qjSs6JOqB7PAkh3DpA7gqezqA55+iJ6L+UFfktIhQ5oMrr5OSv2F/ztoJ+E huPMvwGWKRf5Ar3lkyCHMsVs8/YbybP5I2aS5LRMBPyOVCUXts7+4UIOoxxTX1qYIkUp V60Q== X-Gm-Message-State: AOAM531QyTgAHzQt3GkKjdtzPpms9Gy36hRakB0Kd79enQA6fP1kyD9M hc/kmCP4shyFBmp05d0YCbrJgD0xnE8= X-Google-Smtp-Source: ABdhPJzuyU7Tx+d0kBSi/AR8n9ZNxuu1xE3wR/U6J7zU7z1x6lL1y67OY7Uk3strgDo9iGJCEEdLHQ== X-Received: by 2002:a17:902:e889:b0:151:a56d:eb8f with SMTP id w9-20020a170902e88900b00151a56deb8fmr8694194plg.142.1647791737273; Sun, 20 Mar 2022 08:55:37 -0700 (PDT) Received: from localhost ([14.139.187.71]) by smtp.gmail.com with ESMTPSA id o24-20020a17090a5b1800b001c6aaafa5fbsm7209763pji.24.2022.03.20.08.55.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 20 Mar 2022 08:55:37 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8?= =?utf-8?q?rgensen?= , Jesper Dangaard Brouer Subject: [PATCH bpf-next v3 05/13] bpf: Allow storing referenced kptr in map Date: Sun, 20 Mar 2022 21:25:02 +0530 Message-Id: <20220320155510.671497-6-memxor@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220320155510.671497-1-memxor@gmail.com> References: <20220320155510.671497-1-memxor@gmail.com> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=18542; h=from:subject; bh=ikKDWsO8n1tQZ31MblSQi8BLHzBlR9gwzkSc8pPnxy8=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBiN00ysVXL0rnNPMacyQt7R6X5gQzrTMUGZEGab5tH vhcX+5aJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCYjdNMgAKCRBM4MiGSL8RyjFtEA Cz5kaAwUdtx6Vw7/IL0mEhWymyYeNhVVTl3F0mRNT2pgnP7wgwfLROWIRmUbvMbhqb0CXIGhuzCjBk wDSk8lmzr675vUuBCPzBaRNT+AKocNfw444b2YuRJnJjPEoXSBq3qTrzGCr9zW+gxTuSA6SHKOYRuR /CJn+8mJ4euhArWkRjIEoVBqFrWjah2VCtKSIGH7Dh3ZzapClYiakUN4vakMzjh8fNA2Pcm8rowfKL pT1OnP0L1fDt3jm+VKP+Dg4RM3P/q9Woe7uKGE3iVc1/lkUwa382wstMYSX01c7EsSjusiZW3gv7mF 33hMOVYD7Aj8Sxod29WDN/ga8FNWAPuNpG80Ke9GXjdduqtKYGW0kaVHdXeohCad5a65oUYjMU9GX7 YmJ4P80oy0K2Ko4AOYDmPMLOdzH114fLepR+PP1lpHnaZQOlT0CCjxCKEHIOPEYujtHLz69AzzzudM gIbGbsstbxfLHfpffvhiFDEPPpiMHq02KgW0mZ22bzWpKxlD/p2igkp5KGp0OfwLRM2ly+NRQSTr+M 74FXVv51fojWG0ph4of4tHEKCsvgBQC3gZJ4vvkiM2eU1q3BAKka26Yxd6Dkt6mr3xMCWfb/3WmHZo gVj+nkuZYk9oXlh/JlCsRCou+qZKPQs6HVAnul4jBSkV2xG1wHMd/oBHrRvA== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Extending the code in previous commit, introduce referenced kptr support, which needs to be tagged using 'kptr_ref' tag instead. Unlike unreferenced kptr, referenced kptr have a lot more restrictions. In addition to the type matching, only a newly introduced bpf_kptr_xchg helper is allowed to modify the map value at that offset. This transfers the referenced pointer being stored into the map, releasing the references state for the program, and returning the old value and creating new reference state for the returned pointer. Similar to unreferenced pointer case, return value for this case will also be PTR_TO_BTF_ID_OR_NULL. The reference for the returned pointer must either be eventually released by calling the corresponding release function, otherwise it must be transferred into another map. It is also allowed to call bpf_kptr_xchg with a NULL pointer, to clear the value, and obtain the old value if any. BPF_LDX, BPF_STX, and BPF_ST cannot access referenced kptr. A future commit will permit using BPF_LDX for such pointers, but attempt at making it safe, since the lifetime of object won't be guaranteed. There are valid reasons to enforce the restriction of permitting only bpf_kptr_xchg to operate on referenced kptr. The pointer value must be consistent in face of concurrent modification, and any prior values contained in the map must also be released before a new one is moved into the map. To ensure proper transfer of this ownership, bpf_kptr_xchg returns the old value, which the verifier would require the user to either free or move into another map, and releases the reference held for the pointer being moved in. In the future, direct BPF_XCHG instruction may also be permitted to work like bpf_kptr_xchg helper. Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/bpf.h | 7 ++ include/uapi/linux/bpf.h | 12 ++++ kernel/bpf/btf.c | 11 ++- kernel/bpf/helpers.c | 22 ++++++ kernel/bpf/verifier.c | 128 ++++++++++++++++++++++++++++----- tools/include/uapi/linux/bpf.h | 12 ++++ 6 files changed, 175 insertions(+), 17 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 48ddde854d67..6814e4885fab 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -160,10 +160,15 @@ enum { BPF_MAP_VALUE_OFF_MAX = 8, }; +enum { + BPF_MAP_VALUE_OFF_F_REF = (1U << 0), +}; + struct bpf_map_value_off_desc { u32 offset; u32 btf_id; struct btf *btf; + int flags; }; struct bpf_map_value_off { @@ -413,6 +418,7 @@ enum bpf_arg_type { ARG_PTR_TO_STACK, /* pointer to stack */ ARG_PTR_TO_CONST_STR, /* pointer to a null terminated read-only string */ ARG_PTR_TO_TIMER, /* pointer to bpf_timer */ + ARG_PTR_TO_KPTR, /* pointer to kptr */ __BPF_ARG_TYPE_MAX, /* Extended arg_types. */ @@ -422,6 +428,7 @@ enum bpf_arg_type { ARG_PTR_TO_SOCKET_OR_NULL = PTR_MAYBE_NULL | ARG_PTR_TO_SOCKET, ARG_PTR_TO_ALLOC_MEM_OR_NULL = PTR_MAYBE_NULL | ARG_PTR_TO_ALLOC_MEM, ARG_PTR_TO_STACK_OR_NULL = PTR_MAYBE_NULL | ARG_PTR_TO_STACK, + ARG_PTR_TO_BTF_ID_OR_NULL = PTR_MAYBE_NULL | ARG_PTR_TO_BTF_ID, /* This must be the last entry. Its purpose is to ensure the enum is * wide enough to hold the higher bits reserved for bpf_type_flag. diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 7604e7d5438f..b4e89da75d77 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -5143,6 +5143,17 @@ union bpf_attr { * The **hash_algo** is returned on success, * **-EOPNOTSUP** if the hash calculation failed or **-EINVAL** if * invalid arguments are passed. + * + * void *bpf_kptr_xchg(void *map_value, void *ptr) + * Description + * Exchange kptr at pointer *map_value* with *ptr*, and return the + * old value. *ptr* can be NULL, otherwise it must be a referenced + * pointer which will be released when this helper is called. + * Return + * The old value of kptr (which can be NULL). The returned pointer + * if not NULL, is a reference which must be released using its + * corresponding release function, or moved into a BPF map before + * program exit. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -5339,6 +5350,7 @@ union bpf_attr { FN(copy_from_user_task), \ FN(skb_set_tstamp), \ FN(ima_file_hash), \ + FN(kptr_xchg), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 92afbec0a887..e36ad26a5a6e 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -3175,6 +3175,7 @@ enum { struct btf_field_info { const struct btf_type *type; u32 off; + int flags; }; static int btf_find_field_struct(const struct btf *btf, const struct btf_type *t, @@ -3191,6 +3192,8 @@ static int btf_find_field_struct(const struct btf *btf, const struct btf_type *t static int btf_find_field_kptr(const struct btf *btf, const struct btf_type *t, u32 off, int sz, struct btf_field_info *info) { + int flags; + /* For PTR, sz is always == 8 */ if (!btf_type_is_ptr(t)) return BTF_FIELD_IGNORE; @@ -3201,7 +3204,11 @@ static int btf_find_field_kptr(const struct btf *btf, const struct btf_type *t, /* Reject extra tags */ if (btf_type_is_type_tag(btf_type_by_id(btf, t->type))) return -EINVAL; - if (strcmp("kptr", __btf_name_by_offset(btf, t->name_off))) + if (!strcmp("kptr", __btf_name_by_offset(btf, t->name_off))) + flags = 0; + else if (!strcmp("kptr_ref", __btf_name_by_offset(btf, t->name_off))) + flags = BPF_MAP_VALUE_OFF_F_REF; + else return -EINVAL; /* Get the base type */ @@ -3213,6 +3220,7 @@ static int btf_find_field_kptr(const struct btf *btf, const struct btf_type *t, info->type = t; info->off = off; + info->flags = flags; return BTF_FIELD_FOUND; } @@ -3415,6 +3423,7 @@ struct bpf_map_value_off *btf_find_kptr(const struct btf *btf, tab->off[i].offset = info_arr[i].off; tab->off[i].btf_id = id; tab->off[i].btf = off_btf; + tab->off[i].flags = info_arr[i].flags; tab->nr_off = i + 1; } return tab; diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 315053ef6a75..2e95f94d4efa 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -1374,6 +1374,26 @@ void bpf_timer_cancel_and_free(void *val) kfree(t); } +BPF_CALL_2(bpf_kptr_xchg, void *, map_value, void *, ptr) +{ + unsigned long *kptr = map_value; + + return xchg(kptr, (unsigned long)ptr); +} + +static u32 bpf_kptr_xchg_btf_id; + +const struct bpf_func_proto bpf_kptr_xchg_proto = { + .func = bpf_kptr_xchg, + .gpl_only = false, + .ret_type = RET_PTR_TO_BTF_ID_OR_NULL, + .ret_btf_id = &bpf_kptr_xchg_btf_id, + .arg1_type = ARG_PTR_TO_KPTR, + .arg2_type = ARG_PTR_TO_BTF_ID_OR_NULL, + .arg2_btf_id = &bpf_kptr_xchg_btf_id, + .arg2_release = true, +}; + const struct bpf_func_proto bpf_get_current_task_proto __weak; const struct bpf_func_proto bpf_get_current_task_btf_proto __weak; const struct bpf_func_proto bpf_probe_read_user_proto __weak; @@ -1452,6 +1472,8 @@ bpf_base_func_proto(enum bpf_func_id func_id) return &bpf_timer_start_proto; case BPF_FUNC_timer_cancel: return &bpf_timer_cancel_proto; + case BPF_FUNC_kptr_xchg: + return &bpf_kptr_xchg_proto; default: break; } diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index b8cd34607215..f731a0b45acb 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -258,6 +258,7 @@ struct bpf_call_arg_meta { struct btf *ret_btf; u32 ret_btf_id; u32 subprogno; + struct bpf_map_value_off_desc *kptr_off_desc; }; struct btf *btf_vmlinux; @@ -480,7 +481,8 @@ static bool is_release_function(enum bpf_func_id func_id) { return func_id == BPF_FUNC_sk_release || func_id == BPF_FUNC_ringbuf_submit || - func_id == BPF_FUNC_ringbuf_discard; + func_id == BPF_FUNC_ringbuf_discard || + func_id == BPF_FUNC_kptr_xchg; } static bool may_be_acquire_function(enum bpf_func_id func_id) @@ -500,7 +502,8 @@ static bool is_acquire_function(enum bpf_func_id func_id, if (func_id == BPF_FUNC_sk_lookup_tcp || func_id == BPF_FUNC_sk_lookup_udp || func_id == BPF_FUNC_skc_lookup_tcp || - func_id == BPF_FUNC_ringbuf_reserve) + func_id == BPF_FUNC_ringbuf_reserve || + func_id == BPF_FUNC_kptr_xchg) return true; if (func_id == BPF_FUNC_map_lookup_elem && @@ -3510,10 +3513,12 @@ int check_ptr_off_reg(struct bpf_verifier_env *env, static int map_kptr_match_type(struct bpf_verifier_env *env, struct bpf_map_value_off_desc *off_desc, - struct bpf_reg_state *reg, u32 regno) + struct bpf_reg_state *reg, u32 regno, + bool ref_ptr) { const char *targ_name = kernel_type_name(off_desc->btf, off_desc->btf_id); const char *reg_name = ""; + bool fixed_off_ok = true; if (reg->type != PTR_TO_BTF_ID && reg->type != PTR_TO_BTF_ID_OR_NULL) goto bad_type; @@ -3525,7 +3530,26 @@ static int map_kptr_match_type(struct bpf_verifier_env *env, /* We need to verify reg->type and reg->btf, before accessing reg->btf */ reg_name = kernel_type_name(reg->btf, reg->btf_id); - if (__check_ptr_off_reg(env, reg, regno, true)) + if (ref_ptr) { + if (!reg->ref_obj_id) { + verbose(env, "R%d must be referenced %s%s\n", regno, + reg_type_str(env, PTR_TO_BTF_ID), targ_name); + return -EACCES; + } + /* reg->off can be used to store pointer to a certain type formed by + * incrementing pointer of a parent structure the object is embedded in, + * e.g. map may expect unreferenced struct path *, and user should be + * allowed a store using &file->f_path. However, in the case of + * referenced pointer, we cannot do this, because the reference is only + * for the parent structure, not its embedded object(s), and because + * the transfer of ownership happens for the original pointer to and + * from the map (before its eventual release). + */ + if (reg->off) + fixed_off_ok = false; + } + /* var_off is rejected by __check_ptr_off_reg for PTR_TO_BTF_ID */ + if (__check_ptr_off_reg(env, reg, regno, fixed_off_ok)) return -EACCES; if (!btf_struct_ids_match(&env->log, reg->btf, reg->btf_id, reg->off, @@ -3568,6 +3592,12 @@ static int check_map_kptr_access(struct bpf_verifier_env *env, u32 regno, if (BPF_MODE(insn->code) != BPF_MEM) goto end; + /* We cannot directly access kptr_ref */ + if (off_desc->flags & BPF_MAP_VALUE_OFF_F_REF) { + verbose(env, "accessing referenced kptr disallowed\n"); + return -EACCES; + } + if (class == BPF_LDX) { val_reg = reg_state(env, value_regno); /* We can simply mark the value_regno receiving the pointer @@ -3579,7 +3609,7 @@ static int check_map_kptr_access(struct bpf_verifier_env *env, u32 regno, } else if (class == BPF_STX) { val_reg = reg_state(env, value_regno); if (!register_is_null(val_reg) && - map_kptr_match_type(env, off_desc, val_reg, value_regno)) + map_kptr_match_type(env, off_desc, val_reg, value_regno, false)) return -EACCES; } else if (class == BPF_ST) { if (insn->imm) { @@ -5255,6 +5285,59 @@ static int process_timer_func(struct bpf_verifier_env *env, int regno, return 0; } +static int process_kptr_func(struct bpf_verifier_env *env, int regno, + struct bpf_call_arg_meta *meta) +{ + struct bpf_reg_state *regs = cur_regs(env), *reg = ®s[regno]; + struct bpf_map_value_off_desc *off_desc; + struct bpf_map *map_ptr = reg->map_ptr; + u32 kptr_off; + int ret; + + if (!tnum_is_const(reg->var_off)) { + verbose(env, + "R%d doesn't have constant offset. kptr has to be at the constant offset\n", + regno); + return -EINVAL; + } + if (!map_ptr->btf) { + verbose(env, "map '%s' has to have BTF in order to use bpf_kptr_xchg\n", + map_ptr->name); + return -EINVAL; + } + if (!map_value_has_kptr(map_ptr)) { + ret = PTR_ERR(map_ptr->kptr_off_tab); + if (ret == -E2BIG) + verbose(env, "map '%s' has more than %d kptr\n", map_ptr->name, + BPF_MAP_VALUE_OFF_MAX); + else if (ret == -EEXIST) + verbose(env, "map '%s' has repeating kptr BTF tags\n", map_ptr->name); + else + verbose(env, "map '%s' has no valid kptr\n", map_ptr->name); + return -EINVAL; + } + + meta->map_ptr = map_ptr; + /* Check access for BPF_WRITE */ + meta->raw_mode = true; + ret = check_helper_mem_access(env, regno, sizeof(u64), false, meta); + if (ret < 0) + return ret; + + kptr_off = reg->off + reg->var_off.value; + off_desc = bpf_map_kptr_off_contains(map_ptr, kptr_off); + if (!off_desc) { + verbose(env, "off=%d doesn't point to kptr\n", kptr_off); + return -EACCES; + } + if (!(off_desc->flags & BPF_MAP_VALUE_OFF_F_REF)) { + verbose(env, "off=%d kptr isn't referenced kptr\n", kptr_off); + return -EACCES; + } + meta->kptr_off_desc = off_desc; + return 0; +} + static bool arg_type_is_mem_ptr(enum bpf_arg_type type) { return base_type(type) == ARG_PTR_TO_MEM || @@ -5390,6 +5473,7 @@ static const struct bpf_reg_types func_ptr_types = { .types = { PTR_TO_FUNC } }; static const struct bpf_reg_types stack_ptr_types = { .types = { PTR_TO_STACK } }; static const struct bpf_reg_types const_str_ptr_types = { .types = { PTR_TO_MAP_VALUE } }; static const struct bpf_reg_types timer_types = { .types = { PTR_TO_MAP_VALUE } }; +static const struct bpf_reg_types kptr_types = { .types = { PTR_TO_MAP_VALUE } }; static const struct bpf_reg_types *compatible_reg_types[__BPF_ARG_TYPE_MAX] = { [ARG_PTR_TO_MAP_KEY] = &map_key_value_types, @@ -5417,11 +5501,13 @@ static const struct bpf_reg_types *compatible_reg_types[__BPF_ARG_TYPE_MAX] = { [ARG_PTR_TO_STACK] = &stack_ptr_types, [ARG_PTR_TO_CONST_STR] = &const_str_ptr_types, [ARG_PTR_TO_TIMER] = &timer_types, + [ARG_PTR_TO_KPTR] = &kptr_types, }; static int check_reg_type(struct bpf_verifier_env *env, u32 regno, enum bpf_arg_type arg_type, - const u32 *arg_btf_id) + const u32 *arg_btf_id, + struct bpf_call_arg_meta *meta) { struct bpf_reg_state *regs = cur_regs(env), *reg = ®s[regno]; enum bpf_reg_type expected, type = reg->type; @@ -5474,8 +5560,11 @@ static int check_reg_type(struct bpf_verifier_env *env, u32 regno, arg_btf_id = compatible->btf_id; } - if (!btf_struct_ids_match(&env->log, reg->btf, reg->btf_id, reg->off, - btf_vmlinux, *arg_btf_id)) { + if (meta->func_id == BPF_FUNC_kptr_xchg) { + if (map_kptr_match_type(env, meta->kptr_off_desc, reg, regno, true)) + return -EACCES; + } else if (!btf_struct_ids_match(&env->log, reg->btf, reg->btf_id, reg->off, + btf_vmlinux, *arg_btf_id)) { verbose(env, "R%d is of type %s but %s is expected\n", regno, kernel_type_name(reg->btf, reg->btf_id), kernel_type_name(btf_vmlinux, *arg_btf_id)); @@ -5585,7 +5674,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, */ goto skip_type_check; - err = check_reg_type(env, regno, arg_type, fn->arg_btf_id[arg]); + err = check_reg_type(env, regno, arg_type, fn->arg_btf_id[arg], meta); if (err) return err; @@ -5750,6 +5839,9 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, verbose(env, "string is not zero-terminated\n"); return -EINVAL; } + } else if (arg_type == ARG_PTR_TO_KPTR) { + if (process_kptr_func(env, regno, meta)) + return -EACCES; } return err; @@ -6092,10 +6184,10 @@ static bool check_btf_id_ok(const struct bpf_func_proto *fn) int i; for (i = 0; i < ARRAY_SIZE(fn->arg_type); i++) { - if (fn->arg_type[i] == ARG_PTR_TO_BTF_ID && !fn->arg_btf_id[i]) + if (base_type(fn->arg_type[i]) == ARG_PTR_TO_BTF_ID && !fn->arg_btf_id[i]) return false; - if (fn->arg_type[i] != ARG_PTR_TO_BTF_ID && fn->arg_btf_id[i]) + if (base_type(fn->arg_type[i]) != ARG_PTR_TO_BTF_ID && fn->arg_btf_id[i]) return false; } @@ -6979,21 +7071,25 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn regs[BPF_REG_0].btf_id = meta.ret_btf_id; } } else if (base_type(ret_type) == RET_PTR_TO_BTF_ID) { + struct btf *ret_btf; int ret_btf_id; mark_reg_known_zero(env, regs, BPF_REG_0); regs[BPF_REG_0].type = PTR_TO_BTF_ID | ret_flag; - ret_btf_id = *fn->ret_btf_id; + if (func_id == BPF_FUNC_kptr_xchg) { + ret_btf = meta.kptr_off_desc->btf; + ret_btf_id = meta.kptr_off_desc->btf_id; + } else { + ret_btf = btf_vmlinux; + ret_btf_id = *fn->ret_btf_id; + } if (ret_btf_id == 0) { verbose(env, "invalid return type %u of func %s#%d\n", base_type(ret_type), func_id_name(func_id), func_id); return -EINVAL; } - /* current BPF helper definitions are only coming from - * built-in code with type IDs from vmlinux BTF - */ - regs[BPF_REG_0].btf = btf_vmlinux; + regs[BPF_REG_0].btf = ret_btf; regs[BPF_REG_0].btf_id = ret_btf_id; } else { verbose(env, "unknown return type %u of func %s#%d\n", diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 7604e7d5438f..b4e89da75d77 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -5143,6 +5143,17 @@ union bpf_attr { * The **hash_algo** is returned on success, * **-EOPNOTSUP** if the hash calculation failed or **-EINVAL** if * invalid arguments are passed. + * + * void *bpf_kptr_xchg(void *map_value, void *ptr) + * Description + * Exchange kptr at pointer *map_value* with *ptr*, and return the + * old value. *ptr* can be NULL, otherwise it must be a referenced + * pointer which will be released when this helper is called. + * Return + * The old value of kptr (which can be NULL). The returned pointer + * if not NULL, is a reference which must be released using its + * corresponding release function, or moved into a BPF map before + * program exit. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -5339,6 +5350,7 @@ union bpf_attr { FN(copy_from_user_task), \ FN(skb_set_tstamp), \ FN(ima_file_hash), \ + FN(kptr_xchg), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper From patchwork Sun Mar 20 15:55:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 12786556 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6CA29C433EF for ; Sun, 20 Mar 2022 15:55:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244429AbiCTP5E (ORCPT ); Sun, 20 Mar 2022 11:57:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49270 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245410AbiCTP5E (ORCPT ); Sun, 20 Mar 2022 11:57:04 -0400 Received: from mail-pg1-x543.google.com (mail-pg1-x543.google.com [IPv6:2607:f8b0:4864:20::543]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 58F5354194 for ; Sun, 20 Mar 2022 08:55:41 -0700 (PDT) Received: by mail-pg1-x543.google.com with SMTP id s72so5952160pgc.5 for ; Sun, 20 Mar 2022 08:55:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=nYf48VEX3+P4bixfCnu0l75BtD8ZTpoiibp1OpdQj2I=; b=LrNV5Ne7dD0sFo84fyDb5MnZCbh5gzawC4iz1FGBwqTm7q9Hti9xtmmYtev3QS+p4i JbjoBgGu0zXbcoBXBT/gUxCd6/JeEIIXMewFCoiWwDxXnQKXpSxQGpKDe94Gc+XaJ+ME iDmWwV9oubS3Bey2lXfBo6bRXDMN0dms4IiZyveoVJMXsVGrhhUb+JS34rZI6Pmxg8Cc 6A9XmkdHDFD4xfk2BZrcKhtYktimZ/4sQLvcnGuXHUA6GRHLaOfzBgAIFW6BNyPLQmPW p0/9ldx9p6Q0ODLmgKpF/0Zovn6Tw7qWkxmVwl+Lv2kzE3Je01CFS4XIR9IIQm1GJCEr 3m3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nYf48VEX3+P4bixfCnu0l75BtD8ZTpoiibp1OpdQj2I=; b=mcOQaboNXLdSQ6cqC+Rv4eEQwj/Z4JiGyZoK6yNN+bNK/aDlCtTvt9bZChMXJ890Ha QuwXkV76/8OIbJ2z5xqF3zsmfxyJI+Znftl3a7CGtzrvtMQ4FyaC1DgO2XTFjhb4xCAR TZsLtc8iKLD7fwvLoqwCg7L+51d/jfsUzXRQMfpRyHSh4TsS6I5FR8gbGkcshD8Agr3n DPQCBkJE/sZngWu/mgRSeMwlc27DDVd+X9FWMEfc8cNMZKybttzdB59YGJ3m3+yYkXm7 dOmG5ScM/pyWZDK8CaxgZmz7epD2wErSJ5Cl+qsd2fgUQQs0+llDjefAUYMutzqcDzMW d+Sw== X-Gm-Message-State: AOAM5311RGvq/o77j2IbEpYtp/yy9H7Q7KpoUWPOsYkKQMJbQlY+CpbG k76ubcdZDkBCRrTyhnV1wyuZ/NCERh0= X-Google-Smtp-Source: ABdhPJxyi84SRB7VJQBuMX7ujJJmO3aZi6jQlZaVsWbdy7uT8x4r94sAujoyQf/875hxPCfsgXia4w== X-Received: by 2002:a63:ea51:0:b0:380:7c35:188e with SMTP id l17-20020a63ea51000000b003807c35188emr15143886pgk.607.1647791740661; Sun, 20 Mar 2022 08:55:40 -0700 (PDT) Received: from localhost ([14.139.187.71]) by smtp.gmail.com with ESMTPSA id j23-20020a17090a841700b001c678ac3b4bsm10528299pjn.14.2022.03.20.08.55.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 20 Mar 2022 08:55:40 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8?= =?utf-8?q?rgensen?= , Jesper Dangaard Brouer Subject: [PATCH bpf-next v3 06/13] bpf: Prevent escaping of kptr loaded from maps Date: Sun, 20 Mar 2022 21:25:03 +0530 Message-Id: <20220320155510.671497-7-memxor@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220320155510.671497-1-memxor@gmail.com> References: <20220320155510.671497-1-memxor@gmail.com> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=6017; h=from:subject; bh=KAUHyNQY0gWctZKJXveg9zTRVc8SubaUYgGT8dpTKCc=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBiN00yretAtcKT6OQVTbUkuEa+MqFR5sIMFQtFQJj1 8gFzo1iJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCYjdNMgAKCRBM4MiGSL8RysD0D/ 4xsOi4u6npSmhNpM9NSwi8TB5X4eTg+R16MGaeRmtn1ZhVJf5at+/EGCMShlbcWq9pZrjccQIzP2oe sT6ybQyBUIZQBylxayQGBCMN27I1ldVBIrRt7TaRY7vR3fUvPbISXiI7xYUgsz0cF8VTgZA1res7gn Zgt9mpU7rgloptxhm6Q9EibH48sJ4V9Kq60YX2T8n80sj1raagHS+PShhwenTs08Ew7CRrXEW2ZGeP +XZ2sGX6SoFWGV/1fI+ACJzLYzWYMMLYATRDiC0vcfdrJdk6xbGUOtCL1JUrGgemA/P4byx1BUqsT2 yc1ad00VJ04dJZQZ9hzbnyqi7fx1wewKpaudhrtVPNj22GUgFGtyhf6FgeutQBX/aZQEVRN83eoDva DjyfN9fJgmk0btzuiwSPyycK5neBX8GcWf3Aj05mVEF9TZzG/HzV5Xj7D2SMR9A/1nJQW6GVfGU4xJ L4MH2MYH9QqR2jPFi8df61xCQPVTNQfqfuFG8SDT+LlRAINff/3Mtng8PLqikxUVfXjMDOFaGaHkqt 2aao+eF2Hc2CaF+aoCADzTzJB+Rd0htW04vp9VD3SHdbHmV2kJLaZCfG+zHs09qQd09oXnU38jHZCR iOzQ69WhDkOd30KUADACJhOmtp1UVqNU59Wd1GnAiZej+/ZlHD6PRYq94wbg== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net While we can guarantee that even for unreferenced kptr, the object pointer points to being freed etc. can be handled by the verifier's exception handling (normal load patching to PROBE_MEM loads), we still cannot allow the user to pass these pointers to BPF helpers and kfunc, because the same exception handling won't be done for accesses inside the kernel. The same is true if a referenced pointer is loaded using normal load instruction. Since the reference is not guaranteed to be held while the pointer is used, it must be marked as untrusted. Hence introduce a new type flag, PTR_UNTRUSTED, which is used to mark all registers loading unreferenced and referenced kptr from BPF maps, and ensure they can never escape the BPF program and into the kernel by way of calling stable/unstable helpers. In check_ptr_to_btf_access, the !type_may_be_null check to reject type flags is still correct, as apart from PTR_MAYBE_NULL, only MEM_USER, MEM_PERCPU, and PTR_UNTRUSTED may be set for PTR_TO_BTF_ID. The first two are checked inside the function and rejected using a proper error message, but we still want to allow dereference of untrusted case. Also, we make sure to inherit PTR_UNTRUSTED when chain of pointers are walked, so that this flag is never dropped once it has been set on a PTR_TO_BTF_ID (i.e. trusted to untrusted transition can only be in one direction). In convert_ctx_accesses, extend the switch case to consider untrusted PTR_TO_BTF_ID in addition to normal PTR_TO_BTF_ID for PROBE_MEM conversion for BPF_LDX. Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/bpf.h | 10 +++++++++- kernel/bpf/verifier.c | 34 +++++++++++++++++++++++++++------- 2 files changed, 36 insertions(+), 8 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 6814e4885fab..9d424d567dd3 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -369,7 +369,15 @@ enum bpf_type_flag { */ MEM_PERCPU = BIT(4 + BPF_BASE_TYPE_BITS), - __BPF_TYPE_LAST_FLAG = MEM_PERCPU, + /* PTR is not trusted. This is only used with PTR_TO_BTF_ID, to mark + * unreferenced and referenced kptr loaded from map value using a load + * instruction, so that they can only be dereferenced but not escape the + * BPF program into the kernel (i.e. cannot be passed as arguments to + * kfunc or bpf helpers). + */ + PTR_UNTRUSTED = BIT(5 + BPF_BASE_TYPE_BITS), + + __BPF_TYPE_LAST_FLAG = PTR_UNTRUSTED, }; /* Max number of base types. */ diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index f731a0b45acb..9c5c72ea1d98 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -579,6 +579,8 @@ static const char *reg_type_str(struct bpf_verifier_env *env, strncpy(prefix, "user_", 32); if (type & MEM_PERCPU) strncpy(prefix, "percpu_", 32); + if (type & PTR_UNTRUSTED) + strncpy(prefix, "untrusted_", 32); snprintf(env->type_str_buf, TYPE_STR_BUF_LEN, "%s%s%s", prefix, str[base_type(type)], postfix); @@ -3520,8 +3522,17 @@ static int map_kptr_match_type(struct bpf_verifier_env *env, const char *reg_name = ""; bool fixed_off_ok = true; - if (reg->type != PTR_TO_BTF_ID && reg->type != PTR_TO_BTF_ID_OR_NULL) - goto bad_type; + if (off_desc->flags & BPF_MAP_VALUE_OFF_F_REF) { + if (reg->type != PTR_TO_BTF_ID && + reg->type != (PTR_TO_BTF_ID | PTR_MAYBE_NULL)) + goto bad_type; + } else { /* only unreferenced case accepts untrusted pointers */ + if (reg->type != PTR_TO_BTF_ID && + reg->type != (PTR_TO_BTF_ID | PTR_MAYBE_NULL) && + reg->type != (PTR_TO_BTF_ID | PTR_UNTRUSTED) && + reg->type != (PTR_TO_BTF_ID | PTR_MAYBE_NULL | PTR_UNTRUSTED)) + goto bad_type; + } if (!btf_is_kernel(reg->btf)) { verbose(env, "R%d must point to kernel BTF\n", regno); @@ -3592,9 +3603,11 @@ static int check_map_kptr_access(struct bpf_verifier_env *env, u32 regno, if (BPF_MODE(insn->code) != BPF_MEM) goto end; - /* We cannot directly access kptr_ref */ - if (off_desc->flags & BPF_MAP_VALUE_OFF_F_REF) { - verbose(env, "accessing referenced kptr disallowed\n"); + /* We only allow loading referenced kptr, since it will be marked as + * untrusted, similar to unreferenced kptr. + */ + if (class != BPF_LDX && (off_desc->flags & BPF_MAP_VALUE_OFF_F_REF)) { + verbose(env, "store to referenced kptr disallowed\n"); return -EACCES; } @@ -3604,7 +3617,7 @@ static int check_map_kptr_access(struct bpf_verifier_env *env, u32 regno, * value from map as PTR_TO_BTF_ID, with the correct type. */ mark_btf_ld_reg(env, cur_regs(env), value_regno, PTR_TO_BTF_ID, off_desc->btf, - off_desc->btf_id, PTR_MAYBE_NULL); + off_desc->btf_id, PTR_MAYBE_NULL | PTR_UNTRUSTED); val_reg->id = ++env->id_gen; } else if (class == BPF_STX) { val_reg = reg_state(env, value_regno); @@ -4369,6 +4382,12 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, if (ret < 0) return ret; + /* If this is an untrusted pointer, all pointers formed by walking it + * also inherit the untrusted flag. + */ + if (type_flag(reg->type) & PTR_UNTRUSTED) + flag |= PTR_UNTRUSTED; + if (atype == BPF_READ && value_regno >= 0) mark_btf_ld_reg(env, regs, value_regno, ret, reg->btf, btf_id, flag); @@ -13065,7 +13084,7 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) if (!ctx_access) continue; - switch (env->insn_aux_data[i + delta].ptr_type) { + switch ((int)env->insn_aux_data[i + delta].ptr_type) { case PTR_TO_CTX: if (!ops->convert_ctx_access) continue; @@ -13082,6 +13101,7 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) convert_ctx_access = bpf_xdp_sock_convert_ctx_access; break; case PTR_TO_BTF_ID: + case PTR_TO_BTF_ID | PTR_UNTRUSTED: if (type == BPF_READ) { insn->code = BPF_LDX | BPF_PROBE_MEM | BPF_SIZE((insn)->code); From patchwork Sun Mar 20 15:55:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 12786557 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 339D9C433EF for ; Sun, 20 Mar 2022 15:55:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245410AbiCTP5I (ORCPT ); Sun, 20 Mar 2022 11:57:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49554 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245415AbiCTP5I (ORCPT ); Sun, 20 Mar 2022 11:57:08 -0400 Received: from mail-pj1-x1043.google.com (mail-pj1-x1043.google.com [IPv6:2607:f8b0:4864:20::1043]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9687454196 for ; Sun, 20 Mar 2022 08:55:44 -0700 (PDT) Received: by mail-pj1-x1043.google.com with SMTP id kx5-20020a17090b228500b001c6ed9db871so2161214pjb.1 for ; Sun, 20 Mar 2022 08:55:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=T4H7pdnq3bwIteVvOoef2KNTqot/e0QuKMYjkx98YRQ=; b=f0hCRfMFQb7KSLcoxODvZTGTS7VE/lgAP9C/qokvpRVn/0voMzSa80wObrOuLhe+xS w5TpwOSSRmA1DFYvWxtXAlhQlOyHUcR9ZZe0NSDS9ZbX4dAc+kjm5DCqtsvhn/WqMymS bA6blBLLoOJqH7Ys1//L9Z+Fugza2cqm2dxmDGodyxDbH50b0wsfrLWqPL5EaxA9nqKr EGDfmHRoVHbtyWJhwDieZrrW+4gm22V5VpgDyALVEvhcNwt1rdgl+MfpXsjGjNiSuM20 qg2NKMnq3RKkTJHwcbE35DdgXa9reFnPYEzUoIxBVz0It0U83OzReZHro6nps14IsgYI cc/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=T4H7pdnq3bwIteVvOoef2KNTqot/e0QuKMYjkx98YRQ=; b=MMd+cTlxmAfDxI6wPjGmHx89FsY4P68PbHxH+tosgAWxi7wNWK0kixYrl3DQnOWpZM qNyf0TYp38prWN76U5N5M9BG4UbfleweyL1pj1yCEcIJVSWMa4PwuRllFlVdkPyI3Zp0 9vYBDTl/sZfHZ6ReqCdgf2mrc7i5P9AaLvNrMOgVZPV8YanLff1hdgXiLwBjtsLeFdGY i9rdZzR+XlUEiUm5UD2Ae0r5cdDKuzPQzAKGaK5o9hVOi2MfKObAJf1DPW6axBRCoePh BSdoygV3/uIXqPz/Z51hJ6VlZKdPQ7aLu/SEy+aVV6Jjufpy1hQNfmgt7pmah+MGXReo EZSg== X-Gm-Message-State: AOAM533lLjzlM4MpIDJWyoExgvH2tMOHcMcxIMEgSUSjaT3NawYF/s9E wbuoO+tOAp1ezLgtIQ8p+Ti/PPn0rYo= X-Google-Smtp-Source: ABdhPJyMqc8xr60dW3bA8rY5mclUBAK4eSVgElu1mJHrYkLq7N4kPeApezhVkH90MLXRiEwVsFLYlw== X-Received: by 2002:a17:90b:3508:b0:1c6:e4f9:538b with SMTP id ls8-20020a17090b350800b001c6e4f9538bmr6889826pjb.158.1647791743962; Sun, 20 Mar 2022 08:55:43 -0700 (PDT) Received: from localhost ([14.139.187.71]) by smtp.gmail.com with ESMTPSA id j16-20020a63e750000000b00373598b8cbfsm12270941pgk.74.2022.03.20.08.55.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 20 Mar 2022 08:55:43 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8?= =?utf-8?q?rgensen?= , Jesper Dangaard Brouer Subject: [PATCH bpf-next v3 07/13] bpf: Adapt copy_map_value for multiple offset case Date: Sun, 20 Mar 2022 21:25:04 +0530 Message-Id: <20220320155510.671497-8-memxor@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220320155510.671497-1-memxor@gmail.com> References: <20220320155510.671497-1-memxor@gmail.com> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5763; h=from:subject; bh=RxLjSvyKuEdkZl4+p5jTlytjnaAo9SXxwcUKkicAGLU=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBiN00yX8y4Sla0T5nPy9taGtCoNFvohxuugwzlLQyv bSSs7UaJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCYjdNMgAKCRBM4MiGSL8Ryv69EA CIzGDRc0WZLX1ejlFp3uxvXeDG5rPn/r4FEdOAWILqGZy95qPk//mqYlSdLTMHUJ4cDBCTArxl+zc6 ieNy58mXIj7Gfm7awZFGbCawE/xjZc5qRVdQ+RlgXSgo4bHn9kQJML5sO61EavzeYQ03Vmp628V+IF s0+llBGY3265ApZ7nChNVIzcUtmB9laSyI39XLkzgLk4uEKe299u3Ywk0U1NLvrsvQWrdNn2PELdsZ ph3ivFeVLK1ArOEzElThQ4kRtEiVQQT/FW6pukBYP1SlgEaP8h0yk/ZfwXc/kH/DNcjgTQGdpbmVae FBlJcFl0Car0Q6Zd/84ARGCP6yQrmvUma7LOc5NLFnHmKd+0j1gcbfygmjaKuDcFduviVBVil3BwSg Rq+q2U+CJijiPhEauBQ9X5pAK10UDncAjw5YyQ5YYepZxqh7Jfos88kZnfxYM0fqjQhQPUewy5hYr+ RaiQhLVsWtjvwbuJFKH9X/amttRhLnIDKEOkRfOAen2RK6VlyVMagg6/8hHVJv7BFAKsVymgxNP3dC vyl0PmFX1LhnmFHPeEzod7lqnJk8z5czeu1HTRBRvTVVevGohHkUF3gO1JMpfMOQudl80UlXBp2uuE yKi3L0wTAjJvv3Z7mjUvcZURnHUfqykD2cVfsydgOIGHOkAr0bPgybJOsAMA== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Since now there might be at most 10 offsets that need handling in copy_map_value, the manual shuffling and special case is no longer going to work. Hence, let's generalise the copy_map_value function by using a sorted array of offsets to skip regions that must be avoided while copying into and out of a map value. When the map is created, we populate the offset array in struct map, with one extra element for map->value_size, which is used as the final offset to subtract previous offset from. Then, copy_map_value uses this sorted offset array is used to memcpy while skipping timer, spin lock, and kptr. Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/bpf.h | 55 +++++++++++++++++++++++--------------------- kernel/bpf/syscall.c | 52 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 81 insertions(+), 26 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 9d424d567dd3..6474d2d44b78 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -158,6 +158,10 @@ struct bpf_map_ops { enum { /* Support at most 8 pointers in a BPF map value */ BPF_MAP_VALUE_OFF_MAX = 8, + BPF_MAP_OFF_ARR_MAX = BPF_MAP_VALUE_OFF_MAX + + 1 + /* for bpf_spin_lock */ + 1 + /* for bpf_timer */ + 1, /* for map->value_size sentinel */ }; enum { @@ -206,9 +210,17 @@ struct bpf_map { char name[BPF_OBJ_NAME_LEN]; bool bypass_spec_v1; bool frozen; /* write-once; write-protected by freeze_mutex */ - /* 6 bytes hole */ - - /* The 3rd and 4th cacheline with misc members to avoid false sharing + /* 2 bytes hole */ + struct { + struct { + u32 off; + u8 sz; + } field[BPF_MAP_OFF_ARR_MAX]; + u32 cnt; + } off_arr; + /* 40 bytes hole */ + + /* The 4th and 5th cacheline with misc members to avoid false sharing * particularly with refcounting. */ atomic64_t refcnt ____cacheline_aligned; @@ -250,36 +262,27 @@ static inline void check_and_init_map_value(struct bpf_map *map, void *dst) memset(dst + map->spin_lock_off, 0, sizeof(struct bpf_spin_lock)); if (unlikely(map_value_has_timer(map))) memset(dst + map->timer_off, 0, sizeof(struct bpf_timer)); + if (unlikely(map_value_has_kptr(map))) { + struct bpf_map_value_off *tab = map->kptr_off_tab; + int i; + + for (i = 0; i < tab->nr_off; i++) + *(u64 *)(dst + tab->off[i].offset) = 0; + } } /* copy everything but bpf_spin_lock and bpf_timer. There could be one of each. */ static inline void copy_map_value(struct bpf_map *map, void *dst, void *src) { - u32 s_off = 0, s_sz = 0, t_off = 0, t_sz = 0; + int i; - if (unlikely(map_value_has_spin_lock(map))) { - s_off = map->spin_lock_off; - s_sz = sizeof(struct bpf_spin_lock); - } - if (unlikely(map_value_has_timer(map))) { - t_off = map->timer_off; - t_sz = sizeof(struct bpf_timer); - } + memcpy(dst, src, map->off_arr.field[0].off); + for (i = 1; i < map->off_arr.cnt; i++) { + u32 curr_off = map->off_arr.field[i - 1].off; + u32 next_off = map->off_arr.field[i].off; - if (unlikely(s_sz || t_sz)) { - if (s_off < t_off || !s_sz) { - swap(s_off, t_off); - swap(s_sz, t_sz); - } - memcpy(dst, src, t_off); - memcpy(dst + t_off + t_sz, - src + t_off + t_sz, - s_off - t_off - t_sz); - memcpy(dst + s_off + s_sz, - src + s_off + s_sz, - map->value_size - s_off - s_sz); - } else { - memcpy(dst, src, map->value_size); + curr_off += map->off_arr.field[i - 1].sz; + memcpy(dst + curr_off, src + curr_off, next_off - curr_off); } } void copy_map_value_locked(struct bpf_map *map, void *dst, void *src, diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 5990d6fa97ab..7b32537bd81f 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -30,6 +30,7 @@ #include #include #include +#include #include #include #include @@ -851,6 +852,55 @@ int map_check_no_btf(const struct bpf_map *map, return -ENOTSUPP; } +static int map_off_arr_cmp(const void *_a, const void *_b) +{ + const u32 a = *(const u32 *)_a; + const u32 b = *(const u32 *)_b; + + if (a < b) + return -1; + else if (a > b) + return 1; + return 0; +} + +static void map_populate_off_arr(struct bpf_map *map) +{ + u32 i; + + map->off_arr.cnt = 0; + if (map_value_has_spin_lock(map)) { + i = map->off_arr.cnt; + + map->off_arr.field[i].off = map->spin_lock_off; + map->off_arr.field[i].sz = sizeof(struct bpf_spin_lock); + map->off_arr.cnt++; + } + if (map_value_has_timer(map)) { + i = map->off_arr.cnt; + + map->off_arr.field[i].off = map->timer_off; + map->off_arr.field[i].sz = sizeof(struct bpf_timer); + map->off_arr.cnt++; + } + if (map_value_has_kptr(map)) { + struct bpf_map_value_off *tab = map->kptr_off_tab; + u32 j = map->off_arr.cnt; + + for (i = 0; i < tab->nr_off; i++) { + map->off_arr.field[j + i].off = tab->off[i].offset; + map->off_arr.field[j + i].sz = sizeof(u64); + } + map->off_arr.cnt += tab->nr_off; + } + + map->off_arr.field[map->off_arr.cnt++].off = map->value_size; + if (map->off_arr.cnt == 1) + return; + sort(map->off_arr.field, map->off_arr.cnt, sizeof(map->off_arr.field[0]), + map_off_arr_cmp, NULL); +} + static int map_check_btf(struct bpf_map *map, const struct btf *btf, u32 btf_key_id, u32 btf_value_id) { @@ -1018,6 +1068,8 @@ static int map_create(union bpf_attr *attr) attr->btf_vmlinux_value_type_id; } + map_populate_off_arr(map); + err = security_bpf_map_alloc(map); if (err) goto free_map; From patchwork Sun Mar 20 15:55:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 12786558 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6FE1C433EF for ; Sun, 20 Mar 2022 15:55:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245414AbiCTP5L (ORCPT ); Sun, 20 Mar 2022 11:57:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49774 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245415AbiCTP5L (ORCPT ); Sun, 20 Mar 2022 11:57:11 -0400 Received: from mail-pg1-x541.google.com (mail-pg1-x541.google.com [IPv6:2607:f8b0:4864:20::541]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0290E54188 for ; Sun, 20 Mar 2022 08:55:48 -0700 (PDT) Received: by mail-pg1-x541.google.com with SMTP id s72so5952238pgc.5 for ; Sun, 20 Mar 2022 08:55:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=A+lWQlMxi/mZTiYCtRYuguDDMp67YxOZJ1qDkZ0OBh8=; b=oRaGSaKjDfpxbaDRqk/vE6sLQpZXr3ezgxYdf7h6ZXqTQ0XaZLUi97NeKdClNKnSZB zI7VsQk2OTiSihOvAF5W1IEnO6K0GMQtTAaKLs+dq3ohVeuOqf29a5R42UR/LT95N1ky iPV5Ch3h+qcmNuFmfZPxQLOEGFLCjai2vym2EmILE0bJz7RAf+WP4f1wZT+HAF2edYdZ 8PR2eoqYzeH28tWqSxXQtnPrjqbanAFEzE7mnVvFYJS4vOEF66L12RkD2TzBU8WGcHAm 8eTloakgRh5mg/yhuCXWx/oJvjhhOB7pZsThUu2/UkCQ8SJ+qo3ShKBJuXvZcVGekqBo 92vw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=A+lWQlMxi/mZTiYCtRYuguDDMp67YxOZJ1qDkZ0OBh8=; b=nILwOewtj56Hq0vujg8GN3ykqjmKsvnYfMKpVpEkkI+/EZwafRDaX4Ycq/SRjcm4k0 gM0DsvzOrXNm61dDRq1wWerYFifxC78HANlYIS/ll8psXZSBRD2LcyQ3ELml8OSR8ufD heuk5+WlCPQlid1f0e7rHcOuqz5AvbxAjP4nIKGNEvhvoToU79b9ZByrIZbK1e6FJiIQ EF5lDVnigs9cwKacKks04mM+LU3qD4KR6dw1OPs0qs1E0ihsbr/uDYaTVrDn7lXCPA5h Zdox6UHiCjX1JIG6qfRQyVlJans06LohDw3wBLnkLr+e7DGWmdqjlZMmkzTHX35SOmt9 l8Xg== X-Gm-Message-State: AOAM532pXYmQ/HhJoG3mwRfmGFGpZ56zJnWM5GMUW7YWq/R/D9US0LA8 acQjwg7QlolMJk7MXurcqRcxIiH5w58= X-Google-Smtp-Source: ABdhPJybbP/o80PE1+gvMqqSmHJuorF/GjipK4PDd7hy4vvLuvLQxhZZ5G6oZ5EZjDw1umKx61QRVg== X-Received: by 2002:a63:1b5c:0:b0:382:76f4:c76b with SMTP id b28-20020a631b5c000000b0038276f4c76bmr1932550pgm.424.1647791747342; Sun, 20 Mar 2022 08:55:47 -0700 (PDT) Received: from localhost ([14.139.187.71]) by smtp.gmail.com with ESMTPSA id s25-20020a056a00179900b004f737ce5c1asm16026374pfg.73.2022.03.20.08.55.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 20 Mar 2022 08:55:47 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8?= =?utf-8?q?rgensen?= , Jesper Dangaard Brouer Subject: [PATCH bpf-next v3 08/13] bpf: Populate pairs of btf_id and destructor kfunc in btf Date: Sun, 20 Mar 2022 21:25:05 +0530 Message-Id: <20220320155510.671497-9-memxor@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220320155510.671497-1-memxor@gmail.com> References: <20220320155510.671497-1-memxor@gmail.com> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=6270; h=from:subject; bh=FkCsHbcLcIDitDm954nc+PDuaTQ8gUpPSBycv+UOwts=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBiN00yXbHvchmFGmkgnxuQ5P88XiYVJq23dHsTM+fA jX28nRKJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCYjdNMgAKCRBM4MiGSL8RylWjD/ 0QVMN+hgf12cRvKZjyPqplUgqizDr/QL+Vw8dLJj5/8SPv1urS/MNtLzSRafsQ7pBqv3Bp3XcHwNGq eTcgeYKn6n8uSZn/c2HlxV/i0cK0kIrrA5X0WlaPSdssvVx6+1fqPY4hLkx7WpcoknrAHJ0En375Du Mt+vF9Hxs7zGah+YSpq8snu+B5+iXxoLa4WOarbmT4Zh8QU7oNYdVh21rF++/DUENS9LKB36azlCnA v+CJZ5osbmcme8sJVrMtUfGTj5XdTC/7wDkSKVoaLIT7O1SOCwwPzNB1hAhOHvgo2ng+89byMAeTAK So3+T8q2IHNFxtmXlKvT5ar+030Huosnvo3kNkklYPLCecu0op1fuTUhmHz3LA31EJMPQgARiUPors V3xSXOsr7s+1Pv42yvZV5YZtWZw3w3n5tEJaJOkpwusGITUf6U/B9TvwT/gEkNb+75jEehQKUauPvX 6eF4ctpB/5lh0JSRNVSPqMcXnh2KKRDBPbEEx1XS3P0Ln/IhqwbKMS7q8HyEvK6RTMCKCpBQtMo3Gk blNliJnjrQUvcMT/lS/Scujclgahhqv+Vn+6ZhrXzpdT886Pk1FnJbgWhzAmVXSyZ8ci/A/eV4XT9s fdwV97RcQe5U3O7MYplv1gHKI9/PrDgFagEhvsCny/IbT497F5bsC5TLyiKg== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net To support storing referenced PTR_TO_BTF_ID in maps, we require associating a specific BTF ID with a 'destructor' kfunc. This is because we need to release a live referenced pointer at a certain offset in map value from the map destruction path, otherwise we end up leaking resources. Hence, introduce support for passing an array of btf_id, kfunc_btf_id pairs that denote a BTF ID and its associated release function. Then, add an accessor 'btf_find_dtor_kfunc' which can be used to look up the destructor kfunc of a certain BTF ID. If found, we can use it to free the object from the map free path. The registration of these pairs also serve as a whitelist of structures which are allowed as referenced PTR_TO_BTF_ID in a BPF map, because without finding the destructor kfunc, we will bail and return an error. Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/btf.h | 17 +++++++ kernel/bpf/btf.c | 108 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 125 insertions(+) diff --git a/include/linux/btf.h b/include/linux/btf.h index 5b578dc81c04..ff4be49b7a26 100644 --- a/include/linux/btf.h +++ b/include/linux/btf.h @@ -40,6 +40,11 @@ struct btf_kfunc_id_set { }; }; +struct btf_id_dtor_kfunc { + u32 btf_id; + u32 kfunc_btf_id; +}; + extern const struct file_operations btf_fops; void btf_get(struct btf *btf); @@ -346,6 +351,9 @@ bool btf_kfunc_id_set_contains(const struct btf *btf, enum btf_kfunc_type type, u32 kfunc_btf_id); int register_btf_kfunc_id_set(enum bpf_prog_type prog_type, const struct btf_kfunc_id_set *s); +s32 btf_find_dtor_kfunc(struct btf *btf, u32 btf_id); +int register_btf_id_dtor_kfuncs(const struct btf_id_dtor_kfunc *dtors, u32 add_cnt, + struct module *owner); #else static inline const struct btf_type *btf_type_by_id(const struct btf *btf, u32 type_id) @@ -369,6 +377,15 @@ static inline int register_btf_kfunc_id_set(enum bpf_prog_type prog_type, { return 0; } +static inline s32 btf_find_dtor_kfunc(struct btf *btf, u32 btf_id) +{ + return -ENOENT; +} +static inline int register_btf_id_dtor_kfuncs(const struct btf_id_dtor_kfunc *dtors, + u32 add_cnt, struct module *owner) +{ + return 0; +} #endif #endif diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index e36ad26a5a6e..9cb6f61a50a7 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -207,12 +207,18 @@ enum btf_kfunc_hook { enum { BTF_KFUNC_SET_MAX_CNT = 32, + BTF_DTOR_KFUNC_MAX_CNT = 256, }; struct btf_kfunc_set_tab { struct btf_id_set *sets[BTF_KFUNC_HOOK_MAX][BTF_KFUNC_TYPE_MAX]; }; +struct btf_id_dtor_kfunc_tab { + u32 cnt; + struct btf_id_dtor_kfunc dtors[]; +}; + struct btf { void *data; struct btf_type **types; @@ -228,6 +234,7 @@ struct btf { u32 id; struct rcu_head rcu; struct btf_kfunc_set_tab *kfunc_set_tab; + struct btf_id_dtor_kfunc_tab *dtor_kfunc_tab; /* split BTF support */ struct btf *base_btf; @@ -1614,8 +1621,19 @@ static void btf_free_kfunc_set_tab(struct btf *btf) btf->kfunc_set_tab = NULL; } +static void btf_free_dtor_kfunc_tab(struct btf *btf) +{ + struct btf_id_dtor_kfunc_tab *tab = btf->dtor_kfunc_tab; + + if (!tab) + return; + kfree(tab); + btf->dtor_kfunc_tab = NULL; +} + static void btf_free(struct btf *btf) { + btf_free_dtor_kfunc_tab(btf); btf_free_kfunc_set_tab(btf); kvfree(btf->types); kvfree(btf->resolved_sizes); @@ -7018,6 +7036,96 @@ int register_btf_kfunc_id_set(enum bpf_prog_type prog_type, } EXPORT_SYMBOL_GPL(register_btf_kfunc_id_set); +s32 btf_find_dtor_kfunc(struct btf *btf, u32 btf_id) +{ + struct btf_id_dtor_kfunc_tab *tab = btf->dtor_kfunc_tab; + struct btf_id_dtor_kfunc *dtor; + + if (!tab) + return -ENOENT; + /* Even though the size of tab->dtors[0] is > sizeof(u32), we only need + * to compare the first u32 with btf_id, so we can reuse btf_id_cmp_func. + */ + BUILD_BUG_ON(offsetof(struct btf_id_dtor_kfunc, btf_id) != 0); + dtor = bsearch(&btf_id, tab->dtors, tab->cnt, sizeof(tab->dtors[0]), btf_id_cmp_func); + if (!dtor) + return -ENOENT; + return dtor->kfunc_btf_id; +} + +/* This function must be invoked only from initcalls/module init functions */ +int register_btf_id_dtor_kfuncs(const struct btf_id_dtor_kfunc *dtors, u32 add_cnt, + struct module *owner) +{ + struct btf_id_dtor_kfunc_tab *tab; + struct btf *btf; + u32 tab_cnt; + int ret; + + btf = btf_get_module_btf(owner); + if (!btf) { + if (!owner && IS_ENABLED(CONFIG_DEBUG_INFO_BTF)) { + pr_err("missing vmlinux BTF, cannot register dtor kfuncs\n"); + return -ENOENT; + } + if (owner && IS_ENABLED(CONFIG_DEBUG_INFO_BTF_MODULES)) { + pr_err("missing module BTF, cannot register dtor kfuncs\n"); + return -ENOENT; + } + return 0; + } + if (IS_ERR(btf)) + return PTR_ERR(btf); + + if (add_cnt >= BTF_DTOR_KFUNC_MAX_CNT) { + pr_err("cannot register more than %d kfunc destructors\n", BTF_DTOR_KFUNC_MAX_CNT); + ret = -E2BIG; + goto end; + } + + tab = btf->dtor_kfunc_tab; + /* Only one call allowed for modules */ + if (WARN_ON_ONCE(tab && btf_is_module(btf))) { + ret = -EINVAL; + goto end; + } + + tab_cnt = tab ? tab->cnt : 0; + if (tab_cnt > U32_MAX - add_cnt) { + ret = -EOVERFLOW; + goto end; + } + if (tab_cnt + add_cnt >= BTF_DTOR_KFUNC_MAX_CNT) { + pr_err("cannot register more than %d kfunc destructors\n", BTF_DTOR_KFUNC_MAX_CNT); + ret = -E2BIG; + goto end; + } + + tab = krealloc(btf->dtor_kfunc_tab, + offsetof(struct btf_id_dtor_kfunc_tab, dtors[tab_cnt + add_cnt]), + GFP_KERNEL | __GFP_NOWARN); + if (!tab) { + ret = -ENOMEM; + goto end; + } + + if (!btf->dtor_kfunc_tab) + tab->cnt = 0; + btf->dtor_kfunc_tab = tab; + + memcpy(tab->dtors + tab->cnt, dtors, add_cnt * sizeof(tab->dtors[0])); + tab->cnt += add_cnt; + + sort(tab->dtors, tab->cnt, sizeof(tab->dtors[0]), btf_id_cmp_func, NULL); + + return 0; +end: + btf_free_dtor_kfunc_tab(btf); + btf_put(btf); + return ret; +} +EXPORT_SYMBOL_GPL(register_btf_id_dtor_kfuncs); + #define MAX_TYPES_ARE_COMPAT_DEPTH 2 static From patchwork Sun Mar 20 15:55:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 12786559 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1E45C433EF for ; Sun, 20 Mar 2022 15:55:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245418AbiCTP5Q (ORCPT ); Sun, 20 Mar 2022 11:57:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50050 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245417AbiCTP5P (ORCPT ); Sun, 20 Mar 2022 11:57:15 -0400 Received: from mail-pj1-x1044.google.com (mail-pj1-x1044.google.com [IPv6:2607:f8b0:4864:20::1044]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0BF8954188 for ; Sun, 20 Mar 2022 08:55:52 -0700 (PDT) Received: by mail-pj1-x1044.google.com with SMTP id mm17-20020a17090b359100b001c6da62a559so2742985pjb.3 for ; Sun, 20 Mar 2022 08:55:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JwA6tXeW4rujADpGQJYx5JHai0TWyLc8dgN6taF15IE=; b=oODt10Q4zjgfRuaAQ2rJq9U1cnxWv8ldbVFjDWqbO5rdjSEJVDRFIxs9j7KoUvckgP clH7n9d3KiY77DpPwpwCsqblQXjlO3zbLuDpy9TB+5q7+2dELNBHbYAyRif6AxSv+HkU hrVm6mZw64UxLJPw9qmnK0+OA5+MRSCkjsO+4gibk4OndSsz0HfHCgv+cYBaeSd6bUKW 3I0JewJPl5WEiyYToExSTVwS2cKXmyQu9yi9/LYzmJrGvkgz14DPdQa2Pj11dOmaNdHQ nAf06fAcSgiJi2RprUbsVYl0oWZVyBwodclig+vGLymCOEwjW8c5XO8JxY1uNzGf2zMM Xo2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JwA6tXeW4rujADpGQJYx5JHai0TWyLc8dgN6taF15IE=; b=TgsTAOM/stExlAehCZRQVB22XzJzUO2nytPgJBD0gUllxK2hlE7NEyW0KDoGrRTfI7 KFJAS5ZaIdzyT5BjhIalp90xkHty1RRkNkPi9GUG8+YzrtGDaw/7j/Ou3Mlaexv9QHFx St/kdmyAZWMZ2ZJXl4ZssRdMr65jpvYMWzLyYtSf+WTBB+V3gi84ezukQcMWh4UpLzem Dk1VtsiMQqdOzsoGYUrQ9qVp+ivUaVttN4MsfJP3/nadn2RceykV5zXfBhxvOZT1CaSl lQQZnCrhEUnx4Iyuul2ndLVdx/76P2whSh/D3Z+4DEaVDcKLPq+e04U/epo1ShfIEYXt tdGw== X-Gm-Message-State: AOAM532oBdPegvU7hL7iWK5FN/AOzhcIQeBXzG12BCNI5C3eNY3GKjaO /YX99uHdgKMZ1bbM767qDNND9Vw+Rww= X-Google-Smtp-Source: ABdhPJx9N7Q7DrxTWpjcAOw6zmGxIr/Y4skbYBYeA7/iemhTVZ2v7gstWXN1w0+X/0ePYR+J2opbXg== X-Received: by 2002:a17:90b:3a86:b0:1c6:5971:5980 with SMTP id om6-20020a17090b3a8600b001c659715980mr25558926pjb.68.1647791751313; Sun, 20 Mar 2022 08:55:51 -0700 (PDT) Received: from localhost ([14.139.187.71]) by smtp.gmail.com with ESMTPSA id l10-20020a056a00140a00b004c55d0dcbd1sm16201688pfu.120.2022.03.20.08.55.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 20 Mar 2022 08:55:51 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8?= =?utf-8?q?rgensen?= , Jesper Dangaard Brouer Subject: [PATCH bpf-next v3 09/13] bpf: Wire up freeing of referenced kptr Date: Sun, 20 Mar 2022 21:25:06 +0530 Message-Id: <20220320155510.671497-10-memxor@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220320155510.671497-1-memxor@gmail.com> References: <20220320155510.671497-1-memxor@gmail.com> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=14749; h=from:subject; bh=bsGWMu7O7IOIOHzd460LROXQUnokTF+Q5w3+3PEv6WE=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBiN00yf5ueOdmNO55VEb72j7L13mzQ7326cCT7QVBE bZrgY/iJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCYjdNMgAKCRBM4MiGSL8Ryu8oD/ 0Q0HRU2E5HagZFmjHnovF5rHIPaAzlVBvL01uPKgXu5bOFepQyQM/cXOOP9/GC/9NRRQBUJfOA+gTs PLwQ4Sa4KU7Ytnu4u06/5WK283LoN8GuEeISZmdPNfKFU+hO7JbgJLHBj5wKPiWJiJk+iEEEz3pkS5 2SPXu/YAel4GyJSsF1j2l+cOO7yc7dN/eH+qGV+mL2Zh4qFeMeni8pURSBro8wQWmR/n3mkQaeMzdQ RBfDs124cFB4SRzgEIG1CcNJMsNwtJO0J6JPwUcKG8j/XSrGrom/RVk2Whcrh5z5QSe/763uTkiGGA h2S4elmcNYLTheotRbWZPpS0FOPsYAayZACOS1exvi+4ncNDZaQkVXuyRuHGmUXLRlbCOupTldYfEY Hc6nlLcBHi7gdnRHXvLSi8xBVbkKTfrHclvboMPjnsFhhox3GQa+qZI5gButdbSxeDLJbK7UZ0EAAK o6fqW1MMrTjLo+ZdCLuTRcpEehuWeyh7DL7D/WL7EtyXH6ysap54yXbzPYpMldcvbW5qSMoYXaAk/Z npqCdmKQj3hFLsZhzTDYe6O3addLoFr4uyVf39UNn0e290oB8y2aE7RGFYsVBFZCkDL040urLmSwTB LxTGYuveCHP7GBoXSEt3IKr4fciJ8eOjwI1htLrAJ/qajJqrzUF1UWNqRVaw== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net A destructor kfunc can be defined as void func(type *), where type may be void or any other pointer type as per convenience. In this patch, we ensure that the type is sane and capture the function pointer into off_desc of ptr_off_tab for the specific pointer offset, with the invariant that the dtor pointer is always set when 'kptr_ref' tag is applied to the pointer's pointee type, which is indicated by the flag BPF_MAP_VALUE_OFF_F_REF. Note that only BTF IDs whose destructor kfunc is registered, thus become the allowed BTF IDs for embedding as referenced kptr. Hence it serves the purpose of finding dtor kfunc BTF ID, as well acting as a check against the whitelist of allowed BTF IDs for this purpose. Finally, wire up the actual freeing of the referenced pointer if any at all available offsets, so that no references are leaked after the BPF map goes away and the BPF program previously moved the ownership a referenced pointer into it. The behavior is similar to BPF timers, where bpf_map_{update,delete}_elem will free any existing referenced kptr. The same case is with LRU map's bpf_lru_push_free/htab_lru_push_free functions, which are extended to reset unreferenced and free referenced kptr. Note that unlike BPF timers, kptr is not reset or freed when map uref drops to zero. Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/bpf.h | 4 ++ include/linux/btf.h | 2 + kernel/bpf/arraymap.c | 14 ++++++- kernel/bpf/btf.c | 86 ++++++++++++++++++++++++++++++++++++++++++- kernel/bpf/hashtab.c | 29 ++++++++++----- kernel/bpf/syscall.c | 57 +++++++++++++++++++++++++--- 6 files changed, 173 insertions(+), 19 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 6474d2d44b78..ae52602fdfbf 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -23,6 +23,7 @@ #include #include #include +#include struct bpf_verifier_env; struct bpf_verifier_log; @@ -172,6 +173,8 @@ struct bpf_map_value_off_desc { u32 offset; u32 btf_id; struct btf *btf; + struct module *module; + btf_dtor_kfunc_t dtor; int flags; }; @@ -1551,6 +1554,7 @@ struct bpf_map_value_off_desc *bpf_map_kptr_off_contains(struct bpf_map *map, u3 void bpf_map_free_kptr_off_tab(struct bpf_map *map); struct bpf_map_value_off *bpf_map_copy_kptr_off_tab(const struct bpf_map *map); bool bpf_map_equal_kptr_off_tab(const struct bpf_map *map_a, const struct bpf_map *map_b); +void bpf_map_free_kptr(struct bpf_map *map, void *map_value); struct bpf_map *bpf_map_get(u32 ufd); struct bpf_map *bpf_map_get_with_uref(u32 ufd); diff --git a/include/linux/btf.h b/include/linux/btf.h index ff4be49b7a26..8acf728c8616 100644 --- a/include/linux/btf.h +++ b/include/linux/btf.h @@ -45,6 +45,8 @@ struct btf_id_dtor_kfunc { u32 kfunc_btf_id; }; +typedef void (*btf_dtor_kfunc_t)(void *); + extern const struct file_operations btf_fops; void btf_get(struct btf *btf); diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c index 7f145aefbff8..3cc2884321e7 100644 --- a/kernel/bpf/arraymap.c +++ b/kernel/bpf/arraymap.c @@ -287,10 +287,12 @@ static int array_map_get_next_key(struct bpf_map *map, void *key, void *next_key return 0; } -static void check_and_free_timer_in_array(struct bpf_array *arr, void *val) +static void check_and_free_timer_and_kptr(struct bpf_array *arr, void *val) { if (unlikely(map_value_has_timer(&arr->map))) bpf_timer_cancel_and_free(val + arr->map.timer_off); + if (unlikely(map_value_has_kptr(&arr->map))) + bpf_map_free_kptr(&arr->map, val); } /* Called from syscall or from eBPF program */ @@ -327,7 +329,7 @@ static int array_map_update_elem(struct bpf_map *map, void *key, void *value, copy_map_value_locked(map, val, value, false); else copy_map_value(map, val, value); - check_and_free_timer_in_array(array, val); + check_and_free_timer_and_kptr(array, val); } return 0; } @@ -386,6 +388,7 @@ static void array_map_free_timers(struct bpf_map *map) struct bpf_array *array = container_of(map, struct bpf_array, map); int i; + /* We don't reset or free kptr on uref dropping to zero. */ if (likely(!map_value_has_timer(map))) return; @@ -398,6 +401,13 @@ static void array_map_free_timers(struct bpf_map *map) static void array_map_free(struct bpf_map *map) { struct bpf_array *array = container_of(map, struct bpf_array, map); + int i; + + if (unlikely(map_value_has_kptr(map))) { + for (i = 0; i < array->map.max_entries; i++) + bpf_map_free_kptr(map, array->value + array->elem_size * i); + bpf_map_free_kptr_off_tab(map); + } if (array->map.map_type == BPF_MAP_TYPE_PERCPU_ARRAY) bpf_array_free_percpu(array); diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 9cb6f61a50a7..6227c1be6326 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -3408,6 +3408,7 @@ struct bpf_map_value_off *btf_find_kptr(const struct btf *btf, /* btf_find_field requires array of size max + 1 */ struct btf_field_info info_arr[BPF_MAP_VALUE_OFF_MAX + 1]; struct bpf_map_value_off *tab; + struct module *mod = NULL; int ret, i, nr_off; /* Revisit stack usage when bumping BPF_MAP_VALUE_OFF_MAX */ @@ -3438,16 +3439,99 @@ struct bpf_map_value_off *btf_find_kptr(const struct btf *btf, goto end; } + /* Find and stash the function pointer for the destruction function that + * needs to be eventually invoked from the map free path. + */ + if (info_arr[i].flags & BPF_MAP_VALUE_OFF_F_REF) { + const struct btf_type *dtor_func, *dtor_func_proto; + const struct btf_param *args; + const char *dtor_func_name; + unsigned long addr; + s32 dtor_btf_id; + u32 nr_args; + + /* This call also serves as a whitelist of allowed objects that + * can be used as a referenced pointer and be stored in a map at + * the same time. + */ + dtor_btf_id = btf_find_dtor_kfunc(off_btf, id); + if (dtor_btf_id < 0) { + ret = dtor_btf_id; + btf_put(off_btf); + goto end; + } + + dtor_func = btf_type_by_id(off_btf, dtor_btf_id); + if (!dtor_func || !btf_type_is_func(dtor_func)) { + ret = -EINVAL; + btf_put(off_btf); + goto end; + } + + dtor_func_proto = btf_type_by_id(off_btf, dtor_func->type); + if (!dtor_func_proto || !btf_type_is_func_proto(dtor_func_proto)) { + ret = -EINVAL; + btf_put(off_btf); + goto end; + } + + /* Make sure the prototype of the destructor kfunc is 'void func(type *)' */ + t = btf_type_by_id(off_btf, dtor_func_proto->type); + if (!t || !btf_type_is_void(t)) { + ret = -EINVAL; + btf_put(off_btf); + goto end; + } + + nr_args = btf_type_vlen(dtor_func_proto); + args = btf_params(dtor_func_proto); + + t = NULL; + if (nr_args) + t = btf_type_by_id(off_btf, args[0].type); + /* Allow any pointer type, as width on targets Linux supports + * will be same for all pointer types (i.e. sizeof(void *)) + */ + if (nr_args != 1 || !t || !btf_type_is_ptr(t)) { + ret = -EINVAL; + btf_put(off_btf); + goto end; + } + + if (btf_is_module(btf)) { + mod = btf_try_get_module(off_btf); + if (!mod) { + ret = -ENXIO; + btf_put(off_btf); + goto end; + } + } + + dtor_func_name = __btf_name_by_offset(off_btf, dtor_func->name_off); + addr = kallsyms_lookup_name(dtor_func_name); + if (!addr) { + ret = -EINVAL; + module_put(mod); + btf_put(off_btf); + goto end; + } + tab->off[i].dtor = (void *)addr; + } + tab->off[i].offset = info_arr[i].off; tab->off[i].btf_id = id; tab->off[i].btf = off_btf; + tab->off[i].module = mod; tab->off[i].flags = info_arr[i].flags; tab->nr_off = i + 1; } return tab; end: - while (tab->nr_off--) + while (tab->nr_off--) { btf_put(tab->off[tab->nr_off].btf); + if (tab->off[tab->nr_off].module) + module_put(tab->off[tab->nr_off].module); + } kfree(tab); return ERR_PTR(ret); } diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index 65877967f414..fa4a0a8754c5 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -725,12 +725,16 @@ static int htab_lru_map_gen_lookup(struct bpf_map *map, return insn - insn_buf; } -static void check_and_free_timer(struct bpf_htab *htab, struct htab_elem *elem) +static void check_and_free_timer_and_kptr(struct bpf_htab *htab, + struct htab_elem *elem, + bool free_kptr) { + void *map_value = elem->key + round_up(htab->map.key_size, 8); + if (unlikely(map_value_has_timer(&htab->map))) - bpf_timer_cancel_and_free(elem->key + - round_up(htab->map.key_size, 8) + - htab->map.timer_off); + bpf_timer_cancel_and_free(map_value + htab->map.timer_off); + if (unlikely(map_value_has_kptr(&htab->map)) && free_kptr) + bpf_map_free_kptr(&htab->map, map_value); } /* It is called from the bpf_lru_list when the LRU needs to delete @@ -757,7 +761,7 @@ static bool htab_lru_map_delete_node(void *arg, struct bpf_lru_node *node) hlist_nulls_for_each_entry_rcu(l, n, head, hash_node) if (l == tgt_l) { hlist_nulls_del_rcu(&l->hash_node); - check_and_free_timer(htab, l); + check_and_free_timer_and_kptr(htab, l, true); break; } @@ -829,7 +833,7 @@ static void htab_elem_free(struct bpf_htab *htab, struct htab_elem *l) { if (htab->map.map_type == BPF_MAP_TYPE_PERCPU_HASH) free_percpu(htab_elem_get_ptr(l, htab->map.key_size)); - check_and_free_timer(htab, l); + check_and_free_timer_and_kptr(htab, l, true); kfree(l); } @@ -857,7 +861,7 @@ static void free_htab_elem(struct bpf_htab *htab, struct htab_elem *l) htab_put_fd_value(htab, l); if (htab_is_prealloc(htab)) { - check_and_free_timer(htab, l); + check_and_free_timer_and_kptr(htab, l, true); __pcpu_freelist_push(&htab->freelist, &l->fnode); } else { atomic_dec(&htab->count); @@ -1104,7 +1108,7 @@ static int htab_map_update_elem(struct bpf_map *map, void *key, void *value, if (!htab_is_prealloc(htab)) free_htab_elem(htab, l_old); else - check_and_free_timer(htab, l_old); + check_and_free_timer_and_kptr(htab, l_old, true); } ret = 0; err: @@ -1114,7 +1118,7 @@ static int htab_map_update_elem(struct bpf_map *map, void *key, void *value, static void htab_lru_push_free(struct bpf_htab *htab, struct htab_elem *elem) { - check_and_free_timer(htab, elem); + check_and_free_timer_and_kptr(htab, elem, true); bpf_lru_push_free(&htab->lru, &elem->lru_node); } @@ -1420,7 +1424,10 @@ static void htab_free_malloced_timers(struct bpf_htab *htab) struct htab_elem *l; hlist_nulls_for_each_entry(l, n, head, hash_node) - check_and_free_timer(htab, l); + /* We don't reset or free kptr on uref dropping to zero, + * hence set free_kptr to false. + */ + check_and_free_timer_and_kptr(htab, l, false); cond_resched_rcu(); } rcu_read_unlock(); @@ -1430,6 +1437,7 @@ static void htab_map_free_timers(struct bpf_map *map) { struct bpf_htab *htab = container_of(map, struct bpf_htab, map); + /* We don't reset or free kptr on uref dropping to zero. */ if (likely(!map_value_has_timer(&htab->map))) return; if (!htab_is_prealloc(htab)) @@ -1458,6 +1466,7 @@ static void htab_map_free(struct bpf_map *map) else prealloc_destroy(htab); + bpf_map_free_kptr_off_tab(map); free_percpu(htab->extra_elems); bpf_map_area_free(htab->buckets); for (i = 0; i < HASHTAB_MAP_LOCK_COUNT; i++) diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 7b32537bd81f..3901a049fe2a 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -508,8 +508,11 @@ void bpf_map_free_kptr_off_tab(struct bpf_map *map) if (!map_value_has_kptr(map)) return; for (i = 0; i < tab->nr_off; i++) { + struct module *mod = tab->off[i].module; struct btf *btf = tab->off[i].btf; + if (mod) + module_put(mod); btf_put(btf); } kfree(tab); @@ -524,8 +527,16 @@ struct bpf_map_value_off *bpf_map_copy_kptr_off_tab(const struct bpf_map *map) if (!map_value_has_kptr(map)) return ERR_PTR(-ENOENT); /* Do a deep copy of the kptr_off_tab */ - for (i = 0; i < tab->nr_off; i++) - btf_get(tab->off[i].btf); + for (i = 0; i < tab->nr_off; i++) { + struct module *mod = tab->off[i].module; + struct btf *btf = tab->off[i].btf; + + if (mod && !try_module_get(mod)) { + ret = -ENXIO; + goto end; + } + btf_get(btf); + } size = offsetof(struct bpf_map_value_off, off[tab->nr_off]); new_tab = kzalloc(size, GFP_KERNEL | __GFP_NOWARN); @@ -536,8 +547,14 @@ struct bpf_map_value_off *bpf_map_copy_kptr_off_tab(const struct bpf_map *map) memcpy(new_tab, tab, size); return new_tab; end: - while (i--) - btf_put(tab->off[i].btf); + while (i--) { + struct module *mod = tab->off[i].module; + struct btf *btf = tab->off[i].btf; + + if (mod) + module_put(mod); + btf_put(btf); + } return ERR_PTR(ret); } @@ -557,15 +574,43 @@ bool bpf_map_equal_kptr_off_tab(const struct bpf_map *map_a, const struct bpf_ma return !memcmp(tab_a, tab_b, size); } +/* Caller must ensure map_value_has_kptr is true. Note that this function can be + * called on a map value while the map_value is visible to BPF programs, as it + * ensures the correct synchronization, and we already enforce the same using + * the verifier on the BPF program side, esp. for referenced pointers. + */ +void bpf_map_free_kptr(struct bpf_map *map, void *map_value) +{ + struct bpf_map_value_off *tab = map->kptr_off_tab; + unsigned long *btf_id_ptr; + int i; + + for (i = 0; i < tab->nr_off; i++) { + struct bpf_map_value_off_desc *off_desc = &tab->off[i]; + unsigned long old_ptr; + + btf_id_ptr = map_value + off_desc->offset; + if (!(off_desc->flags & BPF_MAP_VALUE_OFF_F_REF)) { + u64 *p = (u64 *)btf_id_ptr; + + WRITE_ONCE(p, 0); + continue; + } + old_ptr = xchg(btf_id_ptr, 0); + off_desc->dtor((void *)old_ptr); + } +} + /* called from workqueue */ static void bpf_map_free_deferred(struct work_struct *work) { struct bpf_map *map = container_of(work, struct bpf_map, work); security_bpf_map_free(map); - bpf_map_free_kptr_off_tab(map); bpf_map_release_memcg(map); - /* implementation dependent freeing */ + /* implementation dependent freeing, map_free callback also does + * bpf_map_free_kptr_off_tab, if needed. + */ map->ops->map_free(map); } From patchwork Sun Mar 20 15:55:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 12786560 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B7542C433EF for ; Sun, 20 Mar 2022 15:55:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245417AbiCTP5T (ORCPT ); Sun, 20 Mar 2022 11:57:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50258 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245415AbiCTP5S (ORCPT ); Sun, 20 Mar 2022 11:57:18 -0400 Received: from mail-pf1-x443.google.com (mail-pf1-x443.google.com [IPv6:2607:f8b0:4864:20::443]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 687B554188 for ; Sun, 20 Mar 2022 08:55:55 -0700 (PDT) Received: by mail-pf1-x443.google.com with SMTP id p5so7455183pfo.5 for ; Sun, 20 Mar 2022 08:55:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=C0mLN62+cRxqSBr0NxuA9nFQnj29ROBC80G+ZDENw88=; b=kb0URj5ScrM8fx4rqxkSfOcTvDF/2G7B7KJJlTSPDLIv0DQmV0b1lEmTL1hqI1Oqpb FTrtXgD2HYtUvYMxPWUY5XOVQE+uTWFHAhpALaQ9I8nzQL4JQrLB2DuYzVFBwEIwpSm7 SVnzWITB5fReeL6A1jKW1WQm/qWFQ5xg2F83VvgW4nNUNAdz9lCpjtz2cG+Epso8RCNe X6nxhkK4a34k0EZnhZsqX8fN3HXPEquQzLe0BdYRZw1uH1nL0KJQM7CyAm46qHhXWrVP a4yBE7KjM7QvDChjFrFp1JMbhXqwQxDLIoEWW/vJHYH1WIEgoPh5T+0C6B2DoqWcGOW0 Huww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=C0mLN62+cRxqSBr0NxuA9nFQnj29ROBC80G+ZDENw88=; b=JDZKtMmu4vn+A8S/4Y9EouP8Dd0GKoumCDnwjygn03wxetjUPSaz9u6xgBpMS3Gzhf puYGNLd7yYjkd18ED8N1yY83vmmi+M8gPCM9ykqNq81hhLz7hRab2sC3fPKEwMt7oC+4 nJPiVg3kY30QtRz2EmMcnG8NnvjkMh88XIM73bj0gU91nl6ze224QD3g7Kvq3CLf6Xt8 rUptsbIfWFl+3DqVns7TXe1u3iwOQPxWIpcLzy2OQyHTU8ApNJrUvE8pH6KEFcp2UJH9 X2I0F+X6BUu7D/jhFoi76fGfJhe7z5ZIKuGtqJVbe6dtBkCslHNNhhAxu3mATA56R1h6 Nhhw== X-Gm-Message-State: AOAM53318W+CF11M3CeBorHgd7+UaUFWWbpH1aliVdpppnw5Qo9V/U1x sW0WV7Q4/6Y0u22dsnkZGYvvIyjdD7Y= X-Google-Smtp-Source: ABdhPJxpiYpp8RK16IjPEcxdYwPcfgQDHnDhEwe+4xXuyBM1gC13r94hrXTVQoR+rVsCmShRkJVV8A== X-Received: by 2002:a62:1ad3:0:b0:4fa:9adc:7680 with SMTP id a202-20020a621ad3000000b004fa9adc7680mr1960803pfa.25.1647791754759; Sun, 20 Mar 2022 08:55:54 -0700 (PDT) Received: from localhost ([14.139.187.71]) by smtp.gmail.com with ESMTPSA id x2-20020a63aa42000000b0038265eb2495sm2521405pgo.88.2022.03.20.08.55.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 20 Mar 2022 08:55:54 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8?= =?utf-8?q?rgensen?= , Jesper Dangaard Brouer Subject: [PATCH bpf-next v3 10/13] bpf: Teach verifier about kptr_get kfunc helpers Date: Sun, 20 Mar 2022 21:25:07 +0530 Message-Id: <20220320155510.671497-11-memxor@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220320155510.671497-1-memxor@gmail.com> References: <20220320155510.671497-1-memxor@gmail.com> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5735; h=from:subject; bh=RMnLYSm74flSd/a5yI436W0bZ7O4OfWVX+/9PxpWEus=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBiN00yVf1FFW26pPQ3GTDFzGhTyjXGgvKzVqMNbJzh 0Aa1K9+JAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCYjdNMgAKCRBM4MiGSL8RytHQD/ 9w2qFmDC2HlIVAzYYbaab3IEDdVZCdFcznMMjs8Fqdg4Q5wxlylf5Qhz7sGwc48MBnFe5kkQKKnWJ2 j17iRMiNSgTRT1H5ZvcmKMj5zmhLUdZ1EoY3yMMH6OSQTorY/WI0ErPYH7qJdFskIQzN+78sNdQhqQ RFqv2WoCMVHhO2KfeallNK519dezloE4lAvIn7s1tEi48EQP+UX5PhXqtsi6ZdaZTczAcPsWwWcjdo 4XGuUFFMPIV9QuXEHJvrzEiMit4lf/RQVGot4xf1H/V0SjFSKJlEKokKB/yzMcFlWaO/uX8N1qr7rL DvOaQg8l1JvWkb+ZGl6cPK1zXa+P7JBCHqNlWqhpEcbGQsp1wbXflWmkMk/YMrO5p0HaYKU2H5YX43 jRxgiInOnlqQ/D8RGUBPwbyBqVu5OCbCGKYXKdqJ0Fcdra5Jqs8ASuk7Wnv4UuLF89rVWN920jXgs6 LNCuCiYkSntN/R1GNiTTL+fUSGTwfYQFRei2t97vaVwZMmEkTtqcjc9u92KOblquO1EavzqrUhT/Kl 3IrDFU1KERuBTETHePbe7KpyTaOmHSi6VU79DmI7y6fQhLfXy25lrhdEjsZMP7YaXhdlNnjA4X46PI KTr+NVRk+rRucNQ74bYUSC7hJJ+vM/9OS5Yie0vDlH44NAawlsGq/3je/Bdw== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net We introduce a new style of kfunc helpers, namely *_kptr_get, where they take pointer to the map value which points to a referenced kernel pointer contained in the map. Since this is referenced, only bpf_kptr_xchg from BPF side and xchg from kernel side is allowed to change the current value, and each pointer that resides in that location would be referenced, and RCU protected (this must be kept in mind while adding kernel types embeddable as reference kptr in BPF maps). This means that if do the load of the pointer value in an RCU read section, and find a live pointer, then as long as we hold RCU read lock, it won't be freed by a parallel xchg + release operation. This allows us to implement a safe refcount increment scheme. Hence, enforce that first argument of all such kfunc is a proper PTR_TO_MAP_VALUE pointing at the right offset to referenced pointer. For the rest of the arguments, they are subjected to typical kfunc argument checks, hence allowing some flexibility in passing more intent into how the reference should be taken. For instance, in case of struct nf_conn, it is not freed until RCU grace period ends, but can still be reused for another tuple once refcount has dropped to zero. Hence, a bpf_ct_kptr_get helper not only needs to call refcount_inc_not_zero, but also do a tuple match after incrementing the reference, and when it fails to match it, put the reference again and return NULL. This can be implemented easily if we allow passing additional parameters to the bpf_ct_kptr_get kfunc, like a struct bpf_sock_tuple * and a tuple__sz pair. Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/btf.h | 2 ++ kernel/bpf/btf.c | 61 +++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 58 insertions(+), 5 deletions(-) diff --git a/include/linux/btf.h b/include/linux/btf.h index 8acf728c8616..d5d37bfde8df 100644 --- a/include/linux/btf.h +++ b/include/linux/btf.h @@ -17,6 +17,7 @@ enum btf_kfunc_type { BTF_KFUNC_TYPE_ACQUIRE, BTF_KFUNC_TYPE_RELEASE, BTF_KFUNC_TYPE_RET_NULL, + BTF_KFUNC_TYPE_KPTR_ACQUIRE, BTF_KFUNC_TYPE_MAX, }; @@ -35,6 +36,7 @@ struct btf_kfunc_id_set { struct btf_id_set *acquire_set; struct btf_id_set *release_set; struct btf_id_set *ret_null_set; + struct btf_id_set *kptr_acquire_set; }; struct btf_id_set *sets[BTF_KFUNC_TYPE_MAX]; }; diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 6227c1be6326..e11e44f83301 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -6060,11 +6060,11 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, struct bpf_verifier_log *log = &env->log; u32 i, nargs, ref_id, ref_obj_id = 0; bool is_kfunc = btf_is_kernel(btf); + bool rel = false, kptr_get = false; const char *func_name, *ref_tname; const struct btf_type *t, *ref_t; const struct btf_param *args; int ref_regno = 0, ret; - bool rel = false; t = btf_type_by_id(btf, func_id); if (!t || !btf_type_is_func(t)) { @@ -6090,10 +6090,14 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, return -EINVAL; } - /* Only kfunc can be release func */ - if (is_kfunc) + if (is_kfunc) { + /* Only kfunc can be release func */ rel = btf_kfunc_id_set_contains(btf, resolve_prog_type(env->prog), BTF_KFUNC_TYPE_RELEASE, func_id); + kptr_get = btf_kfunc_id_set_contains(btf, resolve_prog_type(env->prog), + BTF_KFUNC_TYPE_KPTR_ACQUIRE, func_id); + } + /* check that BTF function arguments match actual types that the * verifier sees. */ @@ -6122,8 +6126,55 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, if (ret < 0) return ret; - if (btf_get_prog_ctx_type(log, btf, t, - env->prog->type, i)) { + /* kptr_get is only true for kfunc */ + if (i == 0 && kptr_get) { + struct bpf_map_value_off_desc *off_desc; + + if (reg->type != PTR_TO_MAP_VALUE) { + bpf_log(log, "arg#0 expected pointer to map value\n"); + return -EINVAL; + } + + ret = check_mem_reg(env, reg, regno, sizeof(u64)); + if (ret < 0) + return ret; + + /* check_func_arg_reg_off allows var_off for + * PTR_TO_MAP_VALUE, but we need fixed offset to find + * off_desc. + */ + if (!tnum_is_const(reg->var_off)) { + bpf_log(log, "arg#0 must have constant offset\n"); + return -EINVAL; + } + + off_desc = bpf_map_kptr_off_contains(reg->map_ptr, reg->off + reg->var_off.value); + if (!off_desc || !(off_desc->flags & BPF_MAP_VALUE_OFF_F_REF)) { + bpf_log(log, "arg#0 no referenced kptr at map value offset=%llu\n", + reg->off + reg->var_off.value); + return -EINVAL; + } + + if (!btf_type_is_ptr(ref_t)) { + bpf_log(log, "arg#0 BTF type must be a double pointer\n"); + return -EINVAL; + } + + ref_t = btf_type_skip_modifiers(btf, ref_t->type, &ref_id); + ref_tname = btf_name_by_offset(btf, ref_t->name_off); + + if (!btf_type_is_struct(ref_t)) { + bpf_log(log, "kernel function %s args#%d pointer type %s %s is not supported\n", + func_name, i, btf_type_str(ref_t), ref_tname); + return -EINVAL; + } + if (!btf_struct_ids_match(log, btf, ref_id, 0, off_desc->btf, off_desc->btf_id)) { + bpf_log(log, "kernel function %s args#%d expected pointer to %s %s\n", + func_name, i, btf_type_str(ref_t), ref_tname); + return -EINVAL; + } + /* rest of the arguments can be anything, like normal kfunc */ + } else if (btf_get_prog_ctx_type(log, btf, t, env->prog->type, i)) { /* If function expects ctx type in BTF check that caller * is passing PTR_TO_CTX. */ From patchwork Sun Mar 20 15:55:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 12786561 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 631C4C433F5 for ; Sun, 20 Mar 2022 15:56:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245420AbiCTP5W (ORCPT ); Sun, 20 Mar 2022 11:57:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50480 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245415AbiCTP5V (ORCPT ); Sun, 20 Mar 2022 11:57:21 -0400 Received: from mail-pl1-x643.google.com (mail-pl1-x643.google.com [IPv6:2607:f8b0:4864:20::643]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D6ADE54188 for ; Sun, 20 Mar 2022 08:55:58 -0700 (PDT) Received: by mail-pl1-x643.google.com with SMTP id d18so10792109plr.6 for ; Sun, 20 Mar 2022 08:55:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=pZfLmYbolHwmqXMfr7nqaaE34g3CnbTOWtScfGan094=; b=BrDwSz1lGS0KMZSQbROcqoOPNIAd6cd07xLHzZKwkW2k4h4G+mm9Wrj7swH1CV9v6n zo1iti3WpipaSfNmk2jAS5Cq2D/TO+vP0WIjThSuTM8zcdHICtAw/nRseV3Lvj9SVQkx 7W0zjXyOyv/Bhl4Gr+1Tj9nbo24tQMfQK/vtdPxwANOZlaKIgYdeCW8D6AdYOLu0n0Do cl7PYCv+dvzRUk17HYmy1DVIya58ugM/e/Q+T639lkXBfsLdHziOM2+ie2sE0yjdAaDs PpRpU5uSVH2cK4UCyWGavM9DaInc8Y+S5FPp9HBuAcaOK7/InN2anNNXn1PmlT220wF7 JvuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pZfLmYbolHwmqXMfr7nqaaE34g3CnbTOWtScfGan094=; b=e+dJi2x7Y7IdSiBDr4JnBhkjIXzoJgnHXXyvOFyFmyaI92T/1cA4ci/GOdhUC/A7zU bDbTY4G3UjlSkp9+1sMy5a5/ogEX2y6uS/SDEelnawzNg41ljgHXlcZgDWHUhAsGhFX8 71kWfKCkTcUxgR9lyTMy49kBBI7RmJYhgAHhgg2+riENCea3MTBU33InZ5ye9YzRkqFv qou60yzS31Vt7b5ABHoU5QViHN3+f2jG0o3BETZGu9LdzaSnMouD3gMX39h+2XPQDgdG zIvSFo8TSIeFe/aggeCll14RLvz+yj+vLV3pFACjVkBbIo3BQIW6bhzDl7orcvOIRA3n PhgA== X-Gm-Message-State: AOAM531TMO9PtrWScZapVGMJWf+XYDmcGe2aCAJSij/Ab8LACxub+KeL JiH9S4tiPH1NEaAtrMtZnsVVfkBWVyE= X-Google-Smtp-Source: ABdhPJxEuWA9N/Ym2yNCQbI9ehYzm50+vjXhDmHvmJmcoG404U+lYYuKFJeIPUjRirBw8ecTR4859w== X-Received: by 2002:a17:902:8b87:b0:14b:47b3:c0a2 with SMTP id ay7-20020a1709028b8700b0014b47b3c0a2mr8957654plb.51.1647791758279; Sun, 20 Mar 2022 08:55:58 -0700 (PDT) Received: from localhost ([14.139.187.71]) by smtp.gmail.com with ESMTPSA id m17-20020a17090a7f9100b001b9e4d62ed0sm17336506pjl.13.2022.03.20.08.55.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 20 Mar 2022 08:55:58 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8?= =?utf-8?q?rgensen?= , Jesper Dangaard Brouer Subject: [PATCH bpf-next v3 11/13] libbpf: Add kptr type tag macros to bpf_helpers.h Date: Sun, 20 Mar 2022 21:25:08 +0530 Message-Id: <20220320155510.671497-12-memxor@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220320155510.671497-1-memxor@gmail.com> References: <20220320155510.671497-1-memxor@gmail.com> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=907; h=from:subject; bh=dtVrckouAMHnmEAd4Rr2OdBAilXX+AUoTjGvaGbi9os=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBiN00ysrejQAaub14DT4bq6Zig06TX+s1CkhTq0CzC 8ntkqcuJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCYjdNMgAKCRBM4MiGSL8RylDqD/ 476zJfYd3lkGWpUG5RTaArkBIkbPvBL7Fh5e2VR1SlYRJwamRhArVY04KaO8I0bLOM3Uiyl4+npsma FibxdZzJSKThEmt6xea7fFJAA4HZG4AZo/RFaz9EL1yBZF57iDzliG7ngGMWlBhcxLeGgYFDCf4rMO shnJ3FnB93pFg8L+QCFi881ZCg5TqAi2UiHHL6RZ6TjGsSTNV1idK7wRCGXnv0pcBWlbd9IosuQmII L36HbtSspS98vY4FSWUfdczHWj0gd92bvFbszIBOg9rI4kPjKUvkWLOLuQYNZvMemQPEVpTKd7NH8n nuSB1R8EbzlDLmDDWwy94SrqG0DGim1dyaomdXINE8XRIeeeqDyyleRLGZwnyWcWLzmYNx6wuR94yw /pkRgUauiBKDyCM5JIquD7hJ8tEzeiBpQ3JZTQ3bb+ArDpGqBfbjBvxxpa5azn3er3Y0UE3bRoAsZe bt5KUXx9GzLt6tckUKFf3XUIFxW9wvZwSxzVu7t6TmF9n0T0VAUPopEyRfeO1Y9lXRjd1Hb+7asdFR gh+FKVITUcw5q7EWZYNb/v2EuLDVnkIjYyzWKleK1txTO6vtpnoq+Lj+kbgj8uLG2xBsnF09ZNTwC/ /VYqoV0NFFA8mrHBP/KuaHJ1jgucEsTo+zoLwfjZ3WagujrUrokZEjLi6qXA== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Include convenience definitions: __kptr: Unreferenced BTF ID pointer __kptr_ref: Referenced BTF ID pointer Users can use them to tag the pointer type meant to be used with the new support directly in the map value definition. Signed-off-by: Kumar Kartikeya Dwivedi --- tools/lib/bpf/bpf_helpers.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tools/lib/bpf/bpf_helpers.h b/tools/lib/bpf/bpf_helpers.h index 44df982d2a5c..bbae9a057bc8 100644 --- a/tools/lib/bpf/bpf_helpers.h +++ b/tools/lib/bpf/bpf_helpers.h @@ -149,6 +149,8 @@ enum libbpf_tristate { #define __kconfig __attribute__((section(".kconfig"))) #define __ksym __attribute__((section(".ksyms"))) +#define __kptr __attribute__((btf_type_tag("kptr"))) +#define __kptr_ref __attribute__((btf_type_tag("kptr_ref"))) #ifndef ___bpf_concat #define ___bpf_concat(a, b) a ## b From patchwork Sun Mar 20 15:55:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 12786562 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C9256C433EF for ; Sun, 20 Mar 2022 15:56:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245427AbiCTP5f (ORCPT ); Sun, 20 Mar 2022 11:57:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51456 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245436AbiCTP5d (ORCPT ); Sun, 20 Mar 2022 11:57:33 -0400 Received: from mail-pf1-x441.google.com (mail-pf1-x441.google.com [IPv6:2607:f8b0:4864:20::441]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8D7F554697 for ; Sun, 20 Mar 2022 08:56:03 -0700 (PDT) Received: by mail-pf1-x441.google.com with SMTP id s11so13523170pfu.13 for ; Sun, 20 Mar 2022 08:56:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=c0rlVkmTXLMsV3hnUEY6r2LXk7kBTALA/cJR/7tljik=; b=cfit49q3BIadHQ/j3nsC3gUTWqzHNDEzMq9nSob/zCBL45bhuEsstSZiOl9ynbznCP xH09rKqCGQArvmZmnfiasvMzGo0cVw19hiBc0bpIHE0TXSJegmx4GyxDmWOhoHCP2xaU pWvsMFzCiuL46B9X5oEMNpZVE7xxY3ER1vgnXxNFaPF4olK5/ciA0EIXqPGy3xFYEgnk rXPrL/TFBOR18NwcrxUffd7/dcUpBdD0NfFZSItENJ95YhOl8p5vn0ejiQ4zEqoy5Xjp C4NR2EMymZ1jarrymD131PzullinSgoyv+LNDlGGB1QgY4qrFt8ZRaQxoix2mr/AbjPD 1XZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=c0rlVkmTXLMsV3hnUEY6r2LXk7kBTALA/cJR/7tljik=; b=A7IIMS2mEZ6YgamUB5iTergGwyP0gGWWXTo6nCQ8eXps986X8S64wArizcAjOn+78v co1Y0YLQbmY9+SQmLOUXThl70i2YqJn/yC9HJeSDg7inkavl1Bxt2XTh2s04AuoOxSSF GmVBjsMATkNvmcg8Iw+4UVq3GRFSv9hYlxr1gP4TVm4wE2YC73KM/3VntkiaOVNy4Mdx VTyCCb5zMhhwEfS9LNahymc0iKVl6Hz3YI8y/hBXTkxzjs9SyzAdBTZ8Xc2/wElaDISa avsiJlxX3M41DdoY9bucxWSiQuX17rkG5GqiMhPUptnMSq/URLb79jLvZkTnizVozqwi 3qCQ== X-Gm-Message-State: AOAM533sdoTyO04P1SQEVtGYs6Zjo6jPCITk2YfEmF47TYBog6/CbldC 8qKOkctLJ9dLnSuSC96Uf16RNNBzsVk= X-Google-Smtp-Source: ABdhPJyuc9FI+j0CKLRtszcW2MlqWzGr5Glw25aXQUeb4j6eHintqIH99T6pnvTrOlqfbVTXOpXRgg== X-Received: by 2002:a62:38d1:0:b0:4fa:80ad:bf5e with SMTP id f200-20020a6238d1000000b004fa80adbf5emr7427097pfa.69.1647791762823; Sun, 20 Mar 2022 08:56:02 -0700 (PDT) Received: from localhost ([14.139.187.71]) by smtp.gmail.com with ESMTPSA id s11-20020a056a0008cb00b004fa2a3b989dsm15631550pfu.157.2022.03.20.08.56.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 20 Mar 2022 08:56:02 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8?= =?utf-8?q?rgensen?= , Jesper Dangaard Brouer Subject: [PATCH bpf-next v3 12/13] selftests/bpf: Add C tests for kptr Date: Sun, 20 Mar 2022 21:25:09 +0530 Message-Id: <20220320155510.671497-13-memxor@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220320155510.671497-1-memxor@gmail.com> References: <20220320155510.671497-1-memxor@gmail.com> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=6907; h=from:subject; bh=Znt0CMxh3BXXaWdsjTDxTQywTmMa9VB6VICPgyuEem8=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBiN00yE/i1ErgwrSqAqUWpJp/0O397arzWyAZC7w4U jJqIHjyJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCYjdNMgAKCRBM4MiGSL8RyrE+D/ 9cEq8EaLqu6TVssryg0RXTc2Mn9WBKwtxsi0GYiIk6s3pzmKI+EjcwhrykwNIuxzozpAVLvtr+Yjs3 O1n3m4ZZKt5PQYnXJmzulUrxDVmiv1LUcW8dQyhzEahUTIMKlOmZnj8iuW8EvGDspg+ZB/muJdpE3p 15HlH6UmDXPp7+fNmRaa4sfyu7i3ijyk+p23I27jBTQbJ83I6AWCqvPwMKr1rSvVvWr6uz48zPd7ty E9MmjVOvZ4PBhVlpnLqOIDm2NW4WRWs6s4m10QD5tPuNGCHS0c3LMzwvCOowyZaoIfNIiNdz7GfSOC j5ZTTAhU0YPHrfl8f84vzbdYFzQXDAnFkFmF6Miwk6LW/3iDBOfgiKOCAo6nuDkhofqE/aLcnu1yJS pPwryajdsahFDB/oCWpxMIvEDffB85n092i7J3o1wk0bi/ffDF4AFku4UH/ShDMlwN5avYfhHPYq3o PCM7mMnIrvEAUUzIqiPEOYHTfHkIEUyJ+FsOCSVTsPACyKJGbXc7KeYFeIQwfKcYOX6fNXOGRETm2/ fOyfu3cnkY2aCEqsdb0tDHDroritlFzEp0MYfAJZ2OtPhRsSGAOS4OXX/YiLk3uqgF1LYkgTp7RBOC 8s7UnVXSoMaID7rnglMKaBfDwuzjKOdQ96UpdoeCSBzGSesdllVuSPnQsvzQ== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net This uses the __kptr and __kptr_ref macros as well, and tries to test the stuff that is supposed to work, since we have negative tests in test_verifier suite. Also include some code to test map-in-map support, such that the inner_map_meta matches the kptr_off_tab of map added as element. Signed-off-by: Kumar Kartikeya Dwivedi --- .../selftests/bpf/prog_tests/map_kptr.c | 20 ++ tools/testing/selftests/bpf/progs/map_kptr.c | 194 ++++++++++++++++++ 2 files changed, 214 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/map_kptr.c create mode 100644 tools/testing/selftests/bpf/progs/map_kptr.c diff --git a/tools/testing/selftests/bpf/prog_tests/map_kptr.c b/tools/testing/selftests/bpf/prog_tests/map_kptr.c new file mode 100644 index 000000000000..688732295ce9 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/map_kptr.c @@ -0,0 +1,20 @@ +// SPDX-License-Identifier: GPL-2.0 +#include + +#include "map_kptr.skel.h" + +void test_map_kptr(void) +{ + struct map_kptr *skel; + char buf[24]; + int key = 0; + + skel = map_kptr__open_and_load(); + if (!ASSERT_OK_PTR(skel, "map_kptr__open_and_load")) + return; + ASSERT_OK(bpf_map_update_elem(bpf_map__fd(skel->maps.hash_map), &key, buf, 0), + "bpf_map_update_elem hash_map"); + ASSERT_OK(bpf_map_update_elem(bpf_map__fd(skel->maps.hash_malloc_map), &key, buf, 0), + "bpf_map_update_elem hash_malloc_map"); + map_kptr__destroy(skel); +} diff --git a/tools/testing/selftests/bpf/progs/map_kptr.c b/tools/testing/selftests/bpf/progs/map_kptr.c new file mode 100644 index 000000000000..75df3dc05db2 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/map_kptr.c @@ -0,0 +1,194 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include + +struct map_value { + struct prog_test_ref_kfunc __kptr *unref_ptr; + struct prog_test_ref_kfunc __kptr_ref *ref_ptr; +}; + +struct array_map { + __uint(type, BPF_MAP_TYPE_ARRAY); + __type(key, int); + __type(value, struct map_value); + __uint(max_entries, 1); +} array_map SEC(".maps"); + +struct hash_map { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, int); + __type(value, struct map_value); + __uint(max_entries, 1); +} hash_map SEC(".maps"); + +struct hash_malloc_map { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, int); + __type(value, struct map_value); + __uint(max_entries, 1); + __uint(map_flags, BPF_F_NO_PREALLOC); +} hash_malloc_map SEC(".maps"); + +struct lru_hash_map { + __uint(type, BPF_MAP_TYPE_LRU_HASH); + __type(key, int); + __type(value, struct map_value); + __uint(max_entries, 1); +} lru_hash_map SEC(".maps"); + +#define DEFINE_MAP_OF_MAP(map_type, inner_map_type, name) \ + struct { \ + __uint(type, map_type); \ + __uint(max_entries, 1); \ + __uint(key_size, sizeof(int)); \ + __uint(value_size, sizeof(int)); \ + __array(values, struct inner_map_type); \ + } name SEC(".maps") = { \ + .values = { [0] = &inner_map_type }, \ + } + +DEFINE_MAP_OF_MAP(BPF_MAP_TYPE_ARRAY_OF_MAPS, array_map, array_of_array_maps); +DEFINE_MAP_OF_MAP(BPF_MAP_TYPE_ARRAY_OF_MAPS, hash_map, array_of_hash_maps); +DEFINE_MAP_OF_MAP(BPF_MAP_TYPE_ARRAY_OF_MAPS, hash_malloc_map, array_of_hash_malloc_maps); +DEFINE_MAP_OF_MAP(BPF_MAP_TYPE_ARRAY_OF_MAPS, lru_hash_map, array_of_lru_hash_maps); +DEFINE_MAP_OF_MAP(BPF_MAP_TYPE_HASH_OF_MAPS, array_map, hash_of_array_maps); +DEFINE_MAP_OF_MAP(BPF_MAP_TYPE_HASH_OF_MAPS, hash_map, hash_of_hash_maps); +DEFINE_MAP_OF_MAP(BPF_MAP_TYPE_HASH_OF_MAPS, hash_malloc_map, hash_of_hash_malloc_maps); +DEFINE_MAP_OF_MAP(BPF_MAP_TYPE_HASH_OF_MAPS, lru_hash_map, hash_of_lru_hash_maps); + +extern struct prog_test_ref_kfunc *bpf_kfunc_call_test_acquire(unsigned long *sp) __ksym; +extern struct prog_test_ref_kfunc * +bpf_kfunc_call_test_kptr_get(struct prog_test_ref_kfunc **p, int a, int b) __ksym; +extern void bpf_kfunc_call_test_release(struct prog_test_ref_kfunc *p) __ksym; + +static __always_inline +void test_kptr_unref(struct map_value *v) +{ + struct prog_test_ref_kfunc *p; + + p = v->unref_ptr; + /* store untrusted_ptr_or_null_ */ + v->unref_ptr = p; + if (!p) + return; + if (p->a + p->b > 100) + return; + /* store untrusted_ptr_ */ + v->unref_ptr = p; + /* store NULL */ + v->unref_ptr = NULL; +} + +static __always_inline +void test_kptr_ref(struct map_value *v) +{ + struct prog_test_ref_kfunc *p; + + p = v->ref_ptr; + /* store ptr_or_null_ */ + v->unref_ptr = p; + if (!p) + return; + if (p->a + p->b > 100) + return; + /* store NULL */ + p = bpf_kptr_xchg(&v->ref_ptr, NULL); + if (!p) + return; + if (p->a + p->b > 100) { + bpf_kfunc_call_test_release(p); + return; + } + /* store ptr_ */ + v->unref_ptr = p; + bpf_kfunc_call_test_release(p); + + p = bpf_kfunc_call_test_acquire(&(unsigned long){0}); + if (!p) + return; + /* store ptr_ */ + p = bpf_kptr_xchg(&v->ref_ptr, p); + if (!p) + return; + if (p->a + p->b > 100) { + bpf_kfunc_call_test_release(p); + return; + } + bpf_kfunc_call_test_release(p); +} + +static __always_inline +void test_kptr_get(struct map_value *v) +{ + struct prog_test_ref_kfunc *p; + + p = bpf_kfunc_call_test_kptr_get(&v->ref_ptr, 0, 0); + if (!p) + return; + if (p->a + p->b > 100) { + bpf_kfunc_call_test_release(p); + return; + } + bpf_kfunc_call_test_release(p); +} + +static __always_inline +void test_kptr(struct map_value *v) +{ + test_kptr_unref(v); + test_kptr_ref(v); + test_kptr_get(v); +} + +SEC("tc") +int test_map_kptr(struct __sk_buff *ctx) +{ + void *maps[] = { + &array_map, + &hash_map, + &hash_malloc_map, + &lru_hash_map, + }; + struct map_value *v; + int i, key = 0; + + for (i = 0; i < sizeof(maps) / sizeof(*maps); i++) { + v = bpf_map_lookup_elem(&array_map, &key); + if (!v) + return 0; + test_kptr(v); + } + return 0; +} + +SEC("tc") +int test_map_in_map_kptr(struct __sk_buff *ctx) +{ + void *map_of_maps[] = { + &array_of_array_maps, + &array_of_hash_maps, + &array_of_hash_malloc_maps, + &array_of_lru_hash_maps, + &hash_of_array_maps, + &hash_of_hash_maps, + &hash_of_hash_malloc_maps, + &hash_of_lru_hash_maps, + }; + struct map_value *v; + int i, key = 0; + void *map; + + for (i = 0; i < sizeof(map_of_maps) / sizeof(*map_of_maps); i++) { + map = bpf_map_lookup_elem(&array_of_array_maps, &key); + if (!map) + return 0; + v = bpf_map_lookup_elem(map, &key); + if (!v) + return 0; + test_kptr(v); + } + return 0; +} + +char _license[] SEC("license") = "GPL"; From patchwork Sun Mar 20 15:55:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 12786563 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7149DC433F5 for ; Sun, 20 Mar 2022 15:56:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245428AbiCTP5h (ORCPT ); Sun, 20 Mar 2022 11:57:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51648 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245429AbiCTP5g (ORCPT ); Sun, 20 Mar 2022 11:57:36 -0400 Received: from mail-pj1-x1042.google.com (mail-pj1-x1042.google.com [IPv6:2607:f8b0:4864:20::1042]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8F3E6541B8 for ; Sun, 20 Mar 2022 08:56:09 -0700 (PDT) Received: by mail-pj1-x1042.google.com with SMTP id v4so11194187pjh.2 for ; Sun, 20 Mar 2022 08:56:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=NCUuabPk6UPQswaFS+ZrxSbyhGohlaF3VEnbgq4uXX8=; b=UOqUJ32vNhFq5k4Xw+8ADYJw71YshGkwflFh0ZiG1ESgHsGEX8byCaF+jwD2qN00sH htvDmi5PjTn2JeV/0JDON3j7Pg9WI4YdUU7FoRpNR2FR29XaFyb7CQYKLEE4H0DvJmAF siZtB98/TCxA+3+S7Jtxk05obTFOvMuN/nPyrTr6g+4fcyRpt92eKubQ9th0NKQXQtcV KypIKld3w3ussaq29hdUn3l/KpUeiKxfwYoy1Wiena+0IWcHaVaXNvyRJ8wSvOIr4u69 2+WVqJiSimGAxId30tI14xSGPZKkyyfgkcoP5peTSRmJSYBQ0ZeTWExEEp4X8ocTCwyb msUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NCUuabPk6UPQswaFS+ZrxSbyhGohlaF3VEnbgq4uXX8=; b=u/smZIs8t2mD+WkLd0YNRyDdzLt6i2we7DXpQylBiUQrwjXMxVr62Q77TqkHQLNPlF T6OWNBZCvkMLK3vT4S7ssS/dM4k0UENSYcQh8a7Fh3cg24H4PlesWYMpS1HnAZncps9L mXcM5C7UFf12xSgmF1fosLQPoryyS0N9ewiOkijsgv9bLsB3tGPL2S67wjpC3RJX8YZd 8DJ59GxYRHNqe1t+uiEUYseOmcXBGnje+a6ON9mb3vNdHpfcPArYtpAQ8vf+ibmBAp/z bsHNn4+DhyS/wJxtRoL3Anekmk5SUILzSY8vrXxhfFM6kLrivyyTsgzBsE+cKbDZXazB mfFA== X-Gm-Message-State: AOAM5322fURiZWnQbfjhLnQJDMX/bC0LNTJKsoa5V3K17JMsdEVKVEO5 xZjGQdOVZaOSQZjIe7RGpNI726TdTrI= X-Google-Smtp-Source: ABdhPJzK47cBSQKlTRz7WA4el2bDu5XLqRbtaFLFJnfdDXnPdMai1u96OX4p8vzx0j7D5JHHtF/gtQ== X-Received: by 2002:a17:902:c951:b0:154:38b8:aa30 with SMTP id i17-20020a170902c95100b0015438b8aa30mr5947049pla.145.1647791768923; Sun, 20 Mar 2022 08:56:08 -0700 (PDT) Received: from localhost ([14.139.187.71]) by smtp.gmail.com with ESMTPSA id w17-20020a056a0014d100b004f79bb37b54sm17368860pfu.195.2022.03.20.08.56.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 20 Mar 2022 08:56:08 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8?= =?utf-8?q?rgensen?= , Jesper Dangaard Brouer Subject: [PATCH bpf-next v3 13/13] selftests/bpf: Add verifier tests for kptr Date: Sun, 20 Mar 2022 21:25:10 +0530 Message-Id: <20220320155510.671497-14-memxor@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220320155510.671497-1-memxor@gmail.com> References: <20220320155510.671497-1-memxor@gmail.com> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=22624; h=from:subject; bh=hCQOLvvrhpltAyS0uevDP4FoegxWTHzfdFbfr+JhcUg=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBiN00yh29srGsvuicueXm80f9TUq7Metezjpq43Xrw 6xcjh8yJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCYjdNMgAKCRBM4MiGSL8Ryh9rD/ 0aouNCcTeWr2F4vz9lDvyuvxdgqIsEIyg+XMajqduc2cPKco1W9K3wTByiVHn7voOZsNoGSexWWueK MFyVU3y1+TvXCDJCndL8+8ZDEL7CnDl2x9Bc6mHoPei6UN75ioTnKGEbdfnAxF/5kry+2y62fgzoNI 1SktYroUCvdiZIE5UmX9A3Ezj28JKGRopPbOmKpVL8yEAzoTGYmiPprqvgoD6AiHt3tdm1EK7dWbuT LKTDfyhW9LgdS5pEfxjIut7mx/OGZrXiawwuMXhMIuYjGi6r74NyVONFrx+oNMHVmfjgwvmwpDl74Y vsKQgfqBKDQjWMM4x+9LTfa87//zLb+yEIZzUlGBd8NIJMhiaGt1TPspvVVO/zzVbX25daOwRvec2K f8uCtOg3mglT/Tpp/wU1ik6cnrlZxDtsT/0dowJHhUEY4WTOQXHH82wSAqY3/Bt0w8oso8gfzI88ZQ Bwpz1kg3dxfQFvpcxR8Y21fQiaWoH5uuOzxCxAZQ5pu4g4I+JyvC88c6k5QGbA+WJ4LU9QXur+kRN/ Ea/yaPaLM2iQzA9b5gx39QkWmjs0PMqhXVuDkxsQmSIqc7aP1SO8hrtW7OQpmMi/YppkxcXJfmNDWq BDiaYaaLgTXuj6jFHqkNBg+C+w0t3ppUQwQgBf2xZb4gbBynzRaZn9NsJ4uw== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Reuse bpf_prog_test functions to test the support for PTR_TO_BTF_ID in BPF map case, including some tests that verify implementation sanity and corner cases. Signed-off-by: Kumar Kartikeya Dwivedi --- net/bpf/test_run.c | 39 +- tools/testing/selftests/bpf/test_verifier.c | 49 +- .../testing/selftests/bpf/verifier/map_kptr.c | 445 ++++++++++++++++++ 3 files changed, 526 insertions(+), 7 deletions(-) create mode 100644 tools/testing/selftests/bpf/verifier/map_kptr.c diff --git a/net/bpf/test_run.c b/net/bpf/test_run.c index e7b9c2636d10..be1cd7498a4e 100644 --- a/net/bpf/test_run.c +++ b/net/bpf/test_run.c @@ -584,6 +584,12 @@ noinline void bpf_kfunc_call_memb_release(struct prog_test_member *p) { } +noinline struct prog_test_ref_kfunc * +bpf_kfunc_call_test_kptr_get(struct prog_test_ref_kfunc **p, int a, int b) +{ + return &prog_test_struct; +} + struct prog_test_pass1 { int x0; struct { @@ -669,6 +675,7 @@ BTF_ID(func, bpf_kfunc_call_test3) BTF_ID(func, bpf_kfunc_call_test_acquire) BTF_ID(func, bpf_kfunc_call_test_release) BTF_ID(func, bpf_kfunc_call_memb_release) +BTF_ID(func, bpf_kfunc_call_test_kptr_get) BTF_ID(func, bpf_kfunc_call_test_pass_ctx) BTF_ID(func, bpf_kfunc_call_test_pass1) BTF_ID(func, bpf_kfunc_call_test_pass2) @@ -682,6 +689,7 @@ BTF_SET_END(test_sk_check_kfunc_ids) BTF_SET_START(test_sk_acquire_kfunc_ids) BTF_ID(func, bpf_kfunc_call_test_acquire) +BTF_ID(func, bpf_kfunc_call_test_kptr_get) BTF_SET_END(test_sk_acquire_kfunc_ids) BTF_SET_START(test_sk_release_kfunc_ids) @@ -691,8 +699,13 @@ BTF_SET_END(test_sk_release_kfunc_ids) BTF_SET_START(test_sk_ret_null_kfunc_ids) BTF_ID(func, bpf_kfunc_call_test_acquire) +BTF_ID(func, bpf_kfunc_call_test_kptr_get) BTF_SET_END(test_sk_ret_null_kfunc_ids) +BTF_SET_START(test_sk_kptr_acquire_kfunc_ids) +BTF_ID(func, bpf_kfunc_call_test_kptr_get) +BTF_SET_END(test_sk_kptr_acquire_kfunc_ids) + static void *bpf_test_init(const union bpf_attr *kattr, u32 user_size, u32 size, u32 headroom, u32 tailroom) { @@ -1579,14 +1592,30 @@ int bpf_prog_test_run_syscall(struct bpf_prog *prog, static const struct btf_kfunc_id_set bpf_prog_test_kfunc_set = { .owner = THIS_MODULE, - .check_set = &test_sk_check_kfunc_ids, - .acquire_set = &test_sk_acquire_kfunc_ids, - .release_set = &test_sk_release_kfunc_ids, - .ret_null_set = &test_sk_ret_null_kfunc_ids, + .check_set = &test_sk_check_kfunc_ids, + .acquire_set = &test_sk_acquire_kfunc_ids, + .release_set = &test_sk_release_kfunc_ids, + .ret_null_set = &test_sk_ret_null_kfunc_ids, + .kptr_acquire_set = &test_sk_kptr_acquire_kfunc_ids }; +BTF_ID_LIST(bpf_prog_test_dtor_kfunc_ids) +BTF_ID(struct, prog_test_ref_kfunc) +BTF_ID(func, bpf_kfunc_call_test_release) + static int __init bpf_prog_test_run_init(void) { - return register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS, &bpf_prog_test_kfunc_set); + const struct btf_id_dtor_kfunc bpf_prog_test_dtor_kfunc[] = { + { + .btf_id = bpf_prog_test_dtor_kfunc_ids[0], + .kfunc_btf_id = bpf_prog_test_dtor_kfunc_ids[1] + }, + }; + int ret; + + ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS, &bpf_prog_test_kfunc_set); + return ret ?: register_btf_id_dtor_kfuncs(bpf_prog_test_dtor_kfunc, + ARRAY_SIZE(bpf_prog_test_dtor_kfunc), + THIS_MODULE); } late_initcall(bpf_prog_test_run_init); diff --git a/tools/testing/selftests/bpf/test_verifier.c b/tools/testing/selftests/bpf/test_verifier.c index a2cd236c32eb..847402f570bd 100644 --- a/tools/testing/selftests/bpf/test_verifier.c +++ b/tools/testing/selftests/bpf/test_verifier.c @@ -53,7 +53,7 @@ #define MAX_INSNS BPF_MAXINSNS #define MAX_TEST_INSNS 1000000 #define MAX_FIXUPS 8 -#define MAX_NR_MAPS 22 +#define MAX_NR_MAPS 23 #define MAX_TEST_RUNS 8 #define POINTER_VALUE 0xcafe4all #define TEST_DATA_LEN 64 @@ -101,6 +101,7 @@ struct bpf_test { int fixup_map_reuseport_array[MAX_FIXUPS]; int fixup_map_ringbuf[MAX_FIXUPS]; int fixup_map_timer[MAX_FIXUPS]; + int fixup_map_kptr[MAX_FIXUPS]; struct kfunc_btf_id_pair fixup_kfunc_btf_id[MAX_FIXUPS]; /* Expected verifier log output for result REJECT or VERBOSE_ACCEPT. * Can be a tab-separated sequence of expected strings. An empty string @@ -621,8 +622,13 @@ static int create_cgroup_storage(bool percpu) * struct timer { * struct bpf_timer t; * }; + * struct btf_ptr { + * struct prog_test_ref_kfunc __kptr *ptr; + * struct prog_test_ref_kfunc __kptr_ref *ptr; + * } */ -static const char btf_str_sec[] = "\0bpf_spin_lock\0val\0cnt\0l\0bpf_timer\0timer\0t"; +static const char btf_str_sec[] = "\0bpf_spin_lock\0val\0cnt\0l\0bpf_timer\0timer\0t" + "\0btf_ptr\0prog_test_ref_kfunc\0ptr\0kptr\0kptr_ref"; static __u32 btf_raw_types[] = { /* int */ BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */ @@ -638,6 +644,18 @@ static __u32 btf_raw_types[] = { /* struct timer */ /* [5] */ BTF_TYPE_ENC(35, BTF_INFO_ENC(BTF_KIND_STRUCT, 0, 1), 16), BTF_MEMBER_ENC(41, 4, 0), /* struct bpf_timer t; */ + /* struct prog_test_ref_kfunc */ /* [6] */ + BTF_STRUCT_ENC(51, 0, 0), + /* type tag "kptr" */ + BTF_TYPE_TAG_ENC(75, 6), /* [7] */ + /* type tag "kptr_ref" */ + BTF_TYPE_TAG_ENC(80, 6), /* [8] */ + BTF_PTR_ENC(7), /* [9] */ + BTF_PTR_ENC(8), /* [10] */ + /* struct btf_ptr */ /* [11] */ + BTF_STRUCT_ENC(43, 2, 16), + BTF_MEMBER_ENC(71, 9, 0), /* struct prog_test_ref_kfunc __kptr *ptr; */ + BTF_MEMBER_ENC(71, 10, 64), /* struct prog_test_ref_kfunc __kptr_ref *ptr; */ }; static int load_btf(void) @@ -727,6 +745,25 @@ static int create_map_timer(void) return fd; } +static int create_map_kptr(void) +{ + LIBBPF_OPTS(bpf_map_create_opts, opts, + .btf_key_type_id = 1, + .btf_value_type_id = 11, + ); + int fd, btf_fd; + + btf_fd = load_btf(); + if (btf_fd < 0) + return -1; + + opts.btf_fd = btf_fd; + fd = bpf_map_create(BPF_MAP_TYPE_ARRAY, "test_map", 4, 16, 1, &opts); + if (fd < 0) + printf("Failed to create map with btf_id pointer\n"); + return fd; +} + static char bpf_vlog[UINT_MAX >> 8]; static void do_test_fixup(struct bpf_test *test, enum bpf_prog_type prog_type, @@ -754,6 +791,7 @@ static void do_test_fixup(struct bpf_test *test, enum bpf_prog_type prog_type, int *fixup_map_reuseport_array = test->fixup_map_reuseport_array; int *fixup_map_ringbuf = test->fixup_map_ringbuf; int *fixup_map_timer = test->fixup_map_timer; + int *fixup_map_kptr = test->fixup_map_kptr; struct kfunc_btf_id_pair *fixup_kfunc_btf_id = test->fixup_kfunc_btf_id; if (test->fill_helper) { @@ -947,6 +985,13 @@ static void do_test_fixup(struct bpf_test *test, enum bpf_prog_type prog_type, fixup_map_timer++; } while (*fixup_map_timer); } + if (*fixup_map_kptr) { + map_fds[22] = create_map_kptr(); + do { + prog[*fixup_map_kptr].imm = map_fds[22]; + fixup_map_kptr++; + } while (*fixup_map_kptr); + } /* Patch in kfunc BTF IDs */ if (fixup_kfunc_btf_id->kfunc) { diff --git a/tools/testing/selftests/bpf/verifier/map_kptr.c b/tools/testing/selftests/bpf/verifier/map_kptr.c new file mode 100644 index 000000000000..afca65491a18 --- /dev/null +++ b/tools/testing/selftests/bpf/verifier/map_kptr.c @@ -0,0 +1,445 @@ +/* Common tests */ +{ + "map_kptr: BPF_ST imm != 0", + .insns = { + BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), + BPF_LD_MAP_FD(BPF_REG_6, 0), + BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_ST_MEM(BPF_W, BPF_REG_2, 0, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), + BPF_EXIT_INSN(), + BPF_ST_MEM(BPF_DW, BPF_REG_0, 0, 1), + BPF_EXIT_INSN(), + }, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .fixup_map_kptr = { 1 }, + .result = REJECT, + .errstr = "BPF_ST imm must be 0 when storing to kptr at off=0", +}, +{ + "map_kptr: size != bpf_size_to_bytes(BPF_DW)", + .insns = { + BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), + BPF_LD_MAP_FD(BPF_REG_6, 0), + BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_ST_MEM(BPF_W, BPF_REG_2, 0, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), + BPF_EXIT_INSN(), + BPF_ST_MEM(BPF_W, BPF_REG_0, 0, 0), + BPF_EXIT_INSN(), + }, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .fixup_map_kptr = { 1 }, + .result = REJECT, + .errstr = "kptr access size must be BPF_DW", +}, +{ + "map_kptr: map_value non-const var_off", + .insns = { + BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), + BPF_LD_MAP_FD(BPF_REG_6, 0), + BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_ST_MEM(BPF_W, BPF_REG_2, 0, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), + BPF_EXIT_INSN(), + BPF_MOV64_REG(BPF_REG_3, BPF_REG_0), + BPF_LDX_MEM(BPF_DW, BPF_REG_2, BPF_REG_0, 0), + BPF_JMP_IMM(BPF_JNE, BPF_REG_2, 0, 1), + BPF_EXIT_INSN(), + BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_2, 0), + BPF_JMP_IMM(BPF_JLE, BPF_REG_2, 4, 1), + BPF_EXIT_INSN(), + BPF_JMP_IMM(BPF_JGE, BPF_REG_2, 0, 1), + BPF_EXIT_INSN(), + BPF_ALU64_REG(BPF_ADD, BPF_REG_3, BPF_REG_2), + BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_3, 0), + BPF_EXIT_INSN(), + }, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .fixup_map_kptr = { 1 }, + .result = REJECT, + .errstr = "kptr access cannot have variable offset", +}, +{ + "map_kptr: bpf_kptr_xchg non-const var_off", + .insns = { + BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), + BPF_LD_MAP_FD(BPF_REG_6, 0), + BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_ST_MEM(BPF_W, BPF_REG_2, 0, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), + BPF_EXIT_INSN(), + BPF_MOV64_REG(BPF_REG_3, BPF_REG_0), + BPF_LDX_MEM(BPF_DW, BPF_REG_2, BPF_REG_0, 0), + BPF_JMP_IMM(BPF_JNE, BPF_REG_2, 0, 1), + BPF_EXIT_INSN(), + BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_2, 0), + BPF_JMP_IMM(BPF_JLE, BPF_REG_2, 4, 1), + BPF_EXIT_INSN(), + BPF_JMP_IMM(BPF_JGE, BPF_REG_2, 0, 1), + BPF_EXIT_INSN(), + BPF_ALU64_REG(BPF_ADD, BPF_REG_3, BPF_REG_2), + BPF_MOV64_REG(BPF_REG_1, BPF_REG_3), + BPF_MOV64_IMM(BPF_REG_2, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_kptr_xchg), + BPF_EXIT_INSN(), + }, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .fixup_map_kptr = { 1 }, + .result = REJECT, + .errstr = "R1 doesn't have constant offset. kptr has to be at the constant offset", +}, +{ + "map_kptr: unaligned boundary load/store", + .insns = { + BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), + BPF_LD_MAP_FD(BPF_REG_6, 0), + BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_ST_MEM(BPF_W, BPF_REG_2, 0, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), + BPF_EXIT_INSN(), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 7), + BPF_ST_MEM(BPF_DW, BPF_REG_0, 0, 0), + BPF_EXIT_INSN(), + }, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .fixup_map_kptr = { 1 }, + .result = REJECT, + .errstr = "kptr access misaligned expected=0 off=7", +}, +{ + "map_kptr: reject var_off != 0", + .insns = { + BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), + BPF_LD_MAP_FD(BPF_REG_6, 0), + BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_ST_MEM(BPF_W, BPF_REG_2, 0, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), + BPF_EXIT_INSN(), + BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_0, 0), + BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 1), + BPF_EXIT_INSN(), + BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, 0), + BPF_JMP_IMM(BPF_JLE, BPF_REG_2, 4, 1), + BPF_EXIT_INSN(), + BPF_JMP_IMM(BPF_JGE, BPF_REG_2, 0, 1), + BPF_EXIT_INSN(), + BPF_ALU64_REG(BPF_ADD, BPF_REG_1, BPF_REG_2), + BPF_STX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 0), + BPF_EXIT_INSN(), + }, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .fixup_map_kptr = { 1 }, + .result = REJECT, + .errstr = "variable untrusted_ptr_ access var_off=(0x0; 0x7) disallowed", +}, +/* Tests for unreferened PTR_TO_BTF_ID */ +{ + "map_kptr: unref: reject btf_struct_ids_match == false", + .insns = { + BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), + BPF_LD_MAP_FD(BPF_REG_6, 0), + BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_ST_MEM(BPF_W, BPF_REG_2, 0, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), + BPF_EXIT_INSN(), + BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_0, 0), + BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 1), + BPF_EXIT_INSN(), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4), + BPF_STX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 0), + BPF_EXIT_INSN(), + }, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .fixup_map_kptr = { 1 }, + .result = REJECT, + .errstr = "invalid kptr access, R1 type=untrusted_ptr_prog_test_ref_kfunc expected=ptr_prog_test", +}, +{ + "map_kptr: unref: loaded pointer marked as untrusted", + .insns = { + BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), + BPF_LD_MAP_FD(BPF_REG_6, 0), + BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_ST_MEM(BPF_W, BPF_REG_2, 0, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), + BPF_EXIT_INSN(), + BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0), + BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .fixup_map_kptr = { 1 }, + .result = REJECT, + .errstr = "R0 invalid mem access 'untrusted_ptr_or_null_'", +}, +{ + "map_kptr: unref: correct in kernel type size", + .insns = { + BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), + BPF_LD_MAP_FD(BPF_REG_6, 0), + BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_ST_MEM(BPF_W, BPF_REG_2, 0, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), + BPF_EXIT_INSN(), + BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0), + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), + BPF_EXIT_INSN(), + BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 24), + BPF_EXIT_INSN(), + }, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .fixup_map_kptr = { 1 }, + .result = REJECT, + .errstr = "access beyond struct prog_test_ref_kfunc at off 24 size 8", +}, +{ + "map_kptr: unref: inherit PTR_UNTRUSTED on struct walk", + .insns = { + BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), + BPF_LD_MAP_FD(BPF_REG_6, 0), + BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_ST_MEM(BPF_W, BPF_REG_2, 0, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), + BPF_EXIT_INSN(), + BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0), + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), + BPF_EXIT_INSN(), + BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_0, 16), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_this_cpu_ptr), + BPF_EXIT_INSN(), + }, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .fixup_map_kptr = { 1 }, + .result = REJECT, + .errstr = "R1 type=untrusted_ptr_ expected=percpu_ptr_", +}, +{ + "map_kptr: unref: no reference state created", + .insns = { + BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), + BPF_LD_MAP_FD(BPF_REG_6, 0), + BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_ST_MEM(BPF_W, BPF_REG_2, 0, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), + BPF_EXIT_INSN(), + BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0), + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), + BPF_EXIT_INSN(), + BPF_EXIT_INSN(), + }, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .fixup_map_kptr = { 1 }, + .result = ACCEPT, +}, +{ + "map_kptr: unref: bpf_kptr_xchg rejected", + .insns = { + BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), + BPF_LD_MAP_FD(BPF_REG_6, 0), + BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_ST_MEM(BPF_W, BPF_REG_2, 0, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), + BPF_EXIT_INSN(), + BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), + BPF_MOV64_IMM(BPF_REG_2, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_kptr_xchg), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .fixup_map_kptr = { 1 }, + .result = REJECT, + .errstr = "off=0 kptr isn't referenced kptr", +}, +{ + "map_kptr: unref: bpf_kfunc_call_test_kptr_get rejected", + .insns = { + BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), + BPF_LD_MAP_FD(BPF_REG_6, 0), + BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_ST_MEM(BPF_W, BPF_REG_2, 0, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), + BPF_EXIT_INSN(), + BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), + BPF_MOV64_IMM(BPF_REG_2, 0), + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, BPF_PSEUDO_KFUNC_CALL, 0, 0), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .fixup_map_kptr = { 1 }, + .result = REJECT, + .errstr = "arg#0 no referenced kptr at map value offset=0", + .fixup_kfunc_btf_id = { + { "bpf_kfunc_call_test_kptr_get", 13 }, + } +}, +/* Tests for referenced PTR_TO_BTF_ID */ +{ + "map_kptr: ref: loaded pointer marked as untrusted", + .insns = { + BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), + BPF_LD_MAP_FD(BPF_REG_6, 0), + BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_ST_MEM(BPF_W, BPF_REG_2, 0, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), + BPF_EXIT_INSN(), + BPF_MOV64_IMM(BPF_REG_1, 0), + BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_0, 8), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_this_cpu_ptr), + BPF_EXIT_INSN(), + }, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .fixup_map_kptr = { 1 }, + .result = REJECT, + .errstr = "R1 type=untrusted_ptr_or_null_ expected=percpu_ptr_", +}, +{ + "map_kptr: ref: reject off != 0", + .insns = { + BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), + BPF_LD_MAP_FD(BPF_REG_6, 0), + BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_ST_MEM(BPF_W, BPF_REG_2, 0, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), + BPF_EXIT_INSN(), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 8), + BPF_MOV64_REG(BPF_REG_7, BPF_REG_0), + BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), + BPF_MOV64_IMM(BPF_REG_2, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_kptr_xchg), + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), + BPF_EXIT_INSN(), + BPF_MOV64_REG(BPF_REG_1, BPF_REG_7), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 4), + BPF_MOV64_REG(BPF_REG_2, BPF_REG_0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_kptr_xchg), + BPF_EXIT_INSN(), + }, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .fixup_map_kptr = { 1 }, + .result = REJECT, + .errstr = "dereference of modified ptr_ ptr R2 off=4 disallowed", +}, +{ + "map_kptr: ref: reference state created and released on xchg", + .insns = { + BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), + BPF_LD_MAP_FD(BPF_REG_6, 0), + BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_ST_MEM(BPF_W, BPF_REG_2, 0, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), + BPF_EXIT_INSN(), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 8), + BPF_MOV64_REG(BPF_REG_7, BPF_REG_0), + BPF_MOV64_REG(BPF_REG_1, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, -8), + BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, BPF_PSEUDO_KFUNC_CALL, 0, 0), + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), + BPF_EXIT_INSN(), + BPF_MOV64_REG(BPF_REG_1, BPF_REG_7), + BPF_MOV64_REG(BPF_REG_2, BPF_REG_0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_kptr_xchg), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .fixup_map_kptr = { 1 }, + .result = REJECT, + .errstr = "Unreleased reference id=5 alloc_insn=20", + .fixup_kfunc_btf_id = { + { "bpf_kfunc_call_test_acquire", 15 }, + } +}, +{ + "map_kptr: ref: reject STX", + .insns = { + BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), + BPF_LD_MAP_FD(BPF_REG_6, 0), + BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_ST_MEM(BPF_W, BPF_REG_2, 0, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), + BPF_EXIT_INSN(), + BPF_MOV64_REG(BPF_REG_1, 0), + BPF_STX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 8), + BPF_EXIT_INSN(), + }, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .fixup_map_kptr = { 1 }, + .result = REJECT, + .errstr = "store to referenced kptr disallowed", +}, +{ + "map_kptr: ref: reject ST", + .insns = { + BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), + BPF_LD_MAP_FD(BPF_REG_6, 0), + BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_ST_MEM(BPF_W, BPF_REG_2, 0, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), + BPF_EXIT_INSN(), + BPF_ST_MEM(BPF_DW, BPF_REG_0, 8, 0), + BPF_EXIT_INSN(), + }, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .fixup_map_kptr = { 1 }, + .result = REJECT, + .errstr = "store to referenced kptr disallowed", +},