From patchwork Mon Jan 3 15:08:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Patchwork-Id: 12702589 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A25B0C433EF for ; Mon, 3 Jan 2022 15:08:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233867AbiACPI2 (ORCPT ); Mon, 3 Jan 2022 10:08:28 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:38472 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233800AbiACPI1 (ORCPT ); Mon, 3 Jan 2022 10:08:27 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1641222506; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eBJQlk17VYpNq3XuagcLluTSgGk2Y5vu3pcY1OdBztc=; b=Hov9CiIWpJi1wFn22iVFYp8RqT4tUS3Spjqkb4cogBjBgp2LkusnGceV0wZnWh2keVlIkM iYv+NFiJf3jQmjlhPx/LJ6zneB83EFd0ToF6q8r54pAJKP0OanpeIilrlRY26AJFqe7ZfJ v49N1cJMKSlEU4q9DFJjDb0gGzVzotM= Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-607-GO667bn4Nh-TMqCp_pDpdQ-1; Mon, 03 Jan 2022 10:08:25 -0500 X-MC-Unique: GO667bn4Nh-TMqCp_pDpdQ-1 Received: by mail-ed1-f70.google.com with SMTP id g2-20020a056402424200b003f8ee03207eso16224535edb.7 for ; Mon, 03 Jan 2022 07:08:25 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=eBJQlk17VYpNq3XuagcLluTSgGk2Y5vu3pcY1OdBztc=; b=lAYGWF6zfJ9Z4zyZFjMMvUpETJxBzGkMYfl1kRPekrh+H00BN2K70NdgARE3aXzmIW UE+wXoyE3xB4Xmb/XeCu7NoCRuWMkwLdqbchuEyNFEHhzIueCmFzH8PfhO99aNnRghsc H8fHssFLAJCqDh9pNf/cdA3qcmkd4zIV0saBAof1EjrvK1c0CdMFce/tL7gokLLcPm5e pGN5NDP9eWXnda7czK/xD9BzHUdHZ+SGbiU9xhj16fuyTHY9hzRFy77YUnrBnF8F2FTZ hDGwFCEkZyzuODvru30+fRBP2AtsFsS0fRC6sIxm9PSvlJNRbD5BNGWikuRjoScFJNHz R6fw== X-Gm-Message-State: AOAM533qJ+TXG9eAiyMcuf6tgKe8I0DX2dyymqVCyNTyTj9zQbXWWim5 213L7iAYJkgMqts2hTIXp+c//7Zt0oSAOEwEtV32wr0Shfh8EkpIeBAtmANDqRCYRFeBXlvdvxf RTi6Oeo/ZuKrSvgUw X-Received: by 2002:a17:907:76f4:: with SMTP id kg20mr36370190ejc.54.1641222502198; Mon, 03 Jan 2022 07:08:22 -0800 (PST) X-Google-Smtp-Source: ABdhPJw3UVivIiXO5HAhMxK4JvFAQJHcTWVmqehl+Z8texCiZlBu1MGEgxUXYlcSQB0jCmGiMR8OtQ== X-Received: by 2002:a17:907:76f4:: with SMTP id kg20mr36370062ejc.54.1641222500412; Mon, 03 Jan 2022 07:08:20 -0800 (PST) Received: from alrua-x1.borgediget.toke.dk ([2a0c:4d80:42:443::2]) by smtp.gmail.com with ESMTPSA id ht14sm10685790ejc.64.2022.01.03.07.08.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Jan 2022 07:08:20 -0800 (PST) Received: by alrua-x1.borgediget.toke.dk (Postfix, from userid 1000) id 675E2181F2A; Mon, 3 Jan 2022 16:08:19 +0100 (CET) From: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= To: Alexei Starovoitov , Daniel Borkmann , "David S. Miller" , Jakub Kicinski , Jesper Dangaard Brouer , John Fastabend , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , KP Singh Cc: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= , netdev@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH bpf-next v5 1/7] xdp: Allow registering memory model without rxq reference Date: Mon, 3 Jan 2022 16:08:06 +0100 Message-Id: <20220103150812.87914-2-toke@redhat.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220103150812.87914-1-toke@redhat.com> References: <20220103150812.87914-1-toke@redhat.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net The functions that register an XDP memory model take a struct xdp_rxq as parameter, but the RXQ is not actually used for anything other than pulling out the struct xdp_mem_info that it embeds. So refactor the register functions and export variants that just take a pointer to the xdp_mem_info. This is in preparation for enabling XDP_REDIRECT in bpf_prog_run(), using a page_pool instance that is not connected to any network device. Signed-off-by: Toke Høiland-Jørgensen --- include/net/xdp.h | 3 ++ net/core/xdp.c | 92 +++++++++++++++++++++++++++++++---------------- 2 files changed, 65 insertions(+), 30 deletions(-) diff --git a/include/net/xdp.h b/include/net/xdp.h index 447f9b1578f3..8f0812e4996d 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -260,6 +260,9 @@ bool xdp_rxq_info_is_reg(struct xdp_rxq_info *xdp_rxq); int xdp_rxq_info_reg_mem_model(struct xdp_rxq_info *xdp_rxq, enum xdp_mem_type type, void *allocator); void xdp_rxq_info_unreg_mem_model(struct xdp_rxq_info *xdp_rxq); +int xdp_reg_mem_model(struct xdp_mem_info *mem, + enum xdp_mem_type type, void *allocator); +void xdp_unreg_mem_model(struct xdp_mem_info *mem); /* Drivers not supporting XDP metadata can use this helper, which * rejects any room expansion for metadata as a result. diff --git a/net/core/xdp.c b/net/core/xdp.c index 7fe1df85f505..58089f6d2c7a 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -110,20 +110,15 @@ static void mem_allocator_disconnect(void *allocator) mutex_unlock(&mem_id_lock); } -void xdp_rxq_info_unreg_mem_model(struct xdp_rxq_info *xdp_rxq) +void xdp_unreg_mem_model(struct xdp_mem_info *mem) { struct xdp_mem_allocator *xa; - int type = xdp_rxq->mem.type; - int id = xdp_rxq->mem.id; + int type = mem->type; + int id = mem->id; /* Reset mem info to defaults */ - xdp_rxq->mem.id = 0; - xdp_rxq->mem.type = 0; - - if (xdp_rxq->reg_state != REG_STATE_REGISTERED) { - WARN(1, "Missing register, driver bug"); - return; - } + mem->id = 0; + mem->type = 0; if (id == 0) return; @@ -135,6 +130,17 @@ void xdp_rxq_info_unreg_mem_model(struct xdp_rxq_info *xdp_rxq) rcu_read_unlock(); } } +EXPORT_SYMBOL_GPL(xdp_unreg_mem_model); + +void xdp_rxq_info_unreg_mem_model(struct xdp_rxq_info *xdp_rxq) +{ + if (xdp_rxq->reg_state != REG_STATE_REGISTERED) { + WARN(1, "Missing register, driver bug"); + return; + } + + xdp_unreg_mem_model(&xdp_rxq->mem); +} EXPORT_SYMBOL_GPL(xdp_rxq_info_unreg_mem_model); void xdp_rxq_info_unreg(struct xdp_rxq_info *xdp_rxq) @@ -259,28 +265,24 @@ static bool __is_supported_mem_type(enum xdp_mem_type type) return true; } -int xdp_rxq_info_reg_mem_model(struct xdp_rxq_info *xdp_rxq, - enum xdp_mem_type type, void *allocator) +static struct xdp_mem_allocator *__xdp_reg_mem_model(struct xdp_mem_info *mem, + enum xdp_mem_type type, + void *allocator) { struct xdp_mem_allocator *xdp_alloc; gfp_t gfp = GFP_KERNEL; int id, errno, ret; void *ptr; - if (xdp_rxq->reg_state != REG_STATE_REGISTERED) { - WARN(1, "Missing register, driver bug"); - return -EFAULT; - } - if (!__is_supported_mem_type(type)) - return -EOPNOTSUPP; + return ERR_PTR(-EOPNOTSUPP); - xdp_rxq->mem.type = type; + mem->type = type; if (!allocator) { if (type == MEM_TYPE_PAGE_POOL) - return -EINVAL; /* Setup time check page_pool req */ - return 0; + return ERR_PTR(-EINVAL); /* Setup time check page_pool req */ + return NULL; } /* Delay init of rhashtable to save memory if feature isn't used */ @@ -290,13 +292,13 @@ int xdp_rxq_info_reg_mem_model(struct xdp_rxq_info *xdp_rxq, mutex_unlock(&mem_id_lock); if (ret < 0) { WARN_ON(1); - return ret; + return ERR_PTR(ret); } } xdp_alloc = kzalloc(sizeof(*xdp_alloc), gfp); if (!xdp_alloc) - return -ENOMEM; + return ERR_PTR(-ENOMEM); mutex_lock(&mem_id_lock); id = __mem_id_cyclic_get(gfp); @@ -304,15 +306,15 @@ int xdp_rxq_info_reg_mem_model(struct xdp_rxq_info *xdp_rxq, errno = id; goto err; } - xdp_rxq->mem.id = id; - xdp_alloc->mem = xdp_rxq->mem; + mem->id = id; + xdp_alloc->mem = *mem; xdp_alloc->allocator = allocator; /* Insert allocator into ID lookup table */ ptr = rhashtable_insert_slow(mem_id_ht, &id, &xdp_alloc->node); if (IS_ERR(ptr)) { - ida_simple_remove(&mem_id_pool, xdp_rxq->mem.id); - xdp_rxq->mem.id = 0; + ida_simple_remove(&mem_id_pool, mem->id); + mem->id = 0; errno = PTR_ERR(ptr); goto err; } @@ -322,13 +324,43 @@ int xdp_rxq_info_reg_mem_model(struct xdp_rxq_info *xdp_rxq, mutex_unlock(&mem_id_lock); - trace_mem_connect(xdp_alloc, xdp_rxq); - return 0; + return xdp_alloc; err: mutex_unlock(&mem_id_lock); kfree(xdp_alloc); - return errno; + return ERR_PTR(errno); +} + +int xdp_reg_mem_model(struct xdp_mem_info *mem, + enum xdp_mem_type type, void *allocator) +{ + struct xdp_mem_allocator *xdp_alloc; + + xdp_alloc = __xdp_reg_mem_model(mem, type, allocator); + if (IS_ERR(xdp_alloc)) + return PTR_ERR(xdp_alloc); + return 0; +} +EXPORT_SYMBOL_GPL(xdp_reg_mem_model); + +int xdp_rxq_info_reg_mem_model(struct xdp_rxq_info *xdp_rxq, + enum xdp_mem_type type, void *allocator) +{ + struct xdp_mem_allocator *xdp_alloc; + + if (xdp_rxq->reg_state != REG_STATE_REGISTERED) { + WARN(1, "Missing register, driver bug"); + return -EFAULT; + } + + xdp_alloc = __xdp_reg_mem_model(&xdp_rxq->mem, type, allocator); + if (IS_ERR(xdp_alloc)) + return PTR_ERR(xdp_alloc); + + trace_mem_connect(xdp_alloc, xdp_rxq); + return 0; } + EXPORT_SYMBOL_GPL(xdp_rxq_info_reg_mem_model); /* XDP RX runs under NAPI protection, and in different delivery error From patchwork Mon Jan 3 15:08:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Patchwork-Id: 12702585 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46A2CC433F5 for ; Mon, 3 Jan 2022 15:08:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232758AbiACPIY (ORCPT ); Mon, 3 Jan 2022 10:08:24 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:35121 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230130AbiACPIX (ORCPT ); Mon, 3 Jan 2022 10:08:23 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1641222503; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=D3Ki68U13fAIOpoyGaq1fUYBlJK+4OgqlFUiLYU2tac=; b=CoPzRSwo8EjQ477LR/2Uu8rBSUnyzatk8vaSBaMXBuxhTZw3v8eo//Oi6OfnciyCosKr/T TlC2+Qhv2e/wx9PDqxREEquUiFuwU6GxmIGRpVeeeXXRszAFjc7rnV+tXt9M3FA3fqiLYv zcxAXOBhoMVd1ujq4SK0SXcvNoOx3Lw= Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-345-wM-zcc8pNUqSImovm3II0A-1; Mon, 03 Jan 2022 10:08:22 -0500 X-MC-Unique: wM-zcc8pNUqSImovm3II0A-1 Received: by mail-ed1-f71.google.com with SMTP id o20-20020a056402439400b003f83cf1e472so23079650edc.18 for ; Mon, 03 Jan 2022 07:08:22 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=D3Ki68U13fAIOpoyGaq1fUYBlJK+4OgqlFUiLYU2tac=; b=pKIcaw7LjcbUmIV84J75n4vXHBEoedjhrwlLMee9H4tJNn17SdKVH9cOJPdrzuRHkd WG2/GPBGATbd+dVN6SFi4LyaP3cPrsIdsR3ojuUgfNgmieu2zNuQnVDsEjpU4QTLIF05 V0ksrbwvgv1/yBit1mezcwzXcjt+ukX8zv5Q1utgH5PdEu7OrA/WE2MORawzk7qe2a+g jMyfLxwC3ukVnvdhc5EArkTwXP7i9TGyp9jbBQ5dgXIjYnIR5XEw0f7Y778r/Yl1cMJH pv70BZUKywXKLbvqS4pDhWrHa+GqjDJC6C0t6K+YNZAh/gmHfOQec0rbRcjvmUNLz+M/ cduw== X-Gm-Message-State: AOAM531S2AaAynoLjJjDNcHyGj5XIy6r2SB1lzkiU72WUFgFk1/xyTQb FN//QwH7nB+7QaJ7/2kYJU/iczgAOa3lVbOj9E/dzKmqceHKGo/OCT7ySbNCi81Br6P0G+oM7a2 3KQf/SuWEtDBInf6k X-Received: by 2002:a50:fb8f:: with SMTP id e15mr45665082edq.94.1641222501101; Mon, 03 Jan 2022 07:08:21 -0800 (PST) X-Google-Smtp-Source: ABdhPJw40DUBTwyZabag+7r29H+wWRf9tdHz4kapbP/ceiyCGc435YR4PdDcwqr/kdn9Kl4wgvniDQ== X-Received: by 2002:a50:fb8f:: with SMTP id e15mr45665036edq.94.1641222500788; Mon, 03 Jan 2022 07:08:20 -0800 (PST) Received: from alrua-x1.borgediget.toke.dk ([45.145.92.2]) by smtp.gmail.com with ESMTPSA id d17sm10701147ejd.217.2022.01.03.07.08.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Jan 2022 07:08:20 -0800 (PST) Received: by alrua-x1.borgediget.toke.dk (Postfix, from userid 1000) id C3C26181F2B; Mon, 3 Jan 2022 16:08:19 +0100 (CET) From: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= To: Jesper Dangaard Brouer , Ilias Apalodimas , "David S. Miller" , Jakub Kicinski , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Cc: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= , netdev@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH bpf-next v5 2/7] page_pool: Add callback to init pages when they are allocated Date: Mon, 3 Jan 2022 16:08:07 +0100 Message-Id: <20220103150812.87914-3-toke@redhat.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220103150812.87914-1-toke@redhat.com> References: <20220103150812.87914-1-toke@redhat.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Add a new callback function to page_pool that, if set, will be called every time a new page is allocated. This will be used from bpf_test_run() to initialise the page data with the data provided by userspace when running XDP programs with redirect turned on. Acked-by: John Fastabend Signed-off-by: Toke Høiland-Jørgensen Acked-by: Jesper Dangaard Brouer --- include/net/page_pool.h | 2 ++ net/core/page_pool.c | 2 ++ 2 files changed, 4 insertions(+) diff --git a/include/net/page_pool.h b/include/net/page_pool.h index a4082406a003..d807b6800a4a 100644 --- a/include/net/page_pool.h +++ b/include/net/page_pool.h @@ -80,6 +80,8 @@ struct page_pool_params { enum dma_data_direction dma_dir; /* DMA mapping direction */ unsigned int max_len; /* max DMA sync memory size */ unsigned int offset; /* DMA addr offset */ + void (*init_callback)(struct page *page, void *arg); + void *init_arg; }; struct page_pool { diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 1a6978427d6c..f53786f6666d 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -217,6 +217,8 @@ static void page_pool_set_pp_info(struct page_pool *pool, { page->pp = pool; page->pp_magic |= PP_SIGNATURE; + if (pool->p.init_callback) + pool->p.init_callback(page, pool->p.init_arg); } static void page_pool_clear_pp_info(struct page *page) From patchwork Mon Jan 3 15:08:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Patchwork-Id: 12702588 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2673C433FE for ; Mon, 3 Jan 2022 15:08:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233851AbiACPI1 (ORCPT ); Mon, 3 Jan 2022 10:08:27 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:28525 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233807AbiACPI0 (ORCPT ); Mon, 3 Jan 2022 10:08:26 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1641222505; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sq80IA6Io1TllpaI+PVx+jt1JVKEdqHlykzwRmU9cEY=; b=a/+AmGUj6mpAlatmhWtQeTuYLr9NGoAfRTiDDpy7qxWBJsXsrcEu+Y3/7ixBSEzgiADZ6H bcMyn9/TsKjZDvE3kOa647cs1OGzwCtg59ORlWtsS/9e0Uyc2NUhFUwzapqJQxS5r6nRJW H+vBr4z2Pubz7vP4euQ4QSBWIrUcN1c= Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-251-mlVQfIdZPSGoi9OGH1eSng-1; Mon, 03 Jan 2022 10:08:24 -0500 X-MC-Unique: mlVQfIdZPSGoi9OGH1eSng-1 Received: by mail-ed1-f71.google.com with SMTP id eg23-20020a056402289700b003f80a27ca2bso23033962edb.14 for ; Mon, 03 Jan 2022 07:08:24 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=sq80IA6Io1TllpaI+PVx+jt1JVKEdqHlykzwRmU9cEY=; b=3jn66BA9ntd6E3VjolGJQ8hEMY5QISOIYsOf3wV2JWMKKHNHZ+bbcnnM5SVhmTUfHU 9evt3lEBF38k66ty0hV0i9Th5X0MmGA9vk5iK/Nl3Nh19cBPgi8r6xfVCRQv0PHl7Wax 7emAf6anw3rtVqh17CIWV0HZVhxw8P7V7haYTUCTW2/MfIhTWGdX7x3NUHB8MdHheYBY 9UeefyZ+QMaK/Qz3pTuOwGwOGv/xqOBBlSFYLUuJgkKpbGJKJV0l+hVN5IkL1xhVFeit zNLToeil2JU64YRDd+QXq7GM65aWxMMoSOLFw+c0sTksOHdO4NF/DtWFFYEowv0oqljj VAjQ== X-Gm-Message-State: AOAM532475myEVU9dYdh1LByIuHar0BkNpmj1vM8CAfAXEmBGVX7Eop/ CsdMNNeAjPF2pcMMgyxzoW5f5esU4W5usAUwDdGEkq6WMyt8gqek/KSvSkGuVKxjlJ8MpjM/pUC Rkid8UyhImdfFTz6K X-Received: by 2002:a05:6402:27d1:: with SMTP id c17mr43849716ede.128.1641222502351; Mon, 03 Jan 2022 07:08:22 -0800 (PST) X-Google-Smtp-Source: ABdhPJxnA/WIPaTJV/pMImx+DCYnjy8IZQ3jV3GCmmKS5YaF7+NN1UFL3az5p0l2yPu4kXU4XImrKQ== X-Received: by 2002:a05:6402:27d1:: with SMTP id c17mr43849647ede.128.1641222501374; Mon, 03 Jan 2022 07:08:21 -0800 (PST) Received: from alrua-x1.borgediget.toke.dk ([2a0c:4d80:42:443::2]) by smtp.gmail.com with ESMTPSA id h11sm7761316edb.59.2022.01.03.07.08.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Jan 2022 07:08:20 -0800 (PST) Received: by alrua-x1.borgediget.toke.dk (Postfix, from userid 1000) id 4EEE2180300; Mon, 3 Jan 2022 16:08:20 +0100 (CET) From: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= To: Jesper Dangaard Brouer , Ilias Apalodimas , "David S. Miller" , Jakub Kicinski , Alexei Starovoitov , Daniel Borkmann , John Fastabend , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , KP Singh Cc: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= , netdev@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH bpf-next v5 3/7] page_pool: Store the XDP mem id Date: Mon, 3 Jan 2022 16:08:08 +0100 Message-Id: <20220103150812.87914-4-toke@redhat.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220103150812.87914-1-toke@redhat.com> References: <20220103150812.87914-1-toke@redhat.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Store the XDP mem ID inside the page_pool struct so it can be retrieved later for use in bpf_prog_run(). Signed-off-by: Toke Høiland-Jørgensen Acked-by: Jesper Dangaard Brouer --- include/net/page_pool.h | 9 +++++++-- net/core/page_pool.c | 4 +++- net/core/xdp.c | 2 +- 3 files changed, 11 insertions(+), 4 deletions(-) diff --git a/include/net/page_pool.h b/include/net/page_pool.h index d807b6800a4a..79a805542d0f 100644 --- a/include/net/page_pool.h +++ b/include/net/page_pool.h @@ -96,6 +96,7 @@ struct page_pool { unsigned int frag_offset; struct page *frag_page; long frag_users; + u32 xdp_mem_id; /* * Data structure for allocation side @@ -170,9 +171,12 @@ bool page_pool_return_skb_page(struct page *page); struct page_pool *page_pool_create(const struct page_pool_params *params); +struct xdp_mem_info; + #ifdef CONFIG_PAGE_POOL void page_pool_destroy(struct page_pool *pool); -void page_pool_use_xdp_mem(struct page_pool *pool, void (*disconnect)(void *)); +void page_pool_use_xdp_mem(struct page_pool *pool, void (*disconnect)(void *), + struct xdp_mem_info *mem); void page_pool_release_page(struct page_pool *pool, struct page *page); void page_pool_put_page_bulk(struct page_pool *pool, void **data, int count); @@ -182,7 +186,8 @@ static inline void page_pool_destroy(struct page_pool *pool) } static inline void page_pool_use_xdp_mem(struct page_pool *pool, - void (*disconnect)(void *)) + void (*disconnect)(void *), + struct xdp_mem_info *mem) { } static inline void page_pool_release_page(struct page_pool *pool, diff --git a/net/core/page_pool.c b/net/core/page_pool.c index f53786f6666d..7347d5c7dbe0 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -693,10 +693,12 @@ static void page_pool_release_retry(struct work_struct *wq) schedule_delayed_work(&pool->release_dw, DEFER_TIME); } -void page_pool_use_xdp_mem(struct page_pool *pool, void (*disconnect)(void *)) +void page_pool_use_xdp_mem(struct page_pool *pool, void (*disconnect)(void *), + struct xdp_mem_info *mem) { refcount_inc(&pool->user_cnt); pool->disconnect = disconnect; + pool->xdp_mem_id = mem->id; } void page_pool_destroy(struct page_pool *pool) diff --git a/net/core/xdp.c b/net/core/xdp.c index 58089f6d2c7a..7aba35504986 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -320,7 +320,7 @@ static struct xdp_mem_allocator *__xdp_reg_mem_model(struct xdp_mem_info *mem, } if (type == MEM_TYPE_PAGE_POOL) - page_pool_use_xdp_mem(allocator, mem_allocator_disconnect); + page_pool_use_xdp_mem(allocator, mem_allocator_disconnect, mem); mutex_unlock(&mem_id_lock); From patchwork Mon Jan 3 15:08:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Patchwork-Id: 12702586 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE259C433FE for ; Mon, 3 Jan 2022 15:08:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233836AbiACPI0 (ORCPT ); Mon, 3 Jan 2022 10:08:26 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:54078 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232478AbiACPIZ (ORCPT ); Mon, 3 Jan 2022 10:08:25 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1641222505; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VcLBd7sfLQd5mT9HLpqncuK4NzZSAv9ZaVpM25Xng3Y=; b=h6IHT7l3VESBw+vDEg77k6L3tUnYP75nCyIChP5D344CWVgCVaWjtFciWJujHNvp34q5Co PLgQwwfbTG0xL8lNATHVxRmQ+MdIkNomMcr9sbFRr4WASheBjXwU5gOumugkPnrqM1mxwP phKGo1BlIR4DgdCQPoSpc9YyGFmSNGM= Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-632-ACy56AGeNwehXITTQTMIAQ-1; Mon, 03 Jan 2022 10:08:24 -0500 X-MC-Unique: ACy56AGeNwehXITTQTMIAQ-1 Received: by mail-ed1-f72.google.com with SMTP id ch27-20020a0564021bdb00b003f8389236f8so22859054edb.19 for ; Mon, 03 Jan 2022 07:08:23 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=VcLBd7sfLQd5mT9HLpqncuK4NzZSAv9ZaVpM25Xng3Y=; b=KJEdeAnEnnlBkXoOUdxvfDvJhTUNe5iAfA53Zv3girTgPAteriuGYEdQRjXOomE2eu A0Bszz9EBKTPJQv8NtUBIW8hk719RZEQWK5xRGmkERurwwqc3vkuLvat3Tj6tg2IddgT kyow2P3sOUW+Zc3s6+hKRggnkpl4vnGO73dmaFB0yuMoH6KtM6UWF9QbzZYFxbBOoPNY 8WHzGnms4e0SO8IfL2keGNgzwoKpjW1ovv/bs7P/4D9L/FcSvS9bDBj1gLwlc2UR+qRR on/l0oG/dtJJ/hSyhAjSodoz66jx5rIi71ZpgT8lMCchw9aJp3NrZmPSDUqFZSuHKRYZ +XUg== X-Gm-Message-State: AOAM532u9FmIb4Hm/58oP29wARyNnXA1U4V3IOj72gWrjHCtfGxHsEkI btV1zO4Eif3hSc4xa3cARsztQek29T6/Fcd/9aDFCcitavOMAw/JWCnyoHr8O2K6MLsxvteYqy/ KUvgZ5GMNlXBW6EAc X-Received: by 2002:aa7:dd56:: with SMTP id o22mr45854738edw.73.1641222502609; Mon, 03 Jan 2022 07:08:22 -0800 (PST) X-Google-Smtp-Source: ABdhPJzExLOFUCiyhcqUZi2xcTpdt4lezlTcHEA6LCatRyB3+Cfa9wTmh4HeGijksW/jenb9oIvEVQ== X-Received: by 2002:aa7:dd56:: with SMTP id o22mr45854706edw.73.1641222502288; Mon, 03 Jan 2022 07:08:22 -0800 (PST) Received: from alrua-x1.borgediget.toke.dk ([45.145.92.2]) by smtp.gmail.com with ESMTPSA id hd17sm10726076ejc.58.2022.01.03.07.08.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Jan 2022 07:08:21 -0800 (PST) Received: by alrua-x1.borgediget.toke.dk (Postfix, from userid 1000) id F223C181F2A; Mon, 3 Jan 2022 16:08:20 +0100 (CET) From: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , "David S. Miller" , Jakub Kicinski , Jesper Dangaard Brouer Cc: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= , netdev@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH bpf-next v5 4/7] xdp: Move conversion to xdp_frame out of map functions Date: Mon, 3 Jan 2022 16:08:09 +0100 Message-Id: <20220103150812.87914-5-toke@redhat.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220103150812.87914-1-toke@redhat.com> References: <20220103150812.87914-1-toke@redhat.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net All map redirect functions except XSK maps convert xdp_buff to xdp_frame before enqueueing it. So move this conversion of out the map functions and into xdp_do_redirect(). This removes a bit of duplicated code, but more importantly it makes it possible to support caller-allocated xdp_frame structures, which will be added in a subsequent commit. Signed-off-by: Toke Høiland-Jørgensen --- include/linux/bpf.h | 20 ++++++++++---------- kernel/bpf/cpumap.c | 8 +------- kernel/bpf/devmap.c | 32 +++++++++++--------------------- net/core/filter.c | 24 +++++++++++++++++------- 4 files changed, 39 insertions(+), 45 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 26753139d5b4..6e947cd91152 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1669,17 +1669,17 @@ void bpf_patch_call_args(struct bpf_insn *insn, u32 stack_depth); struct btf *bpf_get_btf_vmlinux(void); /* Map specifics */ -struct xdp_buff; +struct xdp_frame; struct sk_buff; struct bpf_dtab_netdev; struct bpf_cpu_map_entry; void __dev_flush(void); -int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, +int dev_xdp_enqueue(struct net_device *dev, struct xdp_frame *xdpf, struct net_device *dev_rx); -int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, +int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_frame *xdpf, struct net_device *dev_rx); -int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, +int dev_map_enqueue_multi(struct xdp_frame *xdpf, struct net_device *dev_rx, struct bpf_map *map, bool exclude_ingress); int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, struct bpf_prog *xdp_prog); @@ -1688,7 +1688,7 @@ int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb, bool exclude_ingress); void __cpu_map_flush(void); -int cpu_map_enqueue(struct bpf_cpu_map_entry *rcpu, struct xdp_buff *xdp, +int cpu_map_enqueue(struct bpf_cpu_map_entry *rcpu, struct xdp_frame *xdpf, struct net_device *dev_rx); int cpu_map_generic_redirect(struct bpf_cpu_map_entry *rcpu, struct sk_buff *skb); @@ -1866,26 +1866,26 @@ static inline void __dev_flush(void) { } -struct xdp_buff; +struct xdp_frame; struct bpf_dtab_netdev; struct bpf_cpu_map_entry; static inline -int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, +int dev_xdp_enqueue(struct net_device *dev, struct xdp_frame *xdpf, struct net_device *dev_rx) { return 0; } static inline -int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, +int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_frame *xdpf, struct net_device *dev_rx) { return 0; } static inline -int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, +int dev_map_enqueue_multi(struct xdp_frame *xdpf, struct net_device *dev_rx, struct bpf_map *map, bool exclude_ingress) { return 0; @@ -1913,7 +1913,7 @@ static inline void __cpu_map_flush(void) } static inline int cpu_map_enqueue(struct bpf_cpu_map_entry *rcpu, - struct xdp_buff *xdp, + struct xdp_frame *xdpf, struct net_device *dev_rx) { return 0; diff --git a/kernel/bpf/cpumap.c b/kernel/bpf/cpumap.c index 0421061d95f1..b3e6b9422238 100644 --- a/kernel/bpf/cpumap.c +++ b/kernel/bpf/cpumap.c @@ -746,15 +746,9 @@ static void bq_enqueue(struct bpf_cpu_map_entry *rcpu, struct xdp_frame *xdpf) list_add(&bq->flush_node, flush_list); } -int cpu_map_enqueue(struct bpf_cpu_map_entry *rcpu, struct xdp_buff *xdp, +int cpu_map_enqueue(struct bpf_cpu_map_entry *rcpu, struct xdp_frame *xdpf, struct net_device *dev_rx) { - struct xdp_frame *xdpf; - - xdpf = xdp_convert_buff_to_frame(xdp); - if (unlikely(!xdpf)) - return -EOVERFLOW; - /* Info needed when constructing SKB on remote CPU */ xdpf->dev_rx = dev_rx; diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index 6feea293ff10..fe019dbdb3f0 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -467,24 +467,19 @@ static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, bq->q[bq->count++] = xdpf; } -static inline int __xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, +static inline int __xdp_enqueue(struct net_device *dev, struct xdp_frame *xdpf, struct net_device *dev_rx, struct bpf_prog *xdp_prog) { - struct xdp_frame *xdpf; int err; if (!dev->netdev_ops->ndo_xdp_xmit) return -EOPNOTSUPP; - err = xdp_ok_fwd_dev(dev, xdp->data_end - xdp->data); + err = xdp_ok_fwd_dev(dev, xdpf->len); if (unlikely(err)) return err; - xdpf = xdp_convert_buff_to_frame(xdp); - if (unlikely(!xdpf)) - return -EOVERFLOW; - bq_enqueue(dev, xdpf, dev_rx, xdp_prog); return 0; } @@ -520,27 +515,27 @@ static u32 dev_map_bpf_prog_run_skb(struct sk_buff *skb, struct bpf_dtab_netdev return act; } -int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, +int dev_xdp_enqueue(struct net_device *dev, struct xdp_frame *xdpf, struct net_device *dev_rx) { - return __xdp_enqueue(dev, xdp, dev_rx, NULL); + return __xdp_enqueue(dev, xdpf, dev_rx, NULL); } -int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, +int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_frame *xdpf, struct net_device *dev_rx) { struct net_device *dev = dst->dev; - return __xdp_enqueue(dev, xdp, dev_rx, dst->xdp_prog); + return __xdp_enqueue(dev, xdpf, dev_rx, dst->xdp_prog); } -static bool is_valid_dst(struct bpf_dtab_netdev *obj, struct xdp_buff *xdp) +static bool is_valid_dst(struct bpf_dtab_netdev *obj, struct xdp_frame *xdpf) { if (!obj || !obj->dev->netdev_ops->ndo_xdp_xmit) return false; - if (xdp_ok_fwd_dev(obj->dev, xdp->data_end - xdp->data)) + if (xdp_ok_fwd_dev(obj->dev, xdpf->len)) return false; return true; @@ -586,14 +581,13 @@ static int get_upper_ifindexes(struct net_device *dev, int *indexes) return n; } -int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, +int dev_map_enqueue_multi(struct xdp_frame *xdpf, struct net_device *dev_rx, struct bpf_map *map, bool exclude_ingress) { struct bpf_dtab *dtab = container_of(map, struct bpf_dtab, map); struct bpf_dtab_netdev *dst, *last_dst = NULL; int excluded_devices[1+MAX_NEST_DEV]; struct hlist_head *head; - struct xdp_frame *xdpf; int num_excluded = 0; unsigned int i; int err; @@ -603,15 +597,11 @@ int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, excluded_devices[num_excluded++] = dev_rx->ifindex; } - xdpf = xdp_convert_buff_to_frame(xdp); - if (unlikely(!xdpf)) - return -EOVERFLOW; - if (map->map_type == BPF_MAP_TYPE_DEVMAP) { for (i = 0; i < map->max_entries; i++) { dst = rcu_dereference_check(dtab->netdev_map[i], rcu_read_lock_bh_held()); - if (!is_valid_dst(dst, xdp)) + if (!is_valid_dst(dst, xdpf)) continue; if (is_ifindex_excluded(excluded_devices, num_excluded, dst->dev->ifindex)) @@ -634,7 +624,7 @@ int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx, head = dev_map_index_hash(dtab, i); hlist_for_each_entry_rcu(dst, head, index_hlist, lockdep_is_held(&dtab->index_lock)) { - if (!is_valid_dst(dst, xdp)) + if (!is_valid_dst(dst, xdpf)) continue; if (is_ifindex_excluded(excluded_devices, num_excluded, diff --git a/net/core/filter.c b/net/core/filter.c index 606ab5a98a1a..81b57ea9ad35 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -3964,12 +3964,24 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, enum bpf_map_type map_type = ri->map_type; void *fwd = ri->tgt_value; u32 map_id = ri->map_id; + struct xdp_frame *xdpf; struct bpf_map *map; int err; ri->map_id = 0; /* Valid map id idr range: [1,INT_MAX[ */ ri->map_type = BPF_MAP_TYPE_UNSPEC; + if (map_type == BPF_MAP_TYPE_XSKMAP) { + err = __xsk_map_redirect(fwd, xdp); + goto out; + } + + xdpf = xdp_convert_buff_to_frame(xdp); + if (unlikely(!xdpf)) { + err = -EOVERFLOW; + goto err; + } + switch (map_type) { case BPF_MAP_TYPE_DEVMAP: fallthrough; @@ -3977,17 +3989,14 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, map = READ_ONCE(ri->map); if (unlikely(map)) { WRITE_ONCE(ri->map, NULL); - err = dev_map_enqueue_multi(xdp, dev, map, + err = dev_map_enqueue_multi(xdpf, dev, map, ri->flags & BPF_F_EXCLUDE_INGRESS); } else { - err = dev_map_enqueue(fwd, xdp, dev); + err = dev_map_enqueue(fwd, xdpf, dev); } break; case BPF_MAP_TYPE_CPUMAP: - err = cpu_map_enqueue(fwd, xdp, dev); - break; - case BPF_MAP_TYPE_XSKMAP: - err = __xsk_map_redirect(fwd, xdp); + err = cpu_map_enqueue(fwd, xdpf, dev); break; case BPF_MAP_TYPE_UNSPEC: if (map_id == INT_MAX) { @@ -3996,7 +4005,7 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, err = -EINVAL; break; } - err = dev_xdp_enqueue(fwd, xdp, dev); + err = dev_xdp_enqueue(fwd, xdpf, dev); break; } fallthrough; @@ -4004,6 +4013,7 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, err = -EBADRQC; } +out: if (unlikely(err)) goto err; From patchwork Mon Jan 3 15:08:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Patchwork-Id: 12702590 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C7454C433EF for ; Mon, 3 Jan 2022 15:08:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233800AbiACPIc (ORCPT ); Mon, 3 Jan 2022 10:08:32 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:37690 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233869AbiACPIa (ORCPT ); Mon, 3 Jan 2022 10:08:30 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1641222509; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lGlVjG9u8jJMDdtq/pUyG8OIadV06H5nUns87f244fk=; b=NZUhJV0QbvVjX4F/mpSC/XBGL4Rdyn7w4Zvn8RP1gFIpxc8nzbyozgo0vLJ749eoVp19l9 Ys/7g91JJnkZmxrM2PZ8vVzpUq25N5DeK/yzMsZj0Hwr7agsxSv49vwspXzstoIwXJxlAa XSnnlwehHnx7FtfELPJJwsA41Si4eC4= Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-1-jvuEIYYBMDmOlpdCnin2RA-1; Mon, 03 Jan 2022 10:08:28 -0500 X-MC-Unique: jvuEIYYBMDmOlpdCnin2RA-1 Received: by mail-ed1-f71.google.com with SMTP id g2-20020a056402424200b003f8ee03207eso16224769edb.7 for ; Mon, 03 Jan 2022 07:08:28 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lGlVjG9u8jJMDdtq/pUyG8OIadV06H5nUns87f244fk=; b=D1abBnLV3WvUUnyIbJyzqIgV4zxzEq9s/fd1xB7sn4SML7Ehv6RMjJMts4s6USymYp 5qW6YQmtaMujtaP8dCLKPo2bs2UAd7XwUXY+fhS78Tpe32HTJfQasfYiNeUgwVuKVYrw eg4i9tkB2AjW2EKLHDsTGihRkC/xDsBgjOK4efmbpOH8Xg63izpjEWcmnolZ/rooAdwd o4KyPuU7QGhcT1cKhi9YoH7zaTHmtP5+2hiPYIjnC6l6NXCVhDyA3yUJKDR0ZCUp0bUY zdo9NSbrqDnwpGFu6jHVNb2jcmgOdA4a2VoJB9Afb5XJHQO1clmHw/VOB6l6w4Cswta0 XVbw== X-Gm-Message-State: AOAM530EhYDVlV/tlTH+2/pFp2+S1oym+DrYumPk6Hh/cJ9/9ucWhd8p g4lN5XEUNy0ljk9U4RXI3/ariSH4CeXEjqYZCzag09Vy+U+vsWJMxGkAu/GmuUV7SDpVfEzqXTy PsYO9RBMTuI94a+5r X-Received: by 2002:a17:906:2cd5:: with SMTP id r21mr37638651ejr.435.1641222505438; Mon, 03 Jan 2022 07:08:25 -0800 (PST) X-Google-Smtp-Source: ABdhPJwrOCi0fyye1AEKcpCPxzHe5FwTTrTTgYNDB1QzJYEp4ptU4OnyPfwpJNZGM5XlsNCsx4yWlQ== X-Received: by 2002:a17:906:2cd5:: with SMTP id r21mr37638524ejr.435.1641222503428; Mon, 03 Jan 2022 07:08:23 -0800 (PST) Received: from alrua-x1.borgediget.toke.dk ([2a0c:4d80:42:443::2]) by smtp.gmail.com with ESMTPSA id v19sm13856217edx.75.2022.01.03.07.08.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Jan 2022 07:08:22 -0800 (PST) Received: by alrua-x1.borgediget.toke.dk (Postfix, from userid 1000) id 67865181F2B; Mon, 3 Jan 2022 16:08:21 +0100 (CET) From: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , "David S. Miller" , Jakub Kicinski , Jesper Dangaard Brouer Cc: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= , netdev@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH bpf-next v5 5/7] xdp: add xdp_do_redirect_frame() for pre-computed xdp_frames Date: Mon, 3 Jan 2022 16:08:10 +0100 Message-Id: <20220103150812.87914-6-toke@redhat.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220103150812.87914-1-toke@redhat.com> References: <20220103150812.87914-1-toke@redhat.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Add an xdp_do_redirect_frame() variant which supports pre-computed xdp_frame structures. This will be used in bpf_prog_run() to avoid having to write to the xdp_frame structure when the XDP program doesn't modify the frame boundaries. Signed-off-by: Toke Høiland-Jørgensen --- include/linux/filter.h | 4 +++ net/core/filter.c | 65 +++++++++++++++++++++++++++++++++++------- 2 files changed, 58 insertions(+), 11 deletions(-) diff --git a/include/linux/filter.h b/include/linux/filter.h index 60eec80fa1d4..71fa57b88bfc 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -1019,6 +1019,10 @@ int xdp_do_generic_redirect(struct net_device *dev, struct sk_buff *skb, int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, struct bpf_prog *prog); +int xdp_do_redirect_frame(struct net_device *dev, + struct xdp_buff *xdp, + struct xdp_frame *xdpf, + struct bpf_prog *prog); void xdp_do_flush(void); /* The xdp_do_flush_map() helper has been renamed to drop the _map suffix, as diff --git a/net/core/filter.c b/net/core/filter.c index 81b57ea9ad35..cceb4cf52519 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -3957,26 +3957,44 @@ u32 xdp_master_redirect(struct xdp_buff *xdp) } EXPORT_SYMBOL_GPL(xdp_master_redirect); -int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, - struct bpf_prog *xdp_prog) +static inline int __xdp_do_redirect_xsk(struct bpf_redirect_info *ri, + struct net_device *dev, + struct xdp_buff *xdp, + struct bpf_prog *xdp_prog) { - struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); enum bpf_map_type map_type = ri->map_type; void *fwd = ri->tgt_value; u32 map_id = ri->map_id; - struct xdp_frame *xdpf; - struct bpf_map *map; int err; ri->map_id = 0; /* Valid map id idr range: [1,INT_MAX[ */ ri->map_type = BPF_MAP_TYPE_UNSPEC; - if (map_type == BPF_MAP_TYPE_XSKMAP) { - err = __xsk_map_redirect(fwd, xdp); - goto out; - } + err = __xsk_map_redirect(fwd, xdp); + if (unlikely(err)) + goto err; + + _trace_xdp_redirect_map(dev, xdp_prog, fwd, map_type, map_id, ri->tgt_index); + return 0; +err: + _trace_xdp_redirect_map_err(dev, xdp_prog, fwd, map_type, map_id, ri->tgt_index, err); + return err; +} + +static __always_inline int __xdp_do_redirect_frame(struct bpf_redirect_info *ri, + struct net_device *dev, + struct xdp_frame *xdpf, + struct bpf_prog *xdp_prog) +{ + enum bpf_map_type map_type = ri->map_type; + void *fwd = ri->tgt_value; + u32 map_id = ri->map_id; + struct bpf_map *map; + int err; + + ri->map_id = 0; /* Valid map id idr range: [1,INT_MAX[ */ + ri->map_type = BPF_MAP_TYPE_UNSPEC; - xdpf = xdp_convert_buff_to_frame(xdp); if (unlikely(!xdpf)) { err = -EOVERFLOW; goto err; @@ -4013,7 +4031,6 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, err = -EBADRQC; } -out: if (unlikely(err)) goto err; @@ -4023,8 +4040,34 @@ int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, _trace_xdp_redirect_map_err(dev, xdp_prog, fwd, map_type, map_id, ri->tgt_index, err); return err; } + +int xdp_do_redirect(struct net_device *dev, struct xdp_buff *xdp, + struct bpf_prog *xdp_prog) +{ + struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + enum bpf_map_type map_type = ri->map_type; + + if (map_type == BPF_MAP_TYPE_XSKMAP) + return __xdp_do_redirect_xsk(ri, dev, xdp, xdp_prog); + + return __xdp_do_redirect_frame(ri, dev, xdp_convert_buff_to_frame(xdp), + xdp_prog); +} EXPORT_SYMBOL_GPL(xdp_do_redirect); +int xdp_do_redirect_frame(struct net_device *dev, struct xdp_buff *xdp, + struct xdp_frame *xdpf, struct bpf_prog *xdp_prog) +{ + struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + enum bpf_map_type map_type = ri->map_type; + + if (map_type == BPF_MAP_TYPE_XSKMAP) + return __xdp_do_redirect_xsk(ri, dev, xdp, xdp_prog); + + return __xdp_do_redirect_frame(ri, dev, xdpf, xdp_prog); +} +EXPORT_SYMBOL_GPL(xdp_do_redirect_frame); + static int xdp_do_generic_redirect_map(struct net_device *dev, struct sk_buff *skb, struct xdp_buff *xdp, From patchwork Mon Jan 3 15:08:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Patchwork-Id: 12702592 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1D382C433EF for ; Mon, 3 Jan 2022 15:08:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233854AbiACPId (ORCPT ); Mon, 3 Jan 2022 10:08:33 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:29367 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233889AbiACPIc (ORCPT ); Mon, 3 Jan 2022 10:08:32 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1641222511; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4GGO6N4bCUKxV1eNBvyBL2tJVvDzjOAuLukvcKL/rog=; b=QVDs4tnQeiYkA5Dt3J8qdCT+/XCdoJZnZTYRy5dqoXYiS9gjfG+h5BFPpjvpFs//3KajII Wno3dng8DJD6YCcm4+EZv8OyAWwWEe2VSnE6wzJyODnW5tLI9+84PQILOrPblq5iNcYE+a l8Afh97mJLtjENTOOTzjFba57aLWGuk= Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-433-wOYoJp6lNUyJxhraW1exHg-1; Mon, 03 Jan 2022 10:08:30 -0500 X-MC-Unique: wOYoJp6lNUyJxhraW1exHg-1 Received: by mail-ed1-f72.google.com with SMTP id dz8-20020a0564021d4800b003f897935eb3so23099777edb.12 for ; Mon, 03 Jan 2022 07:08:30 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4GGO6N4bCUKxV1eNBvyBL2tJVvDzjOAuLukvcKL/rog=; b=yy5lpdXGVt/dCYcRl2RHTGLpSIvzWs3gfpP5Ct/kV1rMr8h9ZOTWRoFizKWsVy1rre QvXf3vwBw+DCkWmIbhQaJYvESHaufu3cj3Svao/qk2nTFjtzcwOGl0MH1kUHwADvogA0 YjQcd3rIvZ8Cdfpzq7FIJTUDzGQ89UQDbN39bw8F8iNRXOJmFEkZx+R2zrFCv5EI3AMk fF9FUXLPWtcVuGZ/4gCu4teTLOenJUCqjoQ41QLUtB2WBA48NuGwnOpPAm/t1Gv4/ogt 9/3VKXqFykGUYa4ZFdVL5ipcwizf+JyPwGD4QRfVXY5CUiMYCRzHJmQrN+k5HZnZQn3V sIzw== X-Gm-Message-State: AOAM533rqi4yvNFP0XovYKnf/8Tle6q04TagOUqwM7wzS8xq2FkMEyu4 t4AcE6ODJbF6mCkg7yzQDxPpZ+x72xaEiSE2FoeYupp5UwwiVQ6j8QjrKaAxPZMAHNsoH1TiKwF 2VMDO7zEwOjH6kTW1 X-Received: by 2002:a05:6402:2787:: with SMTP id b7mr38870203ede.362.1641222506596; Mon, 03 Jan 2022 07:08:26 -0800 (PST) X-Google-Smtp-Source: ABdhPJzK+Txc8/Bne8wUO9SrV2TpsZ6gBn95YOiYiW4pA8MfClahNUVAJoKxvZ1ns/l09jzqHm+S7Q== X-Received: by 2002:a05:6402:2787:: with SMTP id b7mr38870098ede.362.1641222505169; Mon, 03 Jan 2022 07:08:25 -0800 (PST) Received: from alrua-x1.borgediget.toke.dk ([2a0c:4d80:42:443::2]) by smtp.gmail.com with ESMTPSA id oz20sm10660127ejc.60.2022.01.03.07.08.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Jan 2022 07:08:24 -0800 (PST) Received: by alrua-x1.borgediget.toke.dk (Postfix, from userid 1000) id 389CA180300; Mon, 3 Jan 2022 16:08:22 +0100 (CET) From: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , "David S. Miller" , Jakub Kicinski , Jesper Dangaard Brouer Cc: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= , netdev@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH bpf-next v5 6/7] bpf: Add "live packet" mode for XDP in bpf_prog_run() Date: Mon, 3 Jan 2022 16:08:11 +0100 Message-Id: <20220103150812.87914-7-toke@redhat.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220103150812.87914-1-toke@redhat.com> References: <20220103150812.87914-1-toke@redhat.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net This adds support for running XDP programs through bpf_prog_run() in a mode that enables live packet processing of the resulting frames. Previous uses of bpf_prog_run() for XDP returned the XDP program return code and the modified packet data to userspace, which is useful for unit testing of XDP programs. This patch adds a new mode with different semantics. When enabled through the new BPF_F_TEST_XDP_LIVE_FRAMES flag, the XDP program return codes will be honoured: returning XDP_PASS will result in the frame being injected into the networking stack as if it came from the selected networking interface, while returning XDP_TX and XDP_REDIRECT will result in the frame being transmitted out an egress interface. XDP_TX is translated into an XDP_REDIRECT operation, since the real XDP_TX action is only possible from within the network drivers themselves, not from the process context where bpf_prog_run() is executed. To achieve this new mode of operation, we create a page pool instance while setting up the test run, and feed pages from that into the XDP program. The setup cost of this is amortised over the number of repetitions specified by userspace. To support performance testing use case, we further optimise the setup step so that all pages in the pool are pre-initialised with the packet data, and pre-computed context and xdp_frame objects stored at the start of each page. This makes it possible to entirely avoid touching the page content on each XDP program invocation, and enables sending up to 11.5 Mpps/core on my test box. Because the data pages are recycled by the page pool, and the test runner doesn't re-initialise them for each run, subsequent invocations of the XDP program will see the packet data in the state it was after the last time it ran on that particular page. This means that an XDP program that modifies the packet before redirecting it has to be careful about which assumptions it makes about the packet content, but that is only an issue for the most naively written programs. Enabling the new flag is only allowed when not setting ctx_out and data_out in the test specification, since using it means frames will be redirected somewhere else, so they can't be returned. Signed-off-by: Toke Høiland-Jørgensen --- include/uapi/linux/bpf.h | 2 + kernel/bpf/Kconfig | 1 + net/bpf/test_run.c | 287 ++++++++++++++++++++++++++++++++- tools/include/uapi/linux/bpf.h | 2 + 4 files changed, 284 insertions(+), 8 deletions(-) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index b0383d371b9a..5ef20deaf49f 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -1225,6 +1225,8 @@ enum { /* If set, run the test on the cpu specified by bpf_attr.test.cpu */ #define BPF_F_TEST_RUN_ON_CPU (1U << 0) +/* If set, XDP frames will be transmitted after processing */ +#define BPF_F_TEST_XDP_LIVE_FRAMES (1U << 1) /* type for BPF_ENABLE_STATS */ enum bpf_stats_type { diff --git a/kernel/bpf/Kconfig b/kernel/bpf/Kconfig index d24d518ddd63..c8c920020d11 100644 --- a/kernel/bpf/Kconfig +++ b/kernel/bpf/Kconfig @@ -30,6 +30,7 @@ config BPF_SYSCALL select TASKS_TRACE_RCU select BINARY_PRINTF select NET_SOCK_MSG if NET + select PAGE_POOL if NET default n help Enable the bpf() system call that allows to manipulate BPF programs diff --git a/net/bpf/test_run.c b/net/bpf/test_run.c index 46dd95755967..11543819f0ef 100644 --- a/net/bpf/test_run.c +++ b/net/bpf/test_run.c @@ -14,6 +14,7 @@ #include #include #include +#include #include #include #include @@ -52,10 +53,11 @@ static void bpf_test_timer_leave(struct bpf_test_timer *t) rcu_read_unlock(); } -static bool bpf_test_timer_continue(struct bpf_test_timer *t, u32 repeat, int *err, u32 *duration) +static bool bpf_test_timer_continue(struct bpf_test_timer *t, int iterations, + u32 repeat, int *err, u32 *duration) __must_hold(rcu) { - t->i++; + t->i += iterations; if (t->i >= repeat) { /* We're done. */ t->time_spent += ktime_get_ns() - t->time_start; @@ -87,6 +89,267 @@ static bool bpf_test_timer_continue(struct bpf_test_timer *t, u32 repeat, int *e return false; } +/* We put this struct at the head of each page with a context and frame + * initialised when the page is allocated, so we don't have to do this on each + * repetition of the test run. + */ +struct xdp_page_head { + struct xdp_buff orig_ctx; + struct xdp_buff ctx; + struct xdp_frame frm; + u8 data[]; +}; + +struct xdp_test_data { + struct xdp_buff *orig_ctx; + struct xdp_rxq_info rxq; + struct net_device *dev; + struct page_pool *pp; + u32 frame_cnt; +}; + +#define TEST_XDP_FRAME_SIZE (PAGE_SIZE - sizeof(struct xdp_page_head) \ + - sizeof(struct skb_shared_info)) +#define TEST_XDP_BATCH 64 + +static void xdp_test_run_init_page(struct page *page, void *arg) +{ + struct xdp_page_head *head = phys_to_virt(page_to_phys(page)); + struct xdp_buff *new_ctx, *orig_ctx; + u32 headroom = XDP_PACKET_HEADROOM; + struct xdp_test_data *xdp = arg; + size_t frm_len, meta_len; + struct xdp_frame *frm; + void *data; + + orig_ctx = xdp->orig_ctx; + frm_len = orig_ctx->data_end - orig_ctx->data_meta; + meta_len = orig_ctx->data - orig_ctx->data_meta; + headroom -= meta_len; + + new_ctx = &head->ctx; + frm = &head->frm; + data = &head->data; + memcpy(data + headroom, orig_ctx->data_meta, frm_len); + + xdp_init_buff(new_ctx, TEST_XDP_FRAME_SIZE, &xdp->rxq); + xdp_prepare_buff(new_ctx, data, headroom, frm_len, true); + new_ctx->data_meta = new_ctx->data + meta_len; + + xdp_update_frame_from_buff(new_ctx, frm); + frm->mem = new_ctx->rxq->mem; + + memcpy(&head->orig_ctx, new_ctx, sizeof(head->orig_ctx)); +} + +static int xdp_test_run_setup(struct xdp_test_data *xdp, struct xdp_buff *orig_ctx) +{ + struct xdp_mem_info mem = {}; + struct page_pool *pp; + int err; + struct page_pool_params pp_params = { + .order = 0, + .flags = 0, + .pool_size = NAPI_POLL_WEIGHT * 2, + .nid = NUMA_NO_NODE, + .max_len = TEST_XDP_FRAME_SIZE, + .init_callback = xdp_test_run_init_page, + .init_arg = xdp, + }; + + pp = page_pool_create(&pp_params); + if (IS_ERR(pp)) + return PTR_ERR(pp); + + /* will copy 'mem.id' into pp->xdp_mem_id */ + err = xdp_reg_mem_model(&mem, MEM_TYPE_PAGE_POOL, pp); + if (err) { + page_pool_destroy(pp); + return err; + } + xdp->pp = pp; + + /* We create a 'fake' RXQ referencing the original dev, but with an + * xdp_mem_info pointing to our page_pool + */ + xdp_rxq_info_reg(&xdp->rxq, orig_ctx->rxq->dev, 0, 0); + xdp->rxq.mem.type = MEM_TYPE_PAGE_POOL; + xdp->rxq.mem.id = pp->xdp_mem_id; + xdp->dev = orig_ctx->rxq->dev; + xdp->orig_ctx = orig_ctx; + + return 0; +} + +static void xdp_test_run_teardown(struct xdp_test_data *xdp) +{ + struct xdp_mem_info mem = { + .id = xdp->pp->xdp_mem_id, + .type = MEM_TYPE_PAGE_POOL, + }; + + xdp_unreg_mem_model(&mem); +} + +static bool ctx_was_changed(struct xdp_page_head *head) +{ + return head->orig_ctx.data != head->ctx.data || + head->orig_ctx.data_meta != head->ctx.data_meta || + head->orig_ctx.data_end != head->ctx.data_end; +} + +static void reset_ctx(struct xdp_page_head *head) +{ + if (likely(!ctx_was_changed(head))) + return; + + head->ctx.data = head->orig_ctx.data; + head->ctx.data_meta = head->orig_ctx.data_meta; + head->ctx.data_end = head->orig_ctx.data_end; + xdp_update_frame_from_buff(&head->ctx, &head->frm); +} + +static int xdp_recv_frames(struct xdp_frame **frames, int nframes, + struct net_device *dev) +{ + gfp_t gfp = __GFP_ZERO | GFP_ATOMIC; + void *skbs[TEST_XDP_BATCH]; + int i, n; + LIST_HEAD(list); + + n = kmem_cache_alloc_bulk(skbuff_head_cache, gfp, nframes, skbs); + if (unlikely(n == 0)) { + for (i = 0; i < nframes; i++) + xdp_return_frame(frames[i]); + return -ENOMEM; + } + + for (i = 0; i < nframes; i++) { + struct xdp_frame *xdpf = frames[i]; + struct sk_buff *skb = skbs[i]; + + skb = __xdp_build_skb_from_frame(xdpf, skb, dev); + if (!skb) { + xdp_return_frame(xdpf); + continue; + } + + list_add_tail(&skb->list, &list); + } + netif_receive_skb_list(&list); + return 0; +} + +static int xdp_test_run_batch(struct xdp_test_data *xdp, struct bpf_prog *prog, + u32 repeat) +{ + struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + int err = 0, act, ret, i, nframes = 0, batch_sz; + struct xdp_frame *frames[TEST_XDP_BATCH]; + struct xdp_page_head *head; + struct xdp_frame *frm; + bool redirect = false; + struct xdp_buff *ctx; + struct page *page; + + batch_sz = min_t(u32, repeat, TEST_XDP_BATCH); + xdp_set_return_frame_no_direct(); + + for (i = 0; i < batch_sz; i++) { + page = page_pool_dev_alloc_pages(xdp->pp); + if (!page) { + err = -ENOMEM; + goto out; + } + + head = phys_to_virt(page_to_phys(page)); + reset_ctx(head); + ctx = &head->ctx; + frm = &head->frm; + xdp->frame_cnt++; + + act = bpf_prog_run_xdp(prog, ctx); + + /* if program changed pkt bounds we need to update the xdp_frame */ + if (unlikely(ctx_was_changed(head))) { + err = xdp_update_frame_from_buff(ctx, frm); + if (err) { + xdp_return_buff(ctx); + goto out; + } + } + + switch (act) { + case XDP_TX: + /* we can't do a real XDP_TX since we're not in the + * driver, so turn it into a REDIRECT back to the same + * index + */ + ri->tgt_index = xdp->dev->ifindex; + ri->map_id = INT_MAX; + ri->map_type = BPF_MAP_TYPE_UNSPEC; + fallthrough; + case XDP_REDIRECT: + redirect = true; + err = xdp_do_redirect_frame(xdp->dev, ctx, frm, prog); + if (err) { + xdp_return_buff(ctx); + goto out; + } + break; + case XDP_PASS: + frames[nframes++] = frm; + break; + default: + bpf_warn_invalid_xdp_action(NULL, prog, act); + fallthrough; + case XDP_DROP: + xdp_return_buff(ctx); + break; + } + } + +out: + if (redirect) + xdp_do_flush(); + if (nframes) { + ret = xdp_recv_frames(frames, nframes, xdp->dev); + if (ret) + err = ret; + } + + xdp_clear_return_frame_no_direct(); + return err; +} + +static int bpf_test_run_xdp_live(struct bpf_prog *prog, struct xdp_buff *ctx, + u32 repeat, u32 *time) + +{ + struct bpf_test_timer t = { .mode = NO_MIGRATE }; + struct xdp_test_data xdp = {}; + int ret; + + if (!repeat) + repeat = 1; + + ret = xdp_test_run_setup(&xdp, ctx); + if (ret) + return ret; + + bpf_test_timer_enter(&t); + do { + xdp.frame_cnt = 0; + ret = xdp_test_run_batch(&xdp, prog, repeat - t.i); + if (unlikely(ret < 0)) + break; + } while (bpf_test_timer_continue(&t, xdp.frame_cnt, repeat, &ret, time)); + bpf_test_timer_leave(&t); + + xdp_test_run_teardown(&xdp); + return ret; +} + static int bpf_test_run(struct bpf_prog *prog, void *ctx, u32 repeat, u32 *retval, u32 *time, bool xdp) { @@ -118,7 +381,7 @@ static int bpf_test_run(struct bpf_prog *prog, void *ctx, u32 repeat, *retval = bpf_prog_run_xdp(prog, ctx); else *retval = bpf_prog_run(prog, ctx); - } while (bpf_test_timer_continue(&t, repeat, &ret, time)); + } while (bpf_test_timer_continue(&t, 1, repeat, &ret, time)); bpf_reset_run_ctx(old_ctx); bpf_test_timer_leave(&t); @@ -757,13 +1020,14 @@ static void xdp_convert_buff_to_md(struct xdp_buff *xdp, struct xdp_md *xdp_md) int bpf_prog_test_run_xdp(struct bpf_prog *prog, const union bpf_attr *kattr, union bpf_attr __user *uattr) { + bool do_live = (kattr->test.flags & BPF_F_TEST_XDP_LIVE_FRAMES); u32 tailroom = SKB_DATA_ALIGN(sizeof(struct skb_shared_info)); u32 headroom = XDP_PACKET_HEADROOM; u32 size = kattr->test.data_size_in; u32 repeat = kattr->test.repeat; struct netdev_rx_queue *rxqueue; struct xdp_buff xdp = {}; - u32 retval, duration; + u32 retval = 0, duration; struct xdp_md *ctx; u32 max_data_sz; void *data; @@ -773,6 +1037,9 @@ int bpf_prog_test_run_xdp(struct bpf_prog *prog, const union bpf_attr *kattr, prog->expected_attach_type == BPF_XDP_CPUMAP) return -EINVAL; + if (kattr->test.flags & ~BPF_F_TEST_XDP_LIVE_FRAMES) + return -EINVAL; + ctx = bpf_ctx_init(kattr, sizeof(struct xdp_md)); if (IS_ERR(ctx)) return PTR_ERR(ctx); @@ -781,7 +1048,8 @@ int bpf_prog_test_run_xdp(struct bpf_prog *prog, const union bpf_attr *kattr, /* There can't be user provided data before the meta data */ if (ctx->data_meta || ctx->data_end != size || ctx->data > ctx->data_end || - unlikely(xdp_metalen_invalid(ctx->data))) + unlikely(xdp_metalen_invalid(ctx->data)) || + (do_live && (kattr->test.data_out || kattr->test.ctx_out))) goto free_ctx; /* Meta data is allocated from the headroom */ headroom -= ctx->data; @@ -807,7 +1075,10 @@ int bpf_prog_test_run_xdp(struct bpf_prog *prog, const union bpf_attr *kattr, if (repeat > 1) bpf_prog_change_xdp(NULL, prog); - ret = bpf_test_run(prog, &xdp, repeat, &retval, &duration, true); + if (do_live) + ret = bpf_test_run_xdp_live(prog, &xdp, repeat, &duration); + else + ret = bpf_test_run(prog, &xdp, repeat, &retval, &duration, true); /* We convert the xdp_buff back to an xdp_md before checking the return * code so the reference count of any held netdevice will be decremented * even if the test run failed. @@ -905,7 +1176,7 @@ int bpf_prog_test_run_flow_dissector(struct bpf_prog *prog, do { retval = bpf_flow_dissect(prog, &ctx, eth->h_proto, ETH_HLEN, size, flags); - } while (bpf_test_timer_continue(&t, repeat, &ret, &duration)); + } while (bpf_test_timer_continue(&t, 1, repeat, &ret, &duration)); bpf_test_timer_leave(&t); if (ret < 0) @@ -1000,7 +1271,7 @@ int bpf_prog_test_run_sk_lookup(struct bpf_prog *prog, const union bpf_attr *kat do { ctx.selected_sk = NULL; retval = BPF_PROG_SK_LOOKUP_RUN_ARRAY(progs, ctx, bpf_prog_run); - } while (bpf_test_timer_continue(&t, repeat, &ret, &duration)); + } while (bpf_test_timer_continue(&t, 1, repeat, &ret, &duration)); bpf_test_timer_leave(&t); if (ret < 0) diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index b0383d371b9a..5ef20deaf49f 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -1225,6 +1225,8 @@ enum { /* If set, run the test on the cpu specified by bpf_attr.test.cpu */ #define BPF_F_TEST_RUN_ON_CPU (1U << 0) +/* If set, XDP frames will be transmitted after processing */ +#define BPF_F_TEST_XDP_LIVE_FRAMES (1U << 1) /* type for BPF_ENABLE_STATS */ enum bpf_stats_type { From patchwork Mon Jan 3 15:08:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Patchwork-Id: 12702591 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 360C8C4332F for ; Mon, 3 Jan 2022 15:08:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233902AbiACPIf (ORCPT ); Mon, 3 Jan 2022 10:08:35 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:35503 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233861AbiACPIc (ORCPT ); Mon, 3 Jan 2022 10:08:32 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1641222511; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=liH+qN0WmF6KG3mO/mKBj0yNsvaPZ08gu8hDBnDNrZY=; b=Vij0pA4H7U5ZdAx7NRFmnu6z8hJpspKMX+XVb9hrybdR/AMlXXWUuj74BtcndNra6lvWOE JbgcHeCWo4YG6pvRjlba+SiyuvwlYkVv830b6lMFLLUo29+AoWKvM1y6CNoqNV8rdnvHD1 8+Djktdzcf2aZJrEVECiLu5rGN8UezY= Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-401-jxY0Yd49MSmSa9wfIa0q3w-1; Mon, 03 Jan 2022 10:08:30 -0500 X-MC-Unique: jxY0Yd49MSmSa9wfIa0q3w-1 Received: by mail-ed1-f72.google.com with SMTP id g2-20020a056402424200b003f8ee03207eso16224859edb.7 for ; Mon, 03 Jan 2022 07:08:30 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=liH+qN0WmF6KG3mO/mKBj0yNsvaPZ08gu8hDBnDNrZY=; b=XlqDVZhTrWV8UuWfcjT614CBlLpVNK6UixgSVKczhMt7E9Gm94KRULfZhwSM/H5Wsf Q/lIMn7qI5N6eOJjJEB3ScVPEW3BxvRFyIyS+eniJcVjfQKBuwAB7tOeKndsE6SF+AOB +UXylx0Hg6xriR1SZ7qE17Swu1B+CfwcoQzRhA1BaAff07K9iWw/JRj9eHsuuGkEttDJ mDMhFK7AtQAlluQ7K2lSJEkMOcPadhcyF7eP/4LY/qL7BHhJV3ecaaMLukPWcbvqL07e 00KA5TfHnroMEtgSwEq/OFU9VKrdtMg3aTy6o9nWHXo/MQKwXbuCIaZ+am6HGlJASqLw bZ8g== X-Gm-Message-State: AOAM530+6B19715Op5D6es0oLeEMXTDusKNTN6GjU7W3lPyfiZo4eEjA 1oGt1vEf0gWW8J4s4S8LfQ+3xCw8ted6ScxFUSDJWjAEKAugxnYRBC83NoNTZqiwl/kfLB/4tqK DhI/k6OuSct8e308Y X-Received: by 2002:a17:906:d552:: with SMTP id cr18mr36275492ejc.260.1641222507414; Mon, 03 Jan 2022 07:08:27 -0800 (PST) X-Google-Smtp-Source: ABdhPJxsRVCvB++XaDGgB0IoilqLN7chbS8Duehkorg0cuF3gtkSNzcGZOpHIEx/bJEgW06PjwrcLw== X-Received: by 2002:a17:906:d552:: with SMTP id cr18mr36275402ejc.260.1641222506100; Mon, 03 Jan 2022 07:08:26 -0800 (PST) Received: from alrua-x1.borgediget.toke.dk ([2a0c:4d80:42:443::2]) by smtp.gmail.com with ESMTPSA id c12sm13989442edx.80.2022.01.03.07.08.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Jan 2022 07:08:25 -0800 (PST) Received: by alrua-x1.borgediget.toke.dk (Postfix, from userid 1000) id C6B85181F2A; Mon, 3 Jan 2022 16:08:22 +0100 (CET) From: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= To: Alexei Starovoitov , Daniel Borkmann , "David S. Miller" , Jakub Kicinski , Jesper Dangaard Brouer , John Fastabend , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , KP Singh Cc: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= , Shuah Khan , netdev@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH bpf-next v5 7/7] selftests/bpf: Add selftest for XDP_REDIRECT in bpf_prog_run() Date: Mon, 3 Jan 2022 16:08:12 +0100 Message-Id: <20220103150812.87914-8-toke@redhat.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220103150812.87914-1-toke@redhat.com> References: <20220103150812.87914-1-toke@redhat.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net This adds a selftest for the XDP_REDIRECT facility in bpf_prog_run, that redirects packets into a veth and counts them using an XDP program on the other side of the veth pair. Signed-off-by: Toke Høiland-Jørgensen --- .../bpf/prog_tests/xdp_do_redirect.c | 118 ++++++++++++++++++ .../bpf/progs/test_xdp_do_redirect.c | 39 ++++++ 2 files changed, 157 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/xdp_do_redirect.c create mode 100644 tools/testing/selftests/bpf/progs/test_xdp_do_redirect.c diff --git a/tools/testing/selftests/bpf/prog_tests/xdp_do_redirect.c b/tools/testing/selftests/bpf/prog_tests/xdp_do_redirect.c new file mode 100644 index 000000000000..a587c351d495 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/xdp_do_redirect.c @@ -0,0 +1,118 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "test_xdp_do_redirect.skel.h" + +#define SYS(fmt, ...) \ + ({ \ + char cmd[1024]; \ + snprintf(cmd, sizeof(cmd), fmt, ##__VA_ARGS__); \ + if (!ASSERT_OK(system(cmd), cmd)) \ + goto fail; \ + }) + +struct udp_packet { + struct ethhdr eth; + struct ipv6hdr iph; + struct udphdr udp; + __u8 payload[64 - sizeof(struct udphdr) + - sizeof(struct ethhdr) - sizeof(struct ipv6hdr)]; +} __packed; + +static struct udp_packet pkt_udp = { + .eth.h_proto = __bpf_constant_htons(ETH_P_IPV6), + .eth.h_dest = {0x00, 0x11, 0x22, 0x33, 0x44, 0x55}, + .eth.h_source = {0x66, 0x77, 0x88, 0x99, 0xaa, 0xbb}, + .iph.version = 6, + .iph.nexthdr = IPPROTO_UDP, + .iph.payload_len = bpf_htons(sizeof(struct udp_packet) + - offsetof(struct udp_packet, udp)), + .iph.hop_limit = 2, + .iph.saddr.s6_addr16 = {bpf_htons(0xfc00), 0, 0, 0, 0, 0, 0, bpf_htons(1)}, + .iph.daddr.s6_addr16 = {bpf_htons(0xfc00), 0, 0, 0, 0, 0, 0, bpf_htons(2)}, + .udp.source = bpf_htons(1), + .udp.dest = bpf_htons(1), + .udp.len = bpf_htons(sizeof(struct udp_packet) + - offsetof(struct udp_packet, udp)), + .payload = {0x42}, /* receiver XDP program matches on this */ +}; + +#define NUM_PKTS 3 +void test_xdp_do_redirect(void) +{ + struct test_xdp_do_redirect *skel = NULL; + struct xdp_md ctx_in = { .data_end = sizeof(pkt_udp) }; + DECLARE_LIBBPF_OPTS(bpf_test_run_opts, opts, + .data_in = &pkt_udp, + .data_size_in = sizeof(pkt_udp), + .ctx_in = &ctx_in, + .ctx_size_in = sizeof(ctx_in), + .flags = BPF_F_TEST_XDP_LIVE_FRAMES, + .repeat = NUM_PKTS, + ); + int err, prog_fd, ifindex_src, ifindex_dst; + struct bpf_link *link; + + skel = test_xdp_do_redirect__open(); + if (!ASSERT_OK_PTR(skel, "skel")) + return; + + /* We setup a veth pair that we can not only XDP_REDIRECT packets + * between, but also route them. The test packet (defined above) has + * address information so it will be routed back out the same interface + * after it has been received, which will allow it to be picked up by + * the XDP program on the destination interface. + * + * The XDP program we run with bpf_prog_run() will cycle through all + * four return codes (DROP/PASS/TX/REDIRECT), so we should end up with + * NUM_PKTS - 1 packets seen on the dst iface. We match the packets on + * the UDP payload. + */ + SYS("ip link add veth_src type veth peer name veth_dst"); + SYS("ip link set dev veth_src address 00:11:22:33:44:55"); + SYS("ip link set dev veth_dst address 66:77:88:99:aa:bb"); + SYS("ip link set dev veth_src up"); + SYS("ip link set dev veth_dst up"); + SYS("ip addr add dev veth_src fc00::1/64"); + SYS("ip addr add dev veth_dst fc00::2/64"); + SYS("ip neigh add fc00::2 dev veth_src lladdr 66:77:88:99:aa:bb"); + SYS("sysctl -w net.ipv6.conf.all.forwarding=1"); + + ifindex_src = if_nametoindex("veth_src"); + ifindex_dst = if_nametoindex("veth_dst"); + if (!ASSERT_NEQ(ifindex_src, 0, "ifindex_src") || + !ASSERT_NEQ(ifindex_dst, 0, "ifindex_dst")) + goto fail; + + memcpy(skel->rodata->expect_dst, &pkt_udp.eth.h_dest, ETH_ALEN); + skel->rodata->ifindex_out = ifindex_src; + ctx_in.ingress_ifindex = ifindex_src; + + if (!ASSERT_OK(test_xdp_do_redirect__load(skel), "load")) + goto fail; + + link = bpf_program__attach_xdp(skel->progs.xdp_count_pkts, ifindex_dst); + if (!ASSERT_OK_PTR(link, "prog_attach")) + goto fail; + skel->links.xdp_count_pkts = link; + + prog_fd = bpf_program__fd(skel->progs.xdp_redirect_notouch); + err = bpf_prog_test_run_opts(prog_fd, &opts); + if (!ASSERT_OK(err, "prog_run")) + goto fail; + + /* wait for the packets to be flushed */ + kern_sync_rcu(); + + ASSERT_EQ(skel->bss->pkts_seen, NUM_PKTS - 1, "pkt_count"); +fail: + system("ip link del dev veth_src"); + test_xdp_do_redirect__destroy(skel); +} diff --git a/tools/testing/selftests/bpf/progs/test_xdp_do_redirect.c b/tools/testing/selftests/bpf/progs/test_xdp_do_redirect.c new file mode 100644 index 000000000000..f9ea587b0876 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/test_xdp_do_redirect.c @@ -0,0 +1,39 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include + +#define ETH_ALEN 6 +const volatile int ifindex_out; +const volatile __u8 expect_dst[ETH_ALEN]; +volatile int pkts_seen = 0; +volatile int retcode = XDP_DROP; + +SEC("xdp") +int xdp_redirect_notouch(struct xdp_md *xdp) +{ + if (retcode == XDP_REDIRECT) + bpf_redirect(ifindex_out, 0); + return retcode++; +} + +SEC("xdp") +int xdp_count_pkts(struct xdp_md *xdp) +{ + void *data = (void *)(long)xdp->data; + void *data_end = (void *)(long)xdp->data_end; + struct ethhdr *eth = data; + struct ipv6hdr *iph = (void *)(eth + 1); + struct udphdr *udp = (void *)(iph + 1); + __u8 *payload = (void *)(udp + 1); + int i; + + if (payload + 1 > data_end) + return XDP_ABORTED; + + if (iph->nexthdr == IPPROTO_UDP && *payload == 0x42) + pkts_seen++; + + return XDP_PASS; +} + +char _license[] SEC("license") = "GPL";