From patchwork Wed Oct 11 03:43:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 13416551 X-Patchwork-Delegate: kuba@kernel.org Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9AF62C2EB for ; Wed, 11 Oct 2023 03:44:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="j+IKjl74" Received: from mail-pf1-x430.google.com (mail-pf1-x430.google.com [IPv6:2607:f8b0:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D42C7CF for ; Tue, 10 Oct 2023 20:44:00 -0700 (PDT) Received: by mail-pf1-x430.google.com with SMTP id d2e1a72fcca58-690fa0eea3cso5745289b3a.0 for ; Tue, 10 Oct 2023 20:44:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696995840; x=1697600640; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cTCjODxOYvvUIwFtBooSecITmfw9zxGLvgL/AQYXLPc=; b=j+IKjl74uJ6QLf8yvvv1zgBOjj5wOAwuTjbtv2QA1UbBpvFv52IfzUHfoigblZc1pH WZztehjJRCLZ6KCfST15l3l6npRwk7cPb0G6Hp6IvCy0gLqJKCpcIITOyvEDhwLTijb8 KHy4Y2sdn4vr7EJcU+EGcU6OC0vP0I8erle43qNgNk/VTbn9joMsz16Z+SCcQSBi+ZnE UKrW9F0uNCSLlC2+JvUdx60PsgCpltEOVbeqgAH4VDkCNGVqrwQSxXe+DDIg5BhDr8pR vNTuwmdm9Ez1cQAif7QkYD9eJc6FxVM0eybCBefRsO/3tvy3r+3d7x6RkibTertJwcVB J5OA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696995840; x=1697600640; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cTCjODxOYvvUIwFtBooSecITmfw9zxGLvgL/AQYXLPc=; b=LIN9QtJZyRi1Av0oPtCxRBQIxtM5yFKrfVj///eUjncXx/IfpNUH6v/dzzoPT72wLZ oIxBLqLMr0JzNNRX8NSRiffyoAnk3MCz2dihGdCn++LCFyEOwddnHWSAgnTwX6uBeIPh gxh0Scf+imZcMoZQm7B5FdqbM8M98W0HCAUbamK5NEn3yepu2FTsmeDsDR71wrS1gqdW RTHKrC2GRNnD/UTurwD8rw2SVa1txLz6nNDtDBCIfWDTPsVSZpwtJI9i1/JomSJi1kXh FVoOYdtN+I94XdBSWjudXGXJQrR516rUNynAYsXVNSfDdwGgwKHA8rBU7m46BsAKuCgJ 5hZg== X-Gm-Message-State: AOJu0YwGPCmaXxhhnQtimQHGg+THuo1eSfR4gn+kXTN8FXuFNDPE1Hbt 8kcSIyvqkBP7PyPwPZ/ZYkPxPCO0XUObKw== X-Google-Smtp-Source: AGHT+IHMae1JIQoONMnP/2F212m0xc6ZnKS8nE3V1egaMwCvENPf21Sh4YYY5n3ai3TEtco2oAI5RQ== X-Received: by 2002:a05:6a00:22ca:b0:690:454a:dc7b with SMTP id f10-20020a056a0022ca00b00690454adc7bmr22503365pfj.28.1696995840033; Tue, 10 Oct 2023 20:44:00 -0700 (PDT) Received: from wheely.local0.net ([1.128.220.51]) by smtp.gmail.com with ESMTPSA id q30-20020a638c5e000000b0058a9621f583sm7873656pgn.44.2023.10.10.20.43.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Oct 2023 20:43:59 -0700 (PDT) From: Nicholas Piggin To: netdev@vger.kernel.org Cc: Nicholas Piggin , dev@openvswitch.org, Pravin B Shelar , Aaron Conole , "Eelco Chaudron" , "Ilya Maximets" , "Flavio Leitner" Subject: [PATCH 1/7] net: openvswitch: generalise the per-cpu flow key allocation stack Date: Wed, 11 Oct 2023 13:43:38 +1000 Message-ID: <20231011034344.104398-2-npiggin@gmail.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231011034344.104398-1-npiggin@gmail.com> References: <20231011034344.104398-1-npiggin@gmail.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: kuba@kernel.org Rather than an implicit key allocation index based on the recursion level, make this a standalone FIFO allocator. This makes it usable in other places without modifying the recursion accounting. Signed-off-by: Nicholas Piggin --- net/openvswitch/actions.c | 104 ++++++++++++++++++++++++++------------ 1 file changed, 72 insertions(+), 32 deletions(-) diff --git a/net/openvswitch/actions.c b/net/openvswitch/actions.c index fd66014d8a76..bc7a8c2fff91 100644 --- a/net/openvswitch/actions.c +++ b/net/openvswitch/actions.c @@ -59,9 +59,10 @@ struct ovs_frag_data { static DEFINE_PER_CPU(struct ovs_frag_data, ovs_frag_data_storage); -#define DEFERRED_ACTION_FIFO_SIZE 10 #define OVS_RECURSION_LIMIT 5 -#define OVS_DEFERRED_ACTION_THRESHOLD (OVS_RECURSION_LIMIT - 2) +#define NR_FLOW_KEYS 5 +#define DEFERRED_ACTION_FIFO_SIZE 10 + struct action_fifo { int head; int tail; @@ -69,27 +70,64 @@ struct action_fifo { struct deferred_action fifo[DEFERRED_ACTION_FIFO_SIZE]; }; -struct action_flow_keys { - struct sw_flow_key key[OVS_DEFERRED_ACTION_THRESHOLD]; +struct flow_key_stack { + struct sw_flow_key key[NR_FLOW_KEYS]; }; -static struct action_fifo __percpu *action_fifos; -static struct action_flow_keys __percpu *flow_keys; static DEFINE_PER_CPU(int, exec_actions_level); +static struct flow_key_stack __percpu *flow_key_stack; +static DEFINE_PER_CPU(int, flow_keys_allocated); + +static struct action_fifo __percpu *action_fifos; + +/* + * ovs_flow_key_alloc provides a per-CPU sw_flow_key allocator. keys must be + * freed in the reverse order that they were allocated in (i.e., a stack). + */ +static struct sw_flow_key *ovs_flow_key_alloc(void) +{ + struct flow_key_stack *keys = this_cpu_ptr(flow_key_stack); + int level = this_cpu_read(flow_keys_allocated); + + if (unlikely(level >= NR_FLOW_KEYS)) + return NULL; + + __this_cpu_inc(flow_keys_allocated); + + return &keys->key[level]; +} + +static void ovs_flow_key_free(struct sw_flow_key *key) +{ + struct flow_key_stack *keys = this_cpu_ptr(flow_key_stack); + int level = this_cpu_read(flow_keys_allocated); + + /* + * If these debug checks fire then keys will cease being freed + * and the allocator will become exhausted and stop working. This + * gives a graceful failure mode for programming errors. + */ + + if (WARN_ON_ONCE(level == 0)) + return; /* Underflow */ + + if (WARN_ON_ONCE(key != &keys->key[level - 1])) + return; /* Mismatched alloc/free order */ + + __this_cpu_dec(flow_keys_allocated); +} + /* Make a clone of the 'key', using the pre-allocated percpu 'flow_keys' * space. Return NULL if out of key spaces. */ static struct sw_flow_key *clone_key(const struct sw_flow_key *key_) { - struct action_flow_keys *keys = this_cpu_ptr(flow_keys); - int level = this_cpu_read(exec_actions_level); - struct sw_flow_key *key = NULL; + struct sw_flow_key *key; - if (level <= OVS_DEFERRED_ACTION_THRESHOLD) { - key = &keys->key[level - 1]; + key = ovs_flow_key_alloc(); + if (likely(key)) *key = *key_; - } return key; } @@ -1522,9 +1560,10 @@ static int clone_execute(struct datapath *dp, struct sk_buff *skb, { struct deferred_action *da; struct sw_flow_key *clone; + int err = 0; skb = last ? skb : skb_clone(skb, GFP_ATOMIC); - if (!skb) { + if (unlikely(!skb)) { /* Out of memory, skip this action. */ return 0; @@ -1536,26 +1575,27 @@ static int clone_execute(struct datapath *dp, struct sk_buff *skb, * 'flow_keys'. If clone is successful, execute the actions * without deferring. */ - clone = clone_flow_key ? clone_key(key) : key; - if (clone) { - int err = 0; + if (clone_flow_key) { + clone = clone_key(key); + if (unlikely(!clone)) + goto defer; + } else { + clone = key; + } - if (actions) { /* Sample action */ - if (clone_flow_key) - __this_cpu_inc(exec_actions_level); + if (actions) { /* Sample action */ + err = do_execute_actions(dp, skb, clone, actions, len); + } else { /* Recirc action */ + clone->recirc_id = recirc_id; + ovs_dp_process_packet(skb, clone); + } - err = do_execute_actions(dp, skb, clone, - actions, len); + if (clone_flow_key) + ovs_flow_key_free(clone); - if (clone_flow_key) - __this_cpu_dec(exec_actions_level); - } else { /* Recirc action */ - clone->recirc_id = recirc_id; - ovs_dp_process_packet(skb, clone); - } - return err; - } + return err; +defer: /* Out of 'flow_keys' space. Defer actions */ da = add_deferred_actions(skb, key, actions, len); if (da) { @@ -1642,8 +1682,8 @@ int action_fifos_init(void) if (!action_fifos) return -ENOMEM; - flow_keys = alloc_percpu(struct action_flow_keys); - if (!flow_keys) { + flow_key_stack = alloc_percpu(struct flow_key_stack); + if (!flow_key_stack) { free_percpu(action_fifos); return -ENOMEM; } @@ -1654,5 +1694,5 @@ int action_fifos_init(void) void action_fifos_exit(void) { free_percpu(action_fifos); - free_percpu(flow_keys); + free_percpu(flow_key_stack); } From patchwork Wed Oct 11 03:43:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 13416552 X-Patchwork-Delegate: kuba@kernel.org Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 51461D2E9 for ; Wed, 11 Oct 2023 03:44:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="hUEyIAgk" Received: from mail-yb1-xb2c.google.com (mail-yb1-xb2c.google.com [IPv6:2607:f8b0:4864:20::b2c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CF3F594 for ; Tue, 10 Oct 2023 20:44:06 -0700 (PDT) Received: by mail-yb1-xb2c.google.com with SMTP id 3f1490d57ef6-d8168d08bebso6739960276.0 for ; Tue, 10 Oct 2023 20:44:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696995846; x=1697600646; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vb8OHizWZvFBlQGkon8r50KJO0iPtRKwrbcTXutZ2v0=; b=hUEyIAgkRVNt0ni2j30fMJY42jJTYXUQe4qGEfG2UAoHObGrgZefeCnVSavQerrhrW sV3zqOkqHJHS4gZfBNwE2E2FbfAgUq8K88W/GCr32dkzJNymhbGzGcHxgz4gN6ApVyWu Iws+4qiLDJqLdp8H4xQt4hO0bCgZS6A55Wp61qMSJ1tmfZXdLbpOq6umBR4qrPyfqReH eIokvPW8TtAqp0HQ7p1GRB+gVAdj2uQVuu3YtqXHUDrFeO2B27HDYr5ndJq3jZJnJPhI hTBzGePKVD4f2eiUF4BWQQ+s1Qk1QyhHaHKNc0aR6r2tgUfvUQyQqZCQnwP66tczwfHU OhCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696995846; x=1697600646; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vb8OHizWZvFBlQGkon8r50KJO0iPtRKwrbcTXutZ2v0=; b=p8+nJLo3QoP6am9J8QyHwhIzKSL3/WP4pni24k4dyTq+9NYapp4EeTyQ07rD8CQALh ZwOom4epg7xGhczqEIeN4i9wqRYQvoAMU/dUXjBCSGwMkASuhGg/6JiyJjPmwIwxWets hiUjrd3XQk6b+8JqLd+xS/IUYnHQ25G8WA4kSOM3CYyOkGeFmU7H2QA4rf1Cq4XR+rQd xJq2EXG91PFGrrGmrmJEXqMdgQ0gKposU624ogXsCu12srrLOxq/6ujhFQfyOpZ2Lq7L T06OviH9Khzot21hTDRNl9LQZmvMK4CCgsxgBwDlzBFtR7DUDI8tgxGbifZH4bJ11VH3 z5Uw== X-Gm-Message-State: AOJu0YyppW6vnWpNEAhbdQ47locVim9QZLxiEu/+J/AC+Rz0i9tQJ6UM z/grV/05yqicEOLYf+sf0mnE9CiH3qmQHA== X-Google-Smtp-Source: AGHT+IGIEL5xAXUpX1rGYL4QHMgr2UzhKeiAJCCmxhWXNbz8SfmPiq6UtxlyaFh36nhvyH7g8iphGQ== X-Received: by 2002:a5b:4d2:0:b0:d81:504f:f879 with SMTP id u18-20020a5b04d2000000b00d81504ff879mr17613855ybp.28.1696995845737; Tue, 10 Oct 2023 20:44:05 -0700 (PDT) Received: from wheely.local0.net ([1.128.220.51]) by smtp.gmail.com with ESMTPSA id q30-20020a638c5e000000b0058a9621f583sm7873656pgn.44.2023.10.10.20.44.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Oct 2023 20:44:05 -0700 (PDT) From: Nicholas Piggin To: netdev@vger.kernel.org Cc: Nicholas Piggin , dev@openvswitch.org, Pravin B Shelar , Aaron Conole , "Eelco Chaudron" , "Ilya Maximets" , "Flavio Leitner" Subject: [PATCH 2/7] net: openvswitch: Use flow key allocator in ovs_vport_receive Date: Wed, 11 Oct 2023 13:43:39 +1000 Message-ID: <20231011034344.104398-3-npiggin@gmail.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231011034344.104398-1-npiggin@gmail.com> References: <20231011034344.104398-1-npiggin@gmail.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: kuba@kernel.org Rather than allocate the flow key on stack in ovs_vport_receive, use the per-cpu flow key allocator introduced with the previous change. The number of keys are increased because ovs_vport_receive can be in the recursion path too. This brings ovs_vport_receive stack usage from 544 bytes to 64 bytes on ppc64le. Signed-off-by: Nicholas Piggin --- net/openvswitch/actions.c | 6 +++--- net/openvswitch/flow.h | 3 +++ net/openvswitch/vport.c | 27 ++++++++++++++++++++------- 3 files changed, 26 insertions(+), 10 deletions(-) diff --git a/net/openvswitch/actions.c b/net/openvswitch/actions.c index bc7a8c2fff91..7a66574672d3 100644 --- a/net/openvswitch/actions.c +++ b/net/openvswitch/actions.c @@ -60,7 +60,7 @@ struct ovs_frag_data { static DEFINE_PER_CPU(struct ovs_frag_data, ovs_frag_data_storage); #define OVS_RECURSION_LIMIT 5 -#define NR_FLOW_KEYS 5 +#define NR_FLOW_KEYS 10 #define DEFERRED_ACTION_FIFO_SIZE 10 struct action_fifo { @@ -85,7 +85,7 @@ static struct action_fifo __percpu *action_fifos; * ovs_flow_key_alloc provides a per-CPU sw_flow_key allocator. keys must be * freed in the reverse order that they were allocated in (i.e., a stack). */ -static struct sw_flow_key *ovs_flow_key_alloc(void) +struct sw_flow_key *ovs_flow_key_alloc(void) { struct flow_key_stack *keys = this_cpu_ptr(flow_key_stack); int level = this_cpu_read(flow_keys_allocated); @@ -98,7 +98,7 @@ static struct sw_flow_key *ovs_flow_key_alloc(void) return &keys->key[level]; } -static void ovs_flow_key_free(struct sw_flow_key *key) +void ovs_flow_key_free(struct sw_flow_key *key) { struct flow_key_stack *keys = this_cpu_ptr(flow_key_stack); int level = this_cpu_read(flow_keys_allocated); diff --git a/net/openvswitch/flow.h b/net/openvswitch/flow.h index b5711aff6e76..612459518af9 100644 --- a/net/openvswitch/flow.h +++ b/net/openvswitch/flow.h @@ -285,6 +285,9 @@ void ovs_flow_stats_get(const struct sw_flow *, struct ovs_flow_stats *, void ovs_flow_stats_clear(struct sw_flow *); u64 ovs_flow_used_time(unsigned long flow_jiffies); +struct sw_flow_key *ovs_flow_key_alloc(void); +void ovs_flow_key_free(struct sw_flow_key *key); + int ovs_flow_key_update(struct sk_buff *skb, struct sw_flow_key *key); int ovs_flow_key_update_l3l4(struct sk_buff *skb, struct sw_flow_key *key); int ovs_flow_key_extract(const struct ip_tunnel_info *tun_info, diff --git a/net/openvswitch/vport.c b/net/openvswitch/vport.c index 972ae01a70f7..80887a17e23b 100644 --- a/net/openvswitch/vport.c +++ b/net/openvswitch/vport.c @@ -494,7 +494,7 @@ u32 ovs_vport_find_upcall_portid(const struct vport *vport, int ovs_vport_receive(struct vport *vport, struct sk_buff *skb, const struct ip_tunnel_info *tun_info) { - struct sw_flow_key key; + struct sw_flow_key *key; int error; OVS_CB(skb)->input_vport = vport; @@ -509,14 +509,27 @@ int ovs_vport_receive(struct vport *vport, struct sk_buff *skb, tun_info = NULL; } - /* Extract flow from 'skb' into 'key'. */ - error = ovs_flow_key_extract(tun_info, skb, &key); - if (unlikely(error)) { - kfree_skb(skb); - return error; + key = ovs_flow_key_alloc(); + if (unlikely(!key)) { + error = -ENOMEM; + goto err_skb; } - ovs_dp_process_packet(skb, &key); + + /* Extract flow from 'skb' into 'key'. */ + error = ovs_flow_key_extract(tun_info, skb, key); + if (unlikely(error)) + goto err_key; + + ovs_dp_process_packet(skb, key); + ovs_flow_key_free(key); + return 0; + +err_key: + ovs_flow_key_free(key); +err_skb: + kfree_skb(skb); + return error; } static int packet_length(const struct sk_buff *skb, From patchwork Wed Oct 11 03:43:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 13416553 X-Patchwork-Delegate: kuba@kernel.org Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7669EC2EB for ; Wed, 11 Oct 2023 03:44:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="dxuN+fla" Received: from mail-pf1-x436.google.com (mail-pf1-x436.google.com [IPv6:2607:f8b0:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9EA189D for ; Tue, 10 Oct 2023 20:44:11 -0700 (PDT) Received: by mail-pf1-x436.google.com with SMTP id d2e1a72fcca58-690bd8f89baso4917452b3a.2 for ; Tue, 10 Oct 2023 20:44:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696995851; x=1697600651; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3Qou2Xi0iRC4/xhndQ2mmgbtUDHApacfhk0mMbTXCXY=; b=dxuN+fla4ZRQ7k11wm/7zjis+F+8wWbGCgaPNZ4JIsEoDXtRZLHxEy+bZplR2wH0Sm Mt09FVCVE3drzRUyDuhzueofShymjEU9JI1iJX6TO0cHFO6PdZ1CGBA9dGR3t9rsiWpC vhNRmgf44UN142G/dhcJ9vPTM9iGZ9eeZF9t/Xacy3TQJ0b9wS8B8DQPEMvA+G1JI22m VSmYcvpGSKB8URzYmZQR36u8Qvc1BRCReGL8JH9v5mjvez/Dr3MRPmfO9q0QTx8XVeiy qEUvS7sURDM33u6nyXWGssEop6241Jyluaefp1Eyy/TMySDwEQSp0OW77K2m4K7ubxKI DXmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696995851; x=1697600651; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3Qou2Xi0iRC4/xhndQ2mmgbtUDHApacfhk0mMbTXCXY=; b=BAymrvY4bbTaQyDgihQl6MRHV3d6EVNxgNtsZTxXc3a0+InYZaZQ2qRbpLcuVdhHCe 20foImxJisFo/kbRJQvbD8iAfWmzIDMYeUXfryVjEc3AEH+suJcNkS0c1vxiGdjcwIV0 kBYgJLchAvO+Zg99JOmR/gDjZbxPFnVEiCYJVSFvwvRwNHBkrLTnjpcr1fyzF2G38PgS CotIXD620a5VZiJ3vfZO6UzKyOwwWujlb28jYMJCAKexhZrrh9tDRdStnxyr0OLgYPNf WW6CrOGawAE/ivvM44kk3I7KWkHgQh2QRBYBIuDclSPs2by3u1HxiMzOF5rcq8He2R4y Vlag== X-Gm-Message-State: AOJu0YxWedUvF1DGuvdBn+DIXSTOvkPsYObdkylsyLk1o5Qz4G4U8aEX VSU5/nG7RbkxKUaWPc1pr0le8AQQK+zySQ== X-Google-Smtp-Source: AGHT+IETQeg4LGXPttUw1pjj6jgsrPmkurw7I9GSftV80hwfy8cWX8vel1PI3XCZuc9FJJcQoJ6xHw== X-Received: by 2002:a05:6a20:6a28:b0:15e:1351:f33a with SMTP id p40-20020a056a206a2800b0015e1351f33amr20765174pzk.47.1696995850866; Tue, 10 Oct 2023 20:44:10 -0700 (PDT) Received: from wheely.local0.net ([1.128.220.51]) by smtp.gmail.com with ESMTPSA id q30-20020a638c5e000000b0058a9621f583sm7873656pgn.44.2023.10.10.20.44.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Oct 2023 20:44:10 -0700 (PDT) From: Nicholas Piggin To: netdev@vger.kernel.org Cc: Nicholas Piggin , dev@openvswitch.org, Pravin B Shelar , Aaron Conole , "Eelco Chaudron" , "Ilya Maximets" , "Flavio Leitner" , Ilya Maximets Subject: [PATCH 3/7] openvswitch: reduce stack usage in do_execute_actions Date: Wed, 11 Oct 2023 13:43:40 +1000 Message-ID: <20231011034344.104398-4-npiggin@gmail.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231011034344.104398-1-npiggin@gmail.com> References: <20231011034344.104398-1-npiggin@gmail.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: kuba@kernel.org From: Ilya Maximets do_execute_actions() function can be called recursively multiple times while executing actions that require pipeline forking or recirculations. It may also be re-entered multiple times if the packet leaves openvswitch module and re-enters it through a different port. Currently, there is a 256-byte array allocated on stack in this function that is supposed to hold NSH header. Compilers tend to pre-allocate that space right at the beginning of the function: a88: 48 81 ec b0 01 00 00 sub $0x1b0,%rsp NSH is not a very common protocol, but the space is allocated on every recursive call or re-entry multiplying the wasted stack space. Move the stack allocation to push_nsh() function that is only used if NSH actions are actually present. push_nsh() is also a simple function without a possibility for re-entry, so the stack is returned right away. With this change the preallocated space is reduced by 256 B per call: b18: 48 81 ec b0 00 00 00 sub $0xb0,%rsp Signed-off-by: Ilya Maximets --- net/openvswitch/actions.c | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-) diff --git a/net/openvswitch/actions.c b/net/openvswitch/actions.c index 7a66574672d3..be15ef693284 100644 --- a/net/openvswitch/actions.c +++ b/net/openvswitch/actions.c @@ -349,11 +349,18 @@ static int push_eth(struct sk_buff *skb, struct sw_flow_key *key, return 0; } -static int push_nsh(struct sk_buff *skb, struct sw_flow_key *key, - const struct nshhdr *nh) +static noinline_for_stack int push_nsh(struct sk_buff *skb, + struct sw_flow_key *key, + const struct nlattr *a) { + u8 buffer[NSH_HDR_MAX_LEN]; + struct nshhdr *nh = (struct nshhdr *)buffer; int err; + err = nsh_hdr_from_nlattr(a, nh, NSH_HDR_MAX_LEN); + if (err) + return err; + err = nsh_push(skb, nh); if (err) return err; @@ -1477,17 +1484,9 @@ static int do_execute_actions(struct datapath *dp, struct sk_buff *skb, err = pop_eth(skb, key); break; - case OVS_ACTION_ATTR_PUSH_NSH: { - u8 buffer[NSH_HDR_MAX_LEN]; - struct nshhdr *nh = (struct nshhdr *)buffer; - - err = nsh_hdr_from_nlattr(nla_data(a), nh, - NSH_HDR_MAX_LEN); - if (unlikely(err)) - break; - err = push_nsh(skb, key, nh); + case OVS_ACTION_ATTR_PUSH_NSH: + err = push_nsh(skb, key, nla_data(a)); break; - } case OVS_ACTION_ATTR_POP_NSH: err = pop_nsh(skb, key); From patchwork Wed Oct 11 03:43:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 13416554 X-Patchwork-Delegate: kuba@kernel.org Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 23744354EB for ; Wed, 11 Oct 2023 03:44:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="HVcXL7kS" Received: from mail-yw1-x112e.google.com (mail-yw1-x112e.google.com [IPv6:2607:f8b0:4864:20::112e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1EA7692 for ; Tue, 10 Oct 2023 20:44:19 -0700 (PDT) Received: by mail-yw1-x112e.google.com with SMTP id 00721157ae682-5a7c011e113so17542297b3.1 for ; Tue, 10 Oct 2023 20:44:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696995858; x=1697600658; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QC6KbeW6ZXTkOADgcyHrxsp1uDrAyBKZbdYvTZmeLb4=; b=HVcXL7kSXkIP/fQXXkbvkO70Bqbp3BOWOYu889GVmMtiOYeaS7UbsNjBQKm/ebhGiS Lni2Nockd18eywhEHPkgUYq1B5vZ/DqFeNAFfE55qh5u5D/21/dQu2Wb4c3sX0wjcxmx 63cXvFSFPWXWSPKaZ97q9ZRQn2yRvigfOImCm8gLTrbdss88Pxkngu9F53Z/frwo59tG 2vlYx06Dym+5HDWTNvIpoz2egHlErOmWEH/LmK91eiNt/H1zN5faKrZDfj3Y3q2tDtin bln6TjnXLAOyuc7lB0EYFQjhLIkpquvaLj4idAZ4p1bY0hmhuSP2ON3NpOzya/e9aQHZ Arxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696995858; x=1697600658; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QC6KbeW6ZXTkOADgcyHrxsp1uDrAyBKZbdYvTZmeLb4=; b=HhXjNkWDZzWQhxBvxG1cAoNA16+NXUwZsuO2Ttt2Ci6zoKPHrMsEBJkn67f1tw8u8n yaM7qnheEUw3v7Fm0WAWCdb7z0jp56igqHpJzufLYKjcZyC+3JQKjUh9+BbAMjq0lV+h ALhe2rqnMBUILUlF5RnH9bFh0GEpGGs4nWRhcwEuo8ROC8rk0q0Wv3Eom2+WMXSULZTa J/SYbe9ylFPQ8xz9tWgwt0M/ywMTvuAsLQxUp0s6En56QztyfO18V/0Ued4cejVGj0oA YDDdhl83Y/Ak+090UQJ2hQ9vg3sqb2rp6uFRoANvthuEDzb9uUkxFMl9AQPBf1G9oxNU Kp4g== X-Gm-Message-State: AOJu0Yw9ENYXRhn9fBkBN3U5u7wuA0HauoZNkG9xogkaU6oX61jio6tf VeuJmDTRD8UW8W7I6WTfu79OxoDgyr+OgQ== X-Google-Smtp-Source: AGHT+IGR7WzD4+qRbBoPht38+b30c7JemrjbQnAvMIUiH66akqSOfW2CxftoM6OS23X1ivAgsXTMxQ== X-Received: by 2002:a05:6902:1204:b0:d81:c512:e542 with SMTP id s4-20020a056902120400b00d81c512e542mr21283016ybu.31.1696995858061; Tue, 10 Oct 2023 20:44:18 -0700 (PDT) Received: from wheely.local0.net ([1.128.220.51]) by smtp.gmail.com with ESMTPSA id q30-20020a638c5e000000b0058a9621f583sm7873656pgn.44.2023.10.10.20.44.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Oct 2023 20:44:17 -0700 (PDT) From: Nicholas Piggin To: netdev@vger.kernel.org Cc: Nicholas Piggin , dev@openvswitch.org, Pravin B Shelar , Aaron Conole , "Eelco Chaudron" , "Ilya Maximets" , "Flavio Leitner" Subject: [PATCH 4/7] net: openvswitch: Reduce push_nsh stack usage Date: Wed, 11 Oct 2023 13:43:41 +1000 Message-ID: <20231011034344.104398-5-npiggin@gmail.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231011034344.104398-1-npiggin@gmail.com> References: <20231011034344.104398-1-npiggin@gmail.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: kuba@kernel.org Use percpu data to move the large temporary buffer off the push_nsh stack. This reduces stack consumption from 336 bytes to 64 bytes. Signed-off-by: Nicholas Piggin --- net/openvswitch/actions.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/net/openvswitch/actions.c b/net/openvswitch/actions.c index be15ef693284..fa53e22f3ebe 100644 --- a/net/openvswitch/actions.c +++ b/net/openvswitch/actions.c @@ -349,12 +349,18 @@ static int push_eth(struct sk_buff *skb, struct sw_flow_key *key, return 0; } +struct tmp_nsh_hdr { + u8 data[NSH_HDR_MAX_LEN]; +}; + +static DEFINE_PER_CPU(struct tmp_nsh_hdr, tmp_nsh_hdr); + static noinline_for_stack int push_nsh(struct sk_buff *skb, struct sw_flow_key *key, const struct nlattr *a) { - u8 buffer[NSH_HDR_MAX_LEN]; - struct nshhdr *nh = (struct nshhdr *)buffer; + struct tmp_nsh_hdr *hdr = this_cpu_ptr(&tmp_nsh_hdr); + struct nshhdr *nh = (struct nshhdr *)&hdr->data[0]; int err; err = nsh_hdr_from_nlattr(a, nh, NSH_HDR_MAX_LEN); From patchwork Wed Oct 11 03:43:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 13416555 X-Patchwork-Delegate: kuba@kernel.org Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0637DC8FD for ; Wed, 11 Oct 2023 03:44:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="AlG2Oufr" Received: from mail-oa1-x30.google.com (mail-oa1-x30.google.com [IPv6:2001:4860:4864:20::30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5FE2993 for ; Tue, 10 Oct 2023 20:44:24 -0700 (PDT) Received: by mail-oa1-x30.google.com with SMTP id 586e51a60fabf-1e10ba12fd3so3848262fac.1 for ; Tue, 10 Oct 2023 20:44:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696995863; x=1697600663; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=uN4ekHeyBAgLi1+j+3U4VuxqLSHSBtMr1w74fOLOgV4=; b=AlG2OufrLmbu+tXfD6g4rWQU5hEt8IsrF1pYqGtpjsmoNd5Lo3rdkjZWLjaOFoQLZz rXmH6k+Oo2+mbsPdJ73jfw/Tbnq7fZFE9AtzNaoAGzTRcWsP8VyfotBJjlmmt0EE3vCH FEUVoMV71ZOUO5TsUQIdp99bzImSRppyP14CttsYhXA9kdgnRfYKUIYHogepA4h47DWR OSHRxWZMPieEiVTo2ZOt4YR+hCPiXGHppOtNvHwO3nZhlLkOus+Z5iKdKDcorA9tfoOK vQAtpaDCuGEJZnyZjuZwdhP6xMrydfhrZ8cXJezxsubWDBz6yLFjNJq28cO3l1UxfWTQ QD0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696995863; x=1697600663; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uN4ekHeyBAgLi1+j+3U4VuxqLSHSBtMr1w74fOLOgV4=; b=nhOfDsC7GUFN2KAR0mjw8oaK5a9N9aDkTw475RDZFwxJaOlCczarQJLmlVoYrUgJtV 89IUczl4B4Ey0fvszdsVFMXs82elYYDUtf2yxguOyaqeXOzEr8DQ5SY8Hu2241fDoEzm UFELq4eT5JDmAIfFfiyU3TcDT2bdU5/0GRm9u/xMCrWcBOflKuXfuj+rpJllUvZNHJsu Iy3BFnTm5w3xYi7N1l+Rsxb8L1/lTlUlIYbHb61ODuFrSjZPzjm8Jba1Km5/TcJXRd6M +rUut4+lUcs8PfbDHvY9TyDoKs7hTP51Vn16QSTDfm56fFsPfodFyw1qz7cf2bcTvxRa c/KA== X-Gm-Message-State: AOJu0YxqieL0wb8bOe4DSN72qjw/AlL2oFJefaAWxdAHHwB6OWZ5h5Zw I6mPsczwGiNqGH7o9iwtM38Tx59HL+9BCw== X-Google-Smtp-Source: AGHT+IFWmQ1O/yx42gCkftT/nMYXAeFxfbh9K8vwJtnwIPUr1LoWlwfp8fY8lPR/robKbE1Tr/zPOA== X-Received: by 2002:a05:6871:439a:b0:1e9:8d3f:a158 with SMTP id lv26-20020a056871439a00b001e98d3fa158mr997075oab.56.1696995863445; Tue, 10 Oct 2023 20:44:23 -0700 (PDT) Received: from wheely.local0.net ([1.128.220.51]) by smtp.gmail.com with ESMTPSA id q30-20020a638c5e000000b0058a9621f583sm7873656pgn.44.2023.10.10.20.44.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Oct 2023 20:44:23 -0700 (PDT) From: Nicholas Piggin To: netdev@vger.kernel.org Cc: Nicholas Piggin , dev@openvswitch.org, Pravin B Shelar , Aaron Conole , "Eelco Chaudron" , "Ilya Maximets" , "Flavio Leitner" Subject: [PATCH 5/7] net: openvswitch: uninline action execution Date: Wed, 11 Oct 2023 13:43:42 +1000 Message-ID: <20231011034344.104398-6-npiggin@gmail.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231011034344.104398-1-npiggin@gmail.com> References: <20231011034344.104398-1-npiggin@gmail.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: kuba@kernel.org A function tends to use as much stack as the maximum of any control flow path. Compilers can "shrink wrap" to special-case stack allocation, but that only works well for exclusive paths. The switch statement in the loop in do_execute_actions uses as much stack as the maximum of its cases, and so inlining large actions increases overall stack uage. This is particularly bad because the actions that cause recursion are not the largest stack users. Uninline action execution functions, which reduces the stack usage of do_execute_actions from 288 bytes to 112 bytes. Signed-off-by: Nicholas Piggin --- net/openvswitch/actions.c | 69 +++++++++++++++++++++------------------ 1 file changed, 38 insertions(+), 31 deletions(-) diff --git a/net/openvswitch/actions.c b/net/openvswitch/actions.c index fa53e22f3ebe..87ec668d5556 100644 --- a/net/openvswitch/actions.c +++ b/net/openvswitch/actions.c @@ -964,8 +964,9 @@ static void ovs_fragment(struct net *net, struct vport *vport, ovs_kfree_skb_reason(skb, reason); } -static void do_output(struct datapath *dp, struct sk_buff *skb, int out_port, - struct sw_flow_key *key) +static noinline_for_stack void do_output(struct datapath *dp, + struct sk_buff *skb, int out_port, + struct sw_flow_key *key) { struct vport *vport = ovs_vport_rcu(dp, out_port); @@ -995,10 +996,11 @@ static void do_output(struct datapath *dp, struct sk_buff *skb, int out_port, } } -static int output_userspace(struct datapath *dp, struct sk_buff *skb, - struct sw_flow_key *key, const struct nlattr *attr, - const struct nlattr *actions, int actions_len, - uint32_t cutlen) +static noinline_for_stack +int output_userspace(struct datapath *dp, struct sk_buff *skb, + struct sw_flow_key *key, const struct nlattr *attr, + const struct nlattr *actions, int actions_len, + uint32_t cutlen) { struct dp_upcall_info upcall; const struct nlattr *a; @@ -1073,9 +1075,9 @@ static int dec_ttl_exception_handler(struct datapath *dp, struct sk_buff *skb, * Otherwise, sample() should keep 'skb' intact regardless what * actions are executed within sample(). */ -static int sample(struct datapath *dp, struct sk_buff *skb, - struct sw_flow_key *key, const struct nlattr *attr, - bool last) +static noinline_for_stack int sample(struct datapath *dp, struct sk_buff *skb, + struct sw_flow_key *key, + const struct nlattr *attr, bool last) { struct nlattr *actions; struct nlattr *sample_arg; @@ -1104,9 +1106,10 @@ static int sample(struct datapath *dp, struct sk_buff *skb, * Otherwise, clone() should keep 'skb' intact regardless what * actions are executed within clone(). */ -static int clone(struct datapath *dp, struct sk_buff *skb, - struct sw_flow_key *key, const struct nlattr *attr, - bool last) +static noinline_for_stack int clone(struct datapath *dp, + struct sk_buff *skb, + struct sw_flow_key *key, + const struct nlattr *attr, bool last) { struct nlattr *actions; struct nlattr *clone_arg; @@ -1122,8 +1125,9 @@ static int clone(struct datapath *dp, struct sk_buff *skb, !dont_clone_flow_key); } -static void execute_hash(struct sk_buff *skb, struct sw_flow_key *key, - const struct nlattr *attr) +static noinline_for_stack void execute_hash(struct sk_buff *skb, + struct sw_flow_key *key, + const struct nlattr *attr) { struct ovs_action_hash *hash_act = nla_data(attr); u32 hash = 0; @@ -1145,9 +1149,9 @@ static void execute_hash(struct sk_buff *skb, struct sw_flow_key *key, key->ovs_flow_hash = hash; } -static int execute_set_action(struct sk_buff *skb, - struct sw_flow_key *flow_key, - const struct nlattr *a) +static noinline_for_stack int execute_set_action(struct sk_buff *skb, + struct sw_flow_key *flow_key, + const struct nlattr *a) { /* Only tunnel set execution is supported without a mask. */ if (nla_type(a) == OVS_KEY_ATTR_TUNNEL_INFO) { @@ -1165,9 +1169,9 @@ static int execute_set_action(struct sk_buff *skb, /* Mask is at the midpoint of the data. */ #define get_mask(a, type) ((const type)nla_data(a) + 1) -static int execute_masked_set_action(struct sk_buff *skb, - struct sw_flow_key *flow_key, - const struct nlattr *a) +static noinline_for_stack +int execute_masked_set_action(struct sk_buff *skb, struct sw_flow_key *flow_key, + const struct nlattr *a) { int err = 0; @@ -1240,9 +1244,9 @@ static int execute_masked_set_action(struct sk_buff *skb, return err; } -static int execute_recirc(struct datapath *dp, struct sk_buff *skb, - struct sw_flow_key *key, - const struct nlattr *a, bool last) +static noinline_for_stack +int execute_recirc(struct datapath *dp, struct sk_buff *skb, + struct sw_flow_key *key, const struct nlattr *a, bool last) { u32 recirc_id; @@ -1259,9 +1263,10 @@ static int execute_recirc(struct datapath *dp, struct sk_buff *skb, return clone_execute(dp, skb, key, recirc_id, NULL, 0, last, true); } -static int execute_check_pkt_len(struct datapath *dp, struct sk_buff *skb, - struct sw_flow_key *key, - const struct nlattr *attr, bool last) +static noinline_for_stack +int execute_check_pkt_len(struct datapath *dp, struct sk_buff *skb, + struct sw_flow_key *key, const struct nlattr *attr, + bool last) { struct ovs_skb_cb *ovs_cb = OVS_CB(skb); const struct nlattr *actions, *cpl_arg; @@ -1298,7 +1303,8 @@ static int execute_check_pkt_len(struct datapath *dp, struct sk_buff *skb, nla_len(actions), last, clone_flow_key); } -static int execute_dec_ttl(struct sk_buff *skb, struct sw_flow_key *key) +static noinline_for_stack int execute_dec_ttl(struct sk_buff *skb, + struct sw_flow_key *key) { int err; @@ -1558,10 +1564,11 @@ static int do_execute_actions(struct datapath *dp, struct sk_buff *skb, * The execution may be deferred in case the actions can not be executed * immediately. */ -static int clone_execute(struct datapath *dp, struct sk_buff *skb, - struct sw_flow_key *key, u32 recirc_id, - const struct nlattr *actions, int len, - bool last, bool clone_flow_key) +static noinline_for_stack +int clone_execute(struct datapath *dp, struct sk_buff *skb, + struct sw_flow_key *key, u32 recirc_id, + const struct nlattr *actions, int len, bool last, + bool clone_flow_key) { struct deferred_action *da; struct sw_flow_key *clone; From patchwork Wed Oct 11 03:43:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 13416556 X-Patchwork-Delegate: kuba@kernel.org Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 584B5D2E3 for ; Wed, 11 Oct 2023 03:44:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="axGWzn8h" Received: from mail-oi1-x236.google.com (mail-oi1-x236.google.com [IPv6:2607:f8b0:4864:20::236]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3E6D394 for ; Tue, 10 Oct 2023 20:44:29 -0700 (PDT) Received: by mail-oi1-x236.google.com with SMTP id 5614622812f47-3ae2896974bso4385541b6e.0 for ; Tue, 10 Oct 2023 20:44:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696995868; x=1697600668; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BG0lTECjijmEFm9e9LYjeMTQQ21OTNTMYQZ13fnzpuM=; b=axGWzn8hiUh+VppaJb9gJp8LfGPRDN9YmF4pNUE8l9zQO2RbauZLxlP9V9DUjH1FZM tiapdNpFstjlAGu/zlrzIZJ/zilkQMzaNzyOOGUMWaj/nSvib+PmHfVIpW8t+2HFV5N7 bdYtar/9RtShmQ4Gn/c54hgRPInun9QhyIMjBrpa+1kTxmqEUeCU92kNuh4+xV5N4HI+ FourD7bjQbwB91vPnE2lY6SDJqB8A7UN0VtiyOH7Yj7Sn4TDm+0S0HtT5mBUc3U+XdF8 qFTMHXwkhJkTIgtKj6+t4nFtjMy6PgSA8EiSdGAtvBd8O+on6koj8jZYYJkyLfsIHbad VlXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696995868; x=1697600668; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BG0lTECjijmEFm9e9LYjeMTQQ21OTNTMYQZ13fnzpuM=; b=DNNfSEi4lU12NVv4EzicRHiAwlH+v9JC0z0tsY8I46qZ1awYp2ANbg/YZdHD0lW+iV uhr315rGge2r94zRnXiaSJ3Vx+p6ubM7ThYMEYpBSRbMOevZp493FKnDi1Y7bSFDjqTQ JBq4ODgdEqn46XrpbGXqcy4mSxd0YC7AqUUqNEPqPzu72+2iUi5OkeWco9OtMkV0Ogh8 7eXsJYfbdROgePnRlIMfcgamS3amoCSeu3fucVifjQarbczI5tkpcdAEjKurTIh9cCBb yL82ZDttXSo0XLd1apyreaoQWrNQcOSmwqQIbenTyciydrrdrDiQteeI+Y1ANCKepYe+ Ektw== X-Gm-Message-State: AOJu0Yxzxne164/cuELMbvyTvLbhWlFw7xc3sT2mqXsa/0+fLwVBiTP6 FEweKSCWZAfRxQVUHsq2TBhU0RRPcpBELg== X-Google-Smtp-Source: AGHT+IFB+t2SmAwSL29E78ppnJwERh8c1+2v+39QdJb3/4uEQO7vQ9u/QRkvikRsqeYPhd8K3UL+sA== X-Received: by 2002:a05:6808:152a:b0:3a8:6a03:c0c with SMTP id u42-20020a056808152a00b003a86a030c0cmr27307955oiw.27.1696995868320; Tue, 10 Oct 2023 20:44:28 -0700 (PDT) Received: from wheely.local0.net ([1.128.220.51]) by smtp.gmail.com with ESMTPSA id q30-20020a638c5e000000b0058a9621f583sm7873656pgn.44.2023.10.10.20.44.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Oct 2023 20:44:28 -0700 (PDT) From: Nicholas Piggin To: netdev@vger.kernel.org Cc: Nicholas Piggin , dev@openvswitch.org, Pravin B Shelar , Aaron Conole , "Eelco Chaudron" , "Ilya Maximets" , "Flavio Leitner" Subject: [PATCH 6/7] net: openvswitch: uninline ovs_fragment to control stack usage Date: Wed, 11 Oct 2023 13:43:43 +1000 Message-ID: <20231011034344.104398-7-npiggin@gmail.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231011034344.104398-1-npiggin@gmail.com> References: <20231011034344.104398-1-npiggin@gmail.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: kuba@kernel.org ovs_fragment uses a lot of stack, 400 bytes. It is a leaf function but its caller do_output is involved in openvswitch recursion. GCC 13.2 for powerpc64le is not inlining it, but it only has a single call site, so it is liable to being inlined. Mark it noinline_for_stack, to ensure it doesn't bloat stack use in the recursive path. Signed-off-by: Nicholas Piggin --- net/openvswitch/actions.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/net/openvswitch/actions.c b/net/openvswitch/actions.c index 87ec668d5556..ef3a59012d26 100644 --- a/net/openvswitch/actions.c +++ b/net/openvswitch/actions.c @@ -900,9 +900,9 @@ static void prepare_frag(struct vport *vport, struct sk_buff *skb, skb_pull(skb, hlen); } -static void ovs_fragment(struct net *net, struct vport *vport, - struct sk_buff *skb, u16 mru, - struct sw_flow_key *key) +static noinline_for_stack +void ovs_fragment(struct net *net, struct vport *vport, struct sk_buff *skb, + u16 mru, struct sw_flow_key *key) { enum ovs_drop_reason reason; u16 orig_network_offset = 0; From patchwork Wed Oct 11 03:43:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 13416557 X-Patchwork-Delegate: kuba@kernel.org Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D79E2D2E2 for ; Wed, 11 Oct 2023 03:44:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="CCp/9yGs" Received: from mail-oi1-x235.google.com (mail-oi1-x235.google.com [IPv6:2607:f8b0:4864:20::235]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 22EBF92 for ; Tue, 10 Oct 2023 20:44:34 -0700 (PDT) Received: by mail-oi1-x235.google.com with SMTP id 5614622812f47-3af608eb34bso4376314b6e.1 for ; Tue, 10 Oct 2023 20:44:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696995873; x=1697600673; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=icbcgm9xSygTZ84fa5AdWx/TaKno2CPM52PibGcRS+M=; b=CCp/9yGsdONTVBL+uzChuAmaWwuTbJHsNQn6bbHOoWuzy8BdmKGva11/wA2DJ6CPET q0pLUAWFW9skrMzJlYKnh7WbtVWKyJOgSALvagihAaRun/28eUa7SFhN/y0wPzgPH7k2 YLI8aoeA19t6a4OgYGfIZ56vL0fgYhKxHX6hVSl+WIZpen+j0XxSqde0B5AC7B5zOYk9 chBfLWXbGAw1lLUXAFT2jrwLPA08sE5UCPwJJDsuPpB2nNCp16AOE/HVH1I4YOEPdVba txarfkCjYn8gERsXMqZR6X7KuEoS73moB+bz/ERmsrLLQpds6V0qcpXbWsO9dJqvtKKP 4T7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696995873; x=1697600673; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=icbcgm9xSygTZ84fa5AdWx/TaKno2CPM52PibGcRS+M=; b=c2+78R+7NEuROYRySegArggAw0nRE/UhM8i7+RvZa5FvgcNkdVtpx9OogQ7FQts5ib PSZYG6z4vtibzOOdLpcRuA6T7ygewRE5UvFjtz58QxtdKIo757V8uHftgt3rLNlHqH8B CjH811xd3EzQ5FT4+z8s1Y0kEI5ON0zTn76hzXEYPhoPxqLpnWaIeq4/b4UHdF2DkHvg UkjcidOH1pkWcWWEEWffvbIPiVPLII1TySvNpAAoqsHDQxEwhJBM5jlr+PF4k1yNXmkU ZkRM0pEFgNISx2fD26oeSjXgyc3y0QHf7pmYtVAKfh+cU7X2TnH5p3cjenn2Ae6OiQSy W8zA== X-Gm-Message-State: AOJu0YwXPcIZLIB7cINU3X6M05eIvfxJeiTsst7R3GmtVPW/0xnZJiz8 WDombB0M6gWK3gAj3m4SusJG3pCPoC6SVw== X-Google-Smtp-Source: AGHT+IG2DT+GotU2rg/r7ctBPCz3Th6AsDi1xYW5tdM63vjMkGpdFmMl0iu8DNjNiZBKSjfK7EVXKQ== X-Received: by 2002:a05:6808:19a6:b0:3a4:4b42:612b with SMTP id bj38-20020a05680819a600b003a44b42612bmr27095344oib.42.1696995873209; Tue, 10 Oct 2023 20:44:33 -0700 (PDT) Received: from wheely.local0.net ([1.128.220.51]) by smtp.gmail.com with ESMTPSA id q30-20020a638c5e000000b0058a9621f583sm7873656pgn.44.2023.10.10.20.44.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Oct 2023 20:44:32 -0700 (PDT) From: Nicholas Piggin To: netdev@vger.kernel.org Cc: Nicholas Piggin , dev@openvswitch.org, Pravin B Shelar , Aaron Conole , "Eelco Chaudron" , "Ilya Maximets" , "Flavio Leitner" Subject: [PATCH 7/7] net: openvswitch: Reduce stack usage in ovs_dp_process_packet Date: Wed, 11 Oct 2023 13:43:44 +1000 Message-ID: <20231011034344.104398-8-npiggin@gmail.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231011034344.104398-1-npiggin@gmail.com> References: <20231011034344.104398-1-npiggin@gmail.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: kuba@kernel.org The upcall in ovs_dp_process_packet some stack and is not involved in the recursive call. Move it out of line, reducing stack overhead of ovs_dp_process_packet from 144 to 96 bytes. Signed-off-by: Nicholas Piggin --- net/openvswitch/datapath.c | 56 ++++++++++++++++++++++---------------- 1 file changed, 32 insertions(+), 24 deletions(-) diff --git a/net/openvswitch/datapath.c b/net/openvswitch/datapath.c index 11c69415c605..fdc24b1e9bbc 100644 --- a/net/openvswitch/datapath.c +++ b/net/openvswitch/datapath.c @@ -242,6 +242,37 @@ void ovs_dp_detach_port(struct vport *p) ovs_vport_del(p); } +static noinline_for_stack +void do_packet_upcall(struct sk_buff *skb, struct sw_flow_key *key, + const struct vport *p, struct datapath *dp) +{ + struct dp_upcall_info upcall; + int error; + + memset(&upcall, 0, sizeof(upcall)); + upcall.cmd = OVS_PACKET_CMD_MISS; + + if (dp->user_features & OVS_DP_F_DISPATCH_UPCALL_PER_CPU) + upcall.portid = + ovs_dp_get_upcall_portid(dp, smp_processor_id()); + else + upcall.portid = ovs_vport_find_upcall_portid(p, skb); + + upcall.mru = OVS_CB(skb)->mru; + error = ovs_dp_upcall(dp, skb, key, &upcall, 0); + switch (error) { + case 0: + case -EAGAIN: + case -ERESTARTSYS: + case -EINTR: + consume_skb(skb); + break; + default: + kfree_skb(skb); + break; + } +} + /* Must be called with rcu_read_lock. */ void ovs_dp_process_packet(struct sk_buff *skb, struct sw_flow_key *key) { @@ -261,30 +292,7 @@ void ovs_dp_process_packet(struct sk_buff *skb, struct sw_flow_key *key) flow = ovs_flow_tbl_lookup_stats(&dp->table, key, skb_get_hash(skb), &n_mask_hit, &n_cache_hit); if (unlikely(!flow)) { - struct dp_upcall_info upcall; - - memset(&upcall, 0, sizeof(upcall)); - upcall.cmd = OVS_PACKET_CMD_MISS; - - if (dp->user_features & OVS_DP_F_DISPATCH_UPCALL_PER_CPU) - upcall.portid = - ovs_dp_get_upcall_portid(dp, smp_processor_id()); - else - upcall.portid = ovs_vport_find_upcall_portid(p, skb); - - upcall.mru = OVS_CB(skb)->mru; - error = ovs_dp_upcall(dp, skb, key, &upcall, 0); - switch (error) { - case 0: - case -EAGAIN: - case -ERESTARTSYS: - case -EINTR: - consume_skb(skb); - break; - default: - kfree_skb(skb); - break; - } + do_packet_upcall(skb, key, p, dp); stats_counter = &stats->n_missed; goto out; }