From patchwork Thu Jun 27 12:59:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13714336 Received: from mail-ej1-f54.google.com (mail-ej1-f54.google.com [209.85.218.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E2E7517839F; Thu, 27 Jun 2024 12:59:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719493186; cv=none; b=C73qFxWC1Puw8G4/zO8NvJKHrsTIcRptyzov5lihIJkmkDJZF77YA48nFsxTiBERP41v00GwRxtFuMBMFfLh6c4j83J8BDG6SyuJuiISHh/vSHfzwBxVdO/EWzVxFvsXsyrjrkjhwSr8bb0Qi4d2ZWTbrhpCyZ6OJ03DLI/xM9o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719493186; c=relaxed/simple; bh=7ZoYhK/CgONEjG9pC9F7RougkB4yL0GdwFlB4sNvNOc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dwMLEMzk3E9GgZNnrfMnXFts5YXKCaX510dHXb/seJwP12Y53HqQjh8xTVaArE0QBW7SLoFbGPbjPjB0ktM0qpLCmWIeXjRmEO/o2vT3t077pmDfGEzUUd5jXvKTZLkB9Ar8SKtID/yo4WmsreGSJ8ZmTtNRH8JkpiQqeMOyDpU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=LOGo4amE; arc=none smtp.client-ip=209.85.218.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="LOGo4amE" Received: by mail-ej1-f54.google.com with SMTP id a640c23a62f3a-a70c0349b64so560015466b.1; Thu, 27 Jun 2024 05:59:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1719493183; x=1720097983; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=NjjoRZWG8PWTBUhXIjiqjxdChiGsNUGZ3NtShy0+Zv0=; b=LOGo4amEx/e+C/hOGvnBou48TUffSijRn35voo+W1DKS47nL9YezS5frUXTgPAqx1O mADbmE6XUg/1K7+ir0DTz2GHTizRN1Mugfu1pXSUYVeKzhCTvASYKdEj4F7pASeEg2Yk wsRqDJcS+NDTRTl+HusS4I3tIyP+16aM6u350j5ahIPZecbPfdwKsTRN2r/g6ZTTINev kgA0H/Sj4Dth447ui6msdgFMX78WX0lpj7z9XzE5cQdPQ/UfuCg4K4ys1N/tYbK3goJo XzDKoOfgALykBwC3OUIgTcrNLcgjggvT56lXLeLB2Sh6tQBvsT+zoKQLj8BNCtMp4yb/ IlIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719493183; x=1720097983; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NjjoRZWG8PWTBUhXIjiqjxdChiGsNUGZ3NtShy0+Zv0=; b=EqkXMsX5ow1v/nrXXkMXH51apa05w7FEWGG/oJGyBFoQwaQg8HEAMwykyXQ5SKXsBY 3pnchJ0eLkjMB4byl+IIaUBJiyeLCsgDEv1n0cP/W/NYQDypt2cpoRXovK7tmQV+zLJS czS5/83rYj6Lyap+SiEQFIdvij54JazrfwnmjHsMisyWI8zUKgUDs7HmmJFhYIXETvsx Y0RFXHjv3Y+jF4HNLP+EqBNpGEKHmea10jds6FKqQ5YNLTlmzjce8qVJLnBrqR2/KCmt GGSDGxBwDdAbWjmK+/f8hjENlczMR6PCg7BzA6gkyN06e2mN6ZUv1U5bzrEXL5FX8fOl Jt5A== X-Forwarded-Encrypted: i=1; AJvYcCUqWGaB9r9VKwV9/RIukpgklyQ5FLjVP17PaQ9bPA0umAWwRhUf7GSWgQR/AS9Pr/Ny/Uwk4O/Ml7t0TgQQrhAkvEOjyB2u X-Gm-Message-State: AOJu0Yz1ELm/QrwPntz037MQHZ4s/Ls40RYgPyMlIyrh0C+CciA4rK43 TLusYOWPOeBK1eRf40tPKr5UsRLsOIRGe069tizOoRZvKqvWHb2Kss1yqRPH X-Google-Smtp-Source: AGHT+IFEi5ivycykCJ1gWjuYw1W1Rj6SojBpuwYqBU2kN2eq/C5/kdirvvuWLtWKRkZFHGfWDINhkw== X-Received: by 2002:a17:906:318e:b0:a6f:5adc:6533 with SMTP id a640c23a62f3a-a715f9795femr928790266b.46.1719493182718; Thu, 27 Jun 2024 05:59:42 -0700 (PDT) Received: from 127.0.0.1localhost ([163.114.131.193]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a729d7c95a3sm57267766b.194.2024.06.27.05.59.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Jun 2024 05:59:42 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, "David S . Miller" , Jakub Kicinski , David Ahern , Eric Dumazet , Willem de Bruijn Subject: [PATCH net-next 1/5] net: always try to set ubuf in skb_zerocopy_iter_stream Date: Thu, 27 Jun 2024 13:59:41 +0100 Message-ID: <398a9b0f677348f62edf3572a4896ddb0ebfb940.1719190216.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 skb_zcopy_set() does nothing if there is already a ubuf_info associated with an skb, and since ->link_skb should have set it several lines above the check here essentially does nothing and can be removed. It's also safer this way, because even if the callback is faulty we'll have it set. Signed-off-by: Pavel Begunkov Reviewed-by: Willem de Bruijn --- net/core/skbuff.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 2315c088e91d..9f28822dde6f 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -1884,8 +1884,7 @@ int skb_zerocopy_iter_stream(struct sock *sk, struct sk_buff *skb, return err; } - if (!uarg->ops->link_skb) - skb_zcopy_set(skb, uarg, NULL); + skb_zcopy_set(skb, uarg, NULL); return skb->len - orig_len; } EXPORT_SYMBOL_GPL(skb_zerocopy_iter_stream); From patchwork Thu Jun 27 12:59:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13714337 Received: from mail-ej1-f46.google.com (mail-ej1-f46.google.com [209.85.218.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 795E8181CE0; Thu, 27 Jun 2024 12:59:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719493187; cv=none; b=ZjyU7tLGRnlrzUcZl9CHlZMGY10C+2V3UezVuy62QyWPaWj5FcVrFCybOy41VVNNu5SVruPodvCkIEN0CF7rymHh09Y2yw4vGxqeViuEFZy+NZWmFeFpOEa7u7QVJ3JZ26BwGABTmP/vFnsJCHRiD9vawwnDQHDbZfOrsBzWkL0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719493187; c=relaxed/simple; bh=7wf4t3zNVFL5MLPArmxzZz/difFq2GZjTGI/zq/4wFo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=pRNM2lmG2OfWKITjSmAT6esIzD0Prm8Nu5m1zWFEYXeYWV0ox1Vpl7Jm+phQrXnObyOa7BUZaRNqSYzSHtTej2Cs9OyuCZeb9ePEHuQF/ZJaZZf9qCNsSmAc9TmjwkZFKwtMoca/Y8ISTqcOJa/uhlX9JxdODr7QL4ipiyqatAk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=k6Td1TR2; arc=none smtp.client-ip=209.85.218.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="k6Td1TR2" Received: by mail-ej1-f46.google.com with SMTP id a640c23a62f3a-a6fd513f18bso799848166b.3; Thu, 27 Jun 2024 05:59:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1719493183; x=1720097983; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7hgg2HLxbQmSoQjOf3ORs+BE3/TW3TRuVxCZz2YM2kI=; b=k6Td1TR2QX63+vdS7q6Q2IEJYONyoLl8BXJE5AXduqiSg+VbWAyW4R30KGBxHdBsEq 7iav5yjWo2X2Srdnv2urzueMAH87TzoZDYqRZ/gSV0sbqevtpsSa7nYwVfmNUyugVFON QZcVKKj5DFhwPgF+m9BwYUdXp+9oltqXgXk7E42JAyaeXqV42pf8HvUeCQQO914rAhg8 TRYb9yZMx+9jWlI+B6e1fw1DOzw6uJF19G6mcijesxVNcqWM4BPPLK/dtUOepbH/7Nfx l1Y4yP7lz24pfsQYlUPvgAfS3ND07rYfK21mZsw0xKAoshqJ8bpSawCIze2yRgZe3xpp plZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719493183; x=1720097983; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7hgg2HLxbQmSoQjOf3ORs+BE3/TW3TRuVxCZz2YM2kI=; b=tQlpltvq3nhkaEUtPZEuKtSOFPGxZdahTXZqvplzB10W3X1VnfUm9VNZKrWU8EB1qy 3KxeLsp4Qud0TR2ahEDdS9PFJgptIg51mQgMq191T3mSOAXeavLpVnJH/eemebHH9JHO uQM4y7xYAZXQSW2aS97wbGT6jT6WBFDwWGxKOAB6agPtrXqH1Ut8RB+4xPbXH1CVnmzh c7ftmnBkZ53Hsr7VHtdCIt70RESZsiuwkx4qamyAHjUPr5RvUFUrp39wwhd1bgUhBun7 Ql9P7Wojjkaw6DQUhoL9Kv0l5Ya5X7CI7ged41jg2SCzQtn0vyHSWUkoQhiFRaqnKTa/ L88Q== X-Forwarded-Encrypted: i=1; AJvYcCU1h8i6tDqa06N8batGdSPaIVqADL66RIzZwDDpjSg8bFxM3gIzUrwzltbN5DqX6P4wbJX/ui4LdomguIm1keNr1R+LpPvD X-Gm-Message-State: AOJu0Ywi+UxlPu9R+IAqJ2P0zTGtxVH6mWVWw8s+Hh4iWcJsmvv8oEFR Zyeq1spgCuTPOe4Ysh5hcbZ6ARW95g15oBYUB5CEZi5wdRYmEIApoANftiIh X-Google-Smtp-Source: AGHT+IHvtGynC0PYko7Pz/guu5wq17ijkLER0KNUXZnI3f8bZvANVMd95hl74otdAJvA8s2YdeeFjg== X-Received: by 2002:a17:907:d047:b0:a72:744e:6703 with SMTP id a640c23a62f3a-a72744e6addmr601006266b.22.1719493183491; Thu, 27 Jun 2024 05:59:43 -0700 (PDT) Received: from 127.0.0.1localhost ([163.114.131.193]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a729d7c95a3sm57267766b.194.2024.06.27.05.59.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Jun 2024 05:59:43 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, "David S . Miller" , Jakub Kicinski , David Ahern , Eric Dumazet , Willem de Bruijn Subject: [PATCH net-next 2/5] net: split __zerocopy_sg_from_iter() Date: Thu, 27 Jun 2024 13:59:42 +0100 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Split a function out of __zerocopy_sg_from_iter() that only cares about the traditional path with refcounted pages and doesn't need to know about ->sg_from_iter. A preparation patch, we'll improve on the function later. Signed-off-by: Pavel Begunkov Reviewed-by: Willem de Bruijn --- net/core/datagram.c | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/net/core/datagram.c b/net/core/datagram.c index 95f242591fd2..7f7d5da2e406 100644 --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -610,16 +610,10 @@ int skb_copy_datagram_from_iter(struct sk_buff *skb, int offset, } EXPORT_SYMBOL(skb_copy_datagram_from_iter); -int __zerocopy_sg_from_iter(struct msghdr *msg, struct sock *sk, - struct sk_buff *skb, struct iov_iter *from, - size_t length) +static int zerocopy_fill_skb_from_iter(struct sock *sk, struct sk_buff *skb, + struct iov_iter *from, size_t length) { - int frag; - - if (msg && msg->msg_ubuf && msg->sg_from_iter) - return msg->sg_from_iter(sk, skb, from, length); - - frag = skb_shinfo(skb)->nr_frags; + int frag = skb_shinfo(skb)->nr_frags; while (length && iov_iter_count(from)) { struct page *head, *last_head = NULL; @@ -692,6 +686,16 @@ int __zerocopy_sg_from_iter(struct msghdr *msg, struct sock *sk, } return 0; } + +int __zerocopy_sg_from_iter(struct msghdr *msg, struct sock *sk, + struct sk_buff *skb, struct iov_iter *from, + size_t length) +{ + if (msg && msg->msg_ubuf && msg->sg_from_iter) + return msg->sg_from_iter(sk, skb, from, length); + else + return zerocopy_fill_skb_from_iter(sk, skb, from, length); +} EXPORT_SYMBOL(__zerocopy_sg_from_iter); /** From patchwork Thu Jun 27 12:59:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13714338 Received: from mail-ed1-f45.google.com (mail-ed1-f45.google.com [209.85.208.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 289681849D7; Thu, 27 Jun 2024 12:59:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719493188; cv=none; b=ajj/yVDU16/b6OOQgpoOgZZ3TxVpLnnCyuPx7VC0A99HESZc+sPFDAXDbFNQFfMm2GuLw++qwrnYJrISCWJVj4/YXam/16EBGfBP9piKCgoFlVArqE1W5QiBVScF7UlTvOFSdr8ouYNlogUI0w+j/jU317na9FBtYfpP6NjZe2o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719493188; c=relaxed/simple; bh=WZxjExbJY57zIqoo/+Q9dKxeNZQlbSRr32BEuf/2Xpw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YOOamlgjwlaafx/L4+w/z6LJGZ2CUaYCgV096ccdzKoxWNpanfypZ3uml/5Y1p4vhvQ5FRJTtzQkQO5Jl1O1tu5rHNoyRIIncl4ovR6W2+qsWvNyFqeOeM3n38UqpGvKixBb/F1ssIxHJzJPN/69X6gjUvyE13WzsEMRkIci0o8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=iGxz62ry; arc=none smtp.client-ip=209.85.208.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="iGxz62ry" Received: by mail-ed1-f45.google.com with SMTP id 4fb4d7f45d1cf-57d1012e52fso1613146a12.3; Thu, 27 Jun 2024 05:59:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1719493185; x=1720097985; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Fjc6mD01lmIiX8GcrjloiH+OKRZl4anANwhXBkOfgDs=; b=iGxz62ryF96GBKQtDKtG8t33dTCiEM0we3u51384ajByV22dVx2KgarETQNyyylvpV /qlhgW0x1L4nrj5tgjzAG2sfiIU9leFwfs7Ptz9m4DIAkCF+/icPNjYqBAPu3K44jGL6 wsHqj597MIfyKXQw175celdDG+sFGIenzlG/egIgk7aZx/5D4yrwxZfvxHLk/JxIcOib k9DFE2/3DchPt/9DsJ1OxKoTpmPnY4ZQu4rMPcJn31chRrdOL7aVw6eShyXIMzP6fLCU 5X87nPgkXd6NLR5w6xIesRpfJ0g7SFkz5jOscythgA/C3Bd2c1QMVbrkUDOx4eQN4Zxd 3SWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719493185; x=1720097985; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Fjc6mD01lmIiX8GcrjloiH+OKRZl4anANwhXBkOfgDs=; b=v2fZEd81Pil1UYCJBkpTau+/oIUFg9BdIw7WPpXRdVTQ4Z0n4rQKVHCsZV3ILOnoGH A9unVC4JyXP4jCSljSYQhWC8kk5Vff2nKrOLSG6rPQTtMsK3lXet0aarBYosRTqLYrXW ghwo9QChq+YrOvrdQgBB9ylbxle6q7ADK1EFmKTR1EysgqSK6ddszamziK8H6ZNOfIlX GRhUvjz9AAzhl0dO2gxjxF1TEVrAS1gr3zBjooyKjJl1BQu0jAvqZ8gsnNZLZ+cpTQVp 0lH3AkH8sI2vF0KBxE9+b2nMhV98CAl8QKCi4CJ1YncWG7/YlBJi0bGoEjMdq11hDX6l qYIw== X-Forwarded-Encrypted: i=1; AJvYcCUo25U33sAsJ7kf43s1Njib9rE8UAnGHvUjcKCaQ8Vx9q1Ah9buijlFcrnOJWTJ2/pWYGil4kB2xlxJ86m/DOaQGmkmARfp X-Gm-Message-State: AOJu0YyjuBG3Bo0y84Dh4bS37cxjSk8QIZ5XpFevIoIedOMdWpTPoS5y dgQUzXdY+mTffwSFYgshWnIpFAGOfTwn8WcXJzJQwP8LxTyeW1jw76kyDRwG X-Google-Smtp-Source: AGHT+IGbuT+N1shQBBUGGLsCg2q0mwzzoD0BaJYqv6R8boS/hyhHmCkRjehm64p2pcLMeitVmaTCUQ== X-Received: by 2002:a17:907:368a:b0:a72:98a0:2bfc with SMTP id a640c23a62f3a-a7298a02d4amr188678166b.17.1719493184686; Thu, 27 Jun 2024 05:59:44 -0700 (PDT) Received: from 127.0.0.1localhost ([163.114.131.193]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a729d7c95a3sm57267766b.194.2024.06.27.05.59.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Jun 2024 05:59:43 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, "David S . Miller" , Jakub Kicinski , David Ahern , Eric Dumazet , Willem de Bruijn Subject: [PATCH net-next 3/5] net: batch zerocopy_fill_skb_from_iter accounting Date: Thu, 27 Jun 2024 13:59:43 +0100 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Instead of accounting every page range against the socket separately, do it in batch based on the change in skb->truesize. It's also moved into __zerocopy_sg_from_iter(), so that zerocopy_fill_skb_from_iter() is simpler and responsible for setting frags but not the accounting. Signed-off-by: Pavel Begunkov Reviewed-by: Willem de Bruijn --- net/core/datagram.c | 31 ++++++++++++++++++------------- 1 file changed, 18 insertions(+), 13 deletions(-) diff --git a/net/core/datagram.c b/net/core/datagram.c index 7f7d5da2e406..2b24d69b1e94 100644 --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -610,7 +610,7 @@ int skb_copy_datagram_from_iter(struct sk_buff *skb, int offset, } EXPORT_SYMBOL(skb_copy_datagram_from_iter); -static int zerocopy_fill_skb_from_iter(struct sock *sk, struct sk_buff *skb, +static int zerocopy_fill_skb_from_iter(struct sk_buff *skb, struct iov_iter *from, size_t length) { int frag = skb_shinfo(skb)->nr_frags; @@ -621,7 +621,6 @@ static int zerocopy_fill_skb_from_iter(struct sock *sk, struct sk_buff *skb, int refs, order, n = 0; size_t start; ssize_t copied; - unsigned long truesize; if (frag == MAX_SKB_FRAGS) return -EMSGSIZE; @@ -633,17 +632,9 @@ static int zerocopy_fill_skb_from_iter(struct sock *sk, struct sk_buff *skb, length -= copied; - truesize = PAGE_ALIGN(copied + start); skb->data_len += copied; skb->len += copied; - skb->truesize += truesize; - if (sk && sk->sk_type == SOCK_STREAM) { - sk_wmem_queued_add(sk, truesize); - if (!skb_zcopy_pure(skb)) - sk_mem_charge(sk, truesize); - } else { - refcount_add(truesize, &skb->sk->sk_wmem_alloc); - } + skb->truesize += PAGE_ALIGN(copied + start); head = compound_head(pages[n]); order = compound_order(head); @@ -691,10 +682,24 @@ int __zerocopy_sg_from_iter(struct msghdr *msg, struct sock *sk, struct sk_buff *skb, struct iov_iter *from, size_t length) { + unsigned long orig_size = skb->truesize; + unsigned long truesize; + int ret; + if (msg && msg->msg_ubuf && msg->sg_from_iter) return msg->sg_from_iter(sk, skb, from, length); - else - return zerocopy_fill_skb_from_iter(sk, skb, from, length); + + ret = zerocopy_fill_skb_from_iter(skb, from, length); + truesize = skb->truesize - orig_size; + + if (sk && sk->sk_type == SOCK_STREAM) { + sk_wmem_queued_add(sk, truesize); + if (!skb_zcopy_pure(skb)) + sk_mem_charge(sk, truesize); + } else { + refcount_add(truesize, &skb->sk->sk_wmem_alloc); + } + return ret; } EXPORT_SYMBOL(__zerocopy_sg_from_iter); From patchwork Thu Jun 27 12:59:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13714339 Received: from mail-ej1-f52.google.com (mail-ej1-f52.google.com [209.85.218.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EE3C51849FC; Thu, 27 Jun 2024 12:59:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719493190; cv=none; b=V9EDHdj2uIUCdoXp3E8Y1jQQKUykjq5Z11dcBOCTx/vc+rBoTjdlZPDU1ZQigVRclK3CuwABPeEgREuqAL8d/uHoeysiyNNV10CeJbPC7w8/EjHMDAJw1SIlfRqOP7WdlKGlUH72PwIa0AE/TsPiRrvCAQbJnc1Y9+/vb3uq7JY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719493190; c=relaxed/simple; bh=LspUSUQisVXEzuvKb/HhQ7sazs/RezEAljb8kvYO+/E=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ficUMHi58oEtn2lQmJLmwvCwcyO+t1tl68Tl63TDrEbJqvTK+SGTLPqF06Bcb+GOQTMDfOxc6B3VBhMl2ZqQB74Vz5YTdixiU+66r6bEFCYSfbybewUjaI1Hvjo3QnPSl2imMA9ofQ8Onux9CElFKWDKoVOmdWb8jbIbOdFoY0c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=SmOKsqmD; arc=none smtp.client-ip=209.85.218.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="SmOKsqmD" Received: by mail-ej1-f52.google.com with SMTP id a640c23a62f3a-a72510ebc3fso698398366b.2; Thu, 27 Jun 2024 05:59:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1719493187; x=1720097987; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wMm73DkQQbIKqvrdgDxsT4wUsrpx/7Ch+5m3Ck5HfkI=; b=SmOKsqmDk+u+cQy2fQWjr7jwEsos+IX+lnHlhMkJmWVK68yrxQ//Sd3FAsRBCDL7HV 2icjMYR2dTBgWJn8QYbqI8NkDp/ZsxCq5cVpeRaoUTe3lRBXOtyJMgGx9esaPDTtB8H+ zL+b24TITYBLXU5VPxWTw0dUibzOmVC5agrdL1sCmqNDurG+SAx6jRfDYJx+hKWeO4lI 4IdcVdVleJZJqXuOMMCfggFvx97h1OEjVjFo8SDLDVE8cWgLgeosqI91jkM/tQ4WM11O nGutI9/XHrPZuRy7sZKT2TJU6Rwnm9YyvKXFI/mHfwPPyPMJ0Ko0ITItJrvBThF7JSKo 6lnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719493187; x=1720097987; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wMm73DkQQbIKqvrdgDxsT4wUsrpx/7Ch+5m3Ck5HfkI=; b=aNt5VbReomyZ7dJPsdUXmyUs+ul6nvYVTJFKvKqaR2bTJJEnk77DQDset1auj3DDIo KZ7Cbq48sZ3/GHcYlzLio39Dx3FCC3/McQqsD4bGEaPIMuRbhfGCzu9OZG29Md+/061X m46nGjlQIErGYmdtVcxjoBCJWtqSmrhZsOogDSwTkZWwN31dX2RT5gtRsGnTCAGbZ48I ZD/jJ2Mbh2sI7ecFa9dePEvVDJx7WuEpt8Xt9XJ/TcVUUfb1uOBEJWbPF34W8ZvJnqKm Tornl9xG697WSX59IHIfKZ12zWF4geSfxEKTXxB4HVlG3D0wC3T6OmXBo2xrrNAwCdlr BRCw== X-Forwarded-Encrypted: i=1; AJvYcCU8FN7XYqSp7K9Fu5iSAMHMhKRIdq16KqxJ96gdKTDNl998IC6gqypV0K1k7BFFys+8YsbJLf7vgqpCyhdUuStd1I/LAeJP X-Gm-Message-State: AOJu0YxBXksEBgm/sviz54PgazKWbZYGLRFlAiC0ubA4PhTihMGgAVoo Wg0TVtB5UFVL26aJ80IvYyCXFI8tc5byMqiTRKSGZweVapcTeEPgKXCTSKMS X-Google-Smtp-Source: AGHT+IF7w1FciUHwpOTOnu9uCADI6XtyROo6IXITmFVw9xvMf6MobEAU9TB6ZODtgxksRY3Th1iSQg== X-Received: by 2002:a17:906:7f89:b0:a6f:bf0f:4209 with SMTP id a640c23a62f3a-a7242c4b8c4mr1009588566b.42.1719493186099; Thu, 27 Jun 2024 05:59:46 -0700 (PDT) Received: from 127.0.0.1localhost ([163.114.131.193]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a729d7c95a3sm57267766b.194.2024.06.27.05.59.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Jun 2024 05:59:45 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, "David S . Miller" , Jakub Kicinski , David Ahern , Eric Dumazet , Willem de Bruijn Subject: [PATCH net-next 4/5] io_uring/net: move charging socket out of zc io_uring Date: Thu, 27 Jun 2024 13:59:44 +0100 Message-ID: <1e55ad85b726d50a45bdd35fc04e1565d3ba7896.1719190216.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Currently, io_uring's io_sg_from_iter() duplicates the part of __zerocopy_sg_from_iter() charging pages to the socket. It'd be too easy to miss while changing it in net/, the chunk is not the most straightforward for outside users and full of internal implementation details. io_uring is not a good place to keep it, deduplicate it by moving out of the callback into __zerocopy_sg_from_iter(). Signed-off-by: Pavel Begunkov Reviewed-by: Willem de Bruijn --- include/linux/skbuff.h | 3 +++ include/linux/socket.h | 2 +- io_uring/net.c | 16 ++++------------ net/core/datagram.c | 10 +++++----- 4 files changed, 13 insertions(+), 18 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index f4cda3fbdb75..9c29bdd5596d 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -1703,6 +1703,9 @@ int __zerocopy_sg_from_iter(struct msghdr *msg, struct sock *sk, struct sk_buff *skb, struct iov_iter *from, size_t length); +int zerocopy_fill_skb_from_iter(struct sk_buff *skb, + struct iov_iter *from, size_t length); + static inline int skb_zerocopy_iter_dgram(struct sk_buff *skb, struct msghdr *msg, int len) { diff --git a/include/linux/socket.h b/include/linux/socket.h index 89d16b90370b..2a1ff91d1914 100644 --- a/include/linux/socket.h +++ b/include/linux/socket.h @@ -76,7 +76,7 @@ struct msghdr { __kernel_size_t msg_controllen; /* ancillary data buffer length */ struct kiocb *msg_iocb; /* ptr to iocb for async requests */ struct ubuf_info *msg_ubuf; - int (*sg_from_iter)(struct sock *sk, struct sk_buff *skb, + int (*sg_from_iter)(struct sk_buff *skb, struct iov_iter *from, size_t length); }; diff --git a/io_uring/net.c b/io_uring/net.c index 7c98c4d50946..84a7602bcef1 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -1265,14 +1265,14 @@ int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) return io_sendmsg_prep_setup(req, req->opcode == IORING_OP_SENDMSG_ZC); } -static int io_sg_from_iter_iovec(struct sock *sk, struct sk_buff *skb, +static int io_sg_from_iter_iovec(struct sk_buff *skb, struct iov_iter *from, size_t length) { skb_zcopy_downgrade_managed(skb); - return __zerocopy_sg_from_iter(NULL, sk, skb, from, length); + return zerocopy_fill_skb_from_iter(skb, from, length); } -static int io_sg_from_iter(struct sock *sk, struct sk_buff *skb, +static int io_sg_from_iter(struct sk_buff *skb, struct iov_iter *from, size_t length) { struct skb_shared_info *shinfo = skb_shinfo(skb); @@ -1285,7 +1285,7 @@ static int io_sg_from_iter(struct sock *sk, struct sk_buff *skb, if (!frag) shinfo->flags |= SKBFL_MANAGED_FRAG_REFS; else if (unlikely(!skb_zcopy_managed(skb))) - return __zerocopy_sg_from_iter(NULL, sk, skb, from, length); + return zerocopy_fill_skb_from_iter(skb, from, length); bi.bi_size = min(from->count, length); bi.bi_bvec_done = from->iov_offset; @@ -1312,14 +1312,6 @@ static int io_sg_from_iter(struct sock *sk, struct sk_buff *skb, skb->data_len += copied; skb->len += copied; skb->truesize += truesize; - - if (sk && sk->sk_type == SOCK_STREAM) { - sk_wmem_queued_add(sk, truesize); - if (!skb_zcopy_pure(skb)) - sk_mem_charge(sk, truesize); - } else { - refcount_add(truesize, &skb->sk->sk_wmem_alloc); - } return ret; } diff --git a/net/core/datagram.c b/net/core/datagram.c index 2b24d69b1e94..554316c40883 100644 --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -610,8 +610,8 @@ int skb_copy_datagram_from_iter(struct sk_buff *skb, int offset, } EXPORT_SYMBOL(skb_copy_datagram_from_iter); -static int zerocopy_fill_skb_from_iter(struct sk_buff *skb, - struct iov_iter *from, size_t length) +int zerocopy_fill_skb_from_iter(struct sk_buff *skb, + struct iov_iter *from, size_t length) { int frag = skb_shinfo(skb)->nr_frags; @@ -687,11 +687,11 @@ int __zerocopy_sg_from_iter(struct msghdr *msg, struct sock *sk, int ret; if (msg && msg->msg_ubuf && msg->sg_from_iter) - return msg->sg_from_iter(sk, skb, from, length); + ret = msg->sg_from_iter(skb, from, length); + else + ret = zerocopy_fill_skb_from_iter(skb, from, length); - ret = zerocopy_fill_skb_from_iter(skb, from, length); truesize = skb->truesize - orig_size; - if (sk && sk->sk_type == SOCK_STREAM) { sk_wmem_queued_add(sk, truesize); if (!skb_zcopy_pure(skb)) From patchwork Thu Jun 27 12:59:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13714340 Received: from mail-ej1-f42.google.com (mail-ej1-f42.google.com [209.85.218.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 108501802DF; Thu, 27 Jun 2024 12:59:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719493191; cv=none; b=sUUf27ef0B8d9JnaVSbEUSMUJmgR1G1VnEzcOuk92xLixq1MVr54ENBPTACw2M7wwZLRQyy4HipOahDepkHDdWi0XvlvlI1nb4LSfjLxtVWEa14WZO9QChLf+g8AskJwxrL4VLLxeLFNJiYv9c3K4izA1oW98dXpV3pWEhRempQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719493191; c=relaxed/simple; bh=V+7scOsu2EG9oGv6jur44/RxU3MiYpIAUVh4cY5Kd9Q=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cHTEEMF+QZ7alyzcDh3H2/6IZFw+XZ9thxhS/X/qT7c9YBcDTW6NCiXrutsgu5RYv6TmF62WahTMo7lZCPf9amkikS8Cz0vqrIGG8bf+wWoXptTF6WZvMCOJj52xvc6lwTjMeAmNPHlQl2ZqO6G+ajF3lc5d1syLCpm7BYQPScU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=jgjAKBxP; arc=none smtp.client-ip=209.85.218.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="jgjAKBxP" Received: by mail-ej1-f42.google.com with SMTP id a640c23a62f3a-a724b4f1218so676143966b.2; Thu, 27 Jun 2024 05:59:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1719493188; x=1720097988; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Iwlyl7jv323B/JiH9F8ir0gBkVWXQIO2tv0pQQWMneQ=; b=jgjAKBxPxGATBFMu6vrrUAIKuRxwnMdxzBzH02y+0Lq9Q6XDYfekkkCoXdBxOr3Jrq Pb/ji42BWuuZbGT1PfKZTkoEvxyMcAQzeVqYiV85odDSIKhthQK47Lye70XMDuZwexKP aiJF7O6au3EF9Hue5Y8dAUwn2YR6j8IYFJ2Cv3njFSj7sy4/gnK1bQ2iXdNvIit34GhX D9mlrQDwhVdTM2eCFQKCtgcd6UvtMP9TiXnBF08VUWmdQk7hot798elO6IH48K+Cxhpx 55Sa4Fq9yseWyUSJHRHV4qJxmY5PmEXoyMV/KI9/X9uyh5TfvjJ7ahSOJBcdWzl+Z9/C H4sw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719493188; x=1720097988; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Iwlyl7jv323B/JiH9F8ir0gBkVWXQIO2tv0pQQWMneQ=; b=Wg97BohuBfoYdCnoasVStPLZa1DSLQBKMhs8JtRbqFwxnwoOg7RqwlBM1O783Mm/Gy yuB3iugjAX5xpWwmvMWxfGlP8dthsDL9DWRna56DYrFoHSRLTnmF9LvjbOacrnB6BJ/n egvnzl9FwjSg5rlEPpRALYL8PMCK1pTeavKrPIOATxSJACgx4g92wEWor6da0gYCbXfT BKZ229UgC9/S3moBIi2+bRIeUEQgrkN9n8oFCvHm202KPEYQBhynpzVNiOpNmMzl6iCL wmRRTmLGudoYe/hI59g00rvwWM4NgATdDQKqKpun89LmZLCh7e9fnzKKs/lyeMbeNxdU yodw== X-Forwarded-Encrypted: i=1; AJvYcCUUJatMAhHr/ktyIl9OebITjsRNn1xgy++10dZdbF5PaT5znv0N5FqusBBTivTbyCm3We/e0XyaaI8pBoTuCvj/G9USvovz X-Gm-Message-State: AOJu0Ywg4FmxRjjnz10Ou/HZkElzEyois1ClJdsEolT6ZONX7mhH+GGE WXmOI0JODALirGjhlTAIEaBo2Nk/BWaayLBqmwsrL5xwOECBBh1WJp2N3DEr X-Google-Smtp-Source: AGHT+IFTHUJ0n+XF7DXbRHGFvJLep8DObLADpO+rj9XioGpbHJDhuyTnUgZMnQRnKnGmqYkdXb8L3A== X-Received: by 2002:a17:907:a649:b0:a72:7e82:7a15 with SMTP id a640c23a62f3a-a727e827b79mr597962266b.23.1719493188175; Thu, 27 Jun 2024 05:59:48 -0700 (PDT) Received: from 127.0.0.1localhost ([163.114.131.193]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a729d7c95a3sm57267766b.194.2024.06.27.05.59.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Jun 2024 05:59:47 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, "David S . Miller" , Jakub Kicinski , David Ahern , Eric Dumazet , Willem de Bruijn Subject: [PATCH net-next 5/5] net: limit scope of a skb_zerocopy_iter_stream var Date: Thu, 27 Jun 2024 13:59:45 +0100 Message-ID: <3f4a39b3966204f062200caad857ff822f9f2895.1719190216.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 skb_zerocopy_iter_stream() only uses @orig_uarg in the !link_skb path, and we can move the local variable in the appropriate block. Signed-off-by: Pavel Begunkov Reviewed-by: Willem de Bruijn --- net/core/skbuff.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 9f28822dde6f..9b71e4b8796a 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -1856,7 +1856,6 @@ int skb_zerocopy_iter_stream(struct sock *sk, struct sk_buff *skb, struct msghdr *msg, int len, struct ubuf_info *uarg) { - struct ubuf_info *orig_uarg = skb_zcopy(skb); int err, orig_len = skb->len; if (uarg->ops->link_skb) { @@ -1864,6 +1863,8 @@ int skb_zerocopy_iter_stream(struct sock *sk, struct sk_buff *skb, if (err) return err; } else { + struct ubuf_info *orig_uarg = skb_zcopy(skb); + /* An skb can only point to one uarg. This edge case happens * when TCP appends to an skb, but zerocopy_realloc triggered * a new alloc.