From patchwork Tue Apr 11 16:08:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13207807 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D98EC76196 for ; Tue, 11 Apr 2023 16:09:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9C07A900004; Tue, 11 Apr 2023 12:09:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 948E6900002; Tue, 11 Apr 2023 12:09:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7C1FC900004; Tue, 11 Apr 2023 12:09:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 67877900002 for ; Tue, 11 Apr 2023 12:09:18 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 3D80B80472 for ; Tue, 11 Apr 2023 16:09:18 +0000 (UTC) X-FDA: 80669594796.25.DBD2973 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf22.hostedemail.com (Postfix) with ESMTP id 67627C002B for ; Tue, 11 Apr 2023 16:09:15 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Lc3jxrvB; spf=pass (imf22.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681229355; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QAACn2mr7ooi1DdUlemHLPNq4itk2LGg3Qvic6fgj7U=; b=wr5637dTeDBhvNUQMb4DyV7GopDYUDN4DJIYoDIf1lWFq1yiExsGbNbeUSnXS9pRpVyY7j xL+BazQOMHiCedlW7BfxAFE6wKbAefonN1QvEBC4gAnDIRkI/bZYYjCJAdJ9m6jLaXkoYT 3oNBgZ0aDevilWlbzSgr5WRqEcPcoFM= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Lc3jxrvB; spf=pass (imf22.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681229355; a=rsa-sha256; cv=none; b=KJv0dQinoZJjmuhEkE+wk95rbduTEhuiytEoYmTes/IDvC+1N6l7QSRopYCNYmUnEWGYdc Wy7xxT/akrDSpSbMwSiC5PUs7dF4J6OWbkkqJtQGapZ4FXkqKvIrfMY0nIDe5l1SdeyoE2 V4WLWag7qlSLE0amsjAjOVjCENq3RD4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681229354; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QAACn2mr7ooi1DdUlemHLPNq4itk2LGg3Qvic6fgj7U=; b=Lc3jxrvBuCRV0OTew0l+PO9MKhGeM5EAFfV5sFtKyBmwBvVPAoMJkV0t0ZMQyEpLbK1GyZ YI1OCJ91kk8ADNntgQsS7nl7/v3TiF7gC5MarIy5eYp53WU2mKMYQgE6H+MwwRJLBjQuZa IyRPKRvljXZIOGjdtDUGrWnmuunJDVw= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-425-m3bGLW3zMRuPyvCVdlDU1g-1; Tue, 11 Apr 2023 12:09:11 -0400 X-MC-Unique: m3bGLW3zMRuPyvCVdlDU1g-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 77C5083DE48; Tue, 11 Apr 2023 16:09:10 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4197440C20FA; Tue, 11 Apr 2023 16:09:08 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Willem de Bruijn Subject: [PATCH net-next v6 01/18] net: Declare MSG_SPLICE_PAGES internal sendmsg() flag Date: Tue, 11 Apr 2023 17:08:45 +0100 Message-Id: <20230411160902.4134381-2-dhowells@redhat.com> In-Reply-To: <20230411160902.4134381-1-dhowells@redhat.com> References: <20230411160902.4134381-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 67627C002B X-Stat-Signature: a4yybqkokhptu9xosjnw4so64buhsk5z X-HE-Tag: 1681229355-283497 X-HE-Meta: U2FsdGVkX1/8QF4kNL3SXnQ7M/Uu/1xP6qp6xIAcH+T+D0kZbw0XAc7NSSi+JIgHnMvFIMg2Gxe5nBc+94ls4MuYAlg642Vaj0lE5ihn0XW8cmTGpMm4T9Cyfmj3BCgLgprpnYWIDkmKtofgdKRCJCvfbpZocdP2WiygOoWWmPshorQrbL2usb0ajVZZsGjVbJEM2TtcOEw/0iyemfN8wjjtYJPjTNukHhQ9U0Br2B1pU6ax+s9KfPtsv4eWaoc1cyxzJ33Awqrq1vffTrRyOkrEJhdWc1luHj8uvgGdcVEh65jr9Z2XibYVKii5lX1VjBl+c71Rnz0uU0G+UmzB1AmYVkyy+77LFiJZMk5RNrxJMNOBkPztGZs+jt1PcRt8k8oAlIbclsQ6KSdag4T25QzneQJaxX3yxx5x/eMopc/c6oVxBWeQ3Mna4CAUUfTQLHF56i/4pYcxD0k+U0HBrynmLsRqEGEXkycomz6wxGEXCCgP+bvgUDALPZjnQntgjp/OXIJDnQbAsQMsdyrvrlrMuD22ZTZLmied1rRMoohfrfRqhD3H9Aq+3VAP4bANLE3EMy7EeOqkCP8leDBG99VvJyVcjgfCw60JGRGYxMaadOS3VjS/NR0B8ikoN58pwk+7Zl1+UtEAwGxfHrCKsprR2xqf6MgGorjj3fHTk/xDJnW+/ndOUF3tw6avMP6sMWQqWG3shvnAF9S13K9Q2zRSQ7rGxsAEpeoZPQk744qvf1RjMfyslPRJUdvlkxv2IkcdsvXoG3IBWHbEeIA2Gt38jgvJxuYKVs1IKBADiuhcQzZkyCh4zN0i5Pm1LF6oLf1860KEjGvNKidwaMEtPwcNx+nxjO/ppnEK5W5qy826dCPYKJFVXjLs4MUnYqvLGkVKdwxnBxpeum8P60nk/X1Qi0qe/pk3w3xVUzPBiYFjgjx7hZEOxSZaua6w1FPLP1hmkp2/qDfiQlrncSg Sw8WOwu/ 8bsYYhwFmaEwyIDdLkpcnWNWKnozoKLnFuX5I7Dgm/8uwGTVIefmKR2kw/6U+zEGFFwypxARwWFX+JuU4v4eLK5AGH6hG4h3pEHXoYY7BDYqkx9NFbSXjBrEIKBdRVfU997fDfGemdTxFTHvn07UzyA2NNvLzeivaN/xWtu9woLwao4vEMzSI/Hil16eYUa26+ePENSgShjeE20xvbUQXQZtyLuinN+BveeuOzL+JGQyW4+fOmcKHFZa9WwctwipIVvht5+QDazspaZQMGSjKyxwSqHHJ3BwpfSFXKWEOFmOOJ1kc/1F1Ck04/UIKZ4RRVKcTAU0WZ4XCaopaHoOGAFgYpyAg0MFKFW8LQ7cscFwfy6rcds5sFSfVa4NaBzHXfFiKbQ2ikjiQrIYNIt7jFL7oiZRVDDwJtbUchMxcW2j3Mo05J5KyUvIY6GxymLj2jWHVDBASBU0PMMKXLuP2ajEZ6rTFXYgJW4faQSxcmh8an5SI8X7PhRlNIwZl14Iub1da X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Declare MSG_SPLICE_PAGES, an internal sendmsg() flag, that hints to a network protocol that it should splice pages from the source iterator rather than copying the data if it can. This flag is added to a list that is cleared by sendmsg syscalls on entry. This is intended as a replacement for the ->sendpage() op, allowing a way to splice in several multipage folios in one go. Signed-off-by: David Howells Reviewed-by: Willem de Bruijn cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/linux/socket.h | 3 +++ net/socket.c | 2 ++ 2 files changed, 5 insertions(+) diff --git a/include/linux/socket.h b/include/linux/socket.h index 13c3a237b9c9..bd1cc3238851 100644 --- a/include/linux/socket.h +++ b/include/linux/socket.h @@ -327,6 +327,7 @@ struct ucred { */ #define MSG_ZEROCOPY 0x4000000 /* Use user data in kernel path */ +#define MSG_SPLICE_PAGES 0x8000000 /* Splice the pages from the iterator in sendmsg() */ #define MSG_FASTOPEN 0x20000000 /* Send data in TCP SYN */ #define MSG_CMSG_CLOEXEC 0x40000000 /* Set close_on_exec for file descriptor received through @@ -337,6 +338,8 @@ struct ucred { #define MSG_CMSG_COMPAT 0 /* We never have 32 bit fixups */ #endif +/* Flags to be cleared on entry by sendmsg and sendmmsg syscalls */ +#define MSG_INTERNAL_SENDMSG_FLAGS (MSG_SPLICE_PAGES) /* Setsockoptions(2) level. Thanks to BSD these must match IPPROTO_xxx */ #define SOL_IP 0 diff --git a/net/socket.c b/net/socket.c index 73e493da4589..b3fd3f7f7e03 100644 --- a/net/socket.c +++ b/net/socket.c @@ -2136,6 +2136,7 @@ int __sys_sendto(int fd, void __user *buff, size_t len, unsigned int flags, msg.msg_name = (struct sockaddr *)&address; msg.msg_namelen = addr_len; } + flags &= ~MSG_INTERNAL_SENDMSG_FLAGS; if (sock->file->f_flags & O_NONBLOCK) flags |= MSG_DONTWAIT; msg.msg_flags = flags; @@ -2483,6 +2484,7 @@ static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys, } msg_sys->msg_flags = flags; + flags &= ~MSG_INTERNAL_SENDMSG_FLAGS; if (sock->file->f_flags & O_NONBLOCK) msg_sys->msg_flags |= MSG_DONTWAIT; /* From patchwork Tue Apr 11 16:08:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13207808 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38EADC76196 for ; Tue, 11 Apr 2023 16:09:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BBD24900005; Tue, 11 Apr 2023 12:09:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B444F900002; Tue, 11 Apr 2023 12:09:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9BE62900005; Tue, 11 Apr 2023 12:09:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 89C0D900002 for ; Tue, 11 Apr 2023 12:09:24 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 54432120D4C for ; Tue, 11 Apr 2023 16:09:24 +0000 (UTC) X-FDA: 80669595048.10.6330256 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf08.hostedemail.com (Postfix) with ESMTP id 5BA08160003 for ; Tue, 11 Apr 2023 16:09:21 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="Eh4PJA/G"; spf=pass (imf08.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681229361; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GgIr9+xYtjWCQCpR/5vL0XO1pMxYzzO4LkH470whdw8=; b=lJao8zmH5w83amYF1FzajQjSwyLtCL5+cNf5PDOQ8c/n7PmKJIZgJ0mooGfQcxHbxMv0HA AgYSIZIyf9lvOs0TnXgbXjDj7gioyZ2ChlD7MoEf1f9mZlT1J3RjYJjOik4Klak9rSUMp6 N5VkKm341XaGZFk+DrlycJFYk1zDsV0= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="Eh4PJA/G"; spf=pass (imf08.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681229361; a=rsa-sha256; cv=none; b=EKHicg2YehZz5PTmkwBrCbMvHHv37ARdQTYOeAXW53Ggd1ONnVSGMTcT0h0K1pBIqHDPQ+ In8yuasfbPIL68XwO7HJyp9Yb8Ycb51w4xQ+hcByMaNS/HO83cm8QuAnaitFzlhVfMDhkk MDC9ksFw/SdY6oZ8VsqV8VUPE1fRvls= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681229360; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GgIr9+xYtjWCQCpR/5vL0XO1pMxYzzO4LkH470whdw8=; b=Eh4PJA/GhQixkB0zVE79PMEsE2aNXSoi9tyOTIL+iYPFMeC3Q+AQfumM+5x2702XsK+EVg Ucb+ozHqOeqhekPD2VzAqmVMDzsg4zV9/i6vEwxHiUKaDaet1h/3pX2SmcGlzRIlKfRGDp hDNAJWm5kRjtDjh3C6gySIrXk2FdmrY= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-295-4t8stfIHMqWJmxVt1DIeCQ-1; Tue, 11 Apr 2023 12:09:14 -0400 X-MC-Unique: 4t8stfIHMqWJmxVt1DIeCQ-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 667E02807D65; Tue, 11 Apr 2023 16:09:13 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2E90040C83A9; Tue, 11 Apr 2023 16:09:11 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Andrew Morton Subject: [PATCH net-next v6 02/18] mm: Move the page fragment allocator from page_alloc.c into its own file Date: Tue, 11 Apr 2023 17:08:46 +0100 Message-Id: <20230411160902.4134381-3-dhowells@redhat.com> In-Reply-To: <20230411160902.4134381-1-dhowells@redhat.com> References: <20230411160902.4134381-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 5BA08160003 X-Stat-Signature: yhfgrt5ow193t8ttd3i7ruso59ef7hk4 X-Rspam-User: X-HE-Tag: 1681229361-398594 X-HE-Meta: U2FsdGVkX1+HscWtTGhnoIPd/7KFk+de1cmnha9b5suKYNXJvJESD0pNwDGlnYtesPSA08YprvEnusV2ciFJKBCBQbROJacozcA+oOiQSRQBmOAkXhxp17F3L3We2lqnVNxXS/vu63k9dqDbkbFQUAcWNvAekwEbW9vLF8YO86n9/D+ZIj65UJhPYrwDRV7unAmSXzZlZ1WNM0sdDZDzmfEb+xJk2N4z8uQX0SIKjFmsFs4UEuuipI6m0FJkNJIZ0oPKD4J1YkjDNC5UL3dxVNP2GDc98nnfHjrwdnPXo6OuVaBmp3WkPPHgsG08ZC7BnEhUkwEQYlvR03ZXgA82hJy9/3+oIEcWRXXO8q+uDHc6q75q3epskCXEKHTWiLqoCkcTB6/oKVdrQS/doi3nEOzoYnBAqGEMTXOb20JakIFR2349BqkjMN8IcXyLLy1i69oOYgdHxAd69ORm6HwKzRFgipvZWVVNKRYjSCRq8O6ZI9BxIeIsxukim0orp7JWwAMesm+KCiaYnS4fedd+ghJKdjrOo0YquZ9RexhiSFRqxMBFBznqgIWvyL2Nasun7b6lk0Cs3dUa2vNl2YdF/u5zPxRUA9eLQhfAw2i3nZcYjalwcv5I9ggMcHaSp+ncLAbzrrz0TlOZDL+Ju9S883O54eA2FRTqFut93A68Jm1UjofdaezUw4c8Qx2EY9PNVLMa+sPymo07pX71MlayL3FDVkJAJPyTKqL6Bvc5mVETep9e2uKbmKLysv0gSpxc4d431WDoo8ssPu1i0E8IGxE7XfLVXUN99SxcF2aoZbOb+JejomTBO8WfhI7saZmMjhdvZ70lLdHarLiQd/lb9bhq4vwKPz7z1SnfqOKkKbXQoPKbQb+vvtH4lAy4PhOPOmxfqeXNIZ24RbN6jDL/eg3WhExxzBDAfxOIAofV7088Wa3SFzQSjunlnLJpreyaE6ehrOtQwyXGkycZZi7 F/0Io6i/ zigL/UCxHdhKa4xl02cXE2h3dUkjvt1G9lCzFIjY3vHHDG6WueGfA+T88txU6s5OIGtRNRjeQ4hkMWEG122F+5ih/PdKqScXjCFA7ytE+NkZFH/8+2lpdl1KlmQczaPbM+J5O0E0vOXkFwmGDfP/pSPg9QYzXcptKAEcN73G3vGlACqkzWHhPFp1snmrB2KEWeKP0mqBl0c00uj+nj1bg405qjtZbTJ9sKGlFAnOTOs0zB6sA7lLJndkkNN4XiBKAkK4v0G43tmnocgQKxETswTbzQTwndZHnprAGAoYNRTq9/OLldfIuaPQCUYuGrHTV7t3b5YvU7kJgIy3svtXrZ6ds70088aSaWrVaL+G1d8ki+sdj/vctdtTJjB1tFTWlW5E3clMRCexbS0rjkNSC1LfOIND8/HeOUPoVoRDcX6of7wAsGguJhgsVjaJ3KFm/pd8vYifEcrrKSBIyT9X+eNYvtgmY9QgbikA5siqffzrGb9eWuJCTKYt7fzmvx2NmXSkAbEMJZaeZi1Mc1KQWmkIA0Az7dynaZMMU0AiCFHZ9G6qhaKKCCWpudNCcjXusuyNsog3exB+ralRMOa5m7XJgdeXi5yQLZfUL9+5q2DyRFE09p7s5GjRmhGXjzW7iuZOM X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move the page fragment allocator from page_alloc.c into its own file preparatory to changing it. Signed-off-by: David Howells cc: Andrew Morton cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-mm@kvack.org cc: netdev@vger.kernel.org --- mm/Makefile | 2 +- mm/page_alloc.c | 126 ----------------------------------------- mm/page_frag_alloc.c | 131 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 132 insertions(+), 127 deletions(-) create mode 100644 mm/page_frag_alloc.c diff --git a/mm/Makefile b/mm/Makefile index 8e105e5b3e29..4e6dc12b4cbd 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -52,7 +52,7 @@ obj-y := filemap.o mempool.o oom_kill.o fadvise.o \ readahead.o swap.o truncate.o vmscan.o shmem.o \ util.o mmzone.o vmstat.o backing-dev.o \ mm_init.o percpu.o slab_common.o \ - compaction.o \ + compaction.o page_frag_alloc.o \ interval_tree.o list_lru.o workingset.o \ debug.o gup.o mmap_lock.o $(mmu-y) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 7136c36c5d01..d751e750c14b 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -5695,132 +5695,6 @@ void free_pages(unsigned long addr, unsigned int order) EXPORT_SYMBOL(free_pages); -/* - * Page Fragment: - * An arbitrary-length arbitrary-offset area of memory which resides - * within a 0 or higher order page. Multiple fragments within that page - * are individually refcounted, in the page's reference counter. - * - * The page_frag functions below provide a simple allocation framework for - * page fragments. This is used by the network stack and network device - * drivers to provide a backing region of memory for use as either an - * sk_buff->head, or to be used in the "frags" portion of skb_shared_info. - */ -static struct page *__page_frag_cache_refill(struct page_frag_cache *nc, - gfp_t gfp_mask) -{ - struct page *page = NULL; - gfp_t gfp = gfp_mask; - -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - gfp_mask |= __GFP_COMP | __GFP_NOWARN | __GFP_NORETRY | - __GFP_NOMEMALLOC; - page = alloc_pages_node(NUMA_NO_NODE, gfp_mask, - PAGE_FRAG_CACHE_MAX_ORDER); - nc->size = page ? PAGE_FRAG_CACHE_MAX_SIZE : PAGE_SIZE; -#endif - if (unlikely(!page)) - page = alloc_pages_node(NUMA_NO_NODE, gfp, 0); - - nc->va = page ? page_address(page) : NULL; - - return page; -} - -void __page_frag_cache_drain(struct page *page, unsigned int count) -{ - VM_BUG_ON_PAGE(page_ref_count(page) == 0, page); - - if (page_ref_sub_and_test(page, count)) - free_the_page(page, compound_order(page)); -} -EXPORT_SYMBOL(__page_frag_cache_drain); - -void *page_frag_alloc_align(struct page_frag_cache *nc, - unsigned int fragsz, gfp_t gfp_mask, - unsigned int align_mask) -{ - unsigned int size = PAGE_SIZE; - struct page *page; - int offset; - - if (unlikely(!nc->va)) { -refill: - page = __page_frag_cache_refill(nc, gfp_mask); - if (!page) - return NULL; - -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - /* if size can vary use size else just use PAGE_SIZE */ - size = nc->size; -#endif - /* Even if we own the page, we do not use atomic_set(). - * This would break get_page_unless_zero() users. - */ - page_ref_add(page, PAGE_FRAG_CACHE_MAX_SIZE); - - /* reset page count bias and offset to start of new frag */ - nc->pfmemalloc = page_is_pfmemalloc(page); - nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; - nc->offset = size; - } - - offset = nc->offset - fragsz; - if (unlikely(offset < 0)) { - page = virt_to_page(nc->va); - - if (!page_ref_sub_and_test(page, nc->pagecnt_bias)) - goto refill; - - if (unlikely(nc->pfmemalloc)) { - free_the_page(page, compound_order(page)); - goto refill; - } - -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - /* if size can vary use size else just use PAGE_SIZE */ - size = nc->size; -#endif - /* OK, page count is 0, we can safely set it */ - set_page_count(page, PAGE_FRAG_CACHE_MAX_SIZE + 1); - - /* reset page count bias and offset to start of new frag */ - nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; - offset = size - fragsz; - if (unlikely(offset < 0)) { - /* - * The caller is trying to allocate a fragment - * with fragsz > PAGE_SIZE but the cache isn't big - * enough to satisfy the request, this may - * happen in low memory conditions. - * We don't release the cache page because - * it could make memory pressure worse - * so we simply return NULL here. - */ - return NULL; - } - } - - nc->pagecnt_bias--; - offset &= align_mask; - nc->offset = offset; - - return nc->va + offset; -} -EXPORT_SYMBOL(page_frag_alloc_align); - -/* - * Frees a page fragment allocated out of either a compound or order 0 page. - */ -void page_frag_free(void *addr) -{ - struct page *page = virt_to_head_page(addr); - - if (unlikely(put_page_testzero(page))) - free_the_page(page, compound_order(page)); -} -EXPORT_SYMBOL(page_frag_free); - static void *make_alloc_exact(unsigned long addr, unsigned int order, size_t size) { diff --git a/mm/page_frag_alloc.c b/mm/page_frag_alloc.c new file mode 100644 index 000000000000..bee95824ef8f --- /dev/null +++ b/mm/page_frag_alloc.c @@ -0,0 +1,131 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Page fragment allocator + * + * Page Fragment: + * An arbitrary-length arbitrary-offset area of memory which resides within a + * 0 or higher order page. Multiple fragments within that page are + * individually refcounted, in the page's reference counter. + * + * The page_frag functions provide a simple allocation framework for page + * fragments. This is used by the network stack and network device drivers to + * provide a backing region of memory for use as either an sk_buff->head, or to + * be used in the "frags" portion of skb_shared_info. + */ + +#include +#include +#include + +static struct page *__page_frag_cache_refill(struct page_frag_cache *nc, + gfp_t gfp_mask) +{ + struct page *page = NULL; + gfp_t gfp = gfp_mask; + +#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) + gfp_mask |= __GFP_COMP | __GFP_NOWARN | __GFP_NORETRY | + __GFP_NOMEMALLOC; + page = alloc_pages_node(NUMA_NO_NODE, gfp_mask, + PAGE_FRAG_CACHE_MAX_ORDER); + nc->size = page ? PAGE_FRAG_CACHE_MAX_SIZE : PAGE_SIZE; +#endif + if (unlikely(!page)) + page = alloc_pages_node(NUMA_NO_NODE, gfp, 0); + + nc->va = page ? page_address(page) : NULL; + + return page; +} + +void __page_frag_cache_drain(struct page *page, unsigned int count) +{ + VM_BUG_ON_PAGE(page_ref_count(page) == 0, page); + + if (page_ref_sub_and_test(page, count - 1)) + __free_pages(page, compound_order(page)); +} +EXPORT_SYMBOL(__page_frag_cache_drain); + +void *page_frag_alloc_align(struct page_frag_cache *nc, + unsigned int fragsz, gfp_t gfp_mask, + unsigned int align_mask) +{ + unsigned int size = PAGE_SIZE; + struct page *page; + int offset; + + if (unlikely(!nc->va)) { +refill: + page = __page_frag_cache_refill(nc, gfp_mask); + if (!page) + return NULL; + +#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) + /* if size can vary use size else just use PAGE_SIZE */ + size = nc->size; +#endif + /* Even if we own the page, we do not use atomic_set(). + * This would break get_page_unless_zero() users. + */ + page_ref_add(page, PAGE_FRAG_CACHE_MAX_SIZE); + + /* reset page count bias and offset to start of new frag */ + nc->pfmemalloc = page_is_pfmemalloc(page); + nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; + nc->offset = size; + } + + offset = nc->offset - fragsz; + if (unlikely(offset < 0)) { + page = virt_to_page(nc->va); + + if (page_ref_count(page) != nc->pagecnt_bias) + goto refill; + if (unlikely(nc->pfmemalloc)) { + page_ref_sub(page, nc->pagecnt_bias - 1); + __free_pages(page, compound_order(page)); + goto refill; + } + +#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) + /* if size can vary use size else just use PAGE_SIZE */ + size = nc->size; +#endif + /* OK, page count is 0, we can safely set it */ + set_page_count(page, PAGE_FRAG_CACHE_MAX_SIZE + 1); + + /* reset page count bias and offset to start of new frag */ + nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; + offset = size - fragsz; + if (unlikely(offset < 0)) { + /* + * The caller is trying to allocate a fragment + * with fragsz > PAGE_SIZE but the cache isn't big + * enough to satisfy the request, this may + * happen in low memory conditions. + * We don't release the cache page because + * it could make memory pressure worse + * so we simply return NULL here. + */ + return NULL; + } + } + + nc->pagecnt_bias--; + offset &= align_mask; + nc->offset = offset; + + return nc->va + offset; +} +EXPORT_SYMBOL(page_frag_alloc_align); + +/* + * Frees a page fragment allocated out of either a compound or order 0 page. + */ +void page_frag_free(void *addr) +{ + struct page *page = virt_to_head_page(addr); + + __free_pages(page, compound_order(page)); +} +EXPORT_SYMBOL(page_frag_free); From patchwork Tue Apr 11 16:08:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13207809 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 925C4C77B70 for ; Tue, 11 Apr 2023 16:09:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2BC32900006; Tue, 11 Apr 2023 12:09:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 243BE900002; Tue, 11 Apr 2023 12:09:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 09749900006; Tue, 11 Apr 2023 12:09:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id E9009900002 for ; Tue, 11 Apr 2023 12:09:27 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id BB570C0D23 for ; Tue, 11 Apr 2023 16:09:27 +0000 (UTC) X-FDA: 80669595174.28.070D22C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf11.hostedemail.com (Postfix) with ESMTP id 21A8B4002A for ; Tue, 11 Apr 2023 16:09:24 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cagfA6i8; spf=pass (imf11.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681229365; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ivyfYwnUZR0ptY6OLV+4rOfP5TDkuV50aqZyhshAVbQ=; b=oRouEmhza+CEriij66GcK4i9loS006z0KEacV/rpQScYRaxL/LXWJTrGWtFf8pbA2Wu2QB Dd687cV9F4vzRCakgIW7OpzSnXvMDn886ABuYzt+jrJiIPgAGnj3oi4uqHsdYy8cn8icAV RLmWTgcg0GyE3/bR7MQa4Vi3DAUjnbI= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cagfA6i8; spf=pass (imf11.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681229365; a=rsa-sha256; cv=none; b=37bCsb1GfjzlkftDhiSDF2vJo9UR1F2HeG+emLuSIojzHXoyRH/1tBx3rtgKRX2dN82Nyo hkPDvQf77J6LTvqHPZ8cPjIyCDTVM9JO2wfRF0bEcCnbzSKDEo97eIO5ooyzjzQ6Mxv47K LOXXii4vh+S4t9/BBMNk/gcdh0F8I9g= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681229364; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ivyfYwnUZR0ptY6OLV+4rOfP5TDkuV50aqZyhshAVbQ=; b=cagfA6i8tbLCDF84BwooxxX4L5hAgjKtBc6IQEmlAtzKAQgjLOn65uJAuEWc2mE+4YT2oW TCOJk/0wTViNmySDv9xfhx3ej5UUlkYKkuRR4lwOTjg7U/b/vZY+JGkCZgrVGWZ1ng+so1 MpfBQyhx9H0KltTQh5KpmaMdxesClCs= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-180-vRsXjnCWNMOIS6VRPgPWxQ-1; Tue, 11 Apr 2023 12:09:20 -0400 X-MC-Unique: vRsXjnCWNMOIS6VRPgPWxQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A59BA85A588; Tue, 11 Apr 2023 16:09:18 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0CE7714171D5; Tue, 11 Apr 2023 16:09:13 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Jeroen de Borst , Catherine Sullivan , Shailend Chand , Felix Fietkau , John Crispin , Sean Wang , Mark Lee , Lorenzo Bianconi , Matthias Brugger , AngeloGioacchino Del Regno , Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-mediatek@lists.infradead.org, linux-nvme@lists.infradead.org Subject: [PATCH net-next v6 03/18] mm: Make the page_frag_cache allocator use multipage folios Date: Tue, 11 Apr 2023 17:08:47 +0100 Message-Id: <20230411160902.4134381-4-dhowells@redhat.com> In-Reply-To: <20230411160902.4134381-1-dhowells@redhat.com> References: <20230411160902.4134381-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 21A8B4002A X-Stat-Signature: drhfuc4jdjd7wx7x1ybumfuofqmyzeu8 X-Rspam-User: X-HE-Tag: 1681229364-111814 X-HE-Meta: U2FsdGVkX1/oKtRKqP26wrXzzxiZ8+7UrgWz/gYFFOUK+1MSQU9KMlQeHUdhOf7Cqu0bI08zBiD23BBRF4YbdzThAjlJMCvAHbN0iK8+/7LqhtphcNtJWYDkIjR65UnApces2A/GRHpF7J+v86ksWV3Cw8Jzg30RSvodPnZQyz1j2tR+MfLxFV00uOfY9GbOxmRzONbg8QB4u2NuPtrpmdtnA3Fdwrm/Bg5iszZK+v1nCUshON8NvLTU6OFDcrY2Wcr4NdbNFqQEqZSmT6UfAtgt8c5MldL2Xkhuw2j7PJteG3CttcP+qoNxP+sn0mRKOBZlqi5Ka1Ts2nGXuJMyLbFTe5I9cJUGzC+lh/j1xhll+OP/91J8l8mjRNPR3MgIZWffAa6PFYNtGLmsr2sgEb1M5+C92S0lqiGaPfhhM2xys9LkG33I6ckyDij6tZp3tkDND9NoPYhrKMlsRx7m4WDlvWRjfxiBPyuXUaBXQztUOpigR9b1Pc5V/6wa7PBmOaOHAvdx6RyjRWC/A5LsujlJyoiBXzljkJpgZag6wx0yXaQvNUhxLH3DrZlnWuzd8PIpUHH2/poBUHWz5TY8b4evrE6o9eb/AW3NlEYKTiLq1zTSxncwXPMjtSdPkWQkzw3fgRyJPvUbULt+rOaHdEVCMFYuB2cEHAFeQtOhEJSKzw7vmN9zJzlHiq2ec4XwgLwc3VtY2ObYStDlUsP8piAW3Ei20Qf3563pyjbjCUUMH9RRVu3nD+dNCkK+bVivKSoLGQdZ6Qwlxyai8boE+bsEPgCfNZa1q9reECAuMEkTZoGrA0RHHifLgsbX0ohCYT7vN3QYDAhJxpz3ziooXpIjCndBMNqS4SqOxl6WsbS8I3spU7OgTBGt1O0qmFmzg8ZAett3r9r9MV+W8OG4hwPPYfTjeHiJciAKMUpx8fBDFq2/pViEypZByAV63nqYyhGUufxWny6DXb3hn20 9q0cB8Bx lh7Wyi2NHQrJjj5Y7mgVGpyfSV+C73AxwMMlKDF/Qyb6TiD8LJvU+Zxry1GOfMvahIt/AVMdXkU9WmphHx8RHv5PxcXyEWJK23p4xJhHaereqyv1rhgiePzT+p83zjAHQYpvIeb2aY1Gxo9EVPJptjFhlZlq73faF+7u+AthdMTHy00ttSFTWnD7foZsaOEvxg7lW+q3Ot2UDNntVpxkKHuiGKg3VytM46SgAzUD4u7uG1RW9qdLdr9t0TMVQKueiqK8jfF37zPf3LjHO0KqPW6gbtT09FmNuYdjNTkwiCoZyXCyv4HS03mnQTv6I42wr98PAYcKLZBS5W7rhr2MzENFte9ybZg/37muS8ot+UygDWT7uwxvE+VMZHQzc1S9jA4Gy9VSbC0taK2eHZZyamua7Gie0IyUoMRxVOtvMvJuLQoIaMDMkjN/h/mFG+Eh3rfneErv/KqBgcGO7I1UPI/HloT37T+YDHuGEA02vudWoMNVxzUTHznNvyj3aAJKWwTgbtLxgVhXvWAaEORh0bbUV/oxnccOjz9cozDiYdqxHzZidZ9EyUWt0BfmN7hZP6Qk44UPGznG3oXbEcr1j9BiK698Reu8w90A+JKLZtw1Ws1ZG7G9EI2ypg3+Q+VehNyUymsHGiX3wi06aFgwt+kO2XfADDFY3ZhXDBWILP0nfuRn97A6eHb83zF65Ye+QVCaoIc6zW5XaFw2kn+2qghe0vuzHAXQi6LoGOFeryTCg1zoS6RTcm9wy0nFY5KjTaFrVJbqKCj1cG7ShcdDfZ/WTqJ8+d3qN8ol90TDY2u2G2ct/czr6DJFr8+0ogDl0diXfrZbQhskVg854yxfOOPkoAF1EHp53wT3veUtHNXfFdQKS63FBV21cTryLawz29j3s8LMwvr/lfidlSUzHU2YF26dL4rrnrKnC X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Change the page_frag_cache allocator to use multipage folios rather than groups of pages. This reduces page_frag_free to just a folio_put() or put_page(). Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Jeroen de Borst cc: Catherine Sullivan cc: Shailend Chand cc: Felix Fietkau cc: John Crispin cc: Sean Wang cc: Mark Lee cc: Lorenzo Bianconi cc: Matthias Brugger cc: AngeloGioacchino Del Regno cc: Keith Busch cc: Jens Axboe cc: Christoph Hellwig cc: Sagi Grimberg cc: Chaitanya Kulkarni cc: Andrew Morton cc: Matthew Wilcox cc: netdev@vger.kernel.org cc: linux-arm-kernel@lists.infradead.org cc: linux-mediatek@lists.infradead.org cc: linux-nvme@lists.infradead.org cc: linux-mm@kvack.org --- Notes: ver #6) - Removed a couple of leftover page pointer declarations. drivers/net/ethernet/google/gve/gve_main.c | 11 +-- drivers/net/ethernet/mediatek/mtk_wed_wo.c | 17 +--- drivers/nvme/host/tcp.c | 8 +- drivers/nvme/target/tcp.c | 5 +- include/linux/gfp.h | 2 + include/linux/mm_types.h | 13 +-- mm/page_frag_alloc.c | 102 +++++++++++---------- 7 files changed, 67 insertions(+), 91 deletions(-) diff --git a/drivers/net/ethernet/google/gve/gve_main.c b/drivers/net/ethernet/google/gve/gve_main.c index 57ce74315eba..b2fc1a3e6340 100644 --- a/drivers/net/ethernet/google/gve/gve_main.c +++ b/drivers/net/ethernet/google/gve/gve_main.c @@ -1263,17 +1263,10 @@ static void gve_unreg_xdp_info(struct gve_priv *priv) static void gve_drain_page_cache(struct gve_priv *priv) { - struct page_frag_cache *nc; int i; - for (i = 0; i < priv->rx_cfg.num_queues; i++) { - nc = &priv->rx[i].page_cache; - if (nc->va) { - __page_frag_cache_drain(virt_to_page(nc->va), - nc->pagecnt_bias); - nc->va = NULL; - } - } + for (i = 0; i < priv->rx_cfg.num_queues; i++) + page_frag_cache_clear(&priv->rx[i].page_cache); } static int gve_open(struct net_device *dev) diff --git a/drivers/net/ethernet/mediatek/mtk_wed_wo.c b/drivers/net/ethernet/mediatek/mtk_wed_wo.c index 69fba29055e9..d90fea2c7d04 100644 --- a/drivers/net/ethernet/mediatek/mtk_wed_wo.c +++ b/drivers/net/ethernet/mediatek/mtk_wed_wo.c @@ -286,7 +286,6 @@ mtk_wed_wo_queue_free(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) static void mtk_wed_wo_queue_tx_clean(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) { - struct page *page; int i; for (i = 0; i < q->n_desc; i++) { @@ -298,19 +297,12 @@ mtk_wed_wo_queue_tx_clean(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) entry->buf = NULL; } - if (!q->cache.va) - return; - - page = virt_to_page(q->cache.va); - __page_frag_cache_drain(page, q->cache.pagecnt_bias); - memset(&q->cache, 0, sizeof(q->cache)); + page_frag_cache_clear(&q->cache); } static void mtk_wed_wo_queue_rx_clean(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) { - struct page *page; - for (;;) { void *buf = mtk_wed_wo_dequeue(wo, q, NULL, true); @@ -320,12 +312,7 @@ mtk_wed_wo_queue_rx_clean(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) skb_free_frag(buf); } - if (!q->cache.va) - return; - - page = virt_to_page(q->cache.va); - __page_frag_cache_drain(page, q->cache.pagecnt_bias); - memset(&q->cache, 0, sizeof(q->cache)); + page_frag_cache_clear(&q->cache); } static void diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index 42c0598c31f2..05629e83b41d 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -1312,7 +1312,6 @@ static int nvme_tcp_alloc_async_req(struct nvme_tcp_ctrl *ctrl) static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid) { - struct page *page; struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl); struct nvme_tcp_queue *queue = &ctrl->queues[qid]; unsigned int noreclaim_flag; @@ -1323,12 +1322,7 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid) if (queue->hdr_digest || queue->data_digest) nvme_tcp_free_crypto(queue); - if (queue->pf_cache.va) { - page = virt_to_head_page(queue->pf_cache.va); - __page_frag_cache_drain(page, queue->pf_cache.pagecnt_bias); - queue->pf_cache.va = NULL; - } - + page_frag_cache_clear(&queue->pf_cache); noreclaim_flag = memalloc_noreclaim_save(); sock_release(queue->sock); memalloc_noreclaim_restore(noreclaim_flag); diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c index 66e8f9fd0ca7..ae871c31cf00 100644 --- a/drivers/nvme/target/tcp.c +++ b/drivers/nvme/target/tcp.c @@ -1438,7 +1438,6 @@ static void nvmet_tcp_free_cmd_data_in_buffers(struct nvmet_tcp_queue *queue) static void nvmet_tcp_release_queue_work(struct work_struct *w) { - struct page *page; struct nvmet_tcp_queue *queue = container_of(w, struct nvmet_tcp_queue, release_work); @@ -1460,9 +1459,7 @@ static void nvmet_tcp_release_queue_work(struct work_struct *w) if (queue->hdr_digest || queue->data_digest) nvmet_tcp_free_crypto(queue); ida_free(&nvmet_tcp_queue_ida, queue->idx); - - page = virt_to_head_page(queue->pf_cache.va); - __page_frag_cache_drain(page, queue->pf_cache.pagecnt_bias); + page_frag_cache_clear(&queue->pf_cache); kfree(queue); } diff --git a/include/linux/gfp.h b/include/linux/gfp.h index 65a78773dcca..9f77ba6af361 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -314,6 +314,8 @@ static inline void *page_frag_alloc(struct page_frag_cache *nc, return page_frag_alloc_align(nc, fragsz, gfp_mask, ~0u); } +void page_frag_cache_clear(struct page_frag_cache *nc); + extern void page_frag_free(void *addr); #define __free_page(page) __free_pages((page), 0) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 0722859c3647..49a70b3f44a9 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -420,18 +420,13 @@ static inline void *folio_get_private(struct folio *folio) } struct page_frag_cache { - void * va; -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - __u16 offset; - __u16 size; -#else - __u32 offset; -#endif + struct folio *folio; + unsigned int offset; /* we maintain a pagecount bias, so that we dont dirty cache line * containing page->_refcount every time we allocate a fragment. */ - unsigned int pagecnt_bias; - bool pfmemalloc; + unsigned int pagecnt_bias; + bool pfmemalloc; }; typedef unsigned long vm_flags_t; diff --git a/mm/page_frag_alloc.c b/mm/page_frag_alloc.c index bee95824ef8f..ac4cf1eac8ea 100644 --- a/mm/page_frag_alloc.c +++ b/mm/page_frag_alloc.c @@ -16,88 +16,96 @@ #include #include -static struct page *__page_frag_cache_refill(struct page_frag_cache *nc, - gfp_t gfp_mask) +/* + * Allocate a new folio for the frag cache. + */ +static struct folio *page_frag_cache_refill(struct page_frag_cache *nc, + gfp_t gfp_mask) { - struct page *page = NULL; + struct folio *folio = NULL; gfp_t gfp = gfp_mask; #if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - gfp_mask |= __GFP_COMP | __GFP_NOWARN | __GFP_NORETRY | - __GFP_NOMEMALLOC; - page = alloc_pages_node(NUMA_NO_NODE, gfp_mask, - PAGE_FRAG_CACHE_MAX_ORDER); - nc->size = page ? PAGE_FRAG_CACHE_MAX_SIZE : PAGE_SIZE; + gfp_mask |= __GFP_NOWARN | __GFP_NORETRY | __GFP_NOMEMALLOC; + folio = folio_alloc(gfp_mask, PAGE_FRAG_CACHE_MAX_ORDER); #endif - if (unlikely(!page)) - page = alloc_pages_node(NUMA_NO_NODE, gfp, 0); - - nc->va = page ? page_address(page) : NULL; + if (unlikely(!folio)) + folio = folio_alloc(gfp, 0); - return page; + if (folio) + nc->folio = folio; + return folio; } void __page_frag_cache_drain(struct page *page, unsigned int count) { - VM_BUG_ON_PAGE(page_ref_count(page) == 0, page); + struct folio *folio = page_folio(page); - if (page_ref_sub_and_test(page, count - 1)) - __free_pages(page, compound_order(page)); + VM_BUG_ON_FOLIO(folio_ref_count(folio) == 0, folio); + + folio_put_refs(folio, count); } EXPORT_SYMBOL(__page_frag_cache_drain); +void page_frag_cache_clear(struct page_frag_cache *nc) +{ + struct folio *folio = nc->folio; + + if (folio) { + VM_BUG_ON_FOLIO(folio_ref_count(folio) == 0, folio); + folio_put_refs(folio, nc->pagecnt_bias); + nc->folio = NULL; + } +} +EXPORT_SYMBOL(page_frag_cache_clear); + void *page_frag_alloc_align(struct page_frag_cache *nc, unsigned int fragsz, gfp_t gfp_mask, unsigned int align_mask) { - unsigned int size = PAGE_SIZE; - struct page *page; - int offset; + struct folio *folio = nc->folio; + size_t offset; - if (unlikely(!nc->va)) { + if (unlikely(!folio)) { refill: - page = __page_frag_cache_refill(nc, gfp_mask); - if (!page) + folio = page_frag_cache_refill(nc, gfp_mask); + if (!folio) return NULL; -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - /* if size can vary use size else just use PAGE_SIZE */ - size = nc->size; -#endif /* Even if we own the page, we do not use atomic_set(). * This would break get_page_unless_zero() users. */ - page_ref_add(page, PAGE_FRAG_CACHE_MAX_SIZE); + folio_ref_add(folio, PAGE_FRAG_CACHE_MAX_SIZE); /* reset page count bias and offset to start of new frag */ - nc->pfmemalloc = page_is_pfmemalloc(page); + nc->pfmemalloc = folio_is_pfmemalloc(folio); nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; - nc->offset = size; + nc->offset = folio_size(folio); } - offset = nc->offset - fragsz; - if (unlikely(offset < 0)) { - page = virt_to_page(nc->va); - - if (page_ref_count(page) != nc->pagecnt_bias) + offset = nc->offset; + if (unlikely(fragsz > offset)) { + /* Reuse the folio if everyone we gave it to has finished with + * it. + */ + if (!folio_ref_sub_and_test(folio, nc->pagecnt_bias)) { + nc->folio = NULL; goto refill; + } + if (unlikely(nc->pfmemalloc)) { - page_ref_sub(page, nc->pagecnt_bias - 1); - __free_pages(page, compound_order(page)); + __folio_put(folio); + nc->folio = NULL; goto refill; } -#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - /* if size can vary use size else just use PAGE_SIZE */ - size = nc->size; -#endif /* OK, page count is 0, we can safely set it */ - set_page_count(page, PAGE_FRAG_CACHE_MAX_SIZE + 1); + folio_set_count(folio, PAGE_FRAG_CACHE_MAX_SIZE + 1); /* reset page count bias and offset to start of new frag */ nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; - offset = size - fragsz; - if (unlikely(offset < 0)) { + offset = folio_size(folio); + if (unlikely(fragsz > offset)) { /* * The caller is trying to allocate a fragment * with fragsz > PAGE_SIZE but the cache isn't big @@ -107,15 +115,17 @@ void *page_frag_alloc_align(struct page_frag_cache *nc, * it could make memory pressure worse * so we simply return NULL here. */ + nc->offset = offset; return NULL; } } nc->pagecnt_bias--; + offset -= fragsz; offset &= align_mask; nc->offset = offset; - return nc->va + offset; + return folio_address(folio) + offset; } EXPORT_SYMBOL(page_frag_alloc_align); @@ -124,8 +134,6 @@ EXPORT_SYMBOL(page_frag_alloc_align); */ void page_frag_free(void *addr) { - struct page *page = virt_to_head_page(addr); - - __free_pages(page, compound_order(page)); + folio_put(virt_to_folio(addr)); } EXPORT_SYMBOL(page_frag_free); From patchwork Tue Apr 11 16:08:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13207810 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76381C77B6F for ; Tue, 11 Apr 2023 16:09:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 15ADE900007; Tue, 11 Apr 2023 12:09:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0E42B900002; Tue, 11 Apr 2023 12:09:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E2B2A900007; Tue, 11 Apr 2023 12:09:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id CF1F5900002 for ; Tue, 11 Apr 2023 12:09:30 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 9C28780EB6 for ; Tue, 11 Apr 2023 16:09:30 +0000 (UTC) X-FDA: 80669595300.23.A6A0EB0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf03.hostedemail.com (Postfix) with ESMTP id C195D2002B for ; Tue, 11 Apr 2023 16:09:28 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AKFojqa9; spf=pass (imf03.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681229368; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vG+hbfPhdH3XoiTTIP3GEN9NixBWys0PYm40TMzeMsw=; b=E8AGc8yHQoveBC5r34o4MlCQTk1VQNYQFydF5RPvBgymcyfizE0hcqF7/is1jOCgZ2QX3E e13FbIRhH1EYcJoPETtGmYdlxs1ZE4/4PgFg8W5LX90eKTQo7xPhTKJ6LdHDrb3dNYxg46 BJDFi8Id4IHn0Tz2D6NK9N/ciV7ADqo= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AKFojqa9; spf=pass (imf03.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681229368; a=rsa-sha256; cv=none; b=ijg8Vwcip2fpGtpQcrUCDwGsbdLitGIzM7BaNAu7HR1ArwuzBmyuyCbfF3OfPgINRVzQuW w6Wx9f0SozkSZPmB91abCb8uwtMlhQMS8C9hiQtOZhhOQjsCm0QDevv1I3qDjOqcs0EYRI zdtMn5+fXIO0SAEz0HjS+3ZP0ejQzDI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681229367; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vG+hbfPhdH3XoiTTIP3GEN9NixBWys0PYm40TMzeMsw=; b=AKFojqa95wQFOTc1bpSLXBA2ocHqEGlaIkPVEuxS/Y24i66RD6vVKgFvIVwTzMUox68E+N 9XaJuCwY5Jus3WbO7PYYZBVtiA183uSinLWe9MPZc9+V1J5GZ0LpRnN46PpgibIkWKYy0a RBPTZPQwwXMG0O5nSoJGzuHZlaXrFi4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-616-4z8isJvePmeudG4vpT2eFg-1; Tue, 11 Apr 2023 12:09:25 -0400 X-MC-Unique: 4z8isJvePmeudG4vpT2eFg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8241487B2A0; Tue, 11 Apr 2023 16:09:23 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 611BB40C83A9; Tue, 11 Apr 2023 16:09:19 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Jeroen de Borst , Catherine Sullivan , Shailend Chand , Felix Fietkau , John Crispin , Sean Wang , Mark Lee , Lorenzo Bianconi , Matthias Brugger , AngeloGioacchino Del Regno , Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-mediatek@lists.infradead.org, linux-nvme@lists.infradead.org Subject: [PATCH net-next v6 04/18] mm: Make the page_frag_cache allocator use per-cpu Date: Tue, 11 Apr 2023 17:08:48 +0100 Message-Id: <20230411160902.4134381-5-dhowells@redhat.com> In-Reply-To: <20230411160902.4134381-1-dhowells@redhat.com> References: <20230411160902.4134381-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: C195D2002B X-Stat-Signature: m5xs6qg3xrtyboxdc13zfyuonb7dumss X-Rspam-User: X-HE-Tag: 1681229368-32334 X-HE-Meta: U2FsdGVkX1/lmhIfwY8py9877aNnCkYN/3GZ6fHV0x6IgBiylCdZz8QDdT2a+8wadKNEml5KCOUFRtE4NUWSLfKY7A0Nmmnj8+/bGB4oXbL7flK3GCrxj2g3bi2gd/JDVWPwofjFJoRQcY7vK0FgUFpE0SGhtdgTtCHLDUb95rkmoXXD7GiBjtSHO4r98TFCGpb06C2IfH/aMjMrF9807ZmblToCZmjtIQ0O80ydQBZW0LEJbQLIhuOaz6jwaLwR0ruhWYb89k0S3msjrir9NCxK+n35iJPcWGuQxEsSbAQhBP/VhX+ogWpw88ojkhWhOKw+jIY3J4aKuC8ddfgpedDel2qZV9lGk0QcSAQ8WE1a7bvmiz3QuVdlGB7fbNMrALN1GlOQbF0Y8xOJBENa7ekhvO8uaMuUL5DuyCP3E3S/kqaE6fhjOaV3d8pWtSIWHs5nNEi290p/tfAukdFuZjYnVkoNhWUU2GNsB2SNK5pWXJWusWThrwj8SF+hF1kfa3ZnJ7xyQFfDsd0+n/9wUJMEcZYZQokjrWAizPFtP50jeQTUyQajsHyNpxo+ytJTOtjziBzYHoH6aLxlTnvtizvhCTRR/uIqP6eexWRz5cMHrSMMd9EZ0jmACksjzdydpt/FUF9VSNH2ag2MJfNeF23xRNnOe3sOaWc5eFGO4A3HsJcmFIKTeimVufYqaooIhhKnbY5QX6yB0ni9l2RszNnuIMuFr9cioBcU1pasnRW2sz5lAijW7s2OUPP1c7avfoNj1uRixpm4lDON0qirf6BWmmvphue1bR1U99mdvb5d06wuNaVYuuCzcS/ckuyMTHLw1F3PEXK6NQdFjZo84mynIosY4x8LChYa50WPCUuii/8L+ATw0696ZGpSUMVDTJCFsmgNlawLAh/fa3r/3eU54iYCx/1atCrvsovCj/ZELRfX/oS+EYCPODLN/o9IkzdcUjP4/BXZeZwuZE+ g8WZxEfB LwppCWs/CvlvCc8YzqH9uMI9A9BjBACiHnGCR12w5i0A8xaytd+WHM4mOud2zT0IhVrlrbT1e7K/fqAYaWgk63CmVWfn7miBXfaNtew2m+CjkUZ3rc9p89LzdRyQT9etaVXJCeHXzUP5u3yYVLZu+L6dSy41/Y/7wwHXJauNu5pcVIm+AWecmNwtLLDVoKbKYA3X4UtUXtU6RVZTng7vGKYAurG4OKdr4VbTFXZDHA78whpvpmCwVxJF/SB7/b4Igh9GB402PY9452Ds7f+glHSUh1fjiSbSi8SK2o0YRZDqosidiIF7fgxvg++prtsHvJTPaCJ6yCFbMThm0p0h/TfpEok2c5z9IVHOeQFpWqsEIpDtzL4PclOVlW1M1dchcNa5R7k371DBX+wNNc7MGLZJmALgJxIkmEIq8FiVB6yGJJHDfqeZy9/+xIHL+z1akaXlkRUgEeebssMz5E1nWXgL1oCsDyNIqFy/uJk/CJ0QdVY9WTiPtGZ3w4QNwGKaPT4RjlOHpTel8TBzHgItM0CBBebpj0+ipQDz96xJh5SlNR6AY6J8Rn5+Il33O2Ss2DcH59AZXoxDlAFy6wD1rMO8qrb1poSTT9Oq+iUsO46O+Hrn+hQZAqliiu/uC1kprhXVM/ELx0idGIkUPqnAQ0YSnimxSHg20hZ8u/gernZnnkjnTTuEov90yu6RvQ5n+m52ScS9kncBl4GOxKl7LUMyPuUzejgm6RZ5BwFJZzWc+oGKC8kQ7waLpsBv4OMrvpXwUYWZsPsfMqyUpMb4NuuiqqV+MinBS/4pDmNhy19ChFIt71pAz1PiR4la/8nj6Gh73imrWTyOg2+P1DoGNINoSTc3ls3jhDM0lUHTvDHw509JqLzEFhC52k5GFjCDG1mquaNaEFThVoYE3iU+UyVZqZ5iugSEVtBrz X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make the page_frag_cache allocator have a separate allocation bucket for each cpu to avoid racing. This means that no lock is required, other than preempt disablement, to allocate from it, though if a softirq wants to access it, then softirq disablement will need to be added. Make the NVMe, mediatek and GVE drivers pass in NULL to page_frag_cache() and use the default allocation buckets rather than defining their own. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Jeroen de Borst cc: Catherine Sullivan cc: Shailend Chand cc: Felix Fietkau cc: John Crispin cc: Sean Wang cc: Mark Lee cc: Lorenzo Bianconi cc: Matthias Brugger cc: AngeloGioacchino Del Regno cc: Keith Busch cc: Jens Axboe cc: Christoph Hellwig cc: Sagi Grimberg cc: Chaitanya Kulkarni cc: Andrew Morton cc: Matthew Wilcox cc: netdev@vger.kernel.org cc: linux-arm-kernel@lists.infradead.org cc: linux-mediatek@lists.infradead.org cc: linux-nvme@lists.infradead.org cc: linux-mm@kvack.org --- drivers/net/ethernet/google/gve/gve.h | 1 - drivers/net/ethernet/google/gve/gve_main.c | 9 - drivers/net/ethernet/google/gve/gve_rx.c | 2 +- drivers/net/ethernet/mediatek/mtk_wed_wo.c | 6 +- drivers/net/ethernet/mediatek/mtk_wed_wo.h | 2 - drivers/nvme/host/tcp.c | 13 +- drivers/nvme/target/tcp.c | 19 +- include/linux/gfp.h | 19 +- mm/page_frag_alloc.c | 196 ++++++++++++++------- net/core/skbuff.c | 32 ++-- 10 files changed, 165 insertions(+), 134 deletions(-) diff --git a/drivers/net/ethernet/google/gve/gve.h b/drivers/net/ethernet/google/gve/gve.h index e214b51d3c8b..5864d723210a 100644 --- a/drivers/net/ethernet/google/gve/gve.h +++ b/drivers/net/ethernet/google/gve/gve.h @@ -250,7 +250,6 @@ struct gve_rx_ring { struct xdp_rxq_info xdp_rxq; struct xdp_rxq_info xsk_rxq; struct xsk_buff_pool *xsk_pool; - struct page_frag_cache page_cache; /* Page cache to allocate XDP frames */ }; /* A TX desc ring entry */ diff --git a/drivers/net/ethernet/google/gve/gve_main.c b/drivers/net/ethernet/google/gve/gve_main.c index b2fc1a3e6340..6c835038a8eb 100644 --- a/drivers/net/ethernet/google/gve/gve_main.c +++ b/drivers/net/ethernet/google/gve/gve_main.c @@ -1261,14 +1261,6 @@ static void gve_unreg_xdp_info(struct gve_priv *priv) } } -static void gve_drain_page_cache(struct gve_priv *priv) -{ - int i; - - for (i = 0; i < priv->rx_cfg.num_queues; i++) - page_frag_cache_clear(&priv->rx[i].page_cache); -} - static int gve_open(struct net_device *dev) { struct gve_priv *priv = netdev_priv(dev); @@ -1352,7 +1344,6 @@ static int gve_close(struct net_device *dev) netif_carrier_off(dev); if (gve_get_device_rings_ok(priv)) { gve_turndown(priv); - gve_drain_page_cache(priv); err = gve_destroy_rings(priv); if (err) goto err; diff --git a/drivers/net/ethernet/google/gve/gve_rx.c b/drivers/net/ethernet/google/gve/gve_rx.c index d1da7413dc4d..7ae8377c394f 100644 --- a/drivers/net/ethernet/google/gve/gve_rx.c +++ b/drivers/net/ethernet/google/gve/gve_rx.c @@ -634,7 +634,7 @@ static int gve_xdp_redirect(struct net_device *dev, struct gve_rx_ring *rx, total_len = headroom + SKB_DATA_ALIGN(len) + SKB_DATA_ALIGN(sizeof(struct skb_shared_info)); - frame = page_frag_alloc(&rx->page_cache, total_len, GFP_ATOMIC); + frame = page_frag_alloc(NULL, total_len, GFP_ATOMIC); if (!frame) { u64_stats_update_begin(&rx->statss); rx->xdp_alloc_fails++; diff --git a/drivers/net/ethernet/mediatek/mtk_wed_wo.c b/drivers/net/ethernet/mediatek/mtk_wed_wo.c index d90fea2c7d04..859f34447f2f 100644 --- a/drivers/net/ethernet/mediatek/mtk_wed_wo.c +++ b/drivers/net/ethernet/mediatek/mtk_wed_wo.c @@ -143,7 +143,7 @@ mtk_wed_wo_queue_refill(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q, dma_addr_t addr; void *buf; - buf = page_frag_alloc(&q->cache, q->buf_size, GFP_ATOMIC); + buf = page_frag_alloc(NULL, q->buf_size, GFP_ATOMIC); if (!buf) break; @@ -296,8 +296,6 @@ mtk_wed_wo_queue_tx_clean(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) skb_free_frag(entry->buf); entry->buf = NULL; } - - page_frag_cache_clear(&q->cache); } static void @@ -311,8 +309,6 @@ mtk_wed_wo_queue_rx_clean(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) skb_free_frag(buf); } - - page_frag_cache_clear(&q->cache); } static void diff --git a/drivers/net/ethernet/mediatek/mtk_wed_wo.h b/drivers/net/ethernet/mediatek/mtk_wed_wo.h index dbcf42ce9173..6f940db67fb8 100644 --- a/drivers/net/ethernet/mediatek/mtk_wed_wo.h +++ b/drivers/net/ethernet/mediatek/mtk_wed_wo.h @@ -210,8 +210,6 @@ struct mtk_wed_wo_queue_entry { struct mtk_wed_wo_queue { struct mtk_wed_wo_queue_regs regs; - struct page_frag_cache cache; - struct mtk_wed_wo_queue_desc *desc; dma_addr_t desc_dma; diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index 05629e83b41d..5a92236db92a 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -147,8 +147,6 @@ struct nvme_tcp_queue { __le32 exp_ddgst; __le32 recv_ddgst; - struct page_frag_cache pf_cache; - void (*state_change)(struct sock *); void (*data_ready)(struct sock *); void (*write_space)(struct sock *); @@ -482,9 +480,8 @@ static int nvme_tcp_init_request(struct blk_mq_tag_set *set, struct nvme_tcp_queue *queue = &ctrl->queues[queue_idx]; u8 hdgst = nvme_tcp_hdgst_len(queue); - req->pdu = page_frag_alloc(&queue->pf_cache, - sizeof(struct nvme_tcp_cmd_pdu) + hdgst, - GFP_KERNEL | __GFP_ZERO); + req->pdu = page_frag_alloc(NULL, sizeof(struct nvme_tcp_cmd_pdu) + hdgst, + GFP_KERNEL | __GFP_ZERO); if (!req->pdu) return -ENOMEM; @@ -1300,9 +1297,8 @@ static int nvme_tcp_alloc_async_req(struct nvme_tcp_ctrl *ctrl) struct nvme_tcp_request *async = &ctrl->async_req; u8 hdgst = nvme_tcp_hdgst_len(queue); - async->pdu = page_frag_alloc(&queue->pf_cache, - sizeof(struct nvme_tcp_cmd_pdu) + hdgst, - GFP_KERNEL | __GFP_ZERO); + async->pdu = page_frag_alloc(NULL, sizeof(struct nvme_tcp_cmd_pdu) + hdgst, + GFP_KERNEL | __GFP_ZERO); if (!async->pdu) return -ENOMEM; @@ -1322,7 +1318,6 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid) if (queue->hdr_digest || queue->data_digest) nvme_tcp_free_crypto(queue); - page_frag_cache_clear(&queue->pf_cache); noreclaim_flag = memalloc_noreclaim_save(); sock_release(queue->sock); memalloc_noreclaim_restore(noreclaim_flag); diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c index ae871c31cf00..d6cc557cc539 100644 --- a/drivers/nvme/target/tcp.c +++ b/drivers/nvme/target/tcp.c @@ -143,8 +143,6 @@ struct nvmet_tcp_queue { struct nvmet_tcp_cmd connect; - struct page_frag_cache pf_cache; - void (*data_ready)(struct sock *); void (*state_change)(struct sock *); void (*write_space)(struct sock *); @@ -1312,25 +1310,25 @@ static int nvmet_tcp_alloc_cmd(struct nvmet_tcp_queue *queue, c->queue = queue; c->req.port = queue->port->nport; - c->cmd_pdu = page_frag_alloc(&queue->pf_cache, - sizeof(*c->cmd_pdu) + hdgst, GFP_KERNEL | __GFP_ZERO); + c->cmd_pdu = page_frag_alloc(NULL, sizeof(*c->cmd_pdu) + hdgst, + GFP_KERNEL | __GFP_ZERO); if (!c->cmd_pdu) return -ENOMEM; c->req.cmd = &c->cmd_pdu->cmd; - c->rsp_pdu = page_frag_alloc(&queue->pf_cache, - sizeof(*c->rsp_pdu) + hdgst, GFP_KERNEL | __GFP_ZERO); + c->rsp_pdu = page_frag_alloc(NULL, sizeof(*c->rsp_pdu) + hdgst, + GFP_KERNEL | __GFP_ZERO); if (!c->rsp_pdu) goto out_free_cmd; c->req.cqe = &c->rsp_pdu->cqe; - c->data_pdu = page_frag_alloc(&queue->pf_cache, - sizeof(*c->data_pdu) + hdgst, GFP_KERNEL | __GFP_ZERO); + c->data_pdu = page_frag_alloc(NULL, sizeof(*c->data_pdu) + hdgst, + GFP_KERNEL | __GFP_ZERO); if (!c->data_pdu) goto out_free_rsp; - c->r2t_pdu = page_frag_alloc(&queue->pf_cache, - sizeof(*c->r2t_pdu) + hdgst, GFP_KERNEL | __GFP_ZERO); + c->r2t_pdu = page_frag_alloc(NULL, sizeof(*c->r2t_pdu) + hdgst, + GFP_KERNEL | __GFP_ZERO); if (!c->r2t_pdu) goto out_free_data; @@ -1459,7 +1457,6 @@ static void nvmet_tcp_release_queue_work(struct work_struct *w) if (queue->hdr_digest || queue->data_digest) nvmet_tcp_free_crypto(queue); ida_free(&nvmet_tcp_queue_ida, queue->idx); - page_frag_cache_clear(&queue->pf_cache); kfree(queue); } diff --git a/include/linux/gfp.h b/include/linux/gfp.h index 9f77ba6af361..b208ca315882 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -304,18 +304,19 @@ extern void free_pages(unsigned long addr, unsigned int order); struct page_frag_cache; extern void __page_frag_cache_drain(struct page *page, unsigned int count); -extern void *page_frag_alloc_align(struct page_frag_cache *nc, - unsigned int fragsz, gfp_t gfp_mask, - unsigned int align_mask); - -static inline void *page_frag_alloc(struct page_frag_cache *nc, - unsigned int fragsz, gfp_t gfp_mask) +extern void *page_frag_alloc_align(struct page_frag_cache __percpu *frag_cache, + size_t fragsz, gfp_t gfp, + unsigned long align_mask); +extern void *page_frag_memdup(struct page_frag_cache __percpu *frag_cache, + const void *p, size_t fragsz, gfp_t gfp, + unsigned long align_mask); + +static inline void *page_frag_alloc(struct page_frag_cache __percpu *frag_cache, + size_t fragsz, gfp_t gfp) { - return page_frag_alloc_align(nc, fragsz, gfp_mask, ~0u); + return page_frag_alloc_align(frag_cache, fragsz, gfp, ULONG_MAX); } -void page_frag_cache_clear(struct page_frag_cache *nc); - extern void page_frag_free(void *addr); #define __free_page(page) __free_pages((page), 0) diff --git a/mm/page_frag_alloc.c b/mm/page_frag_alloc.c index ac4cf1eac8ea..7844398afe26 100644 --- a/mm/page_frag_alloc.c +++ b/mm/page_frag_alloc.c @@ -16,25 +16,23 @@ #include #include +static DEFINE_PER_CPU(struct page_frag_cache, page_frag_default_allocator); + /* * Allocate a new folio for the frag cache. */ -static struct folio *page_frag_cache_refill(struct page_frag_cache *nc, - gfp_t gfp_mask) +static struct folio *page_frag_cache_refill(gfp_t gfp) { - struct folio *folio = NULL; - gfp_t gfp = gfp_mask; + struct folio *folio; #if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) - gfp_mask |= __GFP_NOWARN | __GFP_NORETRY | __GFP_NOMEMALLOC; - folio = folio_alloc(gfp_mask, PAGE_FRAG_CACHE_MAX_ORDER); + folio = folio_alloc(gfp | __GFP_NOWARN | __GFP_NORETRY | __GFP_NOMEMALLOC, + PAGE_FRAG_CACHE_MAX_ORDER); + if (folio) + return folio; #endif - if (unlikely(!folio)) - folio = folio_alloc(gfp, 0); - if (folio) - nc->folio = folio; - return folio; + return folio_alloc(gfp, 0); } void __page_frag_cache_drain(struct page *page, unsigned int count) @@ -47,55 +45,68 @@ void __page_frag_cache_drain(struct page *page, unsigned int count) } EXPORT_SYMBOL(__page_frag_cache_drain); -void page_frag_cache_clear(struct page_frag_cache *nc) -{ - struct folio *folio = nc->folio; - - if (folio) { - VM_BUG_ON_FOLIO(folio_ref_count(folio) == 0, folio); - folio_put_refs(folio, nc->pagecnt_bias); - nc->folio = NULL; - } -} -EXPORT_SYMBOL(page_frag_cache_clear); - -void *page_frag_alloc_align(struct page_frag_cache *nc, - unsigned int fragsz, gfp_t gfp_mask, - unsigned int align_mask) +/** + * page_frag_alloc_align - Allocate some memory for use in zerocopy + * @frag_cache: The frag cache to use (or NULL for the default) + * @fragsz: The size of the fragment desired + * @gfp: Allocation flags under which to make an allocation + * @align_mask: The required alignment + * + * Allocate some memory for use with zerocopy where protocol bits have to be + * mixed in with spliced/zerocopied data. Unlike memory allocated from the + * slab, this memory's lifetime is purely dependent on the folio's refcount. + * + * The way it works is that a folio is allocated and fragments are broken off + * sequentially and returned to the caller with a ref until the folio no longer + * has enough spare space - at which point the allocator's ref is dropped and a + * new folio is allocated. The folio remains in existence until the last ref + * held by, say, an sk_buff is discarded and then the page is returned to the + * page allocator. + * + * Returns a pointer to the memory on success and -ENOMEM on allocation + * failure. + * + * The allocated memory should be disposed of with folio_put(). + */ +void *page_frag_alloc_align(struct page_frag_cache __percpu *frag_cache, + size_t fragsz, gfp_t gfp, unsigned long align_mask) { - struct folio *folio = nc->folio; + struct page_frag_cache *nc; + struct folio *folio, *spare = NULL; size_t offset; + void *p; - if (unlikely(!folio)) { -refill: - folio = page_frag_cache_refill(nc, gfp_mask); - if (!folio) - return NULL; - - /* Even if we own the page, we do not use atomic_set(). - * This would break get_page_unless_zero() users. - */ - folio_ref_add(folio, PAGE_FRAG_CACHE_MAX_SIZE); + if (!frag_cache) + frag_cache = &page_frag_default_allocator; + if (WARN_ON_ONCE(fragsz == 0)) + fragsz = 1; + align_mask &= ~3UL; - /* reset page count bias and offset to start of new frag */ - nc->pfmemalloc = folio_is_pfmemalloc(folio); - nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; - nc->offset = folio_size(folio); + nc = get_cpu_ptr(frag_cache); +reload: + folio = nc->folio; + offset = nc->offset; +try_again: + + /* Make the allocation if there's sufficient space. */ + if (fragsz <= offset) { + nc->pagecnt_bias--; + offset = (offset - fragsz) & align_mask; + nc->offset = offset; + p = folio_address(folio) + offset; + put_cpu_ptr(frag_cache); + if (spare) + folio_put(spare); + return p; } - offset = nc->offset; - if (unlikely(fragsz > offset)) { - /* Reuse the folio if everyone we gave it to has finished with - * it. - */ - if (!folio_ref_sub_and_test(folio, nc->pagecnt_bias)) { - nc->folio = NULL; + /* Insufficient space - see if we can refurbish the current folio. */ + if (folio) { + if (!folio_ref_sub_and_test(folio, nc->pagecnt_bias)) goto refill; - } if (unlikely(nc->pfmemalloc)) { __folio_put(folio); - nc->folio = NULL; goto refill; } @@ -105,27 +116,56 @@ void *page_frag_alloc_align(struct page_frag_cache *nc, /* reset page count bias and offset to start of new frag */ nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; offset = folio_size(folio); - if (unlikely(fragsz > offset)) { - /* - * The caller is trying to allocate a fragment - * with fragsz > PAGE_SIZE but the cache isn't big - * enough to satisfy the request, this may - * happen in low memory conditions. - * We don't release the cache page because - * it could make memory pressure worse - * so we simply return NULL here. - */ - nc->offset = offset; + if (unlikely(fragsz > offset)) + goto frag_too_big; + goto try_again; + } + +refill: + if (!spare) { + nc->folio = NULL; + put_cpu_ptr(frag_cache); + + spare = page_frag_cache_refill(gfp); + if (!spare) return NULL; - } + + nc = get_cpu_ptr(frag_cache); + /* We may now be on a different cpu and/or someone else may + * have refilled it + */ + nc->pfmemalloc = folio_is_pfmemalloc(spare); + if (nc->folio) + goto reload; } - nc->pagecnt_bias--; - offset -= fragsz; - offset &= align_mask; + nc->folio = spare; + folio = spare; + spare = NULL; + + /* Even if we own the page, we do not use atomic_set(). This would + * break get_page_unless_zero() users. + */ + folio_ref_add(folio, PAGE_FRAG_CACHE_MAX_SIZE); + + /* Reset page count bias and offset to start of new frag */ + nc->pagecnt_bias = PAGE_FRAG_CACHE_MAX_SIZE + 1; + offset = folio_size(folio); + goto try_again; + +frag_too_big: + /* + * The caller is trying to allocate a fragment with fragsz > PAGE_SIZE + * but the cache isn't big enough to satisfy the request, this may + * happen in low memory conditions. We don't release the cache page + * because it could make memory pressure worse so we simply return NULL + * here. + */ nc->offset = offset; - - return folio_address(folio) + offset; + put_cpu_ptr(frag_cache); + if (spare) + folio_put(spare); + return NULL; } EXPORT_SYMBOL(page_frag_alloc_align); @@ -137,3 +177,25 @@ void page_frag_free(void *addr) folio_put(virt_to_folio(addr)); } EXPORT_SYMBOL(page_frag_free); + +/** + * page_frag_memdup - Allocate a page fragment and duplicate some data into it + * @frag_cache: The frag cache to use (or NULL for the default) + * @fragsz: The amount of memory to copy (maximum 1/2 page). + * @p: The source data to copy + * @gfp: Allocation flags under which to make an allocation + * @align_mask: The required alignment + */ +void *page_frag_memdup(struct page_frag_cache __percpu *frag_cache, + const void *p, size_t fragsz, gfp_t gfp, + unsigned long align_mask) +{ + void *q; + + q = page_frag_alloc_align(frag_cache, fragsz, gfp, align_mask); + if (!q) + return q; + + return memcpy(q, p, fragsz); +} +EXPORT_SYMBOL(page_frag_memdup); diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 050a875d09c5..3d05ed64b606 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -222,13 +222,13 @@ static void *page_frag_alloc_1k(struct page_frag_1k *nc, gfp_t gfp_mask) #endif struct napi_alloc_cache { - struct page_frag_cache page; struct page_frag_1k page_small; unsigned int skb_count; void *skb_cache[NAPI_SKB_CACHE_SIZE]; }; static DEFINE_PER_CPU(struct page_frag_cache, netdev_alloc_cache); +static DEFINE_PER_CPU(struct page_frag_cache, napi_frag_cache); static DEFINE_PER_CPU(struct napi_alloc_cache, napi_alloc_cache); /* Double check that napi_get_frags() allocates skbs with @@ -250,11 +250,9 @@ void napi_get_frags_check(struct napi_struct *napi) void *__napi_alloc_frag_align(unsigned int fragsz, unsigned int align_mask) { - struct napi_alloc_cache *nc = this_cpu_ptr(&napi_alloc_cache); - fragsz = SKB_DATA_ALIGN(fragsz); - return page_frag_alloc_align(&nc->page, fragsz, GFP_ATOMIC, align_mask); + return page_frag_alloc_align(&napi_frag_cache, fragsz, GFP_ATOMIC, align_mask); } EXPORT_SYMBOL(__napi_alloc_frag_align); @@ -264,15 +262,12 @@ void *__netdev_alloc_frag_align(unsigned int fragsz, unsigned int align_mask) fragsz = SKB_DATA_ALIGN(fragsz); if (in_hardirq() || irqs_disabled()) { - struct page_frag_cache *nc = this_cpu_ptr(&netdev_alloc_cache); - - data = page_frag_alloc_align(nc, fragsz, GFP_ATOMIC, align_mask); + data = page_frag_alloc_align(&netdev_alloc_cache, + fragsz, GFP_ATOMIC, align_mask); } else { - struct napi_alloc_cache *nc; - local_bh_disable(); - nc = this_cpu_ptr(&napi_alloc_cache); - data = page_frag_alloc_align(&nc->page, fragsz, GFP_ATOMIC, align_mask); + data = page_frag_alloc_align(&napi_frag_cache, + fragsz, GFP_ATOMIC, align_mask); local_bh_enable(); } return data; @@ -652,7 +647,6 @@ EXPORT_SYMBOL(__alloc_skb); struct sk_buff *__netdev_alloc_skb(struct net_device *dev, unsigned int len, gfp_t gfp_mask) { - struct page_frag_cache *nc; struct sk_buff *skb; bool pfmemalloc; void *data; @@ -677,14 +671,12 @@ struct sk_buff *__netdev_alloc_skb(struct net_device *dev, unsigned int len, gfp_mask |= __GFP_MEMALLOC; if (in_hardirq() || irqs_disabled()) { - nc = this_cpu_ptr(&netdev_alloc_cache); - data = page_frag_alloc(nc, len, gfp_mask); - pfmemalloc = nc->pfmemalloc; + data = page_frag_alloc(&netdev_alloc_cache, len, gfp_mask); + pfmemalloc = folio_is_pfmemalloc(virt_to_folio(data)); } else { local_bh_disable(); - nc = this_cpu_ptr(&napi_alloc_cache.page); - data = page_frag_alloc(nc, len, gfp_mask); - pfmemalloc = nc->pfmemalloc; + data = page_frag_alloc(&napi_frag_cache, len, gfp_mask); + pfmemalloc = folio_is_pfmemalloc(virt_to_folio(data)); local_bh_enable(); } @@ -772,8 +764,8 @@ struct sk_buff *__napi_alloc_skb(struct napi_struct *napi, unsigned int len, } else { len = SKB_HEAD_ALIGN(len); - data = page_frag_alloc(&nc->page, len, gfp_mask); - pfmemalloc = nc->page.pfmemalloc; + data = page_frag_alloc(&napi_frag_cache, len, gfp_mask); + pfmemalloc = folio_is_pfmemalloc(virt_to_folio(data)); } if (unlikely(!data)) From patchwork Tue Apr 11 16:08:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13207811 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EDDF7C76196 for ; Tue, 11 Apr 2023 16:09:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 92504900004; Tue, 11 Apr 2023 12:09:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 88613900002; Tue, 11 Apr 2023 12:09:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6D7AA900004; Tue, 11 Apr 2023 12:09:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 5CCBE900002 for ; Tue, 11 Apr 2023 12:09:33 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 31CBA160E86 for ; Tue, 11 Apr 2023 16:09:33 +0000 (UTC) X-FDA: 80669595426.25.2967E58 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf08.hostedemail.com (Postfix) with ESMTP id 74E76160006 for ; Tue, 11 Apr 2023 16:09:31 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="gwP5/Y9P"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf08.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681229371; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yN2nDE+ENFPKsoCAxX3Et3Pc/YScpeBgJ5SXmU8JiBg=; b=Q2yjRBIweLTGRfLVqAUGRxVSoZJP0G1+MsvuGB7hqwFfEfLlVgWBUTsWyyIu3B8GUeGbPq pqkUqV1SPTN5awx6Ljb/Yh8Ab66KakA/XzaKX+I49uXc0wzVQjdG7fButuGfjmW0JVmTcz OrOvBID0KgUFlbJbXJSTif6MuWnh/UA= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="gwP5/Y9P"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf08.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681229371; a=rsa-sha256; cv=none; b=4MIXqxL9Et5vPB/yJQ3CURWpDsjS52gbZpWUMOooaTChEEcyY9PQyqEm4WF1jsh527MT1q i/JiGdYC/R3PwWFbdeL9ybl6W/fAjC83icrbLDjjwfmd1KoSe/LnwbzSDwTr78HjcNXQlT DgcwfBs9FffFzd9Gu24zk9fw9c2NKtg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681229370; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yN2nDE+ENFPKsoCAxX3Et3Pc/YScpeBgJ5SXmU8JiBg=; b=gwP5/Y9Pm0GmKlyAEIGhxQMpy7kZLjFcxKvgEet8iEh8dJOa6eKIC4p4YFufLCct+oYG3B Zv1931ZBWEpqW2PMc7A0TwSeKtOgp8pJOlsYz5lt9AyeWxoTnhJbgHmp/KK5E1WlEgWhdL ZS3MdD7rTiyiyM8wN09SLaP7u1jDJA4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-673-JgBL6_p-OsWYijzjp0KjTQ-1; Tue, 11 Apr 2023 12:09:27 -0400 X-MC-Unique: JgBL6_p-OsWYijzjp0KjTQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3E43A185A7AC; Tue, 11 Apr 2023 16:09:26 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2233A40BC797; Tue, 11 Apr 2023 16:09:24 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v6 05/18] net: Pass max frags into skb_append_pagefrags() Date: Tue, 11 Apr 2023 17:08:49 +0100 Message-Id: <20230411160902.4134381-6-dhowells@redhat.com> In-Reply-To: <20230411160902.4134381-1-dhowells@redhat.com> References: <20230411160902.4134381-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspamd-Queue-Id: 74E76160006 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: 81qfb5bu9repe6mohspgn44ffc453c6n X-HE-Tag: 1681229371-68390 X-HE-Meta: U2FsdGVkX18mIWRzz3aix/GO8rsw0+kXyJjWsEbVdwVVka99FIRQWEwyg785SHfOfaitJUJkAacYSz7rWYvqt9Vrl9QcYSWzH908tFV7htPRKizx4AYRN91HsmKkcDz9YreEdpk2DMxyJ3GCycHouXU9g8jC+KgOd5WLwYYulA0iv4MSdxC8GYD6CbXx6prCvQX7LyvaVr23UCGALml4gFPdaLRJAI0JJz1GzGP7dNdmQOJHdyLRon+LyrEnMAOK4PrFOSJLSgluhCe4WNvdfPuQ65camXfPGrA3varpgqKMEjAoR5ci803q4XsVrp0j1AAJMl9fPHgB8v2NQujan9lM8KfrtXGbO/7D9IXNx/MEJ0YRlyKsHAv2wX+kCsoIrVH0Cl+WDx6PsDXGzpj9Irvc72AUPsF/EhuTejSCWtA7yN/P3OZiKAcYXaBkqbVXJTLOixUHWoc0pt1vDH97o9jbhKOU6yCRZQj6j8APS29dJVgRglM3mqZEtdJTD7wPyoMcpGQG3Vy38hM51aKh4QFjXmvfQJzZ8lDrerLXscSUwFjjA4hElpqBRQ5Slds1u8ikygSc2iu6AB/Dytdy7GwJpSjnzR85OIcJGlAMK2Exygl5Z00q40DdIFUkU/R7SLHght9oPfbvqA5dmKCD+drNcli26u79A8h6yiIGg7W+rZ6fZSPdThxNyqsNKyOuK7pE/O9UlLe/3D/g4iHo+oP4IoJUKr6tl9JzPg3F1eoJVstxE5y/q0690g5U/Br2YZ4dEQ3++VO8DY6t+vbC0pYFwGQcpSMRAU2T0VsEqRvDLkJrHw9r966iM5RWehLMMYqmsnpNnhsi5TELR0Uo92PMGk58aNEfu6jbNN+/gQk7NJ+VtzHgsAz3xRs5QZx9GAPIl9l8ZjiPikNth6LxaT7/KhRV72Ok2vI5sk8a1Ezi6jMASN5nTSheNnx+cqX2POGEGXQsUJwb9QhDGW2 2uK0wQmr PgHm8OoneQUU5bcSN7OPdpyivt0iJXxsIe5BBB8WacEK7rz3UEet5PbkX69CsQ4e7ZmD6QTQErMhYRSk/SuUA5xvZDByIAIF+bNWhc/7P6Rnw7wcJMUOjtt4gjfFqg/T48+v5M6aSc10wjclKpp3qRG2aOrSdPgPF+iA4kqeCyZiHIvcJ+3MAta0W2TAtE8+7h4nSOy/by0hn+WfeSrBl9ACCM/oasDrHwhVJbuuQIGD1E8rKuHY66WOFV/Fgz46NMGKTtvzAUkRrWja/wourISk0X07ppwXR51WifRiIg+rjHqWNixf2uRczCKjn//B8gWLux8pLKN4PgI+mzG6Dx/O2gsbiEnYBAPtckSb6qdRDxgcVyq3q/xhmNH4AXhOsoJ4bj83Ft0PCUyrJP2pA6s3Yua0K4y7ug24Zhjj14d7Z8vM2ZsiAzdEvQbqURbDZvRPNyX/tE2LKnJv3441Id7WXakg+NWd/bqXR77RaOM+81lQ3fk9qcX03bE5J1zghkgArjxn37hm+V7pshYCsdEX10VInMf+l68LVZqXFjTBA9so= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Pass the maximum number of fragments into skb_append_pagefrags() rather than using MAX_SKB_FRAGS so that it can be used from code that wants to specify sysctl_max_skb_frags. Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/linux/skbuff.h | 2 +- net/core/skbuff.c | 4 ++-- net/ipv4/ip_output.c | 3 ++- net/unix/af_unix.c | 2 +- 4 files changed, 6 insertions(+), 5 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 82511b2f61ea..6e508274d2a5 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -1364,7 +1364,7 @@ static inline int skb_pad(struct sk_buff *skb, int pad) #define dev_kfree_skb(a) consume_skb(a) int skb_append_pagefrags(struct sk_buff *skb, struct page *page, - int offset, size_t size); + int offset, size_t size, size_t max_frags); struct skb_seq_state { __u32 lower_offset; diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 3d05ed64b606..d96175f58ca4 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -4132,13 +4132,13 @@ unsigned int skb_find_text(struct sk_buff *skb, unsigned int from, EXPORT_SYMBOL(skb_find_text); int skb_append_pagefrags(struct sk_buff *skb, struct page *page, - int offset, size_t size) + int offset, size_t size, size_t max_frags) { int i = skb_shinfo(skb)->nr_frags; if (skb_can_coalesce(skb, i, page, offset)) { skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], size); - } else if (i < MAX_SKB_FRAGS) { + } else if (i < max_frags) { skb_zcopy_downgrade_managed(skb); get_page(page); skb_fill_page_desc_noacc(skb, i, page, offset, size); diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index 22a90a9392eb..bd002222cf2d 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -1450,7 +1450,8 @@ ssize_t ip_append_page(struct sock *sk, struct flowi4 *fl4, struct page *page, if (len > size) len = size; - if (skb_append_pagefrags(skb, page, offset, len)) { + if (skb_append_pagefrags(skb, page, offset, len, + MAX_SKB_FRAGS)) { err = -EMSGSIZE; goto error; } diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c index fb31e8a4409e..ea45dcc80232 100644 --- a/net/unix/af_unix.c +++ b/net/unix/af_unix.c @@ -2349,7 +2349,7 @@ static ssize_t unix_stream_sendpage(struct socket *socket, struct page *page, newskb = NULL; } - if (skb_append_pagefrags(skb, page, offset, size)) { + if (skb_append_pagefrags(skb, page, offset, size, MAX_SKB_FRAGS)) { tail = skb; goto alloc_skb; } From patchwork Tue Apr 11 16:08:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13207812 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7C0AC76196 for ; Tue, 11 Apr 2023 16:09:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7C31E900003; Tue, 11 Apr 2023 12:09:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 74DC7900002; Tue, 11 Apr 2023 12:09:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5C6A5900003; Tue, 11 Apr 2023 12:09:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 48FD7900002 for ; Tue, 11 Apr 2023 12:09:38 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 1E34E406AC for ; Tue, 11 Apr 2023 16:09:38 +0000 (UTC) X-FDA: 80669595636.19.6CFB07D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf23.hostedemail.com (Postfix) with ESMTP id 32A75140024 for ; Tue, 11 Apr 2023 16:09:35 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=A9Qmljxc; spf=pass (imf23.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681229376; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rzqtbLu7PzckfDCHcN9H9EFIWHgmlYZvGgMWnBPjkms=; b=WwpgtuqwDWa5f3BSJyn5PJfVG/q8cSfqqYj3pFPigiD4VX0qPFLUi2qhO9qvhdaDbcBx8D QL6tsgei7bEZRsdlq07/bWvcOQO2llcFf/cSKwvXpzWUc+ndj/+MgsiRJrXqJV7HnnkBZQ zPVIgHTloBKPfGpzSu0WHK90mmKD2EE= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=A9Qmljxc; spf=pass (imf23.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681229376; a=rsa-sha256; cv=none; b=AakIVhpWUtYZa4Not0meyrNSRnR/GXSJljxaUdYaT0TMfaHs91E2NfAU9KCRBU9JzLmP34 J9RJNlGz+eu9rckLhBbdVxNlAxjgdNhgAW8Uzo9rYXaFVQ78Lqe22YiBKOxkhfQXgdFtDG UbxU0lQ5Os5BadZOD2SNnRaU7VilkQA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681229375; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rzqtbLu7PzckfDCHcN9H9EFIWHgmlYZvGgMWnBPjkms=; b=A9Qmljxc9zP557HL0YMNMsEfIeXam7pOSYPTM8JoK3uVZLDqBOKmFNa8hJr9Zig0P9MT/3 snI5z4rYzv+FMUE71TceSgptT4falGuzfSqE64geKwWTSvYKd9xqZaDA83+wdddkLrpdGn iEuoWiWbK88Pa/EfIs6AAWcBvcGERlM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-361-5X01tIs_NgyU79LDUXo_dg-1; Tue, 11 Apr 2023 12:09:31 -0400 X-MC-Unique: 5X01tIs_NgyU79LDUXo_dg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 168421010424; Tue, 11 Apr 2023 16:09:29 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id EE86140C83A9; Tue, 11 Apr 2023 16:09:26 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v6 06/18] net: Add a function to splice pages into an skbuff for MSG_SPLICE_PAGES Date: Tue, 11 Apr 2023 17:08:50 +0100 Message-Id: <20230411160902.4134381-7-dhowells@redhat.com> In-Reply-To: <20230411160902.4134381-1-dhowells@redhat.com> References: <20230411160902.4134381-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 32A75140024 X-Stat-Signature: pyhs3qi47u91xz41ac55p4e1mkyukmtw X-HE-Tag: 1681229375-572613 X-HE-Meta: U2FsdGVkX1+1rVOwRp5bzlhIms5FEKigGTWLWRIojiOE74VB7QkYFCDBEHAfcV22hdXVmi0zf0hWNHxyE8yy6Z/VUbtw7w43opjutqRkMnB3gt4jhFl3xdTupopcg/R51fkNb3z7FON6XnSZDq2ueuF6RMTniF02/Z2STwVEEWZwl2fwXsan9IGLB6FKX1ha0Ivm2NG1IPUENGQ7cfKEF5DA9n1AEZa7JkWhwX8vEDBLw1Gwyvyrl4+UpCHkbicpugc47i2PtGdtgav3qFu25sDExOYqN1curtJGehK25NyCgPd8x9UigEbg3haDdyCF5U0fvuvHeLw5s8MaryMJfsHupPbh58Kji3ylhVHttGwyzdNUdY9aRFZOLDX5oUXMu5OkvjoW1rodZLH3df99tLARkgy/fYxbWzYaS0t+H+8nrChM43V6CdZcQKqFGIMP6lvJhLvr9/pGAsloKA8X4gULrM8qEWbzdH+ZKnKWIKxmJy1brgqIFb2ychQQiiNf9iIOgU3Qz6mtKv8FVRZnLn0i8c1vLiEE7YsUDD1FkUwq50c2aNzcromGf+9qqwNER8brLdeOPE3ZPJdG13kgSy/UThtWfLzLhcbTfrSPofKYPTrCzlp/7TZw7BGxZSw6VRWqAAhXRWq0hLOsVVav2iROWVApWJxN0cWoD4cz3Bv6yJPj7IWogoH/gWZkjF2OzOSJTWuT6gup9oWcZNr7bStQEEN4JSmQ6gdwM2TH+ThthtuA4ekZKjsPvxmdLwr10ZFTN7YJC5JPgNm94iZ2BurTfJI0qTV1MRWd5R2r31ItDGx+WQAEYJ7LjpDV+7S6GQQeCjLzAo2FJXltJDs/wcn34i5fi8G0tWtTVPKNPO9jzHUqSvaXdctJCCufT5cRyG8rqhSAJ2qWesky5PpNDs/9mJ46Sc3WNFwdocRgLFwsFuxB/6zYRACu9na9pYAuyguF22wiBcoHK16dGwx JNhS79To zgByKjeH6iMZHfv1KWYz2ils27RqB+phpIwgu6bG+f2lCrrppVshboRQ5XGZwOMsIsvSDVdtCmlMAG1K5oRk7Ryp8wkKjlgkj7mx+Emi3F6vn7eta4/qgjkNEi+gJvPZkpiB4m1V4km3tcvdlWYe8d8aU+MipWr9/Efz62QEjj25gUzc/Xkp49fmJfK0iKO5ox01TjvIVUqv5xil5lb/DoZDdFXe8dAIqtilozyfbdagem/5MsHOo/ViW3M4ejz2glosbNaaH7EQzKuTW5orOxyLee/Xn+cFPQc+N/1P7TbFRpdlAlTGX1+rdQAeL2UMFFAefs5DqCne+BneSUh7pKIfodVBc6BKvCgdtZ0Pl2vs2VAzzuCturtdVM1py0hLFJ0PZCm8U5WXM3g3Ghy/LzvbIGrOa4C5U0SYbbJB2O0Mkvl4vBmTYCepkjAzpyJVoybaPvdOQ4TWl7GNmQCpNGDT0DGrrEX6c6HSxx7jumoc2GvXdFhzSH++IVxyeUHqBR0m7 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a function to handle MSG_SPLICE_PAGES being passed internally to sendmsg(). Pages are spliced into the given socket buffer if possible and copied in if not (ie. they're slab pages or have a zero refcount). Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/linux/skbuff.h | 3 ++ net/core/skbuff.c | 110 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 113 insertions(+) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 6e508274d2a5..add43417b798 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -5070,5 +5070,8 @@ static inline void skb_mark_for_recycle(struct sk_buff *skb) } #endif +ssize_t skb_splice_from_iter(struct sk_buff *skb, struct iov_iter *iter, + ssize_t maxsize, gfp_t gfp); + #endif /* __KERNEL__ */ #endif /* _LINUX_SKBUFF_H */ diff --git a/net/core/skbuff.c b/net/core/skbuff.c index d96175f58ca4..c90fc48a63a5 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -6838,3 +6838,113 @@ nodefer: __kfree_skb(skb); if (unlikely(kick) && !cmpxchg(&sd->defer_ipi_scheduled, 0, 1)) smp_call_function_single_async(cpu, &sd->defer_csd); } + +static void skb_splice_csum_page(struct sk_buff *skb, struct page *page, + size_t offset, size_t len) +{ + const char *kaddr; + __wsum csum; + + kaddr = kmap_local_page(page); + csum = csum_partial(kaddr + offset, len, 0); + kunmap_local(kaddr); + skb->csum = csum_block_add(skb->csum, csum, skb->len); +} + +/** + * skb_splice_from_iter - Splice (or copy) pages to skbuff + * @skb: The buffer to add pages to + * @iter: Iterator representing the pages to be added + * @maxsize: Maximum amount of pages to be added + * @gfp: Allocation flags + * + * This is a common helper function for supporting MSG_SPLICE_PAGES. It + * extracts pages from an iterator and adds them to the socket buffer if + * possible, copying them to fragments if not possible (such as if they're slab + * pages). + * + * Returns the amount of data spliced/copied or -EMSGSIZE if there's + * insufficient space in the buffer to transfer anything. + */ +ssize_t skb_splice_from_iter(struct sk_buff *skb, struct iov_iter *iter, + ssize_t maxsize, gfp_t gfp) +{ + struct page *pages[8], **ppages = pages; + unsigned int i; + ssize_t spliced = 0, ret = 0; + size_t frag_limit = READ_ONCE(sysctl_max_skb_frags); + + while (iter->count > 0) { + ssize_t space, nr; + size_t off, len; + + ret = -EMSGSIZE; + space = frag_limit - skb_shinfo(skb)->nr_frags; + if (space < 0) + break; + + /* We might be able to coalesce without increasing nr_frags */ + nr = clamp_t(size_t, space, 1, ARRAY_SIZE(pages)); + + len = iov_iter_extract_pages(iter, &ppages, maxsize, nr, 0, &off); + if (len <= 0) { + ret = len ?: -EIO; + break; + } + + if (space == 0 && + !skb_can_coalesce(skb, skb_shinfo(skb)->nr_frags, + pages[0], off)) { + iov_iter_revert(iter, len); + break; + } + + i = 0; + do { + struct page *page = pages[i++]; + size_t part = min_t(size_t, PAGE_SIZE - off, len); + bool put = false; + + if (!sendpage_ok(page)) { + const void *p = kmap_local_page(page); + void *q; + + q = page_frag_memdup(NULL, p + off, part, gfp, + ULONG_MAX); + kunmap_local(p); + if (!q) { + iov_iter_revert(iter, len); + ret = -ENOMEM; + goto out; + } + page = virt_to_page(q); + off = offset_in_page(q); + put = true; + } + + ret = skb_append_pagefrags(skb, page, off, part, + frag_limit); + if (put) + put_page(page); + if (ret < 0) { + iov_iter_revert(iter, len); + goto out; + } + + if (skb->ip_summed == CHECKSUM_NONE) + skb_splice_csum_page(skb, page, off, part); + + off = 0; + spliced += part; + maxsize -= part; + len -= part; + } while (len > 0); + + if (maxsize <= 0) + break; + } + +out: + skb_len_add(skb, spliced); + return spliced ?: ret; +} From patchwork Tue Apr 11 16:08:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13207813 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E10CC77B6F for ; Tue, 11 Apr 2023 16:09:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DDDD4900005; Tue, 11 Apr 2023 12:09:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D6714900002; Tue, 11 Apr 2023 12:09:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BE097900008; Tue, 11 Apr 2023 12:09:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id A181C900002 for ; Tue, 11 Apr 2023 12:09:39 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 60D20160E86 for ; Tue, 11 Apr 2023 16:09:39 +0000 (UTC) X-FDA: 80669595678.30.758BB3D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf07.hostedemail.com (Postfix) with ESMTP id 647D540026 for ; Tue, 11 Apr 2023 16:09:37 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hytVOjGA; spf=pass (imf07.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681229377; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3NDU5jXq/4v/LV+bxGx3XkpL7tqbKHeQbhHOvrmy/nQ=; b=bjf9QNIKtQqW7sjxUjn+ZBTTcD0WQa2j2w3kWXy7Alm0Arxw593M0l+APhlp/JmmpXRcsW +fN9Lw714uE+DjQ/t200FIfHhd9H1e0+DRmJ8vt3CA/xVhjDQ/nHvQPPQTlqPUnYtgxWTS PUFNwm7wEGHi1SpxbqqBB0gPVUktuGs= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hytVOjGA; spf=pass (imf07.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681229377; a=rsa-sha256; cv=none; b=caDbM7jNfWFJyc/ROR3hCdnvmomnvlEKQRk0eQ2fov9ye+sM+MiVgZDYlks1mHF0EgRYMZ RC9z+7GjSBFU4UtC79yK2QMeKvIVprySP8WRvetHTmg6LvFml9RzlaliFOY61/ThqTPq2X T1dYq9ofgnm6X4BWZWd99ybNY7QrgFo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681229376; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3NDU5jXq/4v/LV+bxGx3XkpL7tqbKHeQbhHOvrmy/nQ=; b=hytVOjGAex0cUxEtQ1FByuVKvN45p4Ei8vXgVb9rqSacclLQuO6nWSWntSPlTFgDDmyUD7 n5phY8S+eZojgp/HfZY9qU4kmt7AIxudh52PgnozeatuAud/yNdeYiq5p1on6jUfuL6AAC VZ9cK5dWqCHoDqYQQP1qyus47APqJ2c= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-515-CzdXFcUzNB6gKH1eCyNJrg-1; Tue, 11 Apr 2023 12:09:32 -0400 X-MC-Unique: CzdXFcUzNB6gKH1eCyNJrg-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C58DD38149B1; Tue, 11 Apr 2023 16:09:31 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id AA4EE47CDC; Tue, 11 Apr 2023 16:09:29 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v6 07/18] tcp: Support MSG_SPLICE_PAGES Date: Tue, 11 Apr 2023 17:08:51 +0100 Message-Id: <20230411160902.4134381-8-dhowells@redhat.com> In-Reply-To: <20230411160902.4134381-1-dhowells@redhat.com> References: <20230411160902.4134381-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: 4rhap5i87pttpjjk7c6ok8zaokruiybm X-Rspamd-Queue-Id: 647D540026 X-HE-Tag: 1681229377-286266 X-HE-Meta: U2FsdGVkX1/1/p0fbUGTj2N5KIilr5aT3vawULO4H2eH5cjVi1HJfZFywQBMCLoMADac5MF5nyv2f3eqBTlSPN3uXNOLqO1Kggti9CVlzCpLOrCWKmMk2fbp7diQixj5pz1Wuto2S9SbMojXCn0aIXSJ4zGVgiCBgdp1plKt04CCBBGnOWW1RurQNrp6nKB0Z9PdCdAlaXg8VNYvoeixBpeXIPju3wyzKu6ZjR2FegU7gbUt47HyHGq+X93ZhWvsDOltJs7D8GTELtCMAovHsoQzj7a5DcAvCJzwyh1l+XodW+7b/fn1ExDXy9GcAVzTkhrNZHjoVBTZRN5K9g+8P/N7e4FbP6fUGCMb2R7J8bayw6FGVfJyGHQhXWI2iB7h3BIQLGnf0tuQOuN0b3kN4plaAK/J9/gwpOuKe3LMAsIBZC1rBj0DiU3JapmM1hbflyUa1yUi/fXXhT2VAQDk5X++tiZuI77XZr2QN6xjHzSeTK+c3ei5lZagEvX67+BS8GkjEm8L/wkYPsy4PZ8Oz6cDoj+32JpeZDGYaulFZkI4f6dZqCyRhIue7gidxuOunS1r5a6jeNqXgO99KJoTjOXZWqIQBaT4PjaFD0P4WiO1NikJ8DMEPGRUMx3+H9vXDf3JVPwtt6j/GvvhG7L8E0gZjYAHh+I/m5gVByce+jqTuHKrxnfNO4fmN9AkP6snLMzWorQK8smEWzjKO0oyiNk9mkTtSh6z/JUAd8khROxrU8ZRlsyMf4e+0s5WswGz6t+GCKVeoBYUST/XOJqLX9sc7dOM01MoBU44f6iPGQQmVbNSYg4ErmR5+p8hMoNB6ETE5n0YAmzGqv1DC7RBIh0Ug6ppo4LH9F9m4r/T0M66huIqU9pHwIlNOZNWwFywBjnEq1kPCTryahEYVD4HlVQFN49fWx4IZZ5ZzpJkQJma9poo7DtG9z4xA/pYrGrw4tHtPRruUa32U8/LNWH ZLZ+CLR6 yhuJ4X6FrQlj5QQ0betZb63W1UQgS7URDz4xPRVT0mXF2uv0ihtNIGHIgSOU0/T+VlHS3Jz63AN92hHszOlvHndkoRcWUBiqPaZD6vLnrWRUQ37AT/5fP1zySJRN7SvuoiPWAEMqEbOKCN75lVLbN6hSbQd69wfHE6p1cZ911ZmpIP/4fg8LfAZfCaN6xS1bqkGfEPOps3O8zdXCcbkgOVaG7dmwRkj1SVQGcRMtNyI9tm69k2sMJmfAKLycHIKsN6QQm6RPwT8A62lzH5Idn7SjgczJNpH+b1FccS3PzYbo1ChguSh98b/8CC9/z5W8OETDF59gilZy2gIeK5AW/4tDw7b7A72KXNzVaGPY3gzOW7SsV+eaaGEbbI5bB82UZDIzxQA3owNkAGrzcU0+wnFRb86g/Yq35wr+22sGg17rc3q8O7mPMjnWjnbVUSizpyVaCIQm/Io4WmoOd8hEJRX3IzZrSiPlhz1bDuevzRDly6KX1lWhmPthS/W/9F1aMRQpp X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make TCP's sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced or copied (if it cannot be spliced) from the source iterator. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #6) - Use common helper. net/ipv4/tcp.c | 43 ++++++++++++++++++++++++++++++++++++------- 1 file changed, 36 insertions(+), 7 deletions(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index fd68d49490f2..0b2213da5aaf 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -1221,7 +1221,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) int flags, err, copied = 0; int mss_now = 0, size_goal, copied_syn = 0; int process_backlog = 0; - bool zc = false; + int zc = 0; long timeo; flags = msg->msg_flags; @@ -1232,17 +1232,22 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (msg->msg_ubuf) { uarg = msg->msg_ubuf; net_zcopy_get(uarg); - zc = sk->sk_route_caps & NETIF_F_SG; + if (sk->sk_route_caps & NETIF_F_SG) + zc = 1; } else if (sock_flag(sk, SOCK_ZEROCOPY)) { uarg = msg_zerocopy_realloc(sk, size, skb_zcopy(skb)); if (!uarg) { err = -ENOBUFS; goto out_err; } - zc = sk->sk_route_caps & NETIF_F_SG; - if (!zc) + if (sk->sk_route_caps & NETIF_F_SG) + zc = MSG_ZEROCOPY; + else uarg_to_msgzc(uarg)->zerocopy = 0; } + } else if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES) && size) { + if (sk->sk_route_caps & NETIF_F_SG) + zc = MSG_SPLICE_PAGES; } if (unlikely(flags & MSG_FASTOPEN || inet_sk(sk)->defer_connect) && @@ -1305,7 +1310,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) goto do_error; while (msg_data_left(msg)) { - int copy = 0; + ssize_t copy = 0; skb = tcp_write_queue_tail(sk); if (skb) @@ -1346,7 +1351,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (copy > msg_data_left(msg)) copy = msg_data_left(msg); - if (!zc) { + if (zc == 0) { bool merge = true; int i = skb_shinfo(skb)->nr_frags; struct page_frag *pfrag = sk_page_frag(sk); @@ -1391,7 +1396,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) page_ref_inc(pfrag->page); } pfrag->offset += copy; - } else { + } else if (zc == MSG_ZEROCOPY) { /* First append to a fragless skb builds initial * pure zerocopy skb */ @@ -1412,6 +1417,30 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (err < 0) goto do_error; copy = err; + } else if (zc == MSG_SPLICE_PAGES) { + /* Splice in data if we can; copy if we can't. */ + if (tcp_downgrade_zcopy_pure(sk, skb)) + goto wait_for_space; + copy = tcp_wmem_schedule(sk, copy); + if (!copy) + goto wait_for_space; + + err = skb_splice_from_iter(skb, &msg->msg_iter, copy, + sk->sk_allocation); + if (err < 0) { + if (err == -EMSGSIZE) { + tcp_mark_push(tp, skb); + goto new_segment; + } + goto do_error; + } + copy = err; + + if (!(flags & MSG_NO_SHARED_FRAGS)) + skb_shinfo(skb)->flags |= SKBFL_SHARED_FRAG; + + sk_wmem_queued_add(sk, copy); + sk_mem_charge(sk, copy); } if (!copied) From patchwork Tue Apr 11 16:08:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13207814 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A0B7C77B70 for ; Tue, 11 Apr 2023 16:09:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 070F1900008; Tue, 11 Apr 2023 12:09:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F3D23900002; Tue, 11 Apr 2023 12:09:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D67BD900008; Tue, 11 Apr 2023 12:09:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id BF561900002 for ; Tue, 11 Apr 2023 12:09:41 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 99A9B160E96 for ; Tue, 11 Apr 2023 16:09:41 +0000 (UTC) X-FDA: 80669595762.29.8680843 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf27.hostedemail.com (Postfix) with ESMTP id C8F1040031 for ; Tue, 11 Apr 2023 16:09:39 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="gC/3zTPK"; spf=pass (imf27.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681229379; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3Xe0/5mGUyG55HKKyDAIMffk7S2lIM9iOEXDW9mWCVU=; b=XV6JtK4fZ3DrI8F4lmYKB7+M6XoGeEv86Q37Y9fvkyByLghTAgZgSK7q0OFY2jZMtltPn8 rH3w9vjLwF1VKHaEVou/hOgaTc9lB0dvD9ZIBt5bZhM4YCB9qzKScpDTrrrFbK36yjYzJG Eaw+zj22mG6HPIIrL5yCz4MJPx4wQcA= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="gC/3zTPK"; spf=pass (imf27.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681229379; a=rsa-sha256; cv=none; b=GF8sH9VzELE1KGRyxiePfH/EiWL4tojmduuk6ODjqsxvDgOp1LVx62shaMfSY0x602Ebkw aKGEyJq+xjfp1jFE4649Qh30CTuz7hoH2TVeyjgPuYd1CiVSDEU4bDm5APXaWf9vE1lu8i DIwv5V7veVtr4e8jmyH2X+DrOlATTe0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681229379; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3Xe0/5mGUyG55HKKyDAIMffk7S2lIM9iOEXDW9mWCVU=; b=gC/3zTPKSgJCar4MZRWA8jwUXFVWb1qCArs4AIFx42Q/dKCm9mcJN46nyls7MR524xvpCN jRiE0YSu2G5CQioPLQacEXgjb/cVR1TPJWw0QArnK9xYAz0qcMR0DFLbkuqrtc04PFRdZQ dNQp437vDPJXNP90NvQM05SixBUqi4Y= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-672-pWkhccMyO0y1D5fU_obMdg-1; Tue, 11 Apr 2023 12:09:35 -0400 X-MC-Unique: pWkhccMyO0y1D5fU_obMdg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A0EC82807D62; Tue, 11 Apr 2023 16:09:34 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 83F282027041; Tue, 11 Apr 2023 16:09:32 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v6 08/18] tcp: Convert do_tcp_sendpages() to use MSG_SPLICE_PAGES Date: Tue, 11 Apr 2023 17:08:52 +0100 Message-Id: <20230411160902.4134381-9-dhowells@redhat.com> In-Reply-To: <20230411160902.4134381-1-dhowells@redhat.com> References: <20230411160902.4134381-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: dmbgcic3z9xkeg6b8yaucxzdqniuokei X-Rspamd-Queue-Id: C8F1040031 X-HE-Tag: 1681229379-267334 X-HE-Meta: U2FsdGVkX1/UPoVzPxkLVNk+PJz3I1DieUVQmVwpeVhqjFjUPeLNU7e9xoeXSfYIVMNu5xrtXlfpuJoPn4xnnz9SKaVTjbG+yRhurJud2S7Otu8+izySepR0C7CDN/Ty7EenyatDNR1TROYC8XMyCGtJHhJnFo+udtsCpoz7TqVtnZ/WwbUvS7qzTG1gSGw56J0Pq/AwrCakIgDLLtYyvuCrl1IE5mx0XuNDAbUAUar70JAYiHpvupFkRgEmZYkqEzdlcROCFcK9Fbrz/uv80xSrqfJMG+DI9CvAZRXdIUaPUU132T7IHJTi7qaERNVkcq1R9GOwzHNugBi9Hqb0WyBtPn3uQZ+QcgiC5sxVXZvziMjl9WVdzRc2jUoMegi+7TsuOS+saV6DLERlr1UWpeR+kq+940bsiXM5DQbRrBmBulMZ+NtcuRHFy2URVv8/IE3A+VhiL7FDO+d55OG+Huc2ONkk6ks+QmIMQ0Gii+ZZz2W0VucO8pJ9tP4RHricrhgTK3xd4QmsebpGnYq1+aA8SBTci6lyC9594ZvkQN2oEDq9+nXTYRM8mWLNnwtApaNxdUxJSGblTbPY9L8sTnczO+GitOZogSgSJpxguo3wONpkVjNwKLKu5SlG5pLlCHC0AZf3WR6E1zRIFoKy5edUVlHN8I0Z3ZGTyUvoA3Xy8H5H+dTLSD1wqDmLS3ARYrPdUjeTd6cWtkLDtxMhnL24zIOISJztC7cMfWlMw5qvkAdHZdE01MdAuXz2JLyvjB4rOQsCAb/mbYmVNp9U9ODfhv1o5ty6bY4HqN24IpbHngS0/WukHwqSF32FTFa1WHXxBJKpkGxk+dkVaKY3CvKRx8Om80WG5qzJtj4i9brYg7j9kfivP0hlLrjmoyUnY70Cg0SaC5M3ZASE2KGAnrxAZ1uILiEzh6EyA2Z/Ht43nj7eNeqpruHx+XWWwcGjtRy8R8LcnL4vlskCmjY +5M8iV0W P2ATTApz7Gz6ugCwUh9i2SO/ICYKqKQ8MqEE5E1Xvfup3Rm8PlduP4KufIUe9uBeatBsNzQ/VPp7tSmIIsxzrnWz5O45SbCOIiKIt8PHy7jjGwd/WGk0/BeDr0AUGTmalSCpVbWdpPiLrnZCKXYGsiCJIoJJF0wRIIyxYSy7OU8/Ju1HEjw/xmrnl6aAlZzfbs67Wc94UzWEckASYY3+ArmLQ8D62cDxmtIndeO+b+KwwV+EDl68EYM4VPwqE2aH+scqwh0A+9PfuH0USjhTFkW8OYfYE3E3h6A6Tawh7mcxK/hI4RlwDtpk2/tTT2T34LHIPhOhdIEN9CmgD7CtVVNvEtX/7/qNdOJqLIEBnxyS9ttrEupyLLIwVOCrXVmrzppwn6s9Ts+4skAh5/r8NbzLWETMupczzVIci5AoVEEYFgx9puJD82ScoWvVIOcng2fMVk2XVIzmDRxCecF3HXc1pv26qtpE+QYq+5hqvEmJaz+02en79lOJ96UpeA8z4zesA X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert do_tcp_sendpages() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. do_tcp_sendpages() can then be inlined in subsequent patches into its callers. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/ipv4/tcp.c | 158 +++---------------------------------------------- 1 file changed, 7 insertions(+), 151 deletions(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 0b2213da5aaf..0187988514b1 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -972,163 +972,19 @@ static int tcp_wmem_schedule(struct sock *sk, int copy) return min(copy, sk->sk_forward_alloc); } -static struct sk_buff *tcp_build_frag(struct sock *sk, int size_goal, int flags, - struct page *page, int offset, size_t *size) -{ - struct sk_buff *skb = tcp_write_queue_tail(sk); - struct tcp_sock *tp = tcp_sk(sk); - bool can_coalesce; - int copy, i; - - if (!skb || (copy = size_goal - skb->len) <= 0 || - !tcp_skb_can_collapse_to(skb)) { -new_segment: - if (!sk_stream_memory_free(sk)) - return NULL; - - skb = tcp_stream_alloc_skb(sk, 0, sk->sk_allocation, - tcp_rtx_and_write_queues_empty(sk)); - if (!skb) - return NULL; - -#ifdef CONFIG_TLS_DEVICE - skb->decrypted = !!(flags & MSG_SENDPAGE_DECRYPTED); -#endif - tcp_skb_entail(sk, skb); - copy = size_goal; - } - - if (copy > *size) - copy = *size; - - i = skb_shinfo(skb)->nr_frags; - can_coalesce = skb_can_coalesce(skb, i, page, offset); - if (!can_coalesce && i >= READ_ONCE(sysctl_max_skb_frags)) { - tcp_mark_push(tp, skb); - goto new_segment; - } - if (tcp_downgrade_zcopy_pure(sk, skb)) - return NULL; - - copy = tcp_wmem_schedule(sk, copy); - if (!copy) - return NULL; - - if (can_coalesce) { - skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], copy); - } else { - get_page(page); - skb_fill_page_desc_noacc(skb, i, page, offset, copy); - } - - if (!(flags & MSG_NO_SHARED_FRAGS)) - skb_shinfo(skb)->flags |= SKBFL_SHARED_FRAG; - - skb->len += copy; - skb->data_len += copy; - skb->truesize += copy; - sk_wmem_queued_add(sk, copy); - sk_mem_charge(sk, copy); - WRITE_ONCE(tp->write_seq, tp->write_seq + copy); - TCP_SKB_CB(skb)->end_seq += copy; - tcp_skb_pcount_set(skb, 0); - - *size = copy; - return skb; -} - ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct tcp_sock *tp = tcp_sk(sk); - int mss_now, size_goal; - int err; - ssize_t copied; - long timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT); - - if (IS_ENABLED(CONFIG_DEBUG_VM) && - WARN_ONCE(!sendpage_ok(page), - "page must not be a Slab one and have page_count > 0")) - return -EINVAL; - - /* Wait for a connection to finish. One exception is TCP Fast Open - * (passive side) where data is allowed to be sent before a connection - * is fully established. - */ - if (((1 << sk->sk_state) & ~(TCPF_ESTABLISHED | TCPF_CLOSE_WAIT)) && - !tcp_passive_fastopen(sk)) { - err = sk_stream_wait_connect(sk, &timeo); - if (err != 0) - goto out_err; - } + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; - sk_clear_bit(SOCKWQ_ASYNC_NOSPACE, sk); + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); - mss_now = tcp_send_mss(sk, &size_goal, flags); - copied = 0; + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - err = -EPIPE; - if (sk->sk_err || (sk->sk_shutdown & SEND_SHUTDOWN)) - goto out_err; - - while (size > 0) { - struct sk_buff *skb; - size_t copy = size; - - skb = tcp_build_frag(sk, size_goal, flags, page, offset, ©); - if (!skb) - goto wait_for_space; - - if (!copied) - TCP_SKB_CB(skb)->tcp_flags &= ~TCPHDR_PSH; - - copied += copy; - offset += copy; - size -= copy; - if (!size) - goto out; - - if (skb->len < size_goal || (flags & MSG_OOB)) - continue; - - if (forced_push(tp)) { - tcp_mark_push(tp, skb); - __tcp_push_pending_frames(sk, mss_now, TCP_NAGLE_PUSH); - } else if (skb == tcp_send_head(sk)) - tcp_push_one(sk, mss_now); - continue; - -wait_for_space: - set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); - tcp_push(sk, flags & ~MSG_MORE, mss_now, - TCP_NAGLE_PUSH, size_goal); - - err = sk_stream_wait_memory(sk, &timeo); - if (err != 0) - goto do_error; - - mss_now = tcp_send_mss(sk, &size_goal, flags); - } - -out: - if (copied) { - tcp_tx_timestamp(sk, sk->sk_tsflags); - if (!(flags & MSG_SENDPAGE_NOTLAST)) - tcp_push(sk, flags, mss_now, tp->nonagle, size_goal); - } - return copied; - -do_error: - tcp_remove_empty_skb(sk); - if (copied) - goto out; -out_err: - /* make sure we wake any epoll edge trigger waiter */ - if (unlikely(tcp_rtx_and_write_queues_empty(sk) && err == -EAGAIN)) { - sk->sk_write_space(sk); - tcp_chrono_stop(sk, TCP_CHRONO_SNDBUF_LIMITED); - } - return sk_stream_error(sk, flags, err); + return tcp_sendmsg_locked(sk, &msg, size); } EXPORT_SYMBOL_GPL(do_tcp_sendpages); From patchwork Tue Apr 11 16:08:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13207815 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A95B2C76196 for ; Tue, 11 Apr 2023 16:09:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4691F900009; Tue, 11 Apr 2023 12:09:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3F153900002; Tue, 11 Apr 2023 12:09:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 291A1900009; Tue, 11 Apr 2023 12:09:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 14652900002 for ; Tue, 11 Apr 2023 12:09:45 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id C880F80E7B for ; Tue, 11 Apr 2023 16:09:44 +0000 (UTC) X-FDA: 80669595888.18.09CB1C0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf29.hostedemail.com (Postfix) with ESMTP id 11A30120031 for ; Tue, 11 Apr 2023 16:09:42 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="F7I7K/Ph"; spf=pass (imf29.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681229383; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rjJlYTxOmtT7wdKdf7S5Ju9QN7AtCPTA0VxjqyvYpeA=; b=I7S+S/NaJB7lljsO76+fhZoYvs8mQiM51QSkaHGrC227YOuKQNKTY/P2lP1WYsARMp5wNr g3QwXeJE4jt83FY6bwgkLHoj4J7tKTjd3ITBqGkMNfuaFCZtilphsopdOPSKIB14VdzgeV K9FqQehYvtaMVe7g+4VwfmM9I0vzN+k= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="F7I7K/Ph"; spf=pass (imf29.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681229383; a=rsa-sha256; cv=none; b=pwH6dhfJLn1j2XPTrtr0C63gTcd9Xn7jCqK0qYum6agNg/CBoKOB1OkCojwYL/scUDZEz3 jKfe4o/1F2C6xq4zzCyjEV8Ml/2hbo5uET4Qr3SPqELCNopnkwv43NJwt2Wu3v6/AWi8BW qkUBo2O+xG8tqdyLDkQvVl2h6baq7Ts= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681229382; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rjJlYTxOmtT7wdKdf7S5Ju9QN7AtCPTA0VxjqyvYpeA=; b=F7I7K/PhBye+yTjJhV8A7UC83T7LrJKnBBSg/eJIwkw8GwP0DYiWxunOTP1zmqov/Vh0JH LQfT3qOQtkS0JbNI0Vpg/8W9NsbVd8B4EbEQQENuEizCbQWuUH9lzGH4slAFqGawChxcyY MHm74AL1mY1Y7Zy1TLRicDj5C7ZEXH0= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-548-cmT0nmxaN_-wKHRfTU4xrw-1; Tue, 11 Apr 2023 12:09:39 -0400 X-MC-Unique: cmT0nmxaN_-wKHRfTU4xrw-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id ABF7538149AF; Tue, 11 Apr 2023 16:09:37 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 47A44C15BBA; Tue, 11 Apr 2023 16:09:35 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, John Fastabend , Jakub Sitnicki , bpf@vger.kernel.org Subject: [PATCH net-next v6 09/18] tcp_bpf: Inline do_tcp_sendpages as it's now a wrapper around tcp_sendmsg Date: Tue, 11 Apr 2023 17:08:53 +0100 Message-Id: <20230411160902.4134381-10-dhowells@redhat.com> In-Reply-To: <20230411160902.4134381-1-dhowells@redhat.com> References: <20230411160902.4134381-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: cwbrg6kr15bkp9te6yiid8crtnq876e9 X-Rspamd-Queue-Id: 11A30120031 X-HE-Tag: 1681229382-768335 X-HE-Meta: U2FsdGVkX1+KS+rNtfB8bCnsTQhX8kcGx24wUKNRmNcjcV5uQyjFiuey3fG/h0SYY9GGZE56bFRexBSQFJQsswOt4ub53h4yeCTJyiEfMcT2iYUoNQRhUXRLiH/EiCL35Kc83Rkrfy7c+DGAICS3MVyiWqH4lgx0FZ8M3G8CjJefSDOhrSYUKAjIMg80zbAv5A8KUcCkEuBDOsf0ChiTKZq933BUjsARUG79U11BlNGrqTFhgn0hh+pcD9MfShksA0aofyRTsUn8EV/okgGDabhPAYFHddKl6eC8BlBWOSAHZs/bQ5I2GRFtHypiv49IiEp7tFYr20OvEOgtqEzqsbzFsA/QFmTpRQz5z2LyT9pFHBCsI6YR+9ru+zkx9VBjiG8OEqXFDIbQWqqp06b416x9hg9tZpL/EUsv/0GG664+u7fwJfEhmDxEZ6x7WjSdvFt+n4gfGE/kKxp8sNiaxkLXisOKyIUqOpzyNAAcKDyFjmuTmulNUfAblT/ntIUYqLUGCqZgGIFG7WZ2+tOe2mHI51lae+axMYJcC0gs4Xt9kMbRwWxX1PvhvK/R49mGSKNkoCHnXgjQL9iM+UNQIH1+AUj2MKqBla7GpjpANsZN+7ZwVhAQrXslN/wepIuFfOYWesB29aaCqf9Ypt0arCG3E03vmqMhfHbdeGC7+RSfc0XtKY776duLinZ6MlvIGMIGS48YkKfg7yb+8wh5pkwuUUu5dC1+9iD5+3wxcm5uiuXHbpAJedFh6XUP0RUU8LQkXifjzipNNDZqrJCXDAZzox6nwwdUV9ya/Wnc/XJPrn8DA7stz7mKayPuKVsPqqZqj8R0Y3FHaVaFVEKzjG+w/Fls2SiUMPpS3/WK2Vopy9InVttStHyI3WCEjO/JKVmVBocVunMN0eBOCoysqgb9DV96/pZuW1zpbuVsOI2ILuw0Bn0VN/mJtn9IPYPe7Pc/svK+ChKr2L6zzz1 eJ3ueoI5 dC+dHqdRi6BLXrV+1cdBfNbS0D21e8ZILKB1z1GN2d0ZmcfzAzJX7xAh1mBWMHTTcozM1p36vwANLrjWogOQDFPj8mDlV33voqw/bRKzqj1IzuJ0LjVj/LhnTjDN/trdSwXHZ4Z0PdvU/SwGK4+55OJ4dt276z/NsMpWsRzjt8aXthNptRgdwj2eOYzLxrhEU+0TN5YyXWeJlKTKQWzWGxKaGO+AGT0eGQXiM2NPVnU46pt3GcGrUFBH8ND2exd71ijEU1sE9ieWDol13ZifSRbJDmvDGuxYvFv1R8uBeC1B+XUQ3adLDaNjnDmNCp8my4eXrl94masISfbk3+fp5AM+atTzFzCsKpgslCsFZ02DUwXhtGGQPl3ZYYbzuybbTcXsgsSGVDXF1kU+Zgj89R3NG4MWhSR2z6mgvlFmrjXDa0cBekKLyRDtWpz8mJ+vs1JguWgguQrcuWZEyAAM7bzOhEAHoMSTBSxYYrDJMK4sCL/6PODzFk2QxjJLxufYZ4OUyFUP1fJHwre/p5ZJZQDP+q4HiCA/qYwV7A1bWWMyB2YR90dc3RPLApQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: John Fastabend cc: Jakub Sitnicki cc: "David S. Miller" cc: Eric Dumazet cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org cc: bpf@vger.kernel.org --- net/ipv4/tcp_bpf.c | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/net/ipv4/tcp_bpf.c b/net/ipv4/tcp_bpf.c index ebf917511937..24bfb885777e 100644 --- a/net/ipv4/tcp_bpf.c +++ b/net/ipv4/tcp_bpf.c @@ -72,11 +72,13 @@ static int tcp_bpf_push(struct sock *sk, struct sk_msg *msg, u32 apply_bytes, { bool apply = apply_bytes; struct scatterlist *sge; + struct msghdr msghdr = { .msg_flags = flags | MSG_SPLICE_PAGES, }; struct page *page; int size, ret = 0; u32 off; while (1) { + struct bio_vec bvec; bool has_tx_ulp; sge = sk_msg_elem(msg, msg->sg.start); @@ -88,16 +90,18 @@ static int tcp_bpf_push(struct sock *sk, struct sk_msg *msg, u32 apply_bytes, tcp_rate_check_app_limited(sk); retry: has_tx_ulp = tls_sw_has_ctx_tx(sk); - if (has_tx_ulp) { - flags |= MSG_SENDPAGE_NOPOLICY; - ret = kernel_sendpage_locked(sk, - page, off, size, flags); - } else { - ret = do_tcp_sendpages(sk, page, off, size, flags); - } + if (has_tx_ulp) + msghdr.msg_flags |= MSG_SENDPAGE_NOPOLICY; + if (flags & MSG_SENDPAGE_NOTLAST) + msghdr.msg_flags |= MSG_MORE; + + bvec_set_page(&bvec, page, size, off); + iov_iter_bvec(&msghdr.msg_iter, ITER_SOURCE, &bvec, 1, size); + ret = tcp_sendmsg_locked(sk, &msghdr, size); if (ret <= 0) return ret; + if (apply) apply_bytes -= ret; msg->sg.size -= ret; @@ -404,7 +408,7 @@ static int tcp_bpf_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) long timeo; int flags; - /* Don't let internal do_tcp_sendpages() flags through */ + /* Don't let internal sendpage flags through */ flags = (msg->msg_flags & ~MSG_SENDPAGE_DECRYPTED); flags |= MSG_NO_SHARED_FRAGS; From patchwork Tue Apr 11 16:08:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13207816 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52F7AC76196 for ; Tue, 11 Apr 2023 16:09:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DDF93900004; Tue, 11 Apr 2023 12:09:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D6728900002; Tue, 11 Apr 2023 12:09:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BBA77900004; Tue, 11 Apr 2023 12:09:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A2D73900002 for ; Tue, 11 Apr 2023 12:09:49 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 5B1BEC04A0 for ; Tue, 11 Apr 2023 16:09:49 +0000 (UTC) X-FDA: 80669596098.03.6563533 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf16.hostedemail.com (Postfix) with ESMTP id 4D5B1180030 for ; Tue, 11 Apr 2023 16:09:46 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gfBuvmFK; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf16.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681229387; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tYeoTX9B8dad49qWPHePVLYzX6oyR1jBlVymL6Q5Zsc=; b=skCZ4y1842lNLlashf6DinVASPqaGRkDYJEY41cYgZ2SD3PgQIFhpTqmJrE+63EQKfkJu4 +a0/8i2H1l+FksUq8ySzeHbI4POYJDaRHwiRd2Ga9gVQuAfMstQcc+rxbVMqDKC2Z2DkOk nSGIOkjOgkwm2jb4sOPv/jdkUa0i8jw= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gfBuvmFK; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf16.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681229387; a=rsa-sha256; cv=none; b=Hs7rGHsWEfxC8ZIVIZ+SOCAA6aLUeKQlLhj4t0+EODVdMw0tL9gwbnmIFxctBiRM+8AUZp PAKfnd0BBYzD/PcmdAe9JB79gtPmqZnpyCLbW9+y5ox6Vw2Oq5ozhSkdEdmmnHE+jOxvhc 4hJkXEsWE7HTmP85ZWHAuGY3AkGJIpk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681229386; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tYeoTX9B8dad49qWPHePVLYzX6oyR1jBlVymL6Q5Zsc=; b=gfBuvmFKv3z1jYlUKaNFNCsekjxaS7TqinG+//1KoVR7TTxnBGOY7B2biLteXD6440bKHS N0Cy5xKV66E9O05yUfm5dhEGy/E9w3MIFLAUqWVyNhTLsZhyCdyZS81eiogrMJE2aeeZ6y WM1i7q9vqsjJL/NQq6gq93zsoedGXNY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-77-UXK5c0oIPSW0QoQ1NoAdzQ-1; Tue, 11 Apr 2023 12:09:42 -0400 X-MC-Unique: UXK5c0oIPSW0QoQ1NoAdzQ-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 16E3B87B2A7; Tue, 11 Apr 2023 16:09:41 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6348847CE1; Tue, 11 Apr 2023 16:09:38 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Steffen Klassert , Herbert Xu Subject: [PATCH net-next v6 10/18] espintcp: Inline do_tcp_sendpages() Date: Tue, 11 Apr 2023 17:08:54 +0100 Message-Id: <20230411160902.4134381-11-dhowells@redhat.com> In-Reply-To: <20230411160902.4134381-1-dhowells@redhat.com> References: <20230411160902.4134381-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Rspamd-Queue-Id: 4D5B1180030 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: hdx8e93k7559rjpk77m6dg93n5hkiy5n X-HE-Tag: 1681229386-994465 X-HE-Meta: U2FsdGVkX1/WOMs2giT3F/I8Yf6DreRoSUUUvTGFUlUygBorbH37YbY5tro2e7bo31f4Owy42O4iG+xNscURlQ+Fzoku05s3y7ohjXvH/cmAqbmlf+rZQOtZkEnrKo0BU2fxN11K8i7g2IrULr+P44QuZ3nUMj6j+a14D3FO4ySXBNC++9cf/RBPZyB9ders3S2W91M0ZSBosFFEcMceqb0zVBFnS2BxjAWyZfbx9f4vCNQNDCQnllylQJfgi5fVloUNiApXZ2+9QlHU73ONPRJfUWuswqYge0CYTIuVZ6osvBA1xsM0lpRoA/mw7LDHsfsc5JUgg7PDRkGRdzpHajE6F8uS+6+14QNHLTjOH1R9k8MYyw8InGuJVXJK8KUuQLli0Ohf8MGksYoJT3/LbTKv8kM/canIRCHKNhqmCUQWvFaQ33xIu1SkLVmqO9zR3s8ltx7YYKba7TUOPBQS7PX3m5UuNTX6eaYSoCy8S2vRER8Eh7Zf7XQHpuneydU0KnaSYkepCWFGo8/fsGN52Fc4cmBydKBU/QbzgAI0MSveO1H4zo+uC40hbnW0GXPBdBsQ3cKNrcgi0f6F1nNttuZ+Tg0Bv1/2S4DltTcMVxM8wSYgcwF4F6Qr77gJXXRV8rIc7W+bxiXYLt7GwFvqsOUOYZ/ozbLvD6RK1ToLaalAKhudV7CzwNLn6ZCXjbrIx9ar7ggK9qod8UPWrWAkDL4zIf+34bMcJ33FvmGukodp5ePQ3vZQ7N4zArDABfGaKKI1Uu9wp6783Pnwm+MehX5QlzPrlZqDks4XhyZj8B5KHI409S89qtPDVela1DewoqDTTRlUa3rL+7BnN0e9XfFCEX8MpRwTJaj3r9R83wUw8KJYgeOgSqw8l2evCiL/aIWRey7CQF2j7/VZkzIIG72TTYFZtFRsLTfUlFmY1iZRTI2+oA23tx5FVWhlPUFMQVb/NADilxi/Bj0/TmI qoaSyNPc J8vBxlBDlGkIjw+tCOLeflhGVZRYD+2WAs0F5bA9pLuQR1NrRqCZn2bQinkruofPETiicYLZ4m2tgXnySUq1Zjy04sMoYUp5vJrF7QG+QG/+U1vRGemgAU5Imdi0oD3rIURv6043cxt8G8J/AoDqxv9xedGOtu3mrTMRQ+riqnHDyymPN8CwVXNWKurtn1pbTsaMiII9u80X74W940IdNFOU6A34eYjOgVUnlDpXv/ef3D+4Wz+mK7XKmeihFu/DBFtVvKOQg385ppGOMk0TK3g6AxL2fycOewhSkIkHuFKMxQHdP6Af+/LdjOUDR92oeVbRq0Ltji7Z4Mw8DIlH5Gqh7i+sFWFWph6NIY37bSmzhAFPYpQPOgIL0+kCGzRadVlNsOFQpdnlfugcyjyb39VJQ5JQHxPxoJAXrXcArOk7dAyb7UbSCRxHPUxJpoiY8J1PAQuspUcp2y2H6MmfH3n1RwTBFbnvhyQXpGQWGzo4iBXc9MnPvIzicnLEfdEDJnaTZ467WBtBHUyEY3WK8vyk/b/PP/iRfbTyzV2mE5dIgbBBHu7kSmo77q+PIGI3dBuPbCBkBBOeEYP13WYtcuHGS264Jiyc6hxUZ2SatJkE/CRKsU3SLHkn7FA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it, allowing do_tcp_sendpages() to be removed. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: Steffen Klassert cc: Herbert Xu cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/xfrm/espintcp.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/net/xfrm/espintcp.c b/net/xfrm/espintcp.c index 872b80188e83..3504925babdb 100644 --- a/net/xfrm/espintcp.c +++ b/net/xfrm/espintcp.c @@ -205,14 +205,16 @@ static int espintcp_sendskb_locked(struct sock *sk, struct espintcp_msg *emsg, static int espintcp_sendskmsg_locked(struct sock *sk, struct espintcp_msg *emsg, int flags) { + struct msghdr msghdr = { .msg_flags = flags | MSG_SPLICE_PAGES, }; struct sk_msg *skmsg = &emsg->skmsg; struct scatterlist *sg; int done = 0; int ret; - flags |= MSG_SENDPAGE_NOTLAST; + msghdr.msg_flags |= MSG_SENDPAGE_NOTLAST; sg = &skmsg->sg.data[skmsg->sg.start]; do { + struct bio_vec bvec; size_t size = sg->length - emsg->offset; int offset = sg->offset + emsg->offset; struct page *p; @@ -220,11 +222,13 @@ static int espintcp_sendskmsg_locked(struct sock *sk, emsg->offset = 0; if (sg_is_last(sg)) - flags &= ~MSG_SENDPAGE_NOTLAST; + msghdr.msg_flags &= ~MSG_SENDPAGE_NOTLAST; p = sg_page(sg); retry: - ret = do_tcp_sendpages(sk, p, offset, size, flags); + bvec_set_page(&bvec, p, size, offset); + iov_iter_bvec(&msghdr.msg_iter, ITER_SOURCE, &bvec, 1, size); + ret = tcp_sendmsg_locked(sk, &msghdr, size); if (ret < 0) { emsg->offset = offset - sg->offset; skmsg->sg.start += done; From patchwork Tue Apr 11 16:08:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13207817 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32D38C77B70 for ; Tue, 11 Apr 2023 16:09:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C0B37900007; Tue, 11 Apr 2023 12:09:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B92C1900002; Tue, 11 Apr 2023 12:09:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A3429900007; Tue, 11 Apr 2023 12:09:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 8CB47900002 for ; Tue, 11 Apr 2023 12:09:52 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 5DC97C0E7B for ; Tue, 11 Apr 2023 16:09:52 +0000 (UTC) X-FDA: 80669596224.23.E36BD10 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf18.hostedemail.com (Postfix) with ESMTP id 107721C0020 for ; Tue, 11 Apr 2023 16:09:48 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=GmR5nHIA; spf=pass (imf18.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681229389; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3FA5Yjp/rAglxaSde3E7QeP89aqCLnwKrUfyvWNxLew=; b=lXtfXzIiTVZdRlTKdwIgWoyImr4cQmkM3QVYm6sq4pwvDRH43K2sfHisnv/8KjA44i2Ht6 foTKH5Yt2xzgC4j6A581Np3BHGu88j8fMFjEghwAc5md20q/Ff1troS2BnBfdjCRYWsbfC JKEhZpig6J+iZnF7Kz+ObM13kLC4pqA= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=GmR5nHIA; spf=pass (imf18.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681229389; a=rsa-sha256; cv=none; b=kpaT+EpPsUKlg1JjJhwSQLMEkwf0V7tFFxCn2aIobenhN74Lyr+Gc7tGF8pO20K3Yyyr2L /JlhYeHukrTi5+Dba4AFKNGEF/zHiKC/ycUivab0jwXePKyrJdXNCQaINjtKgsYp1s8M0Q 1iKyY494iXy6p+YsgCO4iRgQAtVocKc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681229388; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3FA5Yjp/rAglxaSde3E7QeP89aqCLnwKrUfyvWNxLew=; b=GmR5nHIAH4Z6dPxqltjuLWqI1H+Zy4qCw0ViFLsO0Rp+nvZfo5k2eQ8aW/wu+odw0j1cGH 7vQYc4KSVNVJ3BIe2iBYTwEj0qrKxEYWO+7SLXhQlS5Lis4cE/g0GIDh3Uxz7CFnwU41se /D44SCOg7FavjeKaBUO7s470+wj9IsY= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-256-hC-bCB2tOPuewyxBc3G-bQ-1; Tue, 11 Apr 2023 12:09:45 -0400 X-MC-Unique: hC-bCB2tOPuewyxBc3G-bQ-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 028F638149AF; Tue, 11 Apr 2023 16:09:44 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id A8ED8492B01; Tue, 11 Apr 2023 16:09:41 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Boris Pismenny , John Fastabend Subject: [PATCH net-next v6 11/18] tls: Inline do_tcp_sendpages() Date: Tue, 11 Apr 2023 17:08:55 +0100 Message-Id: <20230411160902.4134381-12-dhowells@redhat.com> In-Reply-To: <20230411160902.4134381-1-dhowells@redhat.com> References: <20230411160902.4134381-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Stat-Signature: ehzrxcq8xfzikg8q8h6d3yuagyyow1wp X-Rspam-User: X-Rspamd-Queue-Id: 107721C0020 X-Rspamd-Server: rspam06 X-HE-Tag: 1681229388-910573 X-HE-Meta: U2FsdGVkX1+c3Z9m+8Y3GhNNtB2tzHsoG34GfxLQBffNYjR2I3qqwUSKs1UULIxGUfWoobGbn+8mp8Ewmqm+i/0ULWI3Lx+ywO+wFc59hc+eioGfQnIh5JMrGCIeP0l/lCaLqLkMwG4Cl8XEnGq4aSwGbYUcLHhwiStv+diGU+PihAS9tzQZwHtnOMYSR/v8quZ0fANQlVA6V4tQN+7CDEef+GbqVGu08wjBdykETsShL/QWkSfSpJBu++Hs4WtrsSDVPVRUX/TMCta5YkjoTWfORi9vpRT+exBEU4F6joHZC4JDRroZE5rX8JuVa2atITTgqz0eNJCQAcI6DmFrxLWgk8t1Yt1cFTIRFUhCDxWptNYoGYK9jR5ox7B+pPqAA41gCoNpREdWLU7g2aa0qcinOKKt6Gx+sekCKsyDQaqk9144by2OQ1WKQPVsQ37PN6HWn4NSfSGIF/SKjQk1YIWY7okToeXj2C6pfHCCHHjJk6vKZr4r3iCzVvynu4TwY0YnbWA5lK1WDvJCueJ0W71F/23AP3tZxe+vHPbvxyrvtX0oOe4eh6pAfVGuv13YaN1qn5Ioxchx8urcScMGwqfnBWKMRCHaKfgJ9W1h/+DkhGquK1/PGR2FJ/22QAjkdAzNQhIM6ygKVZq8Bdxcy9xukpAM7ImmDRB9b6v7IQgsvrqQhuppgONysK2UemevoVspS7skIkINtf1+o4J3C9l2WsqmAlZK/qVU27EnxF6tUwe0VTqaIP+s0dGXFxc1D2RBSYmlD3lxGZmZPnRqv/dik1JOOXsu/NbpoVzOgjrsB7J+muIGfJnJA3cg0zPz58DxA5ry7TLT8W5BUIouHJUpcwHavdo2EFyTasR+fykGuLMdkaGOFwCfyJO7/KiwRhCDhf+ccpivGt2cfgVqYlk8CEJmdbcgTwAyKqEl9wpk65GAiZoMriPa2m5iHi4cnKYHWylx5v7aIXQ725o 0VP5bsdP RPTezUY/KU14IUEWf6sv4neN+erdO+vGUZZYsdSV03eeFpgyh8l7xBdA9SPM5ZJuwEM84HOnfw2gjJMwWldyrtqEpWjEveBMZBHyAoQW3XxGt0912IXOHHKLOw6z7XRZEU3sl81SiAsHm0lKzGynw45lPh0YQNYRpTzGVwBKuZjuoXxOZ+pMbLDcGwY0LDcE6Qcub9ErFDq1VAvJn5lv7wAM+GJbuwwdCJYvkMg9DH0I8uzi8sLNGpFwS31kH8y3roqB5wOM+v16dPIRo+a2EA5p9auNInO01a+WxHV5epUOoD0kOM/ivJhxoEQkHKlHU3M1MZnRG8NitavCrs3yR+fcNwMNNi1bZHEeo9X+NG7Fa6KW6ypKBsQiP6W+ZZGF7LNGlrgH+3yvyy/9qRyKmhapF1NOeFKmRjrwSBNqe3bcr2yjZEKj0c1kpLSyKzgVgXOu6itRWZUVMWUEnp7pSBPp4wUMgDfrVD/qowLmXJGEWYS+ky+WFlyGdRyPiDryhoC7iWUD4Tv28tD8WriyFoje7FhJTzfmqeXMYMBKvpRlqrDdhAVmLQ5nGzg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it, allowing do_tcp_sendpages() to be removed. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: "David S. Miller" cc: Eric Dumazet cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/net/tls.h | 2 +- net/tls/tls_main.c | 24 +++++++++++++++--------- 2 files changed, 16 insertions(+), 10 deletions(-) diff --git a/include/net/tls.h b/include/net/tls.h index 154949c7b0c8..d31521c36a84 100644 --- a/include/net/tls.h +++ b/include/net/tls.h @@ -256,7 +256,7 @@ struct tls_context { struct scatterlist *partially_sent_record; u16 partially_sent_offset; - bool in_tcp_sendpages; + bool splicing_pages; bool pending_open_record_frags; struct mutex tx_lock; /* protects partially_sent_* fields and diff --git a/net/tls/tls_main.c b/net/tls/tls_main.c index b32c112984dd..1d0e318d7977 100644 --- a/net/tls/tls_main.c +++ b/net/tls/tls_main.c @@ -124,7 +124,10 @@ int tls_push_sg(struct sock *sk, u16 first_offset, int flags) { - int sendpage_flags = flags | MSG_SENDPAGE_NOTLAST; + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = MSG_SENDPAGE_NOTLAST | MSG_SPLICE_PAGES | flags, + }; int ret = 0; struct page *p; size_t size; @@ -133,16 +136,19 @@ int tls_push_sg(struct sock *sk, size = sg->length - offset; offset += sg->offset; - ctx->in_tcp_sendpages = true; + ctx->splicing_pages = true; while (1) { if (sg_is_last(sg)) - sendpage_flags = flags; + msg.msg_flags = flags; /* is sending application-limited? */ tcp_rate_check_app_limited(sk); p = sg_page(sg); retry: - ret = do_tcp_sendpages(sk, p, offset, size, sendpage_flags); + bvec_set_page(&bvec, p, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + + ret = tcp_sendmsg_locked(sk, &msg, size); if (ret != size) { if (ret > 0) { @@ -154,7 +160,7 @@ int tls_push_sg(struct sock *sk, offset -= sg->offset; ctx->partially_sent_offset = offset; ctx->partially_sent_record = (void *)sg; - ctx->in_tcp_sendpages = false; + ctx->splicing_pages = false; return ret; } @@ -168,7 +174,7 @@ int tls_push_sg(struct sock *sk, size = sg->length; } - ctx->in_tcp_sendpages = false; + ctx->splicing_pages = false; return 0; } @@ -246,11 +252,11 @@ static void tls_write_space(struct sock *sk) { struct tls_context *ctx = tls_get_ctx(sk); - /* If in_tcp_sendpages call lower protocol write space handler + /* If splicing_pages call lower protocol write space handler * to ensure we wake up any waiting operations there. For example - * if do_tcp_sendpages where to call sk_wait_event. + * if splicing pages where to call sk_wait_event. */ - if (ctx->in_tcp_sendpages) { + if (ctx->splicing_pages) { ctx->sk_write_space(sk); return; } From patchwork Tue Apr 11 16:08:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13207818 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1D6C1C77B6F for ; Tue, 11 Apr 2023 16:09:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A85EF90000A; Tue, 11 Apr 2023 12:09:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A0E03900002; Tue, 11 Apr 2023 12:09:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8870090000A; Tue, 11 Apr 2023 12:09:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 72416900002 for ; Tue, 11 Apr 2023 12:09:54 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 4D6C8120DA5 for ; Tue, 11 Apr 2023 16:09:54 +0000 (UTC) X-FDA: 80669596308.24.B7AAA79 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf11.hostedemail.com (Postfix) with ESMTP id D93654001A for ; Tue, 11 Apr 2023 16:09:50 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Bg46C7SU; spf=pass (imf11.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681229390; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rMhKzvuG7s1VjfB4heYphcINA7j4rSAbIEKmxFNidXY=; b=GosgysCVytl5bXQ0OWPYdrAnpop9n5XE99mQOe3wM3c699kC8WErkUcAbllF0lblPHsvnP 7FNfT7+o1LINxKWqn8PuvTwxpwfOKhiNg6/WThmDsP1M2bQOn0PJkpLoknEr9+YcNpR6vy oRo0dK/Fqm/nIBRJoPNgvKQ6MOjR1r8= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Bg46C7SU; spf=pass (imf11.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681229390; a=rsa-sha256; cv=none; b=2ntiTvRxkFMcj7oCq1XqOrb556bMMnMiGXhzVWxOfeLbLlQFUfBt2Kr6A/xQ9kJRZ/E1Dw n6vBQzTzJkaIKFi7XGk5shg+48leXg5Tq3mqJeS6a6Qbuq2h+qQw+YT86O5nEb0auSoR7x Ed17uMH01rU3+t5UYP43B5xx0+SIURM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681229390; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rMhKzvuG7s1VjfB4heYphcINA7j4rSAbIEKmxFNidXY=; b=Bg46C7SUgV/OdaFKIDQ1M25tlYJAcRUQmwNd0/2nWQWvREF8NJK4Gchgoq5omK3V9zxZ3s rawl5095LXzqzSFMPK5bwkB89MeQK58UgHplr3/CNYjsTQdGVWkBYfEZDD8wK3/Xr3jt30 z8dtb6hhRJUbGsvfSSxdw28jscTb3Xc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-552-Zov1PRTBPSW9DUg0cv7v9g-1; Tue, 11 Apr 2023 12:09:48 -0400 X-MC-Unique: Zov1PRTBPSW9DUg0cv7v9g-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 794D6185A7A2; Tue, 11 Apr 2023 16:09:47 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id B3F9D47CDC; Tue, 11 Apr 2023 16:09:44 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Bernard Metzler , Jason Gunthorpe , Leon Romanovsky , Tom Talpey , linux-rdma@vger.kernel.org Subject: [PATCH net-next v6 12/18] siw: Inline do_tcp_sendpages() Date: Tue, 11 Apr 2023 17:08:56 +0100 Message-Id: <20230411160902.4134381-13-dhowells@redhat.com> In-Reply-To: <20230411160902.4134381-1-dhowells@redhat.com> References: <20230411160902.4134381-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: D93654001A X-Stat-Signature: gbt9d8zn9i5ayqd553jt8am44u4y6q4d X-HE-Tag: 1681229390-909163 X-HE-Meta: U2FsdGVkX19J9Y3JZ55qNG9NIU/xnX2ISjC9SEYwK1qBBKyrAwywlsUtpDx4lRLHwFa7RlE//TM0V0aUtLWsA6QzpSIiVQ9He1CA3UKfohvu1Jt5OlZRUp0eEs4Tw6SpGafegQ2NYWXIOxvgV6cnOzKP80na86FII2uKlzATJ+OYho4ue8DMaJcvneMhxGP6kuRBP/pJ7PeJVRiDLSxHRntfvRl5l6Wulp9mgJJ2jNURcG/Kd/KYNARxhiKZaO0SnH2OcD5vILYnxlGIOmDcsOkBPflFNzkLwcDKXRP44jEsB2SXsSYbw2dNUlGntXDfNxmVuFHQoBcUIb7CbJrd6rxl2rUCS4HA0cimYEXrF1cxGvwp6G5IGNgF73ZqXjzfa2VB7hRMHOrH2hbFQJrJyskoUnnS950c/jDM25k6SmA2xm9fEZD6V/tyOU5TzWnGn2dQvZnwoLqSDwkLeNLJVMAE4UyiWlg6Tk4u40+yw28k8Nnts07TTiX5+ivvHWyTRTTDeRqex1civBI4Oqbx9NaZQzr6MNh9xfUT/PNDu49yzdJfNI2UwyrSde4cTDw0GEQ47DG0YmGjz+52cY8tv7mqyzMx4hZnRtrtSQEuF5l6jo6M5zafdS1B96kT9LUh2hCEZeIRYShS00VDvyavH/qJw/9ixfPp1i4dKOTEQXjf+kfza3ATnJX5cWuolMcVMYwD68dRw6ucbhCwV3r+5ipjfU6bMEgtZs+uBsmhVilwWtRlOnIwOJjZ4g2/218qGmNyrq8yHv7DYhiSg19uxM0QrOwOkO6docXWTVGEQBn6Cj8jfSdf0ReoLU3KCc+zbZLW5O5zmRnInJ7iTk+Ymb9SnGavbimk6BsI7IpdwszjSCirCSRPCmtge4axBOh1TsODzXQpMnnQd5IBY5HGSb2/1DAXYc335zPqBe+Q+7/Xwxnc00mbn6XmHw4K9mZPfa+IXuB7futMVE4SF6Z 9ATfI0i+ HqTtsEuGar/WqmLEpAZW2Q27LdPSMxIXVEZCqPu0vH08eYsb0R1JAGt5sljCyP2OingYg/9DAs1Ub7mnje/kFucjNNZ8K9t3hSsCB3EXGql0ApWjWBTHADIg2X2UUWdB0PwZIQ44jEsZhoqDc2G4gdeqfNd/Sg6b3s4+V/dG0y2J0MBgtPWSuvd1SCU4Xdsn9PhKH0iCLZpxC4KVqDp/RI2x253k5kjWfBeZt8HjKnew80XhsPX3PkjvsgZgDvnW9iFfU9409cceAkE4r2bsyTwN6BTuSB5lxHLM3kLFgO4OLI1wBDjcguh/kK7ONcr4u5tWZkzE+9bodukV78P1031MMiGuAtmD/VRtGD+v1cPSlbWkGVxwMwfbLQ86on2icyUXXCu8bOCnja0tRrC2WvYadQ+eWax0p6j3L6pVNc4pAClbWMWUz38UfU5b4HQhTL8qi8aPS/VIdgay4T0mBr3dSlFAEb9L4ojRlECLicGwnkTguuCYOCmszs4mIeRQusYwi6rGGjzVeaA70gQSBJHihNqP+7ULKPqey4N49hbMjVJRTXxOUFFrswBzsPcddkSezjHrgzIXCzT5UlWQfomQH7z9c0pCGVWZPtcrhNdHYn0XI1mN9ZOswaK14VBsjKE1QUjoX2IC/OYw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it, allowing do_tcp_sendpages() to be removed. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells Reviewed-by: Bernard Metzler cc: Jason Gunthorpe cc: Leon Romanovsky cc: Tom Talpey cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-rdma@vger.kernel.org cc: netdev@vger.kernel.org Reviewed-by: Tom Talpey --- Notes: ver #6) - Don't clear MSG_SPLICE_PAGES on the last page. drivers/infiniband/sw/siw/siw_qp_tx.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/sw/siw/siw_qp_tx.c b/drivers/infiniband/sw/siw/siw_qp_tx.c index 05052b49107f..5552e60bb927 100644 --- a/drivers/infiniband/sw/siw/siw_qp_tx.c +++ b/drivers/infiniband/sw/siw/siw_qp_tx.c @@ -313,7 +313,7 @@ static int siw_tx_ctrl(struct siw_iwarp_tx *c_tx, struct socket *s, } /* - * 0copy TCP transmit interface: Use do_tcp_sendpages. + * 0copy TCP transmit interface: Use MSG_SPLICE_PAGES. * * Using sendpage to push page by page appears to be less efficient * than using sendmsg, even if data are copied. @@ -324,20 +324,27 @@ static int siw_tx_ctrl(struct siw_iwarp_tx *c_tx, struct socket *s, static int siw_tcp_sendpages(struct socket *s, struct page **page, int offset, size_t size) { + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = (MSG_MORE | MSG_DONTWAIT | MSG_SENDPAGE_NOTLAST | + MSG_SPLICE_PAGES), + }; struct sock *sk = s->sk; - int i = 0, rv = 0, sent = 0, - flags = MSG_MORE | MSG_DONTWAIT | MSG_SENDPAGE_NOTLAST; + int i = 0, rv = 0, sent = 0; while (size) { size_t bytes = min_t(size_t, PAGE_SIZE - offset, size); if (size + offset <= PAGE_SIZE) - flags = MSG_MORE | MSG_DONTWAIT; + msg.msg_flags &= ~MSG_SENDPAGE_NOTLAST; tcp_rate_check_app_limited(sk); + bvec_set_page(&bvec, page[i], bytes, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + try_page_again: lock_sock(sk); - rv = do_tcp_sendpages(sk, page[i], offset, bytes, flags); + rv = tcp_sendmsg_locked(sk, &msg, size); release_sock(sk); if (rv > 0) { From patchwork Tue Apr 11 16:08:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13207820 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37BBFC77B74 for ; Tue, 11 Apr 2023 16:10:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 778BF90000D; Tue, 11 Apr 2023 12:10:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6D98390000C; Tue, 11 Apr 2023 12:10:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 552B690000D; Tue, 11 Apr 2023 12:10:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 4326C900002 for ; Tue, 11 Apr 2023 12:10:03 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 05F48120D94 for ; Tue, 11 Apr 2023 16:10:03 +0000 (UTC) X-FDA: 80669596686.09.807D0D1 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf20.hostedemail.com (Postfix) with ESMTP id 125031C000C for ; Tue, 11 Apr 2023 16:10:00 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=fW7FwjWH; spf=pass (imf20.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681229401; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WkdEEEXdr/3qYwqcKdd5Ao06l5CmC0hih/jlnjAFNWQ=; b=VXVjzN08Z3t/EeEZXYLRqNvSNzTpr3IY6dlpwwnX1KYrrnDB/+IB36aCqYZGg0usY3lTn6 3KVp/2sTw4Mv3PYcghkDPWsj3Pf7TNw6XPA5E3vZ7OIhae7xKbbUZTgFjFtddIPrIUUXC+ iFezjqpOkHAzIJe1vX6FwyyeOqxoaaA= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=fW7FwjWH; spf=pass (imf20.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681229401; a=rsa-sha256; cv=none; b=CvQtNYH6Q08WjCMu9nwdSP1StULWVTIWVpdY3gfuBak0i1tGdiyj2pNkQVeqtlfWpgDdvv e9S9FFohIzOyiLYKGauzwvn0rwYb1507ZAIMdf7sKnkOMlMOld8wIhjJ9DhqV8xXOGPMjr qtgx27Ao++LRbfL8oP0iBWZq7huv13c= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681229400; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WkdEEEXdr/3qYwqcKdd5Ao06l5CmC0hih/jlnjAFNWQ=; b=fW7FwjWHbTX0vrMWciM4WxtstED6IYg0dTM9/IHNAp/UvOD0ndCngkEdFjQLnLVeWp0gWO KEl84czKk1sKSwI9kAVs2bzLyxtUghL9hbqyVwdImLCujopHqsziVBiMGaagDVAbfJyLe4 ku4U/H1NWxY6WB3dFEjSIlUaOumaQis= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-74-AxOMgkEwP0O6PEpafr2joA-1; Tue, 11 Apr 2023 12:09:51 -0400 X-MC-Unique: AxOMgkEwP0O6PEpafr2joA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 340FA38149B1; Tue, 11 Apr 2023 16:09:50 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1948C1121320; Tue, 11 Apr 2023 16:09:48 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v6 13/18] tcp: Fold do_tcp_sendpages() into tcp_sendpage_locked() Date: Tue, 11 Apr 2023 17:08:57 +0100 Message-Id: <20230411160902.4134381-14-dhowells@redhat.com> In-Reply-To: <20230411160902.4134381-1-dhowells@redhat.com> References: <20230411160902.4134381-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: 16trah56shuwef8syikqet7rjmtjzh6w X-Rspamd-Queue-Id: 125031C000C X-HE-Tag: 1681229400-683018 X-HE-Meta: U2FsdGVkX18rdhu5YfhmrTTDdfUBAuviceObUcOUbyjMfEfYqGxb7JJLwPdCXIoeGwhVW7XTQ+1pMWqB6mdwYfVahfuXreRw+EliR+IUTr3c2xBsILg7x/2Q5AU9hb5ijzvMWXOxXARB/I8Qxb90yOquCSZffumISck9QW9IvrkYOQYcD5RqTafbzGGrXet+0iAoEjyFajGKx+mRbPWEEBbo8VTnYPmSl59sPq925eT4x3DyrrrFj7fOQxotaHzjov9pQW34fR734P+b+3Nv5CbqmOyzvaG2AzSJWrtFb7W6EawC85xsBc1AlCPdOyFeJQ4EV5g3/5cBUa6tp5w8atp31IG6Dd2mpr4UA+HjHx1uovwZC8GhOt6PKZucHIrX/VTMQajrVlnJ6IcdMxfrvDVjndOAI+Zj7/LMGDpP1AiFeoUVnJneXpy/TtnPdJtykZLvXQPudd/eQNKrUt++VXFPzpgfN2vaf5sN/HOrUfPtxRvlMEY7ouEvOuhwlQ80yyI1poy5bGbRC0ghC1h5beYuWh1Jrmju7kmPunfZDKPJ2NJdCrUGCNVl6k8v41CRGL6aGdqs/2pURJfeG+ddlgw9C5NFvjK2QP1Gf8SPk9QSRZLge1wzJZiAtX3CkapNmzCq1vY5MKzwwQzIishcSwhySj2yYBNT5peLbt4aM1Wg4nazxRdMrdVkdn9ut3xGoTijh3hPrU8CLQIdwnefkDeLklKZ/vftlR8vH9PHQp0gwNZss83YLkeTWqh770MRtFjB8cXysOW+SWKuWLgdgrXfT5QkFaYibLtOOmUSvSYlv/DAB7j3GGrxVtsOwcCcJ+SdGZ4jiU4cVe/ev02HjWj5aehcePV1+KVFW0MLRlbDmGaa3sSxI0XKg4Nk+ydiJd0Bxl8EYpRwCVRtLkD5Meme6Fjw67jj1Lkh/u6WvF9huAMfN3sVZK16dkTXti/MqfZzmMWRbpSsV3KIQvx Ihwrp5yM zL91R4BVOFT/Gyo9paqAynM3ryPAobr4rdfcD5Jqh1QBe54IJTWpYvUkfCiWJS6eoWmwjWcPdQZiKc/IMmo3PuEC1ZngdpYGbuDo8myXmdUkcar0tXUt0iZbs+NF0ayrkapDqjXMz99zpV/fe7Mfo1bSTXvqrwkGcx/Gh5XIJdjztR5kcTpn5KI+n4+efofYR1rHQz0k8ysu6y5vDSxYzuJLILA/y9LhxmR4QkbCCi4QZ3OkNIEzNiKdZyFsoNqTkz7J61knk9SXlAMxjiS+DUM4Xfbqzf3DTl18ZaGKu1i3WJDvKMXwvWSSlKDbwHPiYlRhUlSMggQoHuq2YpVJ4VygYKsX/ghU+U9fm7EPKVfTPtjfKojVGw2fd4PKOrvud1P2pzUO8eGTyUMaeaHeridYcU4hCLb2FO6cgGbBiwF4zY3GyBxHBnmnDBX8z3TDbckRR5WidkW0XL0v/GgY7uDmlkvy7KCIrLmVmgpgENKi4FFE3PYXLR1a+EDx31Np54aJP X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Fold do_tcp_sendpages() into its last remaining caller, tcp_sendpage_locked(). Signed-off-by: David Howells cc: Eric Dumazet cc: David Ahern cc: "David S. Miller" cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/net/tcp.h | 2 -- net/ipv4/tcp.c | 21 +++++++-------------- 2 files changed, 7 insertions(+), 16 deletions(-) diff --git a/include/net/tcp.h b/include/net/tcp.h index a0a91a988272..11c62d37f3d5 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -333,8 +333,6 @@ int tcp_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags); int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset, size_t size, int flags); -ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, - size_t size, int flags); int tcp_send_mss(struct sock *sk, int *size_goal, int flags); void tcp_push(struct sock *sk, int flags, int mss_now, int nonagle, int size_goal); diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 0187988514b1..8f39e6024768 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -972,12 +972,17 @@ static int tcp_wmem_schedule(struct sock *sk, int copy) return min(copy, sk->sk_forward_alloc); } -ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, - size_t size, int flags) +int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset, + size_t size, int flags) { struct bio_vec bvec; struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; + if (!(sk->sk_route_caps & NETIF_F_SG)) + return sock_no_sendpage_locked(sk, page, offset, size, flags); + + tcp_rate_check_app_limited(sk); /* is sending application-limited? */ + bvec_set_page(&bvec, page, size, offset); iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); @@ -986,18 +991,6 @@ ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, return tcp_sendmsg_locked(sk, &msg, size); } -EXPORT_SYMBOL_GPL(do_tcp_sendpages); - -int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset, - size_t size, int flags) -{ - if (!(sk->sk_route_caps & NETIF_F_SG)) - return sock_no_sendpage_locked(sk, page, offset, size, flags); - - tcp_rate_check_app_limited(sk); /* is sending application-limited? */ - - return do_tcp_sendpages(sk, page, offset, size, flags); -} EXPORT_SYMBOL_GPL(tcp_sendpage_locked); int tcp_sendpage(struct sock *sk, struct page *page, int offset, From patchwork Tue Apr 11 16:08:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13207819 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 50FB6C76196 for ; Tue, 11 Apr 2023 16:10:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E538590000B; Tue, 11 Apr 2023 12:10:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DDBFA900002; Tue, 11 Apr 2023 12:10:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C7CEC90000B; Tue, 11 Apr 2023 12:10:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id B70BD900002 for ; Tue, 11 Apr 2023 12:10:02 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 62A3240E95 for ; Tue, 11 Apr 2023 16:10:02 +0000 (UTC) X-FDA: 80669596644.19.74FD634 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf24.hostedemail.com (Postfix) with ESMTP id 7CFCB18000F for ; Tue, 11 Apr 2023 16:10:00 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WuaAYZ3N; spf=pass (imf24.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681229400; a=rsa-sha256; cv=none; b=wdqZ6hX/DhhNHMpdr6oht2tF2kQxHCA9arwnhY5u0nljSnDgyAd6qiCEVC2BVcjU19GqfH rV/dL3VczSKIqSGq0+fZ/yUnflm6vFjppV0pJQH4rgb4sNXhoCTInwVNshBMrEHFUMCwIN 3cbcWQG0XhuBCmIPLnx3rqXnMHVD7ug= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WuaAYZ3N; spf=pass (imf24.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681229400; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ixYp8U9bxWbRy0uTYM5jwtBZUcxG981BMDx6Mvgz/Cw=; b=QQ5+oICtlrAHCOBPxnTnatesWjYxqvUh9W69o4nS7XdJISwXOqsCfz906mf+gIgAWRgb/A CsKEUoNE35rsz9/pmiQMCfF9biF9Wu4I4KbsoITrxU9XV9AwYHNMDHD6ixXovaAE1L18GS vlD5071AIFmbOLvIglJqv9cphLB1uSQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681229399; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ixYp8U9bxWbRy0uTYM5jwtBZUcxG981BMDx6Mvgz/Cw=; b=WuaAYZ3NT7WFuCW+k+XLr9wmf0D7M7rxm1AGwjNWkuOWwQ8Hhm3ZPNGVTBw1ugBOVaHh1G h2VIOgcGvBETvusSdPq0nxiZBc8rvWo4+rWccoU+A11/B298lxijsnViUqux2kaRvKn3x0 r5Su1OvsAe8JMMaltE4oMMrhMoYvKGo= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-668-6XnKugzfOdqRWqhI-2hRlw-1; Tue, 11 Apr 2023 12:09:54 -0400 X-MC-Unique: 6XnKugzfOdqRWqhI-2hRlw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0FCA51C05143; Tue, 11 Apr 2023 16:09:53 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id E6A1A2027040; Tue, 11 Apr 2023 16:09:50 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v6 14/18] ip, udp: Support MSG_SPLICE_PAGES Date: Tue, 11 Apr 2023 17:08:58 +0100 Message-Id: <20230411160902.4134381-15-dhowells@redhat.com> In-Reply-To: <20230411160902.4134381-1-dhowells@redhat.com> References: <20230411160902.4134381-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspam-User: X-Rspamd-Queue-Id: 7CFCB18000F X-Rspamd-Server: rspam01 X-Stat-Signature: uuzby5bcmj19tnz4ekg38o37s8coinnm X-HE-Tag: 1681229400-777937 X-HE-Meta: U2FsdGVkX19mXgRh2Mg4N+Vgd38cH9pSTttUIpxaP6LLFDwtjID+fzXzHXZECYQy9c+rKQsYBjk85O48zlP9sz1D1KXynJXEzZBztTPH/wc9csTsre9XDn2jdKaKFjd02oOd8ap6xNoYSWilxgC5c7J427p7TFQBMC1/Ul76ONQLcQnx4kJgsNkETSqdJkqP1OVUhc6loKq2m85tbOKfzPB963Yn1zDTpLAbNWp+MJrqF9S96jYnZfIZxQrhfvZJSz/Z7VbYS5EfEVSkSY15uvvL+KfvKsUefEBdhzEyS2STyGF0/XVtfTqQ5p3nJWV8Q4fKt4IY1BJDJPzgOJbbtRR6NCupveP0OYK4e+w5VnKol0ff+7DABWHDecPHjxzbiJWQazAoePXDOvbTthYFJ5u0zhkMKOvwHiF0EenyWB9b4XpdTyz0XQxzLwMSZQFYNjFbvSERMy/1jl9epUzl+Jai5YIZSLIk06nwLPUmJMe/JRkxNJWC+loMlQpB00c2YO5DtNMEX8beZpI471Z8G0mkjkeBYJGYQC31u1YUVuYB6vWEq56px/pibdPIHl2q2cKwvnNU+PSJoLMnLhh4y+6gWUfqjMu7eNpqU1viOq6mbU75JXjjNAP0af6yhYsRJ5vE/jCbxLBYgNgdOCvKCjIUiDVnCwuBDd5fRxb19NtyWmu7znL45rmoLpAyrH7lfpdZZ6f8pMJX1rp7c+jBEmVm990kwdoF1Mq5EIwSbsxYA7OOQLVaOpryYAolPwkq7XbjvIxEjKcvKfNrREb3v8DLQXJQcl+aCKXqbrMP91ypQjtnKUe3vMYMeu1yHiuy3ccSHXxrjyC5s/i9u9Vl6qfK371M257yy+qOZuWq25o5y5DbGbjlJRS889phzcCg7NaTggGzVZ6A+EtAgcMv7AbNo3Wrs5Qm/2Hajii1zKFwacnYm/yoLWxtGe0AgchLb07v78+4kZtXhKK7I+m rq5fifuo q9O8+A0NT8BlZomFw2rzcd4UYCtgmSqiQFX8S+9k/0KW+4eFh4zm3ZiT1TG10JHi7AVOjL1Y8c5uxgVI6AxzQyG6pYwS5v7yhZTsyERzXK3FHot4RbLczfmxy2QETCZFBi8+zLzUHl9iInHlxQpSIO9g6sxr+VOqGdossOgzqXSB6hpb9dGGBoR1nXMLWXfOBmjwbLun8RYrzpIZIZ6dQvnqRCoCacF5MR9nLYQevLTmSSUTQb+FD2HQBbU8Rcz+ow7pu8ho9JeGzdZwa5gBzBf3ncspFntors+ItkVKvIxZSs5naVsiUQg2Zp9oWD26zkuiVz/eC44yoCIzZ7/daWKO7KcEi6UkgEWtHtmvqSFJ3166Br5QfhaZ1zZyjHE6tEgzaDKZjKW7WBe/lKDINUv0JkfLfDb8t2Rt2SENnwgXtaJVbLyuZQQ+Au4wozUWeJK8X7sFQMqZWSOATTqCOlcD6w4rPPA0n5x9dsEQYYDIw6uW9wlwaCxI1Hy1w+ufFeXTVj1rXGDcgUQO9Nwuq8EWhJQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make IP/UDP sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Willem de Bruijn cc: David Ahern cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #6) - Use common helper. net/ipv4/ip_output.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index bd002222cf2d..b594da6d6bc1 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -1048,6 +1048,14 @@ static int __ip_append_data(struct sock *sk, skb_zcopy_set(skb, uarg, &extra_uref); } } + } else if ((flags & MSG_SPLICE_PAGES) && length) { + if (inet->hdrincl) + return -EPERM; + if (rt->dst.dev->features & NETIF_F_SG) + /* We need an empty buffer to attach stuff to */ + paged = true; + else + flags &= ~MSG_SPLICE_PAGES; } cork->length += length; @@ -1207,6 +1215,15 @@ static int __ip_append_data(struct sock *sk, err = -EFAULT; goto error; } + } else if (flags & MSG_SPLICE_PAGES) { + struct msghdr *msg = from; + + err = skb_splice_from_iter(skb, &msg->msg_iter, copy, + sk->sk_allocation); + if (err < 0) + goto error; + copy = err; + wmem_alloc_delta += copy; } else if (!zc) { int i = skb_shinfo(skb)->nr_frags; From patchwork Tue Apr 11 16:08:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13207821 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1259CC77B75 for ; Tue, 11 Apr 2023 16:10:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9E3A8900002; Tue, 11 Apr 2023 12:10:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 97CF390000E; Tue, 11 Apr 2023 12:10:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 661D8900002; Tue, 11 Apr 2023 12:10:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 455BA90000C for ; Tue, 11 Apr 2023 12:10:03 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 0EE6EC0586 for ; Tue, 11 Apr 2023 16:10:03 +0000 (UTC) X-FDA: 80669596686.10.7D86E2A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf13.hostedemail.com (Postfix) with ESMTP id 2CA282000A for ; Tue, 11 Apr 2023 16:10:00 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=jDogx6jL; spf=pass (imf13.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681229401; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TEhCbtxSk4TMLvu3MRYCcpyvU5YZgAJ2W7fFU6gbWbo=; b=WdBpDV3+Y7JeX0SWtxU0bLlFpblN704G8gGly/IB1iT6OxQyOSLDjanH+4bQNZIQXp0x8a gbTRA5nAZKlehzZZgSD/OxUkx/JZH722hIQPfZ0u4qNhlrqImywts9RQw88tv8RWSFUysT wPn03Mv9gjruAWmB4daLbOTOCf/m0XY= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=jDogx6jL; spf=pass (imf13.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681229401; a=rsa-sha256; cv=none; b=Oa0THOQh5kaAyyA3iDJ/DwdyaUFGzsM9uxt3PJLYXq75m4bW3WaOGzW4+JvT7TgJQFSGP4 u6mSA6r3lZZqNEXSJyfceo2TxRxtT/Z5IAWygQT3oVagK3r10M2qzXAj3MhStUXG3Nt0KJ WAGbY+m561BJStldbfZcfYnyHT1D4mY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681229400; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TEhCbtxSk4TMLvu3MRYCcpyvU5YZgAJ2W7fFU6gbWbo=; b=jDogx6jLrYmnikRKHODKH31/xuW1J6Pce6xbYd/QySlQk08qQgacPuqbrsBex+WZwnMXio DvcAIBsFTnkcwkBDFW2uh+1nmW8zGeJCJl/GGZrEX+8ofqjIQQY6hMvGBwxJAIMQI30p2c KcvvJwQrg91Zi+ucd2UxSOYAC2af2jA= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-260-nNzlZ67MPNC336uggXNYTg-1; Tue, 11 Apr 2023 12:09:56 -0400 X-MC-Unique: nNzlZ67MPNC336uggXNYTg-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id BE0ED2807D62; Tue, 11 Apr 2023 16:09:55 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id A2A98C15BB8; Tue, 11 Apr 2023 16:09:53 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v6 15/18] ip6, udp6: Support MSG_SPLICE_PAGES Date: Tue, 11 Apr 2023 17:08:59 +0100 Message-Id: <20230411160902.4134381-16-dhowells@redhat.com> In-Reply-To: <20230411160902.4134381-1-dhowells@redhat.com> References: <20230411160902.4134381-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 2CA282000A X-Stat-Signature: pbgwddzg1o168zx4qdfkoz5mqx3zaskx X-HE-Tag: 1681229400-96543 X-HE-Meta: U2FsdGVkX18rkgpuaBO3jIHfBMB2HDgpDxl00uXAZTG9c5VEfTZZpBo5Cou01+MEwwVmcvNNZ3aGAtc6m+XQyy+FDf5A00ITHuLpi8xWX9knLzZ9t9gVyzlySweuPuVNNzBbRrMNHCaihZqpaAHrnlv5PkH60V2VJo26gI3z9Rm+Zq2RQFfr9m0HvCCx8ygZvthw+h2GlmQ37WQzRf/WsJC7oDGGVdwoH5IffROq8T9XylXf5dTTpXdKFk5ev6DBlOLPUrc6ms/d08Y0lBf7hUDB2PsB1tC+9LP0AbVGndIAy3HbXiHI4ol1gcwW0NM9zF8SFwu73OEX0kChvuyBNGIRgi8coyyoBFFQTBWrjtCEEaEWmu0qvdlON8x3oCXS500YyGdsYBiXiWuSGpidp2iGxD1Liv2AR0NRn8J3YCFIh1XlBAee6sXIIo0PHOSWzWWbKKMcvX1RawtEvg1L1cCdjIPPTBbCnL092Hsz4MN1MVA3+ZYYNSmdV0PzHovt0WzN4R7138l4erF1gwnZRaH25NUeqmJKshJy5fCPVs8g4DsDpyXUM9zQHusXXGDm3shCcMOgdFs6GSQQw1mT1QnumxnlUEeJLoUrcqxNPz1Uy05VzUv80aPZP3MP/I7oXLSqIl+VCQU/5a+m1+DBZeL4MRIEINgxvD4cbt/PMftsii8F4IHXDO7CuTxIykUCigNG27wxsF+8l/hzSqHpTUQJ3NLakNK85VF7gXRPp+maaEhb8pNWtAsqKyyK5sfxkak/dw748DjEykd19a8BtkK/CUkbMwBMces3HGDFJZgwEPhfQ9qUecKzWux40fWk2og1K3JeOb7QV7/ZMOgCyIJIk3jg97DDWjCUevqXNROM6e61t3GPk9PMhYldJ+XqarpV5R2sfGY0UXsedXwO2yEz/s138924A1UMrA1nyZTIkSHGEwRNIUaMsut8XofkaJQMHv1JBUjeXA01mVD /V4JT0fv 3cBy01dzqaxIqzEbHcry59xP/KXm725VmPA12zy6BFnKMYR+vwsaHvrqe3EUaCeobmOjXDYISIpzQO88zet7sBpIs3O1XmfM+2XJvLDlhMsN9hfVUsaVpmh+LLRKgUClycJEX+j3tUD10XsgoogLomhKo7ookd8CaXx8axDJ6DFyo59sFHeCqxpTXmm9dpoCbWQteeccEw9nXKT8SbZIn1OBtJ/pQshT1fqM3Bd+ZbbkthMWqXlwG2gMfYO23I6z7kcoVcaQLEyCMm+7Jep0/7FRS1Ynui0ATpeONQF6ZkROGSb8I5xzBMX/jKa/5b/ehOoPQOt5EBKzyMrFRkaUd+4If36Nz9d63ZcxqU1nYRa66Ii+PLJOsGa+EtH1ljZm3igDGTkRu4PFsOSCdMfm63083ZdfvOeXQLDNYYb9XN3CjfcJOukmzrjLY2Fq1zT6jaaC0tUUjPUw4EcU/Hf2lGrsDitX3v04fGOETHtxDKqS97genmygrnwfHKwLWwVppIqTqBs9yNWoAzEglkGDbDydFYA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make IP6/UDP6 sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator if possible, copying the data if not. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Willem de Bruijn cc: David Ahern cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #6) - Use common helper. net/ipv6/ip6_output.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c index 0b6140f0179d..4709abee5966 100644 --- a/net/ipv6/ip6_output.c +++ b/net/ipv6/ip6_output.c @@ -1589,6 +1589,14 @@ static int __ip6_append_data(struct sock *sk, skb_zcopy_set(skb, uarg, &extra_uref); } } + } else if ((flags & MSG_SPLICE_PAGES) && length) { + if (inet_sk(sk)->hdrincl) + return -EPERM; + if (rt->dst.dev->features & NETIF_F_SG) + /* We need an empty buffer to attach stuff to */ + paged = true; + else + flags &= ~MSG_SPLICE_PAGES; } /* @@ -1778,6 +1786,15 @@ static int __ip6_append_data(struct sock *sk, err = -EFAULT; goto error; } + } else if (flags & MSG_SPLICE_PAGES) { + struct msghdr *msg = from; + + err = skb_splice_from_iter(skb, &msg->msg_iter, copy, + sk->sk_allocation); + if (err < 0) + goto error; + copy = err; + wmem_alloc_delta += copy; } else if (!zc) { int i = skb_shinfo(skb)->nr_frags; From patchwork Tue Apr 11 16:09:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13207822 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 505DDC77B6F for ; Tue, 11 Apr 2023 16:10:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D69AD90000C; Tue, 11 Apr 2023 12:10:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CF397900005; Tue, 11 Apr 2023 12:10:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AA99C90000C; Tue, 11 Apr 2023 12:10:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 93DEB900005 for ; Tue, 11 Apr 2023 12:10:07 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 21205A0D10 for ; Tue, 11 Apr 2023 16:10:07 +0000 (UTC) X-FDA: 80669596854.25.7F860E2 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf28.hostedemail.com (Postfix) with ESMTP id 1562FC000B for ; Tue, 11 Apr 2023 16:10:04 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LV1JTcfR; spf=pass (imf28.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681229405; a=rsa-sha256; cv=none; b=eNqE4l9ctkACrryVfw1Z2S4KPwkb6ePuD2VCLlVAbCR+Jv0IJZ3nju13OKG5drT8+15X7I XsyN8ktocvd0r/nLLUrsBaRePdAPy/6nOUhjzMWE+6s21NaaKDQSoXG7bNZASt64XGpI/O 6txaxFWB7wktAKB2hkPHaKTYKF9gvYA= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LV1JTcfR; spf=pass (imf28.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681229405; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=arPLIYOMzcdUPWLjuF22tHo0vP4jVzFFThikgCzP+CE=; b=2CuPZo+CKFoPD9dtCl++UYutcfpCtHs7pLfhqvXeAG5Ntf6o8mD8Nbt69DS7KREGk5TmIn UpUTGyY10gBL/6XeeqVDXwLsHAuKpoZ0oRhsuUMqiBBkg9cQph28Hji/Sh1G4Ol68WIpE7 6NBweumM7ZT17TAByP5RzeKL7vWccEs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681229404; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=arPLIYOMzcdUPWLjuF22tHo0vP4jVzFFThikgCzP+CE=; b=LV1JTcfR/6Kw0N4Gd/STvV8OTD269CDWekAmtJN0Nydjtd5P79JyuoFEPp2i4QkjBeGYcK qTt0FXfN7rqGFqpgRWdezSlGq58IxHK5696uzJpRRH+m/KqYc3v239FHGlothX7o5AW+by /G9t2roaXkMODoFfK4MPLmYZdHnpqNs= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-224-FiOc-vDHMJaCmdO7Voy_uA-1; Tue, 11 Apr 2023 12:10:01 -0400 X-MC-Unique: FiOc-vDHMJaCmdO7Voy_uA-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8FEBC38149AF; Tue, 11 Apr 2023 16:09:58 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 75192C15BB8; Tue, 11 Apr 2023 16:09:56 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v6 16/18] udp: Convert udp_sendpage() to use MSG_SPLICE_PAGES Date: Tue, 11 Apr 2023 17:09:00 +0100 Message-Id: <20230411160902.4134381-17-dhowells@redhat.com> In-Reply-To: <20230411160902.4134381-1-dhowells@redhat.com> References: <20230411160902.4134381-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Rspam-User: X-Rspamd-Queue-Id: 1562FC000B X-Rspamd-Server: rspam01 X-Stat-Signature: s7eprinuxyoo6gnjgyqbwc7osfui3d4q X-HE-Tag: 1681229404-623992 X-HE-Meta: U2FsdGVkX182mg0/XMvSiuoGqoF9J/SoKMkC8q/7k27NtWoOAB96eIdazENOlBKUrmSMMDuH+a2z/DoGtTimjW9cx+hEuzd74G5nq45lFWQE6Mghbbb77A1+E3KA2Dwq7DaXfg3fcnQm1ITUo1W3nUzV7aOxIpX0yhtrMkGnLiec7jLBzRDsk7wls6kr9XKd37ZrZWDivxnP6UwJptsaNiCM77FiUpaT2vPXdW3g1MqM3K5S3j84Qu3TzGO7PtHLzpn8OjQ/B/ORYSDBkWs2COs5/iMTGTiSkFwshxJTI5zdTP3pTgtDNIlF+d2UugTq76FHApK0kkL5Gdcb+WZ3Vu2teDlaJf2g0fOOee+5zhSwu1X3QXx/7W1ljYXaJdQ8czgHr3CLakwYXVLYTk/pEf43jzX3zVrUlAe7ZvQ9pGms4qh7FWWVx27G/VDzkGHoARHthKSHtiemEDwKLB9V+o1xkEHpHj5xHFH4AiJBCUqJvU0LIA0Yp3R3owKa8qHQqfbFAyrZQ01tkvcS0RptupYZLFbLbI0d+HUe+7wIDezdFhK1J3GDC7KyXdvA+qp0v+toIh08eMQMfQvnK5VdDT8gI2O1YwzZ+914/3oUX8coKncQjfnPqnsD07bcNJa0aeBZWNlwWK+JdQ2b6U1ODycC5HyA8AfYoL/xo9F+qJJNteYu3JkMZSM8VQbDXlFupAnLvPtqYeskjDyDeVY6ORpHUoM8RmzBnwf4BwKDfI7kvThOjR+nnUQy0uf3JUBaZjhx3/sJUKXGnwsyjcjPffGsTWyQY6BfPgPBWhceDFO/D6fXyPIVqBSsySw4AIagZ+muLW0RkNbJs2NqcAU5COQqg5foAWc1zTgMso6xLf3SNbye4w0QT197CguZxTe46JtYLyuIYQbVH6sjTA2xdOo0ryusq3WVbr/pMxRRRgGCZ4+fMhEjll4bmSIUz71dQq7F0bJ4QT9C5KSAjLH dpU5cl61 VsJr5fcbVC1cXkPKxa9kWdJKMyejjEQKYCeEh3CyzVY2KgZhrLX+J2A1g9TRGNn22kuMSs2RePhecPQaHHfDXzx56chD/QUNhtt/jvewGfzu4GZ5qbyPiW5tSUqXbO6pIUp9kvoDNPvYGf+lyVgSD4Lg5rsqYTxzjNbqJo0A/fM9K3k3nRAjJ0Q/w1BLbwX66b28/JeQ4A2XJkhQIaBl3CqaKjTpL9xyydM6/ilk73qjD6NRlI37JXJGMqqxuFo+qG4rQkcUjuME3zZIt8FaEA4Z+dfpPerS78Qrihfy5ioBoC7ZPomXElXCS+gQs8syMGlDmzHqxs2G7lLAwgiSulwjhxidE8AfwKaCCpKni5CHky3CmNm9lItKB9Fu+EgeFLjvBboCwVeIpHU5txpwcnOccSX3vCtxDdmQp7RAa30Ik4YpeMH2G4hwC1C1FiX74WPnPQWxFnBosOhMRPmpRy9NtEQCvZs3R5vozbObyKZWudn/6ex5GO5xV+xzQTrD6/1lyWlpdt+A8goJN+ESjtboFzA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert udp_sendpage() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Willem de Bruijn cc: David Ahern cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #6) - udp_sendpage() shouldn't lock the socket around udp_sendpage(). - udp_sendpage() should only set MSG_MORE if MSG_SENDPAGE_NOTLAST is set. net/ipv4/udp.c | 51 ++++++-------------------------------------------- 1 file changed, 6 insertions(+), 45 deletions(-) diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c index aa32afd871ee..2879dc6d66ea 100644 --- a/net/ipv4/udp.c +++ b/net/ipv4/udp.c @@ -1332,54 +1332,15 @@ EXPORT_SYMBOL(udp_sendmsg); int udp_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct inet_sock *inet = inet_sk(sk); - struct udp_sock *up = udp_sk(sk); - int ret; + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES }; if (flags & MSG_SENDPAGE_NOTLAST) - flags |= MSG_MORE; - - if (!up->pending) { - struct msghdr msg = { .msg_flags = flags|MSG_MORE }; - - /* Call udp_sendmsg to specify destination address which - * sendpage interface can't pass. - * This will succeed only when the socket is connected. - */ - ret = udp_sendmsg(sk, &msg, 0); - if (ret < 0) - return ret; - } - - lock_sock(sk); + msg.msg_flags |= MSG_MORE; - if (unlikely(!up->pending)) { - release_sock(sk); - - net_dbg_ratelimited("cork failed\n"); - return -EINVAL; - } - - ret = ip_append_page(sk, &inet->cork.fl.u.ip4, - page, offset, size, flags); - if (ret == -EOPNOTSUPP) { - release_sock(sk); - return sock_no_sendpage(sk->sk_socket, page, offset, - size, flags); - } - if (ret < 0) { - udp_flush_pending_frames(sk); - goto out; - } - - up->len += size; - if (!(READ_ONCE(up->corkflag) || (flags&MSG_MORE))) - ret = udp_push_pending_frames(sk); - if (!ret) - ret = size; -out: - release_sock(sk); - return ret; + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return udp_sendmsg(sk, &msg, size); } #define UDP_SKB_IS_STATELESS 0x80000000 From patchwork Tue Apr 11 16:09:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13207823 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A690CC76196 for ; Tue, 11 Apr 2023 16:10:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CF3A0900005; Tue, 11 Apr 2023 12:10:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C7CAC280001; Tue, 11 Apr 2023 12:10:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A816690000E; Tue, 11 Apr 2023 12:10:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 8F45B900005 for ; Tue, 11 Apr 2023 12:10:08 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 61E131605DA for ; Tue, 11 Apr 2023 16:10:08 +0000 (UTC) X-FDA: 80669596896.10.A960757 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf22.hostedemail.com (Postfix) with ESMTP id A6183C000A for ; Tue, 11 Apr 2023 16:10:06 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Jiv24f9K; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf22.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681229406; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ARU/CvO1sKmmPal1qpPfpSnT+ugsfyzFtga1wKQzNmo=; b=EuCdeV5iiCiMenS0BhYBlfgS6QMMcg2z79qNPro7/MeCg/g9lUf6bp/NkbFa8DuLpDBgdr BJQl4I7jdH7KbNZlyxmax9AGTaTmt3oUU7f5u+mtiAHAeb9PXzTlxh+Af4VyeZaz7tbtAO 6IiHhgJHxWLxEEJqHfVUqfsFRMzYcq4= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Jiv24f9K; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf22.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681229406; a=rsa-sha256; cv=none; b=8SF7huUbZNtIjct4XKZVGeoOHoKcAkSmNqe8N43wwQ1FrRPQLQ8QSeenYTw8UQqYksMG4J uZtDGMpPkw8kDO92IkuIkOmmMxsVhs2aOioS4ctRVgccNf7McInvqUiJXi74rDGI40wSFk tWCbYQbfuJMtabnkpHZTgZkH62aZPYc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681229406; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ARU/CvO1sKmmPal1qpPfpSnT+ugsfyzFtga1wKQzNmo=; b=Jiv24f9KxTrQi7ADMn1DW/gqnhlxuXKmMcdSgjogKkQOZVmFvpnhjqgt6aR0tHlqBEz4vF PPErbqMuH5g5nEYCzJegzg75gbWHg4ZkLn325xpuRJ64BKquffRYUAN8p5TpXbYNOGonns wJBOx5kAFWCowQ2+3zsPdRSM7Ss0eIU= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-658-J-1BN9gyNjm90d1w1KToOA-1; Tue, 11 Apr 2023 12:10:02 -0400 X-MC-Unique: J-1BN9gyNjm90d1w1KToOA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4E66E1C0896B; Tue, 11 Apr 2023 16:10:01 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3155B1402403; Tue, 11 Apr 2023 16:09:59 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v6 17/18] ip: Remove ip_append_page() Date: Tue, 11 Apr 2023 17:09:01 +0100 Message-Id: <20230411160902.4134381-18-dhowells@redhat.com> In-Reply-To: <20230411160902.4134381-1-dhowells@redhat.com> References: <20230411160902.4134381-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: A6183C000A X-Stat-Signature: zzkhsx56kuhjypm3odum8c8fpg6o5gz4 X-HE-Tag: 1681229406-982633 X-HE-Meta: U2FsdGVkX19n4rRVH1aeUkT20mPqDYlzNlJFnqVFcCKslylPCLgMQ9ZLYSzBBDQ6RhKfOPRx+9nnQdmgfn4qmCcmAfvhusrRK9OIH0wj6bK5zoDxFuDoJLMuumtbNIHYTGBlzZCsprEU4TWx6DIVYSjQVGZsdmRJ36Qp6Z7r+UQejIX2w9HARJFdvR8bZ718opmrMSFnOuWdrsNVUMNDqWW6sCNgUAgx6mJfleSHMlc3mRACULFA7TJ7UVy/NkTYRm7eu6MUHC/++GPoqmuk1oZl8a84KhNcuYIOEJ/uDKu4zaNX1YYMao32S7xKP/eiX4m3TwLMKjFBxrt35ejxeb2FaPHMVmC+m3BvbEwz38GtqMtXwPJI1k2+roRZiCaaHzmTDEh5jcN4ldXAwuLc1KUgG5iImlY1kWbKq4soMrSCZ2pS5uBtuRu+wEeLWwpruxKWzXIsd5K30TCo/1cHUkoCP1xG2uMH3ZD7wwb1aRu9ib6/lVCkJe1G5TA+KiAIb/zqXkoM5VZH/oMGustz2g0ao3iG9yCDay0fvZ7mGiOX6tagJG18EowqddTfw3ivFc0rNKGvoQ1ezgL3hK2gZ2xxSvTbWqCYj5tFor+lxX6nM53STtTCfSdAep7gP7/Evm/waukmJbPp8bYq+xd0c+CHg0wYeGjXgvG0Nuc24qMMfh7pdDhJmnH74602rOPjV1CWQ8JVZ0wtxXz7N6pyPwrGhuOAOLwtXlIoq1fCjwrU8lu2B661XMMajb5emDUqauzskYEQfKzC09AI0G27zAePKrUCcHXQo1H/yZf6om6n36K/GHqunY7ChGow8rKBbiF10Bq76gbPiCa8BwAWvRn9v+dCB+kaWL3oQLrUZhSNgRdAkulpNJhLB+EUbILU+ShMcOXA8AAaDV9ZVeGXL5+RjdqeoQL9GL5Vpum4xhdR3eV4qeiRNhcGhjuDDk/a5tAfmy+8fDHkjCpQVUl 9VIqDkhH pgXrkzGw2LUev+BZkub7eh7+J97cixMj6w0bof3cXCxqfZeU1U5SvNa/6xSbf7rd5sSaAKG/ZKHLd4f5mUkmZw5avT4RiTGX4dRknpUhfOh2eoM2sxkzydvGrBt+ap83EkvczP2yUs8x1kuS17mM2yLWxrNTPhgHjTN9EHE12A43BEbzDDLWoEP+f5VGxQh9MAUROjjSje+crX9Oe3aJnk4VI7ILeVPvJIyVZRq55ypEzqj5tnmhz8JMNt8anQ5ncf/y+D9l1/lIpMkd1v/9sHcSl0/c0lVYzwwkrHp9UVQB4FjtCmWZXY83L1TZfln669oHAHuPdC/WMwIRramyQ13lWN6R48sCPNuR+QuB7W7kZRkKLUj5l1ZPlU0cjWSdThuNOINuH73p8JewnxLysvS2qQUl1ax1rN4/7PIoSyrB/2BnQIGTy8ja/TJc0mwbnRb4cmhvVNtiToKhqJAQGe37cAex24Wtc+GYgmtOmdJvXOJffesRGAAdoneKV/BID+hMLu2DSpgAa67k9lQkj1en+sg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: ip_append_page() is no longer used with the removal of udp_sendpage(), so remove it. Signed-off-by: David Howells cc: Willem de Bruijn cc: David Ahern cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/net/ip.h | 2 - net/ipv4/ip_output.c | 137 ++----------------------------------------- 2 files changed, 4 insertions(+), 135 deletions(-) diff --git a/include/net/ip.h b/include/net/ip.h index c3fffaa92d6e..7627a4df893b 100644 --- a/include/net/ip.h +++ b/include/net/ip.h @@ -220,8 +220,6 @@ int ip_append_data(struct sock *sk, struct flowi4 *fl4, unsigned int flags); int ip_generic_getfrag(void *from, char *to, int offset, int len, int odd, struct sk_buff *skb); -ssize_t ip_append_page(struct sock *sk, struct flowi4 *fl4, struct page *page, - int offset, size_t size, int flags); struct sk_buff *__ip_make_skb(struct sock *sk, struct flowi4 *fl4, struct sk_buff_head *queue, struct inet_cork *cork); diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index b594da6d6bc1..a02d621e4b94 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -1327,10 +1327,10 @@ static int ip_setup_cork(struct sock *sk, struct inet_cork *cork, } /* - * ip_append_data() and ip_append_page() can make one large IP datagram - * from many pieces of data. Each pieces will be holded on the socket - * until ip_push_pending_frames() is called. Each piece can be a page - * or non-page data. + * ip_append_data() can make one large IP datagram from many pieces of + * data. Each piece will be held on the socket until + * ip_push_pending_frames() is called. Each piece can be a page or + * non-page data. * * Not only UDP, other transport protocols - e.g. raw sockets - can use * this interface potentially. @@ -1363,135 +1363,6 @@ int ip_append_data(struct sock *sk, struct flowi4 *fl4, from, length, transhdrlen, flags); } -ssize_t ip_append_page(struct sock *sk, struct flowi4 *fl4, struct page *page, - int offset, size_t size, int flags) -{ - struct inet_sock *inet = inet_sk(sk); - struct sk_buff *skb; - struct rtable *rt; - struct ip_options *opt = NULL; - struct inet_cork *cork; - int hh_len; - int mtu; - int len; - int err; - unsigned int maxfraglen, fragheaderlen, fraggap, maxnonfragsize; - - if (inet->hdrincl) - return -EPERM; - - if (flags&MSG_PROBE) - return 0; - - if (skb_queue_empty(&sk->sk_write_queue)) - return -EINVAL; - - cork = &inet->cork.base; - rt = (struct rtable *)cork->dst; - if (cork->flags & IPCORK_OPT) - opt = cork->opt; - - if (!(rt->dst.dev->features & NETIF_F_SG)) - return -EOPNOTSUPP; - - hh_len = LL_RESERVED_SPACE(rt->dst.dev); - mtu = cork->gso_size ? IP_MAX_MTU : cork->fragsize; - - fragheaderlen = sizeof(struct iphdr) + (opt ? opt->optlen : 0); - maxfraglen = ((mtu - fragheaderlen) & ~7) + fragheaderlen; - maxnonfragsize = ip_sk_ignore_df(sk) ? 0xFFFF : mtu; - - if (cork->length + size > maxnonfragsize - fragheaderlen) { - ip_local_error(sk, EMSGSIZE, fl4->daddr, inet->inet_dport, - mtu - (opt ? opt->optlen : 0)); - return -EMSGSIZE; - } - - skb = skb_peek_tail(&sk->sk_write_queue); - if (!skb) - return -EINVAL; - - cork->length += size; - - while (size > 0) { - /* Check if the remaining data fits into current packet. */ - len = mtu - skb->len; - if (len < size) - len = maxfraglen - skb->len; - - if (len <= 0) { - struct sk_buff *skb_prev; - int alloclen; - - skb_prev = skb; - fraggap = skb_prev->len - maxfraglen; - - alloclen = fragheaderlen + hh_len + fraggap + 15; - skb = sock_wmalloc(sk, alloclen, 1, sk->sk_allocation); - if (unlikely(!skb)) { - err = -ENOBUFS; - goto error; - } - - /* - * Fill in the control structures - */ - skb->ip_summed = CHECKSUM_NONE; - skb->csum = 0; - skb_reserve(skb, hh_len); - - /* - * Find where to start putting bytes. - */ - skb_put(skb, fragheaderlen + fraggap); - skb_reset_network_header(skb); - skb->transport_header = (skb->network_header + - fragheaderlen); - if (fraggap) { - skb->csum = skb_copy_and_csum_bits(skb_prev, - maxfraglen, - skb_transport_header(skb), - fraggap); - skb_prev->csum = csum_sub(skb_prev->csum, - skb->csum); - pskb_trim_unique(skb_prev, maxfraglen); - } - - /* - * Put the packet on the pending queue. - */ - __skb_queue_tail(&sk->sk_write_queue, skb); - continue; - } - - if (len > size) - len = size; - - if (skb_append_pagefrags(skb, page, offset, len, - MAX_SKB_FRAGS)) { - err = -EMSGSIZE; - goto error; - } - - if (skb->ip_summed == CHECKSUM_NONE) { - __wsum csum; - csum = csum_page(page, offset, len); - skb->csum = csum_block_add(skb->csum, csum, skb->len); - } - - skb_len_add(skb, len); - refcount_add(len, &sk->sk_wmem_alloc); - offset += len; - size -= len; - } - return 0; - -error: - cork->length -= size; - IP_INC_STATS(sock_net(sk), IPSTATS_MIB_OUTDISCARDS); - return err; -} - static void ip_cork_release(struct inet_cork *cork) { cork->flags &= ~IPCORK_OPT; From patchwork Tue Apr 11 16:09:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13207824 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B2B5C76196 for ; Tue, 11 Apr 2023 16:10:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DB661280002; Tue, 11 Apr 2023 12:10:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D4035280001; Tue, 11 Apr 2023 12:10:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B93C2280002; Tue, 11 Apr 2023 12:10:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A1201280001 for ; Tue, 11 Apr 2023 12:10:11 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 45C3E1A0E22 for ; Tue, 11 Apr 2023 16:10:11 +0000 (UTC) X-FDA: 80669597022.02.3E42392 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf13.hostedemail.com (Postfix) with ESMTP id 6D81C20014 for ; Tue, 11 Apr 2023 16:10:09 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="b7m2R9e/"; spf=pass (imf13.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681229409; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NUYtJHWQHrlxATmFk6Zj0PzfCOzptdZvrIQb9VoBGYI=; b=KSdEo1IH4uRcEuA3Dz1DsASWYUFpqud6KSv8jws1BjglZSLx+fFpNgr4eptXJd0IDdHwsR DqRrAPQLFuHC9snt0BX59Na6fjmL9N+rhbCoEK/g6y9bNbR8u7XmIpbOS8QKM/7FyUlNLn CqrOa06wb92t/rjWRDC+NPFxzq8xgA8= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="b7m2R9e/"; spf=pass (imf13.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681229409; a=rsa-sha256; cv=none; b=mbdrCt3naplK1oJmeVbEcxhc6/yRc8HZP02F8c9IntGDs7IfN4VmlJu1UxQ8WjQqeXBUf7 /02Lkk43hq5pPN5wHgpcqHcbr02bC0rHUHS7ljV48r8tKcvVL4gWKa7yo8pmYtaZ+A5w09 Q1orWb3lH4pr0nFTIkw69DanIUnOvG8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1681229408; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NUYtJHWQHrlxATmFk6Zj0PzfCOzptdZvrIQb9VoBGYI=; b=b7m2R9e/eJ4PoPqlu1YgFpaLK4EVA2A0zuh4xQ0T0d28OSQ5SQR6XfIbZLoPHYj/XZKNtn LI5yODD5ZgA9+OMK88Vt4aVu/k1qvAxXlZ7/jTb+y/Q+eoqpzLu/tk1/UEczqgCBj/SLIE n/a+xyTBVQPuxNZE0g6dJ5VXEm0kr44= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-344-v6vTLpRuNC-A8BVR-fqWyA-1; Tue, 11 Apr 2023 12:10:05 -0400 X-MC-Unique: v6vTLpRuNC-A8BVR-fqWyA-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3EF7F101A551; Tue, 11 Apr 2023 16:10:04 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0B5F52166B31; Tue, 11 Apr 2023 16:10:01 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Kuniyuki Iwashima Subject: [PATCH net-next v6 18/18] af_unix: Support MSG_SPLICE_PAGES Date: Tue, 11 Apr 2023 17:09:02 +0100 Message-Id: <20230411160902.4134381-19-dhowells@redhat.com> In-Reply-To: <20230411160902.4134381-1-dhowells@redhat.com> References: <20230411160902.4134381-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Stat-Signature: rto3o9bmbzn1baihiykfwe4eskxmkfic X-Rspam-User: X-Rspamd-Queue-Id: 6D81C20014 X-Rspamd-Server: rspam06 X-HE-Tag: 1681229409-922795 X-HE-Meta: U2FsdGVkX194jRdZLVU48Qpn2GpdZhzPEZ2QINy5HQVynjbXbH27eY23yZyCBorZYvI5951xzdXKxtF1YZyJ/dLXSnamo1nRYijT7Etr3zYzwx+4wUyHbMGy8aLl+q1GAaBW+G8Xa0Jqcj9wyIOrzRnn0hxIVru/uQDmy4zP6nuFZRpgJZ6i374iXkLdWMseWr5uk5aD+L6Y7I0hzRaY1zf9gJPm/kwUg422kaVlVo72J/vm9lLRQLZTJJAPZXk572UUqQ+3loKJbsusno1q1htlbZbCkqELsLgXqhnbaKJolNEocifLBHzkF58tw5Q5vKJwgK2p6SLzWIEJVexz5b6NRIQiHGYkuFn8+W8Q3UiQ39wx8o3zkyRVImhpOV0sVy7gzFNmcbrs+fbciJoK2TwsU3KBAG7+Y2V000knL4ms1YDialRo+MW7AjvZSK/qxjRK3PYtptiZJl2DDQb4RG/gbq2LLmbITgrxqjNqMVFvlb22GQhSgG7UzZCLiYdSMhcgUNMuAxahlOZQFATQEKsR6TXtjK5bhAoLg46T2fz1c8uime0DTSIE1F9HTdrQ4ZiTAvUWu88E5ZDnYBPs8rmXprDbreVcsdNuDcTgZKDw6a+HLvXAxL847aZSNK+9Vc2ep1V65sOJXtWzYGzVg7kuBv9r5BF+76Xgj+5FPvUZuCDKxnL4Nf5pL1Kdzb5XQZTKMXJ4KVuQqZFVGRcwEbKt5EtbVoscyF5ihzrvidhCIbE+GvGLLp5dSc3BopWJuUXt6RkcGn7lQoMTU1DevPk8YVkMn36/r3l4awF2yeqrpezl7aAcMcokhUNdTke/wkr4tyKlxhpJUvy4MYSUNzCgtKB+hOXSnjCU8cPr31C2zQ2JYP+QhH374gzjaIZiY8AjnvEZCdzvsZkL2mArH0v8ypdBRNSl9udOm7RtbIkPGu2DXIV39GSm8zkDiPlZYrNVADurrAy2fVYUNGo K1OPA6oo 3OSULVIGBOtLRv/d/5FefpYRbF8Pnlivl8iBZO/5NVIjQQuUamZz8oNFtomSqXmWv9XUfPPpSdWNUSBZN9LnIPAz4sLxsFcGyFZmv3I6S4zmQqILp9F4xz4QU8FqLCd0rzYchO8xIMcZUxzMRRUBhypaRKnAdcNmVUwGSsYqGrWHB7hoZmBxksSQ7XuWLacXJBJ7XbV3FO27pk+pGVYeA6788XiWp8GRTMaICaPUhGxa5eAR1ixZM0uor3s/0DAVHRDaQTki7ZW0NTe83zCAWwqfLP6ofxEiywjPaVrvHwMv2GQVc+aOdZ23YwpX4jjWS/96QZLqBhDPEbdH/Rv39q7rgZQn/YsEd8HtW0yR+5mUhNIoNEASKg+fTDAeHbyxhYPj+aC6/KeTKL0CsL2WQQ2FMEwsiF4muSVrkZsw1NSvaF2ptH7hX3YBDpUrKpcfAy4raykXnpt9AUqlMEmJK4QE9bD9Q9BZJ2xojQTHiePtq1+Cn+oYLyQ0aLJcDKxscJ0K2suVi6IYifhdK8cFjiaYFXw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make AF_UNIX sendmsg() support MSG_SPLICE_PAGES, splicing in pages from the source iterator if possible and copying the data in otherwise. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Kuniyuki Iwashima cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #6) - Use common helper. net/unix/af_unix.c | 49 +++++++++++++++++++++++++++++++--------------- 1 file changed, 33 insertions(+), 16 deletions(-) diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c index ea45dcc80232..e1f962180e55 100644 --- a/net/unix/af_unix.c +++ b/net/unix/af_unix.c @@ -2200,19 +2200,25 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, while (sent < len) { size = len - sent; - /* Keep two messages in the pipe so it schedules better */ - size = min_t(int, size, (sk->sk_sndbuf >> 1) - 64); + if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES)) { + skb = sock_alloc_send_pskb(sk, 0, 0, + msg->msg_flags & MSG_DONTWAIT, + &err, 0); + } else { + /* Keep two messages in the pipe so it schedules better */ + size = min_t(int, size, (sk->sk_sndbuf >> 1) - 64); - /* allow fallback to order-0 allocations */ - size = min_t(int, size, SKB_MAX_HEAD(0) + UNIX_SKB_FRAGS_SZ); + /* allow fallback to order-0 allocations */ + size = min_t(int, size, SKB_MAX_HEAD(0) + UNIX_SKB_FRAGS_SZ); - data_len = max_t(int, 0, size - SKB_MAX_HEAD(0)); + data_len = max_t(int, 0, size - SKB_MAX_HEAD(0)); - data_len = min_t(size_t, size, PAGE_ALIGN(data_len)); + data_len = min_t(size_t, size, PAGE_ALIGN(data_len)); - skb = sock_alloc_send_pskb(sk, size - data_len, data_len, - msg->msg_flags & MSG_DONTWAIT, &err, - get_order(UNIX_SKB_FRAGS_SZ)); + skb = sock_alloc_send_pskb(sk, size - data_len, data_len, + msg->msg_flags & MSG_DONTWAIT, &err, + get_order(UNIX_SKB_FRAGS_SZ)); + } if (!skb) goto out_err; @@ -2224,13 +2230,24 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, } fds_sent = true; - skb_put(skb, size - data_len); - skb->data_len = data_len; - skb->len = size; - err = skb_copy_datagram_from_iter(skb, 0, &msg->msg_iter, size); - if (err) { - kfree_skb(skb); - goto out_err; + if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES)) { + err = skb_splice_from_iter(skb, &msg->msg_iter, size, + sk->sk_allocation); + if (err < 0) { + kfree_skb(skb); + goto out_err; + } + size = err; + refcount_add(size, &sk->sk_wmem_alloc); + } else { + skb_put(skb, size - data_len); + skb->data_len = data_len; + skb->len = size; + err = skb_copy_datagram_from_iter(skb, 0, &msg->msg_iter, size); + if (err) { + kfree_skb(skb); + goto out_err; + } } unix_state_lock(other);