From patchwork Thu Mar 16 15:25:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13177801 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A504EC6FD1F for ; Thu, 16 Mar 2023 15:26:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1503B900004; Thu, 16 Mar 2023 11:26:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0D9EB900002; Thu, 16 Mar 2023 11:26:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E955F900004; Thu, 16 Mar 2023 11:26:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id DA53B900002 for ; Thu, 16 Mar 2023 11:26:34 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id B2D8E406B6 for ; Thu, 16 Mar 2023 15:26:34 +0000 (UTC) X-FDA: 80575138308.13.55BF5FE Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf12.hostedemail.com (Postfix) with ESMTP id D60564001B for ; Thu, 16 Mar 2023 15:26:32 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=P9OodY0O; spf=pass (imf12.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678980392; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Jp818uqIiXwVZdp69UNUHXRADr94Nvu0NDkz6WG9MO0=; b=sxwdFrRja9itqAD4B9MKEXmSTR9fITwut0PGSAcmQcElFXSLA3IW7AylrMa1ugt799GRh+ +wJPIfUkVRp0JTzMuV78KyJ4FklumxPThM2I+4jUH5VIC4AJukVzElFOnT+xaiAw/D1/bW rX2buUjg/PPqqa5BAwFnyxaTFmhehM0= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=P9OodY0O; spf=pass (imf12.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678980392; a=rsa-sha256; cv=none; b=FbDE6uXinWzsXbhA/R4FwtnrtRgAUhLrLE5iSp7BrjqsOv72OT54L0uUFRLfCqQqueZ8Hs rngPwy64zPp7HvCcMnPKjL7Yh+V/Ks2MtGNBr9BVny/ktdzLZpQEKb14PDdXywQ8ZtYamw pDlO5ULOObPLMKCAA8xaz6pdezUj9rU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678980392; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Jp818uqIiXwVZdp69UNUHXRADr94Nvu0NDkz6WG9MO0=; b=P9OodY0ODYfBX4tnHl4JLkzp0eN1ZSyiziX/3i9/A04Cx1cALbelYHWDeLe12qi9YBsVRb OCrc+AX30SKxgpXfjRUGDbFrlU6iHgFHOGOwwPckpwwnCcUGm9xs5HzsnxDBDlj9qBQ/CB GN+TpQOeB5RExBBcoxWLIeCtWaw8EZI= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-208-5oyLTBYpPFiR48_BvOmftQ-1; Thu, 16 Mar 2023 11:26:26 -0400 X-MC-Unique: 5oyLTBYpPFiR48_BvOmftQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E00D528237D2; Thu, 16 Mar 2023 15:26:25 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2204B40C6E67; Thu, 16 Mar 2023 15:26:24 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH 01/28] net: Declare MSG_SPLICE_PAGES internal sendmsg() flag Date: Thu, 16 Mar 2023 15:25:51 +0000 Message-Id: <20230316152618.711970-2-dhowells@redhat.com> In-Reply-To: <20230316152618.711970-1-dhowells@redhat.com> References: <20230316152618.711970-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: ehimqyi3311jz1r6yt1rusoxdpscwzzt X-Rspamd-Queue-Id: D60564001B X-HE-Tag: 1678980392-950171 X-HE-Meta: U2FsdGVkX189W9PcfXPjwAmNxQruBtxxtBvH0JngaO/mUpBwiwNNyNRMLne8jAnpBI//zVHmWLp2vLdzPpyA+1oZOJj6Rf7KwQFvHv/DzFj+DapJPJDsmE61k/ZD3uer3idip61bqqcIxVwuaAESvq0d6QqKR1eFU1t1bN11F0IiAw5mPrChRx6rstntcbS8Isvhx569JWktADk66rylHEcNqg9HfeidUGxliAiSEy8qAfLqCkgMn2Lfn2mr6zQGB0389upYuOnA0aPeD6pT7G0Ia9bky39+hNqFM16OYnboJNCzcbvKLxCQOCTqQYYa99iKPv24pQVk3d1xLGOiIQDBJ04b/iVX0qTkPhu2sC+/eJo9h/MN2TfBYsPlZdrc0aBO/SIgHZl6KC11HmpmgrIeOAEwl05U4hcUu/u9r5NzCfiNcoHg9IVyx7rofV3MI09MshielY9wxEe4ixQ1TQsliPmmbUnM63tGS/EMhxKxzg0+wDvpbZW9M2K15IHYzAJZNzl6Q4Ut0HtFRFNP9Ml+m/9+ZEVCLfQGBQ55m+5PxzOuFVZZpNdGzB275zJ0m71TQdAxDTa/8WpWVsmHDwe7WI81qtdwQ/ZU6PEZM0JHOeJuN47cSYVxfxIyblR9eEdRR7B8JqKKQ4W/hL/MvqJGVPyks9npPmoXBd8OSwxszoQt54nEu8FUNC/m2GqgtMEj/eEzRQANfOIJRfDYoP0BT9OaUiZ/U0vpUVhDVF9/9d83SvDnEEGBFbDaYN1Jt0TQDkVf0LFYAc0y30eQ1rfZOMN0QbicZhIS5Jr8Od1qjS3LzHypirZu9qNfQ8AXLtVltSQSaK+IAvXO9+eMowU/EtZ7dUFL1YRiYaQxFtSI2H45s1jbSVTRGWBeFWA+l6jHnUsEZ9zd1tm2UTNhCtziuIx+mI+gmqJ79xrR9DjYwT28FWYxVqI9w526GQ7CfVoltsacd/zlRzXsFGC CjC+LCF1 hL52brVCP696y/FGgfzT9KEHwRGqBQUS9QEyGajedRJsj1BNQkpZfJnL92djrSbsVB/iEvdHc5dBRhLvJ9Gw2/mVyf7a/PYzRUCqDZA/gUXprHwnopqXg7f7xxitoU+ngrKb9snlPgeCH83chA/rIDryKwqLmKAltul47kLt/kYuXr5GHUBbscFJ1ocm+U51YPe3WVrd83CGlcUL122geLSLQcTSt3uV5sNoVhyh8StHVyO+Iq65d1FXIBJqjgqIKtXtf9SjPfYFsfeXdaOnkS23JCgJStkoNhWlEhZtoXJrCJwJSbWYbT6+g0Njq4ZjXJj53Kn9/X/OwOW9FJbDU9H8hHRO20dKIufPmUzrbbM+/V8RCTCTov6ih4QxIcZsSbgWM7qqlnpGUnrWa3EbFnHwlbMU9K7M23Gw5e++m++TdwdWOD+6PnZnUZskg3KO5t19H X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Declare MSG_SPLICE_PAGES, an internal sendmsg() flag, that hints to a network protocol that it should splice pages from the source iterator rather than copying the data if it can. This is intended as a replacement for the ->sendpage() op, allowing a way to splice in several multipage folios in one go. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/linux/socket.h | 1 + 1 file changed, 1 insertion(+) diff --git a/include/linux/socket.h b/include/linux/socket.h index 13c3a237b9c9..a67d02da3c54 100644 --- a/include/linux/socket.h +++ b/include/linux/socket.h @@ -327,6 +327,7 @@ struct ucred { */ #define MSG_ZEROCOPY 0x4000000 /* Use user data in kernel path */ +#define MSG_SPLICE_PAGES 0x8000000 /* Splice the pages from the iterator in sendmsg() */ #define MSG_FASTOPEN 0x20000000 /* Send data in TCP SYN */ #define MSG_CMSG_CLOEXEC 0x40000000 /* Set close_on_exec for file descriptor received through From patchwork Thu Mar 16 15:25:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13177802 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 387F3C6FD19 for ; Thu, 16 Mar 2023 15:26:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D369A900005; Thu, 16 Mar 2023 11:26:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CE6F4900002; Thu, 16 Mar 2023 11:26:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B87D9900005; Thu, 16 Mar 2023 11:26:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A6F07900002 for ; Thu, 16 Mar 2023 11:26:39 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 795EC1408E0 for ; Thu, 16 Mar 2023 15:26:39 +0000 (UTC) X-FDA: 80575138518.30.BF76720 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf22.hostedemail.com (Postfix) with ESMTP id 9C905C001B for ; Thu, 16 Mar 2023 15:26:36 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=JwxTFk+d; spf=pass (imf22.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678980396; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hvyVk2Tm5p69fLI8jM9sCOn8INezqsFYzfTdfGnmvK4=; b=NEkDzxTbw0qWP4kVd81iyO/hpBnljZn4ty3DYORKYyf6oqP5VDIFHsIyHLABISdBLCRlHG RjmcoDOPRTtS25efVzMAjpmj7Bg/OVH2RNjPLHJxYZUaWIkFhaxI5CMeoBo8Zr6KysLjRT gzIuo91c4QgnS2FOLV9aHMBowKSnP7c= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=JwxTFk+d; spf=pass (imf22.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678980396; a=rsa-sha256; cv=none; b=jKMZ9AM6azfPN4fzf4Q64oMNdkQimkT9PxQhZJ0UhpXVlyCHrcfa2S8IP/4uE2QCt3vqa8 gU3Rr9c6B/uAt4gQR+xvpd+gOszJRKyAJUHFi2xWM36O2Fws9Qb8RsTMDQxWOOl98Yr5Is gioDngL5TgCbhsEou311dP1VhAfIjqY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678980396; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hvyVk2Tm5p69fLI8jM9sCOn8INezqsFYzfTdfGnmvK4=; b=JwxTFk+d2LnIJIRxrp2jNx8AQVkomOKQFjaTc8lDWrzxY2S4ty9fBbY0jiK6tUVM2JrgIj GBGMxyPAJ1Wr7qc3wVs48bsTDPZ9Qdt2IMv/IxFUG8GEyYzHY0toDWeerqj1OwIqoZ46Si NPHfAys7gjJ9R1R+I6BrgIs4is3SGaM= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-523-Km8kakyRNF2yUKxaser2SQ-1; Thu, 16 Mar 2023 11:26:29 -0400 X-MC-Unique: Km8kakyRNF2yUKxaser2SQ-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id ADED738149BC; Thu, 16 Mar 2023 15:26:28 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7ACDF492B00; Thu, 16 Mar 2023 15:26:26 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Bernard Metzler , Tom Talpey , linux-rdma@vger.kernel.org Subject: [RFC PATCH 02/28] Add a special allocator for staging netfs protocol to MSG_SPLICE_PAGES Date: Thu, 16 Mar 2023 15:25:52 +0000 Message-Id: <20230316152618.711970-3-dhowells@redhat.com> In-Reply-To: <20230316152618.711970-1-dhowells@redhat.com> References: <20230316152618.711970-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 9C905C001B X-Rspam-User: X-Stat-Signature: frf3yqwefz438s4zk8tu5en83jq6rmzi X-HE-Tag: 1678980396-260296 X-HE-Meta: U2FsdGVkX18savIKWGecsAMAJKPFkAmxJZxp4gXqDBYxbvrBVGV175lTOuGLRDd3A2/686UUylPu4N/59vOLK5EC5OUIbTIubEPiM3xn/aUv2kFmwD39UL7xH6mx5O/yt6fBxfNIKLdbr71ZReuMOzC8yj+j+Y+EZdr3txFQoRSEqAwGwecybtGzh5AZiDMqCvHYdWZunI8Zbc1NAmoZGHXwzJOYDRvPcj00Go9owbP6TtACkUqxvp/P33DbrxOaaJOS91pg0lStWRZkhgJn7JXEEneGy83uEpShrCLDD8fv6YTtzb3Nv3L4HFCQ4fXYCBlTq8fkf1ByrlmioN8MzuEYUvXWunnpEX0e3JN9bzSbd2Gi2uMrrlmkAuFJroz5T/a5RAcOJricXadDfhaH9jM8+MzAVaHPYDAVgrWdmLaFD7cek5jvSuh6ZiOiZUhnXkuHaV4cSZsnBDI2V6HD942DwVEhsCL2VTVMHt/8KNgQpf3WWyJK1kyaHzNIKbMpKvok487+pJD10z020vJ1uLcTjr+msBnGA7YDLF6LoDF8xL3+zR9v83Oyo5AI/0P0QhjTLIQ4swSO5ivu1btEZXp1a+70fVsvfMQl7zk4SGm1Jz1aLebCN7ws8j4oZZZxgUwrnxdiPlw9R0Ndz6NMXDETZnWuiEXbHnsbs7QqTIxzftT4yfdZqFsdLsmYuHmbqSSrhTWPGagQYrXdU8hvhpwOqj08ORcNGm0jcHfoxOviLclKd0g4I2bw3/rk0iKCvKmIJYfGEG0QUanyAOItjQWiBgyq4HHQdsUK/jrCpvHbn86+fbKGYscnx+K9JdsnU2DSWkBmCXPR/dt5JT4lPK2LlLlclWx7Ym/yez/Y7GiV7r/tX9i9f/b6ncWRvoOGVOlBEepx0BV+csXqL046S1Z+6k+DNvo+Ko6L0WnOj0vWZhmeNqZNhpm8W3+fmpTQaYE5ke65gvCzWSJ8CPo 9yxXTH0L VQTKN4DN29JmpDh3yGzdf1T+ShLvBtNF6fqQFz++hYyPqT9nVIXAyUU6rCbo2qP9R2q0g2RUilar/skMI6UnMh19i0J3jhvv5KvgipVgf9Y6J8TJmD+CqxqfqgBEV/waenTwQXIWcQDWioc8z+sl1bT/sC0+8pxMi8u2fNlm+BWuGHDlTX/R5sEc1TxtCUPhegs9oRkFjEYPCTR1Nm8iCkkdYCQQTzid450XSrPbyhkyrAmmtqrh13fs9y9ro9Gl8wtT+GeSnn9K9G6L7qkqGWlbvnHCCKDkKcTH3ha2/knxHBXo+fyL/2lRIj93Zx1d3ZNyzfJ19UhRR3vbM0L+sYAjEFZ9AW6fnn0xbyXGI/7BD2HJVY4tu8KZdK9TDaXiG6pHJwlqoHHV9pSsvMlDV1OVaRz5pRsibJrdCiSKsKy6002VvaGB2YjWOoq6h29PF/0sy5y9+HplpmgqGXMgeVTis1cDeDxl/c96eMG8tG4nJXFB0bb2ooqK2kIcAPwQZ/BRHVcqSKGCGqrbqrW8LcAVXwg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If a network protocol sendmsg() sees MSG_SPLICE_DATA, it expects that the iterator is of ITER_BVEC type and that all the pages can have refs taken on them with get_page() and discarded with put_page(). Bits of network filesystem protocol data, however, are typically contained in slab memory for which the cleanup method is kfree(), not put_page(), so this doesn't work. Provide a simple allocator, zcopy_alloc(), that allocates a page at a time per-cpu and sequentially breaks off pieces and hands them out with a ref as it's asked for them. The caller disposes of the memory it was given by calling put_page(). When a page is all parcelled out, it is abandoned by the allocator and another page is obtained. The page will get cleaned up when the last skbuff fragment is destroyed. A helper function, zcopy_memdup() is provided to call zcopy_alloc() and copy the data it is given into it. [!] I'm not sure this is the best way to do things. A better way might be to make the network protocol look at the page and copy it if it's a slab object rather than taking a ref on it. Signed-off-by: David Howells cc: Bernard Metzler cc: Tom Talpey cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-rdma@vger.kernel.org cc: netdev@vger.kernel.org --- include/linux/zcopy_alloc.h | 16 +++++ mm/Makefile | 2 +- mm/zcopy_alloc.c | 129 ++++++++++++++++++++++++++++++++++++ 3 files changed, 146 insertions(+), 1 deletion(-) create mode 100644 include/linux/zcopy_alloc.h create mode 100644 mm/zcopy_alloc.c diff --git a/include/linux/zcopy_alloc.h b/include/linux/zcopy_alloc.h new file mode 100644 index 000000000000..8eb205678073 --- /dev/null +++ b/include/linux/zcopy_alloc.h @@ -0,0 +1,16 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Defs for for zerocopy filler fragment allocator. + * + * Copyright (C) 2023 Red Hat, Inc. All Rights Reserved. + * Written by David Howells (dhowells@redhat.com) + */ + +#ifndef _LINUX_ZCOPY_ALLOC_H +#define _LINUX_ZCOPY_ALLOC_H + +struct bio_vec; + +int zcopy_alloc(size_t size, struct bio_vec *bvec, gfp_t gfp); +int zcopy_memdup(size_t size, const void *p, struct bio_vec *bvec, gfp_t gfp); + +#endif /* _LINUX_ZCOPY_ALLOC_H */ diff --git a/mm/Makefile b/mm/Makefile index 8e105e5b3e29..3848f43751ee 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -52,7 +52,7 @@ obj-y := filemap.o mempool.o oom_kill.o fadvise.o \ readahead.o swap.o truncate.o vmscan.o shmem.o \ util.o mmzone.o vmstat.o backing-dev.o \ mm_init.o percpu.o slab_common.o \ - compaction.o \ + compaction.o zcopy_alloc.o \ interval_tree.o list_lru.o workingset.o \ debug.o gup.o mmap_lock.o $(mmu-y) diff --git a/mm/zcopy_alloc.c b/mm/zcopy_alloc.c new file mode 100644 index 000000000000..7b219392e829 --- /dev/null +++ b/mm/zcopy_alloc.c @@ -0,0 +1,129 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Allocator for zerocopy filler fragments + * + * Copyright (C) 2023 Red Hat, Inc. All Rights Reserved. + * Written by David Howells (dhowells@redhat.com) + * + * Provide a facility whereby pieces of bufferage can be allocated for + * insertion into bio_vec arrays intended for zerocopying, allowing protocol + * stuff to be mixed in with data. + * + * Unlike objects allocated from the slab, the lifetime of these pieces of + * buffer are governed purely by the refcount of the page in which they reside. + */ + +#include +#include +#include +#include +#include + +struct zcopy_alloc_info { + struct folio *folio; /* Page currently being allocated from */ + struct folio *spare; /* Spare page */ + unsigned int used; /* Amount of folio used */ + spinlock_t lock; /* Allocation lock (needs bh-disable) */ +}; + +static struct zcopy_alloc_info __percpu *zcopy_alloc_info; + +static int __init zcopy_alloc_init(void) +{ + zcopy_alloc_info = alloc_percpu(struct zcopy_alloc_info); + if (!zcopy_alloc_info) + panic("Unable to set up zcopy_alloc allocator\n"); + return 0; +} +subsys_initcall(zcopy_alloc_init); + +/** + * zcopy_alloc - Allocate some memory for use in zerocopy + * @size: The amount of memory (maximum 1/2 page). + * @bvec: Where to store the details of the memory + * @gfp: Allocation flags under which to make an allocation + * + * Allocate some memory for use with zerocopy where protocol bits have to be + * mixed in with spliced/zerocopied data. Unlike memory allocated from the + * slab, this memory's lifetime is purely dependent on the folio's refcount. + * + * The way it works is that a folio is allocated and pieces are broken off + * sequentially and given to the allocators with a ref until it no longer has + * enough spare space, at which point the allocator's ref is dropped and a new + * folio is allocated. The folio remains in existence until the last ref held + * by, say, a sk_buff is discarded and then the page is returned to the + * allocator. + * + * Returns 0 on success and -ENOMEM on allocation failure. If successful, the + * details of the allocated memory are placed in *%bvec. + * + * The allocated memory should be disposed of with folio_put(). + */ +int zcopy_alloc(size_t size, struct bio_vec *bvec, gfp_t gfp) +{ + struct zcopy_alloc_info *info; + struct folio *folio, *spare = NULL; + size_t full_size = round_up(size, 8); + + if (WARN_ON_ONCE(full_size > PAGE_SIZE / 2)) + return -ENOMEM; /* Allocate pages */ + +try_again: + info = get_cpu_ptr(zcopy_alloc_info); + + folio = info->folio; + if (folio && folio_size(folio) - info->used < full_size) { + folio_put(folio); + folio = info->folio = NULL; + } + if (spare && !info->spare) { + info->spare = spare; + spare = NULL; + } + if (!folio && info->spare) { + folio = info->folio = info->spare; + info->spare = NULL; + info->used = 0; + } + if (folio) { + bvec_set_folio(bvec, folio, size, info->used); + info->used += full_size; + if (info->used < folio_size(folio)) + folio_get(folio); + else + info->folio = NULL; + } + + put_cpu_ptr(zcopy_alloc_info); + if (folio) { + if (spare) + folio_put(spare); + return 0; + } + + spare = folio_alloc(gfp, 0); + if (!spare) + return -ENOMEM; + goto try_again; +} +EXPORT_SYMBOL(zcopy_alloc); + +/** + * zcopy_memdup - Allocate some memory for use in zerocopy and fill it + * @size: The amount of memory to copy (maximum 1/2 page). + * @p: The source data to copy + * @bvec: Where to store the details of the memory + * @gfp: Allocation flags under which to make an allocation + */ +int zcopy_memdup(size_t size, const void *p, struct bio_vec *bvec, gfp_t gfp) +{ + void *q; + + if (zcopy_alloc(size, bvec, gfp) < 0) + return -ENOMEM; + + q = kmap_local_folio(page_folio(bvec->bv_page), bvec->bv_offset); + memcpy(q, p, size); + kunmap_local(q); + return 0; +} +EXPORT_SYMBOL(zcopy_memdup); From patchwork Thu Mar 16 15:25:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13177803 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98DA3C6FD1F for ; Thu, 16 Mar 2023 15:26:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 324EF900006; Thu, 16 Mar 2023 11:26:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2FC19900002; Thu, 16 Mar 2023 11:26:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 19D55900006; Thu, 16 Mar 2023 11:26:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 0C5E3900002 for ; Thu, 16 Mar 2023 11:26:42 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id E2B13ABCDC for ; Thu, 16 Mar 2023 15:26:40 +0000 (UTC) X-FDA: 80575138560.14.E0918B2 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf24.hostedemail.com (Postfix) with ESMTP id 3075C18001F for ; Thu, 16 Mar 2023 15:26:39 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YkrzPxR8; spf=pass (imf24.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678980399; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jyeyKmuw2xi8Qa/hUDKrm3skE7lHLzOU8pZdDmCLqIg=; b=WXa5fy/42wdJp3QNCsMFYDP/GgH+jWFdozvCXXLC3qAiXj1NeIy3INxt0mGaoJejL01ady CPJmFeqJol1i3t+Qvnh9+9wtwXKUkihjISEkvYhPc58tiOGf0cQCs0eCSGwoz6mLptC2BN Uhq0t0H9NUvdjILxRpoT3UoKgVCp31U= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YkrzPxR8; spf=pass (imf24.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678980399; a=rsa-sha256; cv=none; b=IFHoHIgLpW/5Pe20++3HpRsKJzOSK1v9lotOMryKx0kn9ml5t8uHhrRvHYViG+AIgd5Tcg uXwF59/Zg/qq+AhTkJurhZETa/OCb1PQ76ezddU2zD2yVAdc9PfNrkkR1p93N7nVUH6enR djQPFYBxGAc24cgcbuYtNTN7lqbgrvU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678980398; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jyeyKmuw2xi8Qa/hUDKrm3skE7lHLzOU8pZdDmCLqIg=; b=YkrzPxR85OLrGZARm0ze9hel3sIx1LJv2yW1jtvE1W87uWAJQr1I2xeQpOtdJcqwZtEtRR Ou1ZongmodYW1l41r1L5uDVN+kr1gzsyIG1CuVVFHgjNubUFyeoDrnCqYQ9cS2LMBQMwjK tEohZKVG18F4MUtRnuu0jOAWwFR5anw= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-473-njCfjHCZMGSNtwp8bzoGhw-1; Thu, 16 Mar 2023 11:26:32 -0400 X-MC-Unique: njCfjHCZMGSNtwp8bzoGhw-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2889C38149BF; Thu, 16 Mar 2023 15:26:31 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6026F492B00; Thu, 16 Mar 2023 15:26:29 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH 03/28] tcp: Support MSG_SPLICE_PAGES Date: Thu, 16 Mar 2023 15:25:53 +0000 Message-Id: <20230316152618.711970-4-dhowells@redhat.com> In-Reply-To: <20230316152618.711970-1-dhowells@redhat.com> References: <20230316152618.711970-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 3075C18001F X-Stat-Signature: g3rf18c11j3rs48w3wzc3e7cyge9toq7 X-HE-Tag: 1678980399-531518 X-HE-Meta: U2FsdGVkX18LsuhrtFM2Z3O4UH878AE5Cq7lXmvbpzzyPa84V48PKkFJi5KS5M8onSyItOfGQc9lFEkrb4xfMtDmVt5d7UMRLJpfHYvThwR2elZx0N1mmZYLmLup75P87VVdOK0ZoZBMyFlRRMdv3/V35Z3p00Ij03uOMiJsYJ7I4xU1BX6Bo+nLEVXHItNRCCvfZSkE2SjWM44ehIzxeuhlS9p4tl4G5mGJ5SHJyHXm9BsTbFp99M3gsoefsnFOWXdzEcrntcXHpib38X7YMlSQPipBT3Sk7fWPLydwGbbZBgARXRh+fOVfhVzX/mHy0MkuNGoOSW5mFlBdsAkS+uQCPFj6Q2Q53AtDi3Vy20jYcxMEw4GJk7XbAXgepkp8SSV/XLCemFLMOBOgniZyeaOZek/maUthfwgjORdXbhMNnEpyjDUVD0B2Qz8YQ9EAuVingCYdIsoofFHbXWtvf4n81l8FXbR7qwbvxdU88RMB2EnQRLUzGQEFuUKgj9Wh0lUSNMah786r469efLI9yy7lvwqLqrj1HDvYSOdKDuXgw/UWm+fI+wo0qKo/sfcA2eUGWq33h8ZHDb8yMSdj7RtIT8ZaFCvvgCVnhgwzLKit0sQNmmKA8YcbIKfJrz9TSqHU5J7CdTR/DfmkAN8xf/kD/dXUWSUawGwpwAfI0o/9IKPJxU2VUx/tMYNfcHZWfmRQ0mlpaqXzbZ9PQGfokckUwDjZWswFfGpN6dDCVwx+2HqtUS10W3xwU5r69amE1sfaeFu8ioKFqrnMzEydptaSFZG0zC9l2AKkjYPsI11M3McrDo030JTn7e402Xx2SpCsQOeJn7kQKLoAihQ4CKI5eyZ5iIG+gsvLrlXxeHNMvWkBregnnhrKD/tzFrAa98zQUmQNz+rMFsMAnmkenBg2vlEkFcSAeyUx/xey675/ch6oQ72LZSCjRsTuGupkFvSnaBVXzIseZCM8bJF 5xL/uvwl hmOqLK0xqu8cuKFFly0m2NeYLgatcfdjxVuc6Cghz5Jn+MTjn5IQPsoZ9feZ5+V8M0gDaOkkRwH+AlDigwHYy9Bq1tX4YJOPhPxvfebjCsmngrVFCeYtGJvR7fSgxozBHri5Ds+j1UkZkaR5NF7RMjRYiNi7GsMUgbmbAbjbaol0+dFUCXtl9qu/lIk9d2yiIp3wbbdFxw58ld7Rh++bBjxjivl06TIErUfT+CAy01NTv2hJ+byicAzkJk5YDvQ+pTZXBWsVR24gsNu17Il61Mf2ItAFU7iRk2HNYhGw8a4oY+uIAlGPfUXVCMRrumU9a23I9nJapIj/irVsvtNCgsIxdgoqlzGzDUQ/bjJ1HBsHX1iS5JJavYwSVoMf03hvVziIeBDzOtgpepDcIGuE+txVAUzz8Uzj+CHoIOMRwkt63ELWq17DngKyM9fYoVIhbRWZ/ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make TCP's sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator if possible (the iterator must be ITER_BVEC and the pages must be spliceable). This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/ipv4/tcp.c | 59 +++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 53 insertions(+), 6 deletions(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 288693981b00..77c0c69208a5 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -1220,7 +1220,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) int flags, err, copied = 0; int mss_now = 0, size_goal, copied_syn = 0; int process_backlog = 0; - bool zc = false; + int zc = 0; long timeo; flags = msg->msg_flags; @@ -1231,17 +1231,24 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (msg->msg_ubuf) { uarg = msg->msg_ubuf; net_zcopy_get(uarg); - zc = sk->sk_route_caps & NETIF_F_SG; + if (sk->sk_route_caps & NETIF_F_SG) + zc = 1; } else if (sock_flag(sk, SOCK_ZEROCOPY)) { uarg = msg_zerocopy_realloc(sk, size, skb_zcopy(skb)); if (!uarg) { err = -ENOBUFS; goto out_err; } - zc = sk->sk_route_caps & NETIF_F_SG; - if (!zc) + if (sk->sk_route_caps & NETIF_F_SG) + zc = 1; + else uarg_to_msgzc(uarg)->zerocopy = 0; } + } else if (unlikely(flags & MSG_SPLICE_PAGES) && size) { + if (!iov_iter_is_bvec(&msg->msg_iter)) + return -EINVAL; + if (sk->sk_route_caps & NETIF_F_SG) + zc = 2; } if (unlikely(flags & MSG_FASTOPEN || inet_sk(sk)->defer_connect) && @@ -1345,7 +1352,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (copy > msg_data_left(msg)) copy = msg_data_left(msg); - if (!zc) { + if (zc == 0) { bool merge = true; int i = skb_shinfo(skb)->nr_frags; struct page_frag *pfrag = sk_page_frag(sk); @@ -1390,7 +1397,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) page_ref_inc(pfrag->page); } pfrag->offset += copy; - } else { + } else if (zc == 1) { /* First append to a fragless skb builds initial * pure zerocopy skb */ @@ -1411,6 +1418,46 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (err < 0) goto do_error; copy = err; + } else if (zc == 2) { + /* Splice in data. */ + const struct bio_vec *bv = msg->msg_iter.bvec; + size_t seg = iov_iter_single_seg_count(&msg->msg_iter); + size_t off = bv->bv_offset + msg->msg_iter.iov_offset; + bool can_coalesce; + int i = skb_shinfo(skb)->nr_frags; + + if (copy > seg) + copy = seg; + + can_coalesce = skb_can_coalesce(skb, i, bv->bv_page, off); + if (!can_coalesce && i >= READ_ONCE(sysctl_max_skb_frags)) { + tcp_mark_push(tp, skb); + goto new_segment; + } + if (tcp_downgrade_zcopy_pure(sk, skb)) + goto wait_for_space; + + copy = tcp_wmem_schedule(sk, copy); + if (!copy) + goto wait_for_space; + + if (can_coalesce) { + skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], copy); + } else { + get_page(bv->bv_page); + skb_fill_page_desc_noacc(skb, i, bv->bv_page, off, copy); + } + iov_iter_advance(&msg->msg_iter, copy); + + if (!(flags & MSG_NO_SHARED_FRAGS)) + skb_shinfo(skb)->flags |= SKBFL_SHARED_FRAG; + + skb->len += copy; + skb->data_len += copy; + skb->truesize += copy; + sk_wmem_queued_add(sk, copy); + sk_mem_charge(sk, copy); + } if (!copied) From patchwork Thu Mar 16 15:25:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13177805 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA571C6FD1F for ; Thu, 16 Mar 2023 15:26:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B63EB940008; Thu, 16 Mar 2023 11:26:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B13DF940007; Thu, 16 Mar 2023 11:26:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7B6C9940008; Thu, 16 Mar 2023 11:26:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 6D737940007 for ; Thu, 16 Mar 2023 11:26:44 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 434D11C6BE7 for ; Thu, 16 Mar 2023 15:26:44 +0000 (UTC) X-FDA: 80575138728.07.35758ED Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf06.hostedemail.com (Postfix) with ESMTP id 77B8018001C for ; Thu, 16 Mar 2023 15:26:41 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RoNMUykP; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf06.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678980401; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+jxYzo5nnHDy2PfhlAaDDi56GwEdHMnIZIIG4o6/j/8=; b=pg58plum51ohWRwd9akFCl+fz1etZ3ZQXKxukA13yPFoQfqqYkopnySXCKQBGUrDSd4+pN lh09jwlQkxnpm6Sk/HBbAhfxHny1m3DIieDbA6TAIh/QZnVRRQ479gTusfB1cHx4Ggds3v HLOM/DHGqyGqISfDzufB6HCwt7UMvGA= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RoNMUykP; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf06.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678980401; a=rsa-sha256; cv=none; b=g6n9L+iiMXU4dWJqSQw2f7Mx9JlRGRihR/gJU4ZDJHBaSbfoa0eB2YtHTP8PiNQhaqERpk ZgIC2/ECj1dY6g8N5dYaIZ+TUx9xZoyPb3yk/Ey3GqUL3BEzaaro0ZTEd2SefVs8ZMH4Ol W83rtKEhD4L4+TxnkWKQ265hd/3Du6g= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678980400; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+jxYzo5nnHDy2PfhlAaDDi56GwEdHMnIZIIG4o6/j/8=; b=RoNMUykPJOhcWe7SxWO3nIyBLi8NRr7w0dwMLv7au36yZNA0Bsa3eDsPz/6W6FMkvYYvLg zOTLMtaeyDlh8Vw+xLdK9GI40somX9eekPirYrsXggN0P7ryUcP+vqYKI3rNLHsReBiRWV +KyKyGleZ495dCnHzJrkEbdSKpAOo1k= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-59-UIpyeEHENxKhYGTwrn-jBw-1; Thu, 16 Mar 2023 11:26:38 -0400 X-MC-Unique: UIpyeEHENxKhYGTwrn-jBw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 82972101A552; Thu, 16 Mar 2023 15:26:33 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id B820F35453; Thu, 16 Mar 2023 15:26:31 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH 04/28] tcp: Convert do_tcp_sendpages() to use MSG_SPLICE_PAGES Date: Thu, 16 Mar 2023 15:25:54 +0000 Message-Id: <20230316152618.711970-5-dhowells@redhat.com> In-Reply-To: <20230316152618.711970-1-dhowells@redhat.com> References: <20230316152618.711970-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 77B8018001C X-Stat-Signature: miqg15ixeiq393jsorf7heyki9doufew X-HE-Tag: 1678980401-492389 X-HE-Meta: U2FsdGVkX18ISzdhisahQ3jRySFHxTprw9WD6SUxNVHEcwdZxUft41KZ2vKyRl0CztbVREvhjt8ulwTvoTypkOTDcHzfDouJSIU1yl0huotflHLpcMIX/qJ/tMbsUCx8B+vUGelex5lBPE0Kba30ZakWOdYuNFrEubysqxmI5suA01eU1bCqa0fBtfOEU13p87p22Wphuc+1zwpG/kAWa+f+HkGOPvhrgIr/0MC/SwqaiPIN0TnW6yEqorPHPX8aoiSNIuF5VBoo3G6Wwfp1CkdBQWlXE5RoPXwkExlTXhj370gcjYp5FMYv/t8zevlebtM0QjKCKQIfLcSb0eVL0ju7W2nnBjov6VzbAULPMFYtXZ2lrGq6GlgjcMk9xXpmTBK7wMdwQeLf/ekbSrCUNi/KAwfzawSORMxFvVdvKx9GiL02h2y2M1w3f6vSWV50mXM5VmXNm9/HTCru0D8Y1WB/xIUDlFL3KM68TKnm737ENV/G0mP7LslC9eTFU0fJZ2emjMJ/6kg1+O8d1YUej8ATFFjKrsowSNUss7LbYNtjOjWeY4m0TFECiGIqxGr0XebIs1ZO3cyFs4LoAjK9O2p7H8m1xyB9pTmhDmF1EERm2kCn51e28WrpWumNJ36GojRNs3lS8aZ4zhE3yKnARGraSIQDNBlVHXsy2fdIUJ3giyJP6hvSDcgj0egTrI9gAwN0LnHk5HA+VaH5vSgowdTw/rVeFrcI7rK4kQx9eVvcoee3jenWAxwZ9Geykkl0KWdOpc/QTGabxDO130l77usS95SNe6NSTQz5AoEYu4ZLYPZzSuyjlofVMWX5GhOK/HFioeAhIqvi/QZG8OkYkEvQ8E+6E8akx9nssv77JwYZh/eXPOxh39BBwwZ3EDAxY90bzrViNI0zP1o7J06F+Pa8VCXQKIsD2jKkCcCVCHXTj5c3SK5qwq++HhnEcudBl2mJ+SBYeQevMME0ygP 4MFRa+K+ O2MlAc73zzkAeCv2fK9hJj6ntMkNyv3L3GiXTbuN1aeIjNhMoj1z/gav7v0M4TdrkMiRuUZ47WSgayF8UA+7d/zkQ3j0r7ZLPKC4uMIgTRLena56JiQH2Qfp9BVwhz3n/jyTK/0pe78bJLSXiOsH6ITzFXl1jc0AMGV63xp96/UkC9px++eYvFnj9jRZc92JudtuVOGdfgFStThhgi+SUXxel7b1Qs9o7bwkfmyYvGHVUzC2vFyvqO8g+d55ymABuWvA77C7YFB/D6JZDAbhRJsMhiFanwParYlc2PO+3ky+eugv8Kpfsm/hzWYhCwoG4Jqb9ROftEpBb4xPgSqGNiZx3nGIah4PtsyrZLs7iVel9Updd73X596cDCrE1LzS45dEACqjOXTm9uHiTn4ZwUikMSO01j9jx38nIOuHtDFtZ+Qkl8wtGFgWB6Wv5D0bjS5Ih X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert do_tcp_sendpages() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. do_tcp_sendpages() can then be inlined in subsequent patches into its callers. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/ipv4/tcp.c | 160 +++---------------------------------------------- 1 file changed, 9 insertions(+), 151 deletions(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 77c0c69208a5..7c3acc5673e9 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -971,163 +971,21 @@ static int tcp_wmem_schedule(struct sock *sk, int copy) return min(copy, sk->sk_forward_alloc); } -static struct sk_buff *tcp_build_frag(struct sock *sk, int size_goal, int flags, - struct page *page, int offset, size_t *size) -{ - struct sk_buff *skb = tcp_write_queue_tail(sk); - struct tcp_sock *tp = tcp_sk(sk); - bool can_coalesce; - int copy, i; - - if (!skb || (copy = size_goal - skb->len) <= 0 || - !tcp_skb_can_collapse_to(skb)) { -new_segment: - if (!sk_stream_memory_free(sk)) - return NULL; - - skb = tcp_stream_alloc_skb(sk, 0, sk->sk_allocation, - tcp_rtx_and_write_queues_empty(sk)); - if (!skb) - return NULL; - -#ifdef CONFIG_TLS_DEVICE - skb->decrypted = !!(flags & MSG_SENDPAGE_DECRYPTED); -#endif - tcp_skb_entail(sk, skb); - copy = size_goal; - } - - if (copy > *size) - copy = *size; - - i = skb_shinfo(skb)->nr_frags; - can_coalesce = skb_can_coalesce(skb, i, page, offset); - if (!can_coalesce && i >= READ_ONCE(sysctl_max_skb_frags)) { - tcp_mark_push(tp, skb); - goto new_segment; - } - if (tcp_downgrade_zcopy_pure(sk, skb)) - return NULL; - - copy = tcp_wmem_schedule(sk, copy); - if (!copy) - return NULL; - - if (can_coalesce) { - skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], copy); - } else { - get_page(page); - skb_fill_page_desc_noacc(skb, i, page, offset, copy); - } - - if (!(flags & MSG_NO_SHARED_FRAGS)) - skb_shinfo(skb)->flags |= SKBFL_SHARED_FRAG; - - skb->len += copy; - skb->data_len += copy; - skb->truesize += copy; - sk_wmem_queued_add(sk, copy); - sk_mem_charge(sk, copy); - WRITE_ONCE(tp->write_seq, tp->write_seq + copy); - TCP_SKB_CB(skb)->end_seq += copy; - tcp_skb_pcount_set(skb, 0); - - *size = copy; - return skb; -} - ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct tcp_sock *tp = tcp_sk(sk); - int mss_now, size_goal; - int err; - ssize_t copied; - long timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT); - - if (IS_ENABLED(CONFIG_DEBUG_VM) && - WARN_ONCE(!sendpage_ok(page), - "page must not be a Slab one and have page_count > 0")) - return -EINVAL; - - /* Wait for a connection to finish. One exception is TCP Fast Open - * (passive side) where data is allowed to be sent before a connection - * is fully established. - */ - if (((1 << sk->sk_state) & ~(TCPF_ESTABLISHED | TCPF_CLOSE_WAIT)) && - !tcp_passive_fastopen(sk)) { - err = sk_stream_wait_connect(sk, &timeo); - if (err != 0) - goto out_err; - } - - sk_clear_bit(SOCKWQ_ASYNC_NOSPACE, sk); - - mss_now = tcp_send_mss(sk, &size_goal, flags); - copied = 0; - - err = -EPIPE; - if (sk->sk_err || (sk->sk_shutdown & SEND_SHUTDOWN)) - goto out_err; - - while (size > 0) { - struct sk_buff *skb; - size_t copy = size; - - skb = tcp_build_frag(sk, size_goal, flags, page, offset, ©); - if (!skb) - goto wait_for_space; - - if (!copied) - TCP_SKB_CB(skb)->tcp_flags &= ~TCPHDR_PSH; - - copied += copy; - offset += copy; - size -= copy; - if (!size) - goto out; - - if (skb->len < size_goal || (flags & MSG_OOB)) - continue; - - if (forced_push(tp)) { - tcp_mark_push(tp, skb); - __tcp_push_pending_frames(sk, mss_now, TCP_NAGLE_PUSH); - } else if (skb == tcp_send_head(sk)) - tcp_push_one(sk, mss_now); - continue; - -wait_for_space: - set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); - tcp_push(sk, flags & ~MSG_MORE, mss_now, - TCP_NAGLE_PUSH, size_goal); - - err = sk_stream_wait_memory(sk, &timeo); - if (err != 0) - goto do_error; + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = flags | MSG_SPLICE_PAGES, + }; - mss_now = tcp_send_mss(sk, &size_goal, flags); - } + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); -out: - if (copied) { - tcp_tx_timestamp(sk, sk->sk_tsflags); - if (!(flags & MSG_SENDPAGE_NOTLAST)) - tcp_push(sk, flags, mss_now, tp->nonagle, size_goal); - } - return copied; + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; -do_error: - tcp_remove_empty_skb(sk); - if (copied) - goto out; -out_err: - /* make sure we wake any epoll edge trigger waiter */ - if (unlikely(tcp_rtx_and_write_queues_empty(sk) && err == -EAGAIN)) { - sk->sk_write_space(sk); - tcp_chrono_stop(sk, TCP_CHRONO_SNDBUF_LIMITED); - } - return sk_stream_error(sk, flags, err); + return tcp_sendmsg_locked(sk, &msg, size); } EXPORT_SYMBOL_GPL(do_tcp_sendpages); From patchwork Thu Mar 16 15:25:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13177804 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 645DBC6FD19 for ; Thu, 16 Mar 2023 15:26:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 01741900002; Thu, 16 Mar 2023 11:26:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F3248940007; Thu, 16 Mar 2023 11:26:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DAC70900007; Thu, 16 Mar 2023 11:26:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id C574B900002 for ; Thu, 16 Mar 2023 11:26:43 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 8900E4044E for ; Thu, 16 Mar 2023 15:26:43 +0000 (UTC) X-FDA: 80575138686.16.7F9C09C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf17.hostedemail.com (Postfix) with ESMTP id D1B5640008 for ; Thu, 16 Mar 2023 15:26:41 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Nf3aoiNh; spf=pass (imf17.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678980401; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=V4A9+ThbnxeOTamldoFRxBPwB7/WQbFmJXJUgDasL0c=; b=BzKLWGCXQtXNifDkEBjMJKTqvygvxDfZtzwfOTzlK8p65znnmPSOjw1CesXcNJkT8ETrSa RH6yhfoIKMS5DCY7s9Fo9xP/iIBOtFnWiw+npG7TslgEfzYumpcPTwdQwW5S2/AfDrV1U5 f9J5CfgEcj8HOC9zTzFaT+O+iNqDyUU= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Nf3aoiNh; spf=pass (imf17.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678980401; a=rsa-sha256; cv=none; b=lxiIFOG1vysGLmh7sMpMTXCtanpLz0o1c4mVfCl5Ix4sWqOMjVzyIeIXKBlwX+ZVtYxNCK rg2PN6jC09FuaLGiCrLJlA+/qRqcvHbfDd4+Y3bbayJPz2lViCRptwBEmcWbkdZQeMlqXy VPRZur1hEA6a4qxUmIV6WoZh7Gj8jMk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678980400; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=V4A9+ThbnxeOTamldoFRxBPwB7/WQbFmJXJUgDasL0c=; b=Nf3aoiNhicefF8RGLcDts9UYB7b6F/T0NFzyTq6NRIWTKZTBx7EyePt+JOkf/0dI7C/Ywg yV6S+Bm51lYWCoBEY37oMPS1izbWlkt0SdfatF5XrDz0rarOcn8xAagIQMXB6Cp7+GyRdM BkuHJNDPt+OW23wJjaiSRGiiDvIAvPE= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-592-b1SC9PDcPYSYvZwVDt6M2w-1; Thu, 16 Mar 2023 11:26:37 -0400 X-MC-Unique: b1SC9PDcPYSYvZwVDt6M2w-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 31C5638149B9; Thu, 16 Mar 2023 15:26:36 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1CF1E1121315; Thu, 16 Mar 2023 15:26:34 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, John Fastabend , Jakub Sitnicki , bpf@vger.kernel.org Subject: [RFC PATCH 05/28] tcp_bpf: Inline do_tcp_sendpages as it's now a wrapper around tcp_sendmsg Date: Thu, 16 Mar 2023 15:25:55 +0000 Message-Id: <20230316152618.711970-6-dhowells@redhat.com> In-Reply-To: <20230316152618.711970-1-dhowells@redhat.com> References: <20230316152618.711970-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: dewn7rydqdkkwwa78zcypc57bfcihhgg X-Rspamd-Queue-Id: D1B5640008 X-HE-Tag: 1678980401-308982 X-HE-Meta: U2FsdGVkX18t89INxATVA7yQcjG/wYnVx6ppFM5vdlZRZqhnbDsZqX7gMN6yUF+3hfA5Uuj1IhUtahtnPz5u5/j+tspafyJ8wa6v8tV9L3yuyjOl1mfaLkKWCKhLlXghqsf2OqaXlciN9wCfqECZgkBSgK/fHdiJR3a+gk5DT7VpSny16XiWYbPcXibnwRj3a9fadFErPOYTegLW2Ogog9sTDK61d1sV5ubBMeaF2FFO6F6ESSRX7y/di/1gWHWOC8ucFrfPFkR3arPmEGOeJe273vEkHFcLegcYt2EbQDfL9gR4y+eNaUgtvvfoc1ZGQyVb2/GVo50xf205SVXU9qGiAavIueNjWdFMXTlV5xeSB+O10TivKXFmyks27BOtcPBePt8P+n+o1U89YRJFSjCHroB5GFNSkA9Xm7rVwjLdP9CfyR5TodOCSxguYb+xADr2UCX3hEKtTL9dDg30WVB6emWVGNKSgddSx+gPH+nIk0Qo9+Dj6DkBUgLoADC5cOp+7Xblztmt5b46RofT8TPOXcakBcbJOgtRpGHWpn5Zxq1EPLjCGKz6GqfV3LKjOmAr1O1V3jZTDvkXCE+xtenFzCBjf2kW36iIFvsfJTV3z+t3xZ6K/hJfLj21prYBpnXTdFBYkDb52n4GJg/1mUotk+JJ3HNd3xRLXkQtSM5rYj4BOLEVzw4TocrQBAckhYNC5J5++IhREqTFzF06Jtav7DXSeLiei3DqhdLy43s5/g7OrIFPI5JTzlyRMKqwRPu+i7rBWgN2MzerKxnQmmRLyIxI4Y588wksIHPTO1ezqe2l+IllNegoH/OP785+yEluXjP3D3l2l2ABJrl+ROyET2bN8ZdlpnZc7SEAGLVOKkQp4y9M8IteCNF0WsrUE0kwatOq/xsRrrA2Otca/YkInoikIe/J5fIZyQ9ppogiQCacY+Ac0NZFfCXRvKc/V25Ex78FZuCbWqbEzee hRE6kShx 2nMiv4M3EfFeVFZAgRh4dBCxrt5rZO4mOovNWcfBc5PERNc8K1ureZyGD8Rq4559uNpjVNencmW//1d94NYHxBNakeq6L5Osq9LEoF3BETST8FiKdg0rd9zx0S1hCfJbIi1IyOqF/XEJ8d4Ccn9n8fjgbOKwBh65qvD1uVgWV8dPjAq0fhP+27ZdrdS9yi38rSYCoIzEgFJqAO9NbyCD2FEr0nIbYEcQ1joz9Ktaho7wB/1P0ZisKllHh7bx968RZK1xull9M0LnrFpQ1BWlPE7dUXuIJKKXGrMeDjpDBkq6AhfuCBl2wzPHm7IYGDRw65tPwktTiDDOHf0w//skWm6/ZEY3JDbHYGV606x76wmMsNdO7BvUGErb5V0e0o053a8zIU4MTwTZtHbKgx4IevKW7TrWFlC0mxh8L+sS+ArtLe9BRb/gO0hfOxcrM22OAP//9rRZrAvnb5cRv8n+bQmP5ND4nMjxFdR93uupO32YC4WiFRM4KTqRikRKiAdt4oPMBE980TZpjcn20f5ieScWmpYSYOpFMroFbZqiMTxxn+GYQmtMUFbsnKw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: John Fastabend cc: Jakub Sitnicki cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org cc: bpf@vger.kernel.org --- net/ipv4/tcp_bpf.c | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/net/ipv4/tcp_bpf.c b/net/ipv4/tcp_bpf.c index cf26d65ca389..7f17134637eb 100644 --- a/net/ipv4/tcp_bpf.c +++ b/net/ipv4/tcp_bpf.c @@ -72,11 +72,13 @@ static int tcp_bpf_push(struct sock *sk, struct sk_msg *msg, u32 apply_bytes, { bool apply = apply_bytes; struct scatterlist *sge; + struct msghdr msghdr = { .msg_flags = flags | MSG_SPLICE_PAGES, }; struct page *page; int size, ret = 0; u32 off; while (1) { + struct bio_vec bvec; bool has_tx_ulp; sge = sk_msg_elem(msg, msg->sg.start); @@ -88,16 +90,18 @@ static int tcp_bpf_push(struct sock *sk, struct sk_msg *msg, u32 apply_bytes, tcp_rate_check_app_limited(sk); retry: has_tx_ulp = tls_sw_has_ctx_tx(sk); - if (has_tx_ulp) { - flags |= MSG_SENDPAGE_NOPOLICY; - ret = kernel_sendpage_locked(sk, - page, off, size, flags); - } else { - ret = do_tcp_sendpages(sk, page, off, size, flags); - } + if (has_tx_ulp) + msghdr.msg_flags |= MSG_SENDPAGE_NOPOLICY; + if (flags & MSG_SENDPAGE_NOTLAST) + msghdr.msg_flags |= MSG_MORE; + + bvec_set_page(&bvec, page, size, off); + iov_iter_bvec(&msghdr.msg_iter, ITER_SOURCE, &bvec, 1, size); + ret = tcp_sendmsg_locked(sk, &msghdr, size); if (ret <= 0) return ret; + if (apply) apply_bytes -= ret; msg->sg.size -= ret; @@ -398,7 +402,7 @@ static int tcp_bpf_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) long timeo; int flags; - /* Don't let internal do_tcp_sendpages() flags through */ + /* Don't let internal sendpage flags through */ flags = (msg->msg_flags & ~MSG_SENDPAGE_DECRYPTED); flags |= MSG_NO_SHARED_FRAGS; From patchwork Thu Mar 16 15:25:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13177806 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 693E4C7618D for ; Thu, 16 Mar 2023 15:26:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 067BD940009; Thu, 16 Mar 2023 11:26:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 01703940007; Thu, 16 Mar 2023 11:26:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DD1DF940009; Thu, 16 Mar 2023 11:26:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id CDB91940007 for ; Thu, 16 Mar 2023 11:26:46 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id AA2121C64A0 for ; Thu, 16 Mar 2023 15:26:46 +0000 (UTC) X-FDA: 80575138812.13.10B717D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf25.hostedemail.com (Postfix) with ESMTP id EEE52A001B for ; Thu, 16 Mar 2023 15:26:43 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YibKidY1; spf=pass (imf25.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678980404; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DjALLcCTAkJ9GFGzQm0mxszTSA2UBcCYteYf8nsJBJs=; b=LcAsV0v3134j5eSAsUO2v1/vZKRQsq2RdqrS8sgJO6jT7uETeS+8zRuGff7QkpzDO3zfIB C8GtBKrDOqF03lZfUJ75uKLbW1s2in973pkpja4KezRlQ1VOlaofxbsoHCB2udIpklVrqR yWsq5iLm8aRDqeVuO4mQlyMoUi/GmGs= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YibKidY1; spf=pass (imf25.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678980404; a=rsa-sha256; cv=none; b=HtlBT9roo0uPcB0zlm8NBgP5ImIWX/Ed2qmQ6ugYuTx0r/ex8oxuWEsbkv6uCV2rli8FnH 8VXbGszeHAuo5Hop+VRPLnB4sKkOqN2Jq1IXTAyRr3ZpDjYIKFx6iV4PREikRN1OV/1eQO gHBCeMU4ZT9/OZbMaBl4BDhLUVhMHUQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678980403; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DjALLcCTAkJ9GFGzQm0mxszTSA2UBcCYteYf8nsJBJs=; b=YibKidY1Dmv7PHLzBMqh1fCh7tjC8dHSN/r4N98+33o/2stv/b20iVLWpKBCjDF1WgsHVY n2zJ0Z0Q7/p557yb7y5L4872pJh8+XTB5cc7FJl0/8pDIIS5rbOhD/4EFPwIdxewE4UM6O EJ56ufjvGNhoIwcUyilsByyvE91grtU= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-403-UpLocroTPSOLnFgApGw-Ag-1; Thu, 16 Mar 2023 11:26:40 -0400 X-MC-Unique: UpLocroTPSOLnFgApGw-Ag-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id ECF4C858289; Thu, 16 Mar 2023 15:26:38 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id D7E9940D1C8; Thu, 16 Mar 2023 15:26:36 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Steffen Klassert , Herbert Xu Subject: [RFC PATCH 06/28] espintcp: Inline do_tcp_sendpages() Date: Thu, 16 Mar 2023 15:25:56 +0000 Message-Id: <20230316152618.711970-7-dhowells@redhat.com> In-Reply-To: <20230316152618.711970-1-dhowells@redhat.com> References: <20230316152618.711970-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: afz54yzbdt6mxbqs4es3y78i3dgaoe8b X-Rspamd-Queue-Id: EEE52A001B X-HE-Tag: 1678980403-479216 X-HE-Meta: U2FsdGVkX1+f5FwjRiRKW+P9sEn25Kzxgx9nkAWj2flYlXsK2cPrwPJJ3nvV1FwaxTUV9yaZnTBq16yE9bdqhv5o/lqFltmX3KKneMztLGVx8bk1rvcvCVDHfdyTuUaSukv1K8kj8HWYiSYghZXwAAqTs06BoSnvXuJCEaMFC8OedbbL5jmNfW53BqRxzY+kvyNz6ar0q1ZIZv0Z2AqOh0pj+ybJX/4MTwJHWr81URt9b/n+gg443DeUWRtg8Yyijacih6lgLQabnM5rP0uOIp0wxA7ikPVYHJSvd81iCmyVBg53JKzXTgDQsatuAW9nt6nRmudKJNpIFxBu25Sv7xYLCc3Meaj7HFLXSlfCJhsEXp2U56JjymOnnkwPg2ECEzouJNXg0r/b/DgNG5uLy5xru9bop1iHfqSjwrhyuJ+g/jnwPieDr1glWU1lYybbeVRxeWA1Sw7JVJ3Sz7Xqc/5OFGi3/zyslxxaL2odK0tiRtfHpjbk017NXPnH4reXp+8zkDi8dRSrNdoWtiDCke9jys5vhTEGByRXX6/m15TO1kuHMhJpIOg1WwO16TnSjvyfWa17EAmAWyVg+Vl3akb7Jfi7+Y1ATZXh4NKx/lpyVWE8HjhXmX/y9fwlN5H3ipUur9p8SP8QYitY6FNvxaNNGjifVBE4mQZfMNB7X/rWtC93yh1TOzDqEr/sG4rzGqq/Y9gFlhcdIumKLfIyFlhvIHT3gde1gCqYlUSZbHWt2WvBadFX5hjODOAtVqHTIBdx/OuIqghwXu7yd2kCX7ni24FScKufLXyMebIjDldvzVXzwrzjbVa6SB1GdvVwN03NrkYDu7G9ATZ0pvZtc22aDYbZyooPMuH+651W7z4wPuleNCHhmoBcjaG/1M2Rz8bjBGyERC1D3WVtNEe27yRdSgKEU/xF/1MuGuy1cy3tMY9WE7YGwb+JWfOdgIG9/yp1nBKFIcHaEhr7TiX q+6/3IaV 09PiOQSvurgg7PFifP9dt4EYh3Cn5LJe4S0dI4embHtb+Yg1gkUpIqChTqMVguU8RzE1x0BnBNWUO41LjgPbItNKk7BBPHd8flxR9uL8dt5XxM6xIodhXztmyImDpBkXZgVonN2pFkW1ujHfDEOH9Em4y5GHaQPZA/4OKNobsalIx7ZPTD+0iThkqRD5z8Ow5GXRSmjShuCN88SkiWF36VsbVPixmT//+Kgs5gVwDTNzbIuyV86P+fHan92hs7Nz3GqV/2Dd7qfmavBTXLWZlX6WxoFlIEIYlje91RZG7ktcIq6bZ69SBcXmRQXEbgbrziBQDixhpTktIoR1w4iDotUkwzzMkypo3gZUzEYoWZZTu9zFY606vJulB5og/sKRsH+z+Hf6iWPt9dFrskOtfNlczESs/UrHjE/rj0w/TRBrTk3QG9ngf8qtI+bT473DdpClyrulbZj1Hb2fUHeoJEpHndp7aO3aaCWfywTzpKmaTtziq8tWEPJPD1F8RLkofLHDaKhR+9pHUc75xa44/KZoLB0zJRGbgZXd/tBQVV0DDwi+mIJzX8PBGZQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it, allowing do_tcp_sendpages() to be removed. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: Steffen Klassert cc: Herbert Xu cc: Eric Dumazet cc: "David S. Miller" cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/xfrm/espintcp.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/net/xfrm/espintcp.c b/net/xfrm/espintcp.c index 872b80188e83..3504925babdb 100644 --- a/net/xfrm/espintcp.c +++ b/net/xfrm/espintcp.c @@ -205,14 +205,16 @@ static int espintcp_sendskb_locked(struct sock *sk, struct espintcp_msg *emsg, static int espintcp_sendskmsg_locked(struct sock *sk, struct espintcp_msg *emsg, int flags) { + struct msghdr msghdr = { .msg_flags = flags | MSG_SPLICE_PAGES, }; struct sk_msg *skmsg = &emsg->skmsg; struct scatterlist *sg; int done = 0; int ret; - flags |= MSG_SENDPAGE_NOTLAST; + msghdr.msg_flags |= MSG_SENDPAGE_NOTLAST; sg = &skmsg->sg.data[skmsg->sg.start]; do { + struct bio_vec bvec; size_t size = sg->length - emsg->offset; int offset = sg->offset + emsg->offset; struct page *p; @@ -220,11 +222,13 @@ static int espintcp_sendskmsg_locked(struct sock *sk, emsg->offset = 0; if (sg_is_last(sg)) - flags &= ~MSG_SENDPAGE_NOTLAST; + msghdr.msg_flags &= ~MSG_SENDPAGE_NOTLAST; p = sg_page(sg); retry: - ret = do_tcp_sendpages(sk, p, offset, size, flags); + bvec_set_page(&bvec, p, size, offset); + iov_iter_bvec(&msghdr.msg_iter, ITER_SOURCE, &bvec, 1, size); + ret = tcp_sendmsg_locked(sk, &msghdr, size); if (ret < 0) { emsg->offset = offset - sg->offset; skmsg->sg.start += done; From patchwork Thu Mar 16 15:25:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13177807 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF634C6FD1F for ; Thu, 16 Mar 2023 15:26:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5BD9094000A; Thu, 16 Mar 2023 11:26:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 56DE1940007; Thu, 16 Mar 2023 11:26:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 40D7194000A; Thu, 16 Mar 2023 11:26:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 2CE2B940007 for ; Thu, 16 Mar 2023 11:26:49 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 00C331A095F for ; Thu, 16 Mar 2023 15:26:48 +0000 (UTC) X-FDA: 80575138896.15.6D5F543 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf12.hostedemail.com (Postfix) with ESMTP id 3D7494000F for ; Thu, 16 Mar 2023 15:26:47 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=G79rlx14; spf=pass (imf12.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678980407; a=rsa-sha256; cv=none; b=1em+gigfElt4SNq6vTbPZxG/meayMIwXAAEFs9kkj11X+PFgYedqoESzYEyLyeSSi7ywdL FupcRScU2pa51AvXNVMSXrK6yPDcHrZEwX6jLnOKEtqlrCfdDtSUFeR2MAON3mCJkPf4Z+ mdZY7g35Z+Kb9alxmiA6vGUQsLkrfiM= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=G79rlx14; spf=pass (imf12.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678980407; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OLS5n4UNczR7Lntud/yq4c19A8akH99w6SN5hz2uxtc=; b=SiNaJtC+GvvWE+EIJQOFEzAnJJZgtbwpm78hKMBW/rwFD6JOa4yXM2dDbuTVs0dKv1ZA+v +5F8cO7Mx51/SYcyQ0CrAdK1qJCJCmjRL0DpnouvAQq+e8rodgfnS8wMexXuQ21DonMfj+ 8mnwdwmvruRCOBKY6EgkDKwM5fKZvHM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678980406; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OLS5n4UNczR7Lntud/yq4c19A8akH99w6SN5hz2uxtc=; b=G79rlx14kSMgRSiToCTS+Zbw1bkikBVAPcY+wFLOs40nnDUhLXpjBfhOk8/82baCnYK2vY s3sYkq8YLncmcwrCmFyo31hZs4QsQv15kprqom4cKun6+RRv9XKgB5k/sFDeSiWsJ8GNTF pS3D6zVpTqYMuosN8pHOPmGmbVh9LmY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-391-UMelVK58POioH5w_UJM1sw-1; Thu, 16 Mar 2023 11:26:42 -0400 X-MC-Unique: UMelVK58POioH5w_UJM1sw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7F1BE811E7E; Thu, 16 Mar 2023 15:26:41 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 871AD40C6E67; Thu, 16 Mar 2023 15:26:39 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Boris Pismenny , John Fastabend Subject: [RFC PATCH 07/28] tls: Inline do_tcp_sendpages() Date: Thu, 16 Mar 2023 15:25:57 +0000 Message-Id: <20230316152618.711970-8-dhowells@redhat.com> In-Reply-To: <20230316152618.711970-1-dhowells@redhat.com> References: <20230316152618.711970-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspam-User: X-Rspamd-Queue-Id: 3D7494000F X-Rspamd-Server: rspam01 X-Stat-Signature: 7kp43iey1dsczsypb4pp6k6zmgtu8hzy X-HE-Tag: 1678980407-930222 X-HE-Meta: U2FsdGVkX1+ZFNVai1LBQ6AEAir6cbT75rtBRXHjZrlWNVI/xB/QWqt6pwIwRMSZTc6CUuqyd+WfPRk0HlWkWp1Z0/xHFY8fJbmeBHWnBz+WGneR44SBKOpUjlzOfbszRsEeKRlwMNHEwWXK+lA22ud74aBd/maL8rYCDxO6dcNo0vK2Pvm7aBEAovMWftALAmHGWGjo9qL6hT1M9brEEYtHZSbvKxRIDNfznTRMzhO/BPICE2CWfNLMmhRIX6/BgqVC6GNI6iHVZbmF6oODegWA5nKzIh+79ce0HxO3PHNNJTv03sKXpUonEch5ibSriI35yl/spltsIthy8ixO5VNNRqW2RqpPDeFGL1XcpUbadc2EeRv8pGAdxzQujdK0iBhRB1kbTL4UGuRaKMZB9VCWB/kDXjbFwYDHfU0m5yq7D+p/87K3sLnUoIzHlLTfLORQc5qvdkJo4Hd/Ejb4PO2Fwjw/PITbKWwq82htwKBbLja8P6rIkVGqwHcoFZjM5VFipZkFj+drrZaoIjdFIdBqjd4fiLQRRDUZ/HcdyfdoyofTo60YYWtYVmkWJQKqxmzUm97pq8m7FR1rC7BxC0BXuXXov9dumO3iqGiE5tPr1ycNfQ9cTB4Op2gsHB1qxumVaEkkF7yzjmNES4eyHHplbbrinFfOw1/M5aG/vAHL8w1JWaDL3K4hgo8L+gErHgb52291oraWLM5NsVSKZoqgioqRJb6j/QSRj4cRKTeMLeuaQMVMy56gUd9J8iWU0kH0UA11Fzn5AIUVtOMmkfQt9D+EZ/J/rOFsKWugermBv2fcu7RS8Pp2Yrx8/7T5RWjCs6Ke1W60is66G2KIsnWzxwo9OKyl8xQiYoZfDWGV+VRK3eB4+inGDIYU5fVxL8G0l5Zz/1nG8dz5zU2C0G6KKLkzXJX3xodypXzfAjHxd3O+kgA5SRtU40YXai7p5UYkpQ1jdrkvkJwcwrL xRdZ1wVz rDTbMJRjorC/6Ch0BH/JKLrsO+BixHP7eD38asUo7sqkVwzyox5X3xMJTPu+OywZci375DDM+u6x5zBou6LNbtZc1syAT+m4+BIkh8Xm8TAtV/E3H/MZuq3ofZgFLrN1Nw6SSDDFGGe/byV0kpWmqme3kpk5BED9/dbFKvItce2tBblT29IdiA8+/byCEGYmg78lvP1Ni9DlfG9aNOCdlmcVWOTjw08UV8KQ9Thi+Vkb4kivdMZxWjr5wAT/rKnvUton21h9mrB9Z6LJQAR7Bylndas3lpDa+d3MUsqsnPgN9GKJOMWThckeSibqLwKXStRlTH3KTqaT3zMbpRsAS8xwQB0hqftpaF0D8uindKYMEmUNn/UxOnQQVktEv/a+USiogmkTdvDKVVaVc8BttMDnwvOjfY0Mih2QU1/oMspUFSBflm1aE31IkHZpuPkGGCoyvqg5pQ8BDYTEOEF6l7pFY0G5K1KBHhRqrAG1r/oMcL1DH1zfP41e7uE4G3ioTbeuHxMWp7O67u+CkNPBmQgdGrsGNJjNZdn4bDR/bA7QfuFYPLll3c0DTSQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it, allowing do_tcp_sendpages() to be removed. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: "David S. Miller" cc: Eric Dumazet cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/net/tls.h | 2 +- net/tls/tls_main.c | 24 +++++++++++++++--------- 2 files changed, 16 insertions(+), 10 deletions(-) diff --git a/include/net/tls.h b/include/net/tls.h index 154949c7b0c8..d31521c36a84 100644 --- a/include/net/tls.h +++ b/include/net/tls.h @@ -256,7 +256,7 @@ struct tls_context { struct scatterlist *partially_sent_record; u16 partially_sent_offset; - bool in_tcp_sendpages; + bool splicing_pages; bool pending_open_record_frags; struct mutex tx_lock; /* protects partially_sent_* fields and diff --git a/net/tls/tls_main.c b/net/tls/tls_main.c index 3735cb00905d..8802b4f8b652 100644 --- a/net/tls/tls_main.c +++ b/net/tls/tls_main.c @@ -124,7 +124,10 @@ int tls_push_sg(struct sock *sk, u16 first_offset, int flags) { - int sendpage_flags = flags | MSG_SENDPAGE_NOTLAST; + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = flags | MSG_SPLICE_PAGES | MSG_SENDPAGE_NOTLAST, + }; int ret = 0; struct page *p; size_t size; @@ -133,16 +136,19 @@ int tls_push_sg(struct sock *sk, size = sg->length - offset; offset += sg->offset; - ctx->in_tcp_sendpages = true; + ctx->splicing_pages = true; while (1) { if (sg_is_last(sg)) - sendpage_flags = flags; + msg.msg_flags = flags | MSG_SPLICE_PAGES; /* is sending application-limited? */ tcp_rate_check_app_limited(sk); p = sg_page(sg); retry: - ret = do_tcp_sendpages(sk, p, offset, size, sendpage_flags); + bvec_set_page(&bvec, p, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + + ret = tcp_sendmsg_locked(sk, &msg, size); if (ret != size) { if (ret > 0) { @@ -154,7 +160,7 @@ int tls_push_sg(struct sock *sk, offset -= sg->offset; ctx->partially_sent_offset = offset; ctx->partially_sent_record = (void *)sg; - ctx->in_tcp_sendpages = false; + ctx->splicing_pages = false; return ret; } @@ -168,7 +174,7 @@ int tls_push_sg(struct sock *sk, size = sg->length; } - ctx->in_tcp_sendpages = false; + ctx->splicing_pages = false; return 0; } @@ -246,11 +252,11 @@ static void tls_write_space(struct sock *sk) { struct tls_context *ctx = tls_get_ctx(sk); - /* If in_tcp_sendpages call lower protocol write space handler + /* If splicing_pages call lower protocol write space handler * to ensure we wake up any waiting operations there. For example - * if do_tcp_sendpages where to call sk_wait_event. + * if splicing pages where to call sk_wait_event. */ - if (ctx->in_tcp_sendpages) { + if (ctx->splicing_pages) { ctx->sk_write_space(sk); return; } From patchwork Thu Mar 16 15:25:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13177808 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9AB0BC7618D for ; Thu, 16 Mar 2023 15:26:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3C2CE94000B; Thu, 16 Mar 2023 11:26:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 37166940007; Thu, 16 Mar 2023 11:26:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2397194000B; Thu, 16 Mar 2023 11:26:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 14CEA940007 for ; Thu, 16 Mar 2023 11:26:52 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id DDD36A0981 for ; Thu, 16 Mar 2023 15:26:51 +0000 (UTC) X-FDA: 80575139022.19.20B8868 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf17.hostedemail.com (Postfix) with ESMTP id 0047640004 for ; Thu, 16 Mar 2023 15:26:49 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="J1GYn3/L"; spf=pass (imf17.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678980410; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2QKge+op/Hl6BMrdVZzEnNbF4IaGzItNcXZtJUNSCtc=; b=vM/PbsxC/J7UGcZU7ApTMD7X5r2iv657YlroeMu9JlVWAnyQxiXUwFBWOFBx/W0nmqxGCT R2scZ2isSimIDVlJ8ydIlHufNbIJ7yp05pHGH8pjREzaYN+Ab8C/nPQLxuSXc208us0HxE k0kZY8ZQ4ryO3O7iz5BtTlawW/5gNUg= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="J1GYn3/L"; spf=pass (imf17.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678980410; a=rsa-sha256; cv=none; b=d7O99DXXP/M+6N8OepYQeGPcG6q88yEAY31byyRhvk+4lNKVAmONTWkYCDWq/PQwaa/u3Q QLyOLx1f0EUti7xEytg9dxletqf2NYYeehES3CMsM54CUIVEb854MmIdKunGuoz90L5M4M +XRhHgyt2r0X4waO4x/z9MjF7kdh15c= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678980409; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2QKge+op/Hl6BMrdVZzEnNbF4IaGzItNcXZtJUNSCtc=; b=J1GYn3/LlekdLm/osM4OpklCLUbohKququFhDhuGOiFPqtLqkJsHP4i5XEeKKJIneLSisT HePekccN/bWWejtrkEgjTXQI9cwJ3Oj1V+ug7yWsyae8LPbpL7Sq8pfMgHK5WGIZF8vj0O l9fC/bxVXNtssAmTvXMNNiesZWEAeuI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-632-75WtpKLgNty4TtjE6A487g-1; Thu, 16 Mar 2023 11:26:45 -0400 X-MC-Unique: 75WtpKLgNty4TtjE6A487g-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4412F858F09; Thu, 16 Mar 2023 15:26:44 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 319582166B26; Thu, 16 Mar 2023 15:26:42 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Bernard Metzler , Tom Talpey , linux-rdma@vger.kernel.org Subject: [RFC PATCH 08/28] siw: Inline do_tcp_sendpages() Date: Thu, 16 Mar 2023 15:25:58 +0000 Message-Id: <20230316152618.711970-9-dhowells@redhat.com> In-Reply-To: <20230316152618.711970-1-dhowells@redhat.com> References: <20230316152618.711970-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 0047640004 X-Rspam-User: X-Stat-Signature: xen87i9k8mrgrnkr3stxj6ws3qrrtfmc X-HE-Tag: 1678980409-10236 X-HE-Meta: U2FsdGVkX1/cKQuvNReI5FID+v7BO2vukw5OdlAsL+fykYfuvVObkOVYREbOrNq35EgY5B7rxoSh/RdZWvYEfwbvS1/3fA2KFXFY8lT7uZ7buTxXlM5n/UDuhA+QNT85KhuTm/vj0J65Y/1SPE+A8LZxOaDtoEsb+IuqnpHr5Zgz2J6S62lbIL080bMA2bUor9pYRTjX/smizsQyE4Jac7ubbpKTldMJ1p9JGJzNqktkXeM5bOOzcm3UkWc0a/kT/hcTcTQjMHJ3TGH3+IMWsPB6KmuHc+6zyqXe/xF80bvrAS+ldqg5lLxOXjCdRP4xKpm4wPETfP48kQHKlIEvyInwK6uuqjtXLqI2EHNk1W+msojcVZNM+j/wC/eb6hWAtB6fHSe4EcWzz+fq/yZ8xXe6dJmX7y6RQ3k705Sgk7FsugJKhom35WpRPNuaVRV4FH11KYbGumyowiwtvEtLOXN2BKmXi77b9/HtilVfq7aUKPUx/FHg0XJw4kTGjLwnSmhyQt4JAF/lK7JZC+Ryq8dEVV0Egd92wiGIZnr+eFALinEDVfTopkytNa+eXldLjpjddJbu/iH+fhwlFrOUnHLYCUBhbOStmFk7xP62DbovavxGkD0YjGtJNbyFRIhidxIr/4ZOeMqEqkGIwnPZRCKno98gV/4QifTNmWUeWEHDDx/SqQpdDv9coRwXMd4n94iwEybwGZ57v4pIti6rAzwgUEpU92+62XNJVGuTdRQvCCHjv8T5HTbl+75dmfRYv3lO1Y0Xff2NFJCGjT160JuWEzW16uk2CfjRWPQDaTHM9C12SIs82eESCjxfT4jtrLPO3i5sS9w/OFcW2NNC3vHeIN0MgFrGyulCZd54OmgYERGNQdChh8zTwvsqdAeD+HvA+tJHclRdbVvDhDvd1zdqsY1MbJFDWuEEQG0WoWOiNXvjaMsLTaFsYFb3/eU+Ymh44zii7gITLrU+i02 eGrS/HUI zMvU8m3URh1kQXB33l0CwZ2yzY2PI2T3gpbvsBuH6WgV+6ZCkLwQP+eqXlY6PRSIVlXkwpVAOnAKNtLsb/EB3XzuksW5k1R2OKAMuAHouAnxaXqYRasTdZUunfUuJJfIIiHwMl/XWLFUO4lhOGKDPl1NbchKCZMhE/YPgPVGw0j9jR8Jo8CbyLBVMXUIgtqncmVmletpw/+xkZ0ZVe7LO2cFl1qvzTnPxhWthm3BmHyMGFIWEqM6/xCEs+vUOMaPhwuVzcK84Kk5PSCWZxJQLVJACHgNJ03fDISwFIa5EW6Fzw94m1XhPXS4q9ngVLz/i9MBm1b4boSUnehAu0EcaZUrpdJmZUEBd6dWLl2JZB3mDre9tl091O/TGEeVrnnuFMrvfwA1BkpU/YbZ+j5/7Xnm97KgzBkh/0041526oSs07Qwy5FlYuVziQRJgHIbEnMnPos8gpjGl5wZ6boh9RMiKXZxzny9yE/6wvYnx3py0O169pCBOoe8VHMjUZ0DIUv/pgVNNkMWlKaos= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it, allowing do_tcp_sendpages() to be removed. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: Bernard Metzler cc: Tom Talpey cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-rdma@vger.kernel.org cc: netdev@vger.kernel.org --- drivers/infiniband/sw/siw/siw_qp_tx.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/sw/siw/siw_qp_tx.c b/drivers/infiniband/sw/siw/siw_qp_tx.c index 05052b49107f..8fc179321e2b 100644 --- a/drivers/infiniband/sw/siw/siw_qp_tx.c +++ b/drivers/infiniband/sw/siw/siw_qp_tx.c @@ -313,7 +313,7 @@ static int siw_tx_ctrl(struct siw_iwarp_tx *c_tx, struct socket *s, } /* - * 0copy TCP transmit interface: Use do_tcp_sendpages. + * 0copy TCP transmit interface: Use MSG_SPLICE_PAGES. * * Using sendpage to push page by page appears to be less efficient * than using sendmsg, even if data are copied. @@ -324,20 +324,27 @@ static int siw_tx_ctrl(struct siw_iwarp_tx *c_tx, struct socket *s, static int siw_tcp_sendpages(struct socket *s, struct page **page, int offset, size_t size) { + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = (MSG_SPLICE_PAGES | MSG_MORE | MSG_DONTWAIT | + MSG_SENDPAGE_NOTLAST), + }; struct sock *sk = s->sk; - int i = 0, rv = 0, sent = 0, - flags = MSG_MORE | MSG_DONTWAIT | MSG_SENDPAGE_NOTLAST; + int i = 0, rv = 0, sent = 0; while (size) { size_t bytes = min_t(size_t, PAGE_SIZE - offset, size); if (size + offset <= PAGE_SIZE) - flags = MSG_MORE | MSG_DONTWAIT; + msg.msg_flags = MSG_SPLICE_PAGES | MSG_MORE | MSG_DONTWAIT; tcp_rate_check_app_limited(sk); + bvec_set_page(&bvec, page[i], bytes, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + try_page_again: lock_sock(sk); - rv = do_tcp_sendpages(sk, page[i], offset, bytes, flags); + rv = tcp_sendmsg_locked(sk, &msg, size); release_sock(sk); if (rv > 0) { From patchwork Thu Mar 16 15:25:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13177809 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C3E1C7618D for ; Thu, 16 Mar 2023 15:26:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DD4B994000C; Thu, 16 Mar 2023 11:26:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D3556940007; Thu, 16 Mar 2023 11:26:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BAEC194000C; Thu, 16 Mar 2023 11:26:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id AC5DB940007 for ; Thu, 16 Mar 2023 11:26:55 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 6F5201C65F5 for ; Thu, 16 Mar 2023 15:26:55 +0000 (UTC) X-FDA: 80575139190.28.0695C01 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf25.hostedemail.com (Postfix) with ESMTP id 82375A000B for ; Thu, 16 Mar 2023 15:26:53 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=SoPbCgCQ; spf=pass (imf25.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678980413; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QhIOxGsJKwUbwgq8gdEm1+0LSvzfey76e2D0sxuOMBQ=; b=AF6lN4pEAbKrBY/4jdG0a5ae1Klik8vbjAsCBKJCrZ8qDRgYoZIvOOIuO/Q6zXYHipskPm V4IqoMes8TFFfxtkWMZporWQONQHlyHFe3rIf4wHevIcXqoLnaBjGCUO6odc0CeYNeuCQX rptudEdNCVnj2CyoTvKS6XhJTWv+iKA= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=SoPbCgCQ; spf=pass (imf25.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678980413; a=rsa-sha256; cv=none; b=N9BVbvuOBgO2GbPARnMAYFgGNNWzvpeYVrn+D3QV00PWvUX2l0jfR1EuJEsy30zGV2Pg6G u/I8KpPHms/XhAycXAZvTac/YTptCBlyOEP/DCvXz7AyJTPpBqQ9BWoiA230w2GPv+rasN KVW6SfzY36zbYGebB3H/N0n9U0mcvLk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678980412; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QhIOxGsJKwUbwgq8gdEm1+0LSvzfey76e2D0sxuOMBQ=; b=SoPbCgCQJ3oGC1CEGNZgd7g5eCleJvkgjyX2SKE6xidH21Bpmhlhkwb5ZNJv2ZPk+DgQvc iEhk9j+s2WIDwOjj/e/nMhYE6CUH6fNC6Gi7Hm/EKR8+kyWNO0GQJRjO9WpinEVzeWpZmi PI21bLB5j7sFZwX2/R2pMnPBkpSsw+M= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-393-LNOwTe9INm-jM2IkB3Pl_A-1; Thu, 16 Mar 2023 11:26:47 -0400 X-MC-Unique: LNOwTe9INm-jM2IkB3Pl_A-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A44A8811E7D; Thu, 16 Mar 2023 15:26:46 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id DA02540D1C7; Thu, 16 Mar 2023 15:26:44 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH 09/28] tcp: Fold do_tcp_sendpages() into tcp_sendpage_locked() Date: Thu, 16 Mar 2023 15:25:59 +0000 Message-Id: <20230316152618.711970-10-dhowells@redhat.com> In-Reply-To: <20230316152618.711970-1-dhowells@redhat.com> References: <20230316152618.711970-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Rspamd-Queue-Id: 82375A000B X-Stat-Signature: q3t9xtngsitei663tfa36tjsimjbbcmq X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1678980413-984186 X-HE-Meta: U2FsdGVkX18fui/+8v3+oSODurRFczxUS96sdVpfJjUhC1DLu5maZOHDm0UBDnfS/n8xe9aURLevpFT+S2kMy+wE9tP2DlF/k0F2XmkvqnObEax1LFtPTvg13sRtEdKGlrgDnNrf+0LPObsKIPSFApJrRkRzPt8E69gefSW2KK2FHuoyqzVFggKMeXCEsrjdInsXT64ZpnsShuDUFgOnNeLoTn/BbOje+e6YZesddgkko3ikv/m7jJHh+4LnlG4hXIcpTNBFkm8DYCh4MocKqNKdedg7doGxq7h5bIXag8dCW20pk6p23ZhjhImVmDMir/asL+Ej8+vHKGN6b7Bknv0kZ6CFxl4dcLWGjaRV3jlWBpkj8V1fDRw3itRr8Yv4eZTCbCgLc0jhUCTeI5F445klfUIB8BB0NrfnrESwZKEMacoCX3ZN1rlXdCmtvCJvvpR730QrWw4ArOwJ9GBlEO82GJ7CrrWdQs1/W+i+5hOAVw1kktokdDC/GwrN+SWDnnd3Ajl2CiNUZboiQmCeHgb6eDluOPLBXZ2sRgbEZj98RbvuuHzrwFiAzKY7xqq0UzTNU2XaxdiWTWDU6l3aNC30p/PsBlJ6tSJIsvKmiOZoEx8SYNHAtYDHm2WtiMo7dM5nvCWosfj+LsSgn6gn4XDWxKs9vaEQ0HXli/ic/b8h7DU/yUb/JkUwVo0O+hTgEoOrIOEDDe4LKOZp08iDo0j4JKWwKCUJAcyAaQc0K/1X5s9uhVQFpJg0vOe/Ig61c7luuPS6p3XtrvJrCDBos25NOKsvFEgBG4+LfZUwJiOHAHFiiapZxyoIIP5f+WCJU8Vg0752oy45hHXiO8WEgMcGH3EaFh4qyg6hAvSbGlBwD6hpYLtVgLujD4xLtKCUQ325C2fybnGCW40eWR6qyjp1GZHDr4HyG1yLY5pAIHYWhSqHp8wh6z2rvYUVlXqrk/WhNr2eTW7NlB4LHb1 bvQ9L/WZ rr8Om8CzUBUI9uWhUj+65wqJrvhe1+pKgXoQcYB3VCVjYzdvY4xGEkmdVz0KfIHfF7gDT/GggzqbBjp8RDu8qppGex9PQwUxOQr1I1zVcLA8X+M/OzUWnyLSySuZ2AIKUB76VN2s+9OqsQzyL80HKmdNy7ik7eojjXqq2mgK1iRFysZyqS5VyAixSIsPK7bs2/LtILVahFKIiPMtfQZ+eMLmBpc9Zzo/IULu5VNHynjLq0u66DhHAYguCdxJAH8EDMC5qKqgYGYMpy+b3uzk2cv7kkG9gr3WE82JhgQk6L1gelzoHYC8GIS4U/9U2i4HyM62tgheWIpnTSKBy/2C0Bf+S1/yx6wJm+E44HqniBAF/QXyG5Jx+zSZK5fqqD9dK/X9miVB++hVvgx48PfAOJvTdEXhAZFkKrJ1/80/mVs4F5IfIn7EIjb1eGRrSOvg4YLCv X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Fold do_tcp_sendpages() into its last remaining caller, tcp_sendpage_locked(). Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/net/tcp.h | 2 -- net/ipv4/tcp.c | 21 +++++++-------------- 2 files changed, 7 insertions(+), 16 deletions(-) diff --git a/include/net/tcp.h b/include/net/tcp.h index db9f828e9d1e..844bc8e6a714 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -333,8 +333,6 @@ int tcp_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags); int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset, size_t size, int flags); -ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, - size_t size, int flags); int tcp_send_mss(struct sock *sk, int *size_goal, int flags); void tcp_push(struct sock *sk, int flags, int mss_now, int nonagle, int size_goal); diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 7c3acc5673e9..f1454e4497df 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -971,14 +971,19 @@ static int tcp_wmem_schedule(struct sock *sk, int copy) return min(copy, sk->sk_forward_alloc); } -ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, - size_t size, int flags) +int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset, + size_t size, int flags) { struct bio_vec bvec; struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; + if (!(sk->sk_route_caps & NETIF_F_SG)) + return sock_no_sendpage_locked(sk, page, offset, size, flags); + + tcp_rate_check_app_limited(sk); /* is sending application-limited? */ + bvec_set_page(&bvec, page, size, offset); iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); @@ -987,18 +992,6 @@ ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, return tcp_sendmsg_locked(sk, &msg, size); } -EXPORT_SYMBOL_GPL(do_tcp_sendpages); - -int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset, - size_t size, int flags) -{ - if (!(sk->sk_route_caps & NETIF_F_SG)) - return sock_no_sendpage_locked(sk, page, offset, size, flags); - - tcp_rate_check_app_limited(sk); /* is sending application-limited? */ - - return do_tcp_sendpages(sk, page, offset, size, flags); -} EXPORT_SYMBOL_GPL(tcp_sendpage_locked); int tcp_sendpage(struct sock *sk, struct page *page, int offset, From patchwork Thu Mar 16 15:26:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13177810 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1E52C6FD19 for ; Thu, 16 Mar 2023 15:27:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8060494000E; Thu, 16 Mar 2023 11:27:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7685F94000D; Thu, 16 Mar 2023 11:27:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6091594000E; Thu, 16 Mar 2023 11:27:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 4DEE8940007 for ; Thu, 16 Mar 2023 11:27:00 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 2AEFCA0917 for ; Thu, 16 Mar 2023 15:27:00 +0000 (UTC) X-FDA: 80575139400.06.A18C8F3 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf11.hostedemail.com (Postfix) with ESMTP id 70CB140007 for ; Thu, 16 Mar 2023 15:26:58 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dML2t9Qr; spf=pass (imf11.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678980418; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GYA122zSJc6JqJKbnjvsKVO49kEubsqnJ80XNajw+5k=; b=5OmD3SLTHFa0n7R/l5Vq6ZLt6zHP541u8wJAUavGLGyJR0ZONvr3uWc9ZtIgtvSTuNQ7t1 K6ZWhJsL8T6TfztCY2sWHj0SNsYFnO2Xau7JvE+1cNbIenJdBvPv90Le+opyh2R4bFmjDj murKPW3/SQO34HryT5BNvVmLZ8o6TGw= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dML2t9Qr; spf=pass (imf11.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678980418; a=rsa-sha256; cv=none; b=pVNwgBvHzUWaQOuKD6QBxsAXBN0A5x3RtzAe/KHJRxsDYFKXzjkBuIMmSKJMu0PiZwEOPu PREIFnYesO7DdoXkWbAoMk0qrD2Uxhx6rsLa7OZHBvZUEje4tfk3n5JN7uW6F2vlimCyQx 8QkbahbxKtAxV7Zz013v76XAXgqqweI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678980417; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GYA122zSJc6JqJKbnjvsKVO49kEubsqnJ80XNajw+5k=; b=dML2t9Qrc7YKyD/Sqt/ql5EPJcBAveStpXqd09CkMFao/umfAG1ZDxPnCoamLBJto1S5W3 dbDv6AvW7RZRDo795BdwR/DmfBMJt2uAIqq5AgUCpwgw9lo//7fvInrhXNqmDsSrZZn2fF J9Tw2atiXkRnaBL0qwrD1E2Nw2H8zJE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-489-LkkusC7mPii-DXgerJyhEQ-1; Thu, 16 Mar 2023 11:26:52 -0400 X-MC-Unique: LkkusC7mPii-DXgerJyhEQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 431AA101A553; Thu, 16 Mar 2023 15:26:49 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5D14C1121315; Thu, 16 Mar 2023 15:26:47 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Willem de Bruijn Subject: [RFC PATCH 10/28] ip, udp: Support MSG_SPLICE_PAGES Date: Thu, 16 Mar 2023 15:26:00 +0000 Message-Id: <20230316152618.711970-11-dhowells@redhat.com> In-Reply-To: <20230316152618.711970-1-dhowells@redhat.com> References: <20230316152618.711970-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 70CB140007 X-Stat-Signature: juixoaxmudet6rxbk7j11ijkgkokkz7d X-HE-Tag: 1678980418-544188 X-HE-Meta: U2FsdGVkX18uHsBjSBC3IHqEIV1JATNyWP8SYcmx4P2plxMWNpFuziFV3oU71UNFqrZc2/OWiSmqKqtMv4y4eQm/p5HX1tW4StMBeipdGgS+jNbBsDUSTFZkub7WR+VB01+UuXddojcwMINnpXVWNA6ezqxdac7G07ZzVW7Y5/FrdgThsYm3o4PCSn6GnzxNVC0IdB+5bG5GaIFGMSBpUHvTYUNA/m9fQLcOKAgggYLxaIysNIvSvqgYy0KkhpBHChhe3dCPIf+4Rq6ppFnZx13qQMu0Hr7WLg1KN+Yau0c+J5EjtJoILX6O5eMSNZ+zuq0TmGj1IqsEpVgU1MRTfEgI+VuH4e2JxWslWXis70ppapNVBv+38pKOix7k5G5vfEn/aJiGcxN/h6Qpar0whmEfu3NtqTg8NUQSInrZQRCuuNENaR+++Lg/6ThRzs82cAninp5LUB1Vq3AIb5Y/n7dBoJhG8DCO8HSLeL+RELwVClDFOFS+OMcGY8/fDqCkFYlFJ1kGVF21Ci//prEQZXIZqAgYgmS7IwTl1AcaCPVYdM6/O0qMI7EgEbl09hc7oolWhkcK7pUEoXmc2sNs6Jnjp76zEF2+oqZR2FughT243tSBU6Xt/weQq+MM7vENpuHy/GgGDXiqbAWy8Xmimn+QqGA0LxgS88ijuMC3NSaUWwj5A5VijhaS6bB/kyIc5hxYaAWikl13aheIJIcJpS4D4IsGiuLLI5rJqjpZaLQZoaijBW5/50B7PaNkj0dGk7n1AYUpuGmhCpWCvfInKNLntUf69XrXNIxJDsZU25Tw1LHn8wHRroWt3YH9aW9rws8OYBCNoc85Rqjx3+Pyjv7LclKWbxEjR1DUBt1GzWcbO/E7ZY97y6q46okY5Zz/1NzE/Q2oNl4pL1/dLVwpKH8SWjmnjDqVBP9t8+fEFHBdzVd7QrCb2/rH7751UaWEBkg66DuQwt8MN4x/NMd DoF/u1Ml FRXi5M8H46eVzm2gDnECGMWQ487xIE5Aua7wRRpGs0EagvPQJFJw8l1VI+zO2JKo2azkZHnRHcE3sBMbNtMlYPrDZxTv3RCBBY5nyvEgT+XHH4XdzrwDWsbPCRnXZsY8so8RjzladdiRUo4pGoxVhZLlWRm3+6c5IfkityBqQiCTZzPzxaR7Jz7qloD1v+VDbywobrZAPdtB5O+pc0bGKEt+F91eUoQxvQt/WrlA5uiCrm/i8etTIPzSc+Oco0YRdFgqwlAx5gE12l+4W7zokzVZZS+CGjm1m11UqVZcex//BzgtPu7E7eRd+FdNJx+yO6kU+ivDpFrma14N6GTNqaIpy+E3VHFAlJdqn6ZLbXWkMhhbXc5Sa64mcA3OstXAywTx5+1HJdvGbL7qKJvUJ3r8GQue4+OWbdL7fl95d19fkv7mXDBx3CZo+uI8ru1HffG7hzMZ8VKC6AQwLWd6BUm4iEvpQGxlKOCEQZxgEXJ2GWBPdpjbsIoynsdO7/Uxck5SGDobCsqR+b38+apoYxwWQ9Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make IP/UDP sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator if possible (the iterator must be ITER_BVEC and the pages must be spliceable). This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Willem de Bruijn cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/ipv4/ip_output.c | 89 ++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 86 insertions(+), 3 deletions(-) diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index 4e4e308c3230..721d7e4343ed 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -977,7 +977,7 @@ static int __ip_append_data(struct sock *sk, int err; int offset = 0; bool zc = false; - unsigned int maxfraglen, fragheaderlen, maxnonfragsize; + unsigned int maxfraglen, fragheaderlen, maxnonfragsize, xlength; int csummode = CHECKSUM_NONE; struct rtable *rt = (struct rtable *)cork->dst; unsigned int wmem_alloc_delta = 0; @@ -1017,6 +1017,7 @@ static int __ip_append_data(struct sock *sk, (!exthdrlen || (rt->dst.dev->features & NETIF_F_HW_ESP_TX_CSUM))) csummode = CHECKSUM_PARTIAL; + xlength = length; if ((flags & MSG_ZEROCOPY) && length) { struct msghdr *msg = from; @@ -1047,6 +1048,16 @@ static int __ip_append_data(struct sock *sk, skb_zcopy_set(skb, uarg, &extra_uref); } } + } else if ((flags & MSG_SPLICE_PAGES) && length) { + struct msghdr *msg = from; + + if (!iov_iter_is_bvec(&msg->msg_iter)) + return -EINVAL; + if (inet->hdrincl) + return -EPERM; + if (!(rt->dst.dev->features & NETIF_F_SG)) + return -EOPNOTSUPP; + xlength = transhdrlen; /* We need an empty buffer to attach stuff to */ } cork->length += length; @@ -1074,6 +1085,50 @@ static int __ip_append_data(struct sock *sk, unsigned int alloclen, alloc_extra; unsigned int pagedlen; struct sk_buff *skb_prev; + + if (unlikely(flags & MSG_SPLICE_PAGES)) { + skb_prev = skb; + fraggap = skb_prev->len - maxfraglen; + + alloclen = fragheaderlen + hh_len + fraggap + 15; + skb = sock_wmalloc(sk, alloclen, 1, sk->sk_allocation); + if (unlikely(!skb)) { + err = -ENOBUFS; + goto error; + } + + /* + * Fill in the control structures + */ + skb->ip_summed = CHECKSUM_NONE; + skb->csum = 0; + skb_reserve(skb, hh_len); + + /* + * Find where to start putting bytes. + */ + skb_put(skb, fragheaderlen + fraggap); + skb_reset_network_header(skb); + skb->transport_header = (skb->network_header + + fragheaderlen); + if (fraggap) { + skb->csum = skb_copy_and_csum_bits( + skb_prev, maxfraglen, + skb_transport_header(skb), + fraggap); + skb_prev->csum = csum_sub(skb_prev->csum, + skb->csum); + pskb_trim_unique(skb_prev, maxfraglen); + } + + /* + * Put the packet on the pending queue. + */ + __skb_queue_tail(&sk->sk_write_queue, skb); + continue; + } + xlength = length; + alloc_new_skb: skb_prev = skb; if (skb_prev) @@ -1085,7 +1140,7 @@ static int __ip_append_data(struct sock *sk, * If remaining data exceeds the mtu, * we know we need more fragment(s). */ - datalen = length + fraggap; + datalen = xlength + fraggap; if (datalen > mtu - fragheaderlen) datalen = maxfraglen - fragheaderlen; fraglen = datalen + fragheaderlen; @@ -1099,7 +1154,7 @@ static int __ip_append_data(struct sock *sk, * because we have no idea what fragment will be * the last. */ - if (datalen == length + fraggap) + if (datalen == xlength + fraggap) alloc_extra += rt->dst.trailer_len; if ((flags & MSG_MORE) && @@ -1206,6 +1261,34 @@ static int __ip_append_data(struct sock *sk, err = -EFAULT; goto error; } + } else if (flags & MSG_SPLICE_PAGES) { + struct msghdr *msg = from; + struct iov_iter *iter = &msg->msg_iter; + const struct bio_vec *bv = iter->bvec; + + if (iov_iter_count(iter) <= 0) { + err = -EIO; + goto error; + } + + copy = iov_iter_single_seg_count(&msg->msg_iter); + + err = skb_append_pagefrags(skb, bv->bv_page, + bv->bv_offset + iter->iov_offset, + copy); + if (err < 0) + goto error; + + if (skb->ip_summed == CHECKSUM_NONE) { + __wsum csum; + csum = csum_page(bv->bv_page, + bv->bv_offset + iter->iov_offset, copy); + skb->csum = csum_block_add(skb->csum, csum, skb->len); + } + + iov_iter_advance(iter, copy); + skb_len_add(skb, copy); + refcount_add(copy, &sk->sk_wmem_alloc); } else if (!zc) { int i = skb_shinfo(skb)->nr_frags; From patchwork Thu Mar 16 15:26:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13177811 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15FA2C7618D for ; Thu, 16 Mar 2023 15:27:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A00BA940007; Thu, 16 Mar 2023 11:27:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 965CB94000F; Thu, 16 Mar 2023 11:27:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6A57D940007; Thu, 16 Mar 2023 11:27:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4FDC094000D for ; Thu, 16 Mar 2023 11:27:00 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 2D50D1A0827 for ; Thu, 16 Mar 2023 15:27:00 +0000 (UTC) X-FDA: 80575139400.04.926DE05 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf26.hostedemail.com (Postfix) with ESMTP id 294FA140018 for ; Thu, 16 Mar 2023 15:26:57 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="YKsn3kU/"; spf=pass (imf26.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678980418; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5bgiBObt8qBauZcUWYsV1T0v3n3OqeoexntQtIJIREA=; b=laeUsLmjTagEpDVgDutbyJSYZjGXzAiWxumwBIG+GLHrNcCjbiLvFgUieT40+rAkH+QV0N O8O5KoatnD0NMICZBSstmChshT1E0uumBtQSQX2jACwSOxi6SDFDBywNJc6GdN7bMxNCke gZnbZSRTCnM6WJf6TpJQLdsk5AN9hGg= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="YKsn3kU/"; spf=pass (imf26.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678980418; a=rsa-sha256; cv=none; b=bdRYqKN7AXcKzXeFjGcSWwN5tkPZchv1qksomfcEsL8bwlMs7V2a97g2aXMIpb4/EHrckx drzqtnqJxeiaUDJPz4sdCLE0pv7PIDqRV1+GtN5WEyRYzLbXKJLeMF2ItZYlSpIGhEYiHA NaAW4nhidEFJC7HqdNaZgu9DQ6B2DDo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678980417; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5bgiBObt8qBauZcUWYsV1T0v3n3OqeoexntQtIJIREA=; b=YKsn3kU/5PAlFOh9yEwDxs3M4b8ejujkcrxpyXBn3SKGESmwhe6klFskkLAuOqzr+0SumB gPv2Dz2y6zYjRFYEnu7iMILDf3qLM/M79idct75Rwiw+Ke4TRNDmcbcHSiYsq+68eHMFHa do4VQIpP9cZkbwp5P912PX9Oay7wV5Q= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-383-a51AlkUUOb-NXl1w0lE3CA-1; Thu, 16 Mar 2023 11:26:53 -0400 X-MC-Unique: a51AlkUUOb-NXl1w0lE3CA-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 568E3185A7A7; Thu, 16 Mar 2023 15:26:52 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id D49BC492B02; Thu, 16 Mar 2023 15:26:49 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Willem de Bruijn Subject: [RFC PATCH 11/28] udp: Convert udp_sendpage() to use MSG_SPLICE_PAGES Date: Thu, 16 Mar 2023 15:26:01 +0000 Message-Id: <20230316152618.711970-12-dhowells@redhat.com> In-Reply-To: <20230316152618.711970-1-dhowells@redhat.com> References: <20230316152618.711970-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Rspamd-Queue-Id: 294FA140018 X-Stat-Signature: qso8zffq6fpium5koex58pzmag9m4xn3 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1678980417-653430 X-HE-Meta: U2FsdGVkX1+dO/Wi71VgvJy+gpDFkjB6yZeKAqXdlGfbEAFoT6LJGW9S87LX1F2JZKR9Ae8mKxlxvzbqC7BDO74f69zFEc78fwjsUBo950ZrdGn7O1ZzCdO0Q7Ee7SV4ftoAw+KfZkuMc2c+iC0B/BpscHMlqmsEqGU6vTTfWtJ4R/uoOLiQdC1677pgDrtL0snQZYNgP6UafnvGnH7JDEU0sR4eWLBUrky5I8FguC6IAzf/t9RNJUqMoy9nFM+EIGmC5XUxQRbMzt/GuRJolGTAx8p8DZJZrUnxt5PykburSuoDc0E5h0XUEKx3fSBF52Z25gurjrVZxFRHjgDjY32JJivVcoNTxyh8G75i3Zos0KY6EeGjVVpzzFaDGGg9+RDqr5S2IEtsNMESQSeU3NcH7TimQVFrQXExq1xsQi3ocHiDqLbdLoersTEJ0zZzkqH16MNYiC2GxUgb+8QQUyvJsS+lNk+McvAcLr8WTkbu5vNkbUOivZ7fvkWZPD+wbqb3fPZYkYi2ASteyPjXl8VZk3+GLoivX0AaccYnm9KQPVS/neFGlNll7nZBsIqXX12rxwYyMzWTkSg7IQOobt2YHcS+rlLAtl9uhEUbrEtN0qfRoaOgT1IQtUsoVjoXUCt0/CUZNsTCYc3lsqgIO+wolHuddmYTtdI/8hxQ56S54kKUd4yHmzKkhybIpc4hqMzOwDV2jAX6JYcCA118iwvgJOTH5jv5Y7nIhYCJxjVFirDHCz48v0YaHtTypgHBK4qFYo8AkbrHneu/OqN2IBwrqlfdPwS+Qqo8xOyl3T+aRLxnbflXgiqYelDjgIgFfZmVHw+gsOtIkvFbcVIpqTqeEX4AxN9e4eMeEki0TL5BLna2E39Hez3mlVafaTcO3q71rfFi3wXyE8rQAJPD0gYaXoaVG7LtvxeJ/bN8zEWTWe0hBfhSTT/OOtHj4qbIVqsouiP3weHvwavurv7 yl5UyW6A ZffyedTjIfuYMXkFoDRcmp0CJhxjMUUPZHPEeZkkp/5tQacdg8c8sZebY/IQb9ZFwKyHOb0VnG1wRScNJ+6zACORbLQuaHxBaXqE7Ul288MVXAcVAvMGHz2FX6TgRvwguVC3W5WVKhYEuANPctOXr0YPtkR8vdcglxF05bwmZvvyySyxSfCTTzgXkVpuyVS7csV+oLUM62+jrF+4DtVOlL1VluRy2U9Exg74MnbyTTR1DY7s5eWw8z2z40MdoZU43LhxuFr+ymPvC8YffQ26ZVD3wcvS6iG0OTLdJln17R4fZJdsnio2SKVVEqZa/RuIQJb7XKPaUQVkVE8ZP4WJGx83vJFWjxiOUZvvlSNjMDrgurQcezJgE3rnLyLSKO4Y7QZL870stO6kia9O4TcpY9u3yzvWAi6rtOcSuTL7CEoJSsh2gN09QO5AiUfut1NPGUmwCOTXDAYTzmHAObKAZmMzkBSx8hRSIkdcYP4ewF+nlMEd7KwXFByk70qx0Dba4ZQIJ+omYkaM/3xPNQ4JjZkVKQw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert udp_sendpage() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Willem de Bruijn cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/ipv4/udp.c | 50 +++++++++----------------------------------------- 1 file changed, 9 insertions(+), 41 deletions(-) diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c index c605d171eb2d..097feb92e215 100644 --- a/net/ipv4/udp.c +++ b/net/ipv4/udp.c @@ -1332,52 +1332,20 @@ EXPORT_SYMBOL(udp_sendmsg); int udp_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct inet_sock *inet = inet_sk(sk); - struct udp_sock *up = udp_sk(sk); + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = flags | MSG_SPLICE_PAGES | MSG_MORE + }; int ret; - if (flags & MSG_SENDPAGE_NOTLAST) - flags |= MSG_MORE; + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); - if (!up->pending) { - struct msghdr msg = { .msg_flags = flags|MSG_MORE }; - - /* Call udp_sendmsg to specify destination address which - * sendpage interface can't pass. - * This will succeed only when the socket is connected. - */ - ret = udp_sendmsg(sk, &msg, 0); - if (ret < 0) - return ret; - } + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; lock_sock(sk); - - if (unlikely(!up->pending)) { - release_sock(sk); - - net_dbg_ratelimited("cork failed\n"); - return -EINVAL; - } - - ret = ip_append_page(sk, &inet->cork.fl.u.ip4, - page, offset, size, flags); - if (ret == -EOPNOTSUPP) { - release_sock(sk); - return sock_no_sendpage(sk->sk_socket, page, offset, - size, flags); - } - if (ret < 0) { - udp_flush_pending_frames(sk); - goto out; - } - - up->len += size; - if (!(READ_ONCE(up->corkflag) || (flags&MSG_MORE))) - ret = udp_push_pending_frames(sk); - if (!ret) - ret = size; -out: + ret = udp_sendmsg(sk, &msg, size); release_sock(sk); return ret; } From patchwork Thu Mar 16 15:26:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13177812 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 058E9C6FD1F for ; Thu, 16 Mar 2023 15:27:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 76ADB94000F; Thu, 16 Mar 2023 11:27:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 71B8B94000D; Thu, 16 Mar 2023 11:27:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5BBCA94000F; Thu, 16 Mar 2023 11:27:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 49B6A94000D for ; Thu, 16 Mar 2023 11:27:01 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 0998580A6D for ; Thu, 16 Mar 2023 15:27:01 +0000 (UTC) X-FDA: 80575139442.24.E2BEE6A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf02.hostedemail.com (Postfix) with ESMTP id 568C98001C for ; Thu, 16 Mar 2023 15:26:59 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VMk6gPea; spf=pass (imf02.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678980419; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0tjSspNdCEV0klTPAiqijmVmAHGr/T2RpULqY6WmgRo=; b=4ehj8d+y155O3jtdGRrIGBk9PbpyJ5pYOvsxXxEQX7lkZ4C1EitFgsBozWmAd3tY1MvrIY oLYt97Bo+gjOY/WNgdzMemZAW0eCVKdGJTvrxa0QHaxFEGiX2L4pORL+dKJ79YPrDw9mB2 X3tv3L5CffUk57At/jFJ1R+76kIUZcw= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VMk6gPea; spf=pass (imf02.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678980419; a=rsa-sha256; cv=none; b=284F9bqA2dUkhU1zzH0v8jV/rnpEKQ5hOOgAFFlvM/rSJFk8Z/P3J8UAA/xbyceo9nyInM qndk6KBzq6OTV8nAhxZ+o5RYpbFVTmKnfFXN/3Ak7xlUCzJayoZqa+LjUiJ+Gnin4Qamh/ 69nEDbmu9/OfbR2oPToBTID2z1pMZgo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678980418; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0tjSspNdCEV0klTPAiqijmVmAHGr/T2RpULqY6WmgRo=; b=VMk6gPea/tO8X2//9s1aGA5A67Nq/0LNAJBchTL8RsM+Kx1kQTWr+viQUAP1vM0Dlm6Qss YWjwp0TQu+4L7HIxhmGsw0K0W69Av69XMBO8MQmQrpNHKBlxdw2bIe783T/4pMw1zEJ2lC LvkULVtIEgyOZHU94l9IAi1BLcQiWjo= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-84-pxJorDNpMoumBCRZt3rTBA-1; Thu, 16 Mar 2023 11:26:55 -0400 X-MC-Unique: pxJorDNpMoumBCRZt3rTBA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C928228237DA; Thu, 16 Mar 2023 15:26:54 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0A22240C6F87; Thu, 16 Mar 2023 15:26:52 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH 12/28] af_unix: Support MSG_SPLICE_PAGES Date: Thu, 16 Mar 2023 15:26:02 +0000 Message-Id: <20230316152618.711970-13-dhowells@redhat.com> In-Reply-To: <20230316152618.711970-1-dhowells@redhat.com> References: <20230316152618.711970-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 568C98001C X-Stat-Signature: fb4t94xng54gzd94qf513bunjx1dad3r X-HE-Tag: 1678980419-218649 X-HE-Meta: U2FsdGVkX1+HDUF8S1caYOMtdkA+sPE1a8WlnpWgExfNto7t+GM5ArIeGxU1ITldU1U9UNtRSoUgTnPlMMyXVs2GeQw1alXnsF8cHLWNC0eIs9RtqhXFpAqgGiGvUJTCpAYfZ6X8PCj5N2HwQrNx4kH8c6BP6TxpWdZtp7OkmPG9gz8Oj6/sckC+5uRY2MZgLtYvOZa0vyd352YeFO0ON78KDdae1XqKSxs6KT1B4jrFD6DZycaPhU9S30qMDvZclXGnoQebdebe6g/4VtK0gAMki3i326pLBXFcm8m71whdLxIqtnM3TfC2dU5gNuSB+g29wTAgmEZBMPvv92UzdqMwKiKXWIxZD0hnP4um6F5CeY9Wy8j8brGU3tSOf2wxx3eFV3tjsCRVZ23ImRIgluuwgR8bO2aHGwtcCLoBFx7gSXfjqc8ePZcGytjlP/gW1rFlYmRmyg7RCdv8dtOVwWpxwHyZBlNHysMhKO14jZtRtErOa0zrp9n6zacGN2KZ9Obdu0dZ+z8c6S5hYdvci67I6sj0M2LUC74kLJFWkU767SVED/+LvCdhNi08s+NXOOJFEIzMdnej5qViDknC9Er3HmMOy5KV6dEttYI7mL0/GLG83b4oEXUMI0A17gpvNXgJ/rb63YiXxq7uEdO+mOjpx0yCFNH4Z8cUbA7+lSH6KaE3uWdcdq9tTxv5u2X8wrTkbca38zy0PA4FtZAZp7kg2x/vO5zijJJJsCq0U0BwLI/iXfrxEPbuIKNSj1beFJjyY994PTdG8JxfFK9hs3qZTo/HkQUGOjpSkGDDVFAK6kDyvC91N12yVnlQ85ZeJJ8ZB3PuEkkGpJ4qCbmm/Fd++DG5K24BOtGEfZdpfkPLKa/tIBYv7pKBoHcDp4nJZGIy58ncWI+pT2XMG6XDINDYrseOw2+kuAFw1hqS6atbMzy9d8p1lYBWuKDCmw3/v4IxNDGn5Z7fUSF9PQF OIJ45G9h GvYCbnwEoLg//pM4EU8MxU0Jp14ybL6RSD1dZ8K70rHj7fEYnbzNFQQTeQqMG103PVaCQCfJblQw/B7AbIUJJ2idC/m70XntxkD91BW7Ap9MvqKB303PluiP6ZZ+QewDvIb0raGRc+oQJwnQptkHlY9mDmOAp8j/8KqD1rlF5BJXlL10A77ecd6ro0ZXdjixStDV3e2dhp5ee3z2oiDvqBs81t73QvIkNK5+B7Krc4gISqHeme39QQBS95ctWlR0hM/pmVJ2dO1xFdprXTAST43sPasVX9LxTSrd7bnuJiF6PF/WcoSV/BaCibh55ZxBmf5qwH1G+1k9Ysgh+imm81U+g6hl7AmaHCYysrJjX1EoDTdfkHUM4/eTSLjy5sk0upU8vJhAyoFJwTC68QFlhfTq/oRsoiRW/vFbX2Rloh0Z3RHKE67wxJ4JJLt/4La45/N0B X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make AF_UNIX sendmsg() support MSG_SPLICE_PAGES, splicing in pages from the source iterator if given and if ITER_BVEC and copying the data in otherwise. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/unix/af_unix.c | 84 +++++++++++++++++++++++++++++++++++++--------- 1 file changed, 68 insertions(+), 16 deletions(-) diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c index 347122c3575e..6f3454db9c53 100644 --- a/net/unix/af_unix.c +++ b/net/unix/af_unix.c @@ -2151,6 +2151,44 @@ static int queue_oob(struct socket *sock, struct msghdr *msg, struct sock *other } #endif +/* + * Extract pages from a BVEC-type iterator and add them to the socket buffer. + */ +static ssize_t unix_extract_bvec_to_skb(struct sk_buff *skb, + struct iov_iter *iter, ssize_t maxsize) +{ + const struct bio_vec *bv = iter->bvec; + unsigned long start = iter->iov_offset; + unsigned int i; + ssize_t ret = 0; + + for (i = 0; i < iter->nr_segs; i++) { + size_t off, len; + + len = bv[i].bv_len; + if (start >= len) { + start -= len; + continue; + } + + len = min_t(size_t, maxsize, len - start); + off = bv[i].bv_offset + start; + + if (skb_append_pagefrags(skb, bv->bv_page, off, len) < 0) + break; + + ret += len; + maxsize -= len; + if (maxsize <= 0) + break; + start = 0; + } + + if (ret > 0) + iov_iter_advance(iter, ret); + return ret; +} + static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, size_t len) { @@ -2194,19 +2232,25 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, while (sent < len) { size = len - sent; - /* Keep two messages in the pipe so it schedules better */ - size = min_t(int, size, (sk->sk_sndbuf >> 1) - 64); + if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES)) { + skb = sock_alloc_send_pskb(sk, 0, 0, + msg->msg_flags & MSG_DONTWAIT, + &err, 0); + } else { + /* Keep two messages in the pipe so it schedules better */ + size = min_t(int, size, (sk->sk_sndbuf >> 1) - 64); - /* allow fallback to order-0 allocations */ - size = min_t(int, size, SKB_MAX_HEAD(0) + UNIX_SKB_FRAGS_SZ); + /* allow fallback to order-0 allocations */ + size = min_t(int, size, SKB_MAX_HEAD(0) + UNIX_SKB_FRAGS_SZ); - data_len = max_t(int, 0, size - SKB_MAX_HEAD(0)); + data_len = max_t(int, 0, size - SKB_MAX_HEAD(0)); - data_len = min_t(size_t, size, PAGE_ALIGN(data_len)); + data_len = min_t(size_t, size, PAGE_ALIGN(data_len)); - skb = sock_alloc_send_pskb(sk, size - data_len, data_len, - msg->msg_flags & MSG_DONTWAIT, &err, - get_order(UNIX_SKB_FRAGS_SZ)); + skb = sock_alloc_send_pskb(sk, size - data_len, data_len, + msg->msg_flags & MSG_DONTWAIT, &err, + get_order(UNIX_SKB_FRAGS_SZ)); + } if (!skb) goto out_err; @@ -2218,13 +2262,21 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, } fds_sent = true; - skb_put(skb, size - data_len); - skb->data_len = data_len; - skb->len = size; - err = skb_copy_datagram_from_iter(skb, 0, &msg->msg_iter, size); - if (err) { - kfree_skb(skb); - goto out_err; + if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES)) { + size = unix_extract_bvec_to_skb(skb, &msg->msg_iter, size); + skb->data_len += size; + skb->len += size; + skb->truesize += size; + refcount_add(size, &sk->sk_wmem_alloc); + } else { + skb_put(skb, size - data_len); + skb->data_len = data_len; + skb->len = size; + err = skb_copy_datagram_from_iter(skb, 0, &msg->msg_iter, size); + if (err) { + kfree_skb(skb); + goto out_err; + } } unix_state_lock(other); From patchwork Thu Mar 16 15:26:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13177815 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 74041C6FD1F for ; Thu, 16 Mar 2023 15:27:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 63323940011; Thu, 16 Mar 2023 11:27:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5E19B94000A; Thu, 16 Mar 2023 11:27:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2FBDB940011; Thu, 16 Mar 2023 11:27:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 1AEAE94000A for ; Thu, 16 Mar 2023 11:27:11 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id F0B131A0601 for ; Thu, 16 Mar 2023 15:27:10 +0000 (UTC) X-FDA: 80575139820.25.335582A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf04.hostedemail.com (Postfix) with ESMTP id 2E0F34001F for ; Thu, 16 Mar 2023 15:27:09 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dewr0Sha; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf04.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678980429; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Fl0FVU0Ldf/kGpij9uVxFzVxVMUJY5FbZMiRTa3smnM=; b=4EGiV/dJRcGUJDRuF0paiComOufov9UB2kjOzKu6CFKqkiJBsJDX6B29OKJ2Rp6dIFvheE lIUnlsY/SDBMfFplQgGEiFZ8wGUBZoBXusssy+xeK5wskVO7oRM8EiVPB6ay+JyRaeqh5R mVgEttIuWdmnFk6BvdCx28c70m2nLv4= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dewr0Sha; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf04.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678980429; a=rsa-sha256; cv=none; b=UFQHNUP3DRcHgxuZw0hPDGADm2hPi22GgO2W7cwFjp+0PP//dkeVo1yY3x/uUELD4fqcQP ygBB1TvlebCBPkt/QYinH4ZVBwE7UBobeMsj0JMyxVibZn5iOFgpOBRLJpcIAFHPeBou9C Pf2iBxa5ynkLLbgQLk1W7oHMK7//WkM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678980427; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Fl0FVU0Ldf/kGpij9uVxFzVxVMUJY5FbZMiRTa3smnM=; b=dewr0ShahuoRosngOMtTCAP1LLI7V0r0a86ZQvdcBNMHsF2/2yN3IjDGL8Pfzd8PvbDuSs QfESLIzEyLT5SE+QvaTF2heQ906ljLGejIxgOs3i547l04BpBM0PNgGzI8nfl1X/oj4LwY 95K4gNbGjRXD43Ykk6qjkL+fTfFZle4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-216-khorhuokMxiDfSWLdIPQyg-1; Thu, 16 Mar 2023 11:27:00 -0400 X-MC-Unique: khorhuokMxiDfSWLdIPQyg-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6213F96DC83; Thu, 16 Mar 2023 15:26:57 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 64DEF35453; Thu, 16 Mar 2023 15:26:55 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Herbert Xu , linux-crypto@vger.kernel.org Subject: [RFC PATCH 13/28] crypto: af_alg: Indent the loop in af_alg_sendmsg() Date: Thu, 16 Mar 2023 15:26:03 +0000 Message-Id: <20230316152618.711970-14-dhowells@redhat.com> In-Reply-To: <20230316152618.711970-1-dhowells@redhat.com> References: <20230316152618.711970-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 2E0F34001F X-Stat-Signature: w4uja4nmxrduo1hm9crnty7rzkox6pjp X-HE-Tag: 1678980429-89704 X-HE-Meta: U2FsdGVkX1+N2lw3AwPMUL1bbMT3XUWixNVV74gsj44o1gdnPbQlRX4iYNtnBbMOBZA2xvv41qEcQo9i6LLYqk+JJ6f4jb0grFkjHeaP0shpoib9r25Ej6fSlGGt8evPlRKEBVnLca38cBztfP0hHr/qTVN5noepoa1JYMdGENEqYIeY6qbLc3rFkjRljGoBgtSIhfifnL0a+3yYOOe1HKWtVXE0CrrLsIJlYmaEazDEUZwMZutg2na6mX4kCDsULcU3m6Ex+fDt7br2OFL10jQvWC7G6SIPvIK5yhYUuiheq5j9Ee4arxo3Y7UPN+TO1EBf40smo5AgIS8EeW4K0EekCqt9JvApr1i0UmIyVFexpL8/i6o17WvndstIras5jymMmgldjI7la9tJL08Lq0HDeuKjNodKtt7L0oV+0fcL2KQr03YyMf6VlONkLIIJs9u2H4hfodGmbbYweeuquhw8MaUMgLB9jb0je154FWZqi7zAxcEf6ap1mzDy2126zXpxWrgI3czqrYLPgPc+beYUH6fh8v/HXDGOotTM7fR2R7vi0J29fxK8PEBH9GK5Usj4lIfitNWB8Hb9rczrpnNFKRjAIuM/XAmD6iUSkZURFtJmjbQvhfXqjLB6yi+KOyXGcvIk9vvEIQk0+LvS97WMFE45P4IuOcXEgUFERWd1A5scbonDpouaMT8V/l0S1jGdf48+HHmT8nQuMUab9BcC5JcegoT4ktfk/Wv4+RxQvDEBEtzI64FKv8vnSWOiYi6AZPUJqAcfC+rLvEqUwbC97cHmkV8B6Ej5nufSU3itHbfvCAakomnZVxmmxEZPOx1FT0aIHH6iBKdm8yI4egufHujP/8J+PKyW6izPgMuMbXpQ88/lcuYMiU0QqW/k7jIZUZ1JXhMTiMgT1BMCfK0iufJM9WfLqRSMvCuKyCjgUKPiVHVb6KdTCJv7aLBqMLcAfhQiXWQOIECImJq HEEqgIK7 q915vClsM8fQ2o4LhkJx7jBERjMT9dda7J44xqTQUogD0aapNrTcuyyDPg9tAYIfYziCpN1rBI7aNuyakkjC2HV/a6KLKBCEy+VaTnyiaj//C/lcuFPE61gUBufiUMBi7SmNGcsffnSNTuproHE0E9Af1b+yOiU2ABobPU9Zz/3MI09u0hw/rbgB56Oi+FYbKu1Tlfs+WOu9PhtFv4dIc/XTPbOfaR7w/eCZxZV1hP7fuzi0opMx7HrSCt10axfcIVqJercvY/4Hy82n6OlXqUodkqbbsZEYT1lPTbS/ULczkGoXiU+OESuE8x1GeReKVi4zFO6Dw8W+c/uXnF/sh/qJekAOOp62ys2dG1Y9St5xOpPERpD0Jky6kDX0NQeJhInyYXqdl2/uzPxWSQtRVeksh8nCP1OL3dd0yDaUJn35fAe54nebRsqAvKOAW6g6eUR2UMQFsRA+XL5eB7Km60Mt8zsmOKH+e7cJRwLOms7Yx3vEPxVzj2+m3cw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Put the loop in af_alg_sendmsg() into an if-statement to indent it to make the next patch easier to review as that will add another branch to handle MSG_SPLICE_PAGES to the if-statement. Signed-off-by: David Howells cc: Herbert Xu cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-crypto@vger.kernel.org cc: netdev@vger.kernel.org --- crypto/af_alg.c | 50 +++++++++++++++++++++++++------------------------ 1 file changed, 26 insertions(+), 24 deletions(-) diff --git a/crypto/af_alg.c b/crypto/af_alg.c index 5f7252a5b7b4..feb989b32606 100644 --- a/crypto/af_alg.c +++ b/crypto/af_alg.c @@ -1060,35 +1060,37 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size, if (sgl->cur) sg_unmark_end(sg + sgl->cur - 1); - do { - struct page *pg; - unsigned int i = sgl->cur; + if (1 /* TODO check MSG_SPLICE_PAGES */) { + do { + struct page *pg; + unsigned int i = sgl->cur; - plen = min_t(size_t, len, PAGE_SIZE); + plen = min_t(size_t, len, PAGE_SIZE); - pg = alloc_page(GFP_KERNEL); - if (!pg) { - err = -ENOMEM; - goto unlock; - } + pg = alloc_page(GFP_KERNEL); + if (!pg) { + err = -ENOMEM; + goto unlock; + } - sg_assign_page(sg + i, pg); + sg_assign_page(sg + i, pg); - err = memcpy_from_msg(page_address(sg_page(sg + i)), - msg, plen); - if (err) { - __free_page(sg_page(sg + i)); - sg_assign_page(sg + i, NULL); - goto unlock; - } + err = memcpy_from_msg(page_address(sg_page(sg + i)), + msg, plen); + if (err) { + __free_page(sg_page(sg + i)); + sg_assign_page(sg + i, NULL); + goto unlock; + } - sg[i].length = plen; - len -= plen; - ctx->used += plen; - copied += plen; - size -= plen; - sgl->cur++; - } while (len && sgl->cur < MAX_SGL_ENTS); + sg[i].length = plen; + len -= plen; + ctx->used += plen; + copied += plen; + size -= plen; + sgl->cur++; + } while (len && sgl->cur < MAX_SGL_ENTS); + } if (!size) sg_mark_end(sg + sgl->cur - 1); From patchwork Thu Mar 16 15:26:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13177813 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49117C6FD19 for ; Thu, 16 Mar 2023 15:27:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D5F47940010; Thu, 16 Mar 2023 11:27:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D0DC994000D; Thu, 16 Mar 2023 11:27:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BAF0B940010; Thu, 16 Mar 2023 11:27:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id A540C94000D for ; Thu, 16 Mar 2023 11:27:07 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 68F441202F6 for ; Thu, 16 Mar 2023 15:27:07 +0000 (UTC) X-FDA: 80575139694.30.0EB108A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf27.hostedemail.com (Postfix) with ESMTP id 68D9240004 for ; Thu, 16 Mar 2023 15:27:05 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=e9+4k6Yl; spf=pass (imf27.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678980425; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5aQYhc2sqiUIyaN0Flc/U8EQ5lVituTw9/JGYM47q4M=; b=DV7Hm6LPVe6723vRU7I48x6EvHTMaBr49uK0/C293C2mdogpE2JwpjWWjxBlGCZRl1YlHp CMI245npKpI+cnJrDTezLzlTOuFrpfUVwi9h9ehK4377nSp+2iRouVVfc9RnaCaX3dqyu/ 58ZpLR1LOxBPcxCIgnyvnqjXd9IhK6Q= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=e9+4k6Yl; spf=pass (imf27.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678980425; a=rsa-sha256; cv=none; b=CA7z9ymWg6vNqJzM1ecMnq/dLhoBIy+BmawqBUhYNHc118zc3RBRuA0/GyOVYDebj/Uq27 2fH3Cr1jRnoOzvA68uZN5Z5bljm0/KCjlRcsUbLVxg4IFw36QEysgYhFOa+JD+mgo+Dc7L 7/iYDR+zLJQ7JE/3nY/M+HOHHJJ6N+U= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678980424; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5aQYhc2sqiUIyaN0Flc/U8EQ5lVituTw9/JGYM47q4M=; b=e9+4k6Ylvp95kzf9VibtNuX1qMV7e7J6q3mlf2wx4DLpuRYbCl2mk8kFCJVVelP+X4ieH1 DLi3ROQIeTnUaLe+L4ySKlxQ7Nxz7IpBqPsvxYBEzQ1YRPF95nR56LVXWnGdP5neax8x6m sY24GGGAkTryUsGM3YhCW3BTxuYOtS8= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-659-dhLffNlOOvmeJOxSB9I_Hg-1; Thu, 16 Mar 2023 11:27:01 -0400 X-MC-Unique: dhLffNlOOvmeJOxSB9I_Hg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id F07DA1C09066; Thu, 16 Mar 2023 15:26:59 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id F24A11121315; Thu, 16 Mar 2023 15:26:57 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Herbert Xu , linux-crypto@vger.kernel.org Subject: [RFC PATCH 14/28] crypto: af_alg: Support MSG_SPLICE_PAGES Date: Thu, 16 Mar 2023 15:26:04 +0000 Message-Id: <20230316152618.711970-15-dhowells@redhat.com> In-Reply-To: <20230316152618.711970-1-dhowells@redhat.com> References: <20230316152618.711970-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: db488hs59n55hcsjq1qc39y7nw8r6doq X-Rspamd-Queue-Id: 68D9240004 X-HE-Tag: 1678980425-64260 X-HE-Meta: U2FsdGVkX1/UoMmlSFKUHbVsGYwS9sAW3OQ0mE1XqMcYhqnLuJxPGqeP7in2SbQMl+jRvgQgJYiemMDgFFiPYgPw4gDIfy0TX68ZVTXiAcBfBOOnel2cY2NpBSO1SzNifUlSBqPenkSPNFKwPyiE220E5fSEBOc/GGckHxpvSxA5Y0WQCF+eX8wliRQy2kpWsvC/F+5u5txXv96s4lOanED/3+4/1ib8dP4IonsC2DpX00ZTgCDjrxe03ea92RbFJqDjHX2fulZUWOjWFFbFeBIsztFOHzhqqTK3dWqMmWWGIOoK8n7YicGexoTP1R66lzpaYz7+0qhHoPnspHEAvVdkQVnBveRcQZjahRLw/8y9V/J1FtCEtEs16KSoqLNKjhwG8bmX0qGtvFyge2yOonTtKuQAO6zEGIXhdvPBIPq+9Y9LN3f57L5Cqcp0xS5+krec2eFgT0IrC7otXCggKOEN8T/bdmgQ5yBkO0JF3YINxNvCp6lWiHhjczEhUfEAuFXOXTAdf7vIilvbUUVNIfKUF2vJgjbAvK3F4Ztl064VrIcJghLsIhfOgzSkQiuJx695aI8u3ZcP2+FJbo9GfhaTQ6QdK/9nhxt3V2gj5lOyl21hzWSukjTsYfrBFQfxx218NYVknqqBaweMhThFsioHpcn46Ihmu+F1wjywdoCQRU7exwUre7vDTfu5Jxo4RbjATU1eR/J6tAR3+RMnq1DodFZ2xPL/RQ1vMZr3ip4Ll95qAT5TqSegLW8N3icaimlV6V1VpL+83kcgjPX86chfokDrPha9jZ6BjMZGwTm19qKevANhEecjEHTAoBVYPo4Zh25nnl2nylkkDsLSJCENYGae81JB17CMxyEOl5rdb/ZkmsrZ1o7ShKJ5HZgXuZrDcONAJSlBT45eopMvhNrhjvbbH5oNwA/NHvQ/aCOA8WeJgPNVNDUdZ0mSB1HUfsKJadDglc1AjzFdg5w byXOi8C6 4C5WaoK7xqcYoCmakgUvIpwQ2Xq0QdpJ1gw6oRqSxyyQuIltVzchUr8QZQkit05wds/u/LhSlUn+2wh2f3B+xHDe0GrK87wuK1XItdEsQvmy9pz9iBV+busDKL90csftbDnNFQpN5a2Lgr0sjLDAKqY9Y0CHz3gw+G4my2fjLrrlLfaskk07HzAja1raX4/vXZgkiulLE2bHIat04IeBwyHyteQGHrbLPp+BmhsniUhqo7oFLXrN5YM1X4fTfX167eKdaQVsdkHugbakP3RmWW32XqDhiHn5fQTeIUC6bDXyFzET9U9l2pbfFA2S2wzF8A1sU/fRDSDsutD+4i9qJUlh8ATIaaeeoTLvrwemm/2jE0A5PvtHabexSCTw5Hh0l+PUqEiPnxP8+GLcsUTaakCwMWY+pnNE1ID7jjZk9R68CbC5oOUFUUdbXHJ5QbC8MojXQ4gs/CgTNxYcjnG3GPRNpmQMY85OlX3daaGoEGXrGVRSMpZApw+Efyg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make AF_ALG sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator if possible (the iterator must be ITER_BVEC and the pages must be spliceable). This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. [!] Note that this makes use of netfs_extract_iter_to_sg() from netfslib. This probably needs moving to core code somewhere. Signed-off-by: David Howells cc: Herbert Xu cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-crypto@vger.kernel.org cc: netdev@vger.kernel.org --- crypto/Kconfig | 1 + crypto/af_alg.c | 29 +++++++++++++++++++++++++++-- crypto/algif_aead.c | 22 +++++++++++----------- crypto/algif_skcipher.c | 8 ++++---- 4 files changed, 43 insertions(+), 17 deletions(-) diff --git a/crypto/Kconfig b/crypto/Kconfig index 9c86f7045157..8c04ecbb4395 100644 --- a/crypto/Kconfig +++ b/crypto/Kconfig @@ -1297,6 +1297,7 @@ menu "Userspace interface" config CRYPTO_USER_API tristate + select NETFS_SUPPORT # for netfs_extract_iter_to_sg() config CRYPTO_USER_API_HASH tristate "Hash algorithms" diff --git a/crypto/af_alg.c b/crypto/af_alg.c index feb989b32606..80ab4f6e018c 100644 --- a/crypto/af_alg.c +++ b/crypto/af_alg.c @@ -22,6 +22,7 @@ #include #include #include +#include #include #include #include @@ -970,6 +971,10 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size, bool init = false; int err = 0; + if ((msg->msg_flags & MSG_SPLICE_PAGES) && + !iov_iter_is_bvec(&msg->msg_iter)) + return -EINVAL; + if (msg->msg_controllen) { err = af_alg_cmsg_send(msg, &con); if (err) @@ -1015,7 +1020,7 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size, while (size) { struct scatterlist *sg; size_t len = size; - size_t plen; + ssize_t plen; /* use the existing memory in an allocated page */ if (ctx->merge) { @@ -1060,7 +1065,27 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size, if (sgl->cur) sg_unmark_end(sg + sgl->cur - 1); - if (1 /* TODO check MSG_SPLICE_PAGES */) { + if (msg->msg_flags & MSG_SPLICE_PAGES) { + struct sg_table sgtable = { + .sgl = sg, + .nents = sgl->cur, + .orig_nents = sgl->cur, + }; + + plen = netfs_extract_iter_to_sg(&msg->msg_iter, len, + &sgtable, MAX_SGL_ENTS, 0); + if (plen < 0) { + err = plen; + goto unlock; + } + + for (; sgl->cur < sgtable.nents; sgl->cur++) + get_page(sg_page(&sg[sgl->cur])); + len -= plen; + ctx->used += plen; + copied += plen; + size -= plen; + } else { do { struct page *pg; unsigned int i = sgl->cur; diff --git a/crypto/algif_aead.c b/crypto/algif_aead.c index 42493b4d8ce4..279eb17a1dfc 100644 --- a/crypto/algif_aead.c +++ b/crypto/algif_aead.c @@ -9,8 +9,8 @@ * The following concept of the memory management is used: * * The kernel maintains two SGLs, the TX SGL and the RX SGL. The TX SGL is - * filled by user space with the data submitted via sendpage/sendmsg. Filling - * up the TX SGL does not cause a crypto operation -- the data will only be + * filled by user space with the data submitted via sendpage. Filling up + * the TX SGL does not cause a crypto operation -- the data will only be * tracked by the kernel. Upon receipt of one recvmsg call, the caller must * provide a buffer which is tracked with the RX SGL. * @@ -113,19 +113,19 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, } /* - * Data length provided by caller via sendmsg/sendpage that has not - * yet been processed. + * Data length provided by caller via sendmsg that has not yet been + * processed. */ used = ctx->used; /* - * Make sure sufficient data is present -- note, the same check is - * also present in sendmsg/sendpage. The checks in sendpage/sendmsg - * shall provide an information to the data sender that something is - * wrong, but they are irrelevant to maintain the kernel integrity. - * We need this check here too in case user space decides to not honor - * the error message in sendmsg/sendpage and still call recvmsg. This - * check here protects the kernel integrity. + * Make sure sufficient data is present -- note, the same check is also + * present in sendmsg. The checks in sendmsg shall provide an + * information to the data sender that something is wrong, but they are + * irrelevant to maintain the kernel integrity. We need this check + * here too in case user space decides to not honor the error message + * in sendmsg and still call recvmsg. This check here protects the + * kernel integrity. */ if (!aead_sufficient_data(sk)) return -EINVAL; diff --git a/crypto/algif_skcipher.c b/crypto/algif_skcipher.c index ee8890ee8f33..021f9ce7e87c 100644 --- a/crypto/algif_skcipher.c +++ b/crypto/algif_skcipher.c @@ -9,10 +9,10 @@ * The following concept of the memory management is used: * * The kernel maintains two SGLs, the TX SGL and the RX SGL. The TX SGL is - * filled by user space with the data submitted via sendpage/sendmsg. Filling - * up the TX SGL does not cause a crypto operation -- the data will only be - * tracked by the kernel. Upon receipt of one recvmsg call, the caller must - * provide a buffer which is tracked with the RX SGL. + * filled by user space with the data submitted via sendmsg. Filling up the TX + * SGL does not cause a crypto operation -- the data will only be tracked by + * the kernel. Upon receipt of one recvmsg call, the caller must provide a + * buffer which is tracked with the RX SGL. * * During the processing of the recvmsg operation, the cipher request is * allocated and prepared. As part of the recvmsg operation, the processed From patchwork Thu Mar 16 15:26:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13177814 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DAA7C7618D for ; Thu, 16 Mar 2023 15:27:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9B48094000D; Thu, 16 Mar 2023 11:27:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 93CC194000A; Thu, 16 Mar 2023 11:27:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7B71794000D; Thu, 16 Mar 2023 11:27:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 699ED94000A for ; Thu, 16 Mar 2023 11:27:10 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 442CD404C4 for ; Thu, 16 Mar 2023 15:27:10 +0000 (UTC) X-FDA: 80575139820.04.75C93F2 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf07.hostedemail.com (Postfix) with ESMTP id 6B8254001F for ; Thu, 16 Mar 2023 15:27:08 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Vn2guJKd; spf=pass (imf07.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678980428; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=t96KWWuCikizvNC1y3JF3FIfjq2ZpnpzfAnEQzXudzo=; b=oHTpm+b1ATfWvvri6rZ/Qg/25dE0zSUgq9tmdOBQ5UQ6PrM1YQmVuZqGs5tykKL657lj/s GoCy/k5e9Ez7wOcWJNAWM9WqcIDUplTottgPf3oat25Arh4L1TcIDTCdmosfoqPCuO4zky 41K1IiZFAIVH/1SLD/D5fSkMQD8U3xw= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Vn2guJKd; spf=pass (imf07.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678980428; a=rsa-sha256; cv=none; b=y8e3MJ5dYt8CCrwBdI1HbbBFTwtvKjtKMp6L5khoZcMaU2jWczxuBGa7gzd8IBv6CUhGg4 q0EybL9vcSrz+UzczSut7lj2vd5iuc62GmNI+ddOfZYORInqwwwMPHylZoQoosAYXS3cWG kBS67Z05VeUmb5dh4NjV9scv7OdKHs8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678980427; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=t96KWWuCikizvNC1y3JF3FIfjq2ZpnpzfAnEQzXudzo=; b=Vn2guJKd3tx2oyLb7btA2qRRF7Chji2NqCvIHWps6cnSwHEekofUTrM+jmlStFOxGu8f0W mzZ9a/t2o49u34uWn+EUrkUjt2c/XLMCTsnrVvFou+LqgpIIfSW9dxdHCF0QW1lHLe0b0a q687woUJM2OGD8veQ2PleUQJK0bh5zU= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-638-J7ghSUrYNvWcgXhEzMBmmA-1; Thu, 16 Mar 2023 11:27:03 -0400 X-MC-Unique: J7ghSUrYNvWcgXhEzMBmmA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A60961C09044; Thu, 16 Mar 2023 15:27:02 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id A9D8E40C6E67; Thu, 16 Mar 2023 15:27:00 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Herbert Xu , linux-crypto@vger.kernel.org Subject: [RFC PATCH 15/28] crypto: af_alg: Convert af_alg_sendpage() to use MSG_SPLICE_PAGES Date: Thu, 16 Mar 2023 15:26:05 +0000 Message-Id: <20230316152618.711970-16-dhowells@redhat.com> In-Reply-To: <20230316152618.711970-1-dhowells@redhat.com> References: <20230316152618.711970-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: yw6rxw41czgw7wohd757cbkepq569eo3 X-Rspamd-Queue-Id: 6B8254001F X-HE-Tag: 1678980428-385558 X-HE-Meta: U2FsdGVkX1/JWcc8PG1c9CPh9sIdvAYeteGwD2FivpramRHRas7uK3BwRThjyZOcDwLKtx1VyunuS+2iHYH3oEJHgTdr0aEOajiOO8JjpfxYKnb1w4sXk1fxkftz0cTUuKp/Qj8zIUQZZIXzM2ySHCchzb/RERYHJruEq1dy0ikAPPHYqKwuoffW39CdTpz3+H9mE1jS2GVWDYpdSRiT2SjoB6qbERfFRPttiEa/0yrf0Kq15R/VYt4mFZr+2OHsfARB7kHNoIqD5lSJo+EMCx/v312cbWaaRV4rdbyAqTyccsi+fWn5T9Z4clsFR4OWWWDQNeeHK23q0rllWnFap8qLydGeTlkTkwb7T2C/hWo5I6Ai9SkP/myVpCQqqO2YFcEBmZ+UMGF4jlOM9ga6ru6g0VQ5APbe7OZffxFcv00KeKa05ueqx8TMSY7fmIkKYFcpWWw9eF/byLX+Znz84g0XlA+gdTf+L5ezRTXX2GEh5MuEkAXxya00Fqa2anNkxSpYVUk29HJptfL+xFgbW3OtS9G0F3YrLkNn+EG4/MTv2jAlpXgaN56bBI/f/m+NXRRFl9s7+qAjXSYjyHk7BCZaofhObcShyQ4ttdW5k0sR1PGo5YGsseuKDubrDQFIEqfkQFwGhmTTztxIF5s3CvHkwJj116gJXS5DuzaUQX11npG5J+Rqz0CIp5dV0eHOXipTbMrjl1sbzM6VfJSPZwnV6QpoSyHMdq+NjRGV4R/AiFXw51ga78h9354mja2kt8oU3u5ZycSQH3475SxJKAdoeddQFLuYUWok4TFUJZMhjVrHUOKG2EWj5VExziOn9jSxz8CGv9puIeIO1ElKbJdeySa2jgZMWorSr5iFEBMica15dIxHIgwAtR2MsJlNaGD1Y+coG55Pp/YBXe9z+Ppcee4HToZhZgGFHJyG84RKGUtYba+VXKh4GeEPoF1wlm4g/6n1estlw4IyD5J beWCLjAY 2QQNk1o5bCN26OxvzZ9PT/RKz+W92eMs1BX7NKiB6LCHsWn7D46jOTF/YPuyrtnxbamT2eJ6Hoz58WDqrqDq2MrJQSIqgkC/3F3FMWTsFroDaFNikETv91w9+8Uzw28HfPmS/hu+giVwVwJo3HJc0EESrkrVYSI/oBlVvCxB2QW7D274XlV0LCyJE+pEdecBQrlZYgvByqle9YPjXX2OE6QoLxQd5oORjwNQY22kP3LW61A4gDxuvhQXs318OHfctwIr4t0rDAHx3KJoFlotHGlJH85Vxw2fM1Xe1OPjfhDn7o+KgMpLYW9M1Oer3NzU1k+quf11Fh1/sluEooKyIBrNSMFEem4IF+q/JETvZLzer++iL2r5FEUmaPVartKI0Tz0MjxdYOkgooWpw/spCwlvaz/GrIqDKeLWBqW5HPYB4N0hLMtZCkDxgPMZTMXv+jvnYPAWVuoytb8lwBv49TAkR/8cCT3WpxEUUcBpRhqYMuGnz1xK7grwfTQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert af_alg_sendpage() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. [!] Note that this makes use of netfs_extract_iter_to_sg() from netfslib. This probably needs moving to core code somewhere. Signed-off-by: David Howells cc: Herbert Xu cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-crypto@vger.kernel.org cc: netdev@vger.kernel.org --- crypto/af_alg.c | 53 +++++++++---------------------------------------- 1 file changed, 9 insertions(+), 44 deletions(-) diff --git a/crypto/af_alg.c b/crypto/af_alg.c index 80ab4f6e018c..0e77fce60876 100644 --- a/crypto/af_alg.c +++ b/crypto/af_alg.c @@ -1148,53 +1148,18 @@ EXPORT_SYMBOL_GPL(af_alg_sendmsg); ssize_t af_alg_sendpage(struct socket *sock, struct page *page, int offset, size_t size, int flags) { - struct sock *sk = sock->sk; - struct alg_sock *ask = alg_sk(sk); - struct af_alg_ctx *ctx = ask->private; - struct af_alg_tsgl *sgl; - int err = -EINVAL; - - if (flags & MSG_SENDPAGE_NOTLAST) - flags |= MSG_MORE; - - lock_sock(sk); - if (!ctx->more && ctx->used) - goto unlock; - - if (!size) - goto done; - - if (!af_alg_writable(sk)) { - err = af_alg_wait_for_wmem(sk, flags); - if (err) - goto unlock; - } - - err = af_alg_alloc_tsgl(sk); - if (err) - goto unlock; - - ctx->merge = 0; - sgl = list_entry(ctx->tsgl_list.prev, struct af_alg_tsgl, list); - - if (sgl->cur) - sg_unmark_end(sgl->sg + sgl->cur - 1); + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = flags | MSG_SPLICE_PAGES, + }; - sg_mark_end(sgl->sg + sgl->cur); + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); - get_page(page); - sg_set_page(sgl->sg + sgl->cur, page, size, offset); - sgl->cur++; - ctx->used += size; - -done: - ctx->more = flags & MSG_MORE; - -unlock: - af_alg_data_wakeup(sk); - release_sock(sk); + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - return err ?: size; + return sock_sendmsg(sock, &msg); } EXPORT_SYMBOL_GPL(af_alg_sendpage); From patchwork Thu Mar 16 15:26:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13177817 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DD2C2C7618D for ; Thu, 16 Mar 2023 15:27:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2BA54900003; Thu, 16 Mar 2023 11:27:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 215C794000A; Thu, 16 Mar 2023 11:27:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F0925900003; Thu, 16 Mar 2023 11:27:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D3C90900008 for ; Thu, 16 Mar 2023 11:27:15 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 9E0E3C0802 for ; Thu, 16 Mar 2023 15:27:15 +0000 (UTC) X-FDA: 80575140030.12.8AF82DD Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf06.hostedemail.com (Postfix) with ESMTP id 946D218001C for ; Thu, 16 Mar 2023 15:27:12 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=eUhtBbjZ; spf=pass (imf06.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678980433; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pTLvmMXI2CBo7xhNFEE5mUeBoYhMgPol6qu4jiB+mT4=; b=Ajkyevmc5khJM6xAaQO3uFKe3kNdupO4RWMtNjiSxA1HGAWNszsbP6OKLxiZL/bOHwLYZN Am0uZJ/f241y1cFomDrCXjL6MwrPoa3QjudRV9JiiuhLaz+t+/cpgLkkCP3VySte/jOncY sw4IhZp2TjYW6h01Ed0QjvlYQyIylE4= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=eUhtBbjZ; spf=pass (imf06.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678980433; a=rsa-sha256; cv=none; b=hhemR757DwRpcLYWMI7m4mt8f+TTU2goQpL2yd2i8Eks6CQhlDUOSYPXPLWkHQxGSvIN06 8dCGIk9mPWAsyc9dXfpOUvHqt84PF8smE3Zv41t/DtaroZYSK5VvCTzJuwkcyjLzzKFOap Iw6mSbHLCYr0t04zFX7sfHA0l5qtW5I= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678980432; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pTLvmMXI2CBo7xhNFEE5mUeBoYhMgPol6qu4jiB+mT4=; b=eUhtBbjZXxWGFaYMmS2xtQoNZnJXy/Rrr9vhwpJCEVutL8cp9Jk1S7WHrPI6vvkESgqqXn dB4XBLxCPJ2+KGa2WpK0b02xTYdVV6L+ooW3nDimFr+aFmZk2qwT14OYX0rmvW2VU3S1az LBEoa6oJ3xdTBnS7tNVF8UQJy3uDt0s= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-115-U9_GV7XGNjaKDSHHiGPkTA-1; Thu, 16 Mar 2023 11:27:06 -0400 X-MC-Unique: U9_GV7XGNjaKDSHHiGPkTA-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 100621C09067; Thu, 16 Mar 2023 15:27:05 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 42A90492B00; Thu, 16 Mar 2023 15:27:03 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH 16/28] splice, net: Use sendmsg(MSG_SPLICE_PAGES) rather than ->sendpage() Date: Thu, 16 Mar 2023 15:26:06 +0000 Message-Id: <20230316152618.711970-17-dhowells@redhat.com> In-Reply-To: <20230316152618.711970-1-dhowells@redhat.com> References: <20230316152618.711970-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Stat-Signature: tayq6xtgybchyrw561gbcjjcwd6zxwmg X-Rspam-User: X-Rspamd-Queue-Id: 946D218001C X-Rspamd-Server: rspam06 X-HE-Tag: 1678980432-747666 X-HE-Meta: U2FsdGVkX1+qdQkl3qk3zJg78JixRVB65IE3LPcgSlEQzr+TP/9zeuq1esAoPm3fRSRLA+/5taH9idxI3jCfwLkHfB/mnVVadGrx53ThnTnBdaENEQtzUSMQ0LS3kphqyzuvcCCgsucLQURF9v2PoUf4tbHWp89oer4+WZcTqsdX1TSam1NrR1VumaNW4rOt0WPKwV7GDYKyl/Bc6pRaYM8CidHGFfbggWAN5c4W89rlgf0CTPWCgO8+PGofzIVxSqewFMe1zGgv9v9pOoRx8a3YhbvY/7Dt69mf8g7iIEk1V2hdu7E4fD1/Ur0A2PzBLqiX0YUaL3nomvFotjdrD2o3VwYZkhlDH3hzZu5xRoM1+ye6LC2OoCps+XMGXdon1gf6f7ILqHZWcnbkXNZBjj9IO0iDCvLmyjRi6pjgws07BmHZMVd8QboWyEVWfEQhfm7UH0oayBResS64hXscyaVAyrlauJ5vAYyXOa/HF9rJXVBq4xjxZ5VyZ+03C78MK5kAC7T1MrclR1Rx07EkgedIT16NHOgaL7wFtkd09w/jRllnpyFZzWa7seEIIoILPmp4zWKX0LfKca0+MWVnF8LnLTPfruCsNfxafajFDU6yHEYtRF/LrdbYozjSa05jVv4dm8VR/7kpS+7ekTD071/DilQJp7QD0YCScgQTNnOzd8/CoNVz9IsWwCAMJmTL85l/YNF4IIKRX99ZxjH3yIciCe+6G4jt42T9T/svJXiuCLSFgKsfTQuubTw+PX0Qcwgk89gaS/psyPH94gXqzFu3fOOnGX8JnQNOdYF8ENWD6Kqd2l8TQ8HyIZqruTH36unwVDQj3ENBFdUzQ+pX+s4Z+fkOBIzTGzPILi9MlCNdPcnfvBTGi8kz2GzPLBvg0GxOSUHmjNBEnkrqMSq2GnYCTwAeUhDUl97JuXZXNAx4x3dnlMPWF5NEFSW94p6+9Kkex20jywcnR8QEwwp BulocXc7 veEAzzeNRd9oIBVoUZjeiyualzEA/O9LlHSFMlx3ICGBzPpaOCJb5z7lVMWgYGil6IThjG6SoB70kALW8DukFKkxx+zaY6dnbuw6XoFtj+iHWak9+QSCRIp8m24jQ+zeVUufdt5HnjN3ThUc3/UZhR0eRkQAWZKakCeZnsMy/VpP1VZ1e+6n3CujSxl1+QHQWe5PCBZ49P501/F6E3Qj/3tmGKvsI+NMge4fUP6iPAF8PC8D4zz/LJJZKZ9LAWRIDYlCPdiTev8YirvA8BAp6aJdTEAoJSoW6JDzjBdnHkFTLopTwAp7sqe4Pxz4NYaKqtMSNAL/93DGcVYXbuDkSrWE1xgAz4CXRm666uxtxkV5ipYsyJ4pOEd/3+Gs1Yuq7bO6IvEcB9ebYEVoCfoJ5UZFMp2YnrjUOeAJ1GBTU3msn/H7f7Ac/F42QPLTm22ImpeXI X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use sendmsg(MSG_SPLICE_PAGES) rather than ->sendpage() to splice data from a pipe to a socket. This paves the way for passing in multiple pages at once from a pipe and the handling of multipage folios. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- fs/splice.c | 42 +++++++++++++++++++++++------------------- include/linux/fs.h | 2 -- include/linux/splice.h | 2 ++ net/socket.c | 26 ++------------------------ 4 files changed, 27 insertions(+), 45 deletions(-) diff --git a/fs/splice.c b/fs/splice.c index f46dd1fb367b..23ead122d631 100644 --- a/fs/splice.c +++ b/fs/splice.c @@ -32,6 +32,7 @@ #include #include #include +#include #include #include @@ -410,29 +411,32 @@ const struct pipe_buf_operations nosteal_pipe_buf_ops = { }; EXPORT_SYMBOL(nosteal_pipe_buf_ops); +#ifdef CONFIG_NET /* * Send 'sd->len' bytes to socket from 'sd->file' at position 'sd->pos' * using sendpage(). Return the number of bytes sent. */ -static int pipe_to_sendpage(struct pipe_inode_info *pipe, - struct pipe_buffer *buf, struct splice_desc *sd) +static int pipe_to_sendmsg(struct pipe_inode_info *pipe, + struct pipe_buffer *buf, struct splice_desc *sd) { - struct file *file = sd->u.file; - loff_t pos = sd->pos; - int more; - - if (!likely(file->f_op->sendpage)) - return -EINVAL; + struct socket *sock = sock_from_file(sd->u.file); + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = MSG_SPLICE_PAGES, + }; - more = (sd->flags & SPLICE_F_MORE) ? MSG_MORE : 0; + if (sd->flags & SPLICE_F_MORE) + msg.msg_flags |= MSG_MORE; if (sd->len < sd->total_len && pipe_occupancy(pipe->head, pipe->tail) > 1) - more |= MSG_SENDPAGE_NOTLAST; + msg.msg_flags |= MSG_MORE; - return file->f_op->sendpage(file, buf->page, buf->offset, - sd->len, &pos, more); + bvec_set_page(&bvec, buf->page, sd->len, buf->offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, sd->len); + return sock_sendmsg(sock, &msg); } +#endif static void wakeup_pipe_writers(struct pipe_inode_info *pipe) { @@ -614,7 +618,7 @@ static void splice_from_pipe_end(struct pipe_inode_info *pipe, struct splice_des * Description: * This function does little more than loop over the pipe and call * @actor to do the actual moving of a single struct pipe_buffer to - * the desired destination. See pipe_to_file, pipe_to_sendpage, or + * the desired destination. See pipe_to_file, pipe_to_sendmsg, or * pipe_to_user. * */ @@ -795,8 +799,9 @@ iter_file_splice_write(struct pipe_inode_info *pipe, struct file *out, EXPORT_SYMBOL(iter_file_splice_write); +#ifdef CONFIG_NET /** - * generic_splice_sendpage - splice data from a pipe to a socket + * splice_to_socket - splice data from a pipe to a socket * @pipe: pipe to splice from * @out: socket to write to * @ppos: position in @out @@ -808,13 +813,12 @@ EXPORT_SYMBOL(iter_file_splice_write); * is involved. * */ -ssize_t generic_splice_sendpage(struct pipe_inode_info *pipe, struct file *out, - loff_t *ppos, size_t len, unsigned int flags) +ssize_t splice_to_socket(struct pipe_inode_info *pipe, struct file *out, + loff_t *ppos, size_t len, unsigned int flags) { - return splice_from_pipe(pipe, out, ppos, len, flags, pipe_to_sendpage); + return splice_from_pipe(pipe, out, ppos, len, flags, pipe_to_sendmsg); } - -EXPORT_SYMBOL(generic_splice_sendpage); +#endif static int warn_unsupported(struct file *file, const char *op) { diff --git a/include/linux/fs.h b/include/linux/fs.h index c85916e9f7db..f3ccc243851e 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2740,8 +2740,6 @@ extern ssize_t generic_file_splice_read(struct file *, loff_t *, struct pipe_inode_info *, size_t, unsigned int); extern ssize_t iter_file_splice_write(struct pipe_inode_info *, struct file *, loff_t *, size_t, unsigned int); -extern ssize_t generic_splice_sendpage(struct pipe_inode_info *pipe, - struct file *out, loff_t *, size_t len, unsigned int flags); extern long do_splice_direct(struct file *in, loff_t *ppos, struct file *out, loff_t *opos, size_t len, unsigned int flags); diff --git a/include/linux/splice.h b/include/linux/splice.h index 8f052c3dae95..e6153feda86c 100644 --- a/include/linux/splice.h +++ b/include/linux/splice.h @@ -87,6 +87,8 @@ extern long do_splice(struct file *in, loff_t *off_in, extern long do_tee(struct file *in, struct file *out, size_t len, unsigned int flags); +extern ssize_t splice_to_socket(struct pipe_inode_info *pipe, struct file *out, + loff_t *ppos, size_t len, unsigned int flags); /* * for dynamic pipe sizing diff --git a/net/socket.c b/net/socket.c index 6bae8ce7059e..1b48a976b8cc 100644 --- a/net/socket.c +++ b/net/socket.c @@ -57,6 +57,7 @@ #include #include #include +#include #include #include #include @@ -126,8 +127,6 @@ static long compat_sock_ioctl(struct file *file, unsigned int cmd, unsigned long arg); #endif static int sock_fasync(int fd, struct file *filp, int on); -static ssize_t sock_sendpage(struct file *file, struct page *page, - int offset, size_t size, loff_t *ppos, int more); static ssize_t sock_splice_read(struct file *file, loff_t *ppos, struct pipe_inode_info *pipe, size_t len, unsigned int flags); @@ -162,8 +161,7 @@ static const struct file_operations socket_file_ops = { .mmap = sock_mmap, .release = sock_close, .fasync = sock_fasync, - .sendpage = sock_sendpage, - .splice_write = generic_splice_sendpage, + .splice_write = splice_to_socket, .splice_read = sock_splice_read, .show_fdinfo = sock_show_fdinfo, }; @@ -1062,26 +1060,6 @@ int kernel_recvmsg(struct socket *sock, struct msghdr *msg, } EXPORT_SYMBOL(kernel_recvmsg); -static ssize_t sock_sendpage(struct file *file, struct page *page, - int offset, size_t size, loff_t *ppos, int more) -{ - struct socket *sock; - int flags; - int ret; - - sock = file->private_data; - - flags = (file->f_flags & O_NONBLOCK) ? MSG_DONTWAIT : 0; - /* more is a combination of MSG_MORE and MSG_SENDPAGE_NOTLAST */ - flags |= more; - - ret = kernel_sendpage(sock, page, offset, size, flags); - - if (trace_sock_send_length_enabled()) - call_trace_sock_send_length(sock->sk, ret, 0); - return ret; -} - static ssize_t sock_splice_read(struct file *file, loff_t *ppos, struct pipe_inode_info *pipe, size_t len, unsigned int flags) From patchwork Thu Mar 16 15:26:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13177816 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BDE1C6FD1F for ; Thu, 16 Mar 2023 15:27:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 088C5900008; Thu, 16 Mar 2023 11:27:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 01557900007; Thu, 16 Mar 2023 11:27:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D3564900007; Thu, 16 Mar 2023 11:27:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id C470A900003 for ; Thu, 16 Mar 2023 11:27:15 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 88887C0A51 for ; Thu, 16 Mar 2023 15:27:15 +0000 (UTC) X-FDA: 80575140030.17.7A6C0C7 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf14.hostedemail.com (Postfix) with ESMTP id 0140E100012 for ; Thu, 16 Mar 2023 15:27:11 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OebEA5mr; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf14.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678980433; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BwLwJU8AwjD+1uYy2mnm7V08kEOLBE2Z8d7v3yKcDiA=; b=dgRAyfZs+LppNaA+BUGGaX5arzSeVC93Gesd2r2IrYfbW4SjPibZng5GUj45bTUSrhm8sR RQ7ZH455rTxIZWsCGheKsuwvQ0gIqrFe4Jih4z5VKAdrnum1NZBKXiPCW+jiyl4FxlzuXZ XkgeomseUXNSdCNE1tKZVaZNeCYktM4= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OebEA5mr; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf14.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678980433; a=rsa-sha256; cv=none; b=tS/5UQrFMFef7ua3GvPNCJgUHWTHTW4bB42TEJinBMB+gfmZmmtpLjHuC1zFmb+VKPvrZ8 /HmPBGI99or1IAxl4iqVrQ0A1XieGrhSFyTOq/cay+Mrcqek/cGJ/9cwJI6ekogT7nj+Je MTpasDZiAqh6fXwItzhFYZ289E/Juyo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678980431; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BwLwJU8AwjD+1uYy2mnm7V08kEOLBE2Z8d7v3yKcDiA=; b=OebEA5mr7UE+rAclMBk+x/YTRN8d9f4zlUintOG3/yQOW9MmkayMqwC0/mzMs+pgTmB8UC kmzoLXTw6ZYDMikWQoYygovps+ixJCa49MXfjzzQaPenC4tDB5S9M1rTC6hZrw8oV4bmhS 4572gBgYWJliJEKT1MKQteATyHO8/qs= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-16-J4DCKPwsOV6EioNFlaizgw-1; Thu, 16 Mar 2023 11:27:08 -0400 X-MC-Unique: J4DCKPwsOV6EioNFlaizgw-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6A67038149BC; Thu, 16 Mar 2023 15:27:07 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id A0039492B00; Thu, 16 Mar 2023 15:27:05 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH 17/28] Remove file->f_op->sendpage Date: Thu, 16 Mar 2023 15:26:07 +0000 Message-Id: <20230316152618.711970-18-dhowells@redhat.com> In-Reply-To: <20230316152618.711970-1-dhowells@redhat.com> References: <20230316152618.711970-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Rspamd-Queue-Id: 0140E100012 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: yziuote1adppiwnjxqbm61fm7yoyh4gs X-HE-Tag: 1678980431-576715 X-HE-Meta: U2FsdGVkX1+yV28QfxifQi4oUJU8e/KQsN/rwuA4c2R2uLUk+weSnbtJCsqvAB1DFIBZcoB5O54eLgxOlZtu+ifBc2/CYgKz3GMdqV1ISAxcrclz8LMMVgUx3N0owXTUB60/PXT6sH3riv2U4qm4As7FOTsL9Hmi1td0FSYP3qKTA5dySZ5d3Xw7RMn68Rtrz1VQmrOwMHR3m19VtzFp2M9w0jC9ZEm+DmhOuAV6j10hfQ0h6ZUGHg26ipEOQDBeUJhmz2Z48mIKe+xkd50LkLSm+LYDrGOKJBj7edglnj0efv2UfBE/3yKPsMaiih9ekxt70GTfmpGdP9egNZcmywmuL5FWNppDo6mCtKIQBbVN6h3mPLIC+xCNpMWMy8kl5PyvTphQDQDkV3K3AMIGBlSpCj/n5YBbKyHR7vbRdHckrMXrYQiwWETaL8ln8BmDd/3qFsuMXAeVx+vIhdNVomr5ls6B3Y7WIYK8zFMmIz5fUL9WCenqUIfsJDl5Gd7VCLZ++LntsFX3LxFIYfdTktFh5N+IRZl9Z3SbrVCyen6KuTbBeHP1SRUVrdf5Nz5vwL28E1aQAKS8tdR0R/ViONJ+0Cn2cRbsYc6OecfNX37KDjAPAvcfOoan3LV8kiwE9hNFqZ2ZP5sf9kdGubYPvgiStRgSeAG+jrnw/Lc9Go2OaFk/wq7UhHr5V6IB5fMXOhD+8hfl8fmI8zjxwDLX5l/8Kt9laBW5NiiHlZopel3fFoCPJwn+EtTuLHTVJMou34bva7hQQJQy6BOaHv0ZoN1DlttOMwiQekni07SkIq4/aJxmggenok+jxcc4sOrnP90KlHZQ7Awyo8KjBfzOCPRFoqh7xuwqJE+FLG9xIsx8cXBuWUP0X5sfN/V7e6bUuLzxAtikETwh0VZ4B67icJJsQ7BZVg07jrXPWWD6+EzxIo+IM9Jml683fmZXnQvyMncxn/StqQyOVlHsITq XXj/npQD mFDJsY2y6IYcFHT0gIsyv6/JPhSZe/ckkui+v6q1ImVGptXNUZhVxY7BuD9Sk10cfb7sobdOTS0NAXpNi+V+7qK+hTElWCs/DvMB94vEar5zS/JW9j2LCq/Kv2gHuQ/Qk4Q42ToZ/dIgb+SspFe9CtG36LvohBVhxh1hHV8+4I1L9bcoIIs+yXNcpIhnI0Z4DZ+AQ3tces9S0u83QuOH8ECZzV9bHLB3/Awa1K0XiyYA7RqtxDPNYKvTzchSkT7pG4BZjH2hCkWIDrjEH8ePU+o2zQibPH8GI8G9Aoy/n1oSQCSS7+KbMFpt8SIJsL5CyiJE6Q35ajICljs9D/NIDb7oaHtwPTfePeslZm//wU0KIQWe+inu2rxXTXykVwRWc5ZXAfyqpd+lAO4muR95rahZX/eB4LbcrJcgMsKlnZuV0sQ62ZQ+LJ/MThTzlIraskWWc X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Remove file->f_op->sendpage as splicing to a socket now calls sendmsg rather than sendpage. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/linux/fs.h | 1 - 1 file changed, 1 deletion(-) diff --git a/include/linux/fs.h b/include/linux/fs.h index f3ccc243851e..a9f1b2543d2c 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -1773,7 +1773,6 @@ struct file_operations { int (*fsync) (struct file *, loff_t, loff_t, int datasync); int (*fasync) (int, struct file *, int); int (*lock) (struct file *, int, struct file_lock *); - ssize_t (*sendpage) (struct file *, struct page *, int, size_t, loff_t *, int); unsigned long (*get_unmapped_area)(struct file *, unsigned long, unsigned long, unsigned long, unsigned long); int (*check_flags)(int); int (*flock) (struct file *, int, struct file_lock *); From patchwork Thu Mar 16 15:26:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13177818 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51455C7618E for ; Thu, 16 Mar 2023 15:27:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DD3F6900009; Thu, 16 Mar 2023 11:27:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D8491900007; Thu, 16 Mar 2023 11:27:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BD5A8900009; Thu, 16 Mar 2023 11:27:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id A6677900007 for ; Thu, 16 Mar 2023 11:27:19 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 7DB3C1C6CE5 for ; Thu, 16 Mar 2023 15:27:19 +0000 (UTC) X-FDA: 80575140198.29.E916E70 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf06.hostedemail.com (Postfix) with ESMTP id 89C89180017 for ; Thu, 16 Mar 2023 15:27:17 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=d7ygn5xO; spf=pass (imf06.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678980437; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3nPvkTYaIi99z8jZpnwnKSywFWTrpbYCfoV76x5ycHw=; b=2/1XkgR8HTLYy52tFMnJxcPv2gQf40oJvK5CHwmJ0/CEHLX8YiGZvZ538OY0N59Rn2uJ9l z9MCBURjSQwLIADyQjL/H7PykKBMiqA2r8Fh6pDpz7OQJEc9XwYUp5KEeetTL3MbPghKqD Pdcy4y9Tb3mSEb5esMr2DPpHP99lsV0= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=d7ygn5xO; spf=pass (imf06.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678980437; a=rsa-sha256; cv=none; b=yhvMUVut9E8LIZ3vDL7t367EhSD6R7sJN4kFsU2wVh5E1I6Ir7yVP39PZPPDYO1OIMcsQp rwKjX4SwLqatcy0tqmSCHWEJuxMfzMnBHRDJ2LBlh97WUWXg4t+iZBlgbML1w9rc4zH2ZH 3XZ5sPvvpeUCqkoL8QoAZ4UmWumC/8I= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678980436; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3nPvkTYaIi99z8jZpnwnKSywFWTrpbYCfoV76x5ycHw=; b=d7ygn5xOjFU/vguCqr4AeKU6NzylUjmjpaH9nDCfpdtuS9sg3ZB4J7bHaEDCC7PqNxXkHC KDuNPvqOrpZyl+mKIzQOv7SWcz/O5pQC39ZXz+dZOvJukDMg1lqRfchqCFifgxQdYbfrv6 3gyRIxWuNfT62n3dbfss5t3Nl7QEEYw= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-77-Nfq8OMIzNR-misbqjnyc9g-1; Thu, 16 Mar 2023 11:27:11 -0400 X-MC-Unique: Nfq8OMIzNR-misbqjnyc9g-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3A73085A5A3; Thu, 16 Mar 2023 15:27:10 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 24F3A40C6E68; Thu, 16 Mar 2023 15:27:08 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Bernard Metzler , Tom Talpey , linux-rdma@vger.kernel.org Subject: [RFC PATCH 18/28] siw: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage to transmit Date: Thu, 16 Mar 2023 15:26:08 +0000 Message-Id: <20230316152618.711970-19-dhowells@redhat.com> In-Reply-To: <20230316152618.711970-1-dhowells@redhat.com> References: <20230316152618.711970-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspamd-Queue-Id: 89C89180017 X-Stat-Signature: rhzee9m181cjuo6mn7obn5393j1ecp67 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1678980437-172411 X-HE-Meta: U2FsdGVkX1+q/aVz5p2/a/VKPDUakjQeWyOXvSvXWWwzAEr1OUSRvv2P2altmfabrXpXQPaVVwZjdMW3EQoHqnuZuEkZ8432dGZz3n0Nyn1xNUv2DHoa+Nhka7Z/c+MYgN0s8TA2TIiYsrBHY60RfAnej8WK+mpS8/UI2f7QA7dDtQ2YI1oK+2W3t7/d0KQXw+i/7eh+pRFbem/ZKhu2rv/ShJe+QUrxsTuqpbzzqNsE+Mt9waYDTTUnBO9H0P+fp0t7/SVZ/Qpfof8bDNIss1m78F0A1p6FmPgvxIt5ZqLqHi4kys4irnL8WldIiA/PlPhyywD3KfUWZ9Umu+xnDukDu9sJNUV9XEHt8m4Fsd7GS2V27wWu/P7CPAFy1uyGUz0iyTpkTT2iFt+BXQgdQeGCbIdPqtHax0DcG3rE2VL+mK7OPY9fm9CvISMrnvYpvqBwN13SfIdNNQyfyHAXKn5D++gXM4g7WK1sGc6tW5j/6JuVXE+mWeGM/1uVFRzw8dd1vhMtdIzOWQTz2WPDoi8oQZ7kEZfyaWeK1+JdYtlWTf5Ypa61D0OQUPbFlAkv6gNmgMaMX5jWeeYuJwl2a/3jtsm5WT9514ArU9/GXBcsUrCATdL294Zqy636DzGqQHgy/90JeaoA5DZq9z4KetuEbOURWZacR3hhMu+wxHsabdnnFcbQIZiFYZjUYC4Iz6OcWTl4C9S89ejr8YeUyG4Ym9uDMMFi839HV05org5wfD/kRl+QD685zd/yjirECUA03frLmlYf8pv56iYQkFJvIsg5sgOQjIaxPQkeDcHiCSKs6CIidRKFLkATHMoSWOnVhMNdurzCnNfJvEWyV193TCtPTYJmoY2+JTMbCZwVKeb5dbb787nDCT+V2CjLHeHnyq25UHJXORspLT3Z5gJ5mYKX5EfOcEowJWjVfy2LX4K6Nc3xudcou8i/KVaNxIJkoOKK3GSyuxlgAdS Q0Vtg4pS v5exs+OyfpdceqOsIEvvqvWGwCOmeCDcGO2YErZ7E7HF85TBXvR+Q8j1tUouYnaKrNFZYlTvGcEyv9S+Vyz14JOsgsjSrPqsGUNMeDpvaL3tZ0jvU9WsL0z/kS2+r4aH5DxwgdyUSA+lGr3MpsbL5Qi45sapOrvZmOrn39r/eV3udjM7FL6g83itAi3tBNLenz4ORvkl8XI5kmW14e8Nv5bzId+Cw4rMC6uNlXrdfKYIkQ8jam7BxjRJN62bQGgb3MXNthwk0Sw5vnrxlt7pTBNXZOP7jG9/0SQ5fvOOAF5S5CgSn//QVYPX+BZULm76ajjxxLffHoMRgTCfAHj8quLUh/W6qUEbagcZKGQKNZ9hKwX5LVgh4y5FM9gM06GWDJTjg7PnDIF3jsBzHimdzZxnmh6ga4T54SYDIJmATVcvrm2kG4sp0wIbRXH0TtzMmAbjF1+uApU017nu+dX8p2/d5dQuQMZVciRqmzBbLRP5qWJcs8PHJrGY+RUVm81HBkT8pDidWDFzdTqwaS9N6a9TYTQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When transmitting data, call down into TCP using a single sendmsg with MSG_SPLICE_PAGES to indicate that content should be spliced rather than performing several sendmsg and sendpage calls to transmit header, data pages and trailer. To make this work, the data is assembled in a bio_vec array and attached to a BVEC-type iterator. The header and trailer (if present) are copied into memory acquired from zcopy_alloc() which just breaks a page up into small pieces that can be freed with put_page(). Signed-off-by: David Howells cc: Bernard Metzler cc: Tom Talpey cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-rdma@vger.kernel.org cc: netdev@vger.kernel.org --- drivers/infiniband/sw/siw/siw_qp_tx.c | 231 +++++--------------------- 1 file changed, 46 insertions(+), 185 deletions(-) diff --git a/drivers/infiniband/sw/siw/siw_qp_tx.c b/drivers/infiniband/sw/siw/siw_qp_tx.c index 8fc179321e2b..ec4f0ac324ce 100644 --- a/drivers/infiniband/sw/siw/siw_qp_tx.c +++ b/drivers/infiniband/sw/siw/siw_qp_tx.c @@ -8,6 +8,7 @@ #include #include #include +#include #include #include @@ -312,114 +313,8 @@ static int siw_tx_ctrl(struct siw_iwarp_tx *c_tx, struct socket *s, return rv; } -/* - * 0copy TCP transmit interface: Use MSG_SPLICE_PAGES. - * - * Using sendpage to push page by page appears to be less efficient - * than using sendmsg, even if data are copied. - * - * A general performance limitation might be the extra four bytes - * trailer checksum segment to be pushed after user data. - */ -static int siw_tcp_sendpages(struct socket *s, struct page **page, int offset, - size_t size) -{ - struct bio_vec bvec; - struct msghdr msg = { - .msg_flags = (MSG_SPLICE_PAGES | MSG_MORE | MSG_DONTWAIT | - MSG_SENDPAGE_NOTLAST), - }; - struct sock *sk = s->sk; - int i = 0, rv = 0, sent = 0; - - while (size) { - size_t bytes = min_t(size_t, PAGE_SIZE - offset, size); - - if (size + offset <= PAGE_SIZE) - msg.msg_flags = MSG_SPLICE_PAGES | MSG_MORE | MSG_DONTWAIT; - - tcp_rate_check_app_limited(sk); - bvec_set_page(&bvec, page[i], bytes, offset); - iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); - -try_page_again: - lock_sock(sk); - rv = tcp_sendmsg_locked(sk, &msg, size); - release_sock(sk); - - if (rv > 0) { - size -= rv; - sent += rv; - if (rv != bytes) { - offset += rv; - bytes -= rv; - goto try_page_again; - } - offset = 0; - } else { - if (rv == -EAGAIN || rv == 0) - break; - return rv; - } - i++; - } - return sent; -} - -/* - * siw_0copy_tx() - * - * Pushes list of pages to TCP socket. If pages from multiple - * SGE's, all referenced pages of each SGE are pushed in one - * shot. - */ -static int siw_0copy_tx(struct socket *s, struct page **page, - struct siw_sge *sge, unsigned int offset, - unsigned int size) -{ - int i = 0, sent = 0, rv; - int sge_bytes = min(sge->length - offset, size); - - offset = (sge->laddr + offset) & ~PAGE_MASK; - - while (sent != size) { - rv = siw_tcp_sendpages(s, &page[i], offset, sge_bytes); - if (rv >= 0) { - sent += rv; - if (size == sent || sge_bytes > rv) - break; - - i += PAGE_ALIGN(sge_bytes + offset) >> PAGE_SHIFT; - sge++; - sge_bytes = min(sge->length, size - sent); - offset = sge->laddr & ~PAGE_MASK; - } else { - sent = rv; - break; - } - } - return sent; -} - #define MAX_TRAILER (MPA_CRC_SIZE + 4) -static void siw_unmap_pages(struct kvec *iov, unsigned long kmap_mask, int len) -{ - int i; - - /* - * Work backwards through the array to honor the kmap_local_page() - * ordering requirements. - */ - for (i = (len-1); i >= 0; i--) { - if (kmap_mask & BIT(i)) { - unsigned long addr = (unsigned long)iov[i].iov_base; - - kunmap_local((void *)(addr & PAGE_MASK)); - } - } -} - /* * siw_tx_hdt() tries to push a complete packet to TCP where all * packet fragments are referenced by the elements of one iovec. @@ -439,15 +334,13 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) { struct siw_wqe *wqe = &c_tx->wqe_active; struct siw_sge *sge = &wqe->sqe.sge[c_tx->sge_idx]; - struct kvec iov[MAX_ARRAY]; - struct page *page_array[MAX_ARRAY]; + struct bio_vec bvec[MAX_ARRAY]; struct msghdr msg = { .msg_flags = MSG_DONTWAIT | MSG_EOR }; int seg = 0, do_crc = c_tx->do_crc, is_kva = 0, rv; unsigned int data_len = c_tx->bytes_unsent, hdr_len = 0, trl_len = 0, sge_off = c_tx->sge_off, sge_idx = c_tx->sge_idx, pbl_idx = c_tx->pbl_idx; - unsigned long kmap_mask = 0L; if (c_tx->state == SIW_SEND_HDR) { if (c_tx->use_sendpage) { @@ -457,10 +350,12 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) c_tx->state = SIW_SEND_DATA; } else { - iov[0].iov_base = - (char *)&c_tx->pkt.ctrl + c_tx->ctrl_sent; - iov[0].iov_len = hdr_len = - c_tx->ctrl_len - c_tx->ctrl_sent; + const void *hdr = &c_tx->pkt.ctrl + c_tx->ctrl_sent; + + hdr_len = c_tx->ctrl_len - c_tx->ctrl_sent; + rv = zcopy_memdup(hdr_len, hdr, &bvec[0], GFP_NOFS); + if (rv < 0) + goto done; seg = 1; } } @@ -478,28 +373,9 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) } else { is_kva = 1; } - if (is_kva && !c_tx->use_sendpage) { - /* - * tx from kernel virtual address: either inline data - * or memory region with assigned kernel buffer - */ - iov[seg].iov_base = - (void *)(uintptr_t)(sge->laddr + sge_off); - iov[seg].iov_len = sge_len; - - if (do_crc) - crypto_shash_update(c_tx->mpa_crc_hd, - iov[seg].iov_base, - sge_len); - sge_off += sge_len; - data_len -= sge_len; - seg++; - goto sge_done; - } while (sge_len) { size_t plen = min((int)PAGE_SIZE - fp_off, sge_len); - void *kaddr; if (!is_kva) { struct page *p; @@ -512,33 +388,12 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) p = siw_get_upage(mem->umem, sge->laddr + sge_off); if (unlikely(!p)) { - siw_unmap_pages(iov, kmap_mask, seg); wqe->processed -= c_tx->bytes_unsent; rv = -EFAULT; goto done_crc; } - page_array[seg] = p; - - if (!c_tx->use_sendpage) { - void *kaddr = kmap_local_page(p); - - /* Remember for later kunmap() */ - kmap_mask |= BIT(seg); - iov[seg].iov_base = kaddr + fp_off; - iov[seg].iov_len = plen; - - if (do_crc) - crypto_shash_update( - c_tx->mpa_crc_hd, - iov[seg].iov_base, - plen); - } else if (do_crc) { - kaddr = kmap_local_page(p); - crypto_shash_update(c_tx->mpa_crc_hd, - kaddr + fp_off, - plen); - kunmap_local(kaddr); - } + + bvec_set_page(&bvec[seg], p, plen, fp_off); } else { /* * Cast to an uintptr_t to preserve all 64 bits @@ -552,12 +407,15 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) * bits on a 64 bit platform and 32 bits on a * 32 bit platform. */ - page_array[seg] = virt_to_page((void *)(va & PAGE_MASK)); - if (do_crc) - crypto_shash_update( - c_tx->mpa_crc_hd, - (void *)va, - plen); + bvec_set_virt(&bvec[seg], (void *)va, plen); + } + + if (do_crc) { + void *kaddr = kmap_local_page(bvec[seg].bv_page); + crypto_shash_update(c_tx->mpa_crc_hd, + kaddr + bvec[seg].bv_offset, + bvec[seg].bv_len); + kunmap_local(kaddr); } sge_len -= plen; @@ -567,13 +425,12 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) if (++seg > (int)MAX_ARRAY) { siw_dbg_qp(tx_qp(c_tx), "to many fragments\n"); - siw_unmap_pages(iov, kmap_mask, seg-1); wqe->processed -= c_tx->bytes_unsent; rv = -EMSGSIZE; goto done_crc; } } -sge_done: + /* Update SGE variables at end of SGE */ if (sge_off == sge->length && (data_len != 0 || wqe->processed < wqe->bytes)) { @@ -582,15 +439,8 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) sge_off = 0; } } - /* trailer */ - if (likely(c_tx->state != SIW_SEND_TRAILER)) { - iov[seg].iov_base = &c_tx->trailer.pad[4 - c_tx->pad]; - iov[seg].iov_len = trl_len = MAX_TRAILER - (4 - c_tx->pad); - } else { - iov[seg].iov_base = &c_tx->trailer.pad[c_tx->ctrl_sent]; - iov[seg].iov_len = trl_len = MAX_TRAILER - c_tx->ctrl_sent; - } + /* Set the CRC in the trailer */ if (c_tx->pad) { *(u32 *)c_tx->trailer.pad = 0; if (do_crc) @@ -603,23 +453,31 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) else if (do_crc) crypto_shash_final(c_tx->mpa_crc_hd, (u8 *)&c_tx->trailer.crc); - data_len = c_tx->bytes_unsent; + /* Copy the trailer and add it to the output list */ + if (likely(c_tx->state != SIW_SEND_TRAILER)) { + void *trl = &c_tx->trailer.pad[4 - c_tx->pad]; - if (c_tx->use_sendpage) { - rv = siw_0copy_tx(s, page_array, &wqe->sqe.sge[c_tx->sge_idx], - c_tx->sge_off, data_len); - if (rv == data_len) { - rv = kernel_sendmsg(s, &msg, &iov[seg], 1, trl_len); - if (rv > 0) - rv += data_len; - else - rv = data_len; - } + trl_len = MAX_TRAILER - (4 - c_tx->pad); + rv = zcopy_memdup(trl_len, trl, &bvec[seg], GFP_NOFS); + if (rv < 0) + goto done_crc; } else { - rv = kernel_sendmsg(s, &msg, iov, seg + 1, - hdr_len + data_len + trl_len); - siw_unmap_pages(iov, kmap_mask, seg); + void *trl = &c_tx->trailer.pad[c_tx->ctrl_sent]; + + trl_len = MAX_TRAILER - c_tx->ctrl_sent; + rv = zcopy_memdup(trl_len, trl, &bvec[seg], GFP_NOFS); + if (rv < 0) + goto done_crc; } + + data_len = c_tx->bytes_unsent; + + if (c_tx->use_sendpage) + msg.msg_flags |= MSG_SPLICE_PAGES; + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, bvec, seg + 1, + hdr_len + data_len + trl_len); + rv = sock_sendmsg(s, &msg); + if (rv < (int)hdr_len) { /* Not even complete hdr pushed or negative rv */ wqe->processed -= data_len; @@ -680,6 +538,9 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s) } done_crc: c_tx->do_crc = 0; + if (c_tx->state == SIW_SEND_HDR) + folio_put(page_folio(bvec[0].bv_page)); + folio_put(page_folio(bvec[seg].bv_page)); done: return rv; } From patchwork Thu Mar 16 15:26:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13177820 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 158B1C6FD19 for ; Thu, 16 Mar 2023 15:27:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A059D900005; Thu, 16 Mar 2023 11:27:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9B5FE900002; Thu, 16 Mar 2023 11:27:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 804F5900005; Thu, 16 Mar 2023 11:27:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 6FEAD900002 for ; Thu, 16 Mar 2023 11:27:27 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 344C7140991 for ; Thu, 16 Mar 2023 15:27:27 +0000 (UTC) X-FDA: 80575140534.14.3CCDCA2 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf21.hostedemail.com (Postfix) with ESMTP id 4D8B31C0014 for ; Thu, 16 Mar 2023 15:27:25 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=eRGlOvPD; spf=pass (imf21.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678980445; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mcxOsbWhFQyukCMWrPOKlIpXJGoD9GGTGJfL3NnBqs4=; b=IPkwfDveh8xpjJ1fkqupxYo6Jnnm8bBIIapepovxo4deZsQAtIBzs0CL9Pf2NQsK3d/4Bi Pa4x2EkmQ/DoE9hUqh5JO+SFEcJ3Z3H/8JZAhLlZQShLBEGhQGW3Zhs4AYJhePVjOmkVl3 niNFYyjg30/tEYtNak6e/JvFwx4l9Dk= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=eRGlOvPD; spf=pass (imf21.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678980445; a=rsa-sha256; cv=none; b=kK2OT5Y49kb2hZwz5ZNJXpnSvOR2flku2cAQ1ir8/bAFzqxOzNVRf+nB6fyOZCYRfk/XqW UketU9ggvwj9xVQ8POa0SbnIIXJf2oHbaiIcPYiruTI5ku6LGVZSfXPVzLOiDtRFjfuI3Z rierKme9Ys2m9wyI9z/cbqMuMohTtoQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678980444; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mcxOsbWhFQyukCMWrPOKlIpXJGoD9GGTGJfL3NnBqs4=; b=eRGlOvPDm/+4ju+dd/VcUjcJspRw7+NuF1FdDNaKy8PTijhnDlZJEHNttHQGf1LNbB3VFo uQ1FKl46BAmwqvkosI1aS7dYIYJGzrwiSNxI1Hk7ZZTborz7PA6SHZkH4bDSevD7yNXyh0 gfZ7Jn1uCaXu/MnVpGH/fIUnLjqUfUg= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-558-q3lDEYBpM3ippJJ8OA4ctw-1; Thu, 16 Mar 2023 11:27:13 -0400 X-MC-Unique: q3lDEYBpM3ippJJ8OA4ctw-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E04E7185A792; Thu, 16 Mar 2023 15:27:12 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id CADBAC15BA0; Thu, 16 Mar 2023 15:27:10 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Ilya Dryomov , Xiubo Li , ceph-devel@vger.kernel.org Subject: [RFC PATCH 19/28] ceph: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage Date: Thu, 16 Mar 2023 15:26:09 +0000 Message-Id: <20230316152618.711970-20-dhowells@redhat.com> In-Reply-To: <20230316152618.711970-1-dhowells@redhat.com> References: <20230316152618.711970-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 4D8B31C0014 X-Stat-Signature: 161fsoanpopmnf3ypaxpr6qf7gd8g3sh X-HE-Tag: 1678980445-750529 X-HE-Meta: U2FsdGVkX1+lyN1+sC6NbTAGyM57tDdRHGIu9kPFnqJGfEN+qh+6n20nJQ6njsFWPZyC/HjHlTL/LL/4J8FgyUGXE7giSuuZuZjmLwvZ5c49R3nCIWcUk+2/9x7IhKVYCUeha2XDBNo5WITkCM9ohiR4qMEqO2WRTLT9kl+vdssHvTmXeBgUcwCH/4c5rOz0FGhnO13W+qNh7wJzI3bCn+aONebm85fZEto+LXeqV33BQftrgHj1hUdFAfEN7NXv96PDBf0J2Q4l3JXEz9hPxOqMbl7eNVAxSt4i5cL2sGyoJY25QDG8Gq44L6KYbsSgmQ/NPFmX8YzFgizViLpWWNPCzJetbGpZDEZCJsfBQ6VTuduxZO1r2B/LKvyQP6/phuuocnliBCe8kAKPqt90UoG3m6+Vv2jGIhIGMKyETVe0Sb/8w0g5FF8hqWxMSM2AZi/jfd3G5tX0nTXp7VA0phXstFYUo3JxLIul9Ph89h+CZoyulH3EsWM/hebhWn/hbmaDQJxBJRJT6FBDWE7Pe86G5Kj9yQV90MIpOYAf5e403Z0jGBB5VIEei832sn2NT8nXQ7elYcD9+Olvfcb+pQzatlPWPi2KkhcrTGwIKMP1ol78JWR9N3chFBwmxCDT7XCEvZrOpt+SdKlnzL7ZBEjr3bUCj1o/H4q2PCgmoag5UCMQqcu8rTlMTpPSfbK1+30xqE/CfLPIkFSpIieTueF/wLflLVqNeYp4mNPTiOmBxtlaPGBfw9+xJ7OHoSQATTge6q/74axlbKmznnOm6mXLkH2Sy9vrMwnOFkkF2+P66eAvxXF+H+3sbbuBI3/6SYtxTGGw/rV9fdWrWJEOtVlKvKstOtRxvjIRM0abDHhtZlY/6ZjVbQlu2UsCRVFuOLOv6ztsS4wRgsItP+der2ljWRbxfmDZHyvw0LLS0nB1CrknzhZiUHZ5s9S/GcvlwFra5bHxcs3ytAGjZMw HUgDDXD2 hqn/icC5TBpZY6XzYjLjRQnXReEa8ro/LZ9eetajcpZ6FJeICii4I1wmzG4ongiVBwjui+7FofPtw7t8VaFw7LxsV4wd6wMrcQ7udw2vxsh5AVZjy3jUyFowuGvNPt44dTmxCgm0JGigDFtWVCKVICMN5/hvr7MfbO73D5FbBhtE1ok0YKbhU9g/FylD88LwIwvkhyJz8f2NDF4TcezXMQzUWj+P+DwxsF/LFXjgtriHlrj7k48QXTpCzPogOStDvT2Qy8F/Z3bw88+xu5HGHfjFikuXXVY7UC1UaI9e8Tkl0mNqI9c8mjdezz0q0Cu1O59hgrqcINaFQQ4VfBEUqvHIORECYnE1M0LGcfMSufGE0xJwWPh9FtigLH24yP8K8WGsAvl37GMxkBa1SK/qao04mctu6rOj02JVoRNWqKpwM80hMQgA5jbpefPeRVsu342XXKgHx8xRj2qwJmznH7XKzXw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use sendmsg() and MSG_SPLICE_PAGES rather than sendpage in ceph when transmitting data. For the moment, this can only transmit one page at a time because of the architecture of net/ceph/, but if write_partial_message_data() can be given a bvec[] at a time by the iteration code, this would allow pages to be sent in a batch. Signed-off-by: David Howells cc: Ilya Dryomov cc: Xiubo Li cc: Jeff Layton cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: ceph-devel@vger.kernel.org cc: netdev@vger.kernel.org --- net/ceph/messenger_v1.c | 58 ++++++++++++++--------------------------- 1 file changed, 19 insertions(+), 39 deletions(-) diff --git a/net/ceph/messenger_v1.c b/net/ceph/messenger_v1.c index d664cb1593a7..b2d801a49122 100644 --- a/net/ceph/messenger_v1.c +++ b/net/ceph/messenger_v1.c @@ -74,37 +74,6 @@ static int ceph_tcp_sendmsg(struct socket *sock, struct kvec *iov, return r; } -/* - * @more: either or both of MSG_MORE and MSG_SENDPAGE_NOTLAST - */ -static int ceph_tcp_sendpage(struct socket *sock, struct page *page, - int offset, size_t size, int more) -{ - ssize_t (*sendpage)(struct socket *sock, struct page *page, - int offset, size_t size, int flags); - int flags = MSG_DONTWAIT | MSG_NOSIGNAL | more; - int ret; - - /* - * sendpage cannot properly handle pages with page_count == 0, - * we need to fall back to sendmsg if that's the case. - * - * Same goes for slab pages: skb_can_coalesce() allows - * coalescing neighboring slab objects into a single frag which - * triggers one of hardened usercopy checks. - */ - if (sendpage_ok(page)) - sendpage = sock->ops->sendpage; - else - sendpage = sock_no_sendpage; - - ret = sendpage(sock, page, offset, size, flags); - if (ret == -EAGAIN) - ret = 0; - - return ret; -} - static void con_out_kvec_reset(struct ceph_connection *con) { BUG_ON(con->v1.out_skip); @@ -464,7 +433,6 @@ static int write_partial_message_data(struct ceph_connection *con) struct ceph_msg *msg = con->out_msg; struct ceph_msg_data_cursor *cursor = &msg->cursor; bool do_datacrc = !ceph_test_opt(from_msgr(con->msgr), NOCRC); - int more = MSG_MORE | MSG_SENDPAGE_NOTLAST; u32 crc; dout("%s %p msg %p\n", __func__, con, msg); @@ -482,6 +450,10 @@ static int write_partial_message_data(struct ceph_connection *con) */ crc = do_datacrc ? le32_to_cpu(msg->footer.data_crc) : 0; while (cursor->total_resid) { + struct bio_vec bvec; + struct msghdr msghdr = { + .msg_flags = MSG_SPLICE_PAGES | MSG_SENDPAGE_NOTLAST, + }; struct page *page; size_t page_offset; size_t length; @@ -494,9 +466,12 @@ static int write_partial_message_data(struct ceph_connection *con) page = ceph_msg_data_next(cursor, &page_offset, &length); if (length == cursor->total_resid) - more = MSG_MORE; - ret = ceph_tcp_sendpage(con->sock, page, page_offset, length, - more); + msghdr.msg_flags |= MSG_MORE; + + bvec_set_page(&bvec, page, length, page_offset); + iov_iter_bvec(&msghdr.msg_iter, ITER_SOURCE, &bvec, 1, length); + + ret = sock_sendmsg(con->sock, &msghdr); if (ret <= 0) { if (do_datacrc) msg->footer.data_crc = cpu_to_le32(crc); @@ -526,7 +501,10 @@ static int write_partial_message_data(struct ceph_connection *con) */ static int write_partial_skip(struct ceph_connection *con) { - int more = MSG_MORE | MSG_SENDPAGE_NOTLAST; + struct bio_vec bvec; + struct msghdr msghdr = { + .msg_flags = MSG_SPLICE_PAGES | MSG_SENDPAGE_NOTLAST | MSG_MORE, + }; int ret; dout("%s %p %d left\n", __func__, con, con->v1.out_skip); @@ -534,9 +512,11 @@ static int write_partial_skip(struct ceph_connection *con) size_t size = min(con->v1.out_skip, (int)PAGE_SIZE); if (size == con->v1.out_skip) - more = MSG_MORE; - ret = ceph_tcp_sendpage(con->sock, ceph_zero_page, 0, size, - more); + msghdr.msg_flags &= ~MSG_SENDPAGE_NOTLAST; + bvec_set_page(&bvec, ZERO_PAGE(0), size, 0); + iov_iter_bvec(&msghdr.msg_iter, ITER_SOURCE, &bvec, 1, size); + + ret = sock_sendmsg(con->sock, &msghdr); if (ret <= 0) goto out; con->v1.out_skip -= ret; From patchwork Thu Mar 16 15:26:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13177823 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0062C7618D for ; Thu, 16 Mar 2023 15:27:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 67FF0940012; Thu, 16 Mar 2023 11:27:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6086894000A; Thu, 16 Mar 2023 11:27:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4D077940012; Thu, 16 Mar 2023 11:27:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 3DFA294000A for ; Thu, 16 Mar 2023 11:27:35 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 0637A81442 for ; Thu, 16 Mar 2023 15:27:35 +0000 (UTC) X-FDA: 80575140870.27.15DAB1C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf17.hostedemail.com (Postfix) with ESMTP id 4FC2B4001B for ; Thu, 16 Mar 2023 15:27:33 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UKS4AMgT; spf=pass (imf17.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678980453; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cqdl8fmtEvVwpgPJdtUvP824my58WFWl8Wp/Vt/z1XI=; b=aK+RzZmWt9PnltsivbteD8FdohleBno1QVqOHfwe+puJVCeUhEcIRLh65rHeJZyZplVbQG aV8erqVWqj7qRVHWyit/l5nB0PxL/Tz/w4sEGOloXW2wuYBvu6LV4G7XCN7Iyc+XzEAbUF qHJL+S7ay80gz30LZpGArpDn+ZwoGNY= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UKS4AMgT; spf=pass (imf17.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678980453; a=rsa-sha256; cv=none; b=KKkA44EbUqo85QgrBTpV49r2ZB6CNqkz3RmCXvwxO2ROkLhTPDvaj2mZm4v5eEpWXptr4Z 2vS9h+9vojnq2TlriDvv/k+8KXXuCynKRgo5i7oQefkyGd7TKXAVXfZ3q7TW372bJh+dWh b/bAQoNmRoXl8rkel3X+6ri7VXS1XkE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678980452; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cqdl8fmtEvVwpgPJdtUvP824my58WFWl8Wp/Vt/z1XI=; b=UKS4AMgTmDX4aGS/5f48Hd0GpunzLUJayFXbjTcp4Y84/Ue+dykGoE3vzUykMjbUoFtQPp 1FQy5wRUQx/YCn5w4LPFhRy8yALte6I+FXZtBEI67o7VYSNAPqddFj3J2PpiYc90TSuGaV XMmHyMk7/j4h3UXb/eRasedTwcYyQlI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-644-1WZJi73lMMOMNIGU82y0ZA-1; Thu, 16 Mar 2023 11:27:22 -0400 X-MC-Unique: 1WZJi73lMMOMNIGU82y0ZA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B5D4E185A78F; Thu, 16 Mar 2023 15:27:15 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9C1A1140EBF4; Thu, 16 Mar 2023 15:27:13 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, "Martin K. Petersen" , linux-scsi@vger.kernel.org, target-devel@vger.kernel.org Subject: [RFC PATCH 20/28] iscsi: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage Date: Thu, 16 Mar 2023 15:26:10 +0000 Message-Id: <20230316152618.711970-21-dhowells@redhat.com> In-Reply-To: <20230316152618.711970-1-dhowells@redhat.com> References: <20230316152618.711970-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 4FC2B4001B X-Stat-Signature: dr7nsmfp786otrp61gaawfob836ksz6q X-Rspam-User: X-HE-Tag: 1678980453-64290 X-HE-Meta: U2FsdGVkX19JTLyFgA8m/RXRzpqiIbYOjI1fi07BamN38BzBx9drg+iew2S6OFO2bV2dvwWC6QsEkBIbhBgEiOi7OTTi88NOBPENH/soAvquL3/+6J9Rl/d0QoPcjo20UGZ5yFSSLwjO1eIeGWkMEFFUIE4SAOqwpSzDFBLjKVOnaohaHSrRKFydXgMXnuE0dP4mFyBa2q7oUEY2elEXspOqqFf9mPe5EA/UITJokqf/W9tOJ6LMEmSd/eIsPW/b6Gv7A34HlXmUUDRs8RA1DqA+0rJKlXYWarTfu8a2bRN3HnxIQ57ULyrMiV9YAV6ERqJ6LbCUdbS23t/egHoMv1MzXDHCZL4TH3EGuUsXfh+kPVMbkueglkHrSuIViY9TvKoSYqnbVspzRpovfUo0ocR8Q0KjgbnrpSheC/kso4OCtd6pt0FTk1tf4e0jyrQIuiy8ijXLT1aBwYdQVLFmPMBF6ns4bEHVhVtkBPo977X496xxcNTd+BOXUMPYz3Cn4qfOb2G0mkbnIBeIr2RxinPPJ5o2TsQYdATvmNPiB/sfB0OAeIRqXbVGqNuV4vGq9BssgggMOzS5ZROYG1SZg6naEhO8DnACgr5s9dRCKWdZ/8HBOVDv7vrjImINh+c68vbKE20mPKYU1xFWChjnesSbbyRFGLFt2yXpsIF1gtSQta+qRDayBgB9T/JgZg6o4C02xfCgIYdUgDEG1fmo5RvWMJWx65tZOTALNC33FVS3hHHSxoy8J+o3vusvGNmRE2cpRYb0t5jJHYhko49/ZnJCB9YYZpBTIacwkFNE5uxW+wzkVnugOmVqrl9zr8cTi87Ihemusyim/KBWq3MtEd2EdhTUEbrtCeSVVUThBeQfRYlWgKHkfO7KLlr1/pLyRmypeHGnu3lLYusIEnfGsa1Z6Gaid1RwZ4KoTgc9pxMiGD3i/D7LFiTmBthrIbX3Ct3OX5iQce3x5aM43nA EsGBN2rJ 2zWaLiO6TiwfuWpu7CNtUnTmJ66uS/Ao2Db/gagBAwBtHwJIf/smX4g61pedHLdut7oY9VGRUMZWOoyTE4rJwVbK23x4LsgN++UJaVgaC9+o8gSsYOeU+2N0/3dlnJA2DawqCplWSxrD2UCDCruxOznkVx/g7DpWwZJr7lwZ52mPGLbhxQluQEYYnkQUN9OuRSf8z/2YmR836ijtY/fwwMeLs5J98DbRLhuoLZgiNA8Kxzxjqoe8UY1ObIXQ0KpkmXrlu0FgdihHqwqTdkuKPgZiq6OgEP+mH125X3OKllUaP3uS8bdYCVhpwWsTqaxalsA9m6IXdegtDd83Wy4rgDGSSHEUyWoThX3Mepvyh7g4qKUL2qwofRI/YtEImJZg3uC08LINP4KEsy9Owc9WS/cJLdvaOYWb5t39EXtQk+8eyS7pIJKOzZftf4wibmajskmLIQvF04MEQWM05qPRfL1UaiA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use sendmsg() with MSG_SPLICE_PAGES rather than sendpage. This allows multiple pages and multipage folios to be passed through. TODO: iscsit_fe_sendpage_sg() should perhaps set up a bio_vec array for the entire set of pages it's going to transfer plus two for the header and trailer and use zcopy_alloc() to allocate the header and trailer - and then call sendmsg once for the entire message. Signed-off-by: David Howells cc: "Martin K. Petersen" cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-scsi@vger.kernel.org cc: target-devel@vger.kernel.org cc: netdev@vger.kernel.org --- drivers/target/iscsi/iscsi_target_util.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/drivers/target/iscsi/iscsi_target_util.c b/drivers/target/iscsi/iscsi_target_util.c index 26dc8ed3045b..c7d58e41ac3b 100644 --- a/drivers/target/iscsi/iscsi_target_util.c +++ b/drivers/target/iscsi/iscsi_target_util.c @@ -1078,6 +1078,8 @@ int iscsit_fe_sendpage_sg( struct iscsit_conn *conn) { struct scatterlist *sg = cmd->first_data_sg; + struct bio_vec bvec; + struct msghdr msghdr = { .msg_flags = MSG_SPLICE_PAGES, }; struct kvec iov; u32 tx_hdr_size, data_len; u32 offset = cmd->first_data_sg_off; @@ -1121,17 +1123,17 @@ int iscsit_fe_sendpage_sg( u32 space = (sg->length - offset); u32 sub_len = min_t(u32, data_len, space); send_pg: - tx_sent = conn->sock->ops->sendpage(conn->sock, - sg_page(sg), sg->offset + offset, sub_len, 0); + bvec_set_page(&bvec, sg_page(sg), sub_len, sg->offset + offset); + iov_iter_bvec(&msghdr.msg_iter, ITER_SOURCE, &bvec, 1, sub_len); + + tx_sent = conn->sock->ops->sendmsg(conn->sock, &msghdr, sub_len); if (tx_sent != sub_len) { if (tx_sent == -EAGAIN) { - pr_err("tcp_sendpage() returned" - " -EAGAIN\n"); + pr_err("sendmsg/splice returned -EAGAIN\n"); goto send_pg; } - pr_err("tcp_sendpage() failure: %d\n", - tx_sent); + pr_err("sendmsg/splice failure: %d\n", tx_sent); return -1; } From patchwork Thu Mar 16 15:26:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13177821 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 820CDC6FD1F for ; Thu, 16 Mar 2023 15:27:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1C75E900002; Thu, 16 Mar 2023 11:27:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1760294000A; Thu, 16 Mar 2023 11:27:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F0AE2900003; Thu, 16 Mar 2023 11:27:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id DFA2C900002 for ; Thu, 16 Mar 2023 11:27:29 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id B8CD580B15 for ; Thu, 16 Mar 2023 15:27:29 +0000 (UTC) X-FDA: 80575140618.23.AF19311 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf04.hostedemail.com (Postfix) with ESMTP id 007A94001E for ; Thu, 16 Mar 2023 15:27:27 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WF33x7wE; spf=pass (imf04.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678980448; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cLmy3QT3PxPcXOXP8Tmrcj7ottmHeL6VPbhO66kxXIU=; b=0ks1VeBM3NNAAjx3zCO6vcBR2wdU7ovGgyhNcbWLMWugalgjPT9btuVA336YJsWu1+JVcQ o+hIT1KWvoGVOXd6zGzgDwYxA9VRu9LKLb1VOhQFFXoijM1pa5b6Eu4ntX5GVDyhQiKsfM T/scFo+TGS3TCbkNJnsAt/95pfwE7b4= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WF33x7wE; spf=pass (imf04.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678980448; a=rsa-sha256; cv=none; b=NZZpVTJG0tSjJIuFvGATU8a93rk7uzIN0smhIJ98tblSmNa+tXdFc+wnwxSxcc4Y91owDG ez04TpiRaKdNDHWwBZf97hmjSRD90XQiUF0vg5cEuV+89nTrl/18ZTRmm6h4F3HOSLGO5b z1Jy2xzZjG2wfHitjK4eqBPNYXuoby0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678980447; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cLmy3QT3PxPcXOXP8Tmrcj7ottmHeL6VPbhO66kxXIU=; b=WF33x7wEaldYAMrJeyWUYUhceMAM0aarp7Fb7ZgIaebEjI+5FM/aYMj5WPnyx1SHlwYYjv hBbz3p3mzxVqif6zd0k6gQMc9/lHMB3I4GRmJnScg1kBd16Gak1v0xt9DdBInkm37zp6pa hbEjbV4xhnmdZykZvT9mFimKOHXX3UE= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-531-HxkbzJbbNIOwbn5zC8ZlYg-1; Thu, 16 Mar 2023 11:27:24 -0400 X-MC-Unique: HxkbzJbbNIOwbn5zC8ZlYg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 684F938149BF; Thu, 16 Mar 2023 15:27:18 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 521EB1121315; Thu, 16 Mar 2023 15:27:16 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, John Fastabend , Jakub Sitnicki , bpf@vger.kernel.org Subject: [RFC PATCH 21/28] tcp_bpf: Make tcp_bpf_sendpage() go through tcp_bpf_sendmsg(MSG_SPLICE_PAGES) Date: Thu, 16 Mar 2023 15:26:11 +0000 Message-Id: <20230316152618.711970-22-dhowells@redhat.com> In-Reply-To: <20230316152618.711970-1-dhowells@redhat.com> References: <20230316152618.711970-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: in5iopwds878hrdx576detogge3ifme8 X-Rspamd-Queue-Id: 007A94001E X-HE-Tag: 1678980447-868384 X-HE-Meta: U2FsdGVkX18ySqVZkjcTcuw7hsA/yv0HfB3/wpOjcBvdNma1XTlA7UQx2ClOFuhKI8qKDIUmdN3BMjMs3jzfIm8APqLbcEKB8aPdjKOAa2HfVXocdKyxNKUWRnau5vMI8Dfd9toy33BrvGS+HPZ+8PSKBiUnsgFfHtixKDPIa065Y6wzKXHg9mb+Hbxn3ev3Rm6jDtbESvCAx72RMGP5QucsiYHbVfQBH8vnw7DdcOiogrAqTSktvvWcyXhUtjfrVJEmux/B5MCkBldW0+NrsV6vS1WCASj5QOQ6O2ELdz8oIj8jjcMjvBOq5mCqEm5mcH9KsBz5qXeSTNjK8C7UUhTVMGZIRBSZ2ZdX8CyCGg+kGa5NtGWp96lv1HDdsETK6W3wLq+9V3BIwoTIRyfBANyorwmw/4hzQl4xlEymmvS0r8R9HIulpHHTcvNuTeVCFex+EVKEq2OkYQ1BrYjen3v+sqmX4TWgPwmcAtiQ8TYtfbI5Ej3hhcoQLi/l9+YdYLdeumCD1n6/VMnmsDorEkRzvjZxKsJgRX9GRerMw40Dchk6mSUIZPe9ZjsJndTncqTuvWNhaer2N5zCVgs4ladxJU61PtngzFrSmbAxQVYKQcOJFLKD045CheM0NEasBG1IejY8XjGspfAsCJ/ScyDbapy6roN4Wcdhk+p5nS7lF6GlCuJrqPcpZj8zwMqlfWZ4bVxwAO0IHs4FtgnuKLe/iXphMX8dCkNrjXPertNO9BYmAvy8Yj2kl/1TqibemI/s9kpuyfkJC5Vq67yEne0tGpV+opVMkWUp/+j0ozZgny6+rTjXYhkRUfRgEY1Kn1svDCoI8DlekdHq0LGQ3FHhkimmOyycvIcmjamixPKMR3IB8SKTXdH6n7esuG5kNPR1YvNd1onsY0H5yFrUgvfsLu+SNcwrHEi8yEt2UytNuLCrbGv/WAo5R5LPc/9W7WZ2u6xlMPco7nb5Jdl GGIyUfGF q+QoP0/b2H1q8i/GDN+IZiwZsY47Op+gU2oOz2r3VR/UrTyeSSHWhKzUhOhIWPexFBHxjeg0hXM8yj6qDwqyoUkoO1wg/hhMnqaO7YDATxIEod7WPZq5LSl4e91R5SbVcgWcW+vbS3dyU7qY/V6mPNqieeTpqi+WDh+DFC9KWchEFFHdWUvaYcy5+Bf85BpbHZUxJPf4+/p+U+tdxxtpfUEXkBCNYveJbpImY9OtxmpFdSGVH4oYIzmjOdwyCpPJ7xQD3AgBKtlW1Pm9W16NsbzLgqmlkKUmcZWZCpkDRUt4JKTR3QPof25mZ7aYdX/Vjs7MYO68s2r41UqmNHx/dQNz1JkZW48Z+jFDf8BR8wTA/YPzAKvBc800/z+tol6hDI8RpujWFY3HkFzEeb+5Li0+g+qYvQXLfQJ1fcMmbxzigGwDO70r2AAQnZ8bhzcYlPgKandl52uTA+rRiQbx3VpJCSkfh1sp38VSTxQZdDdR3w89eHlZxQdCtcPhZaal8+IHKUZz9RYJPTHIH464vKFggJCJ+jGWimxASWBGv7Qn/JRi97uwdt+/a0w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Translate tcp_bpf_sendpage() calls to tcp_bpf_sendmsg(MSG_SPLICE_PAGES). Signed-off-by: David Howells cc: John Fastabend cc: Jakub Sitnicki cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: bpf@vger.kernel.org cc: netdev@vger.kernel.org --- net/ipv4/tcp_bpf.c | 49 +++++++++------------------------------------- 1 file changed, 9 insertions(+), 40 deletions(-) diff --git a/net/ipv4/tcp_bpf.c b/net/ipv4/tcp_bpf.c index 7f17134637eb..de37a4372437 100644 --- a/net/ipv4/tcp_bpf.c +++ b/net/ipv4/tcp_bpf.c @@ -485,49 +485,18 @@ static int tcp_bpf_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) static int tcp_bpf_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct sk_msg tmp, *msg = NULL; - int err = 0, copied = 0; - struct sk_psock *psock; - bool enospc = false; - - psock = sk_psock_get(sk); - if (unlikely(!psock)) - return tcp_sendpage(sk, page, offset, size, flags); + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = flags | MSG_SPLICE_PAGES, + }; - lock_sock(sk); - if (psock->cork) { - msg = psock->cork; - } else { - msg = &tmp; - sk_msg_init(msg); - } + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); - /* Catch case where ring is full and sendpage is stalled. */ - if (unlikely(sk_msg_full(msg))) - goto out_err; - - sk_msg_page_add(msg, page, size, offset); - sk_mem_charge(sk, size); - copied = size; - if (sk_msg_full(msg)) - enospc = true; - if (psock->cork_bytes) { - if (size > psock->cork_bytes) - psock->cork_bytes = 0; - else - psock->cork_bytes -= size; - if (psock->cork_bytes && !enospc) - goto out_err; - /* All cork bytes are accounted, rerun the prog. */ - psock->eval = __SK_NONE; - psock->cork_bytes = 0; - } + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - err = tcp_bpf_send_verdict(sk, psock, msg, &copied, flags); -out_err: - release_sock(sk); - sk_psock_put(sk, psock); - return copied ? copied : err; + return tcp_bpf_sendmsg(sk, &msg, size); } enum { From patchwork Thu Mar 16 15:26:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13177819 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51042C6FD1F for ; Thu, 16 Mar 2023 15:27:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DD37B900007; Thu, 16 Mar 2023 11:27:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D8273900005; Thu, 16 Mar 2023 11:27:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C4ABF900007; Thu, 16 Mar 2023 11:27:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id B53AD900005 for ; Thu, 16 Mar 2023 11:27:25 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 8C09D1C6509 for ; Thu, 16 Mar 2023 15:27:25 +0000 (UTC) X-FDA: 80575140450.04.7E2CB1B Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf24.hostedemail.com (Postfix) with ESMTP id C7E84180007 for ; Thu, 16 Mar 2023 15:27:23 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UDhWu41e; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf24.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678980443; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+QyEBFQ78rmkbhEM2WtWHIrO32XG2a7frdMcXNjpZkk=; b=k6oiuUBrKkxtuxlB3OTgVOpHFaM1mTKi8y3eduBzeMJEET65YagjfhSTmVV7jLacYwa0sR vOHOFHr+qr77jFLlpizJVTwLAFqAerrphLZj5Ars6grICdipINiDYGEMRFHMoIuOviet7f LkHLOW0fZSd3eqmBxQ/avx0fLKWXjLQ= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UDhWu41e; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf24.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678980443; a=rsa-sha256; cv=none; b=0sdV2Ipp0XEveKXIFP36lIiqAHy+s0kSjPntIwEXmQ3FBF3IW/0zWIEVYBVzoNSpSpOygO bZsZwYmfRSNnwRZ4NfWjkJUDaMmGo/h67D0yLOOOyC9cEXjT0lQn0Nma9uRO1bAKHhY/2o KiXDeeXP6XVV291lRWhUMNZe+aeqXHE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678980443; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+QyEBFQ78rmkbhEM2WtWHIrO32XG2a7frdMcXNjpZkk=; b=UDhWu41e+HuLxv0jSMIWNleHxQ9X0kS7ElSWAN/rOjJI2RiWLp83E10VKu8ZEkkCGzcLyk ZgWt379liTEkQZCiWc44sdRUvlC2Ukr654S0yNhByIqvDf1XwHife7oczQqLj3QhBtO2FL iuxrRi+HlauQSfc94HQEkqwxwKTjDEk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-166-k012TxuLM5-J9DwPD-MUfg-1; Thu, 16 Mar 2023 11:27:21 -0400 X-MC-Unique: k012TxuLM5-J9DwPD-MUfg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id DE2EC858F0E; Thu, 16 Mar 2023 15:27:20 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1C465202701E; Thu, 16 Mar 2023 15:27:19 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH 22/28] net: Use sendmsg(MSG_SPLICE_PAGES) not sendpage in skb_send_sock() Date: Thu, 16 Mar 2023 15:26:12 +0000 Message-Id: <20230316152618.711970-23-dhowells@redhat.com> In-Reply-To: <20230316152618.711970-1-dhowells@redhat.com> References: <20230316152618.711970-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: C7E84180007 X-Stat-Signature: aod3cut5s3itjyat5zd1ri5cz4kozpyj X-HE-Tag: 1678980443-370235 X-HE-Meta: U2FsdGVkX1/koFamiMqq7FcrbeGbBufHyAsOhG04K20KGOU+ybd6FziaXRJR0Bgzp2d7o9f//jNnLGPSUkRrRYSDp53ug/3anU3O/pInt+xtQdPZ7Nmuce+KivBKUzS02Tm+1iFx+phmKJtH9rlpqLQiOR5c+ftx3jCc6u5xLW1CDnQHCxmvWN4kHy6tns7K1TgcDYnWYqE+F9DG6CHkE1Q5LVbbPbNXxElzX8dv2D0yZD6zFGXbtGIYBhr2CAK/fVQEYvaJSNNVtF9njbaNBkwD4PjTcUgZ+I2JaTC6VQqLR9k+oriDXS4j0SM+4yzvf2zRtGE2wLUMoMjtSs36hJicZxtlESuYHYwo3YXh6nO9IsVh0Z2Xm5vDOn9wi+l4mXQWOo3rAPYlXHL5t+/0NlhxMKxtYEKvZDK2oT8e7nGPGS3BMtYA3gH6yHAK+mcvn3g+awWvh9NugjV46Brfx7tWIIHMw51G4Cgj3YHIag92hS1vC+wzrFmPnmcyOh0bZ99i7zKkyAaw3J2KokxDFqMxkNuSq8t/jKm3XakFUBZV83SQ2IiHNTAvPWTW7nRJ9lbxFilWjI35QoZN8xfE5jknBzsfsmwIqm+BPHaLiBGhH+FT74XQMmEYiwmpu/ycdU+fXib+xQVkRlL4RLw/zHUZrkNTz/MnbkHC3OjIgba9S+UvHZV0kxWiBD6mLy9ERJU9Fwohigwk7CuRhRosMJPeXYAEPX0RRMLKZuOZGIuOkBQDBOFnFs1Q+bXdnustW3kZ/l+RSBjafYsi7QWUSSynEGmLFDuDC8gfflD3tN5vPAZkWIrj5IVa60f8QbEu5VmaKdC+St8r/w5+w9TneIbHaGExFfk0v/ZJEzIGq9vbqEIVp/wpiZ4noMrVLU9FIUX7lfrjg0TmhyZChPCcR3K0rROeKFZiibTuZAKUvaFyS9x7vCCE6LopljgmE0QqjN+iHfSt9F64WH1EoDY /mprfVkJ n3pgaf3vXd55gvfWWIJzCz1Vn2IQSUZkICdbAB05b30lYbQUVRfhqysDQ7kQjmZ/w2Psdjqh+BCoUbSaB4ElbnJF3MEYXbPY+6Obi1TN1q0aSatQi4S9JPn0qXoMnPhcvTSTDgK79l89wBR5L8LMkyoc/GQ5bBY0goElrfhOA35lu9bkCWPQWCR6v2oj9qDxgq2Bc2kV6p7Xsh+DzlQzG8eIW3pCgiL8I8wT8hNEdvVpR/h2VDhCe2vQg5KL6xk4Pwi3AALqJRJoBIeBeI+3wjLQkXaikoNWGV2Fu4XhUR2vwF4BuqxZArmcooNhZnE8Kx8WjMCvF2aF6uBZTASB+Ri/MbRCowr1qkKjtK8MOlaqCVQzL0Kb47j70tYBCvgt8t+gjVL4rK9w/1XIBMKVJvFOCpTpNNoAY4r7XwuAeMy7NH8ul1trHyTMWNi9boEJRyNTV X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use sendmsg() with MSG_SPLICE_PAGES rather than sendpage in skb_send_sock(). This causes pages to be spliced from the source iterator if possible (the iterator must be ITER_BVEC and the pages must be spliceable). This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Note that this could perhaps be improved to fill out a bvec array with all the frags and then make a single sendmsg call, possibly sticking the header on the front also. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/core/skbuff.c | 49 ++++++++++++++++++++++++++--------------------- 1 file changed, 27 insertions(+), 22 deletions(-) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index eb7d33b41e71..9fa333e26b7d 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -2927,32 +2927,32 @@ int skb_splice_bits(struct sk_buff *skb, struct sock *sk, unsigned int offset, } EXPORT_SYMBOL_GPL(skb_splice_bits); -static int sendmsg_unlocked(struct sock *sk, struct msghdr *msg, - struct kvec *vec, size_t num, size_t size) +static int sendmsg_locked(struct sock *sk, struct msghdr *msg) { struct socket *sock = sk->sk_socket; + size_t size = msg_data_left(msg); if (!sock) return -EINVAL; - return kernel_sendmsg(sock, msg, vec, num, size); + + if (!sock->ops->sendmsg_locked) + return sock_no_sendmsg_locked(sk, msg, size); + + return sock->ops->sendmsg_locked(sk, msg, size); } -static int sendpage_unlocked(struct sock *sk, struct page *page, int offset, - size_t size, int flags) +static int sendmsg_unlocked(struct sock *sk, struct msghdr *msg) { struct socket *sock = sk->sk_socket; if (!sock) return -EINVAL; - return kernel_sendpage(sock, page, offset, size, flags); + return sock_sendmsg(sock, msg); } -typedef int (*sendmsg_func)(struct sock *sk, struct msghdr *msg, - struct kvec *vec, size_t num, size_t size); -typedef int (*sendpage_func)(struct sock *sk, struct page *page, int offset, - size_t size, int flags); +typedef int (*sendmsg_func)(struct sock *sk, struct msghdr *msg); static int __skb_send_sock(struct sock *sk, struct sk_buff *skb, int offset, - int len, sendmsg_func sendmsg, sendpage_func sendpage) + int len, sendmsg_func sendmsg) { unsigned int orig_len = len; struct sk_buff *head = skb; @@ -2972,8 +2972,9 @@ static int __skb_send_sock(struct sock *sk, struct sk_buff *skb, int offset, memset(&msg, 0, sizeof(msg)); msg.msg_flags = MSG_DONTWAIT; - ret = INDIRECT_CALL_2(sendmsg, kernel_sendmsg_locked, - sendmsg_unlocked, sk, &msg, &kv, 1, slen); + iov_iter_kvec(&msg.msg_iter, ITER_SOURCE, &kv, 1, slen); + ret = INDIRECT_CALL_2(sendmsg, sendmsg_locked, + sendmsg_unlocked, sk, &msg); if (ret <= 0) goto error; @@ -3004,11 +3005,17 @@ static int __skb_send_sock(struct sock *sk, struct sk_buff *skb, int offset, slen = min_t(size_t, len, skb_frag_size(frag) - offset); while (slen) { - ret = INDIRECT_CALL_2(sendpage, kernel_sendpage_locked, - sendpage_unlocked, sk, - skb_frag_page(frag), - skb_frag_off(frag) + offset, - slen, MSG_DONTWAIT); + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = MSG_SPLICE_PAGES | MSG_DONTWAIT, + }; + + bvec_set_page(&bvec, skb_frag_page(frag), slen, + skb_frag_off(frag) + offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, slen); + + ret = INDIRECT_CALL_2(sendmsg, sendmsg_locked, + sendmsg_unlocked, sk, &msg); if (ret <= 0) goto error; @@ -3045,16 +3052,14 @@ static int __skb_send_sock(struct sock *sk, struct sk_buff *skb, int offset, int skb_send_sock_locked(struct sock *sk, struct sk_buff *skb, int offset, int len) { - return __skb_send_sock(sk, skb, offset, len, kernel_sendmsg_locked, - kernel_sendpage_locked); + return __skb_send_sock(sk, skb, offset, len, sendmsg_locked); } EXPORT_SYMBOL_GPL(skb_send_sock_locked); /* Send skb data on a socket. Socket must be unlocked. */ int skb_send_sock(struct sock *sk, struct sk_buff *skb, int offset, int len) { - return __skb_send_sock(sk, skb, offset, len, sendmsg_unlocked, - sendpage_unlocked); + return __skb_send_sock(sk, skb, offset, len, sendmsg_unlocked); } /** From patchwork Thu Mar 16 15:26:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13177822 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 08401C6FD19 for ; Thu, 16 Mar 2023 15:27:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E22F294000D; Thu, 16 Mar 2023 11:27:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D83B694000A; Thu, 16 Mar 2023 11:27:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C4D1B94000D; Thu, 16 Mar 2023 11:27:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id A719C94000A for ; Thu, 16 Mar 2023 11:27:30 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 6E431ABC40 for ; Thu, 16 Mar 2023 15:27:30 +0000 (UTC) X-FDA: 80575140660.15.9375740 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf26.hostedemail.com (Postfix) with ESMTP id 39111140018 for ; Thu, 16 Mar 2023 15:27:28 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=NMQu4bpN; spf=pass (imf26.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678980448; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zb6mFnKCVLQu6xBmdV/ljtXgUzSvLJ/bZitog7Xho38=; b=jdFjJ4elY0hdquzogrtYm3nboAzMVTdNLopBJ8O5iuEhnKyb3VZwA50cs65e7f2tT4OzbO NVKoi0LFETq3kLpaJ1vrEtHB356RIkrvjnQ5b6PN8rSmWw84mVv5iOXq8MnU1ukJYgq6JX 3GbL0Ihrmk8wUOfa4hPNU5xkbt3a/gc= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=NMQu4bpN; spf=pass (imf26.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678980448; a=rsa-sha256; cv=none; b=MgZ+IYFIhoPSeIQmmw/v1zrvD7DVRLgUY3Ijiweksa4lSUUusvQfEE4znGgnRJU75h26ri z1XJkPa0zUcj6nPcNf15qcqTXFs7Ed5P7f1Fgf0DgLz0ZXGwQy9/XAxLdU7bIZ4JJ9XPr2 LVeMP+vWxXFwtVJHrkoY8JcJPeHbRYU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678980447; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zb6mFnKCVLQu6xBmdV/ljtXgUzSvLJ/bZitog7Xho38=; b=NMQu4bpNobpBTUVnGaqnv7jVeB+Qb0O4YSoquuxZLzWZnnOVWRtQANRHH9iyN5pyQbmcB4 4KPnFgFMXQ6Y3qFEHMQc4/V1U3w4gjlClDZPbHUlBCJgy78DyPJeECzswaI9pgbYx+lYA1 qCYCYZVbc0yCm3/+WqWvofZdzjbgj14= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-584-Wn_Xf5sGMh2vYcAXsHUQRQ-1; Thu, 16 Mar 2023 11:27:24 -0400 X-MC-Unique: Wn_Xf5sGMh2vYcAXsHUQRQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A4CD3857FB3; Thu, 16 Mar 2023 15:27:23 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id A8B7C140EBF4; Thu, 16 Mar 2023 15:27:21 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Herbert Xu , linux-crypto@vger.kernel.org Subject: [RFC PATCH 23/28] algif: Remove hash_sendpage*() Date: Thu, 16 Mar 2023 15:26:13 +0000 Message-Id: <20230316152618.711970-24-dhowells@redhat.com> In-Reply-To: <20230316152618.711970-1-dhowells@redhat.com> References: <20230316152618.711970-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: pmzprgcf8rtqwe8iz7xsoh4ud8rhr36h X-Rspamd-Queue-Id: 39111140018 X-HE-Tag: 1678980448-716146 X-HE-Meta: U2FsdGVkX1/UeJtU3OI2XbsxlLRh+kwQ+Se576tgsXM8RMivQ522Xd88XQKrYI71FmY266IRFVzt0qhesNZXmxvZENVeYi5kiJ+0Ey8oDtJxBsEh9pkoncCUoF4TihYC5PfLrbzhVQvSd6hz6R/HcqbzMA43yJer43bQ9BsPp2MRFbMnxlYEDnjDkkLrf3ynpw/KrgFVkkyp3pU0w6c6trErbTHytpJ9CyLlJNIf+yAVpVl0izEARjLKMGae/n/fxa2bmJZp/VHW8qb8sjcvVKO8XRGSDAyE8ntvWxlJhECVHJZ9tsJrFlwjM1hV0MbyNj+D/RrZDVRJtCgl+xNB/9Vt+sdgZrpjR6ExjQ/Ea55cOCJY5h10q+zWEYPaVEbyVXW1vf5xSeKGe/H/pS4YDAaDJ5Y5EXucfvHlzSSubMsoZeN+V21nazjx9dLleKSNnD0WVl4MzhsnUuOdTCqtbx00ojKbwFhRmozD8cyBxLaCfVxHR0As+imtzTHzF0ZpCw+Og8NuAUD1xBS44n6uwnhwU7Z/KVeM/oaHiw4OJoL4wlXDhrQ/5qZVd4DUwT+r7AZVRFIrfG/6gF9Xtp+n3TMlP8CRLhLHbQlu+y6pCftSTF9tXya+aaeMTRqxPdlDAc+TZWqjyiOU6AgTW9guPXwC41fIYM4zQQZoUzAQsOsUCiE7EmkBRbB2HOJ87WbrCTMtWHmBjvyT93TCoBwqum1DhmUZO209wzOypN/zWBEB0yS0PkV20LVyLwfypPzS3B/k2neZ53Vu3W+KIDoP7G+XLfqtN4SiEmingeLR84dUXgyohQV4P9pWTcWtHs+ZxzQMqUrZvLGtCCuDlHptvi0E5yKB9AGpeyak3CpgfZdVoVuezeyUeW0zc4uvG9YOXoNuiLRH8saAJ+DaiAMR/Tab9IKcVH34ePxnQtY3KbRflnlSuj5X4mwEQpUmyTbc+ZlWCdCfISqOoRkK/ni VNmexE+J vdP7GPifxLVZvbH2v/niTTd7zn7j80Z2NShZWLXXQUZmyMuN1LEEqUu5fpfEkz1zP75ERzOkffarnpsEGf7MSZJ8suuahIM/7Q1ND5IbKhvGNiRaTRO7uswoU8lPc11XHKC3DX1YvTCaKTOlqvRLtETVvr1QXQqc2ooiF1FAAkG0io0DQv/hy3j+ic2vVOUV6nU+LAS4TDQlZMX3FUOJSQYakcFTuLDFICCHXoh90qkBa3NAJkI3Z0EcY3gg2W6WDz5NAjdQEIfLlwVdd5ZE+hw5pO5B02NFtVG/Ilhd4UE5sbCbaxzzK0Vq8tGfOAe28UOc6xegBcIQuGpo0vSo4YwFWqeZ0aj4dgK6TqDYI+x86ZmeXX++GnFu45wsms3by+KWee1JHy+1E7rprcllUQ0mr4JzhtkBejycM/zvNTWH/ztTRzFy7H+RBZa6qb+e03yXAO1eCLPvWmmvdeP4M92n/bDKysM5AEwr1UUSgSVb91TS3f/3b1jwmyA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Remove hash_sendpage*() and use hash_sendmsg() as the latter seems to just use the source pages directly anyway. Signed-off-by: David Howells cc: Herbert Xu cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-crypto@vger.kernel.org cc: netdev@vger.kernel.org --- crypto/algif_hash.c | 66 --------------------------------------------- 1 file changed, 66 deletions(-) diff --git a/crypto/algif_hash.c b/crypto/algif_hash.c index 1d017ec5c63c..52f5828a054a 100644 --- a/crypto/algif_hash.c +++ b/crypto/algif_hash.c @@ -129,58 +129,6 @@ static int hash_sendmsg(struct socket *sock, struct msghdr *msg, return err ?: copied; } -static ssize_t hash_sendpage(struct socket *sock, struct page *page, - int offset, size_t size, int flags) -{ - struct sock *sk = sock->sk; - struct alg_sock *ask = alg_sk(sk); - struct hash_ctx *ctx = ask->private; - int err; - - if (flags & MSG_SENDPAGE_NOTLAST) - flags |= MSG_MORE; - - lock_sock(sk); - sg_init_table(ctx->sgl.sg, 1); - sg_set_page(ctx->sgl.sg, page, size, offset); - - if (!(flags & MSG_MORE)) { - err = hash_alloc_result(sk, ctx); - if (err) - goto unlock; - } else if (!ctx->more) - hash_free_result(sk, ctx); - - ahash_request_set_crypt(&ctx->req, ctx->sgl.sg, ctx->result, size); - - if (!(flags & MSG_MORE)) { - if (ctx->more) - err = crypto_ahash_finup(&ctx->req); - else - err = crypto_ahash_digest(&ctx->req); - } else { - if (!ctx->more) { - err = crypto_ahash_init(&ctx->req); - err = crypto_wait_req(err, &ctx->wait); - if (err) - goto unlock; - } - - err = crypto_ahash_update(&ctx->req); - } - - err = crypto_wait_req(err, &ctx->wait); - if (err) - goto unlock; - - ctx->more = flags & MSG_MORE; - -unlock: - release_sock(sk); - - return err ?: size; -} - static int hash_recvmsg(struct socket *sock, struct msghdr *msg, size_t len, int flags) { @@ -285,7 +233,6 @@ static struct proto_ops algif_hash_ops = { .release = af_alg_release, .sendmsg = hash_sendmsg, - .sendpage = hash_sendpage, .recvmsg = hash_recvmsg, .accept = hash_accept, }; @@ -337,18 +284,6 @@ static int hash_sendmsg_nokey(struct socket *sock, struct msghdr *msg, return hash_sendmsg(sock, msg, size); } -static ssize_t hash_sendpage_nokey(struct socket *sock, struct page *page, - int offset, size_t size, int flags) -{ - int err; - - err = hash_check_key(sock); - if (err) - return err; - - return hash_sendpage(sock, page, offset, size, flags); -} - static int hash_recvmsg_nokey(struct socket *sock, struct msghdr *msg, size_t ignored, int flags) { @@ -387,7 +322,6 @@ static struct proto_ops algif_hash_ops_nokey = { .release = af_alg_release, .sendmsg = hash_sendmsg_nokey, - .sendpage = hash_sendpage_nokey, .recvmsg = hash_recvmsg_nokey, .accept = hash_accept_nokey, }; From patchwork Thu Mar 16 15:26:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13177826 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 343C0C761AF for ; Thu, 16 Mar 2023 15:27:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C72E8900008; Thu, 16 Mar 2023 11:27:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C23F8900003; Thu, 16 Mar 2023 11:27:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AC465900008; Thu, 16 Mar 2023 11:27:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 9BA16900003 for ; Thu, 16 Mar 2023 11:27:40 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 6325A121405 for ; Thu, 16 Mar 2023 15:27:40 +0000 (UTC) X-FDA: 80575141080.09.C15FEE9 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf16.hostedemail.com (Postfix) with ESMTP id 9E95118000C for ; Thu, 16 Mar 2023 15:27:38 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KkGB8Kun; spf=pass (imf16.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678980458; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=14pAi6JHwU3C53h/ZQrPmmNB396J3kzB8Cmz3sg/vjY=; b=lvAI2dCmYatT9ZxEk9/sngQlw7FOU21WLj5vHppXDWVLxyuqBZRaHNrRNUY7vM9fMqEsyi +axF30Ml0haAc7rnOo1FNKBXFRGjG7r9Kzp/0MairB2XoefhjYyk96WUjgaT8IpAfxqhkT /oYocm9jH+bnfMnclFuqRGXw91J9jQE= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KkGB8Kun; spf=pass (imf16.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678980458; a=rsa-sha256; cv=none; b=MUFxDvCjLkytZ5VBzXohH6KIG9qSGAtoUovu4k6I5N82LGOzOj+jySRZZo5qGuyUOPKacc 4zza4IUIvlF9+MqchE5OmTk4CBkVNDJ3JH4RuJVli94REtxLQ4BYicijkYpV7uRjoIXwZW hdWGi4KG2ZsPPVVNMV+wfkduteikRbM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678980457; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=14pAi6JHwU3C53h/ZQrPmmNB396J3kzB8Cmz3sg/vjY=; b=KkGB8Kun1f7XldWy1TObInYwkXKlfO9UNdV8/Nog8OdEIMAU5fOWZTctWt5qCf5kj/E8VP xriZsVP7rJpS5SA7OgoXuhy58lAkJGlzhnWlAoz9Ew/NH/mX1UpEH3bqnbZk7pa+nk/eqn nwYVQkSQIIT8D6u8ADBYUFDmcnF6AHg= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-319-pD9jpABAOzyxINJMnpA1QQ-1; Thu, 16 Mar 2023 11:27:27 -0400 X-MC-Unique: pD9jpABAOzyxINJMnpA1QQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 72108185A790; Thu, 16 Mar 2023 15:27:26 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5A34D2027040; Thu, 16 Mar 2023 15:27:24 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Ilya Dryomov , Xiubo Li , ceph-devel@vger.kernel.org Subject: [RFC PATCH 24/28] ceph: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage() Date: Thu, 16 Mar 2023 15:26:14 +0000 Message-Id: <20230316152618.711970-25-dhowells@redhat.com> In-Reply-To: <20230316152618.711970-1-dhowells@redhat.com> References: <20230316152618.711970-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Stat-Signature: u1qrqkgdjahieexkeuueaostj1ak93sf X-Rspam-User: X-Rspamd-Queue-Id: 9E95118000C X-Rspamd-Server: rspam06 X-HE-Tag: 1678980458-992726 X-HE-Meta: U2FsdGVkX1//7QyyPvgc3WZTiMLM8IT6Ry/WkCAjtLhLx9AxzE6iYfTJsNL4J+grxWfQ3SozgfEaa7sLuPglHTiehc9f7a9ALAoy9Wxtz4q8NYM7cHv8a6uLkOkIsSSJa7AZ7lR13pjgPbxGHixIERdhPIbbPPs6k0PYn39OWBZl1E7Egv7yAs7Te5DNKKbTtF7pPaCJ1jkTXRy7O3Vr0V4d3MN5nol7ko2CMkVzMVU3GcJb7t+l2SjUX7/VILjJafZ/he/hiMvCU/7O2d26vFNgG6D2yjoqBgaU6g8HOw2XZJ/EYi+kjOWI9lnlEA/BDtgjHZschIWZ7llvJlwi/kPI4v7ffOu/JUidJ/ZZMBx64OymaDBz9wOStJquoxdbGjfDC05Xtu8cTLn+S3/miXfANHMOvN3/LlV/uu55ll8MyWXsxiPZG700+KGb1i+GdbCXmrjk/0qr8x/SgdpXIof7zniMnM3QcfCFTyF7hZCPFMyXDYPWhPtBcvJVFSzijEeiNp/W3kzThCPUldxXV/5j0RJhzcfyowYXf2fSjDt8cCQHUJ5d4lT26vB7ZxQ9lnJNe/+mgGzklzpL0VKr+rj65RSrcSEVaeieCHcLGLGjQzpp8fbmvVrm+8grnD3aysjOTIy0vBXyhSjq8cfMsTUiFpSywIyOSDYD13ij6wdkx/xe4BMM0vHAokFzJOqxFlqxJtAmziOXwhEkwaH9vJ8y04gSmJXIf7Ve9fwEens1bX2ro31W2qBOsAmrqcB5mtq+ukp9XhL3UPgCxjgFPCG7wYA3XzjMvZtn9fdDF426tIoWQXmfoV9w0zGNoVzIw59e9IGp12i9N63btjOX+glGM0PD97tgk5wHWpKLhK4uMlOHW3W2wVxt0LUOySIxUQJq+HKFuM0yplBdc633YwzJdXbJm4tcDeQNZct9bDPIPrDud0DJk0KAjG0aC2GeDb5QoWB+lMY5g9yzRs7 4vVBLy0g RYOQqncRExjl2GLVtdeA8scoCLLKto4n5HLd2cLPx5FgM13ZMnkAigMX42PUW/1cq/iStVae3jiEInVBfgTSRTckxoH27mfaiE72OwHRXODUPSRLJvoApNOfCk8goRhfajDeveCR8rEG0wvsciKPpX9FJuP+DoqaZ3jv7CfaVO1Hasmi1jIFQBlbzJq6gl+KPfYZzAoDTJD6AhuBkYTQOiJ+8g79XKLifELict3YSpLPK2GQo/w4Qqv9idz8V+gAizzA77zif9+EonKLhKds6lNZWKrq6hr6ba5wknLNwHnh1eDkcS8KV3JzquHqUAIVnckH75n0yxOLciULdLzdrGQf7DisQLY43J7KTPCoZuYsAjt8/QXkNeP7rZVmfHlEwc8+QfU+QEyQvZs1hR4w+s6iPu3DktHxTEYnCHd6JUUQOfCGYV7zZP8y+Y4YotpkgeNEGWegP1ElMMGxOypWSqPotgA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use sendmsg() and MSG_SPLICE_PAGES rather than sendpage in ceph when transmitting data. For the moment, this can only transmit one page at a time because of the architecture of net/ceph/, but if write_partial_message_data() can be given a bvec[] at a time by the iteration code, this would allow pages to be sent in a batch. Signed-off-by: David Howells cc: Ilya Dryomov cc: Xiubo Li cc: Jeff Layton cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: ceph-devel@vger.kernel.org cc: netdev@vger.kernel.org --- net/ceph/messenger_v2.c | 89 +++++++++-------------------------------- 1 file changed, 18 insertions(+), 71 deletions(-) diff --git a/net/ceph/messenger_v2.c b/net/ceph/messenger_v2.c index 301a991dc6a6..1637a0c21126 100644 --- a/net/ceph/messenger_v2.c +++ b/net/ceph/messenger_v2.c @@ -117,91 +117,38 @@ static int ceph_tcp_recv(struct ceph_connection *con) return ret; } -static int do_sendmsg(struct socket *sock, struct iov_iter *it) -{ - struct msghdr msg = { .msg_flags = CEPH_MSG_FLAGS }; - int ret; - - msg.msg_iter = *it; - while (iov_iter_count(it)) { - ret = sock_sendmsg(sock, &msg); - if (ret <= 0) { - if (ret == -EAGAIN) - ret = 0; - return ret; - } - - iov_iter_advance(it, ret); - } - - WARN_ON(msg_data_left(&msg)); - return 1; -} - -static int do_try_sendpage(struct socket *sock, struct iov_iter *it) -{ - struct msghdr msg = { .msg_flags = CEPH_MSG_FLAGS }; - struct bio_vec bv; - int ret; - - if (WARN_ON(!iov_iter_is_bvec(it))) - return -EINVAL; - - while (iov_iter_count(it)) { - /* iov_iter_iovec() for ITER_BVEC */ - bvec_set_page(&bv, it->bvec->bv_page, - min(iov_iter_count(it), - it->bvec->bv_len - it->iov_offset), - it->bvec->bv_offset + it->iov_offset); - - /* - * sendpage cannot properly handle pages with - * page_count == 0, we need to fall back to sendmsg if - * that's the case. - * - * Same goes for slab pages: skb_can_coalesce() allows - * coalescing neighboring slab objects into a single frag - * which triggers one of hardened usercopy checks. - */ - if (sendpage_ok(bv.bv_page)) { - ret = sock->ops->sendpage(sock, bv.bv_page, - bv.bv_offset, bv.bv_len, - CEPH_MSG_FLAGS); - } else { - iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bv, 1, bv.bv_len); - ret = sock_sendmsg(sock, &msg); - } - if (ret <= 0) { - if (ret == -EAGAIN) - ret = 0; - return ret; - } - - iov_iter_advance(it, ret); - } - - return 1; -} - /* * Write as much as possible. The socket is expected to be corked, * so we don't bother with MSG_MORE/MSG_SENDPAGE_NOTLAST here. * * Return: - * 1 - done, nothing (else) to write + * >0 - done, nothing (else) to write * 0 - socket is full, need to wait * <0 - error */ static int ceph_tcp_send(struct ceph_connection *con) { + struct msghdr msg = { + .msg_iter = con->v2.out_iter, + .msg_flags = CEPH_MSG_FLAGS, + }; int ret; + if (WARN_ON(!iov_iter_is_bvec(&con->v2.out_iter))) + return -EINVAL; + + if (con->v2.out_iter_sendpage) + msg.msg_flags |= MSG_SPLICE_PAGES; + dout("%s con %p have %zu try_sendpage %d\n", __func__, con, iov_iter_count(&con->v2.out_iter), con->v2.out_iter_sendpage); - if (con->v2.out_iter_sendpage) - ret = do_try_sendpage(con->sock, &con->v2.out_iter); - else - ret = do_sendmsg(con->sock, &con->v2.out_iter); + + ret = sock_sendmsg(con->sock, &msg); + if (ret > 0) + iov_iter_advance(&con->v2.out_iter, ret); + else if (ret == -EAGAIN) + ret = 0; + dout("%s con %p ret %d left %zu\n", __func__, con, ret, iov_iter_count(&con->v2.out_iter)); return ret; From patchwork Thu Mar 16 15:26:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13177824 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 232C7C76196 for ; Thu, 16 Mar 2023 15:27:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2886C940013; Thu, 16 Mar 2023 11:27:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1EBA994000A; Thu, 16 Mar 2023 11:27:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 08B67940013; Thu, 16 Mar 2023 11:27:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id EB85194000A for ; Thu, 16 Mar 2023 11:27:35 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id B9D7BABC07 for ; Thu, 16 Mar 2023 15:27:35 +0000 (UTC) X-FDA: 80575140870.20.7F1D317 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf19.hostedemail.com (Postfix) with ESMTP id 18BC11A0011 for ; Thu, 16 Mar 2023 15:27:32 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=T0560YF+; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf19.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678980453; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=92bJ6Ynf1CSTwRAMP7lCPtKM1kI8JBpDgRb6Ls1sSMY=; b=aM7xaOCeE/cUybloDMGdO+JG5TCZw718pggXCbK2sMOIhYn3QzeBuDVOLtdSbcSqzlGkzB yBf1/zgc/FSxU+ux2OaKSm3OEVq/64Oe8VTbxABisseOoErd9aI+yNaf8T+Hc9UKfEJvxL BmrVcnC9sjoHpCGdhTqrGm8lMXeaEc4= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=T0560YF+; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf19.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678980453; a=rsa-sha256; cv=none; b=8hwVy07kytx52juTvjsiZeJovMLCZhwllKRpPbCeR1AfmOCEMpmhM/T39C5oQtgXUZlDWz yQwzO3MTNA6sQR2sbnKk4r8miYVv4ixUvFB8LCJTvrpcowxOEjSwhWdifwrljLmY2aw1jw FTVGlPz6oHZyOecZZMpJR7CQNSbrweU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678980452; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=92bJ6Ynf1CSTwRAMP7lCPtKM1kI8JBpDgRb6Ls1sSMY=; b=T0560YF+UveLKxyIQx6ZoR8+u2KbXX6nJFjTC5hjHOz+cSSMCuSj4NAPRca4lgCrGFvgOY a5IlhQiCB/0Aed3TQL2WKKrNXo6k9MZyAW77v+B0xLHNLqx5CmWnWpMmza8FxDQXdGGKmy oRdN18ieOdADNNnOI62KEaUJGuRjw54= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-135-Uky157iCM7O_1XjnKbwmzg-1; Thu, 16 Mar 2023 11:27:30 -0400 X-MC-Unique: Uky157iCM7O_1XjnKbwmzg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2D062185A791; Thu, 16 Mar 2023 15:27:29 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 150B540B3ED6; Thu, 16 Mar 2023 15:27:26 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Santosh Shilimkar , linux-rdma@vger.kernel.org, rds-devel@oss.oracle.com Subject: [RFC PATCH 25/28] rds: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage Date: Thu, 16 Mar 2023 15:26:15 +0000 Message-Id: <20230316152618.711970-26-dhowells@redhat.com> In-Reply-To: <20230316152618.711970-1-dhowells@redhat.com> References: <20230316152618.711970-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Queue-Id: 18BC11A0011 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: r6xx13ongw7nk9abitzrjybzr6mz5d4g X-HE-Tag: 1678980452-736443 X-HE-Meta: U2FsdGVkX1/wPpA6HPMAKNGiuX5h+opgsDbBEN7rWgCNjYXV37jgMuwjMcxqju3v0Z8wEDgCqpzfFOgIuFVcPj69zJkpBGeQawbYOZVwtEym3eJRQnV3lkTEmTtdlT+cXGNHTYbmB6hsNKsczBLSFdKIKFYlOXYNxqspHk1jzkNfjD7WbJCvCXwSU08zaVXa3hESDlmjBgR98gPBa4i0VrLpqmNue+xAgsMIAZjSc1yW11efiwY3pqQfpcIx2WVNdGKCxa6vwA/c6yvq6ejPAN6c815G29TV+4QiqeyPo27EGC+Y/4T890erUjCUDSKL240w0Wg+TgwKjjDVs52VLRJIcLBoIz2wAK3u/g1ZQrEXt86N6ozABma/2GPhzskegks6QDmva9UEJ7YyLIIJDaWIrAwQNFxjI7PrLfhVoQg4ZrT+wiYhVLXI1QO7Bm1O7waQYhMDE+QvhobR7d0xkyS0eeQ7CYhgViagThDmck5xh+1tnWpkMRWjvAHAnpe4VCsk23m2zwEOkciJs1RuVfiX+HXI7GG9yKTQziH/k/Nl4wyl4d2CClZ/2zr7YyZVyhMyntLXFXSZ9tUj2I1SMf4f5R24dv0DUzJlwGv3N+AcqYET6E/mWk/ZRzLlxvJr6iTUOdpLA/c+EZmJY7FR9OZ+h59oAHbawHJG/vcx/el4O+16zJwWcyOkEGDxgNhafL1wFta4know6bB1ZJhcjo7RyoFpRAx1CKdBf+jtp7S1HojyylHozHO1OtJSY8b0B9ge8kSqGceiF41uvwZ32qUg7ur6N3PgkJzhSWedELXpoOdUyjag34gWRLw7Q1ZOqYQSCFVXzN7DlEGiwcX6DRoyxQj0CpjtDEak0HOoTiY7m9lqCz1o5QHgtVOpFsUqV8doroeVVdOYrAiLViNn2RhptQ3EvlUH4aPRNgNSfd42hKicwwkRIotPHnwZRqTYFl+iGRhqfWLkc/lrFMM wMSnkVo0 T4l4tL1SMmN0qt9au7X8lTOubjHhz20p5kO74S/dFwcfThqibROKtcBKcgTjtf7hvAQMYfVNCqnKPhvgh+6MpYgGB3o4kay34zBowY6OkhT4c12pCg2x5v+aSFD/nFfhYn5r/Da91Tqi9WXBdmDw0GBNSHjULY5O0RFRXaq3abL+1XBM9MNx7r/4Ly0TYk4TL436GaREQUppKSS7OF9cg/yXMJKniCj4YwoshIoNTjv9VMMsqD6YxaZ0hPmM6wS1EJyBYMiUy9OFvy6YN7YgsLlPZ2niX2IE4lrgdlARjo/m99EiLHgkHAdVnUGWzBcMGJ2SAvRAJm9jceFT54yD5fc7WCd/Jqb3ZZRv/lReSOvsqA7N//OhvZaoSZw0X4CEf/VosAVSIDuEytPnfTsY+kRgatGG+1RX32MeH5XOwvZepk9eb2B+p+vhwDFf5i5iKi7TcDUBDTsNEpWC7dzbD17Bi+w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When transmitting data, call down into TCP using a single sendmsg with MSG_SPLICE_PAGES to indicate that content should be spliced rather than performing several sendmsg and sendpage calls to transmit header and data pages. To make this work, the data is assembled in a bio_vec array and attached to a BVEC-type iterator. The header are copied into memory acquired from zcopy_alloc() which just breaks a page up into small pieces that can be freed with put_page(). Signed-off-by: David Howells cc: Santosh Shilimkar cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-rdma@vger.kernel.org cc: rds-devel@oss.oracle.com cc: netdev@vger.kernel.org --- net/rds/tcp_send.c | 80 ++++++++++++++++++++-------------------------- 1 file changed, 35 insertions(+), 45 deletions(-) diff --git a/net/rds/tcp_send.c b/net/rds/tcp_send.c index 8c4d1d6e9249..0d6eb85a930d 100644 --- a/net/rds/tcp_send.c +++ b/net/rds/tcp_send.c @@ -32,6 +32,7 @@ */ #include #include +#include #include #include "rds_single_path.h" @@ -52,29 +53,24 @@ void rds_tcp_xmit_path_complete(struct rds_conn_path *cp) tcp_sock_set_cork(tc->t_sock->sk, false); } -/* the core send_sem serializes this with other xmit and shutdown */ -static int rds_tcp_sendmsg(struct socket *sock, void *data, unsigned int len) -{ - struct kvec vec = { - .iov_base = data, - .iov_len = len, - }; - struct msghdr msg = { - .msg_flags = MSG_DONTWAIT | MSG_NOSIGNAL, - }; - - return kernel_sendmsg(sock, &msg, &vec, 1, vec.iov_len); -} - /* the core send_sem serializes this with other xmit and shutdown */ int rds_tcp_xmit(struct rds_connection *conn, struct rds_message *rm, unsigned int hdr_off, unsigned int sg, unsigned int off) { struct rds_conn_path *cp = rm->m_inc.i_conn_path; struct rds_tcp_connection *tc = cp->cp_transport_data; + struct msghdr msg = { + .msg_flags = MSG_SPLICE_PAGES | MSG_DONTWAIT | MSG_NOSIGNAL, + }; + struct bio_vec *bvec; + unsigned int i, size = 0, ix = 0; + bool free_hdr = false; int done = 0; - int ret = 0; - int more; + int ret = -ENOMEM; + + bvec = kmalloc_array(1 + sg, sizeof(struct bio_vec), GFP_KERNEL); + if (!bvec) + goto out; if (hdr_off == 0) { /* @@ -101,41 +97,30 @@ int rds_tcp_xmit(struct rds_connection *conn, struct rds_message *rm, /* see rds_tcp_write_space() */ set_bit(SOCK_NOSPACE, &tc->t_sock->sk->sk_socket->flags); - ret = rds_tcp_sendmsg(tc->t_sock, - (void *)&rm->m_inc.i_hdr + hdr_off, - sizeof(rm->m_inc.i_hdr) - hdr_off); + ret = zcopy_memdup(sizeof(rm->m_inc.i_hdr) - hdr_off, + (void *)&rm->m_inc.i_hdr + hdr_off, + &bvec[ix], GFP_KERNEL); if (ret < 0) goto out; - done += ret; - if (hdr_off + done != sizeof(struct rds_header)) - goto out; + free_hdr = true; + size += bvec[ix].bv_len; + ix++; } - more = rm->data.op_nents > 1 ? (MSG_MORE | MSG_SENDPAGE_NOTLAST) : 0; - while (sg < rm->data.op_nents) { - int flags = MSG_DONTWAIT | MSG_NOSIGNAL | more; - - ret = tc->t_sock->ops->sendpage(tc->t_sock, - sg_page(&rm->data.op_sg[sg]), - rm->data.op_sg[sg].offset + off, - rm->data.op_sg[sg].length - off, - flags); - rdsdebug("tcp sendpage %p:%u:%u ret %d\n", (void *)sg_page(&rm->data.op_sg[sg]), - rm->data.op_sg[sg].offset + off, rm->data.op_sg[sg].length - off, - ret); - if (ret <= 0) - break; - - off += ret; - done += ret; - if (off == rm->data.op_sg[sg].length) { - off = 0; - sg++; - } - if (sg == rm->data.op_nents - 1) - more = 0; + for (i = sg; i < rm->data.op_nents; i++) { + bvec_set_page(&bvec[ix], + sg_page(&rm->data.op_sg[i]), + rm->data.op_sg[i].length - off, + rm->data.op_sg[i].offset + off); + off = 0; + size += bvec[ix].bv_len; + ix++; } + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, bvec, ix, size); + ret = sock_sendmsg(tc->t_sock, &msg); + rdsdebug("tcp sendmsg-splice %u,%u ret %d\n", ix, size, ret); + out: if (ret <= 0) { /* write_space will hit after EAGAIN, all else fatal */ @@ -158,6 +143,11 @@ int rds_tcp_xmit(struct rds_connection *conn, struct rds_message *rm, } if (done == 0) done = ret; + if (bvec) { + if (free_hdr) + put_page(bvec[0].bv_page); + kfree(bvec); + } return done; } From patchwork Thu Mar 16 15:26:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13177825 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39DCBC761A6 for ; Thu, 16 Mar 2023 15:27:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CC2C5940014; Thu, 16 Mar 2023 11:27:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C4AB094000A; Thu, 16 Mar 2023 11:27:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A9CC6940014; Thu, 16 Mar 2023 11:27:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 980F494000A for ; Thu, 16 Mar 2023 11:27:38 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 649CA1C64E3 for ; Thu, 16 Mar 2023 15:27:38 +0000 (UTC) X-FDA: 80575140996.25.937B5F3 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf27.hostedemail.com (Postfix) with ESMTP id 9B6AF40028 for ; Thu, 16 Mar 2023 15:27:36 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=SRCAwBoe; spf=pass (imf27.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678980456; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6d/Oxl1Cdu8ybDL5tOgPD1gv3cW2fyMj4HBClKMadWo=; b=OkJcHCH2TSte+WvRUMSHnJ/mWhoLSzNjAAoU14fjbtULRSg4wLBFd4Fj8n3tImBSL6tddV TKd0qXmAxGcqizuE2jPBjD5CZkEUAmYweRGqcVN45xqRx/6mdUsx1RpcKmihvgg/vYj6Vf mTxD+wvSXFpF9vlZPqqh149exU+zOts= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=SRCAwBoe; spf=pass (imf27.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678980456; a=rsa-sha256; cv=none; b=OeFTylzNZSc7qyYgq+ynIHEIHa0NxuV3/wruzfi3jKKj3xo4LoKAXG2EOQd/vy6ALp81xL f/3VygYNVQ+IkFcM801oXwFygZYx9NuY3Ya2BXT1+cmHTkHCu9S1dPXMJiJQKD4Aq/8qBW J5Oxa7JeF94bDQwS9mobpPVgQeaHsn0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678980456; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6d/Oxl1Cdu8ybDL5tOgPD1gv3cW2fyMj4HBClKMadWo=; b=SRCAwBoekLpOuPsVR5OQ7R3SouOnwNaWXaP6sertbSCGkP8LBXOQo+XG7IoI9pil9r4TXv WWJrezE2l4JOXC5g0S1UjZsnaii63CcFMU++8ws+tO7PWv+TS5xpQINF1vOQGPJzS7z4IP VoR00+lKhmBV/Q3Wo6mpf8MFGBFAEW8= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-630-rdzabOPcMoiPR4mi-J1pAg-1; Thu, 16 Mar 2023 11:27:32 -0400 X-MC-Unique: rdzabOPcMoiPR4mi-J1pAg-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0D8C828237D5; Thu, 16 Mar 2023 15:27:32 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id DC5562166B26; Thu, 16 Mar 2023 15:27:29 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christine Caulfield , David Teigland , cluster-devel@redhat.com Subject: [RFC PATCH 26/28] dlm: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage Date: Thu, 16 Mar 2023 15:26:16 +0000 Message-Id: <20230316152618.711970-27-dhowells@redhat.com> In-Reply-To: <20230316152618.711970-1-dhowells@redhat.com> References: <20230316152618.711970-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 9B6AF40028 X-Stat-Signature: tpw85ccewemyx5dqabxeghdou7a3fh8j X-HE-Tag: 1678980456-342494 X-HE-Meta: U2FsdGVkX1/ZDiuxIP4OBUaPnRnY61G0WC3Oi7a7tu0jxdU4UySuNCzT4+TmfPujL/uI88kz3VUtofRQ3+1VXHSf/i1CHapr2ZVf+27/rnA3jvK5ISR0Xb1fPK9UIFYSOZQXPF/vG4QRMWsHmHdskC/vdJXqK082ydEuh7t7/G8gR+MiNcvElCwEfBQjwB+5Ef7zBEvGbk3hA46gfp6BYXOvtJ+LjRom+TH2PDLiIQbOLeNrDKv9EQOq3CROpvFQohQ5hSYbC6J1CmT/wSc9ShmS2nso7AiG6FQF6W/og+4dviXoYA9bVhsAkEYWiTgL8XOxoCU0GynhprBKjIHq8Xlk9XZAW+gKKPix4H9Wyo4NdkYgQD9Z0Q8VuBS12NNgJrAh6X01uD5K4ngEdH2kwaakEcOGYjPsag3vHaS47dVWY6ZmB2QanvMVRdm1Evl5qGbd6ikbgPCDOb1OIa1zquhhXkrs4d/kcv70YghwLM0BQxIoD4jOCKS5p3XAv0UXMb9ikyCs536gAyKIYidW9oi3dibaoJNG6f7+E6WPZdxI291YjpMo4R2G1uIn3SY7CyZe+7JYR+AfHTERC/rolwAjDjz5MtQzPsemax5dbGDHNanaqgayOx2/2Nqkb14V5flYi6fdALufNjwayyuS2o/ZSDe7W+nrgunMtuI0tlwSyEXY4nmhYc4EPnS6pVOjQwAwULsM/55pjUjYW6vlFu8gG3nUmkc/a2ozzBWqpPRUobOKezlflx4PDg2CDW9E7x9y+uK3O4ddpO8BW7UhuDRI70mtMHwb8Ty0VFzK2Q57xD+kIhyJimD2FS75y8KCPpJjORcM0Ji/6JWKS4z/cd7nXFI361npQqKL9FDNLdAbJBBnWlKoYpLK32O1TFF7K08u6jch4BNp/DEHHKyDakaPG3whOUoAbz64We+nkjHzaQvbv7MM0ddzDF+dBTCTTYKcqGpdUU8TRF90/sL c449mIKV fLm4A8m6oVnL+5sN/R75sD55skvrXorMNbOW81phcXGbAt7ckEIc4+nxM2YTsWSD8dC02MHZqLigxN2Rtqi7MghKssj+zHwRc7Kf5Uo44DPpG9GlcFjIZjbKrz77JzqcX5hviQO2DD7GorHGRA8QNCRsU0vQL+GxEtV9P3Jmn44fpfDTc++0tJHfykmukCuoJ55OhM0NdDZVv33XFGfEYXxT2U9wez+zJ1PB7MJfOjkaWRRZEgyM+97B/YC9qRRitLEDN21keUL2PFR0Ew1BhsQuJE/+QDaaFWsnnYJTAU/GewVCpsp0erqO+gUYarWO/imV9dB8We/FDA5t8Dr5P5NxXo9Il2/V1R8c03s0PMAKUOtUCNwIksL/SJugj5tkFPx7zufX8Q9RTkf+/lARqr54XyKxa1NdPOZaD2JCCrr7wM5hvga3/X2q8V2to2+l7u49A X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When transmitting data, call down a layer using a single sendmsg with MSG_SPLICE_PAGES to indicate that content should be spliced rather using sendpage. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Christine Caulfield cc: David Teigland cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: cluster-devel@redhat.com cc: netdev@vger.kernel.org --- fs/dlm/lowcomms.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/fs/dlm/lowcomms.c b/fs/dlm/lowcomms.c index a9b14f81d655..9c0c691b6106 100644 --- a/fs/dlm/lowcomms.c +++ b/fs/dlm/lowcomms.c @@ -1394,8 +1394,11 @@ int dlm_lowcomms_resend_msg(struct dlm_msg *msg) /* Send a message */ static int send_to_sock(struct connection *con) { - const int msg_flags = MSG_DONTWAIT | MSG_NOSIGNAL; struct writequeue_entry *e; + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = MSG_SPLICE_PAGES | MSG_DONTWAIT | MSG_NOSIGNAL, + }; int len, offset, ret; spin_lock_bh(&con->writequeue_lock); @@ -1411,8 +1414,9 @@ static int send_to_sock(struct connection *con) WARN_ON_ONCE(len == 0 && e->users == 0); spin_unlock_bh(&con->writequeue_lock); - ret = kernel_sendpage(con->sock, e->page, offset, len, - msg_flags); + bvec_set_page(&bvec, e->page, len, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, len); + ret = sock_sendmsg(con->sock, &msg); trace_dlm_send(con->nodeid, ret); if (ret == -EAGAIN || ret == 0) { lock_sock(con->sock->sk); From patchwork Thu Mar 16 15:26:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13177827 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37454C7618D for ; Thu, 16 Mar 2023 15:27:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C72C4900003; Thu, 16 Mar 2023 11:27:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C2388900002; Thu, 16 Mar 2023 11:27:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A9D13900003; Thu, 16 Mar 2023 11:27:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 99906900002 for ; Thu, 16 Mar 2023 11:27:43 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 69C2D1A07C0 for ; Thu, 16 Mar 2023 15:27:43 +0000 (UTC) X-FDA: 80575141206.09.A684ED3 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf26.hostedemail.com (Postfix) with ESMTP id 9E37914001B for ; Thu, 16 Mar 2023 15:27:40 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=etjMSyAa; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf26.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678980460; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CNHQplOZd5HfwcKa/y8evYEBVb89JfsyQlI6+74E8fk=; b=dh4RvOEG2MLUgXu1nLjFlLBJnQDlHYVCgg6PCrv+nQfKYLyL0m7EgcVAGXdl57geplrSG/ j5iQotr+cqGEU+xRiIfGCg4+Kr6xPWbD4gejtXG5dyLJcI3EFiU4yqjFO7KWpUOaWo5i01 HfQqsJrY4sRUzG8Eew2ZESS87+u/fqo= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=etjMSyAa; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf26.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678980460; a=rsa-sha256; cv=none; b=LFkAdjHKV6ubTQMWLaCxfbF1AFeLX6lFLxvTXkLrNWunClEBOsPc1OsNiOccRz8cv8vFlg tgIx39B9JnCfDNQ9Zhej1AbVqdyeZTvSmS4WivzeWMqRF8WsQvn338nMofOp0WpAteNGwB t/hAUxDz3gyBm2ticP+BzdDWuDOCIs8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678980459; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CNHQplOZd5HfwcKa/y8evYEBVb89JfsyQlI6+74E8fk=; b=etjMSyAaUMrXETAwP6TjkzO4Aq1Ju9tGjaxVgQ5nKvw68CX54Iw1F+JLz+IrK8/iOmUfEv IPrDz4p2ZBaYRZqyhdfW9oFYB1yHbsuAHF37Jlu4xEbeUvVEV8P0XlcRBMgWnHvueFy946 4Ocjn4o3Pt+zn9Lyt9IvrleG7Jpl+h8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-321-2P0S9tmzOBuO6RSNRr9oBQ-1; Thu, 16 Mar 2023 11:27:36 -0400 X-MC-Unique: 2P0S9tmzOBuO6RSNRr9oBQ-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E5750887402; Thu, 16 Mar 2023 15:27:34 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9F26035453; Thu, 16 Mar 2023 15:27:32 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Trond Myklebust , Anna Schumaker , Chuck Lever , linux-nfs@vger.kernel.org Subject: [RFC PATCH 27/28] sunrpc: Use sendmsg(MSG_SPLICE_PAGES) rather then sendpage Date: Thu, 16 Mar 2023 15:26:17 +0000 Message-Id: <20230316152618.711970-28-dhowells@redhat.com> In-Reply-To: <20230316152618.711970-1-dhowells@redhat.com> References: <20230316152618.711970-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 9E37914001B X-Stat-Signature: wauyb7kyw6g6gw4bb9jqnxdb893kpoz6 X-HE-Tag: 1678980460-348316 X-HE-Meta: U2FsdGVkX19RI3Oju24acgTEEGr+FvTQW+n9+/lW3gvN5SDB0PaWK78C/LThbcMsVPRWB+eN3KWliowIKsmDZ2KbkbuKlzrKgRvBib/lMldtmYQjZe3wcDqgKa/l/nV1v+7hXNT59GAyrovTRykkbMxR+WdKyqfldK4fnmRFdjT7r9XXSnz7QvjoOMJlkKuAZ4/xVJue/uFZjL2aB7oog6+LeMGHQL2EwERfELPT52hkFQZH2d+5xDPIh/AWsYPdxK+uBz18wj8z+rYrw7EHgVz1xS6s5QxODx56+cjdRYZQuJu6r1WKvrNs6oMJnE9JH2vJelAGcdZ/fray8abqYoUjSY6sKJ2d0YyAPfyz56eoEoKagl5aDJDT4sasOdEzTItO1udJOp2kf29pupojAIiqa4kYBKzy2ZYh7OH72iNSMAGs3t7zwf4Y7o5a6bXkc7VmBxPPdh9SZbRMS2YUbn23sm9tT+jGONYwD15dlKZdTteFt+8vcWoZcXl3Umkf2ReJ2Nf0FRUFNpwa2Bw4zF5+v2VjgPlrlypl7PkjQ62KQByGWoSOr43f/iBntHXPNkOSuk74HxCN31oguudo0mPTrLqqkCqxaBB/0fFJ7tLjRUaQftpc2UZ8wE9ZQJkd2KbEHzh1g4dLVVfiU/pf1LB2zn+0ZBH5rvdIr/t99CkzD7RKpWMyg3nG/+i8T4MEiWPDsEuTTN+4KBg9UnJp/6GGz6wT6JFvr0gfiFa9S4DyJ/OQ9IkTuIKLIhWiwQ+Cs+qF3lP4O2PFxk4vpuiA4Sy9u7pFGbqEg6QJWIFHAdOLRlPjKgpdrAr7+y6dcEoDBxLup74m2nvG9t4fxAp08foN19kQh6MO/j9JPCPhtC1FGp0qxO5gzVWZFiirhZVXxktsEIQGB1W3ZMVGXMLeI8eL1jNmb1zAkvMgxoEkSc4lJWwopiIgpXqKbdlTwFSJnTppvDzgW0ZlTHvzb2Q MmAxvxWj 6iWsT6Ug9HLfoPlZ0zFaAh6g74vjeW1Lm5Dpi2onMaIPzNBgXkqQ8/JgPOHCESqqikbG6mhS0CFQWYveiA6jnP4X+Q/aGyxHT2xZ4K5taaQGjpOS+dLhMugqgUK5VyDei0LDwmPtza6NFMipe8pUBx031Ge3wRRfJvUmpxKM4sv6uaUHb02PRcXhSvS1XoKNNLoYswGX1aG7kda2dGNkBZSQ7ZUHvsI5rTZNr81PYcxkd8rFHSgdl2AFAd/wEd08ZIZ2MwvXXHtt2A3WQctp0jA6d6wlNnhD3Aj5hgj/q9FeDBTCrr/F19mT0YuAgE3FP2Mmb1AQpcIH2T8lyqE1Jo1D6UVJlGF57qiVqDW6z4lKhUIROIwpJcQGxJ/J/vzNRQLuhZKXSDHevUraSnjZ4+7kQ9RBjA2FjGCxCI44k9Nv9btzpC+3VFFifA4K2Sj94R0DkZlEq88hobHwWeYBHE2qVCvdBG2FL6FV4SSCKsOfsNnN7v2LppLsbF0DRLZ7M4EVbVDND1xrhbs+rVeh5YEX/pA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When transmitting data, call down into TCP using a single sendmsg with MSG_SPLICE_PAGES to indicate that content should be spliced rather than performing several sendmsg and sendpage calls to transmit header, data pages and trailer. To make this work, the data is assembled in a bio_vec array and attached to a BVEC-type iterator. The bio_vec array has two extra slots before the first for headers and one after the last for a trailer. The headers and trailer are copied into memory acquired from zcopy_alloc() which just breaks a page up into small pieces that can be freed with put_page(). Signed-off-by: David Howells cc: Trond Myklebust cc: Anna Schumaker cc: Chuck Lever cc: Jeff Layton cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-nfs@vger.kernel.org cc: netdev@vger.kernel.org --- net/sunrpc/svcsock.c | 70 ++++++++++++-------------------------------- net/sunrpc/xdr.c | 24 ++++++++++++--- 2 files changed, 38 insertions(+), 56 deletions(-) diff --git a/net/sunrpc/svcsock.c b/net/sunrpc/svcsock.c index 03a4f5615086..1fa41ddbc40e 100644 --- a/net/sunrpc/svcsock.c +++ b/net/sunrpc/svcsock.c @@ -36,6 +36,7 @@ #include #include #include +#include #include #include #include @@ -1060,16 +1061,8 @@ static int svc_tcp_recvfrom(struct svc_rqst *rqstp) return 0; /* record not complete */ } -static int svc_tcp_send_kvec(struct socket *sock, const struct kvec *vec, - int flags) -{ - return kernel_sendpage(sock, virt_to_page(vec->iov_base), - offset_in_page(vec->iov_base), - vec->iov_len, flags); -} - /* - * kernel_sendpage() is used exclusively to reduce the number of + * MSG_SPLICE_PAGES is used exclusively to reduce the number of * copy operations in this path. Therefore the caller must ensure * that the pages backing @xdr are unchanging. * @@ -1081,65 +1074,38 @@ static int svc_tcp_sendmsg(struct socket *sock, struct xdr_buf *xdr, { const struct kvec *head = xdr->head; const struct kvec *tail = xdr->tail; - struct kvec rm = { - .iov_base = &marker, - .iov_len = sizeof(marker), - }; struct msghdr msg = { - .msg_flags = 0, + .msg_flags = MSG_SPLICE_PAGES, }; - int ret; + int ret, n = xdr_buf_pagecount(xdr), size; *sentp = 0; ret = xdr_alloc_bvec(xdr, GFP_KERNEL); if (ret < 0) return ret; - ret = kernel_sendmsg(sock, &msg, &rm, 1, rm.iov_len); + ret = zcopy_memdup(sizeof(marker), &marker, &xdr->bvec[-2], GFP_KERNEL); if (ret < 0) return ret; - *sentp += ret; - if (ret != rm.iov_len) - return -EAGAIN; - ret = svc_tcp_send_kvec(sock, head, 0); + ret = zcopy_memdup(head->iov_len, head->iov_base, &xdr->bvec[-1], GFP_KERNEL); if (ret < 0) return ret; - *sentp += ret; - if (ret != head->iov_len) - goto out; - if (xdr->page_len) { - unsigned int offset, len, remaining; - struct bio_vec *bvec; - - bvec = xdr->bvec + (xdr->page_base >> PAGE_SHIFT); - offset = offset_in_page(xdr->page_base); - remaining = xdr->page_len; - while (remaining > 0) { - len = min(remaining, bvec->bv_len - offset); - ret = kernel_sendpage(sock, bvec->bv_page, - bvec->bv_offset + offset, - len, 0); - if (ret < 0) - return ret; - *sentp += ret; - if (ret != len) - goto out; - remaining -= len; - offset = 0; - bvec++; - } - } + ret = zcopy_memdup(tail->iov_len, tail->iov_base, &xdr->bvec[n], GFP_KERNEL); + if (ret < 0) + return ret; - if (tail->iov_len) { - ret = svc_tcp_send_kvec(sock, tail, 0); - if (ret < 0) - return ret; - *sentp += ret; - } + size = sizeof(marker) + head->iov_len + xdr->page_len + tail->iov_len; + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, xdr->bvec - 2, n + 3, size); -out: + ret = sock_sendmsg(sock, &msg); + if (ret < 0) + return ret; + if (ret > 0) + *sentp = ret; + if (ret != size) + return -EAGAIN; return 0; } diff --git a/net/sunrpc/xdr.c b/net/sunrpc/xdr.c index 36835b2f5446..6dff0b4f17b8 100644 --- a/net/sunrpc/xdr.c +++ b/net/sunrpc/xdr.c @@ -145,14 +145,19 @@ xdr_alloc_bvec(struct xdr_buf *buf, gfp_t gfp) { size_t i, n = xdr_buf_pagecount(buf); - if (n != 0 && buf->bvec == NULL) { - buf->bvec = kmalloc_array(n, sizeof(buf->bvec[0]), gfp); + if (buf->bvec == NULL) { + /* Allow for two headers and a trailer to be attached */ + buf->bvec = kmalloc_array(n + 3, sizeof(buf->bvec[0]), gfp); if (!buf->bvec) return -ENOMEM; + buf->bvec += 2; + buf->bvec[-2].bv_page = NULL; + buf->bvec[-1].bv_page = NULL; for (i = 0; i < n; i++) { bvec_set_page(&buf->bvec[i], buf->pages[i], PAGE_SIZE, 0); } + buf->bvec[n].bv_page = NULL; } return 0; } @@ -160,8 +165,19 @@ xdr_alloc_bvec(struct xdr_buf *buf, gfp_t gfp) void xdr_free_bvec(struct xdr_buf *buf) { - kfree(buf->bvec); - buf->bvec = NULL; + if (buf->bvec) { + size_t n = xdr_buf_pagecount(buf); + + if (buf->bvec[-2].bv_page) + put_page(buf->bvec[-2].bv_page); + if (buf->bvec[-1].bv_page) + put_page(buf->bvec[-1].bv_page); + if (buf->bvec[n].bv_page) + put_page(buf->bvec[n].bv_page); + buf->bvec -= 2; + kfree(buf->bvec); + buf->bvec = NULL; + } } /**