From patchwork Thu Mar 16 15:26:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13177827 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37454C7618D for ; Thu, 16 Mar 2023 15:27:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C72C4900003; Thu, 16 Mar 2023 11:27:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C2388900002; Thu, 16 Mar 2023 11:27:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A9D13900003; Thu, 16 Mar 2023 11:27:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 99906900002 for ; Thu, 16 Mar 2023 11:27:43 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 69C2D1A07C0 for ; Thu, 16 Mar 2023 15:27:43 +0000 (UTC) X-FDA: 80575141206.09.A684ED3 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf26.hostedemail.com (Postfix) with ESMTP id 9E37914001B for ; Thu, 16 Mar 2023 15:27:40 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=etjMSyAa; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf26.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678980460; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CNHQplOZd5HfwcKa/y8evYEBVb89JfsyQlI6+74E8fk=; b=dh4RvOEG2MLUgXu1nLjFlLBJnQDlHYVCgg6PCrv+nQfKYLyL0m7EgcVAGXdl57geplrSG/ j5iQotr+cqGEU+xRiIfGCg4+Kr6xPWbD4gejtXG5dyLJcI3EFiU4yqjFO7KWpUOaWo5i01 HfQqsJrY4sRUzG8Eew2ZESS87+u/fqo= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=etjMSyAa; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf26.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678980460; a=rsa-sha256; cv=none; b=LFkAdjHKV6ubTQMWLaCxfbF1AFeLX6lFLxvTXkLrNWunClEBOsPc1OsNiOccRz8cv8vFlg tgIx39B9JnCfDNQ9Zhej1AbVqdyeZTvSmS4WivzeWMqRF8WsQvn338nMofOp0WpAteNGwB t/hAUxDz3gyBm2ticP+BzdDWuDOCIs8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678980459; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CNHQplOZd5HfwcKa/y8evYEBVb89JfsyQlI6+74E8fk=; b=etjMSyAaUMrXETAwP6TjkzO4Aq1Ju9tGjaxVgQ5nKvw68CX54Iw1F+JLz+IrK8/iOmUfEv IPrDz4p2ZBaYRZqyhdfW9oFYB1yHbsuAHF37Jlu4xEbeUvVEV8P0XlcRBMgWnHvueFy946 4Ocjn4o3Pt+zn9Lyt9IvrleG7Jpl+h8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-321-2P0S9tmzOBuO6RSNRr9oBQ-1; Thu, 16 Mar 2023 11:27:36 -0400 X-MC-Unique: 2P0S9tmzOBuO6RSNRr9oBQ-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E5750887402; Thu, 16 Mar 2023 15:27:34 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9F26035453; Thu, 16 Mar 2023 15:27:32 +0000 (UTC) From: David Howells To: Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: David Howells , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Trond Myklebust , Anna Schumaker , Chuck Lever , linux-nfs@vger.kernel.org Subject: [RFC PATCH 27/28] sunrpc: Use sendmsg(MSG_SPLICE_PAGES) rather then sendpage Date: Thu, 16 Mar 2023 15:26:17 +0000 Message-Id: <20230316152618.711970-28-dhowells@redhat.com> In-Reply-To: <20230316152618.711970-1-dhowells@redhat.com> References: <20230316152618.711970-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 9E37914001B X-Stat-Signature: wauyb7kyw6g6gw4bb9jqnxdb893kpoz6 X-HE-Tag: 1678980460-348316 X-HE-Meta: U2FsdGVkX19RI3Oju24acgTEEGr+FvTQW+n9+/lW3gvN5SDB0PaWK78C/LThbcMsVPRWB+eN3KWliowIKsmDZ2KbkbuKlzrKgRvBib/lMldtmYQjZe3wcDqgKa/l/nV1v+7hXNT59GAyrovTRykkbMxR+WdKyqfldK4fnmRFdjT7r9XXSnz7QvjoOMJlkKuAZ4/xVJue/uFZjL2aB7oog6+LeMGHQL2EwERfELPT52hkFQZH2d+5xDPIh/AWsYPdxK+uBz18wj8z+rYrw7EHgVz1xS6s5QxODx56+cjdRYZQuJu6r1WKvrNs6oMJnE9JH2vJelAGcdZ/fray8abqYoUjSY6sKJ2d0YyAPfyz56eoEoKagl5aDJDT4sasOdEzTItO1udJOp2kf29pupojAIiqa4kYBKzy2ZYh7OH72iNSMAGs3t7zwf4Y7o5a6bXkc7VmBxPPdh9SZbRMS2YUbn23sm9tT+jGONYwD15dlKZdTteFt+8vcWoZcXl3Umkf2ReJ2Nf0FRUFNpwa2Bw4zF5+v2VjgPlrlypl7PkjQ62KQByGWoSOr43f/iBntHXPNkOSuk74HxCN31oguudo0mPTrLqqkCqxaBB/0fFJ7tLjRUaQftpc2UZ8wE9ZQJkd2KbEHzh1g4dLVVfiU/pf1LB2zn+0ZBH5rvdIr/t99CkzD7RKpWMyg3nG/+i8T4MEiWPDsEuTTN+4KBg9UnJp/6GGz6wT6JFvr0gfiFa9S4DyJ/OQ9IkTuIKLIhWiwQ+Cs+qF3lP4O2PFxk4vpuiA4Sy9u7pFGbqEg6QJWIFHAdOLRlPjKgpdrAr7+y6dcEoDBxLup74m2nvG9t4fxAp08foN19kQh6MO/j9JPCPhtC1FGp0qxO5gzVWZFiirhZVXxktsEIQGB1W3ZMVGXMLeI8eL1jNmb1zAkvMgxoEkSc4lJWwopiIgpXqKbdlTwFSJnTppvDzgW0ZlTHvzb2Q MmAxvxWj 6iWsT6Ug9HLfoPlZ0zFaAh6g74vjeW1Lm5Dpi2onMaIPzNBgXkqQ8/JgPOHCESqqikbG6mhS0CFQWYveiA6jnP4X+Q/aGyxHT2xZ4K5taaQGjpOS+dLhMugqgUK5VyDei0LDwmPtza6NFMipe8pUBx031Ge3wRRfJvUmpxKM4sv6uaUHb02PRcXhSvS1XoKNNLoYswGX1aG7kda2dGNkBZSQ7ZUHvsI5rTZNr81PYcxkd8rFHSgdl2AFAd/wEd08ZIZ2MwvXXHtt2A3WQctp0jA6d6wlNnhD3Aj5hgj/q9FeDBTCrr/F19mT0YuAgE3FP2Mmb1AQpcIH2T8lyqE1Jo1D6UVJlGF57qiVqDW6z4lKhUIROIwpJcQGxJ/J/vzNRQLuhZKXSDHevUraSnjZ4+7kQ9RBjA2FjGCxCI44k9Nv9btzpC+3VFFifA4K2Sj94R0DkZlEq88hobHwWeYBHE2qVCvdBG2FL6FV4SSCKsOfsNnN7v2LppLsbF0DRLZ7M4EVbVDND1xrhbs+rVeh5YEX/pA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When transmitting data, call down into TCP using a single sendmsg with MSG_SPLICE_PAGES to indicate that content should be spliced rather than performing several sendmsg and sendpage calls to transmit header, data pages and trailer. To make this work, the data is assembled in a bio_vec array and attached to a BVEC-type iterator. The bio_vec array has two extra slots before the first for headers and one after the last for a trailer. The headers and trailer are copied into memory acquired from zcopy_alloc() which just breaks a page up into small pieces that can be freed with put_page(). Signed-off-by: David Howells cc: Trond Myklebust cc: Anna Schumaker cc: Chuck Lever cc: Jeff Layton cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-nfs@vger.kernel.org cc: netdev@vger.kernel.org --- net/sunrpc/svcsock.c | 70 ++++++++++++-------------------------------- net/sunrpc/xdr.c | 24 ++++++++++++--- 2 files changed, 38 insertions(+), 56 deletions(-) diff --git a/net/sunrpc/svcsock.c b/net/sunrpc/svcsock.c index 03a4f5615086..1fa41ddbc40e 100644 --- a/net/sunrpc/svcsock.c +++ b/net/sunrpc/svcsock.c @@ -36,6 +36,7 @@ #include #include #include +#include #include #include #include @@ -1060,16 +1061,8 @@ static int svc_tcp_recvfrom(struct svc_rqst *rqstp) return 0; /* record not complete */ } -static int svc_tcp_send_kvec(struct socket *sock, const struct kvec *vec, - int flags) -{ - return kernel_sendpage(sock, virt_to_page(vec->iov_base), - offset_in_page(vec->iov_base), - vec->iov_len, flags); -} - /* - * kernel_sendpage() is used exclusively to reduce the number of + * MSG_SPLICE_PAGES is used exclusively to reduce the number of * copy operations in this path. Therefore the caller must ensure * that the pages backing @xdr are unchanging. * @@ -1081,65 +1074,38 @@ static int svc_tcp_sendmsg(struct socket *sock, struct xdr_buf *xdr, { const struct kvec *head = xdr->head; const struct kvec *tail = xdr->tail; - struct kvec rm = { - .iov_base = &marker, - .iov_len = sizeof(marker), - }; struct msghdr msg = { - .msg_flags = 0, + .msg_flags = MSG_SPLICE_PAGES, }; - int ret; + int ret, n = xdr_buf_pagecount(xdr), size; *sentp = 0; ret = xdr_alloc_bvec(xdr, GFP_KERNEL); if (ret < 0) return ret; - ret = kernel_sendmsg(sock, &msg, &rm, 1, rm.iov_len); + ret = zcopy_memdup(sizeof(marker), &marker, &xdr->bvec[-2], GFP_KERNEL); if (ret < 0) return ret; - *sentp += ret; - if (ret != rm.iov_len) - return -EAGAIN; - ret = svc_tcp_send_kvec(sock, head, 0); + ret = zcopy_memdup(head->iov_len, head->iov_base, &xdr->bvec[-1], GFP_KERNEL); if (ret < 0) return ret; - *sentp += ret; - if (ret != head->iov_len) - goto out; - if (xdr->page_len) { - unsigned int offset, len, remaining; - struct bio_vec *bvec; - - bvec = xdr->bvec + (xdr->page_base >> PAGE_SHIFT); - offset = offset_in_page(xdr->page_base); - remaining = xdr->page_len; - while (remaining > 0) { - len = min(remaining, bvec->bv_len - offset); - ret = kernel_sendpage(sock, bvec->bv_page, - bvec->bv_offset + offset, - len, 0); - if (ret < 0) - return ret; - *sentp += ret; - if (ret != len) - goto out; - remaining -= len; - offset = 0; - bvec++; - } - } + ret = zcopy_memdup(tail->iov_len, tail->iov_base, &xdr->bvec[n], GFP_KERNEL); + if (ret < 0) + return ret; - if (tail->iov_len) { - ret = svc_tcp_send_kvec(sock, tail, 0); - if (ret < 0) - return ret; - *sentp += ret; - } + size = sizeof(marker) + head->iov_len + xdr->page_len + tail->iov_len; + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, xdr->bvec - 2, n + 3, size); -out: + ret = sock_sendmsg(sock, &msg); + if (ret < 0) + return ret; + if (ret > 0) + *sentp = ret; + if (ret != size) + return -EAGAIN; return 0; } diff --git a/net/sunrpc/xdr.c b/net/sunrpc/xdr.c index 36835b2f5446..6dff0b4f17b8 100644 --- a/net/sunrpc/xdr.c +++ b/net/sunrpc/xdr.c @@ -145,14 +145,19 @@ xdr_alloc_bvec(struct xdr_buf *buf, gfp_t gfp) { size_t i, n = xdr_buf_pagecount(buf); - if (n != 0 && buf->bvec == NULL) { - buf->bvec = kmalloc_array(n, sizeof(buf->bvec[0]), gfp); + if (buf->bvec == NULL) { + /* Allow for two headers and a trailer to be attached */ + buf->bvec = kmalloc_array(n + 3, sizeof(buf->bvec[0]), gfp); if (!buf->bvec) return -ENOMEM; + buf->bvec += 2; + buf->bvec[-2].bv_page = NULL; + buf->bvec[-1].bv_page = NULL; for (i = 0; i < n; i++) { bvec_set_page(&buf->bvec[i], buf->pages[i], PAGE_SIZE, 0); } + buf->bvec[n].bv_page = NULL; } return 0; } @@ -160,8 +165,19 @@ xdr_alloc_bvec(struct xdr_buf *buf, gfp_t gfp) void xdr_free_bvec(struct xdr_buf *buf) { - kfree(buf->bvec); - buf->bvec = NULL; + if (buf->bvec) { + size_t n = xdr_buf_pagecount(buf); + + if (buf->bvec[-2].bv_page) + put_page(buf->bvec[-2].bv_page); + if (buf->bvec[-1].bv_page) + put_page(buf->bvec[-1].bv_page); + if (buf->bvec[n].bv_page) + put_page(buf->bvec[n].bv_page); + buf->bvec -= 2; + kfree(buf->bvec); + buf->bvec = NULL; + } } /**