From patchwork Tue Jun 20 14:53:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13286010 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57E0AEB64D7 for ; Tue, 20 Jun 2023 14:56:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DDF6D8D0005; Tue, 20 Jun 2023 10:56:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D68A08D0001; Tue, 20 Jun 2023 10:56:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C303A8D0005; Tue, 20 Jun 2023 10:56:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id AEAC48D0001 for ; Tue, 20 Jun 2023 10:56:39 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 54B7F1A116B for ; Tue, 20 Jun 2023 14:56:39 +0000 (UTC) X-FDA: 80923427718.21.D8DA0EC Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf14.hostedemail.com (Postfix) with ESMTP id 2F346100004 for ; Tue, 20 Jun 2023 14:56:36 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=K11U+9Mq; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf14.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687272997; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Fr8R9cUCocDj4LYXtGmeXkCgA5OgO4hqxYPqdNjfTX8=; b=oRfPQReHgIszVbnfFvNYfBVaWsYvxIN0x+yvPWnWM+j/yA9AQZVqNylKQX3p4h0CNPQZHz YkdiMJhWQZtCbJam6kDpZl84k7JNxWNBMySTd/LzTIl+693oQDLIZ+zk75Eux77AuumGFb JU751es4d7/KekrQbQ0Piec4DdPwizA= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=K11U+9Mq; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf14.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687272997; a=rsa-sha256; cv=none; b=iBmznTnisgOy9FRgNuP89SZ1FQGEZPdxkK85Fj3pQ0Q6hr1xjZvKJrCwDNMb85bXQHzh28 ZZ/gNhQP/iV38cmNZ1txj/6CArRb6FcmfWOdgR6uN3kJCWnfqBKyZpHp69crqD7m97btyA 5tw3fha7+FJpjKg5mLJ/qEhJ6RdrHKQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687272996; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Fr8R9cUCocDj4LYXtGmeXkCgA5OgO4hqxYPqdNjfTX8=; b=K11U+9MqYcMS/Ny2dYdvbS9vqB/ZmJFQs3131VgiGMKgXUGuyOHaCSa7u2ZpdVDmB1KfUJ YwWTedBB5V1nYxxncvx36o17ft+HmoDeRRX/LXvniPbOWoWktyEZm0LMYlYUwi+5fvXta9 ByesKlN3mzAXlH5oux2oSnOIK4dI17g= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-264-MX17dbQIPO-VqzbjHhAp6A-1; Tue, 20 Jun 2023 10:56:30 -0400 X-MC-Unique: MX17dbQIPO-VqzbjHhAp6A-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 767F18D142C; Tue, 20 Jun 2023 14:54:20 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.4]) by smtp.corp.redhat.com (Postfix) with ESMTP id B6327C1ED97; Tue, 20 Jun 2023 14:54:18 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , Alexander Duyck , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Santosh Shilimkar , linux-rdma@vger.kernel.org, rds-devel@oss.oracle.com Subject: [PATCH net-next v3 08/18] rds: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage Date: Tue, 20 Jun 2023 15:53:27 +0100 Message-ID: <20230620145338.1300897-9-dhowells@redhat.com> In-Reply-To: <20230620145338.1300897-1-dhowells@redhat.com> References: <20230620145338.1300897-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Rspamd-Queue-Id: 2F346100004 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: u9rjbwf6b7zfc8d951ce46sk58rw4gfu X-HE-Tag: 1687272996-82908 X-HE-Meta: U2FsdGVkX1/kdL2xsj0I94t5kRJ1llmI8ojZUC8IOF9ha61cw3Ulb754j5YTrHvrnTh/BBUvsMtep7WgqCpNbQK9niKHgJSNCw5e5nh4OCL0HnzhLV8f3loJCmaQR6QcPvg5tjsHG5Jz+Vwh316CRAB2So4eU6dXSxJJMlMcmcenKd+bat1qyGpkenn7qGUMK41z8A7L2xCRNTM8Z84RQ9fZPI+ziJLKUjFO2lXxWfIKbbvnYAzEFictM0X9U9mhnhhMizJ6oAzXMB5Asguh4sV4QVgx0smc9lByiWeGrFnqz7EGIQX/g1SEkrtpOiiJavxxCtphalYIq2f7pOU8Uz3jF1rjgsirOmYmHAItVSfqj8pZ+n3TLK4pqBQdZhQrXpaaSnAYIybgM2MdA9z0MgKU8HUYwa+UYvtv5cNvmZQXeJxgKbrt88d9QQKjLKvu4Um45Kr33CxM0Cm1JOQSypCiC+dbh8wGb619ixFUOULHkIBuOdanIKts23MZfGz21iDjg73L4vbl7/2JVyCPYn9lAivA1QekwxTzzS9DTpXd1RuyI+O5q5TTJkgV6fddAag4B3CDdi7jUyOQjXuBgFca5sXg2pWNQ6QQRDK+rSdtWtOd9EWMyUrfAO/CWz38TbtAfRnWc7s3RCY7Loxt1zt0DunfGlEmEKJZiqGpD+TCYF6PVURBiVN7x1Fn2XDg10W7/MYOICs07xkVeH6pFo27d7u4ZtTCFy3vJ0eD2jJYA7moDVSJDIsqwn0BsrbMF0rtiqOQuuro4ssYi8uSKUnJI6PyiPB4dmd+KP7r3AVp+1CPIxaEmbJOTjnVLXYTPjtG5Hy/KjlKnVnfnaGNuxJrl/HVY83Eax14koVkZFLO2hvPI3SzgPdP4JCz8AgIVFzGIsy7DNBmGePby9mRXZPZyv8RvYTpoau3ORqATkPCOw0mPiGq6XIUS8t05hEXVS4St1Uqi8RkC+othEw XOHGTkyU KMcJG92GdbA2AuRmMTQVtv3cCiA8dkGRMzkzdr80bmM8+SeuGUBTmRleoZ/a8xF+o4mtKtF0NPVqJDIgo8rm6s9nA2SmVDSOjmomtBBaxzVdI36j8JxUukZkmdy6bXrXPe2PfF5wpx6foxsL44Zel9EpkidHITjjrM0SbplXRUx3nmWBgXVjJq8yxOK3Wfyg0fJkMKxX5w+rmFj2TfyU0HAXOd93HY8h0JMDP5fIIoQa7HJBAbZhsBKm0TMk7uBhEgaj/u+j/oh5Va7kPsl//WgTJ3p8oHx8lhtdc+dSCMWLo9eKCF2Sxq9a9lATDmk6zSQwQ2F2SquZQDVAgPO/MVVZCpQkS0C5oZMmMd8pXRernUZmn+rlCbicuQSdvhKy2nQi4PxlKA+q/E9yfvrRumcUniQApwD/G4mRQy2pidhzRZfxww/iT9aqJ5m/YpaCmbE+iezn8WGEjW7OTc8cqIMqBy26oONij0z1kqfsEXvGxSDpZ04YLzncBGWZg5dOs6wwoNglFygob2sc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When transmitting data, call down into TCP using a single sendmsg with MSG_SPLICE_PAGES to indicate that content should be spliced rather than performing several sendmsg and sendpage calls to transmit header and data pages. To make this work, the data is assembled in a bio_vec array and attached to a BVEC-type iterator. The header are copied into memory acquired from zcopy_alloc() which just breaks a page up into small pieces that can be freed with put_page(). Signed-off-by: David Howells cc: Santosh Shilimkar cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-rdma@vger.kernel.org cc: rds-devel@oss.oracle.com cc: netdev@vger.kernel.org --- net/rds/tcp_send.c | 74 +++++++++++++++++----------------------------- 1 file changed, 27 insertions(+), 47 deletions(-) diff --git a/net/rds/tcp_send.c b/net/rds/tcp_send.c index 8c4d1d6e9249..550390d5ff2b 100644 --- a/net/rds/tcp_send.c +++ b/net/rds/tcp_send.c @@ -52,29 +52,23 @@ void rds_tcp_xmit_path_complete(struct rds_conn_path *cp) tcp_sock_set_cork(tc->t_sock->sk, false); } -/* the core send_sem serializes this with other xmit and shutdown */ -static int rds_tcp_sendmsg(struct socket *sock, void *data, unsigned int len) -{ - struct kvec vec = { - .iov_base = data, - .iov_len = len, - }; - struct msghdr msg = { - .msg_flags = MSG_DONTWAIT | MSG_NOSIGNAL, - }; - - return kernel_sendmsg(sock, &msg, &vec, 1, vec.iov_len); -} - /* the core send_sem serializes this with other xmit and shutdown */ int rds_tcp_xmit(struct rds_connection *conn, struct rds_message *rm, unsigned int hdr_off, unsigned int sg, unsigned int off) { struct rds_conn_path *cp = rm->m_inc.i_conn_path; struct rds_tcp_connection *tc = cp->cp_transport_data; + struct msghdr msg = { + .msg_flags = MSG_SPLICE_PAGES | MSG_DONTWAIT | MSG_NOSIGNAL, + }; + struct bio_vec *bvec; + unsigned int i, size = 0, ix = 0; int done = 0; - int ret = 0; - int more; + int ret = -ENOMEM; + + bvec = kmalloc_array(1 + sg, sizeof(struct bio_vec), GFP_KERNEL); + if (!bvec) + goto out; if (hdr_off == 0) { /* @@ -101,41 +95,26 @@ int rds_tcp_xmit(struct rds_connection *conn, struct rds_message *rm, /* see rds_tcp_write_space() */ set_bit(SOCK_NOSPACE, &tc->t_sock->sk->sk_socket->flags); - ret = rds_tcp_sendmsg(tc->t_sock, - (void *)&rm->m_inc.i_hdr + hdr_off, - sizeof(rm->m_inc.i_hdr) - hdr_off); - if (ret < 0) - goto out; - done += ret; - if (hdr_off + done != sizeof(struct rds_header)) - goto out; + bvec_set_virt(&bvec[ix], (void *)&rm->m_inc.i_hdr + hdr_off, + sizeof(rm->m_inc.i_hdr) - hdr_off); + size += bvec[ix].bv_len; + ix++; } - more = rm->data.op_nents > 1 ? (MSG_MORE | MSG_SENDPAGE_NOTLAST) : 0; - while (sg < rm->data.op_nents) { - int flags = MSG_DONTWAIT | MSG_NOSIGNAL | more; - - ret = tc->t_sock->ops->sendpage(tc->t_sock, - sg_page(&rm->data.op_sg[sg]), - rm->data.op_sg[sg].offset + off, - rm->data.op_sg[sg].length - off, - flags); - rdsdebug("tcp sendpage %p:%u:%u ret %d\n", (void *)sg_page(&rm->data.op_sg[sg]), - rm->data.op_sg[sg].offset + off, rm->data.op_sg[sg].length - off, - ret); - if (ret <= 0) - break; - - off += ret; - done += ret; - if (off == rm->data.op_sg[sg].length) { - off = 0; - sg++; - } - if (sg == rm->data.op_nents - 1) - more = 0; + for (i = sg; i < rm->data.op_nents; i++) { + bvec_set_page(&bvec[ix], + sg_page(&rm->data.op_sg[i]), + rm->data.op_sg[i].length - off, + rm->data.op_sg[i].offset + off); + off = 0; + size += bvec[ix].bv_len; + ix++; } + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, bvec, ix, size); + ret = sock_sendmsg(tc->t_sock, &msg); + rdsdebug("tcp sendmsg-splice %u,%u ret %d\n", ix, size, ret); + out: if (ret <= 0) { /* write_space will hit after EAGAIN, all else fatal */ @@ -158,6 +137,7 @@ int rds_tcp_xmit(struct rds_connection *conn, struct rds_message *rm, } if (done == 0) done = ret; + kfree(bvec); return done; }