From patchwork Fri Mar 7 18:28:53 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 14006946 Received: from mail-ed1-f50.google.com (mail-ed1-f50.google.com [209.85.208.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C478123BCFA for ; Fri, 7 Mar 2025 18:28:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741372108; cv=none; b=U5/feL4o6xL6fj0UZAibQBjwz7JupofeIFUPkGhtgtTZOIl9+EBkoH+HvS2/MeX4rEnLSPAkmtA9HifwKmCmUEu9gChduVarzU2ry53sGwybjNr3nedI2BkEWk+7fiGV4kpGyBW7baMFjO01E84/XBjJfAIGtf9roMb6fmVzkKU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741372108; c=relaxed/simple; bh=8g9zRX6sHAB3sw6JN99B9kMYnKKz/dlCYEj7U1/xouE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=jvK8qfFrBz64+nBlzO1qq2tSZWgONUeRnWQWtQcSkzmpCP7A38Oj3DU/MgOrb5bOvOFrpzUfl7pCHKr85Qik+DpI73lANd6osv6bCZrMGdLOLPuAQCHd3AR49ILR4bTR2LmdcA4oScIYaK897FfrgiltPcXzQ9HKjySehxqQl0k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=BcsdT38e; arc=none smtp.client-ip=209.85.208.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="BcsdT38e" Received: by mail-ed1-f50.google.com with SMTP id 4fb4d7f45d1cf-5e058ca6806so4214781a12.3 for ; Fri, 07 Mar 2025 10:28:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741372105; x=1741976905; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8WP4++kaTGZ4gQVnE5wRyc3mctW9Uz2qpENJImUi5S4=; b=BcsdT38eJIqm9Ik41HF9aXCeI9rfBtzYgE15WNl9ehtnnF2sOpuC+yaindBxbdBtzb LGe4QjkYLoapw3dPNgQcD8RbF/60NCh96JZ0dowsSpp4ntw/NKCAvklgSNn44PxUVuru blDQpR109jf8DaEO6YizsnUTW7mpPS5+Ck67+xKy1HO2vEuDEb4a8DWDXUroIrPKGJKF ntEmiJMV4iOUB+eyWglVGHtefhFnsyZbmIqvV0eqtK9JJdXAgFy/fL9zLnFn/10yOoL7 JnQwVmV+GBRGmtCnqEVs8F5b3zAjEwe4IqsSrGSjBZ3csb/G2CTcixalyGyjYSV/np64 JWUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741372105; x=1741976905; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8WP4++kaTGZ4gQVnE5wRyc3mctW9Uz2qpENJImUi5S4=; b=guWGrAnq5JS8vHz23uNBSCYo9HgnNAsoi30iTSJWf3dTE8nPO0oSoiAZ8GqhgVPxUP NaUZpHl5c0jJag37f9A+2VCs8K6a+ZCAof5j48A6JgznSnFyd/7gY7l23AP453TA7tPu aIFWNH2T0Clht4zBT2tEzcXhPA8AUly/sl/t6h0T5Os0mZEIWMDz2R+TqhhCFfUDrN+T YEiWNyQcUfJhjRzaxBh8hf2JJgCYdjp6JKM0S+wt/LEppAzpce5NR5kPCD+aNub2YwFD va7/cVqBdcTOmxp6PAbaClu4q5pDfWspkIDeqzCvfQ2trwVfXAlagNB2jaAwQN5sVf8n 8C3Q== X-Gm-Message-State: AOJu0YwS/vv9NyMBRzjTN9nJ2R6x6QZjnDFDnG/Yxa9fBhFUAiwLez4M IBI+aBNFunb+Cx8LgZRzTKI4run0tc/YjAzNqA6TCdpE2i0JjLJnqgFn+g== X-Gm-Gg: ASbGncs+4hGGFbijWrqXV2TYDDcEUhstAUnyx2lQpZEl6lbJqHK1kD3wn6KPgS9F7Ba Z0BDKLktQfECAZ8uihg/x5RMk4sSRE/jIrtLQmOiYKuqyNjuYd5vT6x0hToqsAtCJazPEEgeOzt kNHidf7DD09e99K2EOKGY0w0L6dQtvf6whZR7ATRlAylb8Ki+5FXqFRTza++j6BZvVCHPvkSScj 4+VfqlHMbjFGwruLpRv1gv8CvcAJ72oDj7cDRuJ/ovSwi6elhMXR1IC/UFm2J2+D8D4nuDpe5To YWlUHcFE4pFn7LaLQSIh3MxHGco70LsGukuF+QVm3kOmjIS5DMNkTRnIzQ== X-Google-Smtp-Source: AGHT+IEBFcYyeDRCkTpMTF7olUDzL59N4kF633L43rJW8IL+C3+7sCDWlFmGutSjimkQiZwveR0kuw== X-Received: by 2002:a05:6402:2750:b0:5e5:49af:411d with SMTP id 4fb4d7f45d1cf-5e5e22da050mr6789905a12.17.1741372104359; Fri, 07 Mar 2025 10:28:24 -0800 (PST) Received: from 127.0.0.1localhost ([85.255.232.206]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5e5c74a9315sm2883230a12.46.2025.03.07.10.28.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Mar 2025 10:28:23 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH liburing v2 1/4] Add vectored registered buffer req init helpers Date: Fri, 7 Mar 2025 18:28:53 +0000 Message-ID: <2ca386bcdbfe1518822f372f4462c346a40bc4a3.1741372065.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Signed-off-by: Pavel Begunkov --- src/include/liburing.h | 31 +++++++++++++++++++++++++++++++ src/liburing-ffi.map | 3 +++ 2 files changed, 34 insertions(+) diff --git a/src/include/liburing.h b/src/include/liburing.h index d162d0e6..ae2021b9 100644 --- a/src/include/liburing.h +++ b/src/include/liburing.h @@ -556,6 +556,16 @@ IOURINGINLINE void io_uring_prep_read_fixed(struct io_uring_sqe *sqe, int fd, sqe->buf_index = (__u16) buf_index; } +IOURINGINLINE void io_uring_prep_readv_fixed(struct io_uring_sqe *sqe, int fd, + const struct iovec *iovecs, + unsigned nr_vecs, __u64 offset, + int flags, int buf_index) +{ + io_uring_prep_readv2(sqe, fd, iovecs, nr_vecs, offset, flags); + sqe->opcode = IORING_OP_READV_FIXED; + sqe->buf_index = (__u16)buf_index; +} + IOURINGINLINE void io_uring_prep_writev(struct io_uring_sqe *sqe, int fd, const struct iovec *iovecs, unsigned nr_vecs, __u64 offset) @@ -580,6 +590,16 @@ IOURINGINLINE void io_uring_prep_write_fixed(struct io_uring_sqe *sqe, int fd, sqe->buf_index = (__u16) buf_index; } +IOURINGINLINE void io_uring_prep_writev_fixed(struct io_uring_sqe *sqe, int fd, + const struct iovec *iovecs, + unsigned nr_vecs, __u64 offset, + int flags, int buf_index) +{ + io_uring_prep_writev2(sqe, fd, iovecs, nr_vecs, offset, flags); + sqe->opcode = IORING_OP_WRITEV_FIXED; + sqe->buf_index = (__u16)buf_index; +} + IOURINGINLINE void io_uring_prep_recvmsg(struct io_uring_sqe *sqe, int fd, struct msghdr *msg, unsigned flags) { @@ -964,6 +984,17 @@ IOURINGINLINE void io_uring_prep_sendmsg_zc(struct io_uring_sqe *sqe, int fd, sqe->opcode = IORING_OP_SENDMSG_ZC; } +IOURINGINLINE void io_uring_prep_sendmsg_zc_fixed(struct io_uring_sqe *sqe, + int fd, + const struct msghdr *msg, + unsigned flags, + unsigned buf_index) +{ + io_uring_prep_sendmsg_zc(sqe, fd, msg, flags); + sqe->ioprio |= IORING_RECVSEND_FIXED_BUF; + sqe->buf_index = buf_index; +} + IOURINGINLINE void io_uring_prep_recv(struct io_uring_sqe *sqe, int sockfd, void *buf, size_t len, int flags) { diff --git a/src/liburing-ffi.map b/src/liburing-ffi.map index 294d2abf..75c9e9bd 100644 --- a/src/liburing-ffi.map +++ b/src/liburing-ffi.map @@ -230,4 +230,7 @@ LIBURING_2.10 { global: io_uring_register_ifq; io_uring_prep_epoll_wait; + io_uring_prep_writev_fixed; + io_uring_prep_readv_fixed; + io_uring_prep_sendmsg_zc_fixed; } LIBURING_2.9; From patchwork Fri Mar 7 18:28:54 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 14006947 Received: from mail-ed1-f48.google.com (mail-ed1-f48.google.com [209.85.208.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 47FBC19ABC2 for ; Fri, 7 Mar 2025 18:28:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741372111; cv=none; b=qCh57+6MBN6wtWAsjTXjpv91Y29MAxuVFGAOhHdbhdB9W+wP+2K2XgffMP0SN1VYxZWVOUh9sMm0cHFYY3Wt90oT/M0la9Ng5QG3Oe57yRzPRAOqEO5nC9W4D1aICNJjYciipsc3FyI+Ev154Zjs+A9CnyE3UsPiM8Qg7r+gKtE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741372111; c=relaxed/simple; bh=ekUvtFH/SfQXgIHgfIYiXK40M2zqG/dda6QIuokadQY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=XqMUsBPxphG7Pgx8u/4IgvAfUX+6yXcnO3UF7+Ro1Yen8aBKnRgKv7LTk3+fSuI8N17qOfcEvSP82ZIRA7Zd4RUVlKxy9RbOfE/H2kB4jAwE3d1pa3zJx38nzmFgp6skdzipSDl1AQS+OxmD5xbD6Re8OU+Q32HwENzXPBFSAr4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=nf23HyAH; arc=none smtp.client-ip=209.85.208.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="nf23HyAH" Received: by mail-ed1-f48.google.com with SMTP id 4fb4d7f45d1cf-5e60cfef9cfso878695a12.2 for ; Fri, 07 Mar 2025 10:28:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741372108; x=1741976908; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rwjHxuGNfHhbmaESXrubOfAJtb+l3BJVFh0Fi9idhfo=; b=nf23HyAH7mKrLoNXwhYWnrsJpubH5HklOpN/T0LKsmpvh1Lj62jG9/9Q1ZWRyR7N+r sGXRw+9Kb96X2KXY50qcfKMsY8zS3Rcj9rpIoAMtp84YlhPss9r1LpYlYLKegKTlgtwh HCHuuHeRgnlMod4G6ubbzHJxKR0KUQaIhvFjX+00ChsD6OplpOzwZdyz0hldxNaQykes RnTJ/2I4uEofqMK2N01zM0fH1KrV8xz9cSjCfxMeP0u8Xe6w6oynzN/gZmQ6TCABbbZD WAMwyaLSj6El6F6y/B4YnLRGqQbBJeagTUbvBZ+AiQMgjgDOfZprnRqpaF3U4wlaBMrS 3mnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741372108; x=1741976908; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rwjHxuGNfHhbmaESXrubOfAJtb+l3BJVFh0Fi9idhfo=; b=nCdwtZLjF4sc3rDwm/7bnuWZmZl+Yvoaw7l2WnFodyTnJCTSRHV2xbruKtBwM3uWnn OCDLt04iOxegYQ1F8ZbhFYfPMO8aigoOC11hEAA+UrcP/IfetCNKSmB+MqY590o/Svfs kjwDk49g/OeOZ1FyWoQpqSQ3Q2uPviRbvlnqubqL6BRjuyUovD8xr0VxmZWMntYGe5I/ rh8gKdKcRhgjem7x9QQbLlop4PPvq0IfxvSROLEuEmyxwahHI/PE7qjdqVGBNXJn/Csz ciIYvm4sRtTF0wRZ07BdtXBp7qoVz7xz7OKSXfEZ3oSEUZ/H+c/MaHQtB1Lp+XNZdhA3 fN0w== X-Gm-Message-State: AOJu0YzvzOiIx5QmwdCl4m1CYD7VQFVSGAxDDfdVgYH3o3fIHCM+xgsf hlJ+GteBIg4ffs0cWJuyii8iQc9vdRssEExchL89tK/66r66xkXfPJe42A== X-Gm-Gg: ASbGncuHNalEFdbzb+pptTGF7E0LQc286/Oiz0yPvO3UgjSKl/ColQ6d/oRCzKfnbxs 9Qtgq0JYuKARvcoVO526IIIvdRjqxvt2y7XsWzGYzQWBv4yrNkNhPHgB3GDYfdCx8thE71lfzIw 0b4rJ233aV28aeNAF8uqbgBxwX/RQRC6Io6bZtTNG4xjg13DqtSI5niBU0UulLq8bXRFhT+qD9N T29eqsHayiYWnJaPKVlcvKBF4DHvTt33VGvGbO2cDuOW322CdmF6neIysX9XuEt9jxOmTDSFgfB c6FtHOkhqodfwSn3rj7KSYk/Fjq08Jm2gkFNXlRAWsJdBaTfrkkQG0dPzg== X-Google-Smtp-Source: AGHT+IE7ye+INdTsOZPTwhfloDrjww+f07DIiJ7qJ4vC3k60FLRKLcExplaYLKCExgU6ODun/TmzFQ== X-Received: by 2002:a05:6402:1ed3:b0:5e5:9a2b:167a with SMTP id 4fb4d7f45d1cf-5e5e22da458mr5671162a12.17.1741372107714; Fri, 07 Mar 2025 10:28:27 -0800 (PST) Received: from 127.0.0.1localhost ([85.255.232.206]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5e5c74a9315sm2883230a12.46.2025.03.07.10.28.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Mar 2025 10:28:26 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH liburing v2 2/4] test/sendzc: test registered vectored buffers Date: Fri, 7 Mar 2025 18:28:54 +0000 Message-ID: <272e2179217c7ed32d0113411c5d7db652f60f3a.1741372065.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Signed-off-by: Pavel Begunkov --- test/send-zerocopy.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/test/send-zerocopy.c b/test/send-zerocopy.c index c8eafe28..680481a0 100644 --- a/test/send-zerocopy.c +++ b/test/send-zerocopy.c @@ -69,6 +69,7 @@ static size_t page_sz; static char *tx_buffer, *rx_buffer; static struct iovec buffers_iov[__BUF_NR]; +static bool has_regvec; static bool has_sendzc; static bool has_sendmsg; static bool hit_enomem; @@ -96,6 +97,7 @@ static int probe_zc_support(void) has_sendzc = p->ops_len > IORING_OP_SEND_ZC; has_sendmsg = p->ops_len > IORING_OP_SENDMSG_ZC; + has_regvec = p->ops_len > IORING_OP_READV_FIXED; io_uring_queue_exit(&ring); free(p); return 0; @@ -448,6 +450,11 @@ static int do_test_inet_send(struct io_uring *ring, int sock_client, int sock_se else io_uring_prep_sendmsg(sqe, sock_client, &msghdr[i], msg_flags); + if (real_fixed_buf) { + sqe->ioprio |= IORING_RECVSEND_FIXED_BUF; + sqe->buf_index = conf->buf_index; + } + if (!conf->iovec) { io = &iov[i]; iov_len = 1; @@ -619,7 +626,11 @@ static int test_inet_send(struct io_uring *ring) conf.tcp = tcp; regbuf = conf.mix_register || conf.fixed_buf; - if (conf.iovec && (!conf.use_sendmsg || regbuf || conf.cork)) + if (!tcp && conf.long_iovec) + continue; + if (conf.use_sendmsg && regbuf && !has_regvec) + continue; + if (conf.iovec && (!conf.use_sendmsg || conf.cork)) continue; if (!conf.zc) { if (regbuf) @@ -637,7 +648,7 @@ static int test_inet_send(struct io_uring *ring) continue; if (!client_connect && conf.addr == NULL) continue; - if (conf.use_sendmsg && (regbuf || !has_sendmsg)) + if (conf.use_sendmsg && !has_sendmsg) continue; if (msg_zc_set && !conf.zc) continue; From patchwork Fri Mar 7 18:28:55 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 14006948 Received: from mail-ed1-f53.google.com (mail-ed1-f53.google.com [209.85.208.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2574A23BCFA for ; Fri, 7 Mar 2025 18:28:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741372114; cv=none; b=Kg4UyQl8PPReQiXaZ3KLZPGi2iqEZxQRMFpLePunL/ox5zfRLk5H2h4XIu5ZTEAAybpi97DQOx5/QRbCzH1OJd/BpcsP1qK0o5sU9zGeQ0nPEQr5tjAXAr80rqyK7YfHCym7cM7T5sXq5MHm9TLpQUJ7Z3gd/yEXUjHyaBD5944= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741372114; c=relaxed/simple; bh=CdQWfCZybetwuh0QHo4vHCvZ4DVJZbuxlvbLWuDqrUA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qSQeh/d9vawI7GCiezv8U5aqxTfgVY1c/fyLqcQCWhLwGv0q5y8fsUdU7AS3ko0Jl0ry+rmnB+QtzvAId0oNYfmpBqHrVRbaMeIwi7xZvCkP9SAIZRKf2nalJGTFl4KKQKIgRp0z1I5hUh28ZlYZs0Tb2843BhjL7r4Cpxp+j/o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=RkIAEi4h; arc=none smtp.client-ip=209.85.208.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="RkIAEi4h" Received: by mail-ed1-f53.google.com with SMTP id 4fb4d7f45d1cf-5e4ebc78da5so4077118a12.2 for ; Fri, 07 Mar 2025 10:28:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741372110; x=1741976910; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wjAyPpy+7gR3wP8zN7zP6bimQJIBWv2Ey6Arcbs3jQQ=; b=RkIAEi4ha+v0AIpFUNP4CNj9ORkwxPCqfAisUxq4OigeOVejkP2OEm0N6YrZqrwF7d n28buLsgmz8l5/RcEbU6jrBD8UQTBW1WHTWGzhSY+vX+XFtvu00GAeY0frwg+cgth7XA ccTqa2sh/HuBvf1w5LaR5/6Zl7EV6GYlAYTPCPnau1fxRFvyB7vzaCOOl++kHCQdV9GI QKKtxc3Hbs3ox3HDaG2R/UWnDQGhkOinJc5oeS+Gpe3taaaWXCJnTdGavGszk0A04Zv8 yZ3jgvpe5m/rkdsBhllNsBo1nux5HgfIr6AV9V3Ga7JhKv7ZgAWXZjaMb/Nlg+HRwphA 5M0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741372110; x=1741976910; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wjAyPpy+7gR3wP8zN7zP6bimQJIBWv2Ey6Arcbs3jQQ=; b=tR6Y2sor8GDodErDzrNgRnNzFVNcVrW7aDpX6qqUMmG30Leh3kiYN/UhAzS3lpcII7 XbqCnOwYf5MyeOSIClex1YsnLVgbnzp63A9n2ZVWyNiCUlsY+ULMppyXDJsHoJHJIKt8 vj78z9V+bEjceul5pSUcIA1XSCj4l7tzUHNk7fZlM7SEHvh/GVhHhBvyaCLf4mAs1xsT JDayu/VCX4tpmzv7k3Od48a6UcQjaqUjNCqqRQH+0eDEwaS7QY88YR9uspzFX5d7QsX7 wIkANKxgA8vF9OndX55dyS5e5rF/yWYkfFm6n0YnX/6mB83oWftiZDK+p4rTk8xIfKYQ GN/w== X-Gm-Message-State: AOJu0Ywt6iciyry+aMfOIkYjC7XdZgcgjSUNIZZEVN+N+ulj1MN9SpUq RZqCMJKFFXpX4nou+ojmgXzS2KfEk+wxJ7y+ev8PpheV+3NuoOIIsk4pkg== X-Gm-Gg: ASbGncspAeWwIeeLxEadBCP5+xz0xCMGTKoZQ6nDRPDLkGpw2qP2c0Iyfv8S76Li3lC i2QL1LgBK/EafAvj7NNgcTsHVQ8+862CZd9RsdBiIoZ0wEbTuIFhRzjjvLByZM0kXcCKRqOR5cH MTdiCl1xNKS2qGCG26nVnZIkbB5LnT9swdlO9cYDpc2JznmtNOxZPxnqghJcCxaKV04KVuc1bq3 runydmLsnhnbNCQS9PbOTvtwmHjO2raI3A0Cme0TaQuc6njp7MPfEs+nlD6AT0XbqjHrKoTLJkG jcymKSu8T7feDKRivhvfmDnhhd1nvlKLblOrnBpVZRB0ZTOP1k4HR1YkqA== X-Google-Smtp-Source: AGHT+IGzfHzrYdTJCaz6aDLCNCfL7wMrYPBOHj3dJEx1w2UpSwPDnBG/vG8OmTFg0vOcSxQR5jTHBg== X-Received: by 2002:a05:6402:26cf:b0:5e0:8c35:a137 with SMTP id 4fb4d7f45d1cf-5e5e2490c0dmr4679741a12.23.1741372109786; Fri, 07 Mar 2025 10:28:29 -0800 (PST) Received: from 127.0.0.1localhost ([85.255.232.206]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5e5c74a9315sm2883230a12.46.2025.03.07.10.28.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Mar 2025 10:28:28 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH liburing v2 3/4] tests/helpers: add t_create_socketpair_ip Date: Fri, 7 Mar 2025 18:28:55 +0000 Message-ID: X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 create_socketpair_ip() is useful for zerocopy tx testing as it needs TCP sockets and not just AF_UNIX. There will be tx zc tests in more files, so move the function to helpers. Signed-off-by: Pavel Begunkov --- test/helpers.c | 111 ++++++++++++++++++++++++++++++++++++++ test/helpers.h | 5 ++ test/send-zerocopy.c | 123 ++++--------------------------------------- 3 files changed, 127 insertions(+), 112 deletions(-) diff --git a/test/helpers.c b/test/helpers.c index 07186911..b2d386f4 100644 --- a/test/helpers.c +++ b/test/helpers.c @@ -373,3 +373,114 @@ void *aligned_alloc(size_t alignment, size_t size) return ret; } + +int t_create_socketpair_ip(struct sockaddr_storage *addr, + int *sock_client, int *sock_server, + bool ipv6, bool client_connect, + bool msg_zc, bool tcp, const char *name) +{ + socklen_t addr_size; + int family, sock, listen_sock = -1; + int ret; + + memset(addr, 0, sizeof(*addr)); + if (ipv6) { + struct sockaddr_in6 *saddr = (struct sockaddr_in6 *)addr; + + family = AF_INET6; + saddr->sin6_family = family; + saddr->sin6_port = htons(0); + addr_size = sizeof(*saddr); + } else { + struct sockaddr_in *saddr = (struct sockaddr_in *)addr; + + family = AF_INET; + saddr->sin_family = family; + saddr->sin_port = htons(0); + saddr->sin_addr.s_addr = htonl(INADDR_ANY); + addr_size = sizeof(*saddr); + } + + /* server sock setup */ + if (tcp) { + sock = listen_sock = socket(family, SOCK_STREAM, IPPROTO_TCP); + } else { + sock = *sock_server = socket(family, SOCK_DGRAM, 0); + } + if (sock < 0) { + perror("socket"); + return 1; + } + + ret = bind(sock, (struct sockaddr *)addr, addr_size); + if (ret < 0) { + perror("bind"); + return 1; + } + + ret = getsockname(sock, (struct sockaddr *)addr, &addr_size); + if (ret < 0) { + fprintf(stderr, "getsockname failed %i\n", errno); + return 1; + } + + if (tcp) { + ret = listen(sock, 128); + assert(ret != -1); + } + + if (ipv6) { + struct sockaddr_in6 *saddr = (struct sockaddr_in6 *)addr; + + inet_pton(AF_INET6, name, &(saddr->sin6_addr)); + } else { + struct sockaddr_in *saddr = (struct sockaddr_in *)addr; + + inet_pton(AF_INET, name, &saddr->sin_addr); + } + + /* client sock setup */ + if (tcp) { + *sock_client = socket(family, SOCK_STREAM, IPPROTO_TCP); + assert(client_connect); + } else { + *sock_client = socket(family, SOCK_DGRAM, 0); + } + if (*sock_client < 0) { + perror("socket"); + return 1; + } + if (client_connect) { + ret = connect(*sock_client, (struct sockaddr *)addr, addr_size); + if (ret < 0) { + perror("connect"); + return 1; + } + } + if (msg_zc) { +#ifdef SO_ZEROCOPY + int val = 1; + + /* + * NOTE: apps must not set SO_ZEROCOPY when using io_uring zc. + * It's only here to test interactions with MSG_ZEROCOPY. + */ + if (setsockopt(*sock_client, SOL_SOCKET, SO_ZEROCOPY, &val, sizeof(val))) { + perror("setsockopt zc"); + return 1; + } +#else + fprintf(stderr, "no SO_ZEROCOPY\n"); + return 1; +#endif + } + if (tcp) { + *sock_server = accept(listen_sock, NULL, NULL); + if (!*sock_server) { + fprintf(stderr, "can't accept\n"); + return 1; + } + close(listen_sock); + } + return 0; +} diff --git a/test/helpers.h b/test/helpers.h index d0294eba..f8a5c7f2 100644 --- a/test/helpers.h +++ b/test/helpers.h @@ -81,6 +81,11 @@ struct iovec *t_create_buffers(size_t buf_num, size_t buf_size); */ int t_create_socket_pair(int fd[2], bool stream); +int t_create_socketpair_ip(struct sockaddr_storage *addr, + int *sock_client, int *sock_server, + bool ipv6, bool client_connect, + bool msg_zc, bool tcp, const char *name); + /* * Helper for setting up a ring and checking for user privs */ diff --git a/test/send-zerocopy.c b/test/send-zerocopy.c index 680481a0..b505e4d0 100644 --- a/test/send-zerocopy.c +++ b/test/send-zerocopy.c @@ -258,117 +258,6 @@ static int test_send_faults(int sock_tx, int sock_rx) return T_EXIT_PASS; } -static int create_socketpair_ip(struct sockaddr_storage *addr, - int *sock_client, int *sock_server, - bool ipv6, bool client_connect, - bool msg_zc, bool tcp) -{ - socklen_t addr_size; - int family, sock, listen_sock = -1; - int ret; - - memset(addr, 0, sizeof(*addr)); - if (ipv6) { - struct sockaddr_in6 *saddr = (struct sockaddr_in6 *)addr; - - family = AF_INET6; - saddr->sin6_family = family; - saddr->sin6_port = htons(0); - addr_size = sizeof(*saddr); - } else { - struct sockaddr_in *saddr = (struct sockaddr_in *)addr; - - family = AF_INET; - saddr->sin_family = family; - saddr->sin_port = htons(0); - saddr->sin_addr.s_addr = htonl(INADDR_ANY); - addr_size = sizeof(*saddr); - } - - /* server sock setup */ - if (tcp) { - sock = listen_sock = socket(family, SOCK_STREAM, IPPROTO_TCP); - } else { - sock = *sock_server = socket(family, SOCK_DGRAM, 0); - } - if (sock < 0) { - perror("socket"); - return 1; - } - - ret = bind(sock, (struct sockaddr *)addr, addr_size); - if (ret < 0) { - perror("bind"); - return 1; - } - - ret = getsockname(sock, (struct sockaddr *)addr, &addr_size); - if (ret < 0) { - fprintf(stderr, "getsockname failed %i\n", errno); - return 1; - } - - if (tcp) { - ret = listen(sock, 128); - assert(ret != -1); - } - - if (ipv6) { - struct sockaddr_in6 *saddr = (struct sockaddr_in6 *)addr; - - inet_pton(AF_INET6, HOSTV6, &(saddr->sin6_addr)); - } else { - struct sockaddr_in *saddr = (struct sockaddr_in *)addr; - - inet_pton(AF_INET, HOST, &saddr->sin_addr); - } - - /* client sock setup */ - if (tcp) { - *sock_client = socket(family, SOCK_STREAM, IPPROTO_TCP); - assert(client_connect); - } else { - *sock_client = socket(family, SOCK_DGRAM, 0); - } - if (*sock_client < 0) { - perror("socket"); - return 1; - } - if (client_connect) { - ret = connect(*sock_client, (struct sockaddr *)addr, addr_size); - if (ret < 0) { - perror("connect"); - return 1; - } - } - if (msg_zc) { -#ifdef SO_ZEROCOPY - int val = 1; - - /* - * NOTE: apps must not set SO_ZEROCOPY when using io_uring zc. - * It's only here to test interactions with MSG_ZEROCOPY. - */ - if (setsockopt(*sock_client, SOL_SOCKET, SO_ZEROCOPY, &val, sizeof(val))) { - perror("setsockopt zc"); - return 1; - } -#else - fprintf(stderr, "no SO_ZEROCOPY\n"); - return 1; -#endif - } - if (tcp) { - *sock_server = accept(listen_sock, NULL, NULL); - if (!*sock_server) { - fprintf(stderr, "can't accept\n"); - return 1; - } - close(listen_sock); - } - return 0; -} - struct send_conf { bool fixed_buf; bool mix_register; @@ -574,6 +463,16 @@ static int do_test_inet_send(struct io_uring *ring, int sock_client, int sock_se return 0; } +static int create_socketpair_ip(struct sockaddr_storage *addr, + int *sock_client, int *sock_server, + bool ipv6, bool client_connect, + bool msg_zc, bool tcp) +{ + return t_create_socketpair_ip(addr, sock_client, sock_server, ipv6, + client_connect, msg_zc, tcp, + ipv6 ? HOSTV6 : HOST); +} + static int test_inet_send(struct io_uring *ring) { struct send_conf conf; @@ -598,7 +497,7 @@ static int test_inet_send(struct io_uring *ring) continue; #endif ret = create_socketpair_ip(&addr, &sock_client, &sock_server, ipv6, - client_connect, msg_zc_set, tcp); + client_connect, msg_zc_set, tcp); if (ret) { fprintf(stderr, "sock prep failed %d\n", ret); return 1; From patchwork Fri Mar 7 18:28:56 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 14006949 Received: from mail-ed1-f41.google.com (mail-ed1-f41.google.com [209.85.208.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C368323BCFA for ; Fri, 7 Mar 2025 18:28:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741372117; cv=none; b=SgpebakbbwAuEhGASfs/i0p4KnFM18ExlQE6/DoSYZ68QCwO7I3oGSEqPEaZpN7rrpzNdHI6bJliejOs5yX5eVpFVM/GFQnggGa1vC+yTsTcQyLmOBCRtI9tFm3yAq5MLxK8HcZTqjG1x65tBC+ebDODBN2JvV9ko/Rhf5azNuE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741372117; c=relaxed/simple; bh=k2qdaVJX1Oy1PPU7TLsGOIK/h24OVnA/pBE/aQlKXBE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=uXiIX5WA2IMn/ocEDsFW6VMZedOLp1s8JRz+urx8SZdHcYOb9hNuaD4pspTojjEdIzNnpTHnD4UOIpMsVnT5VoyzG+u2Dtxc7gxkLNuRqDOZmO/5ARbfLFy6rlppHiccKMNvyUZpjLqpeLwjF89YmLpqIJwJJopcMofphA92utI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=eCg+GtMm; arc=none smtp.client-ip=209.85.208.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="eCg+GtMm" Received: by mail-ed1-f41.google.com with SMTP id 4fb4d7f45d1cf-5e614da8615so375168a12.1 for ; Fri, 07 Mar 2025 10:28:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741372112; x=1741976912; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Q4DhHv8/7P8vMGPOcDW4/bTrCzDmmW4urYlMU97ASdI=; b=eCg+GtMmL+WV+Vqwwi6XpXxl25dWo/JzN+K+9ctw7gJXSHqNfau5mgCJ1TcIhdsZJh 99E9vSE7jG4Rn4UUxPcJ6Psg4ubCRbvyQ8+aTZI5jDwxk9aeZAu3gNSHl8fbqamKHu2T RVVXaoWNs25amqeZSXaUkd+1tc/k7GdYp1P626Sbq8+hIQHKttCvWweSEXq6XfmJYdsp 60ewX/7t6VnDQ4NS4HmALYkaOx2+2klyidjSUCOa9nTQPZQlngi27ZUF8ScCvtCKq/5j ueTXBOGYa05V/Cp5OAzztUupgVSZI36dCEkMuiNwaPpSH8nLSSQ8XeCEL+Y7Yq/0DNha 0TGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741372112; x=1741976912; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Q4DhHv8/7P8vMGPOcDW4/bTrCzDmmW4urYlMU97ASdI=; b=IH2C0QS8f4cbdt7RDmyUriuWscnO0V5v9DU6EjFBmeWY2gDiRw7cvteKJdQKC2DBXs YgAShYjCaqxfJJdWoDwyWx6bIF7x5fZZSGTBGl2TQIj8qKr40PejyIv7RpSpTfL2FLiW rI/9WujuID17VxCl7SioLdBT811a4QQko/iZISrHFpXUR+zvuWnf4zhOUI88jffDcg7c n4cp3G4aeZIgn6Z93KIaJEYkh7iRKAwHkGOyDnVkHs3Vbac6faD1D1RvqIIzQB9G/vJf b7i0o/GM+1sm7cCzLlYGXUCZOp3L3GZIeCZAYiqw7wK6wiBmkA8D72rdp2tfG7u+boTM Rwqw== X-Gm-Message-State: AOJu0YxeopDx66WXg01dhk1FJ8SVy4w2jxIbvZFBiGj8kDCbpLscwUSN h1JrCTo9LtiKvOEdB0dGdqF0euDiv6E444PBCbsPctBPbA1xCx0/Vtyx4Q== X-Gm-Gg: ASbGncu+bGdT/u4PI9aTu6xEcByyQ8b/udJCoZkMk/L+yyyWxoMzQyVPiGdUirgix73 jkwOiV3rqGN+7Oi2TcYjQnthGj8L+wxF590Ki8xE2QkmEq1LH4hE6pfClXnpyuAD5qIAlVtlxLz WLZ6R+jEhPPhjLDKR03BzNdN5098mKzTf/EwmIaWPThyio0dEkoabv3YS5nSZXREuoMH0LeFi1p /KlwiErIZ2YhYXD48mySMI+uBBFtIkjylYb6wAj5sVBus9oMRb1yrj1fUe1XS2zVB3pnYEYKUYx C2xbP/HY0WmiNVk8cvJmi7jE1VsouMGRheqctHmxD0JRm0yGBYMrXIFAwg== X-Google-Smtp-Source: AGHT+IFdS0G7ZRu29/BUYmqM2a90XDi0ogg7fWUuJMGtVz9eBAiDJY2jPmg4N3b+yKPKvKp8fSgDrQ== X-Received: by 2002:a05:6402:210f:b0:5de:d932:a54c with SMTP id 4fb4d7f45d1cf-5e614ec158amr929297a12.2.1741372112271; Fri, 07 Mar 2025 10:28:32 -0800 (PST) Received: from 127.0.0.1localhost ([85.255.232.206]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5e5c74a9315sm2883230a12.46.2025.03.07.10.28.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Mar 2025 10:28:31 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH liburing v2 4/4] tests: targeted registered vector tests Date: Fri, 7 Mar 2025 18:28:56 +0000 Message-ID: X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 The main part here is to exercise various iovec->bvec reallocation scenarios, but it also checks edge cases and validates final data. Signed-off-by: Pavel Begunkov --- test/Makefile | 1 + test/vec-regbuf.c | 605 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 606 insertions(+) create mode 100644 test/vec-regbuf.c diff --git a/test/Makefile b/test/Makefile index 0367ef72..fb11d16a 100644 --- a/test/Makefile +++ b/test/Makefile @@ -241,6 +241,7 @@ test_srcs := \ wq-aff.c \ xattr.c \ zcrx.c \ + vec-regbuf.c \ # EOL # Please keep this list sorted alphabetically. diff --git a/test/vec-regbuf.c b/test/vec-regbuf.c new file mode 100644 index 00000000..1891e315 --- /dev/null +++ b/test/vec-regbuf.c @@ -0,0 +1,605 @@ +/* SPDX-License-Identifier: MIT */ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "helpers.h" +#include "liburing.h" + +static bool has_regvec; + +struct buf_desc { + char *buf_wr; + char *buf_rd; + size_t size; + + struct io_uring ring; + bool ring_init; + bool fixed; + int buf_idx; + bool rw; +}; + +#define BUF_BASE_IDX 1 +static int page_sz; + +static void probe_support(void) +{ + struct io_uring_probe *p; + struct io_uring ring; + int ret = 0; + + ret = io_uring_queue_init(1, &ring, 0); + if (ret) { + fprintf(stderr, "queue init failed: %d\n", ret); + exit(ret); + } + + p = t_calloc(1, sizeof(*p) + 256 * sizeof(struct io_uring_probe_op)); + ret = io_uring_register_probe(&ring, p, 256); + + /* if we don't have PROBE_REGISTER, we don't have OP_READ/WRITE */ + if (ret == -EINVAL) + goto out; + if (ret) { + fprintf(stderr, "register_probe: %d\n", ret); + goto out; + } + + has_regvec = p->ops_len > IORING_OP_READV_FIXED && + (p->ops[IORING_OP_READV_FIXED].flags & IO_URING_OP_SUPPORTED); +out: + io_uring_queue_exit(&ring); + if (p) + free(p); +} + +static void bind_ring(struct buf_desc *bd, struct io_uring *ring, unsigned buf_idx) +{ + size_t size = bd->size; + struct iovec iov; + int ret; + + iov.iov_len = size; + iov.iov_base = bd->buf_wr; + + ret = io_uring_register_buffers_update_tag(ring, buf_idx, &iov, NULL, 1); + if (ret != 1) { + if (geteuid()) { + fprintf(stderr, "Not root, skipping\n"); + exit(0); + } + fprintf(stderr, "buf reg failed %i\n", ret); + exit(1); + } + bd->buf_idx = buf_idx; +} + +static void reinit_ring(struct buf_desc *bd) +{ + struct io_uring *ring = &bd->ring; + int ret; + + if (bd->ring_init) { + io_uring_queue_exit(ring); + bd->ring_init = false; + } + + ret = io_uring_queue_init(32, ring, 0); + if (ret) { + fprintf(stderr, "ring init error %i\n", ret); + exit(1); + } + + ret = io_uring_register_buffers_sparse(ring, 128); + if (ret) { + fprintf(stderr, "table reg error %i\n", ret); + exit(1); + } + + bind_ring(bd, &bd->ring, BUF_BASE_IDX); + bd->ring_init = true; +} + +static void init_buffers(struct buf_desc *bd, size_t size) +{ + void *start; + void *mem; + + start = mmap(NULL, size + page_sz * 2, PROT_NONE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE, -1, 0); + if (start == MAP_FAILED) { + fprintf(stderr, "Unable to preserve the page mixture memory.\n"); + exit(1); + } + + mem = mmap(start + page_sz, size, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, -1, 0); + if (mem == MAP_FAILED) { + fprintf(stderr, "mmap fail\n"); + exit(1); + } + + memset(bd, 0, sizeof(*bd)); + bd->size = size; + bd->buf_wr = mem; + bd->buf_rd = malloc(size); + if (!bd->buf_rd) { + fprintf(stderr, "malloc fail\n"); + exit(1); + } +} + +static int verify_data(struct buf_desc *bd, struct iovec *wr_vecs, int nr_iovec, + int fd) +{ + int iov_idx, ret; + + for (iov_idx = 0; iov_idx < nr_iovec; iov_idx++) { + struct iovec *vec = &wr_vecs[iov_idx]; + size_t seg_size = vec->iov_len; + size_t read_bytes = 0; + + while (1) { + ret = read(fd, bd->buf_rd + read_bytes, seg_size - read_bytes); + if (ret < 0) { + fprintf(stderr, "read error %i", ret); + return 1; + } + read_bytes += ret; + if (read_bytes == seg_size) + break; + if (ret == 0) { + fprintf(stderr, "can't read %i", ret); + return 2; + } + } + + ret = memcmp(bd->buf_rd, vec->iov_base, seg_size); + if (ret != 0) { + fprintf(stderr, "data mismatch %i\n", ret); + return 3; + } + } + return 0; +} + +struct verify_data { + struct buf_desc *bd; + struct iovec *vecs; + int nr_vec; + int fd; +}; + +static void *verify_thread_cb(void *data) +{ + struct verify_data *vd = data; + int ret; + + ret = verify_data(vd->bd, vd->vecs, vd->nr_vec, vd->fd); + return (void *)(unsigned long)ret; +} + +static int test_rw(struct buf_desc *bd, struct iovec *vecs, int nr_vec, int fd_wr) +{ + unsigned buf_idx = bd->buf_idx; + struct io_uring *ring = &bd->ring; + struct io_uring_sqe *sqe; + struct io_uring_cqe *cqe; + int ret; + + sqe = io_uring_get_sqe(ring); + if (bd->fixed) + io_uring_prep_writev_fixed(sqe, fd_wr, vecs, nr_vec, 0, 0, buf_idx); + else + io_uring_prep_writev(sqe, fd_wr, vecs, nr_vec, 0); + + ret = io_uring_submit(ring); + if (ret != 1) { + fprintf(stderr, "submit failed %i\n", ret); + exit(1); + } + ret = io_uring_wait_cqe(ring, &cqe); + if (ret) { + fprintf(stderr, "wait_cqe=%d\n", ret); + exit(1); + } + + ret = cqe->res; + io_uring_cqe_seen(ring, cqe); + return ret; +} + +static int test_sendzc(struct buf_desc *bd, struct iovec *vecs, int nr_vec, int fd_wr) +{ + unsigned buf_idx = bd->buf_idx; + struct io_uring *ring = &bd->ring; + struct io_uring_sqe *sqe; + struct io_uring_cqe *cqe; + int ret, cqe_ret, more; + struct msghdr msghdr; + + memset(&msghdr, 0, sizeof(msghdr)); + msghdr.msg_iov = vecs; + msghdr.msg_iovlen = nr_vec; + + sqe = io_uring_get_sqe(ring); + if (bd->fixed) + io_uring_prep_sendmsg_zc_fixed(sqe, fd_wr, &msghdr, 0, buf_idx); + else + io_uring_prep_sendmsg_zc(sqe, fd_wr, &msghdr, 0); + + ret = io_uring_submit(ring); + if (ret != 1) { + fprintf(stderr, "submit failed %i\n", ret); + exit(1); + } + ret = io_uring_wait_cqe(ring, &cqe); + if (ret) { + fprintf(stderr, "wait_cqe=%d\n", ret); + exit(1); + } + + cqe_ret = cqe->res; + more = cqe->flags & IORING_CQE_F_MORE; + io_uring_cqe_seen(ring, cqe); + + if (more) { + ret = io_uring_wait_cqe(ring, &cqe); + if (ret) { + fprintf(stderr, "wait_cqe=%d\n", ret); + exit(1); + } + io_uring_cqe_seen(ring, cqe); + } + return cqe_ret; +} + +static int test_vec(struct buf_desc *bd, struct iovec *vecs, int nr_vec, + bool expect_fail, int *cqe_ret) +{ + struct sockaddr_storage addr; + int sock_server, sock_client; + struct verify_data vd; + size_t total_len = 0; + int i, ret; + void *verify_res; + pthread_t th; + + ret = t_create_socketpair_ip(&addr, &sock_client, &sock_server, + true, true, false, true, "::1"); + if (ret) { + fprintf(stderr, "sock prep failed %d\n", ret); + return 1; + } + + for (i = 0; i < bd->size; i++) + bd->buf_wr[i] = i; + memset(bd->buf_rd, 0, bd->size); + + for (i = 0; i < nr_vec; i++) + total_len += vecs[i].iov_len; + + vd.bd = bd; + vd.vecs = vecs; + vd.nr_vec = nr_vec; + vd.fd = sock_server; + + if (!expect_fail) { + ret = pthread_create(&th, NULL, verify_thread_cb, &vd); + if (ret) { + fprintf(stderr, "pthread_create failed %i\n", ret); + return ret; + } + } + + if (bd->rw) + ret = test_rw(bd, vecs, nr_vec, sock_client); + else + ret = test_sendzc(bd, vecs, nr_vec, sock_client); + + *cqe_ret = ret; + + if (!expect_fail && ret != total_len) { + fprintf(stderr, "invalid cqe %i, expected %lu\n", + ret, (unsigned long)total_len); + return 1; + } + + if (!expect_fail) { + pthread_join(th, &verify_res); + ret = (int)(unsigned long)verify_res; + if (ret) { + fprintf(stderr, "verify failed %i\n", ret); + return 1; + } + } + close(sock_client); + close(sock_server); + return 0; +} + +struct work { + struct iovec *vecs; + unsigned nr_vecs; +}; + +static int test_sequence(struct buf_desc *bd, unsigned nr, struct work *ws) +{ + int i, ret; + int cqe_ret; + + reinit_ring(bd); + + for (i = 0; i < nr; i++) { + ret = test_vec(bd, ws[i].vecs, ws[i].nr_vecs, false, &cqe_ret); + if (ret) { + fprintf(stderr, "sequence failed, idx %i/%i\n", i, nr); + return ret; + } + } + return 0; +} + +static void test_basic(struct buf_desc *bd) +{ + void *p = bd->buf_wr; + int ret; + struct iovec iov_page = { .iov_base = p, + .iov_len = page_sz, }; + struct iovec iov_inner = { .iov_base = p + 1, + .iov_len = 3, }; + struct iovec iov_maxbvec = { .iov_base = p + page_sz - 1, + .iov_len = page_sz + 2, }; + struct iovec iov_big = { .iov_base = p, + .iov_len = page_sz * 12 + 33, }; + struct iovec iov_big_unalign = { .iov_base = p + 10, + .iov_len = page_sz * 7 + 41, }; + struct iovec iov_full = { .iov_base = p, + .iov_len = bd->size, }; + struct iovec iov_right1 = { .iov_base = p + bd->size - page_sz + 5, + .iov_len = page_sz - 5 }; + struct iovec iov_right2 = { .iov_base = p + bd->size - page_sz - 5, + .iov_len = page_sz + 5 }; + struct iovec iov_full_unalign = { .iov_base = p + 1, + .iov_len = bd->size - 1, }; + struct iovec vecs[] = { + iov_page, + iov_big, + iov_inner, + iov_big_unalign, + iov_big_unalign, + }; + struct iovec vecs_basic[] = { iov_page, iov_page, iov_page }; + struct iovec vecs_full[] = { iov_full, iov_full, iov_full }; + struct iovec vecs_full_unalign[] = { iov_full_unalign, iov_full_unalign, + iov_full_unalign }; + struct iovec vecs_maxsegs[] = { iov_maxbvec, iov_maxbvec, iov_maxbvec, + iov_maxbvec, iov_maxbvec, iov_maxbvec}; + + ret = test_sequence(bd, 1, (struct work[]) { + { &iov_page, 1 }, + { vecs, 1 }}); + if (ret) { + fprintf(stderr, "seq failure: basic aligned, %i\n", ret); + exit(1); + } + + ret = test_sequence(bd, 2, (struct work[]) { + { vecs, 1 }, + { vecs, 1 }}); + if (ret) { + fprintf(stderr, "seq failure: basic aligned, %i\n", ret); + exit(1); + } + + ret = test_sequence(bd, 2, (struct work[]) { + { vecs + 1, 1 }, + { vecs + 1, 1 }}); + if (ret) { + fprintf(stderr, "seq failure: multi page buffer, %i\n", ret); + exit(1); + } + + ret = test_sequence(bd, 2, (struct work[]) { + { vecs + 2, 1 }, + { vecs + 2, 1 }}); + if (ret) { + fprintf(stderr, "seq failure: misaligned buffer, %i\n", ret); + exit(1); + } + + ret = test_sequence(bd, 2, (struct work[]) { + { vecs + 3, 1 }, + { vecs + 3, 1 }}); + if (ret) { + fprintf(stderr, "seq failure: misaligned multipage buffer, %i\n", ret); + exit(1); + } + + ret = test_sequence(bd, 2, (struct work[]) { + { vecs, 1 }, + { vecs + 3, 1 }}); + if (ret) { + fprintf(stderr, "seq failure: realloc + increase bvec, %i\n", ret); + exit(1); + } + + ret = test_sequence(bd, 2, (struct work[]) { + { vecs + 3, 1 }, + { vecs + 0, 1 }}); + if (ret) { + fprintf(stderr, "seq failure: realloc + decrease bvec, %i\n", ret); + exit(1); + } + + ret = test_sequence(bd, 2, (struct work[]) { + { vecs, 4 }, + { vecs, 4 }}); + if (ret) { + fprintf(stderr, "seq failure: multisegment, %i\n", ret); + exit(1); + } + + ret = test_sequence(bd, 3, (struct work[]) { + { vecs, 2 }, + { vecs, 3 }, + { vecs, 4 }}); + if (ret) { + fprintf(stderr, "seq failure: multisegment 2, %i\n", ret); + exit(1); + } + + ret = test_sequence(bd, 3, (struct work[]) { + { vecs_basic, 1 }, + { vecs_basic, 2 }, + { vecs_basic, 3 }}); + if (ret) { + fprintf(stderr, "seq failure: increase iovec, %i\n", ret); + exit(1); + } + + ret = test_sequence(bd, 3, (struct work[]) { + { vecs_basic, 3 }, + { vecs_basic, 2 }, + { vecs_basic, 1 }}); + if (ret) { + fprintf(stderr, "seq failure: decrease iovec, %i\n", ret); + exit(1); + } + + ret = test_sequence(bd, 3, (struct work[]) { + { &iov_right1, 1 }, + { &iov_right2, 1 }, + { &iov_right1, 1 }}); + if (ret) { + fprintf(stderr, "seq failure: right aligned, %i\n", ret); + exit(1); + } + + ret = test_sequence(bd, 3, (struct work[]) { + { vecs_full, 1 }, + { vecs_full, 1 }, + { vecs_full, 3 }}); + if (ret) { + fprintf(stderr, "seq failure: full size, %i\n", ret); + exit(1); + } + + ret = test_sequence(bd, 3, (struct work[]) { + { vecs_full_unalign, 1 }, + { vecs_full_unalign, 1 }, + { vecs_full_unalign, 3 }}); + if (ret) { + fprintf(stderr, "seq failure: full size unsigned, %i\n", ret); + exit(1); + } + + ret = test_sequence(bd, 3, (struct work[]) { + { vecs_maxsegs, 1 }, + { vecs_maxsegs, 2 }, + { vecs_maxsegs, 3 }}); + if (ret) { + fprintf(stderr, "seq failure: overestimated segments, %i\n", ret); + exit(1); + } + + ret = test_sequence(bd, 3, (struct work[]) { + { vecs_maxsegs, 6 }, + { vecs_maxsegs, 6 }, + { vecs_maxsegs, 6 }}); + if (ret) { + fprintf(stderr, "seq failure: overestimated segments 2, %i\n", ret); + exit(1); + } +} + +static void test_fail(struct buf_desc *bd) +{ + int ret, cqe_ret; + void *p = bd->buf_wr; + struct iovec iov_0len = { .iov_base = p, .iov_len = 0 }; + struct iovec iov_0buf = { .iov_base = 0, .iov_len = 1 }; + struct iovec iov_inv = { .iov_base = (void *)-1U, .iov_len = 1 }; + struct iovec iov_under = { .iov_base = p - 1, .iov_len = 1 }; + struct iovec iov_over = { .iov_base = p + bd->size, .iov_len = 1 }; + struct iovec vecs_0[] = { iov_0len, iov_0len, iov_0len, iov_0len, + iov_0len, iov_0len, iov_0len, iov_0len }; + + reinit_ring(bd); + ret = test_vec(bd, vecs_0, 8, true, &cqe_ret); + if (ret || cqe_ret > 0) { + fprintf(stderr, "0 length test failed %i, cqe %i\n", + ret, cqe_ret); + exit(1); + } + + reinit_ring(bd); + ret = test_vec(bd, &iov_0buf, 1, true, &cqe_ret); + if (ret || cqe_ret >= 0) { + fprintf(stderr, "0 buf test failed %i, cqe %i\n", + ret, cqe_ret); + exit(1); + } + + reinit_ring(bd); + ret = test_vec(bd, &iov_inv, 1, true, &cqe_ret); + if (ret || cqe_ret >= 0) { + fprintf(stderr, "inv buf test failed %i, cqe %i\n", + ret, cqe_ret); + exit(1); + } + + reinit_ring(bd); + ret = test_vec(bd, &iov_under, 1, true, &cqe_ret); + if (ret || cqe_ret >= 0) { + fprintf(stderr, "inv buf underflow failed %i, cqe %i\n", + ret, cqe_ret); + exit(1); + } + + reinit_ring(bd); + ret = test_vec(bd, &iov_over, 1, true, &cqe_ret); + if (ret || cqe_ret >= 0) { + fprintf(stderr, "inv buf overflow failed %i, cqe %i\n", + ret, cqe_ret); + exit(1); + } +} + +int main(void) +{ + struct buf_desc bd = {}; + int i = 0; + + page_sz = sysconf(_SC_PAGESIZE); + + probe_support(); + if (!has_regvec) { + printf("doesn't support registered vector ops, skip\n"); + return 0; + } + + init_buffers(&bd, page_sz * 32); + bd.fixed = true; + + for (i = 0; i < 2; i++) { + bool rw = i & 1; + + bd.rw = rw; + + test_basic(&bd); + test_fail(&bd); + } + + io_uring_queue_exit(&bd.ring); + return 0; +}