From patchwork Mon Aug 12 16:55:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13760857 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9A29D13AA47 for ; Mon, 12 Aug 2024 17:00:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723482058; cv=none; b=sE6a6fnFazLH2GfGAyskGFNkJbswnkSleNuDqlYcUDQCaT5La+dwIYxtleiPiIZL505Cuupp6tfpfvOHLkgakBNQfdUbYjrcmZuWgMa7Huep+oPy9zKZcVH1qM83zsNk21UrXVwn8bkFDxRmC7vNtOfByi+edIatD9tmoYJOjL4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723482058; c=relaxed/simple; bh=kEK9aFaDFaRtL9J2hVxr8jMo8+cADSfUsHnGS+JgYzg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TMwUlRUQjFAqiqbH2KgCZqsyR+ZsY7XKeXnSaPrYggaQTNFfUQUQFuo57ejEMoL6wMvxBvUDiHVFWi9BZGRVTFSRVG860/mPMgNhZUsO/IApafN+VKUtw8KFI+BvBqL+ISnOZBsEewimA2uoM6cHPINX/35jr3M7tj05ospa75U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=q1Jhkl2R; arc=none smtp.client-ip=209.85.214.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="q1Jhkl2R" Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-1fc57d0f15aso3052655ad.2 for ; Mon, 12 Aug 2024 10:00:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1723482053; x=1724086853; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=o80CQNJb0yqYiBgWEpKfZgN9Dt8er/w2Rt4Yd7KyRkU=; b=q1Jhkl2RVXSUJle7AyfzEdSwe8xIE0CMoG2X3G5Z1lEXIQWAeUqm8MkEleSU5yyFyt Cc07b1IpdRTXGMlijlgd5JD0aCwtMpBMQqrq1NtN1dL3hX/LFTVSiJcapT17JMq5LCRd Gx0Dn3vhzrRBtQboXtuish9w7HROVCW0FspQnLrjMlANgkw6O84NcH8oGhEpQQi2tYQp StZFVDQChl3NVhls1KYxCfOo6Gkvm2BuqcUYoogMPG906+/tJ98gmuTMQsnzJYDs/Tjo Fhy5uF5SCD2DzoXv2mE5al6b0HndeVnX79x+ApJvLimuvA0Cyk114OZHaHAM5kseNFx2 1jng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723482053; x=1724086853; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=o80CQNJb0yqYiBgWEpKfZgN9Dt8er/w2Rt4Yd7KyRkU=; b=lNnCmcplOhhmyS4YbhXSTHELHEJefdliN++RsFg7eqV2UvFR/fYEEcEkmmdiE9Y5or QJRJ+jMehu+KfqHyQZGOQP40i/FLCOzoOTzz2QZ4lCCOUC5Uhm5zEqHgArTVD7JIRSzG OHzf0QDXDIo4FJu4eR+k/D7+aHufro5xD/RhxM6NrxwElXdVEP9POdnB3s19AtJ3bC6r rAtLsMTeZATcMKUGR+ZCTRCvDvrkcAfxPYgHtYfRQngdSbEA/+niFOKSq8jtpiR4tm0d oFaagr25NYFmd/aht0/VJrKrOtuF87jWncM6egbzghIT1aiiK4QrzG+eb5dCmp5ZbW42 87Gg== X-Gm-Message-State: AOJu0YwoEZIeX9grAZXIMOxu1yaLvhjT2iEcNC9ZUWPQCTsXfF2dQGqy eQpj+Ds4usLVm5MyznXVw+O4JNPjdU1fA9S9h1NbpkNXqkeB+37K6/gXnfl74Iv0oteBk/CoRHY h X-Google-Smtp-Source: AGHT+IE4+3uAatWw8oHQdwf8QJhtcVULK/+WdIFJJcM2GAhHctCIqITBGiA/PYpEv9hxcVGWMVvhMw== X-Received: by 2002:a17:903:1c2:b0:1f9:b19b:4255 with SMTP id d9443c01a7336-201ca139a9emr6506315ad.4.1723482053305; Mon, 12 Aug 2024 10:00:53 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-200bb9feaa4sm40212725ad.213.2024.08.12.10.00.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Aug 2024 10:00:52 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 1/4] io_uring/kbuf: have io_provided_buffers_select() take buf_sel_arg Date: Mon, 12 Aug 2024 10:55:24 -0600 Message-ID: <20240812170044.93133-2-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240812170044.93133-1-axboe@kernel.dk> References: <20240812170044.93133-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Rather than pass in the iovec in both spots, pass in the buf_sel_arg struct pointer directly. In preparation for needing more of this selection struct off that path. Signed-off-by: Jens Axboe --- io_uring/kbuf.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index 793b2454acca..794a687d8589 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -119,7 +119,7 @@ static void __user *io_provided_buffer_select(struct io_kiocb *req, size_t *len, static int io_provided_buffers_select(struct io_kiocb *req, size_t *len, struct io_buffer_list *bl, - struct iovec *iov) + struct buf_sel_arg *arg) { void __user *buf; @@ -127,8 +127,8 @@ static int io_provided_buffers_select(struct io_kiocb *req, size_t *len, if (unlikely(!buf)) return -ENOBUFS; - iov[0].iov_base = buf; - iov[0].iov_len = *len; + arg->iovs[0].iov_base = buf; + arg->iovs[0].iov_len = *len; return 0; } @@ -296,7 +296,7 @@ int io_buffers_select(struct io_kiocb *req, struct buf_sel_arg *arg, bl->head += ret; } } else { - ret = io_provided_buffers_select(req, &arg->out_len, bl, arg->iovs); + ret = io_provided_buffers_select(req, &arg->out_len, bl, arg); } out_unlock: io_ring_submit_unlock(ctx, issue_flags); @@ -323,7 +323,7 @@ int io_buffers_peek(struct io_kiocb *req, struct buf_sel_arg *arg) } /* don't support multiple buffer selections for legacy */ - return io_provided_buffers_select(req, &arg->max_len, bl, arg->iovs); + return io_provided_buffers_select(req, &arg->max_len, bl, arg); } static int __io_remove_buffers(struct io_ring_ctx *ctx, From patchwork Mon Aug 12 16:55:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13760858 Received: from mail-pj1-f48.google.com (mail-pj1-f48.google.com [209.85.216.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5DD8017C21B for ; Mon, 12 Aug 2024 17:00:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723482058; cv=none; b=psD18EjJjqPQKLcYTduOn6c9NUtjlsDTJ5N3VKcCFWtGtU4lpcKlBGHjsQO9IhOEA7SMAFM+y4iCSktHg6wNHQefVEwHw9iCmIizGkbrkN8w6iujT1Znvle0LJ4qHoqP5q3/oTFu2pCROW4bDiwQugh8EvMLXhRG06UYaBgYBII= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723482058; c=relaxed/simple; bh=LbQqkw7YZP8zPiB3vP6jCdtVMBrcnWjSI4UjUVwzchQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=NAIjBispK48ITm+aF/JQS1zCmws4dcRpdDfRgZdOWWntrXLBpgRYad6SRfz5y9QU7sJoXXKgd5PcxD3XssxxjFyFuCVvj9AnN8htPrv1K8P2RyySnyk/RPMgtTDAx/hpnsg4YuSGo1Lpuiv+V4mekq3oZ0/DUTW+jPLj5mmsy/g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=dbdIElAb; arc=none smtp.client-ip=209.85.216.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="dbdIElAb" Received: by mail-pj1-f48.google.com with SMTP id 98e67ed59e1d1-2cb80633dcfso856614a91.0 for ; Mon, 12 Aug 2024 10:00:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1723482055; x=1724086855; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+k0CzKlomibFd620tzLnsekibzHi84Tk+Peag9/NfCM=; b=dbdIElAbic66FrQjRpHo9stjHgdiz38HkzyWy5/WH0j0gzenXJjCw9FRlJ6g5OnSpq KmAQglT7aqe0GwTdcXok5pWOsCU5MWZLnw8zvAvRLqPn0kK1/q9wWE6pP8gVwv94Rxna HRoGuwk5kgeL5hAB4i0txl0YOIQorMOLF7WILPxCb1at0n5dzqir6OJP8pK5809MPuzY bJ7dtRYGcmS6R0OlFHPIdV+fzZ5DatygsSOYR8p/J0aOuSJNMRSAZFdecbQyKpsUB+lp B8EzA+WgAeJJlNOIZq4wPCVMnk4dQ2pABiicP0aQI4lmtYNJd2tttLzqPKCkRy+cpcWS O3gw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723482055; x=1724086855; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+k0CzKlomibFd620tzLnsekibzHi84Tk+Peag9/NfCM=; b=tWrBHGAM1jCTv2FXMzhnnhXQnKSKmHry97bSWkuCcvJ/utzqRqlwMBIgvfJkasJ7HX fKRkbGQ3KP3Pq22s59KfFQMNJCyTNSALARMk2Razcl9ddE9FvOjSfRIENgAYKGaRCgv5 aFmPd0CtE8Crhg+uZ0Hbn+o/EsXqxGLAj2RUX7vHVmKl/pRxhYgD4i+backa9qTSKkzr RLGhL37r8LOILU9pkq6C4dVDndHBs9SsERaeP1D6pHOObsmzVncOvLrf2m0MqJp1QPbF 1pPxm6Rw1ZTrCtvtkEpCJwFLE9Lkt1iPOhNioE0nmnFYZasGhcQ1AXW73w2RTz4mEN+M F4og== X-Gm-Message-State: AOJu0YxmWVprkJR0aWAJBh3NHOdLDEI+fxu1Wv6PUZNVD+05LmTNBJOq L8MpR09FGVqq3mBthPKbjU87rcTSjsy2DIoePb/oW/n1b9COSZKxTa37A+bxM6X1wFWmP88VOKC l X-Google-Smtp-Source: AGHT+IGZ3MnUtHMXCuEWLVAE2vRG7Kow9Jdh4LkXDlRDJOa8q6+8gnVPt5XhMVLK4eftXigy5itMuA== X-Received: by 2002:a17:902:d4cc:b0:1fd:a7b8:edaf with SMTP id d9443c01a7336-201ca1da02cmr6110495ad.8.1723482055094; Mon, 12 Aug 2024 10:00:55 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-200bb9feaa4sm40212725ad.213.2024.08.12.10.00.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Aug 2024 10:00:53 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 2/4] io_uring/net: pass in io_kiocb to io_bundle_nbufs() Date: Mon, 12 Aug 2024 10:55:25 -0600 Message-ID: <20240812170044.93133-3-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240812170044.93133-1-axboe@kernel.dk> References: <20240812170044.93133-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 In preparation for needing the io_kiocb in there, and kmsg can always be gotten off that in the first place. Signed-off-by: Jens Axboe --- io_uring/net.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index e312fc1ed7de..a6268e62b348 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -459,8 +459,9 @@ static void io_req_msg_cleanup(struct io_kiocb *req, * data in the iter, then loop the segments to figure out how much we * transferred. */ -static int io_bundle_nbufs(struct io_async_msghdr *kmsg, int ret) +static int io_bundle_nbufs(struct io_kiocb *req, int ret) { + struct io_async_msghdr *kmsg = req->async_data; struct iovec *iov; int nbufs; @@ -503,7 +504,7 @@ static inline bool io_send_finish(struct io_kiocb *req, int *ret, goto finish; } - cflags = io_put_kbufs(req, *ret, io_bundle_nbufs(kmsg, *ret), issue_flags); + cflags = io_put_kbufs(req, *ret, io_bundle_nbufs(req, *ret), issue_flags); if (bundle_finished || req->flags & REQ_F_BL_EMPTY) goto finish; @@ -844,7 +845,7 @@ static inline bool io_recv_finish(struct io_kiocb *req, int *ret, cflags |= IORING_CQE_F_SOCK_NONEMPTY; if (sr->flags & IORING_RECVSEND_BUNDLE) { - cflags |= io_put_kbufs(req, *ret, io_bundle_nbufs(kmsg, *ret), + cflags |= io_put_kbufs(req, *ret, io_bundle_nbufs(req, *ret), issue_flags); /* bundle with no more immediate buffers, we're done */ if (req->flags & REQ_F_BL_EMPTY) From patchwork Mon Aug 12 16:55:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13760859 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 15D2713AA47 for ; Mon, 12 Aug 2024 17:00:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723482061; cv=none; b=ZOlMn4nZNtpEOZ04V9V0S99xX3XiCvb1Ti3zf4KynMRLEboF4vTB4JN40i/ZVVy8YYBeGYYvvea82eudBqpNrJuIegdE7+gx3xZXVJi4jIDbgZ9xSsVAlsPe4/bTiPLYg8E53LZ7Wjw1QMiUL0XsNz4heXB8OnthYYI1OHPRr2k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723482061; c=relaxed/simple; bh=CEJQUJaNShFRAoYb7OXpHGCrl24JZf2hvGjozUcru2M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YTkOGun1w0KagCS0qX018lwak8WltWlSIXYXHg7nPcGuXlRjgITggD8B4P/QuTiM1ptkGLJ7lcJSrscv1OT7A9XwP/6wH+THmerREKuX9Lt3ELHUIH/9qD21cuyjB6Ay8jPLND4twwZ8kANtJg7GYbhhu6b1g839g0aNvcOK1Ak= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=AMHt7BVC; arc=none smtp.client-ip=209.85.214.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="AMHt7BVC" Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-1fc5a6147f3so1981935ad.2 for ; Mon, 12 Aug 2024 10:00:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1723482058; x=1724086858; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wYiaDg8G3wV+GXOQbFZjaRsZK4kWIVmhgslAB3aLK3Y=; b=AMHt7BVC2cEhdmb6KDSEeDMTmo2uO2DvcVipJTmJ0EVj/jLjh+jz96jOejaBbmxOGA aAaBkxVia9r3TL7KWBqZKiE5UKtYJNva8VUn4Ihg1EioBZm9BKmJBb5m/NPglqobBWzl uc2emolC+gxxqQ2hSIyVdymeNIQytyC8rT1LoalrXkEnN9bvNga6Un02q0VbgnnrF7YH oGlDCQY3WBGjT+nmMsdWO7iylRUvMFx/wnzHguWpdVJSjXwn38UcbYQts+u0HoslLSGM kolRVa8/sgr7zWUbxSJzlA7yMr8OUFrPDrBOsgXoKeIIDS+Ll0LUpQrSXwsbmzjvpX0v u82w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723482058; x=1724086858; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wYiaDg8G3wV+GXOQbFZjaRsZK4kWIVmhgslAB3aLK3Y=; b=gH5esmkU8aV649UEf7TLCjDlDTNWu6vb4ORHydCZqriFztpQrvccuUj60qOnPnAz8v MUTTZdBOBHiAQmlffmfGNFKysrAnTBBWc2hmln3/suIkSjvoIoOJofhlKa6775iflz42 acMB5UnIE46bCImdGVDVv+XlIrLeKb8R/XG8fk6PiPRNx+xdFBwo+elNtI7zIFSBv6HZ UfW/p40McImjUh+7im0BKpAGs0keOQrms/zicC9WinVdezbxlDDZhDLcRxagZWN9uYq3 W0LdrsVeB4oLkAOVUWYyuAxzWzOlnuKiWgMXz/1FRnjNSvxDEExPqqN+xAqxay5iNVRG 8Law== X-Gm-Message-State: AOJu0Yxdl6PPJjOyFHk4RPQFZSnh33tSmah29W764nMzaVS6xKi9J65w 3qmx3ElzIglHOILlv0uzE6Zll0cYH/D8GSXXClZS3GrDTY3cuxrFBXy/Av6kqqZyBgVHSc5+CSF r X-Google-Smtp-Source: AGHT+IFGG9O47Xicpgk8ao1fk5Sn4Oak9q9cG0urqa70LijIy568merHJCATWHvPHpajQ2sPnY31jw== X-Received: by 2002:a17:902:f20b:b0:1fc:5b41:bac9 with SMTP id d9443c01a7336-201ca1b1405mr6313885ad.7.1723482057826; Mon, 12 Aug 2024 10:00:57 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-200bb9feaa4sm40212725ad.213.2024.08.12.10.00.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Aug 2024 10:00:55 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 3/4] io_uring/kbuf: shrink nr_iovs/mode in struct buf_sel_arg Date: Mon, 12 Aug 2024 10:55:26 -0600 Message-ID: <20240812170044.93133-4-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240812170044.93133-1-axboe@kernel.dk> References: <20240812170044.93133-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 nr_iovs is capped at 1024, and mode only has a few low values. We can safely make them u16, in preparation for adding a few more members. Signed-off-by: Jens Axboe --- io_uring/kbuf.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/io_uring/kbuf.h b/io_uring/kbuf.h index 1d56092d9286..5625ff0e349d 100644 --- a/io_uring/kbuf.h +++ b/io_uring/kbuf.h @@ -59,8 +59,8 @@ struct buf_sel_arg { struct iovec *iovs; size_t out_len; size_t max_len; - int nr_iovs; - int mode; + unsigned short nr_iovs; + unsigned short mode; }; void __user *io_buffer_select(struct io_kiocb *req, size_t *len, From patchwork Mon Aug 12 16:55:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13760860 Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 21F0C186E3B for ; Mon, 12 Aug 2024 17:01:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723482063; cv=none; b=NSr4PO8plfPgJjMH63FBnoVVDKXxtjl0giGoA9jfWfMZuo8dHUrdN+I/rWONbVEjphnH4j8WwNJqWDJ2NVLAHZ6I7PqSS2uQw9lPSOPmMoNn092Lzwqas2xrBTbm5TLOpl1X4j9M/fhVSKGe/8T4VPhpSNzfkXnkMlvs6gbfRy0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723482063; c=relaxed/simple; bh=w341BamM6UwwBNwzUvYf/Bw+djnxjRdy3qmZgJHKG60=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=GW0pnnWO+T4UCifYBvHfggv69Z5zIlaBHlQuXwajkHAoM56YwgFnRA4vGeurdAoTrb2DGHGyXz1xTbWNwYGIGr1kGrRK5a7iWytyPNdHgvxxQaIpKhdiexLrzPaTdAZ4EHGjgnF1HBjB0APXCFvggce8I8Y+TQC74hSGtPK5ImU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=AAvqTT9b; arc=none smtp.client-ip=209.85.215.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="AAvqTT9b" Received: by mail-pg1-f177.google.com with SMTP id 41be03b00d2f7-78f86e56b4cso782476a12.3 for ; Mon, 12 Aug 2024 10:01:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1723482060; x=1724086860; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=NESc3W7wBpHoPLhXz9rZPuRs7hYDx9uq56Y8V2ZMbSo=; b=AAvqTT9bpCHsSSPQZjUawEKo5f7srPrsWuj1EYrDeLKia+StgHEJFkCLMuOdcyrb9t g1w9gVGV9kmL95FHjFk/RtK7GIPtGISuNSLddQriIfWPR0aOTLLLAi773cZOfkkZFV8B sSNIei1nYPYLSlR0Mmt+ckj87xIK4mb1WEFb5epBZQORoJSQRKLndNtsjypmKkNYIkNo KrsQVTtMmtJFQvuBZKzIZNy60Ww5N/aK601jjWs1V6GCOYDrqLhT09DWWeS6R/GuWAnu VXhO1o0tnwO1uhp6HKT+pUOyUvEH388tQWmJio1UMiALzfdrWqpRnDuXTSgquEmW2gWd 7J2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723482060; x=1724086860; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NESc3W7wBpHoPLhXz9rZPuRs7hYDx9uq56Y8V2ZMbSo=; b=f3eAPacv/dIANvXThHMfsDwlFf45wnVVGMRwKqJoL9QHT475c9iO/eYQ/y9yWdxUM8 E8M+5lvjZFt/VzBC8QAWudrI+/ypHMdYTFpvG/LKxZNUUt/rOdTQ1CWW7XtwthaN0Q5X pZZJGo/anQMBjV/UYOvY9oee8nE8QCM9eDFW4ty7fCqZYe8oTSu0rNvTpVv+4wdURuAG ZajmCJiy6Rd8k4kkzL3lUbzMTiV2IrJPMcGz2GZeNNvKIJoWvX7EblcWXF1rkJqAQFas pNQHN8wO2W5/R/pB3tdBA0H+jrcsBcnpJRFYigKBOZjEmMZXM7UxXwOdO+sLh5ePXcAV 5oGQ== X-Gm-Message-State: AOJu0YzayQFZVx6VbyE3GY7T8l5MR4oVOSxrYRhqvoYSD5kGc9Kk1eB/ ysjwSg6kfWo/AWkjcVh/nHrvS3+Arx8IfzPbFrqhpq7D7t3/uyUdU9nIL0DhiPrwkDsnAvfHiIU J X-Google-Smtp-Source: AGHT+IHaPcJnev9bJ9d1Y9OMvmNPmptHCK7rJSgbdYEOoBLxwHh39bfcG+LBibiUALZhCKh/AYUVTg== X-Received: by 2002:a17:902:d4c9:b0:1fc:4377:e6ea with SMTP id d9443c01a7336-201ca1cb2a3mr6165285ad.9.1723482059865; Mon, 12 Aug 2024 10:00:59 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-200bb9feaa4sm40212725ad.213.2024.08.12.10.00.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Aug 2024 10:00:58 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 4/4] io_uring/net: allow coalescing of mapped segments Date: Mon, 12 Aug 2024 10:55:27 -0600 Message-ID: <20240812170044.93133-5-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240812170044.93133-1-axboe@kernel.dk> References: <20240812170044.93133-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 For bundles, when multiple buffers are selected, it's not unlikely that some/all of them will be virtually contigious. If these segments aren't big, then nice wins can be reaped by coalescing them into bigger segments. This makes networking copies more efficient, and reduces the number of iterations that need to be done over an iovec. Ideally, multiple segments that would've been mapped as an ITER_IOVEC before can now be mapped into a single ITER_UBUF iterator. Example from an io_uring network backend receiving data, with various transfer sizes, over a 100G network link. recv size coalesce threads bw cpu usage bw diff ===================================================================== 64 0 1 23GB/sec 100% 64 1 1 46GB/sec 79% +100% 64 0 4 81GB/sec 370% 64 1 4 96GB/sec 160% + 20% 256 0 1 44GB/sec 90% 256 1 1 47GB/sec 48% + 7% 256 0 4 90GB/sec 190% 256 1 4 96GB/sec 120% + 7% 1024 0 1 49GB/sec 60% 1024 1 1 50GB/sec 53% + 2% 1024 0 4 94GB/sec 140% 1024 1 4 96GB/sec 120% + 2% where obviously small buffer sizes benefit the most, but where an efficiency gain is seen even at higher buffer sizes as well. Signed-off-by: Jens Axboe --- io_uring/kbuf.c | 61 +++++++++++++++++++++++++++++++++++++++++++++---- io_uring/kbuf.h | 3 +++ io_uring/net.c | 48 ++++++++++++++++---------------------- io_uring/net.h | 1 + 4 files changed, 80 insertions(+), 33 deletions(-) diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index 794a687d8589..76eade5a5567 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -129,6 +129,7 @@ static int io_provided_buffers_select(struct io_kiocb *req, size_t *len, arg->iovs[0].iov_base = buf; arg->iovs[0].iov_len = *len; + arg->nsegs = 1; return 0; } @@ -194,11 +195,16 @@ void __user *io_buffer_select(struct io_kiocb *req, size_t *len, /* cap it at a reasonable 256, will be one page even for 4K */ #define PEEK_MAX_IMPORT 256 +/* + * Returns how many iovecs were used to fill the range. arg->nsegs contains + * the number of buffers mapped, which may be less than the return value if + * segments were coalesced. + */ static int io_ring_buffers_peek(struct io_kiocb *req, struct buf_sel_arg *arg, struct io_buffer_list *bl) { struct io_uring_buf_ring *br = bl->buf_ring; - struct iovec *iov = arg->iovs; + struct iovec *prev_iov, *iov = arg->iovs; int nr_iovs = arg->nr_iovs; __u16 nr_avail, tail, head; struct io_uring_buf *buf; @@ -239,9 +245,11 @@ static int io_ring_buffers_peek(struct io_kiocb *req, struct buf_sel_arg *arg, if (!arg->max_len) arg->max_len = INT_MAX; + prev_iov = NULL; req->buf_index = buf->bid; do { int len = buf->len; + void __user *ubuf; /* truncate end piece, if needed */ if (len > arg->max_len) { @@ -250,10 +258,20 @@ static int io_ring_buffers_peek(struct io_kiocb *req, struct buf_sel_arg *arg, buf->len = len; } - iov->iov_base = u64_to_user_ptr(buf->addr); - iov->iov_len = len; - iov++; + ubuf = u64_to_user_ptr(buf->addr); + if (prev_iov && + prev_iov->iov_base + prev_iov->iov_len == ubuf && + prev_iov->iov_len + len <= INT_MAX) { + prev_iov->iov_len += len; + } else { + iov->iov_base = ubuf; + iov->iov_len = len; + if (arg->coalesce) + prev_iov = iov; + iov++; + } + arg->nsegs++; arg->out_len += len; arg->max_len -= len; if (!arg->max_len) @@ -266,7 +284,8 @@ static int io_ring_buffers_peek(struct io_kiocb *req, struct buf_sel_arg *arg, req->flags |= REQ_F_BL_EMPTY; req->flags |= REQ_F_BUFFER_RING; - req->buf_list = bl; + if (arg->coalesce) + req->buf_list = bl; return iov - arg->iovs; } @@ -326,6 +345,38 @@ int io_buffers_peek(struct io_kiocb *req, struct buf_sel_arg *arg) return io_provided_buffers_select(req, &arg->max_len, bl, arg); } +int io_buffer_segments(struct io_kiocb *req, int nbytes) +{ + struct io_uring_buf_ring *br; + struct io_buffer_list *bl; + int nbufs = 0; + unsigned bid; + + /* + * Safe to use ->buf_list here, as coalescing can only have happened + * if we remained lock throughout the operation. Unlocked usage must + * not have buf_sel_arg->coalesce set to true + */ + bl = req->buf_list; + if (unlikely(!bl || !(bl->flags & IOBL_BUF_RING))) + return 1; + + bid = req->buf_index; + br = bl->buf_ring; + do { + struct io_uring_buf *buf; + int this_len; + + buf = io_ring_head_to_buf(br, bid, bl->mask); + this_len = min_t(int, buf->len, nbytes); + nbufs++; + bid++; + nbytes -= this_len; + } while (nbytes); + + return nbufs; +} + static int __io_remove_buffers(struct io_ring_ctx *ctx, struct io_buffer_list *bl, unsigned nbufs) { diff --git a/io_uring/kbuf.h b/io_uring/kbuf.h index 5625ff0e349d..d6d5f936fe6f 100644 --- a/io_uring/kbuf.h +++ b/io_uring/kbuf.h @@ -61,8 +61,11 @@ struct buf_sel_arg { size_t max_len; unsigned short nr_iovs; unsigned short mode; + unsigned short nsegs; + bool coalesce; }; +int io_buffer_segments(struct io_kiocb *req, int nbytes); void __user *io_buffer_select(struct io_kiocb *req, size_t *len, unsigned int issue_flags); int io_buffers_select(struct io_kiocb *req, struct buf_sel_arg *arg, diff --git a/io_uring/net.c b/io_uring/net.c index a6268e62b348..f15671f5b118 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -462,33 +462,16 @@ static void io_req_msg_cleanup(struct io_kiocb *req, static int io_bundle_nbufs(struct io_kiocb *req, int ret) { struct io_async_msghdr *kmsg = req->async_data; - struct iovec *iov; - int nbufs; - /* no data is always zero segments, and a ubuf is always 1 segment */ + /* no data is always zero segments */ if (ret <= 0) return 0; - if (iter_is_ubuf(&kmsg->msg.msg_iter)) - return 1; - - iov = kmsg->free_iov; - if (!iov) - iov = &kmsg->fast_iov; - - /* if all data was transferred, it's basic pointer math */ + /* if all data was transferred, we already know the number of buffers */ if (!iov_iter_count(&kmsg->msg.msg_iter)) - return iter_iov(&kmsg->msg.msg_iter) - iov; - - /* short transfer, count segments */ - nbufs = 0; - do { - int this_len = min_t(int, iov[nbufs].iov_len, ret); - - nbufs++; - ret -= this_len; - } while (ret); + return kmsg->nbufs; - return nbufs; + /* short transfer, iterate buffers to find number of segments */ + return io_buffer_segments(req, ret); } static inline bool io_send_finish(struct io_kiocb *req, int *ret, @@ -602,6 +585,7 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) .iovs = &kmsg->fast_iov, .max_len = INT_MAX, .nr_iovs = 1, + .coalesce = !(issue_flags & IO_URING_F_UNLOCKED), }; if (kmsg->free_iov) { @@ -625,6 +609,7 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) req->flags |= REQ_F_NEED_CLEANUP; } sr->len = arg.out_len; + kmsg->nbufs = arg.nsegs; if (ret == 1) { sr->buf = arg.iovs[0].iov_base; @@ -1080,6 +1065,7 @@ static int io_recv_buf_select(struct io_kiocb *req, struct io_async_msghdr *kmsg .iovs = &kmsg->fast_iov, .nr_iovs = 1, .mode = KBUF_MODE_EXPAND, + .coalesce = true, }; if (kmsg->free_iov) { @@ -1095,7 +1081,18 @@ static int io_recv_buf_select(struct io_kiocb *req, struct io_async_msghdr *kmsg if (unlikely(ret < 0)) return ret; - /* special case 1 vec, can be a fast path */ + if (arg.iovs != &kmsg->fast_iov && arg.iovs != kmsg->free_iov) { + kmsg->free_iov_nr = arg.nsegs; + kmsg->free_iov = arg.iovs; + req->flags |= REQ_F_NEED_CLEANUP; + } + kmsg->nbufs = arg.nsegs; + + /* + * Special case 1 vec, can be a fast path. Note that multiple + * contig buffers may get mapped to a single vec, but we can + * still use ITER_UBUF for those. + */ if (ret == 1) { sr->buf = arg.iovs[0].iov_base; sr->len = arg.iovs[0].iov_len; @@ -1103,11 +1100,6 @@ static int io_recv_buf_select(struct io_kiocb *req, struct io_async_msghdr *kmsg } iov_iter_init(&kmsg->msg.msg_iter, ITER_DEST, arg.iovs, ret, arg.out_len); - if (arg.iovs != &kmsg->fast_iov && arg.iovs != kmsg->free_iov) { - kmsg->free_iov_nr = ret; - kmsg->free_iov = arg.iovs; - req->flags |= REQ_F_NEED_CLEANUP; - } } else { void __user *buf; diff --git a/io_uring/net.h b/io_uring/net.h index 52bfee05f06a..b9a453da4a0f 100644 --- a/io_uring/net.h +++ b/io_uring/net.h @@ -9,6 +9,7 @@ struct io_async_msghdr { /* points to an allocated iov, if NULL we use fast_iov instead */ struct iovec *free_iov; int free_iov_nr; + int nbufs; int namelen; __kernel_size_t controllen; __kernel_size_t payloadlen;