From patchwork Mon Apr 8 01:03:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13620339 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6A413ECC for ; Mon, 8 Apr 2024 01:03:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712538235; cv=none; b=pTGrerXA4BTNOiVCaBV+phR0U7GkkO/gegmfYdIp+PoDkz+bTL1hVsNtz6CKtvDj8WNGIfV5GmB5zrST5mMdXohHMusTEuqJfndMZY0O9PDEoJmwIcvkJ91RiHRWHdxus2tG4R4uk79MB62jdmLaa2MjQOPGcfWlRIYK1r5JhDs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712538235; c=relaxed/simple; bh=Oof9IwYPFSTPeqyQDdc6SupoIfPpdAP/Y7xlesSnToE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kXAQqZcN9dv6NyzGsnrspwIkuCD7BeaEFGAKh6lWBJiDwcE9c/69EM4MltU3A0dCxYuAWAuRuEGuwnWNdWqPg1SdxvJa+PbPXVzqr9tB0optR9Ck6b2tdf9qaeUSNpebqhzMUflRF0MtDWnEixqCbiWrvteaFvCzgfze8LIwH54= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=J8hs9DS6; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="J8hs9DS6" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1712538232; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=w5kd2QgByI9r4402byTbewzH9k1skSighT9jtbHwX60=; b=J8hs9DS6k4wjHqNP/anMDJe1qRMELf0G7yGE15iP4T8WwDPyw9puX3vU40VPJIyqVT0+Ag FTtaxndhWHCUCgNcjMgb+6W4CqOvcHcTALFUmwPkr1ettgmGSb+Alp30RLpvNm7/Rei1pD vmidSzLJPCJfw8QT9qqiLvql/+gB/vI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-478-2kYpbW4pNbiLMujqyrIZrA-1; Sun, 07 Apr 2024 21:03:49 -0400 X-MC-Unique: 2kYpbW4pNbiLMujqyrIZrA-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D4976927A69; Mon, 8 Apr 2024 01:03:48 +0000 (UTC) Received: from localhost (unknown [10.72.116.148]) by smtp.corp.redhat.com (Postfix) with ESMTP id 74ACAC27EAA; Mon, 8 Apr 2024 01:03:47 +0000 (UTC) From: Ming Lei To: Jens Axboe , io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Pavel Begunkov , Kevin Wolf , Ming Lei Subject: [PATCH 1/9] io_uring: net: don't check sqe->__pad2[0] for send zc Date: Mon, 8 Apr 2024 09:03:14 +0800 Message-ID: <20240408010322.4104395-2-ming.lei@redhat.com> In-Reply-To: <20240408010322.4104395-1-ming.lei@redhat.com> References: <20240408010322.4104395-1-ming.lei@redhat.com> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.8 send_zc never uses any byte in sqe->__pad2[], and the check is unnecessary, so remove it and prepare for using the last byte as sqe extended flags. Signed-off-by: Ming Lei --- io_uring/net.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/io_uring/net.c b/io_uring/net.c index 97d815d13b6a..9c0567892945 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -997,7 +997,7 @@ int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) zc->done_io = 0; - if (unlikely(READ_ONCE(sqe->__pad2[0]) || READ_ONCE(sqe->addr3))) + if (unlikely(READ_ONCE(sqe->addr3))) return -EINVAL; /* we don't support IOSQE_CQE_SKIP_SUCCESS just yet */ if (req->flags & REQ_F_CQE_SKIP) From patchwork Mon Apr 8 01:03:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13620340 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 20CC163A for ; Mon, 8 Apr 2024 01:04:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712538242; cv=none; b=QYM3NAAt3Xbt+MMuUzUCUZnVGdzcGMW6sV5QVZWqHtcyLzsiXS0iZgXmwYGu3dBqDq8j4s2p1ob7Xbuuwh9vo8mVid2+MrBR5mgBiJYoX88RXCDJPVZ0F52B+nfOz18dx76fSAyXc92+RN87T/O0BlDhIwkqB59hHsZxoP6wTpc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712538242; c=relaxed/simple; bh=vAU4WfxbE5asbbG9GUSc01+FP7Xfa9wIs5vwZzbHqbo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=A/pRTYhjmCGexlc0K3GcJ1p3tWCVhjUAqr8VcbhW7lKzliqGhsnE65b17xOEioVw2QpRGZFkAgtE+IiV25pi9IUj/wt1VGUHQRHXfgXpxOscdFMm581by+AWgkYtzALsVT3oinAxkd1nmbKZOdPg3aSmSrBaj4NcIqUWHLi4Sfo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=U+fRw8AX; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="U+fRw8AX" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1712538240; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qM6NHLD66oZBDenxTut2lYwJA+IsUNu2ImiCulPd8c0=; b=U+fRw8AXPfoI6JgZsMiOU7mDoP3Wyj3k8lAdZqiqVw14opCknzciUKeYCcmFZqEPkVhv+d GD4KBfAAuAn0xeIWdB5GruLk90lpnoeLCjMtK4D+wLUHDcplyIyAoRyjTVuBbQqoOPcbJJ bKmkUmQMtJXNyztTf3hpW1oZGsNHCQk= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-20-KFvhzIdkOgeYE-3bWPYPYQ-1; Sun, 07 Apr 2024 21:03:57 -0400 X-MC-Unique: KFvhzIdkOgeYE-3bWPYPYQ-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A9DAB29AC02D; Mon, 8 Apr 2024 01:03:56 +0000 (UTC) Received: from localhost (unknown [10.72.116.148]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4B270210FD1B; Mon, 8 Apr 2024 01:03:54 +0000 (UTC) From: Ming Lei To: Jens Axboe , io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Pavel Begunkov , Kevin Wolf , Ming Lei Subject: [PATCH 2/9] io_uring: support user sqe ext flags Date: Mon, 8 Apr 2024 09:03:15 +0800 Message-ID: <20240408010322.4104395-3-ming.lei@redhat.com> In-Reply-To: <20240408010322.4104395-1-ming.lei@redhat.com> References: <20240408010322.4104395-1-ming.lei@redhat.com> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.6 sqe->flags is u8, and now we have used 7 bits, so take the last one for extending purpose. If bit7(IOSQE_HAS_EXT_FLAGS_BIT) is 1, it means this sqe carries ext flags from the last byte(.ext_flags), or bit23~bit16 of sqe->uring_cmd_flags for IORING_OP_URING_CMD. io_slot_flags() return value is converted to `ULL` because the affected bits are beyond 32bit now. Signed-off-by: Ming Lei --- include/linux/io_uring_types.h | 6 ++++-- include/uapi/linux/io_uring.h | 13 +++++++++++++ io_uring/filetable.h | 2 +- io_uring/io_uring.c | 14 +++++++++++++- io_uring/uring_cmd.c | 3 ++- 5 files changed, 33 insertions(+), 5 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 3e72fa52f1e3..67347e5d06ec 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -435,6 +435,7 @@ struct io_tw_state { }; enum { + /* 1st byte is from sqe->flags, and 2nd is from sqe ext_flags */ REQ_F_FIXED_FILE_BIT = IOSQE_FIXED_FILE_BIT, REQ_F_IO_DRAIN_BIT = IOSQE_IO_DRAIN_BIT, REQ_F_LINK_BIT = IOSQE_IO_LINK_BIT, @@ -442,9 +443,10 @@ enum { REQ_F_FORCE_ASYNC_BIT = IOSQE_ASYNC_BIT, REQ_F_BUFFER_SELECT_BIT = IOSQE_BUFFER_SELECT_BIT, REQ_F_CQE_SKIP_BIT = IOSQE_CQE_SKIP_SUCCESS_BIT, + REQ_F_SQE_EXT_FLAGS_BIT = IOSQE_HAS_EXT_FLAGS_BIT, - /* first byte is taken by user flags, shift it to not overlap */ - REQ_F_FAIL_BIT = 8, + /* first 2 bytes are taken by user flags, shift it to not overlap */ + REQ_F_FAIL_BIT = 16, REQ_F_INFLIGHT_BIT, REQ_F_CUR_POS_BIT, REQ_F_NOWAIT_BIT, diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index a7f847543a7f..4847d7cf1ac9 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -98,6 +98,10 @@ struct io_uring_sqe { __u64 __pad2[1]; }; __u64 optval; + struct { + __u8 __pad4[15]; + __u8 ext_flags; + }; /* * If the ring is initialized with IORING_SETUP_SQE128, then * this field is used for 80 bytes of arbitrary command data @@ -123,6 +127,7 @@ enum io_uring_sqe_flags_bit { IOSQE_ASYNC_BIT, IOSQE_BUFFER_SELECT_BIT, IOSQE_CQE_SKIP_SUCCESS_BIT, + IOSQE_HAS_EXT_FLAGS_BIT, }; /* @@ -142,6 +147,11 @@ enum io_uring_sqe_flags_bit { #define IOSQE_BUFFER_SELECT (1U << IOSQE_BUFFER_SELECT_BIT) /* don't post CQE if request succeeded */ #define IOSQE_CQE_SKIP_SUCCESS (1U << IOSQE_CQE_SKIP_SUCCESS_BIT) +/* + * sqe ext flags carried in the last byte, or bit23~bit16 of + * sqe->uring_cmd_flags for IORING_URING_CMD. + */ +#define IOSQE_HAS_EXT_FLAGS (1U << IOSQE_HAS_EXT_FLAGS_BIT) /* * io_uring_setup() flags @@ -263,11 +273,14 @@ enum io_uring_op { /* * sqe->uring_cmd_flags top 8bits aren't available for userspace + * bit31 ~ bit24 kernel internal usage + * bit23 ~ bit16 sqe ext flags * IORING_URING_CMD_FIXED use registered buffer; pass this flag * along with setting sqe->buf_index. */ #define IORING_URING_CMD_FIXED (1U << 0) #define IORING_URING_CMD_MASK IORING_URING_CMD_FIXED +#define IORING_URING_CMD_EXT_MASK 0x00ff0000 /* diff --git a/io_uring/filetable.h b/io_uring/filetable.h index b2435c4dca1f..d25247c9b9f5 100644 --- a/io_uring/filetable.h +++ b/io_uring/filetable.h @@ -43,7 +43,7 @@ io_fixed_file_slot(struct io_file_table *table, unsigned i) #define FFS_ISREG 0x2UL #define FFS_MASK ~(FFS_NOWAIT|FFS_ISREG) -static inline unsigned int io_slot_flags(struct io_fixed_file *slot) +static inline unsigned long io_slot_flags(struct io_fixed_file *slot) { return (slot->file_ptr & ~FFS_MASK) << REQ_F_SUPPORT_NOWAIT_BIT; } diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 8df9ad010803..6d4def11aebf 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -109,7 +109,8 @@ IOSQE_IO_HARDLINK | IOSQE_ASYNC) #define SQE_VALID_FLAGS (SQE_COMMON_FLAGS | IOSQE_BUFFER_SELECT | \ - IOSQE_IO_DRAIN | IOSQE_CQE_SKIP_SUCCESS) + IOSQE_IO_DRAIN | IOSQE_CQE_SKIP_SUCCESS | \ + IOSQE_HAS_EXT_FLAGS) #define IO_REQ_CLEAN_FLAGS (REQ_F_BUFFER_SELECTED | REQ_F_NEED_CLEANUP | \ REQ_F_POLLED | REQ_F_INFLIGHT | REQ_F_CREDS | \ @@ -2080,6 +2081,17 @@ static int io_init_req(struct io_ring_ctx *ctx, struct io_kiocb *req, /* enforce forwards compatibility on users */ if (sqe_flags & ~SQE_VALID_FLAGS) return io_init_fail_req(req, -EINVAL); + if (sqe_flags & IOSQE_HAS_EXT_FLAGS) { + u32 sqe_ext_flags; + + if (opcode != IORING_OP_URING_CMD) + sqe_ext_flags = READ_ONCE(sqe->ext_flags); + else + sqe_ext_flags = (READ_ONCE(sqe->uring_cmd_flags) + & IORING_URING_CMD_EXT_MASK) >> 16; + req->flags |= sqe_ext_flags << 8; + } + if (sqe_flags & IOSQE_BUFFER_SELECT) { if (!def->buffer_select) return io_init_fail_req(req, -EOPNOTSUPP); diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c index 334d31dd6628..43b71f29e7b3 100644 --- a/io_uring/uring_cmd.c +++ b/io_uring/uring_cmd.c @@ -202,7 +202,8 @@ int io_uring_cmd_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) if (sqe->__pad1) return -EINVAL; - ioucmd->flags = READ_ONCE(sqe->uring_cmd_flags); + ioucmd->flags = READ_ONCE(sqe->uring_cmd_flags) & + ~IORING_URING_CMD_EXT_MASK; if (ioucmd->flags & ~IORING_URING_CMD_MASK) return -EINVAL; From patchwork Mon Apr 8 01:03:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13620341 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8710E17CD for ; Mon, 8 Apr 2024 01:04:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712538251; cv=none; b=husszPJxylwetWMtriqj5LeE0b4xBmE0q8L2c23XyiKFm2TnKBMyHGcZRXfU9HONsB+kjbK2qE8JFM1bShf7POlP86+vv8d05ni1fywEBHSOifxs8xGEb3Cs4tHu4yjJymaUVPGiXDcDM2SdF55TIdOY8GG4uC9DoJHrO09qUIU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712538251; c=relaxed/simple; bh=guAhS/Fnx0PSMY2Pw5vSYBO2D5nZIMb7GKoSEHwTMx0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=t6qo6nqiHm7GXKIjgnB7M+H/uRdlkn3RLth3BbbmoJkJK9zs/cDQLDuqZQUEy7qXXptphGEd391xv8yw+bEAztq3mvmvSyp0zAFoU4o0TXQqLSubizN997uLO0FqflqWVZAQnUPIm/+tlGpIxRt/403HDZa+R/4GTrO2r+atIoM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Czm8M7/y; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Czm8M7/y" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1712538248; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eK5sUnEeiBFgGX0FYS4kDF4HFOxZuYXlVr4lMLlEA34=; b=Czm8M7/yML1retUXuCg7qT8cpYMVnvjIJmjFHwlHvcPwBzcUmvc0cnSgdMeu1dbgn4NZ0d TVGr2jY2j5LHlEMVNOTiGfJ+gYeBnI4hdjAqLmtZwkY2ra6XTCUvANRUBfJQ1qnFVXr//7 DI9gBUwNiB6U2bu/o78e5P3CqxvZCdY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-536-6nY_BZUJMmio7HPMkqMbBw-1; Sun, 07 Apr 2024 21:04:05 -0400 X-MC-Unique: 6nY_BZUJMmio7HPMkqMbBw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 14FDA10499A1; Mon, 8 Apr 2024 01:04:04 +0000 (UTC) Received: from localhost (unknown [10.72.116.148]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1ED1923D61; Mon, 8 Apr 2024 01:04:02 +0000 (UTC) From: Ming Lei To: Jens Axboe , io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Pavel Begunkov , Kevin Wolf , Ming Lei Subject: [PATCH 3/9] io_uring: add helper for filling cqes in __io_submit_flush_completions() Date: Mon, 8 Apr 2024 09:03:16 +0800 Message-ID: <20240408010322.4104395-4-ming.lei@redhat.com> In-Reply-To: <20240408010322.4104395-1-ming.lei@redhat.com> References: <20240408010322.4104395-1-ming.lei@redhat.com> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.5 No functional change, and prepare for supporting SQE group. Signed-off-by: Ming Lei --- io_uring/io_uring.c | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 6d4def11aebf..c73819c04c0b 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1458,16 +1458,14 @@ static void io_free_batch_list(struct io_ring_ctx *ctx, } while (node); } -void __io_submit_flush_completions(struct io_ring_ctx *ctx) - __must_hold(&ctx->uring_lock) +static inline void io_fill_cqe_lists(struct io_ring_ctx *ctx, + struct io_wq_work_list *list) { - struct io_submit_state *state = &ctx->submit_state; struct io_wq_work_node *node; - __io_cq_lock(ctx); - __wq_list_for_each(node, &state->compl_reqs) { + __wq_list_for_each(node, list) { struct io_kiocb *req = container_of(node, struct io_kiocb, - comp_list); + comp_list); if (!(req->flags & REQ_F_CQE_SKIP) && unlikely(!io_fill_cqe_req(ctx, req))) { @@ -1480,6 +1478,15 @@ void __io_submit_flush_completions(struct io_ring_ctx *ctx) } } } +} + +void __io_submit_flush_completions(struct io_ring_ctx *ctx) + __must_hold(&ctx->uring_lock) +{ + struct io_submit_state *state = &ctx->submit_state; + + __io_cq_lock(ctx); + io_fill_cqe_lists(ctx, &state->compl_reqs); __io_cq_unlock_post(ctx); if (!wq_list_empty(&ctx->submit_state.compl_reqs)) { From patchwork Mon Apr 8 01:03:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13620342 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 58C3D10E9 for ; Mon, 8 Apr 2024 01:04:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712538255; cv=none; b=r8yrM7U7AogUL9OYlh0fgOeZW8HZh1w9BaHmGFTIYgq0WOCyCdHP4+Q0bjGDXt/3lb/W5ksiOXdd9+5gfk1OIOXb4FIZww1GJtx9MA1nK5L3RNIsAvfymDyGBAiWd6+3NG9hHIfQTYfqh+yAg/ncHel2wlkCFiTeWOOWkUIs57Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712538255; c=relaxed/simple; bh=A0m9RFVAWRdlX79tNuHAOT77nPe+Jqk1pLUJTU7vgzo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Qy0VIH9RNqVY8BQ8dIhc7CNhEWD2lkRw7zb8qQwJPPixGqycWCrzkip645hLXl6mgQGAXE3tNJ0uvDJ9rlQlNuyO3+ivnrWY5Ee1UJ2CYJ4cJc5ws6Q4PSNbnGgU15IulOGt8c+sb2TDFabWuhUsw809pwTjDSFbJq95R3LReO4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=G+GKbx9y; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="G+GKbx9y" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1712538253; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=McDk3PqX7U+JoL3gMgs97ctrnawJhamozIBAtYPTYWc=; b=G+GKbx9yor4vwfmTBYfP4cmblt/W72krgfCRhy/W4ZeZWAfmIq91qaIRRaSKsiIWsuY4IN XGHglJZvd7UJaWYqS3UGaerwmZmn1FPaPU6Bi9YMiwF02c0ZWYKs0rHYCZsdFKMruhbxI7 T/gQapdLCS1FH5pnD6MBLyIRHN/B1vI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-606-rmwUIfAwMY-jOdCs2OWGAQ-1; Sun, 07 Apr 2024 21:04:11 -0400 X-MC-Unique: rmwUIfAwMY-jOdCs2OWGAQ-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A03F1927A6B; Mon, 8 Apr 2024 01:04:10 +0000 (UTC) Received: from localhost (unknown [10.72.116.148]) by smtp.corp.redhat.com (Postfix) with ESMTP id B4C31444585; Mon, 8 Apr 2024 01:04:09 +0000 (UTC) From: Ming Lei To: Jens Axboe , io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Pavel Begunkov , Kevin Wolf , Ming Lei Subject: [PATCH 4/9] io_uring: add one output argument to io_submit_sqe Date: Mon, 8 Apr 2024 09:03:17 +0800 Message-ID: <20240408010322.4104395-5-ming.lei@redhat.com> In-Reply-To: <20240408010322.4104395-1-ming.lei@redhat.com> References: <20240408010322.4104395-1-ming.lei@redhat.com> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.10 Add one output argument to io_submit_sqe() for returning how many SQEs handled in this function. Prepare for supporting SQE group, which can include multiple member SQEs to handle in io_submit_sqe(). No functional change. Signed-off-by: Ming Lei --- io_uring/io_uring.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index c73819c04c0b..4969d21ea2f8 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2204,12 +2204,13 @@ static __cold int io_submit_fail_init(const struct io_uring_sqe *sqe, } static inline int io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req, - const struct io_uring_sqe *sqe) + const struct io_uring_sqe *sqe, unsigned int *nr) __must_hold(&ctx->uring_lock) { struct io_submit_link *link = &ctx->submit_state.link; int ret; + *nr = 1; ret = io_init_req(ctx, req, sqe); if (unlikely(ret)) return io_submit_fail_init(sqe, req, ret); @@ -2351,6 +2352,7 @@ int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr) do { const struct io_uring_sqe *sqe; struct io_kiocb *req; + unsigned int done; if (unlikely(!io_alloc_req(ctx, &req))) break; @@ -2363,12 +2365,13 @@ int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr) * Continue submitting even for sqe failure if the * ring was setup with IORING_SETUP_SUBMIT_ALL */ - if (unlikely(io_submit_sqe(ctx, req, sqe)) && + if (unlikely(io_submit_sqe(ctx, req, sqe, &done)) && !(ctx->flags & IORING_SETUP_SUBMIT_ALL)) { - left--; + left -= done; break; } - } while (--left); + left -= done; + } while (left); if (unlikely(left)) { ret -= left; From patchwork Mon Apr 8 01:03:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13620343 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 264831860 for ; Mon, 8 Apr 2024 01:04:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712538265; cv=none; b=CU0RU/VnsjcuuAq+ym0w+u1o4doeqJJoZsLeF2NUfu5nzNxEse/WXYE/5MGyAwCQB6Uph7zoVHMProQJPEGlNNJsl3u6TkWwQzg9tOhDYzf2Gwrg0CsJGojcOPsyztaEUTdLODTlMWMj3+SQ+ehdx4llck3sRwiCDPwSDng0LH8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712538265; c=relaxed/simple; bh=TgRa41j6isTfJsPNxy7+hMO1MyLUepmrgPPzTtLksrU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=myF+T4JRaLGQvWVElQeUs2IopkpvGfTuVCa9/6fCFhQtsvIgQH7td3/N17odQzZokLwGdEljkH9Xx57WZbjwewzaj1T3zwaIl+XRwEX1ueyHel42teAgkHUCAS9GBGx18yC/xhg5EL71Lt0RmOu0OVZI+QvZOpKu/U5M/eza5+M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Ik6rQtDM; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Ik6rQtDM" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1712538262; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gAWK8uGMA1FmyCYD/3aI7hQBXaYJIwKm7EJ5DymaJ1o=; b=Ik6rQtDMHnlwnAwMX7t5JhyukjrkJhEyHt8lHxZPstsUFMnpnfKiyJEO9V71VNjIPMe1qN 7D6WvLt5hQevGd5ycwnSP/JsqT4wbd5SV8rXcnwcyusnPn2PFZ4AGTDj488Ggmy1pPk393 +H3Op7Ef/jX121uLgHKKqOEAJkHwLLg= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-647--cLVtox9PmS_vZoeNL-_kg-1; Sun, 07 Apr 2024 21:04:18 -0400 X-MC-Unique: -cLVtox9PmS_vZoeNL-_kg-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 54B1C8DEF78; Mon, 8 Apr 2024 01:04:18 +0000 (UTC) Received: from localhost (unknown [10.72.116.148]) by smtp.corp.redhat.com (Postfix) with ESMTP id 154BCC0157C; Mon, 8 Apr 2024 01:04:16 +0000 (UTC) From: Ming Lei To: Jens Axboe , io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Pavel Begunkov , Kevin Wolf , Ming Lei Subject: [PATCH 5/9] io_uring: support SQE group Date: Mon, 8 Apr 2024 09:03:18 +0800 Message-ID: <20240408010322.4104395-6-ming.lei@redhat.com> In-Reply-To: <20240408010322.4104395-1-ming.lei@redhat.com> References: <20240408010322.4104395-1-ming.lei@redhat.com> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.8 SQE group is defined as one chain of SQEs starting with the first sqe that has IOSQE_EXT_SQE_GROUP set, and ending with the first subsequent sqe that doesn't have it set, and it is similar with chain of linked sqes. The 1st SQE is group leader, and the other SQEs are group member. The group leader is always freed after all members are completed. Group members aren't submitted until the group leader is completed, and there isn't any dependency among group members, and IOSQE_IO_LINK can't be set for group members, same with IOSQE_IO_DRAIN. Typically the group leader provides or makes resource, and the other members consume the resource, such as scenario of multiple backup, the 1st SQE is to read data from source file into fixed buffer, the other SQEs write data from the same buffer into other destination files. SQE group provides very efficient way to complete this task: 1) fs write SQEs and fs read SQE can be submitted in single syscall, no need to submit fs read SQE first, and wait until read SQE is completed, 2) no need to link all write SQEs together, then write SQEs can be submitted to files concurrently. Meantime application is simplified a lot in this way. Another use case is to for supporting generic device zero copy: - the lead SQE is for providing device buffer, which is owned by device or kernel, can't be cross userspace, otherwise easy to cause leak for devil application or panic - member SQEs reads or writes concurrently against the buffer provided by lead SQE Signed-off-by: Ming Lei --- include/linux/io_uring_types.h | 10 ++ include/uapi/linux/io_uring.h | 4 + io_uring/io_uring.c | 218 ++++++++++++++++++++++++++++++--- io_uring/io_uring.h | 17 ++- 4 files changed, 231 insertions(+), 18 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 67347e5d06ec..7ce4a2d4a8b8 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -444,6 +444,7 @@ enum { REQ_F_BUFFER_SELECT_BIT = IOSQE_BUFFER_SELECT_BIT, REQ_F_CQE_SKIP_BIT = IOSQE_CQE_SKIP_SUCCESS_BIT, REQ_F_SQE_EXT_FLAGS_BIT = IOSQE_HAS_EXT_FLAGS_BIT, + REQ_F_SQE_GROUP_BIT = 8 + IOSQE_EXT_SQE_GROUP_BIT, /* first 2 bytes are taken by user flags, shift it to not overlap */ REQ_F_FAIL_BIT = 16, @@ -474,6 +475,7 @@ enum { REQ_F_CAN_POLL_BIT, REQ_F_BL_EMPTY_BIT, REQ_F_BL_NO_RECYCLE_BIT, + REQ_F_SQE_GROUP_LEAD_BIT, /* not a real bit, just to check we're not overflowing the space */ __REQ_F_LAST_BIT, @@ -497,6 +499,8 @@ enum { REQ_F_BUFFER_SELECT = IO_REQ_FLAG(REQ_F_BUFFER_SELECT_BIT), /* IOSQE_CQE_SKIP_SUCCESS */ REQ_F_CQE_SKIP = IO_REQ_FLAG(REQ_F_CQE_SKIP_BIT), + /* IOSQE_EXT_SQE_GROUP */ + REQ_F_SQE_GROUP = IO_REQ_FLAG(REQ_F_SQE_GROUP_BIT), /* fail rest of links */ REQ_F_FAIL = IO_REQ_FLAG(REQ_F_FAIL_BIT), @@ -552,6 +556,8 @@ enum { REQ_F_BL_EMPTY = IO_REQ_FLAG(REQ_F_BL_EMPTY_BIT), /* don't recycle provided buffers for this request */ REQ_F_BL_NO_RECYCLE = IO_REQ_FLAG(REQ_F_BL_NO_RECYCLE_BIT), + /* sqe group lead */ + REQ_F_SQE_GROUP_LEAD = IO_REQ_FLAG(REQ_F_SQE_GROUP_LEAD_BIT), }; typedef void (*io_req_tw_func_t)(struct io_kiocb *req, struct io_tw_state *ts); @@ -665,6 +671,10 @@ struct io_kiocb { u64 extra1; u64 extra2; } big_cqe; + + /* all SQE group members linked here for group lead */ + struct io_kiocb *grp_link; + atomic_t grp_refs; }; struct io_overflow_cqe { diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 4847d7cf1ac9..c0d34f2a2c17 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -128,6 +128,8 @@ enum io_uring_sqe_flags_bit { IOSQE_BUFFER_SELECT_BIT, IOSQE_CQE_SKIP_SUCCESS_BIT, IOSQE_HAS_EXT_FLAGS_BIT, + + IOSQE_EXT_SQE_GROUP_BIT = 0, }; /* @@ -152,6 +154,8 @@ enum io_uring_sqe_flags_bit { * sqe->uring_cmd_flags for IORING_URING_CMD. */ #define IOSQE_HAS_EXT_FLAGS (1U << IOSQE_HAS_EXT_FLAGS_BIT) +/* defines sqe group */ +#define IOSQE_EXT_SQE_GROUP (1U << IOSQE_EXT_SQE_GROUP_BIT) /* * io_uring_setup() flags diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 4969d21ea2f8..0f41b26723a8 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -147,6 +147,10 @@ static bool io_uring_try_cancel_requests(struct io_ring_ctx *ctx, bool cancel_all); static void io_queue_sqe(struct io_kiocb *req); +static bool io_get_sqe(struct io_ring_ctx *ctx, + const struct io_uring_sqe **sqe); +static int io_init_req(struct io_ring_ctx *ctx, struct io_kiocb *req, + const struct io_uring_sqe *sqe); struct kmem_cache *req_cachep; static struct workqueue_struct *iou_wq __ro_after_init; @@ -925,10 +929,189 @@ bool io_req_post_cqe(struct io_kiocb *req, s32 res, u32 cflags) return posted; } +static void __io_req_failed(struct io_kiocb *req, s32 res, bool skip_cqe) +{ + const struct io_cold_def *def = &io_cold_defs[req->opcode]; + + lockdep_assert_held(&req->ctx->uring_lock); + + if (skip_cqe) + req->flags |= REQ_F_FAIL | REQ_F_CQE_SKIP; + else + req_set_fail(req); + io_req_set_res(req, res, io_put_kbuf(req, IO_URING_F_UNLOCKED)); + if (def->fail) + def->fail(req); +} + +void io_req_defer_failed(struct io_kiocb *req, s32 res) + __must_hold(&ctx->uring_lock) +{ + __io_req_failed(req, res, false); + io_req_complete_defer(req); +} + +static void io_req_defer_failed_sliently(struct io_kiocb *req, s32 res) + __must_hold(&ctx->uring_lock) +{ + __io_req_failed(req, res, true); + io_req_complete_defer(req); +} + +/* + * Called after member req is completed, and return the lead request for + * caller to fill cqe & free it really + */ +static inline struct io_kiocb *io_complete_group_member(struct io_kiocb *req) +{ + struct io_kiocb *lead = req->grp_link; + + req->grp_link = NULL; + if (lead && atomic_dec_and_test(&lead->grp_refs)) { + req->grp_link = NULL; + lead->flags &= ~REQ_F_SQE_GROUP_LEAD; + return lead; + } + + return NULL; +} + +/* + * Called after lead req is completed and before posting cqe and freeing, + * for issuing member requests + */ +void io_complete_group_lead(struct io_kiocb *req, unsigned issue_flags) +{ + struct io_kiocb *member = req->grp_link; + + if (!member) + return; + + while (member) { + struct io_kiocb *next = member->grp_link; + + member->grp_link = req; + if (unlikely(req->flags & REQ_F_FAIL)) { + /* + * Now group lead is failed, so simply fail members + * with -EIO, and the application can figure out + * the reason from lead's cqe->res + */ + __io_req_failed(member, -EIO, false); + + if (issue_flags & IO_URING_F_COMPLETE_DEFER) + io_req_complete_defer(member); + else { + member->io_task_work.func = io_req_task_complete; + io_req_task_work_add(member); + } + } else { + trace_io_uring_submit_req(member); + if ((issue_flags & IO_URING_F_COMPLETE_DEFER) && + !(member->flags & REQ_F_FORCE_ASYNC)) + io_queue_sqe(member); + else + io_req_task_queue(member); + } + member = next; + } + req->grp_link = NULL; +} + +static bool sqe_is_group_member(const struct io_uring_sqe *sqe) +{ + return (READ_ONCE(sqe->flags) & IOSQE_HAS_EXT_FLAGS) && + (READ_ONCE(sqe->ext_flags) & IOSQE_EXT_SQE_GROUP); +} + +/* + * Initialize the whole SQE group. + * + * Walk every member in this group even though failure happens. If the lead + * request is failed, CQE is only posted for lead, otherwise, CQE is posted + * for every member request. Member requests aren't issued until the lead + * is completed, and the lead request won't be freed until all member + * requests are completed. + * + * The whole group shares the link flag in group lead, and other members + * aren't allowed to set any LINK flag. And only the lead request may + * appear in the submission link list. + */ +static int io_init_req_group(struct io_ring_ctx *ctx, struct io_kiocb *lead, + int *nr, int lead_ret) + __must_hold(&ctx->uring_lock) +{ + bool more = true; + int cnt = 0; + + lead->grp_link = NULL; + do { + const struct io_uring_sqe *sqe; + struct io_kiocb *req = NULL; + int ret = -ENOMEM; + + io_alloc_req(ctx, &req); + + if (unlikely(!io_get_sqe(ctx, &sqe))) { + if (req) + io_req_add_to_cache(req, ctx); + break; + } + + /* one group ends with !IOSQE_EXT_SQE_GROUP */ + if (!sqe_is_group_member(sqe)) + more = false; + + if (req) { + ret = io_init_req(ctx, req, sqe); + /* + * Both IO_LINK and IO_DRAIN aren't allowed for + * group member, and the boundary has to be in + * the lead sqe, so the whole group shares + * same IO_LINK and IO_DRAIN. + */ + if (!ret && (req->flags & (IO_REQ_LINK_FLAGS | + REQ_F_IO_DRAIN))) + ret = -EINVAL; + if (!more) + req->flags |= REQ_F_SQE_GROUP; + if (unlikely(ret)) { + /* + * The lead will be failed, so don't post + * CQE for any member + */ + io_req_defer_failed_sliently(req, ret); + } else { + req->grp_link = lead->grp_link; + lead->grp_link = req; + } + } + cnt += 1; + if (ret) + lead_ret = ret; + } while (more); + + /* Mark lead if we get members, otherwise degrade it to normal req */ + if (cnt > 0) { + lead->flags |= REQ_F_SQE_GROUP_LEAD; + atomic_set(&lead->grp_refs, cnt); + *nr += cnt; + } else { + lead->flags &= ~REQ_F_SQE_GROUP; + } + + return lead_ret; +} + static void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) { struct io_ring_ctx *ctx = req->ctx; + if (req_is_group_lead(req)) { + io_complete_group_lead(req, issue_flags); + return; + } + /* * All execution paths but io-wq use the deferred completions by * passing IO_URING_F_COMPLETE_DEFER and thus should not end up here. @@ -960,20 +1143,6 @@ static void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) req_ref_put(req); } -void io_req_defer_failed(struct io_kiocb *req, s32 res) - __must_hold(&ctx->uring_lock) -{ - const struct io_cold_def *def = &io_cold_defs[req->opcode]; - - lockdep_assert_held(&req->ctx->uring_lock); - - req_set_fail(req); - io_req_set_res(req, res, io_put_kbuf(req, IO_URING_F_UNLOCKED)); - if (def->fail) - def->fail(req); - io_req_complete_defer(req); -} - /* * Don't initialise the fields below on every allocation, but do that in * advance and keep them valid across allocations. @@ -1459,7 +1628,8 @@ static void io_free_batch_list(struct io_ring_ctx *ctx, } static inline void io_fill_cqe_lists(struct io_ring_ctx *ctx, - struct io_wq_work_list *list) + struct io_wq_work_list *list, + struct io_wq_work_list *grp) { struct io_wq_work_node *node; @@ -1477,6 +1647,13 @@ static inline void io_fill_cqe_lists(struct io_ring_ctx *ctx, io_req_cqe_overflow(req); } } + + if (grp && req_is_group_member(req)) { + struct io_kiocb *lead = io_complete_group_member(req); + + if (lead) + wq_list_add_head(&lead->comp_list, grp); + } } } @@ -1484,14 +1661,19 @@ void __io_submit_flush_completions(struct io_ring_ctx *ctx) __must_hold(&ctx->uring_lock) { struct io_submit_state *state = &ctx->submit_state; + struct io_wq_work_list list = {NULL, NULL}; __io_cq_lock(ctx); - io_fill_cqe_lists(ctx, &state->compl_reqs); + io_fill_cqe_lists(ctx, &state->compl_reqs, &list); + if (!wq_list_empty(&list)) + io_fill_cqe_lists(ctx, &list, NULL); __io_cq_unlock_post(ctx); - if (!wq_list_empty(&ctx->submit_state.compl_reqs)) { + if (!wq_list_empty(&state->compl_reqs)) { io_free_batch_list(ctx, state->compl_reqs.first); INIT_WQ_LIST(&state->compl_reqs); + if (!wq_list_empty(&list)) + io_free_batch_list(ctx, list.first); } ctx->submit_state.cq_flush = false; } @@ -2212,6 +2394,8 @@ static inline int io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req, *nr = 1; ret = io_init_req(ctx, req, sqe); + if (req->flags & REQ_F_SQE_GROUP) + ret = io_init_req_group(ctx, req, nr, ret); if (unlikely(ret)) return io_submit_fail_init(sqe, req, ret); diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 1eb65324792a..99eeb4eee219 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -104,6 +104,7 @@ bool __io_alloc_req_refill(struct io_ring_ctx *ctx); bool io_match_task_safe(struct io_kiocb *head, struct task_struct *task, bool cancel_all); +void io_complete_group_lead(struct io_kiocb *req, unsigned int issue_flags); enum { IO_EVENTFD_OP_SIGNAL_BIT, @@ -113,6 +114,16 @@ enum { void io_eventfd_ops(struct rcu_head *rcu); void io_activate_pollwq(struct io_ring_ctx *ctx); +static inline bool req_is_group_lead(struct io_kiocb *req) +{ + return req->flags & REQ_F_SQE_GROUP_LEAD; +} + +static inline bool req_is_group_member(struct io_kiocb *req) +{ + return !req_is_group_lead(req) && (req->flags & REQ_F_SQE_GROUP); +} + static inline void io_lockdep_assert_cq_locked(struct io_ring_ctx *ctx) { #if defined(CONFIG_PROVE_LOCKING) @@ -355,7 +366,11 @@ static inline void io_req_complete_defer(struct io_kiocb *req) lockdep_assert_held(&req->ctx->uring_lock); - wq_list_add_tail(&req->comp_list, &state->compl_reqs); + if (unlikely(req_is_group_lead(req))) + io_complete_group_lead(req, IO_URING_F_COMPLETE_DEFER | + IO_URING_F_NONBLOCK); + else + wq_list_add_tail(&req->comp_list, &state->compl_reqs); } static inline void io_commit_cqring_flush(struct io_ring_ctx *ctx) From patchwork Mon Apr 8 01:03:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13620344 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E3CA0BE55 for ; Mon, 8 Apr 2024 01:04:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712538289; cv=none; b=YM7odLeof/pi0DoagbjqH8m/miJg3ubBqniHxkFk4NsALTHFxvpvMQ296u7RKgEDZVB1wIcN0Ht6NipCUjDcu/L/0H8wYIMYPiCIWvDiTY00RLvlLcSUI45FbRWp+5N+FwLEWHEb6AUn5FkHDitR/UIzyd9PGyGUoVlbcD9AfwM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712538289; c=relaxed/simple; bh=guSN1HQo6pnNdn98fyI12+qKFruRdwZES/QQ9bH7k9o=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=AS7dJ/4yN53AFNYqVcLP4TAmTSbwcrdkEr3IYSDgXh12QByFd00uVTIv5JP74GgkbUfX9INdZnG90iyUuGPQI91qWvU+OrODWtyf/GzLevjDPw4mcM7A1LJ0ei5UAW5DXcIJNOndcN360ufVgvANamIyGi1Z7XymCio8qONDl2I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=YyXXc8lL; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="YyXXc8lL" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1712538287; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FRszjCsOSQdBH4b2T0C/YjzJh80frntnMrhRjjQPemw=; b=YyXXc8lLG7NcrQajJOkx4YUeH3xU68EKzFa9NPzzgwwLpty6VZrYPlY8JcapOB/8tk/X8V EVorttnoKEpLeBVxfcY0CD5C4cZF/p92iA8o5DIBWFIp6dCc2PBZTFDvgpWZwiRe5aleB0 qsASr8xPMHyWYh8L0TzS7TazyHeVQ0c= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-222-NTP9nq8mO0uSxI5wVBcbxQ-1; Sun, 07 Apr 2024 21:04:42 -0400 X-MC-Unique: NTP9nq8mO0uSxI5wVBcbxQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0BE9C888080; Mon, 8 Apr 2024 01:04:42 +0000 (UTC) Received: from localhost (unknown [10.72.116.148]) by smtp.corp.redhat.com (Postfix) with ESMTP id C086E40AE782; Mon, 8 Apr 2024 01:04:40 +0000 (UTC) From: Ming Lei To: Jens Axboe , io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Pavel Begunkov , Kevin Wolf , Ming Lei Subject: [PATCH 6/9] io_uring: support providing sqe group buffer Date: Mon, 8 Apr 2024 09:03:19 +0800 Message-ID: <20240408010322.4104395-7-ming.lei@redhat.com> In-Reply-To: <20240408010322.4104395-1-ming.lei@redhat.com> References: <20240408010322.4104395-1-ming.lei@redhat.com> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 SQE group introduces one new mechanism to share resource among one group of requests, and all member requests can consume the resource provided by group lead efficiently and concurrently. This patch uses the added sqe group feature to share kernel buffer in sqe group: - the group lead provides kernel buffer to member requests - member requests use the provided buffer to do FS or network IO, or more operations in future - this kernel buffer is returned back after member requests use it up This way looks a bit similar with kernel's pipe/splice, but there are some important differences: - splice is for transferring data between two FDs via pipe, and fd_out can only read data from pipe; this feature can borrow buffer from group lead to members, so member request can write data to this buffer if the provided buffer is allowed to write to. - splice implements data transfer by moving pages between subsystem and pipe, that means page ownership is transferred, and this way is one of the most complicated thing of splice; this patch supports scenarios in which the buffer can't be transferred, and buffer is only borrowed to member requests, and is returned back after member requests consume the provided buffer, so buffer lifetime is simplified a lot. Especially the buffer is guaranteed to be returned back. - splice can't run in async way basically It can help to implement generic zero copy between device and related operations, such as ublk, fuse, vdpa, even network receive or whatever. Signed-off-by: Ming Lei --- include/linux/io_uring_types.h | 30 ++++++++++++++++ io_uring/io_uring.c | 10 +++++- io_uring/kbuf.c | 62 ++++++++++++++++++++++++++++++++++ io_uring/kbuf.h | 12 +++++++ io_uring/net.c | 29 ++++++++++++++++ io_uring/opdef.c | 5 +++ io_uring/opdef.h | 2 ++ io_uring/rw.c | 20 ++++++++++- 8 files changed, 168 insertions(+), 2 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 7ce4a2d4a8b8..e632c3584800 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -6,6 +6,7 @@ #include #include #include +#include #include enum { @@ -39,6 +40,22 @@ enum io_uring_cmd_flags { IO_URING_F_COMPAT = (1 << 12), }; +/* buffer provided from kernel */ +struct io_uring_kernel_buf { + unsigned long len; + unsigned short nr_bvecs; + unsigned short dir; /* ITER_SOURCE or ITER_DEST */ + + /* offset in the 1st bvec */ + unsigned int offset; + const struct bio_vec *bvec; + + /* private field, user don't touch it */ + struct bio_vec __bvec[]; +}; + +typedef void (io_uring_buf_giveback_t) (const struct io_uring_kernel_buf *); + struct io_wq_work_node { struct io_wq_work_node *next; }; @@ -476,6 +493,7 @@ enum { REQ_F_BL_EMPTY_BIT, REQ_F_BL_NO_RECYCLE_BIT, REQ_F_SQE_GROUP_LEAD_BIT, + REQ_F_GROUP_KBUF_BIT, /* not a real bit, just to check we're not overflowing the space */ __REQ_F_LAST_BIT, @@ -558,6 +576,8 @@ enum { REQ_F_BL_NO_RECYCLE = IO_REQ_FLAG(REQ_F_BL_NO_RECYCLE_BIT), /* sqe group lead */ REQ_F_SQE_GROUP_LEAD = IO_REQ_FLAG(REQ_F_SQE_GROUP_LEAD_BIT), + /* group lead provides kbuf for members, set for both lead and member */ + REQ_F_GROUP_KBUF = IO_REQ_FLAG(REQ_F_GROUP_KBUF_BIT), }; typedef void (*io_req_tw_func_t)(struct io_kiocb *req, struct io_tw_state *ts); @@ -641,6 +661,15 @@ struct io_kiocb { * REQ_F_BUFFER_RING is set. */ struct io_buffer_list *buf_list; + + /* + * store kernel buffer provided by sqe group lead, valid + * IFF REQ_F_GROUP_KBUF + * + * The buffer meta is immutable since it is shared by + * all member requests + */ + const struct io_uring_kernel_buf *grp_kbuf; }; union { @@ -675,6 +704,7 @@ struct io_kiocb { /* all SQE group members linked here for group lead */ struct io_kiocb *grp_link; atomic_t grp_refs; + io_uring_buf_giveback_t *grp_kbuf_ack; }; struct io_overflow_cqe { diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 0f41b26723a8..596c4442c3c6 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -114,7 +114,7 @@ #define IO_REQ_CLEAN_FLAGS (REQ_F_BUFFER_SELECTED | REQ_F_NEED_CLEANUP | \ REQ_F_POLLED | REQ_F_INFLIGHT | REQ_F_CREDS | \ - REQ_F_ASYNC_DATA) + REQ_F_ASYNC_DATA | REQ_F_GROUP_KBUF) #define IO_REQ_CLEAN_SLOW_FLAGS (REQ_F_REFCOUNT | REQ_F_LINK | REQ_F_HARDLINK |\ IO_REQ_CLEAN_FLAGS) @@ -384,6 +384,9 @@ static bool req_need_defer(struct io_kiocb *req, u32 seq) static void io_clean_op(struct io_kiocb *req) { + if (req->flags & REQ_F_GROUP_KBUF) + io_group_kbuf_drop(req); + if (req->flags & REQ_F_BUFFER_SELECTED) { spin_lock(&req->ctx->completion_lock); io_kbuf_drop(req); @@ -989,8 +992,13 @@ void io_complete_group_lead(struct io_kiocb *req, unsigned issue_flags) while (member) { struct io_kiocb *next = member->grp_link; + const struct io_issue_def *def = &io_issue_defs[member->opcode]; member->grp_link = req; + if ((req->flags & REQ_F_GROUP_KBUF) && def->accept_group_kbuf) { + member->flags |= REQ_F_GROUP_KBUF; + member->grp_kbuf_ack = NULL; + } if (unlikely(req->flags & REQ_F_FAIL)) { /* * Now group lead is failed, so simply fail members diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index 3846a055df44..ab175fa6d57c 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -672,3 +672,65 @@ int io_pbuf_mmap(struct file *file, struct vm_area_struct *vma) io_put_bl(ctx, bl); return ret; } + +int io_provide_group_kbuf(struct io_kiocb *req, + const struct io_uring_kernel_buf *grp_kbuf, + io_uring_buf_giveback_t grp_kbuf_ack) +{ + if (unlikely(!req_is_group_lead(req))) + return -EINVAL; + + /* + * Borrow this buffer from one kernel subsystem, and return them + * by calling `grp_kbuf_ack` when the group lead is freed. + * + * Not like pipe/splice, this kernel buffer is always owned by the + * provider, and has to be returned back. + */ + req->grp_kbuf = grp_kbuf; + req->grp_kbuf_ack = grp_kbuf_ack; + req->flags |= REQ_F_GROUP_KBUF; + + return 0; +} + +int io_import_group_kbuf(struct io_kiocb *req, unsigned long buf_off, + unsigned int len, int dir, struct iov_iter *iter) +{ + struct io_kiocb *lead = req->grp_link; + const struct io_uring_kernel_buf *kbuf; + unsigned long offset; + + WARN_ON_ONCE(!(req->flags & REQ_F_GROUP_KBUF)); + + if (!req_is_group_member(req)) + return -EINVAL; + + if (!lead || !req_is_group_lead(lead) || !lead->grp_kbuf) + return -EINVAL; + + /* req->fused_cmd_kbuf is immutable */ + kbuf = lead->grp_kbuf; + offset = kbuf->offset; + + if (!kbuf->bvec) + return -EINVAL; + + if (dir != kbuf->dir) + return -EINVAL; + + if (unlikely(buf_off > kbuf->len)) + return -EFAULT; + + if (unlikely(len > kbuf->len - buf_off)) + return -EFAULT; + + /* don't use io_import_fixed which doesn't support multipage bvec */ + offset += buf_off; + iov_iter_bvec(iter, dir, kbuf->bvec, kbuf->nr_bvecs, offset + len); + + if (offset) + iov_iter_advance(iter, offset); + + return 0; +} diff --git a/io_uring/kbuf.h b/io_uring/kbuf.h index 5a9635ee0217..5d731ed52a28 100644 --- a/io_uring/kbuf.h +++ b/io_uring/kbuf.h @@ -64,6 +64,12 @@ struct io_buffer_list *io_pbuf_get_bl(struct io_ring_ctx *ctx, unsigned long bgid); int io_pbuf_mmap(struct file *file, struct vm_area_struct *vma); +int io_provide_group_kbuf(struct io_kiocb *req, + const struct io_uring_kernel_buf *grp_kbuf, + io_uring_buf_giveback_t grp_kbuf_ack); +int io_import_group_kbuf(struct io_kiocb *req, unsigned long buf_off, + unsigned int len, int dir, struct iov_iter *iter); + static inline bool io_kbuf_recycle_ring(struct io_kiocb *req) { /* @@ -145,4 +151,10 @@ static inline unsigned int io_put_kbuf(struct io_kiocb *req, __io_put_kbuf(req, issue_flags); return ret; } + +static inline void io_group_kbuf_drop(struct io_kiocb *req) +{ + if (req->grp_kbuf_ack) + req->grp_kbuf_ack(req->grp_kbuf); +} #endif diff --git a/io_uring/net.c b/io_uring/net.c index 9c0567892945..5371f0f2c0d8 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -78,6 +78,13 @@ struct io_sr_msg { */ #define MULTISHOT_MAX_RETRY 32 +#define user_ptr_to_u64(x) ( \ +{ \ + typecheck(void __user *, (x)); \ + (u64)(unsigned long)(x); \ +} \ +) + int io_shutdown_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) { struct io_shutdown *shutdown = io_kiocb_to_cmd(req, struct io_shutdown); @@ -479,6 +486,14 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) (sr->flags & IORING_RECVSEND_POLL_FIRST)) return -EAGAIN; + if (req->flags & REQ_F_GROUP_KBUF) { + ret = io_import_group_kbuf(req, + user_ptr_to_u64(sr->buf), + sr->len, ITER_SOURCE, + &kmsg->msg.msg_iter); + if (unlikely(ret)) + return ret; + } flags = sr->msg_flags; if (issue_flags & IO_URING_F_NONBLOCK) flags |= MSG_DONTWAIT; @@ -927,6 +942,11 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags) &kmsg->msg.msg_iter); if (unlikely(ret)) goto out_free; + } else if (req->flags & REQ_F_GROUP_KBUF) { + ret = io_import_group_kbuf(req, user_ptr_to_u64(sr->buf), + sr->len, ITER_DEST, &kmsg->msg.msg_iter); + if (unlikely(ret)) + goto out_free; } kmsg->msg.msg_inq = -1; @@ -1125,8 +1145,17 @@ static int io_send_zc_import(struct io_kiocb *req, struct io_async_msghdr *kmsg) if (unlikely(ret)) return ret; kmsg->msg.sg_from_iter = io_sg_from_iter; + } else if (req->flags & REQ_F_GROUP_KBUF) { + struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); + + ret = io_import_group_kbuf(req, user_ptr_to_u64(sr->buf), + sr->len, ITER_SOURCE, &kmsg->msg.msg_iter); + if (unlikely(ret)) + return ret; + kmsg->msg.sg_from_iter = io_sg_from_iter; } else { io_notif_set_extended(sr->notif); + ret = import_ubuf(ITER_SOURCE, sr->buf, sr->len, &kmsg->msg.msg_iter); if (unlikely(ret)) return ret; diff --git a/io_uring/opdef.c b/io_uring/opdef.c index a16f73938ebb..705f0333f9e0 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -246,6 +246,7 @@ const struct io_issue_def io_issue_defs[] = { .ioprio = 1, .iopoll = 1, .iopoll_queue = 1, + .accept_group_kbuf = 1, .async_size = sizeof(struct io_async_rw), .prep = io_prep_read, .issue = io_read, @@ -260,6 +261,7 @@ const struct io_issue_def io_issue_defs[] = { .ioprio = 1, .iopoll = 1, .iopoll_queue = 1, + .accept_group_kbuf = 1, .async_size = sizeof(struct io_async_rw), .prep = io_prep_write, .issue = io_write, @@ -281,6 +283,7 @@ const struct io_issue_def io_issue_defs[] = { .pollout = 1, .audit_skip = 1, .ioprio = 1, + .accept_group_kbuf = 1, #if defined(CONFIG_NET) .async_size = sizeof(struct io_async_msghdr), .prep = io_sendmsg_prep, @@ -296,6 +299,7 @@ const struct io_issue_def io_issue_defs[] = { .buffer_select = 1, .audit_skip = 1, .ioprio = 1, + .accept_group_kbuf = 1, #if defined(CONFIG_NET) .async_size = sizeof(struct io_async_msghdr), .prep = io_recvmsg_prep, @@ -423,6 +427,7 @@ const struct io_issue_def io_issue_defs[] = { .pollout = 1, .audit_skip = 1, .ioprio = 1, + .accept_group_kbuf = 1, #if defined(CONFIG_NET) .async_size = sizeof(struct io_async_msghdr), .prep = io_send_zc_prep, diff --git a/io_uring/opdef.h b/io_uring/opdef.h index 7ee6f5aa90aa..a53970655c82 100644 --- a/io_uring/opdef.h +++ b/io_uring/opdef.h @@ -29,6 +29,8 @@ struct io_issue_def { unsigned iopoll_queue : 1; /* vectored opcode, set if 1) vectored, and 2) handler needs to know */ unsigned vectored : 1; + /* opcodes which accept provided group kbuf */ + unsigned accept_group_kbuf : 1; /* size of async data needed, if any */ unsigned short async_size; diff --git a/io_uring/rw.c b/io_uring/rw.c index 3134a6ece1be..f1052af40563 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -235,7 +235,8 @@ static int io_prep_rw_setup(struct io_kiocb *req, int ddir, bool do_import) if (io_rw_alloc_async(req)) return -ENOMEM; - if (!do_import || io_do_buffer_select(req)) + if (!do_import || io_do_buffer_select(req) || + (req->flags & REQ_F_GROUP_KBUF)) return 0; rw = req->async_data; @@ -603,11 +604,16 @@ static inline loff_t *io_kiocb_ppos(struct kiocb *kiocb) */ static ssize_t loop_rw_iter(int ddir, struct io_rw *rw, struct iov_iter *iter) { + struct io_kiocb *req = cmd_to_io_kiocb(rw); struct kiocb *kiocb = &rw->kiocb; struct file *file = kiocb->ki_filp; ssize_t ret = 0; loff_t *ppos; + /* group buffer is kernel buffer and doesn't have userspace addr */ + if (req->flags & REQ_F_GROUP_KBUF) + return -EOPNOTSUPP; + /* * Don't support polled IO through this interface, and we can't * support non-blocking either. For the latter, this just causes @@ -813,6 +819,11 @@ static int __io_read(struct io_kiocb *req, unsigned int issue_flags) ret = io_import_iovec(ITER_DEST, req, io, issue_flags); if (unlikely(ret < 0)) return ret; + } else if (req->flags & REQ_F_GROUP_KBUF) { + ret = io_import_group_kbuf(req, rw->addr, rw->len, ITER_DEST, + &io->iter); + if (unlikely(ret)) + return ret; } ret = io_rw_init_file(req, FMODE_READ); @@ -995,6 +1006,13 @@ int io_write(struct io_kiocb *req, unsigned int issue_flags) ssize_t ret, ret2; loff_t *ppos; + if (req->flags & REQ_F_GROUP_KBUF) { + ret = io_import_group_kbuf(req, rw->addr, rw->len, ITER_SOURCE, + &io->iter); + if (unlikely(ret)) + return ret; + } + ret = io_rw_init_file(req, FMODE_WRITE); if (unlikely(ret)) return ret; From patchwork Mon Apr 8 01:03:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13620345 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 199BD15CE for ; Mon, 8 Apr 2024 01:04:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712538295; cv=none; b=QP8umf5KepLg4MrN9Pi4mXY4Zlkp5Y8Toy0Y1krahJnpZoMAOskRrcIQXh0NyJ540qGh8JP7VsL/CEp4l0AJXcLDINBDsqjZ8F01R00sBQoXl6B19zYSgPsJQ3VVR9Usr7o07NUyw8+K+8NfGt9QT9h4XrE6SkTlyiFA4xWaq3A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712538295; c=relaxed/simple; bh=tbxDyxHAlcNFeQnBURwM1YgU/WULb6cwg6iZD7VQov0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SMeHruvde5qoSqK5ORmDxcy65+tNJzRUqHrcVoJkWUAR/uTh7vA/qgvKRjVIX9x6xOZbMoey3pNBE1VM4JYzSjDSOuo1TRGMrMDBO95GtupNwzjfkGiq1ysb/TFAcLUJsoUjarhNLy5FXfMhcC4bkSwIj4GSTUrmeVWsaZ9wyl4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=I0d2KEAe; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="I0d2KEAe" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1712538293; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MZqNAycV/VAYvzRE10L9enQpc5mGiIyBt3NJ38rT53g=; b=I0d2KEAernt0dsiHJ7qZkgODN+3bYCfb52vXIp7bR98OMP0qNZkfxNEE0x7SBc/8ybTJ1F RjKfW9dRWZAwMHoHVuVByI6mBatKTgwnqx3YyVGNm+HbcDtddQcDzSQUoLLiVK4Im9otfB Y2QA+WL0m+P6IoCtDBCXD5nixgL5mH0= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-237-PsKzz67XNC-qlylK31yMjA-1; Sun, 07 Apr 2024 21:04:48 -0400 X-MC-Unique: PsKzz67XNC-qlylK31yMjA-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 63ECE3800082; Mon, 8 Apr 2024 01:04:48 +0000 (UTC) Received: from localhost (unknown [10.72.116.148]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7547B210FD1D; Mon, 8 Apr 2024 01:04:47 +0000 (UTC) From: Ming Lei To: Jens Axboe , io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Pavel Begunkov , Kevin Wolf , Ming Lei Subject: [PATCH 7/9] io_uring/uring_cmd: support provide group kernel buffer Date: Mon, 8 Apr 2024 09:03:20 +0800 Message-ID: <20240408010322.4104395-8-ming.lei@redhat.com> In-Reply-To: <20240408010322.4104395-1-ming.lei@redhat.com> References: <20240408010322.4104395-1-ming.lei@redhat.com> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.6 Allow uring command to be group leader for providing kernel buffer, and this way can support generic device zero copy over device buffer. The following patch will use the way to support zero copy for ublk. Signed-off-by: Ming Lei --- include/linux/io_uring/cmd.h | 9 +++++++++ include/uapi/linux/io_uring.h | 7 ++++++- io_uring/uring_cmd.c | 22 ++++++++++++++++++++++ 3 files changed, 37 insertions(+), 1 deletion(-) diff --git a/include/linux/io_uring/cmd.h b/include/linux/io_uring/cmd.h index 447fbfd32215..707660df1083 100644 --- a/include/linux/io_uring/cmd.h +++ b/include/linux/io_uring/cmd.h @@ -48,6 +48,9 @@ void __io_uring_cmd_do_in_task(struct io_uring_cmd *ioucmd, void io_uring_cmd_mark_cancelable(struct io_uring_cmd *cmd, unsigned int issue_flags); +int io_uring_cmd_provide_kbuf(struct io_uring_cmd *ioucmd, + const struct io_uring_kernel_buf *grp_kbuf, + io_uring_buf_giveback_t grp_kbuf_ack); #else static inline int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw, struct iov_iter *iter, void *ioucmd) @@ -67,6 +70,12 @@ static inline void io_uring_cmd_mark_cancelable(struct io_uring_cmd *cmd, unsigned int issue_flags) { } +static inline int io_uring_cmd_provide_kbuf(struct io_uring_cmd *ioucmd, + const struct io_uring_kernel_buf *grp_kbuf, + io_uring_buf_giveback_t grp_kbuf_ack) +{ + return -EOPNOTSUPP; +} #endif /* diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index c0d34f2a2c17..0060971721d0 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -281,9 +281,14 @@ enum io_uring_op { * bit23 ~ bit16 sqe ext flags * IORING_URING_CMD_FIXED use registered buffer; pass this flag * along with setting sqe->buf_index. + * IORING_PROVIDE_GROUP_KBUF this command provides group kernel buffer + * for member requests which can retrieve + * any sub-buffer with offset(sqe->addr) and + * len(sqe->len) */ #define IORING_URING_CMD_FIXED (1U << 0) -#define IORING_URING_CMD_MASK IORING_URING_CMD_FIXED +#define IORING_PROVIDE_GROUP_KBUF (1U << 1) +#define IORING_URING_CMD_MASK (IORING_URING_CMD_FIXED | IORING_PROVIDE_GROUP_KBUF) #define IORING_URING_CMD_EXT_MASK 0x00ff0000 diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c index 43b71f29e7b3..ec03c9fe965c 100644 --- a/io_uring/uring_cmd.c +++ b/io_uring/uring_cmd.c @@ -14,6 +14,7 @@ #include "alloc_cache.h" #include "rsrc.h" #include "uring_cmd.h" +#include "kbuf.h" static struct uring_cache *io_uring_async_get(struct io_kiocb *req) { @@ -174,6 +175,27 @@ void io_uring_cmd_done(struct io_uring_cmd *ioucmd, ssize_t ret, ssize_t res2, } EXPORT_SYMBOL_GPL(io_uring_cmd_done); +/* + * Provide kernel buffer for sqe group members to consume, and the caller + * has to guarantee that the provided buffer and the callback are valid + * until the callback is called. + */ +int io_uring_cmd_provide_kbuf(struct io_uring_cmd *ioucmd, + const struct io_uring_kernel_buf *grp_kbuf, + io_uring_buf_giveback_t grp_kbuf_ack) +{ + struct io_kiocb *req = cmd_to_io_kiocb(ioucmd); + + if (unlikely(!(ioucmd->flags & IORING_PROVIDE_GROUP_KBUF))) + return -EINVAL; + + if (unlikely(!(req->flags & REQ_F_SQE_GROUP))) + return -EINVAL; + + return io_provide_group_kbuf(req, grp_kbuf, grp_kbuf_ack); +} +EXPORT_SYMBOL_GPL(io_uring_cmd_provide_kbuf); + static int io_uring_cmd_prep_setup(struct io_kiocb *req, const struct io_uring_sqe *sqe) { From patchwork Mon Apr 8 01:03:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13620346 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EFF91EBB for ; Mon, 8 Apr 2024 01:05:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712538318; cv=none; b=lylFibfBYJNl18jzDxGbKmVhwIHYgFQLDlQQJY5MQCkS6UoVfBl7Y9BZWFJnRdDl7a/hPfZG79jumsg3paZj3NI6LnLXqXBHtDR8oSHqta0ok6jyUvS2DAMYsV3YMcPzGGAWKMh50UhpTuTPf1SqRhwqMDeBqSD99PJIX/c2aRA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712538318; c=relaxed/simple; bh=BBWIpxcX0WKpXH97dhCfbluTtgAp9lG4cLLprLP9wBU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TaYeoL9mWd9dhuJkiD/6DklrTv1gJRWC7yAwZb17xqvWMB47KlYwIAA9RTHlFrhdQcW8d3ToTH+BJsh5e5uPP9CI5m8jdyVOSRwggjjRftHIdFEQzs3lccR7tsIOO4ZZ7hZEjmjPwDkzboZ2ZSmTUq1zgBx7+rWEMgilb4mLlVo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=bWJsyKrq; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="bWJsyKrq" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1712538315; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dFQvaYUrK19EodWYNVHpNXRGM1Pzyl107BRv7JDZC5s=; b=bWJsyKrqQvmC45+WkbARZsGKqlHufCNawAtUGCHaqHmaajJQKE5T8wAy/JXzczTmwF6odL Eajklv0qR0S3g13zCgeIOIxwsGcU4PtcmfevTMdWox1Ps7TsVQlpUUJBIAK8O0TGwedvXQ ORfA9FgBxeohHi3eJhD4EmUV3FrdI1A= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-550-otSmyAG7O86fa35cBibQFA-1; Sun, 07 Apr 2024 21:05:12 -0400 X-MC-Unique: otSmyAG7O86fa35cBibQFA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 40D613819C69; Mon, 8 Apr 2024 01:05:12 +0000 (UTC) Received: from localhost (unknown [10.72.116.148]) by smtp.corp.redhat.com (Postfix) with ESMTP id D0CED40AE782; Mon, 8 Apr 2024 01:05:10 +0000 (UTC) From: Ming Lei To: Jens Axboe , io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Pavel Begunkov , Kevin Wolf , Ming Lei Subject: [PATCH 8/9] ublk: support provide io buffer Date: Mon, 8 Apr 2024 09:03:21 +0800 Message-ID: <20240408010322.4104395-9-ming.lei@redhat.com> In-Reply-To: <20240408010322.4104395-1-ming.lei@redhat.com> References: <20240408010322.4104395-1-ming.lei@redhat.com> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 Implement uring command's IORING_PROVIDE_GROUP_KBUF, and provide io buffer for userpace to run io_uring operations(FS, network IO), then ublk zero copy can be supported. Signed-off-by: Ming Lei --- drivers/block/ublk_drv.c | 156 ++++++++++++++++++++++++++++++++-- include/uapi/linux/ublk_cmd.h | 7 +- 2 files changed, 152 insertions(+), 11 deletions(-) diff --git a/drivers/block/ublk_drv.c b/drivers/block/ublk_drv.c index bea3d5cf8a83..76e3e9e421f3 100644 --- a/drivers/block/ublk_drv.c +++ b/drivers/block/ublk_drv.c @@ -71,6 +71,8 @@ struct ublk_rq_data { __u64 sector; __u32 operation; __u32 nr_zones; + bool allocated_bvec; + struct io_uring_kernel_buf buf[0]; }; struct ublk_uring_cmd_pdu { @@ -189,6 +191,8 @@ struct ublk_params_header { __u32 types; }; +static inline struct request *__ublk_check_and_get_req(struct ublk_device *ub, + struct ublk_queue *ubq, int tag, size_t offset); static bool ublk_abort_requests(struct ublk_device *ub, struct ublk_queue *ubq); static inline unsigned int ublk_req_build_flags(struct request *req); @@ -566,6 +570,11 @@ static inline bool ublk_need_req_ref(const struct ublk_queue *ubq) return ublk_support_user_copy(ubq); } +static inline bool ublk_support_zc(const struct ublk_queue *ubq) +{ + return ubq->flags & UBLK_F_SUPPORT_ZERO_COPY; +} + static inline void ublk_init_req_ref(const struct ublk_queue *ubq, struct request *req) { @@ -829,6 +838,70 @@ static size_t ublk_copy_user_pages(const struct request *req, return done; } +/* + * The built command buffer is immutable, so it is fine to feed it to + * concurrent io_uring provide buf commands + */ +static int ublk_init_zero_copy_buffer(struct request *req) +{ + struct ublk_rq_data *data = blk_mq_rq_to_pdu(req); + struct io_uring_kernel_buf *imu = data->buf; + struct req_iterator rq_iter; + unsigned int nr_bvecs = 0; + struct bio_vec *bvec; + unsigned int offset; + struct bio_vec bv; + + if (!ublk_rq_has_data(req)) + goto exit; + + rq_for_each_bvec(bv, req, rq_iter) + nr_bvecs++; + + if (!nr_bvecs) + goto exit; + + if (req->bio != req->biotail) { + int idx = 0; + + bvec = kvmalloc_array(nr_bvecs, sizeof(struct bio_vec), + GFP_NOIO); + if (!bvec) + return -ENOMEM; + + offset = 0; + rq_for_each_bvec(bv, req, rq_iter) + bvec[idx++] = bv; + data->allocated_bvec = true; + } else { + struct bio *bio = req->bio; + + offset = bio->bi_iter.bi_bvec_done; + bvec = __bvec_iter_bvec(bio->bi_io_vec, bio->bi_iter); + } + imu->bvec = bvec; + imu->nr_bvecs = nr_bvecs; + imu->offset = offset; + imu->len = blk_rq_bytes(req); + imu->dir = req_op(req) == REQ_OP_READ ? ITER_DEST : ITER_SOURCE; + + return 0; +exit: + imu->bvec = NULL; + return 0; +} + +static void ublk_deinit_zero_copy_buffer(struct request *req) +{ + struct ublk_rq_data *data = blk_mq_rq_to_pdu(req); + struct io_uring_kernel_buf *imu = data->buf; + + if (data->allocated_bvec) { + kvfree(imu->bvec); + data->allocated_bvec = false; + } +} + static inline bool ublk_need_map_req(const struct request *req) { return ublk_rq_has_data(req) && req_op(req) == REQ_OP_WRITE; @@ -840,13 +913,25 @@ static inline bool ublk_need_unmap_req(const struct request *req) (req_op(req) == REQ_OP_READ || req_op(req) == REQ_OP_DRV_IN); } -static int ublk_map_io(const struct ublk_queue *ubq, const struct request *req, +static int ublk_map_io(const struct ublk_queue *ubq, struct request *req, struct ublk_io *io) { const unsigned int rq_bytes = blk_rq_bytes(req); - if (ublk_support_user_copy(ubq)) + if (ublk_support_user_copy(ubq)) { + if (ublk_support_zc(ubq)) { + int ret = ublk_init_zero_copy_buffer(req); + + /* + * The only failure is -ENOMEM for allocating providing + * buffer command, return zero so that we can requeue + * this req. + */ + if (unlikely(ret)) + return 0; + } return rq_bytes; + } /* * no zero copy, we delay copy WRITE request data into ublksrv @@ -864,13 +949,16 @@ static int ublk_map_io(const struct ublk_queue *ubq, const struct request *req, } static int ublk_unmap_io(const struct ublk_queue *ubq, - const struct request *req, + struct request *req, struct ublk_io *io) { const unsigned int rq_bytes = blk_rq_bytes(req); - if (ublk_support_user_copy(ubq)) + if (ublk_support_user_copy(ubq)) { + if (ublk_support_zc(ubq)) + ublk_deinit_zero_copy_buffer(req); return rq_bytes; + } if (ublk_need_unmap_req(req)) { struct iov_iter iter; @@ -1016,6 +1104,7 @@ static inline void __ublk_complete_rq(struct request *req) return; exit: + ublk_deinit_zero_copy_buffer(req); blk_mq_end_request(req, res); } @@ -1658,6 +1747,46 @@ static inline void ublk_prep_cancel(struct io_uring_cmd *cmd, io_uring_cmd_mark_cancelable(cmd, issue_flags); } +static void ublk_io_buf_giveback_cb(const struct io_uring_kernel_buf *buf) +{ + struct ublk_rq_data *data = container_of(buf, struct ublk_rq_data, buf[0]); + struct request *req = blk_mq_rq_from_pdu(data); + struct ublk_queue *ubq = req->mq_hctx->driver_data; + + ublk_put_req_ref(ubq, req); +} + +static int ublk_provide_io_buf(struct io_uring_cmd *cmd, + struct ublk_queue *ubq, int tag) +{ + struct ublk_device *ub = cmd->file->private_data; + struct ublk_rq_data *data; + struct request *req; + + if (!ub) + return -EPERM; + + req = __ublk_check_and_get_req(ub, ubq, tag, 0); + if (!req) + return -EINVAL; + + pr_devel("%s: qid %d tag %u request bytes %u\n", + __func__, tag, ubq->q_id, blk_rq_bytes(req)); + + data = blk_mq_rq_to_pdu(req); + + /* + * io_uring guarantees that the callback will be called after + * the provided buffer is consumed, and it is automatic removal + * before this uring command is freed. + * + * This request won't be completed unless the callback is called, + * so ublk module won't be unloaded too. + */ + return io_uring_cmd_provide_kbuf(cmd, data->buf, + ublk_io_buf_giveback_cb); +} + static int __ublk_ch_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags, const struct ublksrv_io_cmd *ub_cmd) @@ -1674,6 +1803,10 @@ static int __ublk_ch_uring_cmd(struct io_uring_cmd *cmd, __func__, cmd->cmd_op, ub_cmd->q_id, tag, ub_cmd->result); + if ((cmd->flags & IORING_PROVIDE_GROUP_KBUF) && + cmd_op != UBLK_U_IO_PROVIDE_IO_BUF) + return -EOPNOTSUPP; + if (ub_cmd->q_id >= ub->dev_info.nr_hw_queues) goto out; @@ -1709,6 +1842,8 @@ static int __ublk_ch_uring_cmd(struct io_uring_cmd *cmd, ret = -EINVAL; switch (_IOC_NR(cmd_op)) { + case _IOC_NR(UBLK_U_IO_PROVIDE_IO_BUF): + return ublk_provide_io_buf(cmd, ubq, tag); case UBLK_IO_FETCH_REQ: /* UBLK_IO_FETCH_REQ is only allowed before queue is setup */ if (ublk_queue_ready(ubq)) { @@ -2128,11 +2263,14 @@ static void ublk_align_max_io_size(struct ublk_device *ub) static int ublk_add_tag_set(struct ublk_device *ub) { + int zc = !!(ub->dev_info.flags & UBLK_F_SUPPORT_ZERO_COPY); + struct ublk_rq_data *data; + ub->tag_set.ops = &ublk_mq_ops; ub->tag_set.nr_hw_queues = ub->dev_info.nr_hw_queues; ub->tag_set.queue_depth = ub->dev_info.queue_depth; ub->tag_set.numa_node = NUMA_NO_NODE; - ub->tag_set.cmd_size = sizeof(struct ublk_rq_data); + ub->tag_set.cmd_size = struct_size(data, buf, zc); ub->tag_set.flags = BLK_MQ_F_SHOULD_MERGE; ub->tag_set.driver_data = ub; return blk_mq_alloc_tag_set(&ub->tag_set); @@ -2417,8 +2555,12 @@ static int ublk_ctrl_add_dev(struct io_uring_cmd *cmd) goto out_free_dev_number; } - /* We are not ready to support zero copy */ - ub->dev_info.flags &= ~UBLK_F_SUPPORT_ZERO_COPY; + /* zero copy depends on user copy */ + if ((ub->dev_info.flags & UBLK_F_SUPPORT_ZERO_COPY) && + !ublk_dev_is_user_copy(ub)) { + ret = -EINVAL; + goto out_free_dev_number; + } ub->dev_info.nr_hw_queues = min_t(unsigned int, ub->dev_info.nr_hw_queues, nr_cpu_ids); diff --git a/include/uapi/linux/ublk_cmd.h b/include/uapi/linux/ublk_cmd.h index c8dc5f8ea699..897ace0794c2 100644 --- a/include/uapi/linux/ublk_cmd.h +++ b/include/uapi/linux/ublk_cmd.h @@ -94,6 +94,8 @@ _IOWR('u', UBLK_IO_COMMIT_AND_FETCH_REQ, struct ublksrv_io_cmd) #define UBLK_U_IO_NEED_GET_DATA \ _IOWR('u', UBLK_IO_NEED_GET_DATA, struct ublksrv_io_cmd) +#define UBLK_U_IO_PROVIDE_IO_BUF \ + _IOWR('u', 0x23, struct ublksrv_io_cmd) /* only ABORT means that no re-fetch */ #define UBLK_IO_RES_OK 0 @@ -126,10 +128,7 @@ #define UBLKSRV_IO_BUF_TOTAL_BITS (UBLK_QID_OFF + UBLK_QID_BITS) #define UBLKSRV_IO_BUF_TOTAL_SIZE (1ULL << UBLKSRV_IO_BUF_TOTAL_BITS) -/* - * zero copy requires 4k block size, and can remap ublk driver's io - * request into ublksrv's vm space - */ +/* io_uring provide kbuf command based zero copy */ #define UBLK_F_SUPPORT_ZERO_COPY (1ULL << 0) /* From patchwork Mon Apr 8 01:03:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13620347 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A19CB7F8 for ; Mon, 8 Apr 2024 01:05:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712538336; cv=none; b=RAQ/DzEwB/pMeG7VFIwclLEjbK1L4Xjcv/fs+tgakS4H1YFMnyXbFmrlJ9Y7tazcDZS6OLKD/5TABXNhSQ8KEB/ERxx+VfNkYEK/NvzPJdFVtxxSshaLEoSXfjSTBcat43GWyhA8pcO67UyvV5i+5fn/Y4TOpQd+hFEEZ0VluKk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712538336; c=relaxed/simple; bh=+FI9rCcRMO9FRe0PZDi0wWo54+iVj3+d/f6zQWUW80Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=OsUoWMLeaCvDb5KdQC5mk98cocNP44P2D527Q6zy13zZgDwaay3IwfJ5rxSo9b10VdJrro/GemJZBb8myJuOavkjyeweVk2BBRQyz0m5Xcj1IAEv8i3bysNONPbDsJg6g4nDZ1IxhLXawKs2TsQxxKonJgiE7DPSweXwqC7xI0s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=VYFXHpVW; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="VYFXHpVW" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1712538333; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fk7Og0+9kQFsceT+wSVT+18TGLYiV9J02JZHLxr0aUI=; b=VYFXHpVWA+49QTxrPz+TBsw2hvdpwsosxj8l88m3hXDOfZZ1m17HjhIASNfqJmyB7eLB/h v+CbzBLR7gyCUl1rN9D9G7f3GBay9OEkoJrQtcBefUAM64ZhJw0dsFJBjqIBZZUNDZRuP2 /jqrIPl5hcpLeSkQKsvNK2F5Qom1l+s= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-662-RN62w8-aPn2hL5hS2n8D7Q-1; Sun, 07 Apr 2024 21:05:30 -0400 X-MC-Unique: RN62w8-aPn2hL5hS2n8D7Q-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id BA4F6811000; Mon, 8 Apr 2024 01:05:29 +0000 (UTC) Received: from localhost (unknown [10.72.116.148]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0A8B447F; Mon, 8 Apr 2024 01:05:28 +0000 (UTC) From: Ming Lei To: Jens Axboe , io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Pavel Begunkov , Kevin Wolf , Ming Lei Subject: [RFC PATCH 9/9] liburing: support sqe ext_flags & sqe group Date: Mon, 8 Apr 2024 09:03:22 +0800 Message-ID: <20240408010322.4104395-10-ming.lei@redhat.com> In-Reply-To: <20240408010322.4104395-1-ming.lei@redhat.com> References: <20240408010322.4104395-1-ming.lei@redhat.com> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.1 From: Ming Lei Provide helper to set sqe ext_flags. Add one test case to cover sqe group feature by applying sqe group for copying one part of source file into multiple destinations via single syscall. Signed-off-by: Ming Lei --- src/include/liburing.h | 15 ++ src/include/liburing/io_uring.h | 20 +++ test/Makefile | 1 + test/group_cp.c | 260 ++++++++++++++++++++++++++++++++ 4 files changed, 296 insertions(+) create mode 100644 test/group_cp.c diff --git a/src/include/liburing.h b/src/include/liburing.h index 3e47298..5379d53 100644 --- a/src/include/liburing.h +++ b/src/include/liburing.h @@ -368,6 +368,21 @@ IOURINGINLINE void io_uring_sqe_set_flags(struct io_uring_sqe *sqe, sqe->flags = (__u8) flags; } +IOURINGINLINE void io_uring_sqe_set_ext_flags(struct io_uring_sqe *sqe, + unsigned flags) +{ + sqe->ext_flags = (__u8) flags; + sqe->flags |= IOSQE_HAS_EXT_FLAGS; +} + +IOURINGINLINE void io_uring_cmd_set_ext_flags(struct io_uring_sqe *sqe, + unsigned flags) +{ + sqe->uring_cmd_flags &= ~IORING_URING_CMD_EXT_MASK; + sqe->uring_cmd_flags |= ((__u8) flags) << 16; + sqe->flags |= IOSQE_HAS_EXT_FLAGS; +} + IOURINGINLINE void __io_uring_set_target_fixed_file(struct io_uring_sqe *sqe, unsigned int file_index) { diff --git a/src/include/liburing/io_uring.h b/src/include/liburing/io_uring.h index bde1199..b0a0318 100644 --- a/src/include/liburing/io_uring.h +++ b/src/include/liburing/io_uring.h @@ -98,6 +98,10 @@ struct io_uring_sqe { __u64 __pad2[1]; }; __u64 optval; + struct { + __u8 __pad4[15]; + __u8 ext_flags; + }; /* * If the ring is initialized with IORING_SETUP_SQE128, then * this field is used for 80 bytes of arbitrary command data @@ -123,6 +127,9 @@ enum { IOSQE_ASYNC_BIT, IOSQE_BUFFER_SELECT_BIT, IOSQE_CQE_SKIP_SUCCESS_BIT, + IOSQE_HAS_EXT_FLAGS_BIT, + + IOSQE_EXT_SQE_GROUP_BIT = 0, }; /* @@ -142,6 +149,13 @@ enum { #define IOSQE_BUFFER_SELECT (1U << IOSQE_BUFFER_SELECT_BIT) /* don't post CQE if request succeeded */ #define IOSQE_CQE_SKIP_SUCCESS (1U << IOSQE_CQE_SKIP_SUCCESS_BIT) +/* + * sqe ext flags carried in the last byte, or bit23~bit16 of + * sqe->uring_cmd_flags for IORING_URING_CMD. + */ +#define IOSQE_HAS_EXT_FLAGS (1U << IOSQE_HAS_EXT_FLAGS_BIT) +/* defines sqe group */ +#define IOSQE_EXT_SQE_GROUP (1U << IOSQE_EXT_SQE_GROUP_BIT) /* * io_uring_setup() flags @@ -265,8 +279,14 @@ enum io_uring_op { * sqe->uring_cmd_flags * IORING_URING_CMD_FIXED use registered buffer; pass this flag * along with setting sqe->buf_index. + * IORING_PROVIDE_GROUP_KBUF this command provides group kernel buffer + * for member requests which can retrieve + * any sub-buffer with offset(sqe->addr) and + * len(sqe->len) */ #define IORING_URING_CMD_FIXED (1U << 0) +#define IORING_PROVIDE_GROUP_KBUF (1U << 1) +#define IORING_URING_CMD_EXT_MASK 0x00ff0000 /* diff --git a/test/Makefile b/test/Makefile index 32848ec..dd3b394 100644 --- a/test/Makefile +++ b/test/Makefile @@ -207,6 +207,7 @@ test_srcs := \ wakeup-hang.c \ wq-aff.c \ xattr.c \ + group_cp.c \ # EOL all_targets := diff --git a/test/group_cp.c b/test/group_cp.c new file mode 100644 index 0000000..4ac5cdb --- /dev/null +++ b/test/group_cp.c @@ -0,0 +1,260 @@ +/* SPDX-License-Identifier: MIT */ +/* + * Test SQE group feature + * + */ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "liburing.h" +#include "helpers.h" + +#define BUF_SIZE 8192 +#define BUFFERS 6 + + +struct test_data { + const char *src_file; + struct iovec iov[BUFFERS]; + int fd_in, fd_out1, fd_out2; +}; + +static void set_sqe_group(struct io_uring_sqe *sqe) +{ + io_uring_sqe_set_ext_flags(sqe, IOSQE_EXT_SQE_GROUP); +} + +static int check_cqe(struct io_uring *ring, unsigned int nr) +{ + int i, ret; + + for (i = 0; i < nr; ++i) { + struct io_uring_cqe *cqe; + int res; + + ret = io_uring_peek_cqe(ring, &cqe); + res = cqe->res; + if (ret) { + fprintf(stderr, "peek failed: %d\n", ret); + return ret; + } + io_uring_cqe_seen(ring, cqe); + if (res != BUF_SIZE) + fprintf(stderr, "bad result %d, user_data %llx\n", + res, cqe->user_data); + //printf("cqe %lld res %d\n", cqe->user_data, cqe->res); + } + + return 0; +} + +static int prep_test(struct io_uring *ring, struct test_data *data) +{ + int ret, i; + + data->fd_in = open(data->src_file, O_RDONLY | O_DIRECT, 0644); + if (data->fd_in < 0) { + perror("open in"); + return 1; + } + + data->fd_out1 = open(".test_group_cp1", O_RDWR | O_CREAT | O_DIRECT, 0644); + unlink(".test_group_cp1"); + + data->fd_out2 = open(".test_group_cp2", O_RDWR | O_CREAT| O_DIRECT, 0644); + unlink(".test_group_cp2"); + + if (data->fd_out1 < 0 || data->fd_out2 < 0) { + perror("open out"); + return 1; + } + + for (i = 0; i < BUFFERS; i++) { + void *buf; + + assert(!posix_memalign(&buf, 4096, BUF_SIZE)); + data->iov[i].iov_base = buf; + data->iov[i].iov_len = BUF_SIZE; + memset(data->iov[i].iov_base, 0, BUF_SIZE); + } + + ret = io_uring_register_buffers(ring, data->iov, BUFFERS); + if (ret) { + fprintf(stderr, "Error registering buffers: %s", + strerror(-ret)); + return 1; + } + + return 0; +} + +static void unprep_test(struct io_uring *ring, struct test_data *d) +{ + io_uring_unregister_buffers(ring); + close(d->fd_in); + close(d->fd_out1); + close(d->fd_out2); +} + +static unsigned build_group_sqes(struct io_uring *ring, struct test_data *d, + off_t off, int buf_idx, __u8 lead_flags, __u8 mem_flags) +{ + struct io_uring_sqe *sqe, *sqe2, *sqe1; + + sqe = io_uring_get_sqe(ring); + sqe1 = io_uring_get_sqe(ring); + sqe2 = io_uring_get_sqe(ring); + assert(sqe && sqe1 && sqe2); + + io_uring_prep_read_fixed(sqe, d->fd_in, d->iov[buf_idx].iov_base, + BUF_SIZE, 0, buf_idx); + set_sqe_group(sqe); + sqe->user_data = buf_idx + 1; + sqe->flags |= lead_flags; + + io_uring_prep_write_fixed(sqe1, d->fd_out1, d->iov[buf_idx].iov_base, + BUF_SIZE, off, buf_idx); + set_sqe_group(sqe1); + sqe1->user_data = buf_idx + 2; + sqe1->flags |= mem_flags; + + io_uring_prep_write_fixed(sqe2, d->fd_out2, d->iov[buf_idx].iov_base, + BUF_SIZE, off, buf_idx); + sqe2->user_data = buf_idx + 3; + sqe2->flags |= mem_flags; + + return 3; +} + +static int verify_cp(struct io_uring *ring, struct test_data *d, off_t off, + unsigned int buf_idx) +{ + struct io_uring_sqe *sqe2, *sqe1; + int ret; + + sqe1 = io_uring_get_sqe(ring); + sqe2 = io_uring_get_sqe(ring); + assert(sqe1 && sqe2); + + io_uring_prep_read_fixed(sqe1, d->fd_out1, d->iov[buf_idx + 1].iov_base, + BUF_SIZE, off, buf_idx + 1); + sqe1->user_data = buf_idx + 7; + io_uring_prep_read_fixed(sqe2, d->fd_out2, d->iov[buf_idx + 2].iov_base, + BUF_SIZE, off, buf_idx + 2); + sqe1->user_data = buf_idx + 8; + ret = io_uring_submit_and_wait(ring, 2); + if (ret < 0) { + fprintf(stderr, "submit failed: %d\n", ret); + return 1; + } + + ret = check_cqe(ring, 2); + if (ret) + return ret; + + if (memcmp(d->iov[buf_idx].iov_base, d->iov[buf_idx + 1].iov_base, BUF_SIZE)) { + fprintf(stderr, "data not match for destination 1\n"); + return 1; + } + + if (memcmp(d->iov[buf_idx].iov_base, d->iov[buf_idx + 2].iov_base, BUF_SIZE)) { + fprintf(stderr, "data not match for destination 2\n"); + return 1; + } + return 0; +} + +static int test(struct io_uring *ring, struct test_data *d, + __u8 lead_flags, __u8 mem_flags) +{ + unsigned test_link = lead_flags & IOSQE_IO_LINK; + unsigned nr; + int ret; + + if (!test_link) { + nr = build_group_sqes(ring, d, 0, 0, lead_flags, mem_flags); + } else { + /* two groups linked together */ + nr = build_group_sqes(ring, d, 0, 0, lead_flags, mem_flags); + nr += build_group_sqes(ring, d, BUF_SIZE, 3, lead_flags, + mem_flags); + } + + ret = io_uring_submit_and_wait(ring, nr); + if (ret < 0) { + fprintf(stderr, "submit failed: %d\n", ret); + return 1; + } + + if (check_cqe(ring, nr)) + return 1; + + ret = verify_cp(ring, d, 0, 0); + if (ret) + return ret; + if (test_link) + return verify_cp(ring, d, BUF_SIZE, 3); + return 0; +} + +static int run_test(struct io_uring *ring, struct test_data *d, + __u8 lead_flags, __u8 mem_flags) +{ + int ret = test(ring, d, lead_flags, mem_flags); + if (ret) { + fprintf(stderr, "Test failed lead flags %x mem flags %x\n", + lead_flags, mem_flags); + return T_EXIT_FAIL; + } + + exit(0); + return 0; +} + +int main(int argc, char *argv[]) +{ + struct io_uring ring; + struct test_data data = { + .src_file = argv[0], + }; + int ret; + int g_link, g_async, m_async; + + if (argc > 1) + return T_EXIT_SKIP; + + ret = t_create_ring(16, &ring, 0); + if (ret == T_SETUP_SKIP) + return T_EXIT_SKIP; + else if (ret < 0) + return T_EXIT_FAIL; + + ret = prep_test(&ring, &data); + if (ret) { + fprintf(stderr, "Prepare Test failed\n"); + return T_EXIT_FAIL; + } + + for (g_async = 0; g_async < 2; g_async += 1) + for (g_link = 0; g_link < 2; g_link += 1) + for (m_async = 0; m_async < 2; m_async += 1) { + __u8 g_flags = (g_async ? IOSQE_ASYNC : 0) | + (g_link ? IOSQE_IO_LINK : 0); + __u8 m_flags = (g_async ? IOSQE_ASYNC : 0); + + if (run_test(&ring, &data, g_flags, m_flags)) + return T_EXIT_FAIL; + } + unprep_test(&ring, &data); + + io_uring_queue_exit(&ring); + return T_EXIT_PASS; +}