From patchwork Tue Feb 6 16:22:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13547578 Received: from mail-io1-f45.google.com (mail-io1-f45.google.com [209.85.166.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CF477134742 for ; Tue, 6 Feb 2024 16:24:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707236652; cv=none; b=kNH6SdEE5ImICtF5yrvBOO13THBsTcK8MiKgjbOuM7hweKjOWi47k147hLkFcNB+eX71B2Bex2JKQKJmEfM5X70A7bgqPd90nULfZUkcdX9GmqSG05YEcEuWbFHPlUt63Q6UmmvJ2r3Wz0Nk4k4YwvHEhXHODeXJm2GkpzQbVss= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707236652; c=relaxed/simple; bh=pzm1U0+erPaCQ+1X0Bf/YxoiO0lYuIawD/xOu5Fuwgo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HhYEFYGX1uMDK+IiXVsVgCg/MZvhEkcb/efI6i8eCcn4RTd1DXYDWXunbSscoWQtWqnsQNL5i/7NAR6BhQv6uEqdv0CFc/CzdGH3VFkQCqsZNZLxPJ6eQYiI+8xt96P9Kcy52Fh9pulcba8g0c4d6rQEp6PyWCPRdT0LqhX66QU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=TsWqudtL; arc=none smtp.client-ip=209.85.166.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="TsWqudtL" Received: by mail-io1-f45.google.com with SMTP id ca18e2360f4ac-7bff8f21b74so59110039f.0 for ; Tue, 06 Feb 2024 08:24:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1707236648; x=1707841448; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=iS0yZ+NiEuYwLGHRNOCdqliE6h4hbHOCvOVbuM2cTJo=; b=TsWqudtL8TPIxCjZBST0JnurCr7/44gTKUODg8vRG9a0nPocigcyo6j2cY/9LyedS8 8VX9VZ853N8CMup40UxDRt8V903nmB1i16EnC+GqdaBtRstnpj5piMrSY3PNYmr8r/3U i5oMEkQCI4CNRyY3ReZzsWLYOHdRc7miMjaE/WU2PFA1nyTM4cfpL6bnT+FWwyFdyLmH 2CfYf/SWnd0mwM8nHAKQnYYr0ZS6uOw2ZqUgYSfxAZXj4+rjzyK304tLHWAIhYQwCnSy nEmLH8OxW6q7dCgLT4rQJyGcan5zQxp4m85ruS7gtieRPjcbEIf5e2FkMgSiejCfnHkY 6C7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707236648; x=1707841448; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iS0yZ+NiEuYwLGHRNOCdqliE6h4hbHOCvOVbuM2cTJo=; b=nmcGW1Sxv4LhNbxzZlvguoZXwZmfiNW2R1vIR96ERcWTU4oc2SKeUBENv5h5D5ZD7C BXCnb9B1qjoNgXSG82Ket5GV+4z5PMhOKRx61GnOBq55IcrZGyJ8+IUfsPebMa+EDCMT t+Gtfy2ixaQ5K3wbunDhL2UbX+9ABj9ZRMtudT+zzi3MbmVNpqgJrxbtwiE3LLcfVaT5 n16CwwLDgJqtNTAoVGB8YTwUIh47a+JvbTmMuOPYPwIAIzx4SJZ6yrdsNAV1tIny6pG3 dpwjV0Jpny6zRu9Nit07s6o/r4s+V3RmnzQWm5eyiqaVJ5+htY/Rce0iHxiSfNMgokHQ KO2A== X-Gm-Message-State: AOJu0YyPCa5S6aXf8Mx6Z38oLdZeBuo3CCyLLf+TtkQP46d1DmmVUDi7 i7BEsDBhjPKHKmy2OEWTD1V+v6z7NclBalhBpQCO5bjcp1Qr1i8pCg7Y9DRu8Ds/bB5iPibdh4X uldM= X-Google-Smtp-Source: AGHT+IFNmUpPgPW7JkgyHVP9Hu9qvrHciTVmIh82gyH73mcshQjjQHZXosREJPxapqfcbS0GnvZBAA== X-Received: by 2002:a05:6602:70c:b0:7c2:caa4:561a with SMTP id f12-20020a056602070c00b007c2caa4561amr3302256iox.2.1707236648310; Tue, 06 Feb 2024 08:24:08 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id v17-20020a6b5b11000000b007bfe5fb5e0dsm520031ioh.51.2024.02.06.08.24.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 08:24:06 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 1/6] io_uring: expand main struct io_kiocb flags to 64-bits Date: Tue, 6 Feb 2024 09:22:47 -0700 Message-ID: <20240206162402.643507-2-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240206162402.643507-1-axboe@kernel.dk> References: <20240206162402.643507-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 We're out of space here, and none of the flags are easily reclaimable. Bump it to 64-bits and re-arrange the struct a bit to avoid gaps. Add a specific bitwise type for the request flags, io_request_flags_t. This will help catch violations of casting this value to a smaller type on 32-bit archs, like unsigned int. No functional changes intended in this patch. Signed-off-by: Jens Axboe --- include/linux/io_uring_types.h | 87 ++++++++++++++++++--------------- include/trace/events/io_uring.h | 14 +++--- io_uring/filetable.h | 2 +- io_uring/io_uring.c | 9 ++-- 4 files changed, 60 insertions(+), 52 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 854ad67a5f70..5ac18b05d4ee 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -428,7 +428,7 @@ struct io_tw_state { bool locked; }; -enum { +enum io_req_flags { REQ_F_FIXED_FILE_BIT = IOSQE_FIXED_FILE_BIT, REQ_F_IO_DRAIN_BIT = IOSQE_IO_DRAIN_BIT, REQ_F_LINK_BIT = IOSQE_IO_LINK_BIT, @@ -468,70 +468,73 @@ enum { __REQ_F_LAST_BIT, }; +typedef enum io_req_flags __bitwise io_req_flags_t; +#define IO_REQ_FLAG(bitno) ((__force io_req_flags_t) BIT_ULL((bitno))) + enum { /* ctx owns file */ - REQ_F_FIXED_FILE = BIT(REQ_F_FIXED_FILE_BIT), + REQ_F_FIXED_FILE = IO_REQ_FLAG(REQ_F_FIXED_FILE_BIT), /* drain existing IO first */ - REQ_F_IO_DRAIN = BIT(REQ_F_IO_DRAIN_BIT), + REQ_F_IO_DRAIN = IO_REQ_FLAG(REQ_F_IO_DRAIN_BIT), /* linked sqes */ - REQ_F_LINK = BIT(REQ_F_LINK_BIT), + REQ_F_LINK = IO_REQ_FLAG(REQ_F_LINK_BIT), /* doesn't sever on completion < 0 */ - REQ_F_HARDLINK = BIT(REQ_F_HARDLINK_BIT), + REQ_F_HARDLINK = IO_REQ_FLAG(REQ_F_HARDLINK_BIT), /* IOSQE_ASYNC */ - REQ_F_FORCE_ASYNC = BIT(REQ_F_FORCE_ASYNC_BIT), + REQ_F_FORCE_ASYNC = IO_REQ_FLAG(REQ_F_FORCE_ASYNC_BIT), /* IOSQE_BUFFER_SELECT */ - REQ_F_BUFFER_SELECT = BIT(REQ_F_BUFFER_SELECT_BIT), + REQ_F_BUFFER_SELECT = IO_REQ_FLAG(REQ_F_BUFFER_SELECT_BIT), /* IOSQE_CQE_SKIP_SUCCESS */ - REQ_F_CQE_SKIP = BIT(REQ_F_CQE_SKIP_BIT), + REQ_F_CQE_SKIP = IO_REQ_FLAG(REQ_F_CQE_SKIP_BIT), /* fail rest of links */ - REQ_F_FAIL = BIT(REQ_F_FAIL_BIT), + REQ_F_FAIL = IO_REQ_FLAG(REQ_F_FAIL_BIT), /* on inflight list, should be cancelled and waited on exit reliably */ - REQ_F_INFLIGHT = BIT(REQ_F_INFLIGHT_BIT), + REQ_F_INFLIGHT = IO_REQ_FLAG(REQ_F_INFLIGHT_BIT), /* read/write uses file position */ - REQ_F_CUR_POS = BIT(REQ_F_CUR_POS_BIT), + REQ_F_CUR_POS = IO_REQ_FLAG(REQ_F_CUR_POS_BIT), /* must not punt to workers */ - REQ_F_NOWAIT = BIT(REQ_F_NOWAIT_BIT), + REQ_F_NOWAIT = IO_REQ_FLAG(REQ_F_NOWAIT_BIT), /* has or had linked timeout */ - REQ_F_LINK_TIMEOUT = BIT(REQ_F_LINK_TIMEOUT_BIT), + REQ_F_LINK_TIMEOUT = IO_REQ_FLAG(REQ_F_LINK_TIMEOUT_BIT), /* needs cleanup */ - REQ_F_NEED_CLEANUP = BIT(REQ_F_NEED_CLEANUP_BIT), + REQ_F_NEED_CLEANUP = IO_REQ_FLAG(REQ_F_NEED_CLEANUP_BIT), /* already went through poll handler */ - REQ_F_POLLED = BIT(REQ_F_POLLED_BIT), + REQ_F_POLLED = IO_REQ_FLAG(REQ_F_POLLED_BIT), /* buffer already selected */ - REQ_F_BUFFER_SELECTED = BIT(REQ_F_BUFFER_SELECTED_BIT), + REQ_F_BUFFER_SELECTED = IO_REQ_FLAG(REQ_F_BUFFER_SELECTED_BIT), /* buffer selected from ring, needs commit */ - REQ_F_BUFFER_RING = BIT(REQ_F_BUFFER_RING_BIT), + REQ_F_BUFFER_RING = IO_REQ_FLAG(REQ_F_BUFFER_RING_BIT), /* caller should reissue async */ - REQ_F_REISSUE = BIT(REQ_F_REISSUE_BIT), + REQ_F_REISSUE = IO_REQ_FLAG(REQ_F_REISSUE_BIT), /* supports async reads/writes */ - REQ_F_SUPPORT_NOWAIT = BIT(REQ_F_SUPPORT_NOWAIT_BIT), + REQ_F_SUPPORT_NOWAIT = IO_REQ_FLAG(REQ_F_SUPPORT_NOWAIT_BIT), /* regular file */ - REQ_F_ISREG = BIT(REQ_F_ISREG_BIT), + REQ_F_ISREG = IO_REQ_FLAG(REQ_F_ISREG_BIT), /* has creds assigned */ - REQ_F_CREDS = BIT(REQ_F_CREDS_BIT), + REQ_F_CREDS = IO_REQ_FLAG(REQ_F_CREDS_BIT), /* skip refcounting if not set */ - REQ_F_REFCOUNT = BIT(REQ_F_REFCOUNT_BIT), + REQ_F_REFCOUNT = IO_REQ_FLAG(REQ_F_REFCOUNT_BIT), /* there is a linked timeout that has to be armed */ - REQ_F_ARM_LTIMEOUT = BIT(REQ_F_ARM_LTIMEOUT_BIT), + REQ_F_ARM_LTIMEOUT = IO_REQ_FLAG(REQ_F_ARM_LTIMEOUT_BIT), /* ->async_data allocated */ - REQ_F_ASYNC_DATA = BIT(REQ_F_ASYNC_DATA_BIT), + REQ_F_ASYNC_DATA = IO_REQ_FLAG(REQ_F_ASYNC_DATA_BIT), /* don't post CQEs while failing linked requests */ - REQ_F_SKIP_LINK_CQES = BIT(REQ_F_SKIP_LINK_CQES_BIT), + REQ_F_SKIP_LINK_CQES = IO_REQ_FLAG(REQ_F_SKIP_LINK_CQES_BIT), /* single poll may be active */ - REQ_F_SINGLE_POLL = BIT(REQ_F_SINGLE_POLL_BIT), + REQ_F_SINGLE_POLL = IO_REQ_FLAG(REQ_F_SINGLE_POLL_BIT), /* double poll may active */ - REQ_F_DOUBLE_POLL = BIT(REQ_F_DOUBLE_POLL_BIT), + REQ_F_DOUBLE_POLL = IO_REQ_FLAG(REQ_F_DOUBLE_POLL_BIT), /* request has already done partial IO */ - REQ_F_PARTIAL_IO = BIT(REQ_F_PARTIAL_IO_BIT), + REQ_F_PARTIAL_IO = IO_REQ_FLAG(REQ_F_PARTIAL_IO_BIT), /* fast poll multishot mode */ - REQ_F_APOLL_MULTISHOT = BIT(REQ_F_APOLL_MULTISHOT_BIT), + REQ_F_APOLL_MULTISHOT = IO_REQ_FLAG(REQ_F_APOLL_MULTISHOT_BIT), /* recvmsg special flag, clear EPOLLIN */ - REQ_F_CLEAR_POLLIN = BIT(REQ_F_CLEAR_POLLIN_BIT), + REQ_F_CLEAR_POLLIN = IO_REQ_FLAG(REQ_F_CLEAR_POLLIN_BIT), /* hashed into ->cancel_hash_locked, protected by ->uring_lock */ - REQ_F_HASH_LOCKED = BIT(REQ_F_HASH_LOCKED_BIT), + REQ_F_HASH_LOCKED = IO_REQ_FLAG(REQ_F_HASH_LOCKED_BIT), /* don't use lazy poll wake for this request */ - REQ_F_POLL_NO_LAZY = BIT(REQ_F_POLL_NO_LAZY_BIT), + REQ_F_POLL_NO_LAZY = IO_REQ_FLAG(REQ_F_POLL_NO_LAZY_BIT), }; typedef void (*io_req_tw_func_t)(struct io_kiocb *req, struct io_tw_state *ts); @@ -592,15 +595,14 @@ struct io_kiocb { * and after selection it points to the buffer ID itself. */ u16 buf_index; - unsigned int flags; - struct io_cqe cqe; + atomic_t refs; + + io_req_flags_t flags; struct io_ring_ctx *ctx; struct task_struct *task; - struct io_rsrc_node *rsrc_node; - union { /* store used ubuf, so we can prevent reloading */ struct io_mapped_ubuf *imu; @@ -615,18 +617,23 @@ struct io_kiocb { struct io_buffer_list *buf_list; }; + /* for polled requests, i.e. IORING_OP_POLL_ADD and async armed poll */ + struct hlist_node hash_node; + union { /* used by request caches, completion batching and iopoll */ struct io_wq_work_node comp_list; /* cache ->apoll->events */ __poll_t apoll_events; }; - atomic_t refs; - atomic_t poll_refs; + + struct io_rsrc_node *rsrc_node; + + struct io_cqe cqe; + struct io_task_work io_task_work; + atomic_t poll_refs; unsigned nr_tw; - /* for polled requests, i.e. IORING_OP_POLL_ADD and async armed poll */ - struct hlist_node hash_node; /* internal polling, see IORING_FEAT_FAST_POLL */ struct async_poll *apoll; /* opcode allocated if it needs to store data for async defer */ diff --git a/include/trace/events/io_uring.h b/include/trace/events/io_uring.h index 69454f1f98b0..3d7704a52b73 100644 --- a/include/trace/events/io_uring.h +++ b/include/trace/events/io_uring.h @@ -148,7 +148,7 @@ TRACE_EVENT(io_uring_queue_async_work, __field( void *, req ) __field( u64, user_data ) __field( u8, opcode ) - __field( unsigned int, flags ) + __field( io_req_flags_t, flags ) __field( struct io_wq_work *, work ) __field( int, rw ) @@ -167,10 +167,10 @@ TRACE_EVENT(io_uring_queue_async_work, __assign_str(op_str, io_uring_get_opcode(req->opcode)); ), - TP_printk("ring %p, request %p, user_data 0x%llx, opcode %s, flags 0x%x, %s queue, work %p", + TP_printk("ring %p, request %p, user_data 0x%llx, opcode %s, flags 0x%lx, %s queue, work %p", __entry->ctx, __entry->req, __entry->user_data, - __get_str(op_str), - __entry->flags, __entry->rw ? "hashed" : "normal", __entry->work) + __get_str(op_str), (long) __entry->flags, + __entry->rw ? "hashed" : "normal", __entry->work) ); /** @@ -378,7 +378,7 @@ TRACE_EVENT(io_uring_submit_req, __field( void *, req ) __field( unsigned long long, user_data ) __field( u8, opcode ) - __field( u32, flags ) + __field( io_req_flags_t, flags ) __field( bool, sq_thread ) __string( op_str, io_uring_get_opcode(req->opcode) ) @@ -395,10 +395,10 @@ TRACE_EVENT(io_uring_submit_req, __assign_str(op_str, io_uring_get_opcode(req->opcode)); ), - TP_printk("ring %p, req %p, user_data 0x%llx, opcode %s, flags 0x%x, " + TP_printk("ring %p, req %p, user_data 0x%llx, opcode %s, flags 0x%lx, " "sq_thread %d", __entry->ctx, __entry->req, __entry->user_data, __get_str(op_str), - __entry->flags, __entry->sq_thread) + (long) __entry->flags, __entry->sq_thread) ); /* diff --git a/io_uring/filetable.h b/io_uring/filetable.h index b47adf170c31..b2435c4dca1f 100644 --- a/io_uring/filetable.h +++ b/io_uring/filetable.h @@ -17,7 +17,7 @@ int io_fixed_fd_remove(struct io_ring_ctx *ctx, unsigned int offset); int io_register_file_alloc_range(struct io_ring_ctx *ctx, struct io_uring_file_index_range __user *arg); -unsigned int io_file_get_flags(struct file *file); +io_req_flags_t io_file_get_flags(struct file *file); static inline void io_file_bitmap_clear(struct io_file_table *table, int bit) { diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index cd9a137ad6ce..360a7ee41d3a 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1768,9 +1768,9 @@ static void io_iopoll_req_issued(struct io_kiocb *req, unsigned int issue_flags) } } -unsigned int io_file_get_flags(struct file *file) +io_req_flags_t io_file_get_flags(struct file *file) { - unsigned int res = 0; + io_req_flags_t res = 0; if (S_ISREG(file_inode(file)->i_mode)) res |= REQ_F_ISREG; @@ -2171,7 +2171,8 @@ static int io_init_req(struct io_ring_ctx *ctx, struct io_kiocb *req, /* req is partially pre-initialised, see io_preinit_req() */ req->opcode = opcode = READ_ONCE(sqe->opcode); /* same numerical values with corresponding REQ_F_*, safe to copy */ - req->flags = sqe_flags = READ_ONCE(sqe->flags); + sqe_flags = READ_ONCE(sqe->flags); + req->flags = (io_req_flags_t) sqe_flags; req->cqe.user_data = READ_ONCE(sqe->user_data); req->file = NULL; req->rsrc_node = NULL; @@ -4153,7 +4154,7 @@ static int __init io_uring_init(void) BUILD_BUG_ON(SQE_COMMON_FLAGS >= (1 << 8)); BUILD_BUG_ON((SQE_VALID_FLAGS | SQE_COMMON_FLAGS) != SQE_VALID_FLAGS); - BUILD_BUG_ON(__REQ_F_LAST_BIT > 8 * sizeof(int)); + BUILD_BUG_ON(__REQ_F_LAST_BIT > 8 * sizeof(u64)); BUILD_BUG_ON(sizeof(atomic_t) != sizeof(u32)); From patchwork Tue Feb 6 16:22:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13547579 Received: from mail-io1-f46.google.com (mail-io1-f46.google.com [209.85.166.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 634D9134CDF for ; Tue, 6 Feb 2024 16:24:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707236654; cv=none; b=M60ljcL9qjBdRo4whBMzq4ptGcPncgHXCsxB+joJ3nt2qn76xhx4SpYWdwZPF5A7pEONZ+OnBJgJ18IA5UwREEha/Lqj6+c+4w6ynwKl1/BwOWKb/pH5zgLU4cbrM64NQPQTSjDLY8WYvAq09WpR+/dPVjHwTbel6FpOFmAwIeg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707236654; c=relaxed/simple; bh=HhjKqV0xRqk0fe+Dd+ki5LLmlYDvpCTZd1lvic2Xklo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HhWACxi+ZNZZ5QOS2YmvXWwnGufXJPLvlHMWM8DWA9HZiwWp43IVd4X/J24aG/LfqHb32pzXOyWhGFwqaAO6cLs+umbiYNch4hjObHOVHHhWVGyyAPeu/cNkVqctK4axrwi/XOUSVBvHv0tideMqZLStmJz0MshOl0ODcaiSzKg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=b2w/iF0y; arc=none smtp.client-ip=209.85.166.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="b2w/iF0y" Received: by mail-io1-f46.google.com with SMTP id ca18e2360f4ac-7c3df9489d2so27612639f.0 for ; Tue, 06 Feb 2024 08:24:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1707236651; x=1707841451; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BfTea7CHpcEUgf68SvNCkLAG4dRHDhWc4mmdjJHmlfs=; b=b2w/iF0y1ogv4G9wmAKtqEjmmdmECOuw8Qm+wiuq+g0JjFyLsiMDSR5YxGMEG28Spw d8jHlGvN6aqZrRskXqZPEUSXgi8CctQtTn809Y4dxXueOFUXth3eczEDwtBc4zInAkVD Bm+uTu5fmm4XdmFCxxSHLVYXLMKTUl7TvXknFxmyL6XBMSdp3kHlNcuRWHSlTihRbwQg yaX9IHu6KQLfpE1izhgxQzlYc1HvVjnudHnkfSnzhsHoffPU3GrccX2XT+hcMYdUAko3 psTiGKEcJZQH6s6Nc4YNOfPF1h2EfrGqCTVkaYkUpXh8F6Gg9rFV4CK/h9JDyodJdX/U Mu4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707236651; x=1707841451; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BfTea7CHpcEUgf68SvNCkLAG4dRHDhWc4mmdjJHmlfs=; b=ryFF0JFJN8dMnBMDKWAUbxjHhxvpbRoQAW4X8Eri9muBAANn3EJIwIdH2pDs5Ks8Ri IvTi7RPwE+NlWfq59REqtHXc3dICFp9nAhA3eGIASA+jYTFH43D7LvCxbs2RNBBqTNck zQk3lWxk6RPtd90AwbTeYR1hBAqi+n77TNGp4ee929jurZ91xee5SgXkQ4ZvJ2Ob0Kyq SWrco+q6ZEqCAKZLqd5bk1B6Jnc+ArIPjqnCC8exgbpZ0L9rCfW582apt4Lyp41v+Ir9 VoWd6qZtlInbCaqAnkxnrMRXfTMXm4YF1jocCS5n6Vv6EH3roRa7LvcGeX1RwFYYe6F+ 3HHw== X-Gm-Message-State: AOJu0YzmVCf9SNtaIBZj78UQXy9VA1/SnE5YiEAPJ3H9xtTJUe8tqQ1G RRGvd111wOdjgBcqdmwCF4xFjhppE0Rs8Rf4lw8pR7s487rREU42Rxy62Mu+a4URgmeFKnaxLY6 A9NE= X-Google-Smtp-Source: AGHT+IFZFUU19go7kMBjaJ3Yb7Fq7e2N8LWVGkaBHh+0YFhBIEoEel4pgrrC4jksXZtya6PLiBw9HA== X-Received: by 2002:a05:6602:2410:b0:7c3:f2c1:e8aa with SMTP id s16-20020a056602241000b007c3f2c1e8aamr2234585ioa.0.1707236651104; Tue, 06 Feb 2024 08:24:11 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id v17-20020a6b5b11000000b007bfe5fb5e0dsm520031ioh.51.2024.02.06.08.24.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 08:24:08 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 2/6] io_uring: add io_file_can_poll() helper Date: Tue, 6 Feb 2024 09:22:48 -0700 Message-ID: <20240206162402.643507-3-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240206162402.643507-1-axboe@kernel.dk> References: <20240206162402.643507-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 This adds a flag to avoid dipping dereferencing file and then f_op to figure out if the file has a poll handler defined or not. We generally call this at least twice for networked workloads. Signed-off-by: Jens Axboe --- include/linux/io_uring_types.h | 3 +++ io_uring/io_uring.c | 2 +- io_uring/io_uring.h | 12 ++++++++++++ io_uring/kbuf.c | 2 +- io_uring/poll.c | 2 +- io_uring/rw.c | 6 +++--- 6 files changed, 21 insertions(+), 6 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 5ac18b05d4ee..7f06cee02b58 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -463,6 +463,7 @@ enum io_req_flags { REQ_F_SUPPORT_NOWAIT_BIT, REQ_F_ISREG_BIT, REQ_F_POLL_NO_LAZY_BIT, + REQ_F_CAN_POLL_BIT, /* not a real bit, just to check we're not overflowing the space */ __REQ_F_LAST_BIT, @@ -535,6 +536,8 @@ enum { REQ_F_HASH_LOCKED = IO_REQ_FLAG(REQ_F_HASH_LOCKED_BIT), /* don't use lazy poll wake for this request */ REQ_F_POLL_NO_LAZY = IO_REQ_FLAG(REQ_F_POLL_NO_LAZY_BIT), + /* file is pollable */ + REQ_F_CAN_POLL = IO_REQ_FLAG(REQ_F_CAN_POLL_BIT), }; typedef void (*io_req_tw_func_t)(struct io_kiocb *req, struct io_tw_state *ts); diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 360a7ee41d3a..d0e06784926f 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1969,7 +1969,7 @@ void io_wq_submit_work(struct io_wq_work *work) if (req->flags & REQ_F_FORCE_ASYNC) { bool opcode_poll = def->pollin || def->pollout; - if (opcode_poll && file_can_poll(req->file)) { + if (opcode_poll && io_file_can_poll(req)) { needs_poll = true; issue_flags |= IO_URING_F_NONBLOCK; } diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index d5495710c178..2952551fe345 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -5,6 +5,7 @@ #include #include #include +#include #include #include #include "io-wq.h" @@ -398,4 +399,15 @@ static inline size_t uring_sqe_size(struct io_ring_ctx *ctx) return 2 * sizeof(struct io_uring_sqe); return sizeof(struct io_uring_sqe); } + +static inline bool io_file_can_poll(struct io_kiocb *req) +{ + if (req->flags & REQ_F_CAN_POLL) + return true; + if (file_can_poll(req->file)) { + req->flags |= REQ_F_CAN_POLL; + return true; + } + return false; +} #endif diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index 18df5a9d2f5e..71880615bb78 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -180,7 +180,7 @@ static void __user *io_ring_buffer_select(struct io_kiocb *req, size_t *len, req->buf_list = bl; req->buf_index = buf->bid; - if (issue_flags & IO_URING_F_UNLOCKED || !file_can_poll(req->file)) { + if (issue_flags & IO_URING_F_UNLOCKED || !io_file_can_poll(req)) { /* * If we came in unlocked, we have no choice but to consume the * buffer here, otherwise nothing ensures that the buffer won't diff --git a/io_uring/poll.c b/io_uring/poll.c index 7513afc7b702..4afec733fef6 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -727,7 +727,7 @@ int io_arm_poll_handler(struct io_kiocb *req, unsigned issue_flags) if (!def->pollin && !def->pollout) return IO_APOLL_ABORTED; - if (!file_can_poll(req->file)) + if (!io_file_can_poll(req)) return IO_APOLL_ABORTED; if (!(req->flags & REQ_F_APOLL_MULTISHOT)) mask |= EPOLLONESHOT; diff --git a/io_uring/rw.c b/io_uring/rw.c index d5e79d9bdc71..0fb7a045163a 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -682,7 +682,7 @@ static bool io_rw_should_retry(struct io_kiocb *req) * just use poll if we can, and don't attempt if the fs doesn't * support callback based unlocks */ - if (file_can_poll(req->file) || !(req->file->f_mode & FMODE_BUF_RASYNC)) + if (io_file_can_poll(req) || !(req->file->f_mode & FMODE_BUF_RASYNC)) return false; wait->wait.func = io_async_buf_func; @@ -831,7 +831,7 @@ static int __io_read(struct io_kiocb *req, unsigned int issue_flags) * If we can poll, just do that. For a vectored read, we'll * need to copy state first. */ - if (file_can_poll(req->file) && !io_issue_defs[req->opcode].vectored) + if (io_file_can_poll(req) && !io_issue_defs[req->opcode].vectored) return -EAGAIN; /* IOPOLL retry should happen for io-wq threads */ if (!force_nonblock && !(req->ctx->flags & IORING_SETUP_IOPOLL)) @@ -930,7 +930,7 @@ int io_read_mshot(struct io_kiocb *req, unsigned int issue_flags) /* * Multishot MUST be used on a pollable file */ - if (!file_can_poll(req->file)) + if (!io_file_can_poll(req)) return -EBADFD; ret = __io_read(req, issue_flags); From patchwork Tue Feb 6 16:22:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13547580 Received: from mail-io1-f48.google.com (mail-io1-f48.google.com [209.85.166.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B2C29134742 for ; Tue, 6 Feb 2024 16:24:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707236655; cv=none; b=qd+gFtkwsjtymUeq/jfp2CPISAL+KWu5VxJJO5g9f6+nM/o3SM6sadkPuwSchp7ZbhQX6nY1ZwEC6oxabyeJ5/goYKIyuW76Vqh2zpj9QPOv7Ml//593q6S9PMFlJVKMfBHHewvz/L5d0rYbHheL4zVx9qhUF+toXXL2ixx9F34= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707236655; c=relaxed/simple; bh=/56j+vB4HnzzerVXqOnMuM20+9TN+rJNAf3Nfqps0to=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=BlBpkVxVyGvNSAxLQe6lH2A856Zn9CFAJDLdm/PnaoCqBhmfhzUleJYe3g7D9A3wBclXm9wkqdG3SfwIA7qFQ02yfEEc5J2BpDr//4RvMQ3tK+Pcw/jQ9rFzrIXb9X/U0Dwojbnc2dFDk5OiSxMHj2kXOaJEi94Zb86qPZIKoLc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=3co2mPRb; arc=none smtp.client-ip=209.85.166.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="3co2mPRb" Received: by mail-io1-f48.google.com with SMTP id ca18e2360f4ac-7bff8f21b74so59110439f.0 for ; Tue, 06 Feb 2024 08:24:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1707236652; x=1707841452; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WAW6bkbJaWe8E5VD7l9A+IdcMAJxrlkRYErXBCFfjXo=; b=3co2mPRbH0GvuPaMEM7OD7E0BmlVoNrlisGpyatvDVQNSHgn9OcxExB9BzsfrGLhZX SZatPBUn24M+ZA+vmga33OT405zjcY8vl0Gopmx6ULhZli9+7ypyz79XKU6n0e+RUwXo Es5sQ6AO3+mTXVToZySnl43+dSsa5eZnPADHtCxpsihUpsDwfMVMIlNV/gcdDbcPnmIX 0AMphKWAuYHdTZ11FbLW8omkpnZ5rBd/F7ar783D3hPPNjW2MsxRR5wiWQiDNloNk9a9 3YJsSptHmcRkGf2b0sdX3SxDWwxNcRzBWGSdO4aSRp/NLJvIthKZ4OVMaifc8zTH4eVY q1Fw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707236652; x=1707841452; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WAW6bkbJaWe8E5VD7l9A+IdcMAJxrlkRYErXBCFfjXo=; b=swSpKW8g5NCTpiewZDLaoTev17acs6YWPKr8If+XE+DnDgjhVgcUe7LfUxqFcEtpor yLZPH/U2G1NbwGy8Y0KZpUkUUvfFrWb4SMZPXvXZb2nNErpTuqB/jYS2ded6CQiBlCt/ jpkBxFmngX45ulwGBUG1xXHlBC2q2j90zk1jNAnSPXg+suusroMKejLSRhd+1RBkJCdb B5gzYkYwKJ63Agj+jZfNvjOg6YKsYS451qIL3M0oOaCWZnXjdK1vyka7d/IL78pvT4HE gtpIoZFbStRcnAim9qsfX4YH+nV9irqH9TTm+2PkEPpt493JxIFM2rmJ8eYIM+6/rprA XXcw== X-Gm-Message-State: AOJu0Yy4JW90mzcXuivpwas7U1KAvX4ghnhZ60ACwZUE/7QqwnA29fKL NHg9tX8ljEBj1NxtD+daLraRnN+STSjX0SNRL8NovQQdOteKCFFGX4i7xD6udu5ojOU19KrUiww Z8zU= X-Google-Smtp-Source: AGHT+IEmbwHmqj4n8YweiZynfDD7xM1DlTxMiLs1bq7ZuexH74RofmFAdWjE4DLBS7CoD/M1zpEzwQ== X-Received: by 2002:a6b:f308:0:b0:7c3:f836:aed with SMTP id m8-20020a6bf308000000b007c3f8360aedmr438153ioh.0.1707236652374; Tue, 06 Feb 2024 08:24:12 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id v17-20020a6b5b11000000b007bfe5fb5e0dsm520031ioh.51.2024.02.06.08.24.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 08:24:11 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 3/6] io_uring/cancel: don't default to setting req->work.cancel_seq Date: Tue, 6 Feb 2024 09:22:49 -0700 Message-ID: <20240206162402.643507-4-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240206162402.643507-1-axboe@kernel.dk> References: <20240206162402.643507-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Just leave it unset by default, avoiding dipping into the last cacheline (which is otherwise untouched) for the fast path of using poll to drive networked traffic. Add a flag that tells us if the sequence is valid or not, and then we can defer actually assigning the flag and sequence until someone runs cancelations. Signed-off-by: Jens Axboe --- include/linux/io_uring_types.h | 3 +++ io_uring/cancel.c | 3 +-- io_uring/cancel.h | 10 ++++++++++ io_uring/io_uring.c | 1 - io_uring/poll.c | 6 +----- 5 files changed, 15 insertions(+), 8 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 7f06cee02b58..69a043ff8460 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -464,6 +464,7 @@ enum io_req_flags { REQ_F_ISREG_BIT, REQ_F_POLL_NO_LAZY_BIT, REQ_F_CAN_POLL_BIT, + REQ_F_CANCEL_SEQ_BIT, /* not a real bit, just to check we're not overflowing the space */ __REQ_F_LAST_BIT, @@ -538,6 +539,8 @@ enum { REQ_F_POLL_NO_LAZY = IO_REQ_FLAG(REQ_F_POLL_NO_LAZY_BIT), /* file is pollable */ REQ_F_CAN_POLL = IO_REQ_FLAG(REQ_F_CAN_POLL_BIT), + /* cancel sequence is set and valid */ + REQ_F_CANCEL_SEQ = IO_REQ_FLAG(REQ_F_CANCEL_SEQ_BIT), }; typedef void (*io_req_tw_func_t)(struct io_kiocb *req, struct io_tw_state *ts); diff --git a/io_uring/cancel.c b/io_uring/cancel.c index 8a8b07dfc444..acfcdd7f059a 100644 --- a/io_uring/cancel.c +++ b/io_uring/cancel.c @@ -58,9 +58,8 @@ bool io_cancel_req_match(struct io_kiocb *req, struct io_cancel_data *cd) return false; if (cd->flags & IORING_ASYNC_CANCEL_ALL) { check_seq: - if (cd->seq == req->work.cancel_seq) + if (io_cancel_match_sequence(req, cd->seq)) return false; - req->work.cancel_seq = cd->seq; } return true; diff --git a/io_uring/cancel.h b/io_uring/cancel.h index c0a8e7c520b6..76b32e65c03c 100644 --- a/io_uring/cancel.h +++ b/io_uring/cancel.h @@ -25,4 +25,14 @@ void init_hash_table(struct io_hash_table *table, unsigned size); int io_sync_cancel(struct io_ring_ctx *ctx, void __user *arg); bool io_cancel_req_match(struct io_kiocb *req, struct io_cancel_data *cd); +static inline bool io_cancel_match_sequence(struct io_kiocb *req, int sequence) +{ + if ((req->flags & REQ_F_CANCEL_SEQ) && sequence == req->work.cancel_seq) + return true; + + req->flags |= REQ_F_CANCEL_SEQ; + req->work.cancel_seq = sequence; + return false; +} + #endif diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index d0e06784926f..9b499864f10d 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -463,7 +463,6 @@ static void io_prep_async_work(struct io_kiocb *req) req->work.list.next = NULL; req->work.flags = 0; - req->work.cancel_seq = atomic_read(&ctx->cancel_seq); if (req->flags & REQ_F_FORCE_ASYNC) req->work.flags |= IO_WQ_WORK_CONCURRENT; diff --git a/io_uring/poll.c b/io_uring/poll.c index 4afec733fef6..3f3380dc5f68 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -588,10 +588,7 @@ static int __io_arm_poll_handler(struct io_kiocb *req, struct io_poll_table *ipt, __poll_t mask, unsigned issue_flags) { - struct io_ring_ctx *ctx = req->ctx; - INIT_HLIST_NODE(&req->hash_node); - req->work.cancel_seq = atomic_read(&ctx->cancel_seq); io_init_poll_iocb(poll, mask); poll->file = req->file; req->apoll_events = poll->events; @@ -818,9 +815,8 @@ static struct io_kiocb *io_poll_find(struct io_ring_ctx *ctx, bool poll_only, if (poll_only && req->opcode != IORING_OP_POLL_ADD) continue; if (cd->flags & IORING_ASYNC_CANCEL_ALL) { - if (cd->seq == req->work.cancel_seq) + if (io_cancel_match_sequence(req, cd->seq)) continue; - req->work.cancel_seq = cd->seq; } *out_bucket = hb; return req; From patchwork Tue Feb 6 16:22:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13547581 Received: from mail-io1-f45.google.com (mail-io1-f45.google.com [209.85.166.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DB7F8134CFF for ; Tue, 6 Feb 2024 16:24:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707236657; cv=none; b=D3iFsu4SyYk/kO0ReKZzO9SEUsfNGFHBmZmakSwSRhKLkOd2WKJkcpvuJbkSL75XjDVB4L1QI9Cqwq7w2x8LmieE6F73AFd6aAdeQ6m073YV3DJLzpXDUHX7JV+rr4DhuRSAEjew7/dYRTy4xWr4e5XSdCTv0O5BYyc0SfjY0MY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707236657; c=relaxed/simple; bh=+OxtjrK4uEvJXTurejEo5/X8jfGWKREe/uT9iKMACNM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=NyRjh9NSQyacp4iwL8phbUzCgVDdAHFL9AEfX8loBE7nFT6ZC19/18ngBcqj9L+ETtMsVy2Xtna2V1vlTf99RKxBDN9MJpjSywlue2iv40n6P9fIOxyFhcxQ5P2TAn8RUsxoMmhOZfBOsWzV4I696oIrz8Go67bAQ3Y4fO7vwgI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=wuT6IRah; arc=none smtp.client-ip=209.85.166.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="wuT6IRah" Received: by mail-io1-f45.google.com with SMTP id ca18e2360f4ac-7c3d923f7cbso16581039f.0 for ; Tue, 06 Feb 2024 08:24:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1707236654; x=1707841454; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xcXfzwvlM8i1i3PC8yvPbeMAxX3SSeQ1hy/DHx3+Lkc=; b=wuT6IRahhg7jqYAKUrVbHPPiestDiFKx5ogrfDNfrd6i4KkCLZXJEfl89vkuQy65cQ u5YQhPOGge/46xAXvkM0qH0SzcVATcd+8PEMZzg/IdDpekU+gwS5MXvlskBiGT9YQC2J qD26Gvi5LM6giqiW9OWsHgwEEIH8+8KECLvuYZkRpDOdp2/epVI4ieyJkItzAoybA8Ix nVbZ5QBK24jS/gzQNfMC6AM6MN3G2CEK0Cdik6yxOWz/P+hNQFU4WAuouNNge9nIZmtI sXHFXUyhKybIQYCtAnDT6rpjEE8lX5raLTZ2LMeKNrJ3datjDSEpHPcRAH6vz5vlJSYI Rx+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707236654; x=1707841454; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xcXfzwvlM8i1i3PC8yvPbeMAxX3SSeQ1hy/DHx3+Lkc=; b=maDtdcdZGdRmS4+krQZBFct8zKmMJhoGLnSwR01P5v7IA0vWIjLGkN8a8/VFR+U00v xAR/dVCfy04upCvl8lSlMbFB9ZoQQmIaSqxn+ZZ9sorZzYnUU4jDhyx4W5pRCzqomItb Q4c6Hzn4F6fsimtU93sZU+Rtt+JhKGWoM0fop8G/erza6Gk6spHPeo7kIuyRqtuaR30d XNDLFHT/bswFOkdayOf2ExG6wLWVtWdaUTUnnR68k5Hdx6GPpXKpVs59y4R6n2sLdkx+ hROLZ3Y0lDLznhnVc9jA2qpYx1li/g0/hRpjUMWNdaE4rqhBvvCTHmKyBZQo6wFloVPr Rhqw== X-Gm-Message-State: AOJu0YyGZq7RGE1X16nSTzEPb5rscnFsBWT/jFfsPv8qQQ9ZzgSsKgVt F8aUe+sLAQeR6h71L+sDFqVBEW0kxOdOv4aSyvsjp2sEI2uWaT9xMBkzI9Sj+nHVPHOFQBcAlKY cxFo= X-Google-Smtp-Source: AGHT+IFwpEf7QjW3ovkgo0m0mQPhIq3HQbGgfhE6640h7R4jlyGBT5hEQGzToXk7ZtyQQiIPjLKZSg== X-Received: by 2002:a05:6602:2bcf:b0:7c3:eaf9:2cd with SMTP id s15-20020a0566022bcf00b007c3eaf902cdmr3500309iov.2.1707236654608; Tue, 06 Feb 2024 08:24:14 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id v17-20020a6b5b11000000b007bfe5fb5e0dsm520031ioh.51.2024.02.06.08.24.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 08:24:12 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 4/6] io_uring: move io_kiocb->nr_tw into comp_list union Date: Tue, 6 Feb 2024 09:22:50 -0700 Message-ID: <20240206162402.643507-5-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240206162402.643507-1-axboe@kernel.dk> References: <20240206162402.643507-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 comp_list is only used for completion purposes, which it why it currently shares space with apoll_events (which is only used for poll triggering). nr_rw is also not used with comp_list, the former is just used for local task_list wakeup optimizations. This doesn't save any space in io_kiocb, rather it now leaves a 32-bit hole that can be used for something else, when the need arises. Signed-off-by: Jens Axboe --- include/linux/io_uring_types.h | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 69a043ff8460..8c0742f5b57e 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -602,8 +602,6 @@ struct io_kiocb { */ u16 buf_index; - atomic_t refs; - io_req_flags_t flags; struct io_ring_ctx *ctx; @@ -629,8 +627,11 @@ struct io_kiocb { union { /* used by request caches, completion batching and iopoll */ struct io_wq_work_node comp_list; - /* cache ->apoll->events */ - __poll_t apoll_events; + struct { + /* cache ->apoll->events */ + __poll_t apoll_events; + unsigned nr_tw; + }; }; struct io_rsrc_node *rsrc_node; @@ -639,7 +640,7 @@ struct io_kiocb { struct io_task_work io_task_work; atomic_t poll_refs; - unsigned nr_tw; + atomic_t refs; /* internal polling, see IORING_FEAT_FAST_POLL */ struct async_poll *apoll; /* opcode allocated if it needs to store data for async defer */ From patchwork Tue Feb 6 16:22:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13547582 Received: from mail-io1-f49.google.com (mail-io1-f49.google.com [209.85.166.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DD8301350CA for ; Tue, 6 Feb 2024 16:24:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707236659; cv=none; b=o9aA/4Qh0rpftUY9XlSDxApwNQ+Fg3ggEnvANCeQLeiOKnTPdXoXUGuEL/cac3OO6nfscHDe8D/GTir0UIaX1vZfzalphxbqox44bt7NoHRC7GnAsQeswyq+2f4gjywXMI+SzQHs6pF7MVtOiIBbXVVZWnZU/Db55l9iQnMh03g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707236659; c=relaxed/simple; bh=uimgy7+wl4DXZV7rrq9F0Z4HxdrzWqdEH39e6n8xHJ4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=G3MeHqHbiNZAv0L+HfWz7CnO9coiIJebT623/9JNRDfIl001c7ufEVd+nKeIjKHuGFRdPSwBX9iwEFE/lBoDJnDxye8iFTJQ60UtNtXvAUNMwqdg41Veu5dxUo1ticfAkYRwAHsewuJdfO4sZn8YjKDuWuTxmzR0ObbE+ogshlg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=FjAFoiP8; arc=none smtp.client-ip=209.85.166.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="FjAFoiP8" Received: by mail-io1-f49.google.com with SMTP id ca18e2360f4ac-7c3e06c8608so30170839f.1 for ; Tue, 06 Feb 2024 08:24:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1707236656; x=1707841456; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GXvepwDyq3Lcyc7QSbbVA75c61s/YgsZV8j5neg7n9s=; b=FjAFoiP8twD1A5ZMnsWsLzV6UToqBxW/TT2GedmFQwas6HpeieA6urF/UhFutGzZ8M JMVxOoW40WGD7wDxYb91COp9VwO2X8BFQcVHuDIlbsyRNMsl0fIpD7FMfLbv7nqWGRLq OmBX9ZFHcA6y85wa+3D7NvAt3R5sy9bjjZDsivhFE39tAvOd1wzKtaddsXnc8b4YCA8M nLoXTeLCHn37JhutXxPAGYU/2Xpdr6pXOgjNdMa0T0XbrwcrWqmt/1G7r/DRDjmCs1uM Y94ionfsf0nXtm5WC+k0dpx8kix0x5C/mX8KlxTnz9h9z4ViTwgrhmdRV7ZwFgD6U01r 4oQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707236656; x=1707841456; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GXvepwDyq3Lcyc7QSbbVA75c61s/YgsZV8j5neg7n9s=; b=sVZMZjtk0rDoP220SlSbMuKm3rlUuZPVYi13qtNrbEDoNfx8hqJUk7zRD4Wq6Q7bEu U1z0jAknmq3sq8cREIZfOiWA7vWG/nCjyUkfQoQZv1DVU+KGVAJmySaaGcQ49FD71et9 jncXzLbSFRnYnJiFgLe8/10LvezXh3XUCwM2rJ0TtQu44ghSQIrH5l3M3v/q0MSV+Z7d 3r7RhX/rA5SmX/dE+EW6ofieEcDUeyTWbccpFW2IvDxCqdMD5r0zYmqmhak29m4+sAbg /4a6gpOK1KPE8lDDalVMNGqy3qSBWD588M3pGr5a617Inmnp7gMT6aQoxtpSEO3LuyZt oJsg== X-Gm-Message-State: AOJu0YyKxmMfRvYs1EgY+vm34IBQ1jfGfPwEwx9eOS1fBM9Abg46zH6R 0hTeKCfjimO+/p/UTK4sJE320LOO+t/95nk6yuqFmzCDh/LFYYQgA85TGAN5Do6LpnsAvRRRY1K Ck6M= X-Google-Smtp-Source: AGHT+IGfyR1EGiCOCn4A4iJrH7iAFLHvkxW6yM6ldJ47QFsm+vV1eopVDAwGfXekMBIqPNyzEUW89Q== X-Received: by 2002:a6b:db13:0:b0:7c3:edbb:1816 with SMTP id t19-20020a6bdb13000000b007c3edbb1816mr2781257ioc.2.1707236656666; Tue, 06 Feb 2024 08:24:16 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id v17-20020a6b5b11000000b007bfe5fb5e0dsm520031ioh.51.2024.02.06.08.24.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 08:24:14 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 5/6] io_uring: mark the need to lock/unlock the ring as unlikely Date: Tue, 6 Feb 2024 09:22:51 -0700 Message-ID: <20240206162402.643507-6-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240206162402.643507-1-axboe@kernel.dk> References: <20240206162402.643507-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Any of the fast paths will already have this locked, this helper only exists to deal with io-wq invoking request issue where we do not have the ctx->uring_lock held already. This means that any common or fast path will already have this locked, mark it as such. Signed-off-by: Jens Axboe --- io_uring/io_uring.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 2952551fe345..46795ee462df 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -208,7 +208,7 @@ static inline void io_ring_submit_unlock(struct io_ring_ctx *ctx, unsigned issue_flags) { lockdep_assert_held(&ctx->uring_lock); - if (issue_flags & IO_URING_F_UNLOCKED) + if (unlikely(issue_flags & IO_URING_F_UNLOCKED)) mutex_unlock(&ctx->uring_lock); } @@ -221,7 +221,7 @@ static inline void io_ring_submit_lock(struct io_ring_ctx *ctx, * The only exception is when we've detached the request and issue it * from an async worker thread, grab the lock for that case. */ - if (issue_flags & IO_URING_F_UNLOCKED) + if (unlikely(issue_flags & IO_URING_F_UNLOCKED)) mutex_lock(&ctx->uring_lock); lockdep_assert_held(&ctx->uring_lock); } From patchwork Tue Feb 6 16:22:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13547583 Received: from mail-il1-f180.google.com (mail-il1-f180.google.com [209.85.166.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1AA92134CCF for ; Tue, 6 Feb 2024 16:24:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707236662; cv=none; b=tI6fmglHYvavz7WqcnmGNYDwIlFEVVL5mjRgtwmZiSNZnU9aYUipJIhkKc59huT7FMVWSrE4i5cRoSg4pLI+Xqd+KBlCiDeFbUhlehIs8t0703X2wx0WdWP2qwDid+7SQyHSxLqlne8/UPblRaJv9f8Map++2XFiDIg4rA9maa4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707236662; c=relaxed/simple; bh=iW+NquMWGt9Gqk/vv4bbPeGDI6UBStnYtNVupziAj8A=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=aGBKqO+kwzbcwi+RaYNk10WvV8gCkgK4co8f3ccfwnFYYxz0HQ11tC6z3bC76cdygns+DDEy5Sd9jH7ws0YLkj8d7UenGd79ii76pz+hWPHo0U/aG8TK0M9h32UfUWLDdX3yqTKPqy2DuinsGU7W0qtj1xRZh0nLNw6ysfNsbO0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=p5RLuW0J; arc=none smtp.client-ip=209.85.166.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="p5RLuW0J" Received: by mail-il1-f180.google.com with SMTP id e9e14a558f8ab-363cedf6a49so759735ab.1 for ; Tue, 06 Feb 2024 08:24:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1707236660; x=1707841460; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UdoaioIXngn6g/+xmxUBVqHGk8dPSTC8+pDTCg3h3uM=; b=p5RLuW0JLhX4PIzCVvTRDTu2Nh1B9bVjG6BeuWLxUldztZKHtXRA1Xzci+a63eQVBI jWnnZRj5fF0zl+oVaS/Q+jfRrJ3f8i5o+hra51KhBqCu1l+PRUqDXki3N/7I4sWky9Oi P+O4iOYa+SF4ep/l7XsChioVG0A9ZG1IRfSZbY3GprvyHDHrxZKZogEdpJWbp2f/iHgz Y5zyqdE1oFQw0I2Jftam1tYq2k1OP9hHLyDxDqSnS5wHu5gu/IddOqiqvP4r7XsgK6Zr btARqD5QvVVwYphEP8mSae0ql8lb2uhmxUrzr0I7VpTDFInJgQ7e9HfDC8vRBoK4MZ/h wKJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707236660; x=1707841460; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UdoaioIXngn6g/+xmxUBVqHGk8dPSTC8+pDTCg3h3uM=; b=HFOXQD7ZURwexGAm/gYaykbjK5c0REpEHKNMnQ85NyvZ08EVpFsh3PbkyOhldVtLhp sCFz88pGnlG5mXYU+pJEd0P8k0QdGDKDOzIBcq2ZHzSuYLkxh9rSTN1drLeMltmabuP5 V/0NgmmwPRjKIS7k13zmZp8sXZCfeXXKuw28AceSXrs8hFJYCRWHXC3pcpTfnbWpdACJ l8aBhniMt4+7K3p87KJTHxBPYeXk/FvMwT5uQwLlH4RWTsVpEClwCA9fC0KvapDCJfdC azt6nQ3m41kqMOGWQosqB4DA1c4QYwwVMt0amLACXNHtH0+Q4pZbR/CkCAgpjGNH0lon ZUSw== X-Gm-Message-State: AOJu0YyvlpwEpZNHnZ2tlpE7mEjN72Mteb0kzNuiMEDtt+o+9B6BcKMZ kzqrjREMFT8ZXDOgKHq9iuWb+xDdLMf0hsYyxPHgKIwGpwXdOdZIFJdTv4aAYAQxHPJJ9FhbCmC gKAg= X-Google-Smtp-Source: AGHT+IGd2WwoIOPeR6d9HVP4RAsZdPoDivIitV5LvOSXMUv5J7nMyyv+IyilynHHejfy0a4ZdfW79Q== X-Received: by 2002:a6b:f308:0:b0:7c3:f836:aed with SMTP id m8-20020a6bf308000000b007c3f8360aedmr438490ioh.0.1707236659809; Tue, 06 Feb 2024 08:24:19 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id v17-20020a6b5b11000000b007bfe5fb5e0dsm520031ioh.51.2024.02.06.08.24.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 08:24:16 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 6/6] io_uring/rw: remove dead file == NULL check Date: Tue, 6 Feb 2024 09:22:52 -0700 Message-ID: <20240206162402.643507-7-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240206162402.643507-1-axboe@kernel.dk> References: <20240206162402.643507-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Any read/write opcode has needs_file == true, which means that we would've failed the request long before reaching the issue stage if we didn't successfully assign a file. This check has been dead forever, and is really a leftover from generic code. Signed-off-by: Jens Axboe --- io_uring/rw.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/io_uring/rw.c b/io_uring/rw.c index 0fb7a045163a..8ba93fffc23a 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -721,7 +721,7 @@ static int io_rw_init_file(struct io_kiocb *req, fmode_t mode) struct file *file = req->file; int ret; - if (unlikely(!file || !(file->f_mode & mode))) + if (unlikely(!(file->f_mode & mode))) return -EBADF; if (!(req->flags & REQ_F_FIXED_FILE))