From patchwork Thu Nov 14 04:14:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13874534 Received: from mail-wr1-f51.google.com (mail-wr1-f51.google.com [209.85.221.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EB4C41E882F for ; Thu, 14 Nov 2024 04:13:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731557641; cv=none; b=vEYnyhZ/Swr3mCNfGh3xAazXzhUY4YU86r85Z6905dvjC7f5V/RIwzaPgf7biM5iO6amIy7u9BKviTq0pk59kb5QJ8LEf27VoQrSn599HlXqOndxnslpv3S+upkF/BuH7jPTHiGY0Vzis6AloqEZ5v3qZcEpO2z4P5Wice78Vi8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731557641; c=relaxed/simple; bh=eGfmk5pHWRmz/5IEyopxX540NFO23oPagEaL0p8rDAI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=GfXNEe06U03uEhqTREMNhVfa1cW7XXwYXw4/1rAA+ioCDDFrib+OCAMtaN/4STF762FsSG/uwS+NFpyznGLjOZUiPTTSz52CSF5ac7hxWzIxfRfEHZxKayMk3kFdS+zHVtO4iUw3pLrVr7cq//CCwcjGsG280NWk9m7LBe47pN0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=NXk0rxwX; arc=none smtp.client-ip=209.85.221.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="NXk0rxwX" Received: by mail-wr1-f51.google.com with SMTP id ffacd0b85a97d-37ed7eb07a4so125410f8f.2 for ; Wed, 13 Nov 2024 20:13:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1731557638; x=1732162438; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LbuG+/DWjJeYlovcxNfbsJzRVRRM+t1Xz3YpJq8xpS0=; b=NXk0rxwXikeN/V1smIQABraEbwM+1IV3tUczzcRoUK7PPvxEPFIxCEgeHYTAgRtnLy vUCsJ87umG/jZtdOdR2KePjOaRibFVXEGo+lQCZ/FP9kk1jAZxpg8puFr334puCJl1F9 FdYcFOo2AkQimlpXacx1qKyX0Le8RGatGs4+7bHzsrjKZGib7dJFCZH/i7UtTcAt+WkA 5u7L7Lfr10dU7/6RXE/sa5q5QdRF8cpIRVNJmZWyiff16oyq6P5T8Pl2nbcjnhfTA1ug rS1V0drcZFCTbfYmpD8C5M7VeZ47SNSq5u8Y2JdW7FlnO5LEPVsviiC0jdMnQE3un836 ctFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731557638; x=1732162438; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LbuG+/DWjJeYlovcxNfbsJzRVRRM+t1Xz3YpJq8xpS0=; b=swHePCqlVOsh9cbG+i6F3pJFRyCNyuA+AqSeAWV5q/npMYhDJkxdvltjO1BJfceMJ0 HVJLpbXDLx1nc8bZtddxpR0pLctE2pQ+H75KUjRXbfhpvvL0a9VE0xuW7dgCMWQ86X+e vehfXZ2raC7LX8CPhSMDYI0rLbBbtQqbw4t3+aP2NKBR85bT6FioItQicADBEgBY1Glw NHK0iuClpUEkoRFKIq4H4kUEFGl+hTrp9RkhfLiqGuSyrqo6xoU3KDbvFFWjN4THfNJd KE6VM31wms9eC1LcU/lDkjfbAjmGWNUldKmc9qPPXkvbHqKQdO075VY5ZFwT2+CpHZN+ SlSQ== X-Gm-Message-State: AOJu0YzN/adSRlI4aObq6tIJMfgP0sSGOHexehwsH/Gj6scxKtgovqSV WkwUmKZcz3iqrMbAuPQNx5DvsiJm7VPHp1mF3uK6eB8V8ZDJN04AFhGz8w== X-Google-Smtp-Source: AGHT+IGqS0sdBB369z2SSe8Uw+6yFrfB6Z3f7GpElRqlh06aUVjAACDkQx2ilmhxZRUBC3pvBnMS2A== X-Received: by 2002:a5d:47ac:0:b0:37d:5282:1339 with SMTP id ffacd0b85a97d-3820df6136cmr4287581f8f.22.1731557637993; Wed, 13 Nov 2024 20:13:57 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.132.111]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3821ae311fbsm251936f8f.95.2024.11.13.20.13.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Nov 2024 20:13:57 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH 1/6] io_uring: fortify io_pin_pages with a warning Date: Thu, 14 Nov 2024 04:14:20 +0000 Message-ID: X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 We're a bit too frivolous with types of nr_pages arguments, converting it to long and back to int, passing an unsigned int pointer as an int pointer and so on. Shouldn't cause any problem but should be carefully reviewed, but until then let's add a WARN_ON_ONCE check to be more confident callers don't pass poorely checked arguents. Signed-off-by: Pavel Begunkov --- io_uring/memmap.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/io_uring/memmap.c b/io_uring/memmap.c index 85c66fa54956..6ab59c60dfd0 100644 --- a/io_uring/memmap.c +++ b/io_uring/memmap.c @@ -140,6 +140,8 @@ struct page **io_pin_pages(unsigned long uaddr, unsigned long len, int *npages) nr_pages = end - start; if (WARN_ON_ONCE(!nr_pages)) return ERR_PTR(-EINVAL); + if (WARN_ON_ONCE(nr_pages > INT_MAX)) + return ERR_PTR(-EOVERFLOW); pages = kvmalloc_array(nr_pages, sizeof(struct page *), GFP_KERNEL); if (!pages) From patchwork Thu Nov 14 04:14:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13874535 Received: from mail-wm1-f54.google.com (mail-wm1-f54.google.com [209.85.128.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6ED7B1E8826 for ; Thu, 14 Nov 2024 04:14:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731557643; cv=none; b=d9KrPp++ONauOa60fy0v5CL95RndMYq8Oz5i2ecGk6bqAY99TuDZuSvbvK09H56O2l0c78R/AFANopwDZO0H6F7oRODGrJZgjXzXrcnjIO+jyHsU1pfgWv+TlEgr59N3NxkLceFMwX31c4HUm/cuO6p3YyETmcVk2Hf597h9Ivo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731557643; c=relaxed/simple; bh=bOXuPylse+C0vi8bk1k20OU0yKwquVaExqg5AhfYLSY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=k3cz5g9Q/Ukqwt3K+WAACk4kkadAalJYrdGgG/p4OXr5o/XT2HtHfjk6JKX+/dmG6OOT+DW8fXW2tTbSr9PLhyTPfSSXbKNSUVriGgVz9BLGMmE9PwuGe0WQ/S5qF77LD62Dhiv0tVM+SZhuMvmzl6h3426PcxE9enMxj8dAMqY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=CeY6ZZ23; arc=none smtp.client-ip=209.85.128.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="CeY6ZZ23" Received: by mail-wm1-f54.google.com with SMTP id 5b1f17b1804b1-4315e62afe0so1969605e9.1 for ; Wed, 13 Nov 2024 20:14:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1731557639; x=1732162439; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Hwyltf5fgdrVd41AJQ/C7rIRuEDCxBHV6H77DqLRdb4=; b=CeY6ZZ23I7bZ8ULSUFhhZ0JOIQ41+pjd7YdU48OwjyOPzjpvX1QnOVqu0hpI8Z/YO8 oEg27T+1oUZELq5DXx2TK6cEc6Zv8ROMXhUTTId4vBvMorrgCIFJVLWvYuUwiQUC4cow mgPAx5PtQCV3jSKHKIGQurwct5hCu/rt/N5KU+wyvml4kCdnn9x2/CYpT0JUvU1aVxJQ yNeCp8WyfMI1nN+yzs9BUeBjurR+sYYfMc/t7Ogss+ys9kdchhcHordDf1j58J6yEIlB EkTmub1vRognRbh5h/DqIOjB61wOOClU3T+dIBgOWnpclxXxVTdXzp1QPc3PPP36G+rY skAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731557639; x=1732162439; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Hwyltf5fgdrVd41AJQ/C7rIRuEDCxBHV6H77DqLRdb4=; b=XOfRDOs93ah1vRWT1uXGeK7z0X8tjIdu5F1REPP5dUCZMPoU6ncYY8fc0Ud1EVFMLb WAyXjb/kFIGnR0TWaS3wDu/rHCBKTc/DQGJ/iAL1fCSfGx2rRTd82ke8+pCD6ffkmVgJ 7Kp9Bt0zOVTYN8h+KyEjbUbMIsn2LMfiBwV71oEnfSQP7dd2SDFdHx0Wt6MfzQR5IbvB FCq3p9mwI4Tq6F1/qJJsaoLRBuoApGkP1oYU0KiVQzNca9PAS65xXlBfMAf5LQl3y9IW Ad44nB8WiVBQBDNluW7ix1MCFEiJCMcogdeb5dfS76bu4T2cTkjB7TtGxEwHDLnvl2wN bf2g== X-Gm-Message-State: AOJu0YyWH8ABEhDPXdDwQlPLwOqHhpcZo+LF+SqQIz+1kJYnLMPkokNv qzMctYsplCo8rWUL+fe1K0ClJypKy+QkSWmRENntnyRAUCppd94Xm0O5JA== X-Google-Smtp-Source: AGHT+IGKIufDLs3bta+PL+WA+/3gFF+wlnPP7XlwH8u9p++yKtxJ2th3LO/5NHZPaUSIR0vd+/G2JQ== X-Received: by 2002:a05:6000:1564:b0:37d:4d21:350c with SMTP id ffacd0b85a97d-381f186cc7fmr18473685f8f.13.1731557639328; Wed, 13 Nov 2024 20:13:59 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.132.111]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3821ae311fbsm251936f8f.95.2024.11.13.20.13.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Nov 2024 20:13:58 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH 2/6] io_uring: disable ENTER_EXT_ARG_REG for IOPOLL Date: Thu, 14 Nov 2024 04:14:21 +0000 Message-ID: X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 IOPOLL doesn't use the extended arguments, no need for it to support IORING_ENTER_EXT_ARG_REG. Let's disable it for IOPOLL, if anything it leaves more space for future extensions. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index bd71782057de..464a70bde7e6 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -3214,12 +3214,8 @@ static int io_validate_ext_arg(struct io_ring_ctx *ctx, unsigned flags, if (!(flags & IORING_ENTER_EXT_ARG)) return 0; - - if (flags & IORING_ENTER_EXT_ARG_REG) { - if (argsz != sizeof(struct io_uring_reg_wait)) - return -EINVAL; - return PTR_ERR(io_get_ext_arg_reg(ctx, argp)); - } + if (flags & IORING_ENTER_EXT_ARG_REG) + return -EINVAL; if (argsz != sizeof(arg)) return -EINVAL; if (copy_from_user(&arg, argp, sizeof(arg))) From patchwork Thu Nov 14 04:14:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13874536 Received: from mail-wm1-f48.google.com (mail-wm1-f48.google.com [209.85.128.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2821D1E8836 for ; Thu, 14 Nov 2024 04:14:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731557644; cv=none; b=MDT3d0y6XkRI6lVDjE24gd/iBlqhdMe/PRvP3x0Ow2mza7dNI0bkTntf9zhPAb3llRzEa1yfSWVhk8ieIIWHiBtTusZJIFojSu8X7Mh6yqSmDG1xuMGfRPZShSEFmoe10qJtrwBBu3UkSlr2iSe/7kd7rwB6YA+E6t5z+cJpGOU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731557644; c=relaxed/simple; bh=v2IbOLpYo3a22Mop7BMSABJ+5BM1F7i4KaPCY4tFSYM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=r1oMAvTNTNlwQgcpU/PXfPW6xab68JuM2lNY6B1wcm9EsSPkHuzncQxxKVEiJJFY8zgzmCxVBEcB7W3W/8iucIG14PDzxjnA66NPRbun/gdFDcPf+R678JjkPbwLVRASSpDwRwwZkgSpBiac+xq6cDJJpRs3Y/vHJLCtpnodFtE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=XF/Spkln; arc=none smtp.client-ip=209.85.128.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="XF/Spkln" Received: by mail-wm1-f48.google.com with SMTP id 5b1f17b1804b1-43163667f0eso1868585e9.0 for ; Wed, 13 Nov 2024 20:14:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1731557641; x=1732162441; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9nJxolNITZkKjcgVgsRnwP3J3cPf7OR1t7KjI8tbabc=; b=XF/Spklnsvj+kH+6zhRifnhi9M+Eh2MFIenQIVCkLZKKqyb+ZgJ4/+CYKatDsWjXsm yAsdNgU0jVJeD5W254ojZz/ppGd8e8oxDh5B/oQIr34a6Oo01W9JI51TacQxMSYW/8pm j0tgw5IrxgLmxmyzrM4TOfgmoh1PaI5v5uH+RWMWN/aVsKudvlEEmqNoffkN/tI81Ksn h7yy4XlvSupfhRlYvjRLwXEQfsuuvupdO2sLh+98GIIfZv/2yLBC8lP8jniRunAp6p3A KcTDPVwfwJb/5CsZ+7LwUzuyJXX7CHwBGSZzdBLmf020K7CmUdd7D+a6sS/Hq+Uh4rrL kNmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731557641; x=1732162441; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9nJxolNITZkKjcgVgsRnwP3J3cPf7OR1t7KjI8tbabc=; b=V6zvqGX3lIEduaxS0qYtItKi+0sXnl9hCZh6lMulELjQlf0Sw9oxdylpykxw2bRWFY IP92dLjAcpjfjwRyPiw73OKMlQ9PzzjfBsCsHIeFdODlDN7qtQ9dheoEB4rzU0Ph4iG1 DrOMWg99hRwS4J6Ux7eSH3DonkMSd9B7fQPu+oJMd6n9hjpHLEPMDBcvO3U+Eq00fOHm x4FLGYBNrJDPbdPecPqzwQN87q+D1RJfIXxb2GTEHOmhuaA6tjvgd24ulDKcW9UrkVn4 F+rDDM+oLyYu7chBlgt2KATbi45sl8MIH6xh0q1Bk6R0Kn1x6ceXST0QixUpCk5w0J4+ cHag== X-Gm-Message-State: AOJu0YwvuaHtBdUj9BKwqNathkFPq3V1DwKxBIxhmsdXSoh3kTYhC1XJ TGaC+QTCvKBc+LupaN9qj/C4mzwKeRLJ99pFb7tThQMLFnRCdTu9wKMoHw== X-Google-Smtp-Source: AGHT+IFnKH7TGUFUV7SeoTz6S4nXTmBfGp6q1B3AquSV0cOic0sD0LVM3hxNHDXxX60QpOCnEO9dgg== X-Received: by 2002:a05:6000:2cb:b0:374:ba23:4b3f with SMTP id ffacd0b85a97d-382184fd782mr485763f8f.9.1731557640732; Wed, 13 Nov 2024 20:14:00 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.132.111]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3821ae311fbsm251936f8f.95.2024.11.13.20.13.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Nov 2024 20:14:00 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH 3/6] io_uring: temporarily disable registered waits Date: Thu, 14 Nov 2024 04:14:22 +0000 Message-ID: <70b1d1d218c41ba77a76d1789c8641dab0b0563e.1731556844.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Disable wait argument registration as it'll be replaced with a more generic feature. We'll still need IORING_ENTER_EXT_ARG_REG parsing in a few commits so leave it be. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 10 ----- include/uapi/linux/io_uring.h | 3 -- io_uring/io_uring.c | 10 ----- io_uring/register.c | 82 ---------------------------------- io_uring/register.h | 1 - 5 files changed, 106 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 072e65e93105..52a5da99a205 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -330,14 +330,6 @@ struct io_ring_ctx { atomic_t cq_wait_nr; atomic_t cq_timeouts; struct wait_queue_head cq_wait; - - /* - * If registered with IORING_REGISTER_CQWAIT_REG, a single - * page holds N entries, mapped in cq_wait_arg. cq_wait_index - * is the maximum allowable index. - */ - struct io_uring_reg_wait *cq_wait_arg; - unsigned char cq_wait_index; } ____cacheline_aligned_in_smp; /* timeouts */ @@ -431,8 +423,6 @@ struct io_ring_ctx { unsigned short n_sqe_pages; struct page **ring_pages; struct page **sqe_pages; - - struct page **cq_wait_page; }; struct io_tw_state { diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 5d08435b95a8..132f5db3d4e8 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -627,9 +627,6 @@ enum io_uring_register_op { /* resize CQ ring */ IORING_REGISTER_RESIZE_RINGS = 33, - /* register fixed io_uring_reg_wait arguments */ - IORING_REGISTER_CQWAIT_REG = 34, - /* this goes last */ IORING_REGISTER_LAST, diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 464a70bde7e6..286b7bb73978 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2709,7 +2709,6 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) io_alloc_cache_free(&ctx->msg_cache, io_msg_cache_free); io_futex_cache_free(ctx); io_destroy_buffers(ctx); - io_unregister_cqwait_reg(ctx); mutex_unlock(&ctx->uring_lock); if (ctx->sq_creds) put_cred(ctx->sq_creds); @@ -3195,15 +3194,6 @@ void __io_uring_cancel(bool cancel_all) static struct io_uring_reg_wait *io_get_ext_arg_reg(struct io_ring_ctx *ctx, const struct io_uring_getevents_arg __user *uarg) { - struct io_uring_reg_wait *arg = READ_ONCE(ctx->cq_wait_arg); - - if (arg) { - unsigned int index = (unsigned int) (uintptr_t) uarg; - - if (index <= ctx->cq_wait_index) - return arg + index; - } - return ERR_PTR(-EFAULT); } diff --git a/io_uring/register.c b/io_uring/register.c index 45edfc57963a..3c5a3cfb186b 100644 --- a/io_uring/register.c +++ b/io_uring/register.c @@ -570,82 +570,6 @@ static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg) return ret; } -void io_unregister_cqwait_reg(struct io_ring_ctx *ctx) -{ - unsigned short npages = 1; - - if (!ctx->cq_wait_page) - return; - - io_pages_unmap(ctx->cq_wait_arg, &ctx->cq_wait_page, &npages, true); - ctx->cq_wait_arg = NULL; - if (ctx->user) - __io_unaccount_mem(ctx->user, 1); -} - -/* - * Register a page holding N entries of struct io_uring_reg_wait, which can - * be used via io_uring_enter(2) if IORING_GETEVENTS_EXT_ARG_REG is set. - * If that is set with IORING_GETEVENTS_EXT_ARG, then instead of passing - * in a pointer for a struct io_uring_getevents_arg, an index into this - * registered array is passed, avoiding two (arg + timeout) copies per - * invocation. - */ -static int io_register_cqwait_reg(struct io_ring_ctx *ctx, void __user *uarg) -{ - struct io_uring_cqwait_reg_arg arg; - struct io_uring_reg_wait *reg; - struct page **pages; - unsigned long len; - int nr_pages, poff; - int ret; - - if (ctx->cq_wait_page || ctx->cq_wait_arg) - return -EBUSY; - if (copy_from_user(&arg, uarg, sizeof(arg))) - return -EFAULT; - if (!arg.nr_entries || arg.flags) - return -EINVAL; - if (arg.struct_size != sizeof(*reg)) - return -EINVAL; - if (check_mul_overflow(arg.struct_size, arg.nr_entries, &len)) - return -EOVERFLOW; - if (len > PAGE_SIZE) - return -EINVAL; - /* offset + len must fit within a page, and must be reg_wait aligned */ - poff = arg.user_addr & ~PAGE_MASK; - if (len + poff > PAGE_SIZE) - return -EINVAL; - if (poff % arg.struct_size) - return -EINVAL; - - pages = io_pin_pages(arg.user_addr, len, &nr_pages); - if (IS_ERR(pages)) - return PTR_ERR(pages); - ret = -EINVAL; - if (nr_pages != 1) - goto out_free; - if (ctx->user) { - ret = __io_account_mem(ctx->user, 1); - if (ret) - goto out_free; - } - - reg = vmap(pages, 1, VM_MAP, PAGE_KERNEL); - if (reg) { - ctx->cq_wait_index = arg.nr_entries - 1; - WRITE_ONCE(ctx->cq_wait_page, pages); - WRITE_ONCE(ctx->cq_wait_arg, (void *) reg + poff); - return 0; - } - ret = -ENOMEM; - if (ctx->user) - __io_unaccount_mem(ctx->user, 1); -out_free: - io_pages_free(&pages, nr_pages); - return ret; -} - static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode, void __user *arg, unsigned nr_args) __releases(ctx->uring_lock) @@ -840,12 +764,6 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode, break; ret = io_register_resize_rings(ctx, arg); break; - case IORING_REGISTER_CQWAIT_REG: - ret = -EINVAL; - if (!arg || nr_args != 1) - break; - ret = io_register_cqwait_reg(ctx, arg); - break; default: ret = -EINVAL; break; diff --git a/io_uring/register.h b/io_uring/register.h index 3e935e8fa4b2..a5f39d5ef9e0 100644 --- a/io_uring/register.h +++ b/io_uring/register.h @@ -5,6 +5,5 @@ int io_eventfd_unregister(struct io_ring_ctx *ctx); int io_unregister_personality(struct io_ring_ctx *ctx, unsigned id); struct file *io_uring_register_get_file(unsigned int fd, bool registered); -void io_unregister_cqwait_reg(struct io_ring_ctx *ctx); #endif From patchwork Thu Nov 14 04:14:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13874537 Received: from mail-wr1-f46.google.com (mail-wr1-f46.google.com [209.85.221.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C1A511DF72E for ; Thu, 14 Nov 2024 04:14:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731557646; cv=none; b=HEXIRfIo5CL83mqhjmhaK+qNaCPj89KNl+iyhcdDZva2vnhBLgJAcgtW2yM+k5NwMmoirzK6KHrAfXMQNg7clWyWri0HR0aFUxFHKAkqj6Qpamtm/jNWI5M9B6wkv/OQo1WMD/j+4ORis9ytHnzAjFFGNJgxzEgq/BrJYYfi1/8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731557646; c=relaxed/simple; bh=XGNR0R9bVnjdyjqTCXAwbp0v04eZfBEsyas4/N/tT50=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rSzc3QylE2XxaSLxKMgAr1V2EQnODBS9vPatzvYpu0f7gSIe73Q62UfkVJAbLOxVc1FxM28wudyYCfe0ONvUCHpMYY6VfZUplKppXTAidPCcGjGNO5MlnayAcbpbMdxmMSELrH38dnfOs9jme+NIi9w6uEO73x/Wj8X7Hjr67xM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=hpNy1ax5; arc=none smtp.client-ip=209.85.221.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="hpNy1ax5" Received: by mail-wr1-f46.google.com with SMTP id ffacd0b85a97d-382171fe8b1so175713f8f.1 for ; Wed, 13 Nov 2024 20:14:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1731557643; x=1732162443; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QevZ2XM/oLWl+9z9+p8K4+bD59erMc6Ct/g8gJjyWoY=; b=hpNy1ax5ekraWGDxFSrEwZgKqu1lgzFajUOoW96ePdoIR4ey1xkgFTG/gOgFceZ1gu mY2/Mawgz2trT8grZX+3whkqj6MS43tNx2OIeGvZUNv8YtE2R26TehiCpA2yzkr/2pmQ xiEpO59TDkOq9qNXRbtqRKizWaURov4nGBgxZg6YFSnU+Ei8MXeJBy4I1YDgjxeRRDFI 0Qb3Oq/A5FRU4ZF0Dp+PsHUxbkVXWNAcEQVuaLjr/q5KKGN5CGaoYDs4VyK3YCWuCmjH MKb/qCN1HR29jq7U+Di98nJHNvztUzMhvj4K8zkkvFRMpISyJYcFjuSdUKQ+sTkc8C/G Py4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731557643; x=1732162443; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QevZ2XM/oLWl+9z9+p8K4+bD59erMc6Ct/g8gJjyWoY=; b=ARicuM2N6m+SXGy9o/FdH3GR4DPl3vnHZRMA/HAkWeha2IdC4uE+oxmbkRfzc8DutD vtk2cXZEzOqxIi3CDr3kXNZarD2OeVpqjgnL+hzxkihdmt1nCY+a4lQkQtGEsx3opWEc 7VUNzDq1+9jcbVcwbsIrffHJRa7gh1XGhYj4937Eg8WuW5ih912/bskbxVP3SsmfXaUP AlVBoU+ne3HlxtM8IbTpmqBgKdzaRDPoGgaeUB+pbmRVw0vteiU3vf746OGxJpLxzk2p O8CD4UjBsYOng4RMlHBWxLSpVpy3iYAjf1yIZwy5O8u4B5iSxLauJXglvcaoBpIuOIrM 9WTA== X-Gm-Message-State: AOJu0YyyCH4xlvP6/xSmOmJFC7WULlKEi8s/RKVKc80xLKLuJK0jwWy5 xaETX7t8lO3r1b8NjDg/YL3iEJ4sMoKk7a9DtRKDFiu70u54RayR3mejcQ== X-Google-Smtp-Source: AGHT+IF7PNOJCpu6HZRqcfBPV7OvKg5sSIv9Ho53jENvP7WjPzPxMr4yoKYkjxjtzp1cIXNPyNCDKQ== X-Received: by 2002:a05:6000:1568:b0:374:cd3c:db6d with SMTP id ffacd0b85a97d-38213fe958bmr1379505f8f.6.1731557642774; Wed, 13 Nov 2024 20:14:02 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.132.111]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3821ae311fbsm251936f8f.95.2024.11.13.20.14.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Nov 2024 20:14:02 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH 4/6] io_uring: introduce memory regions Date: Thu, 14 Nov 2024 04:14:23 +0000 Message-ID: X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 We've got a good number of mappings we share with the userspace, that includes the main rings, provided buffer rings and at least a couple more types. And all of them duplicate some of the code for page pinning, mmap'ing and attempts to optimise it with huge pages. Introduce a notion of regions. For userspace it's just a new structure called struct io_uring_region_desc which supposed to parameterise all such mapping / queues creations. It either represents a user provided memory, in which case the user_addr field should point to it, or a request to the kernel to creating the memory, in which case the user is supposed to mmap it after using the offset returned in the mmap_offset field. With uniform userspace API we can avoid additional boiler plate code and when we'd be adding some optimisation it'll be applied to all mapping types. Internally, there is a new structure struct io_mapped_region holding all relevant runtime information and some helpers to work with it. This patch limits it to user provided regions, which will be extended as a follow up work. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 6 ++++ include/uapi/linux/io_uring.h | 13 +++++++ io_uring/memmap.c | 65 ++++++++++++++++++++++++++++++++++ io_uring/memmap.h | 14 ++++++++ 4 files changed, 98 insertions(+) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 52a5da99a205..1d3a37234ace 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -75,6 +75,12 @@ struct io_hash_table { unsigned hash_bits; }; +struct io_mapped_region { + struct page **pages; + void *vmap_ptr; + size_t nr_pages; +}; + /* * Arbitrary limit, can be raised if need be */ diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 132f5db3d4e8..7ceeccbbf4cb 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -647,6 +647,19 @@ struct io_uring_files_update { __aligned_u64 /* __s32 * */ fds; }; +enum { + /* initialise with user memory pointed by user_addr */ + IORING_REGION_USER_MEM = 1, +}; + +struct io_uring_region_desc { + __u64 user_addr; + __u64 size; + __u64 flags; + __u64 mmap_offset; + __u64 __resv[4]; +}; + /* * Register a fully sparse file space, rather than pass in an array of all * -1 file descriptors. diff --git a/io_uring/memmap.c b/io_uring/memmap.c index 6ab59c60dfd0..6b03f5641ef3 100644 --- a/io_uring/memmap.c +++ b/io_uring/memmap.c @@ -12,6 +12,7 @@ #include "memmap.h" #include "kbuf.h" +#include "rsrc.h" static void *io_mem_alloc_compound(struct page **pages, int nr_pages, size_t size, gfp_t gfp) @@ -194,6 +195,70 @@ void *__io_uaddr_map(struct page ***pages, unsigned short *npages, return ERR_PTR(-ENOMEM); } +void io_free_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr) +{ + if (mr->pages) + unpin_user_pages(mr->pages, mr->nr_pages); + if (mr->vmap_ptr) + vunmap(mr->vmap_ptr); + if (mr->nr_pages && ctx->user) + __io_unaccount_mem(ctx->user, mr->nr_pages); + + memset(mr, 0, sizeof(*mr)); +} + +int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, + struct io_uring_region_desc *reg) +{ + int pages_accounted = 0; + struct page **pages; + int nr_pages, ret; + void *vptr; + u64 end; + + if (WARN_ON_ONCE(mr->pages || mr->vmap_ptr || mr->nr_pages)) + return -EFAULT; + if (memchr_inv(®->__resv, 0, sizeof(reg->__resv))) + return -EINVAL; + if (reg->flags != IORING_REGION_USER_MEM) + return -EINVAL; + if (!reg->user_addr) + return -EFAULT; + if (!reg->size || reg->mmap_offset) + return -EINVAL; + if ((reg->size >> PAGE_SHIFT) > INT_MAX) + return E2BIG; + if ((reg->user_addr | reg->size) & ~PAGE_MASK) + return -EINVAL; + if (check_add_overflow(reg->user_addr, reg->size, &end)) + return -EOVERFLOW; + + pages = io_pin_pages(reg->user_addr, reg->size, &nr_pages); + if (IS_ERR(pages)) + return PTR_ERR(pages); + + if (ctx->user) { + ret = __io_account_mem(ctx->user, nr_pages); + if (ret) + goto out_free; + pages_accounted = nr_pages; + } + + vptr = vmap(pages, nr_pages, VM_MAP, PAGE_KERNEL); + if (!vptr) + goto out_free; + + mr->pages = pages; + mr->vmap_ptr = vptr; + mr->nr_pages = nr_pages; + return 0; +out_free: + if (pages_accounted) + __io_unaccount_mem(ctx->user, pages_accounted); + io_pages_free(&pages, nr_pages); + return ret; +} + static void *io_uring_validate_mmap_request(struct file *file, loff_t pgoff, size_t sz) { diff --git a/io_uring/memmap.h b/io_uring/memmap.h index 5cec5b7ac49a..f361a635b6c7 100644 --- a/io_uring/memmap.h +++ b/io_uring/memmap.h @@ -22,4 +22,18 @@ unsigned long io_uring_get_unmapped_area(struct file *file, unsigned long addr, unsigned long flags); int io_uring_mmap(struct file *file, struct vm_area_struct *vma); +void io_free_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr); +int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr, + struct io_uring_region_desc *reg); + +static inline void *io_region_get_ptr(struct io_mapped_region *mr) +{ + return mr->vmap_ptr; +} + +static inline bool io_region_is_set(struct io_mapped_region *mr) +{ + return !!mr->nr_pages; +} + #endif From patchwork Thu Nov 14 04:14:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13874538 Received: from mail-wr1-f50.google.com (mail-wr1-f50.google.com [209.85.221.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5DE211E8827 for ; Thu, 14 Nov 2024 04:14:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731557649; cv=none; b=gfl9TbttUFK8yEEm0H/N9WfmAvNGuEIv5RY1LkMtalmO2seo3HRbBOlwln2qmWGPOaC9arKv6IeYNwpXabcG3ezE62vDR2uMAMz4RoS3/zgXwdVnvkHwwESXVBE+eTh3TC07UvwHxVoBq7THyI4uexejNJHsa5pNdU+7WUuozgo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731557649; c=relaxed/simple; bh=/obiZqY5u6S9sB6CawVtRypVIKFQ0tCsRKXtcq4bR9I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=sgaLpNyEhT3rPi+i95LN6FrIPjCZfLKWr3tg3CWBg/LfKMUwqAfe2KrEcCTn+scWncGHBRdobm6+oKK8ffno5XUHK901wsi+yMtkmc8er35iswTYss+6qp16aWdoRabbJCOpaQHUKm+X4UQONzsCHedg5qUhD7jdq/kGHr/tCi0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=fBkRajoV; arc=none smtp.client-ip=209.85.221.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="fBkRajoV" Received: by mail-wr1-f50.google.com with SMTP id ffacd0b85a97d-37d6ff1cbe1so141693f8f.3 for ; Wed, 13 Nov 2024 20:14:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1731557645; x=1732162445; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HzJUtYhk70OYS1BXABOZzA3PP2mUGfbEHdHP14EV0Uc=; b=fBkRajoVo2gogtxZZhXdDqXvy0A9xxkjJdDia9KWvbC3SZzmjeF40DUyv2aF8d75Xl Xp1X5V8lON62QtoIMbsvkxg5g6fM/EK4/XVO0QiSLtFoS60eradiOSGmEx+Z/L2Ipr1A 22EKtlllG53Heb+NXYR3intFxQPewuSKAYsv0uSFsgKA7w+uOJe/BaATYN6YI2nsdXWh bfTcXouToz2l25UWVK5Y+H7U60DVIW5DdrLyWdRMf18YaK3edRPhvoNpS2vAOdt/XIqX UECuGXIKfevSx2dYekDx2W4qSVKnbJJviQ3r7hFIyKjgKLLZzYXYJcqaPBTXP2zs2mjd fQ4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731557645; x=1732162445; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HzJUtYhk70OYS1BXABOZzA3PP2mUGfbEHdHP14EV0Uc=; b=QhNJofnguaigsw4fxLa6YxqJ3d71hfr8NoHrWjBmvKDoBS9rV8s+RLKKL3+t1fthfu 1ztRZfpjQyISvT9+/daHddeEDj3++mLs+BsL2oqZ3vqTczg/2Xej+x0oWLEJnej1sTGs FDM8wnECS2oU89fBljsHzP2ryZzpHSkDNl6XMx0H44otqiA7zkR3rHNvonzsHZiEdd55 ijhWpWTBKRR9NKLA7JYOm/ZqbfBEGJjhDvwkiba5yErNfTVlOFbZAg01LVQJlPjWqPil HLflmUPbbnhJY/OIyFiPsmxVNI+qitK1Vibp4OslxBlKLaU5APcl+nGjkS9j2uOp1RY9 ERRg== X-Gm-Message-State: AOJu0YxqvMngvhWdThVsSaUCXsFH1PdUlTDgO+B1A4/tn7M1RocpcvSE +7QeOW6U0KvjX5YB3XtsY9mk4uQOHgCNBRMNT2knD16+ycNfYcuSUAmQow== X-Google-Smtp-Source: AGHT+IHOVG0IjOQKRQ3aUSzPxndxQTkGVqICejHcKJSt+p8sMETbhyWbOL7jjM3zJfIMulk0zP+WeA== X-Received: by 2002:a5d:6d8c:0:b0:382:d8c:2dc3 with SMTP id ffacd0b85a97d-3820df71a43mr4218824f8f.14.1731557644890; Wed, 13 Nov 2024 20:14:04 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.132.111]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3821ae311fbsm251936f8f.95.2024.11.13.20.14.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Nov 2024 20:14:04 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH 5/6] io_uring: add parameter region registration Date: Thu, 14 Nov 2024 04:14:24 +0000 Message-ID: <481f7a4973b86038f6b03f0d1e9ce4e127ced315.1731556844.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Allow the user to pre-register a region for passing various paramteres. To use it for passing wait loop arguments, which is wired in the following commit, the region has to be registered with the IORING_PARAM_REGION_WAIT_ARG flag set. The flag also requires the context to be currently disabled, i.e. IORING_SETUP_R_DISABLED, to avoid races with otherwise potentially running waiters. This will also be useful in the future for various request / SQE arguments like iovec, the meta read/write API, and also for BPF. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 6 ++++ include/uapi/linux/io_uring.h | 13 ++++++++ io_uring/io_uring.c | 1 + io_uring/register.c | 59 ++++++++++++++++++++++++++++++++++ 4 files changed, 79 insertions(+) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 1d3a37234ace..aa5f5ea98076 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -324,6 +324,9 @@ struct io_ring_ctx { unsigned cq_entries; struct io_ev_fd __rcu *io_ev_fd; unsigned cq_extra; + + void *cq_wait_arg; + size_t cq_wait_size; } ____cacheline_aligned_in_smp; /* @@ -429,6 +432,9 @@ struct io_ring_ctx { unsigned short n_sqe_pages; struct page **ring_pages; struct page **sqe_pages; + + /* used for optimised request parameter and wait argument passing */ + struct io_mapped_region param_region; }; struct io_tw_state { diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 7ceeccbbf4cb..49b94029c137 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -627,6 +627,8 @@ enum io_uring_register_op { /* resize CQ ring */ IORING_REGISTER_RESIZE_RINGS = 33, + IORING_REGISTER_PARAM_REGION = 34, + /* this goes last */ IORING_REGISTER_LAST, @@ -660,6 +662,17 @@ struct io_uring_region_desc { __u64 __resv[4]; }; +enum { + /* expose the region as registered wait arguments */ + IORING_PARAM_REGION_WAIT_ARG = 1, +}; + +struct io_uring_param_region_reg { + __u64 region_uptr; /* struct io_uring_region_desc * */ + __u64 flags; + __u64 __resv[2]; +}; + /* * Register a fully sparse file space, rather than pass in an array of all * -1 file descriptors. diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 286b7bb73978..c640b8a4ceee 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2709,6 +2709,7 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) io_alloc_cache_free(&ctx->msg_cache, io_msg_cache_free); io_futex_cache_free(ctx); io_destroy_buffers(ctx); + io_free_region(ctx, &ctx->param_region); mutex_unlock(&ctx->uring_lock); if (ctx->sq_creds) put_cred(ctx->sq_creds); diff --git a/io_uring/register.c b/io_uring/register.c index 3c5a3cfb186b..d1ba14da37ea 100644 --- a/io_uring/register.c +++ b/io_uring/register.c @@ -570,6 +570,59 @@ static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg) return ret; } +/* + * Register a page holding N entries of struct io_uring_reg_wait, which can + * be used via io_uring_enter(2) if IORING_GETEVENTS_EXT_ARG_REG is set. + * If that is set with IORING_GETEVENTS_EXT_ARG, then instead of passing + * in a pointer for a struct io_uring_getevents_arg, an index into this + * registered array is passed, avoiding two (arg + timeout) copies per + * invocation. + */ +static int io_register_mapped_heap(struct io_ring_ctx *ctx, void __user *uarg) +{ + struct io_uring_param_region_reg __user *reg_uptr = uarg; + struct io_uring_param_region_reg reg; + struct io_uring_region_desc __user *rd_uptr; + struct io_uring_region_desc rd; + int ret; + + if (io_region_is_set(&ctx->param_region)) + return -EBUSY; + if (copy_from_user(®, reg_uptr, sizeof(reg))) + return -EFAULT; + rd_uptr = u64_to_user_ptr(reg.region_uptr); + if (copy_from_user(&rd, rd_uptr, sizeof(rd))) + return -EFAULT; + + if (memchr_inv(®.__resv, 0, sizeof(reg.__resv))) + return -EINVAL; + if (reg.flags != IORING_PARAM_REGION_WAIT_ARG) + return -EINVAL; + + /* + * This ensures there are no waiters. Waiters are unlocked and it's + * hard to synchronise with them, especially if we need to initialise + * the region. + */ + if ((reg.flags & IORING_PARAM_REGION_WAIT_ARG) && + !(ctx->flags & IORING_SETUP_R_DISABLED)) + return -EINVAL; + + ret = io_create_region(ctx, &ctx->param_region, &rd); + if (ret) + return ret; + if (copy_to_user(rd_uptr, &rd, sizeof(rd))) { + io_free_region(ctx, &ctx->param_region); + return -EFAULT; + } + + if (reg.flags & IORING_PARAM_REGION_WAIT_ARG) { + ctx->cq_wait_arg = io_region_get_ptr(&ctx->param_region); + ctx->cq_wait_size = rd.size; + } + return 0; +} + static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode, void __user *arg, unsigned nr_args) __releases(ctx->uring_lock) @@ -764,6 +817,12 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode, break; ret = io_register_resize_rings(ctx, arg); break; + case IORING_REGISTER_PARAM_REGION: + ret = -EINVAL; + if (!arg || nr_args != 1) + break; + ret = io_register_mapped_heap(ctx, arg); + break; default: ret = -EINVAL; break; From patchwork Thu Nov 14 04:14:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13874539 Received: from mail-wr1-f53.google.com (mail-wr1-f53.google.com [209.85.221.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8ECDF1E764D for ; Thu, 14 Nov 2024 04:14:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731557650; cv=none; b=HnkXVnM1FRkCw8/R4W1EGC0wB5WDdC1lQAHOk4dPYZRTHoMFRVaYViCMQFNQIiC6GDNOOq0+juJse9hFs/cg4tR2YQfFSggyXKEf03mAQAltw2TX1hQOMM07/Fz5tQAVlB/EW72Xa9fqMuRQAskd3OqJywqoeIDlMTcLaUWkyvY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731557650; c=relaxed/simple; bh=u3Wc0/xVaiErX3YVMXm4hPEwS5V4DhAIBRegPiFdKGk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cRnEE9zbI6dR4OszKNWZMjrrQev6Dj/9IZWD54r3kSTA/nf72B2/OyrJbf/xYhZfyvneKIdZJraNAfW35nwsuXsktVS+VG5ASInon8gnKWmNuU5CQYuTANZx3PCP2gkRV6cOY/q+Yg97V0hHQMPMsU/x0Ki4jmi4XGs0wnAj5Yk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=X/3tAWrr; arc=none smtp.client-ip=209.85.221.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="X/3tAWrr" Received: by mail-wr1-f53.google.com with SMTP id ffacd0b85a97d-3821a905dbcso100451f8f.0 for ; Wed, 13 Nov 2024 20:14:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1731557647; x=1732162447; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tWEpMmG2R8KKbjhtSOySdI1WYY5PGi2mDcfqLVFJT8Y=; b=X/3tAWrrQMiCMQVbjlggijH/m+jFfN/eoIvOQBXi6UU9mhvpleMKxRoBfW+lqCbzzV kyjbIuMlTYVB9Tw0EjMpxTd/BgsmcJdQ4OL/wr2rOaUy4efJTzoX3Vt75srLJpkFmXnZ FUj7xUX8SqSwv7osOFzIuOAcr/lYZQ1K4abzCUw8z2KOkZ0bs1iqXN2Qz+UjsYc2jMVj Mya8QCrd4JTKZHmYgCWhXv5msmosPywTFCLr8kVcXezp4JPbi3znIu+ZHv4pk1khPcu9 Ww46sw2dxzdlUw4Xl1wDL7Xlo/YntKpKTm0vtK1Tj4s3bhVMiRcf5CbL/x6FLe44hUZV xiJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731557647; x=1732162447; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tWEpMmG2R8KKbjhtSOySdI1WYY5PGi2mDcfqLVFJT8Y=; b=puAXJ3Vh3xwGN6NaaniPNXrFx/Eqpt2APnJ5c04wz6ZgQ/QzNVMYNfIiZ+vHdPlC7F sQuBAE8fWNuY+o2dVwHep3IRiihb4yBwK3JWs+odzxQYVbN7fu4JOO8alq8D8Bu84B0b mvYu/EpdvyE8CmsyeRHBD7koDdXYNrQn2QqjQVrMrho+D1F9bBTh02zEQAbSLI5S+ghu zGeiBFvps2GhlXNm3PytFI/SpLwwm4zvNFHJJwfcz7DTPFpP2Bq0Ri7NIavXPSaTzGLR Eq5Nbgq9Ftn+iAT+RHTRw2X3L9IKLHjyUff/5gQ4dmH4BYP9VckS0iIgKP2NEK5ZY/6V yxgA== X-Gm-Message-State: AOJu0YwYuSP1nhw6cn7AV/DvOW3M6dOmjw8e/ZLlxo24rTbL7x+i8ZbY NpHzXbLzI5mw6hUjN95/BI2YpUbFwLtJJej3HMIzmqzfA0HhiL9xwFfYpw== X-Google-Smtp-Source: AGHT+IFLI0PFWBdtVeYmAZVvdqnP3jTnyclk8AYwqd/aQLzq6fMCSmZgCMnnqODsNsqWS1dfCq/ozg== X-Received: by 2002:a05:6000:1845:b0:374:c4e2:3ca7 with SMTP id ffacd0b85a97d-381f1865142mr19540743f8f.5.1731557646560; Wed, 13 Nov 2024 20:14:06 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.132.111]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3821ae311fbsm251936f8f.95.2024.11.13.20.14.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Nov 2024 20:14:06 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH 6/6] io_uring: enable IORING_ENTER_EXT_ARG_REG back Date: Thu, 14 Nov 2024 04:14:25 +0000 Message-ID: <25f44bd38403f2a87fa1defe2dac84b3b04b2659.1731556844.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 We can reenable IORING_ENTER_EXT_ARG_REG, which is now backed by parameter regions. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index c640b8a4ceee..8b4b866b7761 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -3195,7 +3195,16 @@ void __io_uring_cancel(bool cancel_all) static struct io_uring_reg_wait *io_get_ext_arg_reg(struct io_ring_ctx *ctx, const struct io_uring_getevents_arg __user *uarg) { - return ERR_PTR(-EFAULT); + unsigned long size = sizeof(struct io_uring_reg_wait); + unsigned long offset = (uintptr_t)uarg; + unsigned long end; + + /* also protects from NULL ->cq_wait_arg as the size would be 0 */ + if (unlikely(check_add_overflow(offset, size, &end) || + end >= ctx->cq_wait_size)) + return ERR_PTR(-EFAULT); + + return ctx->cq_wait_arg + offset; } static int io_validate_ext_arg(struct io_ring_ctx *ctx, unsigned flags,