From patchwork Thu Oct 24 17:07:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13849435 Received: from mail-io1-f53.google.com (mail-io1-f53.google.com [209.85.166.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 68B1F1F81A8 for ; Thu, 24 Oct 2024 17:08:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729789720; cv=none; b=MTXvL+Cn08fGHEWJ4382YJEzymPXRBkLT32x1YkjcOMpfGZMN2hQOc7CVvsqFnp7gWMtclIfpO8koBhx45T5WV1Gwr99pEgjDwFWEpOGQHfML9t4KG0Jb1zLZslTZlcHmpzbrqhHkbHFiaiM/1l3ZABgRzl/Ue9BAy9HpSD7P5k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729789720; c=relaxed/simple; bh=eAiFpPAvsaYS5b4WbjNYVCeDTzqN4Sf1Y8gYz+2RCoM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=aakZzwoGoUuxtizKojxyPKyHm3y4Cz1eALABndyJlJxTx0yZ+K9hws02cNiHutotwve9AZCELff+5MOX2glgZyebymQDC2Wh8+nJjTFv19/Joalfax9NSRFYSzimYLM4f/3mcp25pa/hQ3hZQ2HZKDZcKmGXs5Teph+jtyEYmAM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=jYyg2aSy; arc=none smtp.client-ip=209.85.166.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="jYyg2aSy" Received: by mail-io1-f53.google.com with SMTP id ca18e2360f4ac-83aff992087so49709339f.3 for ; Thu, 24 Oct 2024 10:08:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1729789716; x=1730394516; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QifmuGZt0bFh+Z2AH/dkkDK+RN42GNXgBlCpLjpKI7Y=; b=jYyg2aSyh48GDmBo5xEeEfpa05sC/wq3xwUtmrV9NC4cTAHS7pkfGgNsKXOlcfR3t9 RwthkKhpUYcqYhO9gtIOjj2wYhuipogU+dBoN2anRlRZWIWQIWcm3nPePXQA54xsQp37 Gd6QA2KEi21Uvn7agp5Qmxgnjp10HkJKDfqp0T+dbqWM/IdhSua8nsDkxDBQGlJ/cuVE bzEouAPaD5MILKwXBfbwoXqL988K3jOcmnx/ggiOzS6iTXLdtrB1g64fTZS+TR84rggw zjHlqYWRqNPs+LmU0qVSBuYzx94WdDy/Jsulf1K6n2n/NcdxdcWBKu+iJmHKArv48LFz Wwhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729789716; x=1730394516; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QifmuGZt0bFh+Z2AH/dkkDK+RN42GNXgBlCpLjpKI7Y=; b=Y6IU7DcjlaRyTrwBFKGbzLHrks8ExkJVk938CU98thRL6uJGSFEO+oVLmARoWfE3gb bOBGZcMfU9qw3KP4GCx+GdpvVKGMl1RETvS+LcB3rUOI+LDHchj6w4XouYO8kylqValu Gq8nO3hAVk4LYZO/mzI10WxjgniFt3xhhKqxTkPCeydyGQ32sn/rz7Qv23IVojZKyXMR PEQ7bc8au8AP3YAsrDZHtXeTkEQDna1eJi08TuOWEW98RNlOz+rQAXhVaSQl+yghXdHC BflE9kPnrU1+NAqQcLq//3JzN980nall02pnsURonOHXSEtld9O8hX0D4FjcIV4CAMuZ F1lg== X-Gm-Message-State: AOJu0YxzK9hJweHo3K4LAnrTg1fDjE5wKZklfsqLDHTK+4hobFIzr/a0 XSs2UCLJuMUi8NX5I6COUD9vG0x19Jjyp6zMycnh0/mm9H3qtsuiQ9qDtSfbmnvRRzR0u9D1ZTT E X-Google-Smtp-Source: AGHT+IFLilFyzrpOUsc/c+COE+9KN1Pu6NEY8OFB8KEHWAz+wTaKrnph6kHhYXf7iqa+a+NIbzOswA== X-Received: by 2002:a05:6e02:1ca3:b0:3a0:4250:165f with SMTP id e9e14a558f8ab-3a4de6e251dmr30189255ab.0.1729789715885; Thu, 24 Oct 2024 10:08:35 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id e9e14a558f8ab-3a400b63981sm31368045ab.67.2024.10.24.10.08.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Oct 2024 10:08:34 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: jannh@google.com, Jens Axboe Subject: [PATCH 1/4] io_uring: move max entry definition and ring sizing into header Date: Thu, 24 Oct 2024 11:07:36 -0600 Message-ID: <20241024170829.1266002-2-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241024170829.1266002-1-axboe@kernel.dk> References: <20241024170829.1266002-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 In preparation for needing this somewhere else, move the definitions for the maximum CQ and SQ ring size into io_uring.h. Make the rings_size() helper available as well, and have it take just the setup flags argument rather than the fill ring pointer. That's all that is needed. Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 14 ++++++-------- io_uring/io_uring.h | 5 +++++ 2 files changed, 11 insertions(+), 8 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 58b401900b41..6dea5242d666 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -105,9 +105,6 @@ #include "alloc_cache.h" #include "eventfd.h" -#define IORING_MAX_ENTRIES 32768 -#define IORING_MAX_CQ_ENTRIES (2 * IORING_MAX_ENTRIES) - #define SQE_COMMON_FLAGS (IOSQE_FIXED_FILE | IOSQE_IO_LINK | \ IOSQE_IO_HARDLINK | IOSQE_ASYNC) @@ -2667,8 +2664,8 @@ static void io_rings_free(struct io_ring_ctx *ctx) ctx->sq_sqes = NULL; } -static unsigned long rings_size(struct io_ring_ctx *ctx, unsigned int sq_entries, - unsigned int cq_entries, size_t *sq_offset) +unsigned long rings_size(unsigned int flags, unsigned int sq_entries, + unsigned int cq_entries, size_t *sq_offset) { struct io_rings *rings; size_t off, sq_array_size; @@ -2676,7 +2673,7 @@ static unsigned long rings_size(struct io_ring_ctx *ctx, unsigned int sq_entries off = struct_size(rings, cqes, cq_entries); if (off == SIZE_MAX) return SIZE_MAX; - if (ctx->flags & IORING_SETUP_CQE32) { + if (flags & IORING_SETUP_CQE32) { if (check_shl_overflow(off, 1, &off)) return SIZE_MAX; } @@ -2687,7 +2684,7 @@ static unsigned long rings_size(struct io_ring_ctx *ctx, unsigned int sq_entries return SIZE_MAX; #endif - if (ctx->flags & IORING_SETUP_NO_SQARRAY) { + if (flags & IORING_SETUP_NO_SQARRAY) { *sq_offset = SIZE_MAX; return off; } @@ -3434,7 +3431,8 @@ static __cold int io_allocate_scq_urings(struct io_ring_ctx *ctx, ctx->sq_entries = p->sq_entries; ctx->cq_entries = p->cq_entries; - size = rings_size(ctx, p->sq_entries, p->cq_entries, &sq_array_offset); + size = rings_size(ctx->flags, p->sq_entries, p->cq_entries, + &sq_array_offset); if (size == SIZE_MAX) return -EOVERFLOW; diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 9cd9a127e9ed..4a471a810f02 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -65,6 +65,11 @@ static inline bool io_should_wake(struct io_wait_queue *iowq) return dist >= 0 || atomic_read(&ctx->cq_timeouts) != iowq->nr_timeouts; } +#define IORING_MAX_ENTRIES 32768 +#define IORING_MAX_CQ_ENTRIES (2 * IORING_MAX_ENTRIES) + +unsigned long rings_size(unsigned int flags, unsigned int sq_entries, + unsigned int cq_entries, size_t *sq_offset); bool io_cqe_cache_refill(struct io_ring_ctx *ctx, bool overflow); int io_run_task_work_sig(struct io_ring_ctx *ctx); void io_req_defer_failed(struct io_kiocb *req, s32 res); From patchwork Thu Oct 24 17:07:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13849436 Received: from mail-io1-f49.google.com (mail-io1-f49.google.com [209.85.166.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AC38C19DF7A for ; Thu, 24 Oct 2024 17:08:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729789721; cv=none; b=BW5wUqIgb/nrkFaHpiByUCNRSfBiE19EPbT1WEAE8eKtMtT+M+iL/BPEBb8JVJm+o5dLybTQEt1vW6W4n77UQNONvg0B8h5JJmp2GSV5fN9bbn+dBzpVtlMeoWwjprvaBVxFV/IfMCoGv2DeMU9hsjb+HfKAR3RF4DmtT+HnQls= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729789721; c=relaxed/simple; bh=j3VmYIUYlyBTGlqvUIMc0JUgVTjmWoaF5EMC9Dn7Zk0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=GZHcxiklGp7VbA90uU0W+rFs46Nu+EGwSxizSF8CPR0QcW0paMep5ey29H2EiXVFnlqfMYUUyaGlsAQToNIV+eTfM7oKUUd9kghLumTj2+7+lkMzPalLqW6n0uA/774qKtcFdVmtlRhrNfNwIcCT6evb0c1G9wGmMf0XrN99Aj8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=avDo21cG; arc=none smtp.client-ip=209.85.166.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="avDo21cG" Received: by mail-io1-f49.google.com with SMTP id ca18e2360f4ac-83aad99d6bfso44974339f.0 for ; Thu, 24 Oct 2024 10:08:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1729789717; x=1730394517; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5+0vcL80oErDAtwoUFPo8e5XBLS2uO6xNRw7O3/obNg=; b=avDo21cG8vpvWrsdIpdqeZaXBymApLXUNrRsL+ZJ2cMQLj5bNL9fsO1XbzopZZdH5D +HpvX5TUQQTErrZT1+M1ulKcn25VIxd+yQNNXCWv+Dba/wdmHCkOnWIUy3qOE9EKPCJN LuzF8Q7IyRxMeI4UgUUCC6ce1Q2ptc+SgZvdxHYPDlTFYvH5ih/WD68WO8S3OR+rNpng WdJFxvPyTqtNG4gjtw8uyhYGDUkuQEF1UVPmGEGodOfYjdKfFlZ7Fnd6A1Vems7GBLIY uZumlNWGZQ+VtfJ1O9nG3zNS6abhGMDbUK5uNIy0dU5GzjNcvkXxDjwJCOJd2t+1rlcH 78SQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729789717; x=1730394517; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5+0vcL80oErDAtwoUFPo8e5XBLS2uO6xNRw7O3/obNg=; b=bq/UXNyeQLoin3UZyd22//pLMXTx652fG/3s10LW5hA0OD0BrBTwJF9hXtZ3qYYySp zpVAvBjyod5ulhKbMv9adg1CUcfhSqJ74fOLfg+SIO6+ykR/LIrKL/SIaJNZro7bJEgK BrAaqvUBrQI7LpgX3nM8oQQ0DoRCJ87ZuB+rij+ElewwQCYsgnHsgdexSLMrbByUhmYw f5qLnn8JmCciTgUeMWloITS10hR7VRgitHS80bvPpH7TWee6vUYTJ2zk8NGLggS7WKpn 0/oP0dEJUndvPBFAtfJq7o5B0cOfJueUPqdce3uomy3lPDFQ2B57EWaZ0hac2YV+8ODT JOhw== X-Gm-Message-State: AOJu0YwXc1CgaNnZLsu27UFsVbGO3gKdLHd1lFFSPe79zsG+77dcUE1l Yn1HyFMOHEg+afiCiwb6KzlUlY1cPMszIiTjmEyQh7k/MwQa72LRZzPnIikqA5R0av83wiq9NIi f X-Google-Smtp-Source: AGHT+IG70rOEhgDa+XfsRbOn+lxE4A8i/O3PO1R7G4Kg562ovTYd1Y8hXCtyBb19X6WB0CrhG1muVg== X-Received: by 2002:a05:6e02:214d:b0:3a0:9fa5:8f1 with SMTP id e9e14a558f8ab-3a4d59df6d6mr81512095ab.24.1729789717134; Thu, 24 Oct 2024 10:08:37 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id e9e14a558f8ab-3a400b63981sm31368045ab.67.2024.10.24.10.08.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Oct 2024 10:08:36 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: jannh@google.com, Jens Axboe Subject: [PATCH 2/4] io_uring: abstract out a bit of the ring filling logic Date: Thu, 24 Oct 2024 11:07:37 -0600 Message-ID: <20241024170829.1266002-3-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241024170829.1266002-1-axboe@kernel.dk> References: <20241024170829.1266002-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Abstract out a io_uring_fill_params() helper, which fills out the necessary bits of struct io_uring_params. Add it to io_uring.h as well, in preparation for having another internal user of it. Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 70 ++++++++++++++++++++++++++------------------- io_uring/io_uring.h | 1 + 2 files changed, 41 insertions(+), 30 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 6dea5242d666..b5974bdad48b 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -3498,14 +3498,8 @@ static struct file *io_uring_get_file(struct io_ring_ctx *ctx) O_RDWR | O_CLOEXEC, NULL); } -static __cold int io_uring_create(unsigned entries, struct io_uring_params *p, - struct io_uring_params __user *params) +int io_uring_fill_params(unsigned entries, struct io_uring_params *p) { - struct io_ring_ctx *ctx; - struct io_uring_task *tctx; - struct file *file; - int ret; - if (!entries) return -EINVAL; if (entries > IORING_MAX_ENTRIES) { @@ -3547,6 +3541,42 @@ static __cold int io_uring_create(unsigned entries, struct io_uring_params *p, p->cq_entries = 2 * p->sq_entries; } + p->sq_off.head = offsetof(struct io_rings, sq.head); + p->sq_off.tail = offsetof(struct io_rings, sq.tail); + p->sq_off.ring_mask = offsetof(struct io_rings, sq_ring_mask); + p->sq_off.ring_entries = offsetof(struct io_rings, sq_ring_entries); + p->sq_off.flags = offsetof(struct io_rings, sq_flags); + p->sq_off.dropped = offsetof(struct io_rings, sq_dropped); + p->sq_off.resv1 = 0; + if (!(p->flags & IORING_SETUP_NO_MMAP)) + p->sq_off.user_addr = 0; + + p->cq_off.head = offsetof(struct io_rings, cq.head); + p->cq_off.tail = offsetof(struct io_rings, cq.tail); + p->cq_off.ring_mask = offsetof(struct io_rings, cq_ring_mask); + p->cq_off.ring_entries = offsetof(struct io_rings, cq_ring_entries); + p->cq_off.overflow = offsetof(struct io_rings, cq_overflow); + p->cq_off.cqes = offsetof(struct io_rings, cqes); + p->cq_off.flags = offsetof(struct io_rings, cq_flags); + p->cq_off.resv1 = 0; + if (!(p->flags & IORING_SETUP_NO_MMAP)) + p->cq_off.user_addr = 0; + + return 0; +} + +static __cold int io_uring_create(unsigned entries, struct io_uring_params *p, + struct io_uring_params __user *params) +{ + struct io_ring_ctx *ctx; + struct io_uring_task *tctx; + struct file *file; + int ret; + + ret = io_uring_fill_params(entries, p); + if (unlikely(ret)) + return ret; + ctx = io_ring_ctx_alloc(p); if (!ctx) return -ENOMEM; @@ -3630,6 +3660,9 @@ static __cold int io_uring_create(unsigned entries, struct io_uring_params *p, if (ret) goto err; + if (!(p->flags & IORING_SETUP_NO_SQARRAY)) + p->sq_off.array = (char *)ctx->sq_array - (char *)ctx->rings; + ret = io_sq_offload_create(ctx, p); if (ret) goto err; @@ -3638,29 +3671,6 @@ static __cold int io_uring_create(unsigned entries, struct io_uring_params *p, if (ret) goto err; - p->sq_off.head = offsetof(struct io_rings, sq.head); - p->sq_off.tail = offsetof(struct io_rings, sq.tail); - p->sq_off.ring_mask = offsetof(struct io_rings, sq_ring_mask); - p->sq_off.ring_entries = offsetof(struct io_rings, sq_ring_entries); - p->sq_off.flags = offsetof(struct io_rings, sq_flags); - p->sq_off.dropped = offsetof(struct io_rings, sq_dropped); - if (!(ctx->flags & IORING_SETUP_NO_SQARRAY)) - p->sq_off.array = (char *)ctx->sq_array - (char *)ctx->rings; - p->sq_off.resv1 = 0; - if (!(ctx->flags & IORING_SETUP_NO_MMAP)) - p->sq_off.user_addr = 0; - - p->cq_off.head = offsetof(struct io_rings, cq.head); - p->cq_off.tail = offsetof(struct io_rings, cq.tail); - p->cq_off.ring_mask = offsetof(struct io_rings, cq_ring_mask); - p->cq_off.ring_entries = offsetof(struct io_rings, cq_ring_entries); - p->cq_off.overflow = offsetof(struct io_rings, cq_overflow); - p->cq_off.cqes = offsetof(struct io_rings, cqes); - p->cq_off.flags = offsetof(struct io_rings, cq_flags); - p->cq_off.resv1 = 0; - if (!(ctx->flags & IORING_SETUP_NO_MMAP)) - p->cq_off.user_addr = 0; - p->features = IORING_FEAT_SINGLE_MMAP | IORING_FEAT_NODROP | IORING_FEAT_SUBMIT_STABLE | IORING_FEAT_RW_CUR_POS | IORING_FEAT_CUR_PERSONALITY | IORING_FEAT_FAST_POLL | diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 4a471a810f02..e3e6cb14de5d 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -70,6 +70,7 @@ static inline bool io_should_wake(struct io_wait_queue *iowq) unsigned long rings_size(unsigned int flags, unsigned int sq_entries, unsigned int cq_entries, size_t *sq_offset); +int io_uring_fill_params(unsigned entries, struct io_uring_params *p); bool io_cqe_cache_refill(struct io_ring_ctx *ctx, bool overflow); int io_run_task_work_sig(struct io_ring_ctx *ctx); void io_req_defer_failed(struct io_kiocb *req, s32 res); From patchwork Thu Oct 24 17:07:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13849437 Received: from mail-il1-f179.google.com (mail-il1-f179.google.com [209.85.166.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C213B1F9ECC for ; Thu, 24 Oct 2024 17:08:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729789723; cv=none; b=eCfs0M2aeWbYo6ynPZSrsKrWtrBqrlZmIHSPtkRQY8/fxy+YikuY5wTEPAT+/xmKjysyi/pJ2X+cxOBDyPYGYPHKtItcK/6QDXGLN8t/bMavFIXEhFBlNKBeihp7bqfSTe32u99JIjo628V3yaOp7Dm/CuOWvy2887eXqxx9TkY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729789723; c=relaxed/simple; bh=Hc8LWFF+BnTAwlQsvqEqcz1icMxeGM4aq1XGEcW9+IY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=L9KF+LXGVCeiZVk1BM1pi5OdGmhlPU9EcuHF205e2BJXzQJf+F2a88TbNjeRhAt4I/eN1euoE2/mMhGYalSxH9XloQkj6tCH3Ur+HwKe2PFrBMb+H0r4FhGn2n5kRDxl0GSqAIJMinGIA7DXlC1vBCCtfTUj7hDXKKRc59SqIk0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=oEC8Rvs/; arc=none smtp.client-ip=209.85.166.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="oEC8Rvs/" Received: by mail-il1-f179.google.com with SMTP id e9e14a558f8ab-3a3a6abcbf1so4846385ab.2 for ; Thu, 24 Oct 2024 10:08:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1729789719; x=1730394519; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=olc4hQqg9C/G+s6Y4gqlaKc2EU5xK/tf7mC8ufS+JaY=; b=oEC8Rvs/R7cfkdeR4LAE+UqhfwNSjAsUDW6W9CHSs6Ast0t1Lq71x5U5ivoqNkpTf3 V/fZygIY3j9ixboS/TKeN+1CFStGK9omS64T+KsXSWFPZPQBLQmYEQEaCaCAJ7aVpzeO ChQJ5T8KInBIyS1Cyf3K4CXhtXPSR7tpiy4yIjZap6fCoqOdptwEbi3vCLSiYNi86zel 7a5ZlMeoDt2tlJJLVkTBDbLgokana/smhm4xQ1ktGLxy1rrZR3g30VbSU9k2SjkMEo9c PPnIKYFasjwVSv4n670Uu18c+r4Hhy6/H7GghhrsGUe5akR3zB07ADvUCHxbTVpdC9yG sjxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729789719; x=1730394519; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=olc4hQqg9C/G+s6Y4gqlaKc2EU5xK/tf7mC8ufS+JaY=; b=wUzC1s4AHHlRILbeCUcxD5N1b6JS+vhHMHsS3lRK66sT5tSklhNsUO0quPwdP/J0U2 pivZuICaO+Vh5MDR62951DzhX1Eh1MDbXtG0JqBT34IYAqHwr5rECBEZXQhvPk8Gvnub vE8WjPKWKa7Aoms6pQmvqKU8eFoh3YUY75XLGfa01accAS0ffpokbOxDE517r0k2UsJT pSQUgeHbfqd3vIMe7sLQH4nFgaGQKzmvN49wNhPplYoB6BZ+3zLJwL1/xjfHiCiJMzHR M7NrQssAoqaiGmUBPjE82CwxymXQekcqVAhEJrrsanlj5UgFAPY1fpAP6tOn0hvb7ToT wFSg== X-Gm-Message-State: AOJu0Ywxbus8IWcp6Q/WSaEtc124HK23DjLXuOHalhMmDLhxufic1WSf Q3xM6ipnhTnyfKCRJLXZsfOYpioYRJ/PaiBpf94JG0n2f72Km/EA7BOIu3wnjkz3hlvAuehGd4P T X-Google-Smtp-Source: AGHT+IEvKaCJSldSQ/mwldl75w3PUJIuycZLqIFBzWu8kszemQckISZQ70u3Zh1UB8XuL46NnXrhQg== X-Received: by 2002:a05:6e02:1448:b0:3a0:5642:c78 with SMTP id e9e14a558f8ab-3a4d599d224mr76969665ab.15.1729789719148; Thu, 24 Oct 2024 10:08:39 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id e9e14a558f8ab-3a400b63981sm31368045ab.67.2024.10.24.10.08.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Oct 2024 10:08:37 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: jannh@google.com, Jens Axboe Subject: [PATCH 3/4] io_uring/memmap: explicitly return -EFAULT for mmap on NULL rings Date: Thu, 24 Oct 2024 11:07:38 -0600 Message-ID: <20241024170829.1266002-4-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241024170829.1266002-1-axboe@kernel.dk> References: <20241024170829.1266002-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 The later mapping will actually check this too, but in terms of code clarify, explicitly check for whether or not the rings and sqes are valid during validation. That makes it explicit that if they are non-NULL, they are valid and can get mapped. Signed-off-by: Jens Axboe --- io_uring/memmap.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/io_uring/memmap.c b/io_uring/memmap.c index a0f32a255fd1..d614824e17bd 100644 --- a/io_uring/memmap.c +++ b/io_uring/memmap.c @@ -204,11 +204,15 @@ static void *io_uring_validate_mmap_request(struct file *file, loff_t pgoff, /* Don't allow mmap if the ring was setup without it */ if (ctx->flags & IORING_SETUP_NO_MMAP) return ERR_PTR(-EINVAL); + if (!ctx->rings) + return ERR_PTR(-EFAULT); return ctx->rings; case IORING_OFF_SQES: /* Don't allow mmap if the ring was setup without it */ if (ctx->flags & IORING_SETUP_NO_MMAP) return ERR_PTR(-EINVAL); + if (!ctx->sq_sqes) + return ERR_PTR(-EFAULT); return ctx->sq_sqes; case IORING_OFF_PBUF_RING: { struct io_buffer_list *bl; From patchwork Thu Oct 24 17:07:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13849438 Received: from mail-il1-f177.google.com (mail-il1-f177.google.com [209.85.166.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F2CF319DF7A for ; Thu, 24 Oct 2024 17:08:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729789725; cv=none; b=bRONeApwed5h44U/d23LkINzqwVB9CDYJelQIgTLR5aecBdfs6ZXzzjrU7PgEsSE9Rn9/4YsWTq5NT7GmvcxSt+Qxy8fZcLFkAnZXKpPibYwoqXwJruainHonYIEhXvCyQ3FQe2CXqmH2dROI2vLKs1Oi6C3uenSHHeQrELp7B8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729789725; c=relaxed/simple; bh=koXgoU2Bb8svwTSLtxM/QVGMSZAaJ3IvVzIJmcPAGwQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=IduPbwF5lj6r4ZO5ofpl6E0FaDJfmMpvf7NroSYewO7xi4BsomvtWwfjYcjKvgE9ooLmVHwUIVtaGo8CLf5/KKHnG/0S7mjPv+JMNY9aN9UoWwzRJnS0Cxefr/VF7tFYilHEsHSBX+Bq5RWbcHb5BnG8vR5VoOnhh1pSuZRlZX4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=cLDksPDX; arc=none smtp.client-ip=209.85.166.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="cLDksPDX" Received: by mail-il1-f177.google.com with SMTP id e9e14a558f8ab-3a4d1633df9so4668495ab.2 for ; Thu, 24 Oct 2024 10:08:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1729789721; x=1730394521; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=NaQjZr9Vw/7x5N6M9zv4XaxVTX09HoMSr/vVr4eXl+g=; b=cLDksPDXQQdBVvMW9+4Q6/+6eK6A+GZIYz4RnnN9tSUCU5QiGPJkqN46e8DKmK66/G fSwpAZ6nAvaqJKDJea8/8VfCN1MAoJ0kFEuDlqHGtQ5lawu7lZWD3DF5fHXGVS8fPvwE ZZFX9k1R5N4OKDMZoK7vRulNOs6l3H7HGLxPxC0pfN/7+8RVuJCRDkQdp3FYNq0AiNG2 FjPrYfwisQyXeQj7G7Wlww97YiOAMq4bzKcwIlruUO1upeSvAnsBkN05/3PJTPcpbOPi mxVHiFtUHiwLjJq3k9u6CWUd9QqwQjDwDWn82dAM+pfH0hOGQ8yl7dQWytHU+HFtwvL1 QziA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729789721; x=1730394521; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NaQjZr9Vw/7x5N6M9zv4XaxVTX09HoMSr/vVr4eXl+g=; b=Dib+YvWudZ0Bt2lJ/+ts2sP65sbvt7HyYkjeNRkERa96dQwR0oVB/yaVg8lHLHojB0 mSC6mVspcdvcIIFXOPqN01nF3eoI1W+6vU5UbMxQTGJtTgPsDyC8oZI8YcEj0xcP9cAV yND1JU5ZeTcD/S1b9W5fhzsb/9f3nioGNYklcwsJkwu8qYTh5VSWUptGj5HthmKDc5+X Tf3Q2BnXs/JWJ5s4WQCrF7wq6Zk3riNjv1QRHWHAGb+KSuBAAlJofMdEQfjEIfwcx3ZP OhGZEaoyF6eaXqpUIkzGYZIGeAFO8OHt+7EG7duwo+RN5P+dlwLzA9tn3YiXOINwZiAT 5FPA== X-Gm-Message-State: AOJu0YzUy1yNeDdOsr8+YGXiHrpQ/X2M81qWowQqk1NK865fhALrrsQw g88/6pRllDcIC3nQqtJ/BPMVHKa78cnljOsxfso2c3ow9AoWxqxK3CzPfym7iNZj7G2x75zCWlD C X-Google-Smtp-Source: AGHT+IHMttldOxIdCJTVAvDC2yLkmpbqf+mxtExBalXxVWLIY+DbKYHOPMk8QMNTEbxLcMSI2ZjJIw== X-Received: by 2002:a05:6e02:1aa8:b0:3a0:92e5:af68 with SMTP id e9e14a558f8ab-3a4d597b60emr80843095ab.15.1729789721424; Thu, 24 Oct 2024 10:08:41 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id e9e14a558f8ab-3a400b63981sm31368045ab.67.2024.10.24.10.08.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Oct 2024 10:08:39 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: jannh@google.com, Jens Axboe Subject: [PATCH 4/4] io_uring/register: add IORING_REGISTER_RESIZE_RINGS Date: Thu, 24 Oct 2024 11:07:39 -0600 Message-ID: <20241024170829.1266002-5-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241024170829.1266002-1-axboe@kernel.dk> References: <20241024170829.1266002-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Once a ring has been created, the size of the CQ and SQ rings are fixed. Usually this isn't a problem on the SQ ring side, as it merely controls the available number of requests that can be submitted in a single system call, and there's rarely a need to change that. For the CQ ring, it's a different story. For most efficient use of io_uring, it's important that the CQ ring never overflows. This means that applications must size it for the worst case scenario, which can be wasteful. Add IORING_REGISTER_RESIZE_RINGS, which allows an application to resize the existing rings. It takes a struct io_uring_params argument, the same one which is used to setup the ring initially, and resizes rings according to the sizes given. Certain properties are always inherited from the original ring setup, like SQE128/CQE32 and other setup options. The implementation only allows flag associated with how the CQ ring is sized and clamped. Existing unconsumed SQE and CQE entries are copied as part of the process. If either the SQ or CQ resized destination ring cannot hold the entries already present in the source rings, then the operation is failed with -EOVERFLOW. Any register op holds ->uring_lock, which prevents new submissions, and the internal mapping holds the completion lock as well across moving CQ ring state. Signed-off-by: Jens Axboe --- include/uapi/linux/io_uring.h | 3 + io_uring/register.c | 214 ++++++++++++++++++++++++++++++++++ 2 files changed, 217 insertions(+) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 86cb385fe0b5..c4737892c7cd 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -615,6 +615,9 @@ enum io_uring_register_op { /* send MSG_RING without having a ring */ IORING_REGISTER_SEND_MSG_RING = 31, + /* resize CQ ring */ + IORING_REGISTER_RESIZE_RINGS = 33, + /* this goes last */ IORING_REGISTER_LAST, diff --git a/io_uring/register.c b/io_uring/register.c index 52b2f9b74af8..911242f63704 100644 --- a/io_uring/register.c +++ b/io_uring/register.c @@ -29,6 +29,7 @@ #include "napi.h" #include "eventfd.h" #include "msg_ring.h" +#include "memmap.h" #define IORING_MAX_RESTRICTIONS (IORING_RESTRICTION_LAST + \ IORING_REGISTER_LAST + IORING_OP_LAST) @@ -361,6 +362,213 @@ static int io_register_clock(struct io_ring_ctx *ctx, return 0; } +/* + * State to maintain until we can swap. Both new and old state, used for + * either mapping or freeing. + */ +struct io_ring_ctx_rings { + unsigned short n_ring_pages; + unsigned short n_sqe_pages; + struct page **ring_pages; + struct page **sqe_pages; + struct io_uring_sqe *sq_sqes; + struct io_rings *rings; +}; + +static void io_register_free_rings(struct io_uring_params *p, + struct io_ring_ctx_rings *r) +{ + if (!(p->flags & IORING_SETUP_NO_MMAP)) { + io_pages_unmap(r->rings, &r->ring_pages, &r->n_ring_pages, + true); + io_pages_unmap(r->sq_sqes, &r->sqe_pages, &r->n_sqe_pages, + true); + } else { + io_pages_free(&r->ring_pages, r->n_ring_pages); + io_pages_free(&r->sqe_pages, r->n_sqe_pages); + vunmap(r->rings); + vunmap(r->sq_sqes); + } +} + +#define swap_old(ctx, o, n, field) \ + do { \ + (o).field = (ctx)->field; \ + (ctx)->field = (n).field; \ + } while (0) + +#define RESIZE_FLAGS (IORING_SETUP_CQSIZE | IORING_SETUP_CLAMP) +#define COPY_FLAGS (IORING_SETUP_NO_SQARRAY | IORING_SETUP_SQE128 | \ + IORING_SETUP_CQE32 | IORING_SETUP_NO_MMAP) + +static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg) +{ + struct io_ring_ctx_rings o = { }, n = { }, *to_free = NULL; + size_t size, sq_array_offset; + struct io_uring_params p; + unsigned i, tail; + void *ptr; + int ret; + + /* for single issuer, must be owner resizing */ + if (ctx->flags & IORING_SETUP_SINGLE_ISSUER && + current != ctx->submitter_task) + return -EEXIST; + if (copy_from_user(&p, arg, sizeof(p))) + return -EFAULT; + if (p.flags & ~RESIZE_FLAGS) + return -EINVAL; + + /* properties that are always inherited */ + p.flags |= (ctx->flags & COPY_FLAGS); + + ret = io_uring_fill_params(p.sq_entries, &p); + if (unlikely(ret)) + return ret; + + /* nothing to do, but copy params back */ + if (p.sq_entries == ctx->sq_entries && p.cq_entries == ctx->cq_entries) { + if (copy_to_user(arg, &p, sizeof(p))) + return -EFAULT; + return 0; + } + + size = rings_size(p.flags, p.sq_entries, p.cq_entries, + &sq_array_offset); + if (size == SIZE_MAX) + return -EOVERFLOW; + + if (!(p.flags & IORING_SETUP_NO_MMAP)) + n.rings = io_pages_map(&n.ring_pages, &n.n_ring_pages, size); + else + n.rings = __io_uaddr_map(&n.ring_pages, &n.n_ring_pages, + p.cq_off.user_addr, size); + if (IS_ERR(n.rings)) + return PTR_ERR(n.rings); + + n.rings->sq_ring_mask = p.sq_entries - 1; + n.rings->cq_ring_mask = p.cq_entries - 1; + n.rings->sq_ring_entries = p.sq_entries; + n.rings->cq_ring_entries = p.cq_entries; + + if (copy_to_user(arg, &p, sizeof(p))) { + io_register_free_rings(&p, &n); + return -EFAULT; + } + + if (p.flags & IORING_SETUP_SQE128) + size = array_size(2 * sizeof(struct io_uring_sqe), p.sq_entries); + else + size = array_size(sizeof(struct io_uring_sqe), p.sq_entries); + if (size == SIZE_MAX) { + io_register_free_rings(&p, &n); + return -EOVERFLOW; + } + + if (!(p.flags & IORING_SETUP_NO_MMAP)) + ptr = io_pages_map(&n.sqe_pages, &n.n_sqe_pages, size); + else + ptr = __io_uaddr_map(&n.sqe_pages, &n.n_sqe_pages, + p.sq_off.user_addr, + size); + if (IS_ERR(ptr)) { + io_register_free_rings(&p, &n); + return PTR_ERR(ptr); + } + + /* + * If using SQPOLL, park the thread + */ + if (ctx->sq_data) { + mutex_unlock(&ctx->uring_lock); + io_sq_thread_park(ctx->sq_data); + mutex_lock(&ctx->uring_lock); + } + + /* + * We'll do the swap. Clear out existing mappings to prevent mmap + * from seeing them, as we'll unmap them. Any attempt to mmap existing + * rings beyond this point will fail. Not that it could proceed at this + * point anyway, as we'll hold the mmap_sem until we've done the swap. + * Likewise, hold the completion * lock over the duration of the actual + * swap. + */ + mmap_write_lock(current->mm); + spin_lock(&ctx->completion_lock); + o.rings = ctx->rings; + ctx->rings = NULL; + o.sq_sqes = ctx->sq_sqes; + ctx->sq_sqes = NULL; + + /* + * Now copy SQ and CQ entries, if any. If either of the destination + * rings can't hold what is already there, then fail the operation. + */ + n.sq_sqes = ptr; + tail = o.rings->sq.tail; + if (tail - o.rings->sq.head > p.sq_entries) + goto overflow; + for (i = o.rings->sq.head; i < tail; i++) { + unsigned src_head = i & (ctx->sq_entries - 1); + unsigned dst_head = i & n.rings->sq_ring_mask; + + n.sq_sqes[dst_head] = o.sq_sqes[src_head]; + } + n.rings->sq.head = o.rings->sq.head; + n.rings->sq.tail = o.rings->sq.tail; + + tail = o.rings->cq.tail; + if (tail - o.rings->cq.head > p.cq_entries) { +overflow: + /* restore old rings, and return -EOVERFLOW via cleanup path */ + ctx->rings = o.rings; + ctx->sq_sqes = o.sq_sqes; + to_free = &n; + ret = -EOVERFLOW; + goto out; + } + for (i = o.rings->cq.head; i < tail; i++) { + unsigned src_head = i & (ctx->cq_entries - 1); + unsigned dst_head = i & n.rings->cq_ring_mask; + + n.rings->cqes[dst_head] = o.rings->cqes[src_head]; + } + n.rings->cq.head = o.rings->cq.head; + n.rings->cq.tail = o.rings->cq.tail; + /* invalidate cached cqe refill */ + ctx->cqe_cached = ctx->cqe_sentinel = NULL; + + n.rings->sq_dropped = o.rings->sq_dropped; + n.rings->sq_flags = o.rings->sq_flags; + n.rings->cq_flags = o.rings->cq_flags; + n.rings->cq_overflow = o.rings->cq_overflow; + + /* all done, store old pointers and assign new ones */ + if (!(ctx->flags & IORING_SETUP_NO_SQARRAY)) + ctx->sq_array = (u32 *)((char *)n.rings + sq_array_offset); + + ctx->sq_entries = p.sq_entries; + ctx->cq_entries = p.cq_entries; + + ctx->rings = n.rings; + ctx->sq_sqes = n.sq_sqes; + swap_old(ctx, o, n, n_ring_pages); + swap_old(ctx, o, n, n_sqe_pages); + swap_old(ctx, o, n, ring_pages); + swap_old(ctx, o, n, sqe_pages); + to_free = &o; + ret = 0; +out: + spin_unlock(&ctx->completion_lock); + mmap_write_unlock(current->mm); + io_register_free_rings(&p, to_free); + + if (ctx->sq_data) + io_sq_thread_unpark(ctx->sq_data); + + return ret; +} + static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode, void __user *arg, unsigned nr_args) __releases(ctx->uring_lock) @@ -549,6 +757,12 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode, break; ret = io_register_clone_buffers(ctx, arg); break; + case IORING_REGISTER_RESIZE_RINGS: + ret = -EINVAL; + if (!arg || nr_args != 1) + break; + ret = io_register_resize_rings(ctx, arg); + break; default: ret = -EINVAL; break;